2019-06-13 10:21:22 UTC - Ivan Kelly: This looks like a bug I thought was
fixed. Have you set both s3 region and s3 endpoint? Only one of them should be
required, but the bug means that if both aren't set there's a NPE during
offload so it ends up in a hung state (@Alexandre DUVAL)
----
2019-06-13 10:22:55 UTC - Ivan Kelly: For all topics under the namespace.
However, the offload will only trigger when a current segment is rolled, which
may be why you're not seeing on existing topics that are above the threshold.
----
2019-06-13 11:00:56 UTC - Alexandre DUVAL: This is fixed, I just updated
brokers to master.
----
2019-06-13 11:01:26 UTC - Alexandre DUVAL: Hi, what's a segment?
----
2019-06-13 12:02:24 UTC - Ivan Kelly: the backlog for a topic is broken into
segments. by default a new segment is created every 50k messages
----
2019-06-13 12:17:28 UTC - Alexandre DUVAL: ok
----
2019-06-13 15:39:47 UTC - Steve Kim: I want to reduce storage cost of message
retention. Does pulsar support compression of messages on disk? Or is this the
responsibility of bookkeeper (e.g.
<https://github.com/apache/bookkeeper/issues/627>)?
----
2019-06-13 15:42:15 UTC - Ryan Samo: Hey guys, I’m trying to start a
functions-worker but keep getting this error and I’m not sure what might be
missing in the yaml file. I do see that we have nulls in the worker ID and we
are using TLS so the worker Port is null.
Error:
11:23:52.158 [main] ERROR
org.apache.pulsar.functions.worker.FunctionWorkerStarter - Failed to start
function worker
java.lang.NullPointerException: null
at
org.apache.pulsar.functions.worker.rest.WorkerServer.init(WorkerServer.java:93)
~[org.apache.pulsar-pulsar-functions-worker-2.3.2.jar:2.3.2]
at
org.apache.pulsar.functions.worker.rest.WorkerServer.<init>(WorkerServer.java:80)
~[org.apache.pulsar-pulsar-functions-worker-2.3.2.jar:2.3.2]
at org.apache.pulsar.functions.worker.Worker.start(Worker.java:52)
~[org.apache.pulsar-pulsar-functions-worker-2.3.2.jar:2.3.2]
at
org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:67)
[org.apache.pulsar-pulsar-functions-worker-2.3.2.jar:2.3.2]
Any ideas?
----
2019-06-13 15:58:19 UTC - David Kjerrumgaard: @Ryan Samo Can you elaborate on
`we are using TLS` ? Does that mean the broker? Also where are you seeing
`nulls in the worker ID`? Is there additional information around that?
----
2019-06-13 16:01:14 UTC - Ryan Samo: So I figured out that the workerPort was
null. I fixed that and the worker comes online. We are using TLS on the
brokers, this functions worker is running outside the brokers. I also see that
the workPortTls is not used though, only the http workerPort. Is there a way to
only use the TLS port on workers?
----
2019-06-13 16:05:06 UTC - David Kjerrumgaard: Have you performed the steps
listed here to enable TLS for the worker?
<https://pulsar.apache.org/docs/en/functions-worker/#enable-tls-transport-encryption>
----
2019-06-13 16:05:56 UTC - Ryan Samo: yeah I see now what’s going on... so the
non tls port has to be open and then the code adds the tlsport as well...
+1 : David Kjerrumgaard
----
2019-06-13 16:06:29 UTC - Ryan Samo: It’s working, we will just have to block
the non tls port on our end I guess
----
2019-06-13 16:18:25 UTC - Ivan Kelly: pulsar supported end-to-end compression
from producer to consumer, so compression of messages on disk isn't needed
----
2019-06-13 16:19:15 UTC - Ivan Kelly:
<http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ProducerBuilder.html#compressionType-org.apache.pulsar.client.api.CompressionType->
----
2019-06-13 16:23:04 UTC - Grant Wu: @Sijie Guo Just saw
<https://github.com/apache/pulsar/issues/3715#issuecomment-501682320>
----
2019-06-13 16:23:23 UTC - Grant Wu: Does this mean that the issue should not
appear on 2.2.1? Because have been encountering it on 2.2.1
----
2019-06-13 17:00:19 UTC - Ryan Samo: Can you run functions in cluster mode if
the workers are not on the brokers?
----
2019-06-13 17:04:53 UTC - David Kjerrumgaard: You can run functions-worker as a
separate process on separate (non-broker) machines. But I am not sure if that
constitutes "cluster" mode.
----
2019-06-13 17:05:35 UTC - Ryan Samo: Cool, yeah I see that it works if we
deploy in localrun mode, but complains in create mode
----
2019-06-13 17:08:11 UTC - David Kjerrumgaard:
<https://pulsar.apache.org/docs/en/functions-worker/#run-functions-worker-separately>
----
2019-06-13 17:08:46 UTC - David Kjerrumgaard: "In this mode, make sure
functionsWorkerEnabled is set to false, so you won't start functions-worker
with brokers by mistake."
----
2019-06-13 17:09:07 UTC - David Kjerrumgaard: what "complaints" do you see?
----
2019-06-13 17:13:00 UTC - Felix Aballi: @Felix Aballi has joined the channel
----
2019-06-13 17:54:21 UTC - Steve Kim: Sorry, I don't understand your answer. Are
you saying that the messages are stored on disk in compressed form when the
producer enables compression?
----
2019-06-13 18:41:36 UTC - Ryan Samo: So I just started up the test exclamation
function to give it a quick try and it complains “Trying to subscribe with
incompatible schema”. The input topic and output topics are just vanilla byte[]
so am I supposed to pass in a parameter to tell the function that?
----
2019-06-13 18:57:05 UTC - David Kjerrumgaard: @Ryan Samo Which Pulsar version
are you running?
----
2019-06-13 18:57:13 UTC - Ryan Samo: 2.3.2
----
2019-06-13 18:57:16 UTC - David Kjerrumgaard: Are the client and broker the
same version?
----
2019-06-13 18:57:54 UTC - Ryan Samo: Yes, using the functions-worker from the
2.3.2 binaries
----
2019-06-13 18:58:04 UTC - Ryan Samo: Brokers are 2.3.2 as well
----
2019-06-13 19:00:17 UTC - David Kjerrumgaard: So these were pre-existing topics
with no associated schema?
----
2019-06-13 19:00:49 UTC - Ryan Samo: Yes, and then I tried to create new ones
just now. Same issue :(
----
2019-06-13 19:01:02 UTC - Ryan Samo: Non partition
----
2019-06-13 19:02:27 UTC - Ryan Samo: I do have bookies running local to
functions workers and then bookies running to support the brokers. Trying to
separate load from the functions state storage to the bookies on the core
----
2019-06-13 19:02:39 UTC - Ryan Samo: Maybe that is confusing the cluster?
----
2019-06-13 19:04:06 UTC - David Kjerrumgaard: try specifying the `BytesSchema`
schema with the `--schema-type` flag when you run the function
----
2019-06-13 19:04:22 UTC - Ryan Samo: Ok trying now
----
2019-06-13 19:04:28 UTC - David Kjerrumgaard: `pulsar-admin functions localrun `
----
2019-06-13 19:04:36 UTC - Patrick Lange: Are there any plans to release a PyPi
package of the python 2.3.2 client for MacOS?
----
2019-06-13 19:05:41 UTC - Jerry Peng: @Matteo Merli ^^^
----
2019-06-13 19:09:00 UTC - Ryan Samo: So if I don’t fully qualify the
BytesSchema, it complains of the custom class needs to exist in class path
----
2019-06-13 19:10:18 UTC - Ryan Samo: If I fully qualify, it complains
BytesSchema not org.apache.pulsar.functions.api.SerDe
----
2019-06-13 19:11:50 UTC - Jerry Peng: @Ryan Samo can you check what the schema
is on the input topic to the function?
----
2019-06-13 19:12:22 UTC - Jerry Peng:
<http://pulsar.apache.org/docs/en/admin-api-schemas/>
----
2019-06-13 19:12:58 UTC - Jerry Peng: Are you using the pulsar-client cli to
produce to this topic?
----
2019-06-13 19:14:29 UTC - Ryan Samo: Trying now
----
2019-06-13 19:15:25 UTC - Aaron: Who should I contact for documentation
questions and feedback?
----
2019-06-13 19:16:06 UTC - Jerry Peng: @Aaron you can ask them here or in the
users mailing list
----
2019-06-13 19:40:57 UTC - Ryan Samo: The admin cli comes back with “not found”.
Trying work that out now
----
2019-06-13 20:05:38 UTC - Aaron: A few questions on the documentation for the
Java client API:
1. What is a listener thread in the client builder interface?
2. How do listener threads and io threads interact with each other and with the
client's connection with the pulsar server/brokers?
3. Is the maxTotalReceiverQueueSizeAcrossPartitions() a total bound for the
individual queue sizes in each partition (which are controlled by
receiverQueueSize?)? Is size in memory or number of messages?
4. Can consumer.receiver() be threaded? (i.e. what happens if
consumer.receiver() is called multiple times concurrently on the same consumer)
5. Is consumer.receiver() the best way to interact with prefetched messages
from the topic? Or does the async call offer better performance/other
advantages?
----
2019-06-13 20:37:12 UTC - Addison Higham: hrm... so, I am just noticing, while
it appears that the underlying JWT library and the pulsar tool to generate key
pairs support ECDSA for doing JWT tokens, I don't think it actually supports
it... see this line:
<https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/utils/AuthTokenUtils.java#L71>
where when the public key gets decoded, it is hard coded to be RSA
hushed : Grant Wu
confused : David Kjerrumgaard
----
2019-06-13 20:47:07 UTC - Addison Higham: @Matteo Merli ^^ looks like you did
most that work, just want to make sure I am not missing anything, but I assume
that only RSA tokens are known to work? it should be pretty easy to fix for at
least ec keys...
----
2019-06-13 20:55:22 UTC - Matteo Merli: I guess.. :slightly_smiling_face: I
don’t remember any particular reason to restrict to RSA only, other than
simplicity
----
2019-06-13 22:10:57 UTC - Addison Higham:
<https://github.com/apache/pulsar/pull/4528> <- I think that should fix it
+1 : Matteo Merli, Steve Kim, Ali Ahmed
----
2019-06-13 23:20:20 UTC - Sijie Guo: Are you seeing
ArrayIndexOutOfBoundException or other exceptions?
----
2019-06-14 00:20:45 UTC - Grant Wu: Sorry, got this confused with the incorrect
parameter exception
----
