2020-09-02 10:26:36 UTC - Ebere Abanonu: @Addison Higham @Sijie Guo please look into Pulsar presto, when TLS is enabled for ZooKeeper, presto fails to connect. ---- 2020-09-02 11:55:23 UTC - Guillaume: Hi, I'd like to know if Pulsar Functions have good performance ? I mean is it better (performance-wise) to write a Pulsar Function or an external consumer ? Thank you. ---- 2020-09-02 13:42:23 UTC - Jennifer Huang: @Jennifer Huang set the channel topic: Apache Pulsar 2.6.1 release blog: <https://pulsar.apache.org/blog/2020/08/21/Apache-Pulsar-2-6-1/> ---- 2020-09-02 15:19:54 UTC - Nazia Firdous: Sir,Did u figure out the issue? ---- 2020-09-02 16:12:28 UTC - Addison Higham: Topic lookups are performed by the broker to know which topic belong to which broker. There is a limit on the number of concurrent lookups that can be in progress, which is controlled by the `maxConcurrentLookupRequest` property, but it defaults to 50,000. If you are trying to work with hundreds of thousands of topics, you may run into that, but the client should back-off and retry.
If you run into that issue with less topics than that, usually it indicates an issue with brokers and/or zookeeper with it being unable to service metadata requests in a timely manner ---- 2020-09-02 16:16:16 UTC - Addison Higham: Functions internally are a pretty standard consumer and producer, so it can be just as fast depending on what you are doing, but it is important to understand the API, for example, if you do long synchronous operations, it will be slow as the function is called one at a time, though with the newest versions of functions API you can now return a future and it be awaited upon in batches ---- 2020-09-02 16:22:01 UTC - Guillaume: even if used with input pattern there will be no gain in performance than if it were an external consumer ? no magic there then ? ---- 2020-09-02 16:27:58 UTC - Evan Furman: @Addison Higham as Tim mentioned, I’m happy to pull anything that is useful. Let me know how I can help ---- 2020-09-02 16:28:14 UTC - Addison Higham: is the cluster still in that state? ---- 2020-09-02 16:28:29 UTC - Evan Furman: i can get it back there :smile: ---- 2020-09-02 16:29:42 UTC - Evan Furman: working on it now ---- 2020-09-02 16:30:59 UTC - John Harris: @John Harris has joined the channel ---- 2020-09-02 16:33:16 UTC - Tim Haak: @Tim Haak has joined the channel ---- 2020-09-02 16:33:58 UTC - Evan Furman: cc: @Tim Haak ---- 2020-09-02 16:34:34 UTC - Tim Haak: :wave: ---- 2020-09-02 16:36:25 UTC - Addison Higham: how did you fix it when it was in that state? a topic unload? ---- 2020-09-02 16:37:15 UTC - Evan Furman: I tore it down last night. I’m rebuilding the cluster now and will see if I can get it back to where we were. Would it be useful to jump on a zoom call at that point? ---- 2020-09-02 16:38:45 UTC - Addison Higham: yeah, happy to do so. Also, in the meantime if you have logs during the period I can take a peek ---- 2020-09-02 16:38:56 UTC - Addison Higham: err for the last cluster if you have logs :slightly_smiling_face: ---- 2020-09-02 16:43:53 UTC - Evan Furman: Ah, didn’t pull before we tore it down. Hoping I’ll have a broken cluster for you soon :stuck_out_tongue: ---- 2020-09-02 18:33:32 UTC - Lari Hotari: Sounds like "reconsumeLater" would do. <http://pulsar.apache.org/api/client/2.6.0-SNAPSHOT/org/apache/pulsar/client/api/Consumer.html#reconsumeLater-org.apache.pulsar.client.api.Message-long-java.util.concurrent.TimeUnit-|http://pulsar.apache.org/api/client/2.6.0-SNAPSHOT/org/apache/pulsar/client/api/Consumer.html#reconsumeLater-org.apache.pulsar.client.api.Message-long-java.util.concurrent.TimeUnit-> or <https://github.com/apache/pulsar/issues/6448|https://github.com/apache/pulsar/issues/6448> for more details. ---- 2020-09-02 21:13:25 UTC - Evan Furman: @Addison Higham I’ll have to get back with you on this. We are troubleshooting some issues with the deployment which are currently blocking me right now… ---- 2020-09-02 21:19:57 UTC - Addison Higham: okay, no worries, I actually am out for a few hours, but feel free to ping me when you think it might be ---- 2020-09-03 06:30:23 UTC - Takahiro Hozumi: Why does pulsar java client have a complete copy of the avro package in `<http://org.apache.pulsar.shade.org|org.apache.pulsar.shade.org>.apache.avro` , instead of using the official avro package directly? Often I want to take the GenericRecord (a version of org.apache.pulsar.shade) received from the pulsar topic and parsed with AUTO_CONSUME and pass it to a library that needs an instance of the original avro package. ---- 2020-09-03 07:55:22 UTC - Lari Hotari: There's also an unshaded version of the client. It's called "pulsar-client-original" <https://search.maven.org/artifact/org.apache.pulsar/pulsar-client-original/2.6.1/jar> . I was running into some issues with container packaged with Jib plugin in a Gradle build. That was resolved with these tricks: ```dependencies { implementation('org.apache.pulsar:pulsar-client-original') { // exclude circe-checksum since it defaults to "nar" packaging exclude group: 'org.apache.bookkeeper:circe-checksum', module: 'circe-checksum' } // by default circe-checksum will resolve to "nar" packaging. That dependency doesn't work with Jib containers // the workaround is to also add the "jar" packaging explicitly as a dependency implementation 'org.apache.bookkeeper:circe-checksum:4.10.0@jar' }``` eyes : Takahiro Hozumi ---- 2020-09-03 08:07:15 UTC - Jennifer Huang: @Jennifer Huang set the channel topic: - Pulsar Summit Asia 2020 CFP is now open: <http://pulsar.apache.org/blog/2020/09/01/pulsar-summit-asia-2020-cfp/> - Apache Pulsar 2.6.1 release blog: <https://pulsar.apache.org/blog/2020/08/21/Apache-Pulsar-2-6-1/> ---- 2020-09-03 08:13:06 UTC - Lari Hotari: I have been seeing similar issues, but I haven't had the time to report them yet. In one case, the thread dump of one broker contained some hanging threads. The logs of that broker also had errors of zookeeper lookups timing out. I guess thread dumps / heap dumps of the brokers would reveal more diagnostics in this case. ---- 2020-09-03 08:56:34 UTC - Thomas: @Thomas has joined the channel ---- 2020-09-03 09:04:38 UTC - Thomas: Hi! Is there a way to configure pulsar to use storage at another path than `PULSAR_HOME/data`? I'd like to have pulsar-binaries & libs in `/opt` and it's data on `/data/fast-storage` . Any docs on that? ----
