2020-10-20 11:53:24 UTC - Guillaume: Hello, When I create more than 10 topics with policies sequentially I have a lot of Timeout from the broker: ```11:42:12.526 [pulsar-ordered-OrderedExecutor-7-0] WARN org.apache.pulsar.broker.service.BrokerService - Got exception when reading persistence policy for <persistent://xiotnx/agent_ns/action_down-bbd1a2fb-a761-4ca8-8e86-82762d1878a3>: null java.util.concurrent.TimeoutException: null
11:42:12.530 [pulsar-ordered-OrderedExecutor-7-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [<persistent://xiotnx/agent_ns/action_down-bbd1a2fb-a761-4ca8-8e86-82762d1878a3>] Failed to update persistence-policies java.util.concurrent.TimeoutException 11:42:12.526 [ForkJoinPool.commonPool-worker-1] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [<persistent://xiotnx/agent_ns/action_down-c06808d9-8569-4a52-a3d2-b440605fb748>] Policies update failed org.apache.pulsar.broker.service.BrokerServiceException$ServerMetadataException: java.util.concurrent.TimeoutException, scheduled retry in 60 seconds java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$ServerMetadataException: java.util.concurrent.TimeoutException``` Do you know where it comes from ? When those timeout appears broker (deployed on Kubernetes) failed and crash. Thanks for you help. ---- 2020-10-20 16:21:29 UTC - Rong: Quick question: is subscription name/value required for pulsar consumer (client) to receive SUBACK and published topic message? For pulsar-client, pulsar python client, you do have to provide a subscription name in pulsar client call. So I’d like to confirm if that’s true for all kinds of clients? Here is the issue: we are using pulsar mqtt handler(mop) with some open-source clients, one of them (mqtt-box) works as consumer, however, other clients like hivemq-cli, paho-mqtt client couldn’t receive published topic mesg, which don’t provide subscription name in their call to pulsar. The pulsar broker seems to work fine without any errors. Could it be an issue for consumer without subscription name? Now not all mqtt clients provide subscription name as an option… ---- 2020-10-20 16:22:20 UTC - Addison Higham: are you still seeing the issue? that looks like a zookeeper issue at first glance ---- 2020-10-20 16:26:38 UTC - Addison Higham: Hrm... I am not familiar with the details of MoP, but yes, you always need a subscription name, but sometimes that subscription name can be automatically generated. It might be that MoP should be able to handle no subscription and just use a generated name (probably like the client hostname or something) @Penghui Li ^^ if you can provide some context ---- 2020-10-20 16:26:49 UTC - Addison Higham: @Rong what I would suggest is opening up an issue on the MoP project ---- 2020-10-20 16:27:36 UTC - Guillaume: I managed to fix it by increasing CPU allocation in Kubernetes. So it's better now. But grantPermission seems to be a very very slow and CPU consumming process. Is there a way to make it faster or to bulk it ? I am creating a lot of topics and grantPermission on those topics. ---- 2020-10-20 16:29:55 UTC - Rong: Thanks, @Addison Higham I already opened an tix for the issue: <https://github.com/streamnative/mop/issues/20> - after some more investigation, i kind of narrowed down the issue to this one. If you can confirm that subscription name is required for all clients, then it should be easy to find a solution ---- 2020-10-20 16:30:44 UTC - Addison Higham: namespace grants are much less expensive, so if you have many topics, grouping them in a namespace and just granting access to the entire namespace can be useful. There isn't a way to grant in batch to individual topics at the moment. I am sure there could be made some improvements there ---- 2020-10-20 16:32:13 UTC - Guillaume: thanks a lot ---- 2020-10-20 16:34:18 UTC - Olivier Chicha: Hello @Addison Higham After quite some headaches I have succeeded to get mvn install -DskipTests to run successfully I am now encountering an issue that I have no idea how to solve during the mvn install -DskipTests -Pdocker find below the end of the logs of the build ---- 2020-10-20 16:35:12 UTC - Addison Higham: @Olivier Chicha do you have a relatively up-to-date docker installed? ---- 2020-10-20 16:35:39 UTC - Olivier Chicha: yes ---- 2020-10-20 16:35:47 UTC - Olivier Chicha: i can check the version ---- 2020-10-20 16:35:59 UTC - Guillaume: by the way i have 1.5 million topics and subscriptions. I guess this is not an issue for Pulsar ? ---- 2020-10-20 16:36:48 UTC - Olivier Chicha: ```olivier@tchitcha:~/git/pulsar/docker$ docker version Client: Docker Engine - Community Version: 19.03.12 API version: 1.40 Go version: go1.13.10 Git commit: 48a66213fe Built: Mon Jun 22 15:45:36 2020 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 19.03.12 API version: 1.40 (minimum version 1.12) Go version: go1.13.10 Git commit: 48a66213fe Built: Mon Jun 22 15:44:07 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.13 GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429 runc: Version: 1.0.0-rc10 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd docker-init: Version: 0.18.0 GitCommit: fec3683``` ---- 2020-10-20 16:37:26 UTC - Addison Higham: @Olivier Chicha what this issue appears to be specifically is that your docker config is is trying to run the `docker-credential-secretservice` to get a token, you can try doing a `docker logout` which should hopefully clear that ---- 2020-10-20 16:37:56 UTC - Addison Higham: see this issue in another project for same problem: <https://github.com/pgRouting/docker-pgrouting/issues/11> ---- 2020-10-20 16:38:06 UTC - Jack Tantram: @Jack Tantram has joined the channel ---- 2020-10-20 16:39:11 UTC - Olivier Chicha: Ok I will definitively try that thank you very much ---- 2020-10-20 16:39:41 UTC - Addison Higham: It is possible, but requires extra tuning and resources. Usually you will need more resources for zookeeper and more memory and CPU for brokers. You also should tune some of the timeout settings you find in broker.conf ---- 2020-10-20 16:39:52 UTC - Addison Higham: you also need to tune your number of bundles in namespaces ---- 2020-10-20 16:41:51 UTC - Emil: Hello! Given the issue has been reopened <https://github.com/apache/pulsar/issues/5350>, does it mean there is no way to use SerDe in java currently? I'm also getting `User class must be in class path` error for my serde class, even though it's included in the function's jar. ---- 2020-10-20 16:42:18 UTC - Guillaume: by the default timeout is 30seconds, this is not enough ? what would be a good value ? ---- 2020-10-20 16:44:11 UTC - Olivier Chicha: Thanks for the link, the issue described there was exactly my case Sorry to have bothered you with something I could have found by myself ---- 2020-10-20 16:44:31 UTC - Addison Higham: no worries, big java stack traces are quite the ball of mud :wink: ---- 2020-10-20 16:44:35 UTC - Addison Higham: especially in build! wink : Olivier Chicha ---- 2020-10-20 16:48:21 UTC - Sijie Guo: Which pulsar version? Is it 2.5.x or 2.6.x? ---- 2020-10-20 16:48:38 UTC - Emil: 2.5.x ---- 2020-10-20 18:00:05 UTC - timurkh: @timurkh has joined the channel ---- 2020-10-20 18:53:54 UTC - Joshua Decosta: @Addison Higham the examples I've attached show I have 3 topics technically but the message metrics say 4 ---- 2020-10-20 20:24:00 UTC - Pushkar Sawant: I am intermittently seeing timeouts on our producer. There are no errors in logs and the only message seen on broker is about closing existing ledger and opening a new ledger. Has anyone else experienced this? Our issue looks similar to <https://github.com/apache/pulsar/issues/173> ---- 2020-10-20 20:37:19 UTC - Joshua Decosta: Also the pulsar_topics_count on the cluster level just looking at the metrics seems wrong ---- 2020-10-20 21:05:18 UTC - Shivasish: @Shivasish has joined the channel ---- 2020-10-20 21:14:33 UTC - Addison Higham: @Joshua Decosta are you running functions? There also is an internal topic that filter out of the API but may not be filtering on metrics side ---- 2020-10-20 21:15:00 UTC - Joshua Decosta: I’m not running functions ---- 2020-10-20 21:15:19 UTC - Joshua Decosta: At least i don’t think i am. ---- 2020-10-20 21:15:38 UTC - Joshua Decosta: I’ll have to check my configs in helm but i believe I’ve turned it off ---- 2020-10-20 21:16:45 UTC - Addison Higham: okay, functions worker creates a few topics, the extra single topic may be the internal topic. Do you see topic names for any of the topic level metrics? ---- 2020-10-20 21:49:40 UTC - Joshua Decosta: Of the function workers were creating a topic that would show before I’d created a single topic right? If so that wasn’t the case ---- 2020-10-21 00:11:51 UTC - Penghui Li: Thanks @Rong ---- 2020-10-21 03:26:28 UTC - Ming Fang: It looks a similar issue was already created. I added my config to the same issue. <https://github.com/apache/pulsar/issues/4672> ---- 2020-10-21 04:07:56 UTC - VanderChen: Hi, all! I have several simple questions about Pulsar. 1. Is there any method to deploy Pulsar Function Worker with *Java clien*t rather than *CLI tools*. 2. Is there any method using multiple threads in Function Worker ---- 2020-10-21 06:26:19 UTC - Rong: @Penghui Li do you think the above suggestion make sense? Is this something that could be fixed in mop, i.e. having it auto-generate some subscription name when the mqtt clients (often the case) don’t provide it in the subscription? ---- 2020-10-21 06:53:02 UTC - Konrad Łyś: @Konrad Łyś has joined the channel ---- 2020-10-21 06:56:19 UTC - Konrad Łyś: Hi, all! I recently opened an issue for the logs format (<https://github.com/apache/pulsar/issues/8298>) But i would like to know if there is a way to configure logging format for broker and bookkeeper manually? I know there is some configuration for local env, but i need to do it in Kubernetes env. ---- 2020-10-21 07:27:41 UTC - Penghui Li: We have take a deep look at the current implementation, we have found the problem and the reason is not the subscription, the root cause is the client ID of the mqtt client. You can try to use a different client ID to verify, we will push a fix soon. ----
