2020-08-20 10:54:09 UTC - Penghui Li: Here is an issue that tracking topic level policy feature <https://github.com/apache/pulsar/issues/2688>, and it will available in 2.7.0 ---- 2020-08-20 10:57:03 UTC - Penghui Li: Currently, you can use namespace level policy `pulsar-admin namespaces set-dispatch-rate` or `pulsar-admin namespaces set-publish-rate` . Topics under this namespace follows the namespace strategy ---- 2020-08-20 10:57:54 UTC - Raghav: Thank you. Yes as of now using the namespace level setting but was trying to see if there are any config settings at topic level. ---- 2020-08-20 11:00:16 UTC - Raghav: And for consumers there are two settings “set-dispatch-rate” and “set-subscription-dispatch-rate” what is used when? And is there any preference of one over the other? ---- 2020-08-20 11:01:24 UTC - Raghav: I get it..the former used at namespace level and the latter at subscription level. But what is the order of preference ? ---- 2020-08-20 11:04:35 UTC - Raghav: I think I might be wrong here..the set-subscription-dispatch is not taking the subscription name as input(not mandatory). So how do we use this? ---- 2020-08-20 11:24:09 UTC - Manpreet Babbra: Hi, a question on custom Pulsar Connectors.. When running a source connector using localrun you have to specify an `destination-topic-name` . Is it possible for the source to write to different topics instead (e.g. route a message to a topic based on its content) or can we only write to one destination topic? ---- 2020-08-20 11:28:26 UTC - Penghui Li: `set-subscription-dispatch` means all subscriptions follows the dispatch rate of the namespace policy, `set-dispatch` means all topics follows the dispatch rate of the namespace policy. Because of a topic can have multiple subscriptions. ---- 2020-08-20 11:33:51 UTC - Raghav: So for an example lets consider two subscriptions sub-a and sub-b for the same topic. Assuming that sub-b was down for sometime and the backlog was built up, when sub-b comes online it tries to use all of the available bandwidth. So how do I restrict sub-b to limit its bandwidth and at the same time I want it to catchup faster by having more throughput than sub-a. Is that possible with 2.6.0? ---- 2020-08-20 13:20:44 UTC - Addison Higham: yes, sources (and the underlying functions) can use the context object to write to any number of topics :slightly_smiling_face: ---- 2020-08-20 13:23:28 UTC - Manpreet Babbra: Great, thank you @Addison Higham. ---- 2020-08-20 13:59:30 UTC - Olivier Chicha: Hello Pulsar Team, we are facing an issue since we moved to 2.6.0 we can reproduce it systematically. Scenario: we connect using websocket to pulsar: <wss://pulsar.acme.com/ws/v2/reader/persistent/public/default/mytopic?messageId=CPi7ZhACIAI%3D> we receive ```HTTP/1.1 500 Failed to create reader: Exclusive consumer is already connected Content-Length: 499 Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=iso-8859-1 Date: Thu, 20 Aug 2020 11:53:09 GMT Server: Jetty(9.4.20.v20190813) <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 500 Failed to create reader: Exclusive consumer is already connected</title> </head> <body><h2>HTTP ERROR 500</h2> <p>Problem accessing /ws/v2/reader/persistent/infinity/default/Alias_GroupAddressMember_Events. Reason: <pre> Failed to create reader: Exclusive consumer is already connected</pre></p><hr><a href="<http://eclipse.org/jetty>">Powered by Jetty:// 9.4.20.v20190813</a><hr/> </body> </html>``` from there on it is impossible to connect a consumer (in exclusive mode) or a reader to that topic. futhermore the reader subscription is kept in memory and as the reader tries to reconnect the number of subscriptions keep growing.
to solve the issue we have to unload the topic. once the topic has been unloaded we can connect back consumer and readers as long as we don't connect our websocket reader with a messageID. note: I don't know if this messageID is still present in the storage of the bookkeeper, most probably not. *If we don't provide any messageID then everything goes well.* We have never noticed this issue before 2.6.0, I am assuming that this is a regression introduced in this version (we were using 2.5.1 before), but I have no proof. Find in attachement the typical exception we get in the broker after the topic as become "unreachable" Is this a known Issue ? If no, can I create a ticket ? Best regards, Olivier ---- 2020-08-20 14:54:35 UTC - Addison Higham: have you looked in github for any issues? ---- 2020-08-20 16:21:07 UTC - Matt Mitchell: Is there a configuration setting that will enable Pulsar to automatically remove subscriptions that have backlogged messages, but 0 consumers? ---- 2020-08-20 16:30:44 UTC - Addison Higham: I believe `subscriptionExpirationTimeMinutes` (or the namespace setting `pulsar-admin namespaces set-subscription-expiration-time`) will delete even if there is backlog ---- 2020-08-20 16:34:37 UTC - Nate Marshall: @Nate Marshall has joined the channel ---- 2020-08-20 16:36:08 UTC - Madhia Asghar: @Madhia Asghar has joined the channel ---- 2020-08-20 16:40:45 UTC - Madhia Asghar: Hi ---- 2020-08-20 16:46:12 UTC - Renars Grinbergs: @Renars Grinbergs has joined the channel ---- 2020-08-20 16:49:48 UTC - Renars Grinbergs: Hi. Is it possible to run Pulsar without any Bookkeepers if persistent topics are disabled and only non-persistent use-cases are actual? Sorry if this is answered somewhere in docs. Could not find any information yet. ---- 2020-08-20 17:04:14 UTC - Derek Moore: You can also expire messages explicitly for a given subscription on a topic or for all subscriptions on a topic: <https://pulsar.apache.org/docs/en/pulsar-admin/#expire-messages> <https://pulsar.apache.org/docs/en/pulsar-admin/#expire-messages-all-subscriptions> ---- 2020-08-20 17:05:27 UTC - Matt Mitchell: perfect, thanks @Addison Higham and @Derek Moore :thumbsup: ---- 2020-08-20 17:29:19 UTC - Aaron: How does one build sub records with the GenericRecordBuilder in the java client? ---- 2020-08-20 17:48:59 UTC - Addison Higham: :thinking_face: I am not aware of it being an explicit goal to support such a scenario, it is true that nonPersistent topics and subscriptions should not need bookkeeper, but there are likely some parts of pulsar initialization that will try and connect. For example, bookkeeper is also used for storing schemas. I would guess it would probably result in an error on boot. Perhaps @Sijie Guo would know if this has been attempted before. As a bit of a workaround, you could probably start bookkeeper with minimal resources and just a single instance, purely to just satisfy the other needs pulsar has of bookkeeper ---- 2020-08-20 17:50:28 UTC - Cesar Vera Bernal: @Cesar Vera Bernal has joined the channel ---- 2020-08-20 19:10:20 UTC - Matt Mitchell: @Addison Higham is `subscriptionExpirationTimeMinutes` a valid config setting for Pulsar 2.5? ---- 2020-08-20 19:11:55 UTC - Addison Higham: it should be, but the 2.5.x docs don't discuss it ---- 2020-08-20 19:13:00 UTC - Matt Mitchell: Ok got it. I was checking through the output of `./bin/pulsar-admin namespaces` and didn’t see a reference to it there either. I’ll see what happens when attempting to set it. ---- 2020-08-20 19:15:11 UTC - Addison Higham: oh the per namespace setting likely isn't there yet ---- 2020-08-20 19:15:20 UTC - Addison Higham: in 2.5.2 it is likely just global ---- 2020-08-20 19:16:23 UTC - Matt Mitchell: Ok thanks. Is it possible to set that via `pulsar-admin` CLI? ---- 2020-08-20 22:03:21 UTC - Nathan Mills: Can someone help me understand the relationships between `retention policy` `backlog quotas policy` and `ttl`. For example if for a namespace I have my `retentionTimeInMinutes` and `retentionSizeInMB` both set to `-1` . What happens if my `backlog-quota limit` is exceed and the `backlog-quota policy` is `consumer_backlog_eviction` ? ---- 2020-08-20 22:14:37 UTC - Arthur Alacar: @Arthur Alacar has joined the channel ---- 2020-08-20 23:21:09 UTC - Jennifer Huang: @Jennifer Huang set the channel topic: We’re excited to announce StreamNative Cloud, providing Apache Pulsar®-as-a-Service. Check out the news at <https://streamnative.io/blog/tech/2020-08-18-announcing-streamnative-cloud> Guides: <https://streamnative.io/docs/cloud/stable/overview> ---- 2020-08-21 06:37:01 UTC - Lari Hotari: Regarding the 2.6.1 release, I noticed that it's possible to get a hold of 2.6.1 binaries before they are in Maven Central by using <https://repository.apache.org/content/repositories/releases/> as a maven repo. At the moment 2.6.1 seems to be also on Maven Central so that's not needed anymore for testing 2.6.1 . However I'd like to also test the 2.6.1 docker images. Is there a staging repo also for docker images? The 2.6.1 images aren't yet on Docker Hub (<https://hub.docker.com/r/apachepulsar/pulsar/tags>). Previously there has been a delay until images have been available for Pulsar releases. Is that a manual step in the release process? When will the 2.6.1 images be available? I'd like to do some testing with 2.6.1 to see if it fixes a few issues I've been experiencing. ---- 2020-08-21 06:41:56 UTC - Lari Hotari: I did find <https://hub.docker.com/r/streamnative/pulsar/tags?page=1&name=2.6.1|streamnative/pulsar:2.6.1> in docker hub. What are the differences of the streamnative version of Pulsar compared to the <http://apache.org|apache.org> version? ---- 2020-08-21 06:55:52 UTC - Nazia Firdous: Hi Can anyone let me know how to produce and consume a message on different host(like if it is two host h218 and h219)..h218 producing the messages and h219 consuming the messages... ---- 2020-08-21 08:25:33 UTC - Raghav: have you tried running the producer on 218 and consumer on 219 using the cli tool pulsar-client? Did you see any errors? ---- 2020-08-21 08:33:04 UTC - Renars Grinbergs: Thank you. If BK is indeed used for other things than topic persistence then I would prefer it remains HA. ----
