2018-06-15 11:08:19 UTC - Idan: Hi, i would like to know if there is possibility via api to purge queue ? ---- 2018-06-15 11:08:38 UTC - Idan: (java client) ---- 2018-06-15 11:19:29 UTC - Matti-Pekka Laaksonen: With 2.0, the consumers can now listen to multiple topics. Is there any way to access the topic the specific message was written to? ---- 2018-06-15 11:20:01 UTC - Matti-Pekka Laaksonen: Or is the preferred way to distinguish between different types of messages to use message properties? ---- 2018-06-15 11:20:37 UTC - Ivan Kelly: @Idan as in drop all messages in a topic? ---- 2018-06-15 11:21:14 UTC - Idan: @Ivan Kelly i guess yes ---- 2018-06-15 11:21:19 UTC - Idan: delete all messages in topic ---- 2018-06-15 11:21:46 UTC - Ivan Kelly: for one subscription, or all subscriptions? ---- 2018-06-15 11:24:37 UTC - Ivan Kelly: in any case, you want either <http://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Namespaces.html#clearNamespaceBacklog-java.lang.String-> or <http://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Topics.html#expireMessagesForAllSubscriptions-java.lang.String-long-> ---- 2018-06-15 11:25:16 UTC - Ivan Kelly: @Matti-Pekka Laaksonen the data is there for sure, one sec ---- 2018-06-15 11:28:50 UTC - Ivan Kelly: @Matti-Pekka Laaksonen you need to cast the messageId to a org.apache.pulsar.client.impl.TopicMessageIdImpl and call getTopicName() on that ---- 2018-06-15 11:29:40 UTC - Matti-Pekka Laaksonen: @Ivan Kelly Oh, nice. This is exactly what I was looking for. Thank you! ---- 2018-06-15 11:31:26 UTC - Idan: @Ivan Kelly one subscription ---- 2018-06-15 11:31:38 UTC - Idan: all topics in that subscription ---- 2018-06-15 11:31:54 UTC - Idan: ok got it. thanks! ---- 2018-06-15 11:32:05 UTC - Ivan Kelly: <http://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Namespaces.html#clearNamespaceBacklogForSubscription-java.lang.String-java.lang.String-> ---- 2018-06-15 11:32:10 UTC - Ivan Kelly: :thumbsup: ---- 2018-06-15 15:31:46 UTC - Matteo Merli: @Idan In addition to the Java admin lib (and REST) it’s possible to perform these operations through the CLI tool. eg :
``` bin/pulsar-admin topics clear-backlog my-topic -s my-subscription ``` ---- 2018-06-15 16:01:33 UTC - Idan: btw: how can I find out which namesame my topic is using? (using this api: <http://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Namespaces.html#clearNamespaceBacklogForSubscription-java.lang.String-java.lang.String->) ---- 2018-06-15 16:01:56 UTC - Idan: when I consume i never provided namespace only topic name and subscription ---- 2018-06-15 16:02:33 UTC - Idan: consumer = client.newConsumer() .topic(topic) .subscriptionType(subscriptionType) .ackTimeout(ackTimeoutInMillis, TimeUnit.MILLISECONDS) .subscriptionName(subscriptionName) .subscribe(); ---- 2018-06-15 16:02:49 UTC - Matteo Merli: a topic is tied to a namespace, the topic name already includes the namespace. eg: Topic name: `<persistent://my-tenant/my-namespace/my-topic>` Namespace: `my-tenant/my-namespace` ---- 2018-06-15 16:03:13 UTC - Idan: ok i must admit i didnt understand well enough the tenant/namspace subjects. i better go the docs again ---- 2018-06-15 16:03:14 UTC - Idan: to get this clear ---- 2018-06-15 16:03:41 UTC - Idan: (not sure when I set my tenant) ---- 2018-06-15 16:04:42 UTC - Matteo Merli: if you use the short topic names. eg `my-topic`, that is equivalent to use `public/default/my-topic` (or `<persistent://public/default/my-topic`>) ---- 2018-06-15 16:05:12 UTC - Idan: to tenant name in this case would be default? ---- 2018-06-15 16:05:39 UTC - Idan: public ---- 2018-06-15 16:05:50 UTC - Matteo Merli: tenant is `public` — namespace is `public/default` ---- 2018-06-15 16:06:09 UTC - Idan: why would i want multiple tenants. what advantages it can gain ---- 2018-06-15 16:06:16 UTC - Idan: i understand you have tenant and namesapces ---- 2018-06-15 16:06:24 UTC - Idan: where namespaces relayed under tenants ---- 2018-06-15 16:07:40 UTC - Matteo Merli: The idea is that you can have multiple tenants in a single cluster and each tenants is allowed to “administer itself”, create namespace, grant permissions and perform operations (like clear backlog and similar), without the system administrator being involved ---- 2018-06-15 16:08:23 UTC - Idan: ok so it will be better to seperate for example my production env and dev env into diff tenants ---- 2018-06-15 16:08:40 UTC - Idan: like prod-tenant and dev-tenant where i can set diff permissions to each ---- 2018-06-15 16:08:41 UTC - Idan: true? ---- 2018-06-15 16:08:47 UTC - Matteo Merli: yes, that is an good example ---- 2018-06-15 16:08:51 UTC - Idan: (couldnt think about other case) ---- 2018-06-15 16:09:03 UTC - Idan: what about namespaces.. why would I need multiple namespaces? ---- 2018-06-15 16:09:38 UTC - Matteo Merli: for example, in places like Yahoo, there 80-100 different tenants on a Pulsar cluster ---- 2018-06-15 16:09:53 UTC - Matteo Merli: each tenant is a “team” within the company ---- 2018-06-15 16:11:20 UTC - Matteo Merli: each team will either have different applications that use Pulsar, or in any case they will have different environments (prod, staging, dev). For all these usages, they will have a different namespace, within their team specific tenant ---- 2018-06-15 16:12:03 UTC - Idan: ok.. so in my case.. we are new startup we dont have lots of teams if at at.. we do have 3 envs (dev,stg,prod) a setup could be like that: ---- 2018-06-15 16:12:10 UTC - Idan: one tenant with 3 namespaces (for each env) ---- 2018-06-15 16:12:21 UTC - Idan: and ofcourse the same topics under each namespace ---- 2018-06-15 16:13:15 UTC - Matteo Merli: yes, in your case there’s no practical difference between using 1-tenant 3-namespaces vs 3-tenants 1-namespace-each ---- 2018-06-15 16:15:19 UTC - Idan: i understand ---- 2018-06-15 16:15:33 UTC - Idan: ok. btw: i finished to create our dlq mechanisem perhaps you want to review my architecture? ---- 2018-06-15 16:15:38 UTC - Idan: pretty simple ---- 2018-06-15 16:16:05 UTC - Idan: there is one thing iam worried about.. we cant make sure our consumers wont consume dedups ---- 2018-06-15 16:16:50 UTC - Idan: if pulsar will have any glitch and the consumer will consume the same messageId the same.. it will pass our mechanism ---- 2018-06-15 16:22:02 UTC - Matteo Merli: @Idan please share details around your DLQ implementation ---- 2018-06-15 16:31:24 UTC - Idan: ok, so we created shared lib among our consumers. the lib has redis client (assuming each of our services has redis). when we ‘gather’ the messages from pulsar we first determine it’s recieveCounter val by matching the key of messageID. then we update it on redis and return it (we using redi’s MULTI command to incr and expire the key). now we have another method that checks if the message shouldDispatchToDLQ. in case the retry=X (our case is 3) we will automaticlly dispatch the message to a DLQ topic by the convention <topic-name>-DLQ ---- 2018-06-15 16:31:42 UTC - Idan: the dlq has a consumer that will consume and put it somewhere ---- 2018-06-15 16:32:47 UTC - Idan: the only caveat here is that if the same message will be consume by pulsar-client we will consider it as re-try and let it pass through ---- 2018-06-15 16:33:13 UTC - Idan: we cant have both re-try determinator and dedup-checker (until you guys will send us back the retrycounter from the broker ---- 2018-06-15 17:59:08 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FB8L5TX29/-.txt|Untitled> and commented: @Sijie Guo @Ali Ahmed Proxy is starting only pulsar in 6650 and 8080. I am using TLS and is not listening on that port. Above is my conf file. Do I need to update discovery.conf or Client.conf. For starting proxy, I am only configuring proxy.conf? ---- 2018-06-15 18:08:56 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FB9HJFSTZ/-.sh|Untitled> and commented: Within the Proxy Container ---- 2018-06-15 18:19:01 UTC - Sijie Guo: @Karthik Palanivelu: # Enable TLS in the proxy tlsEnabledInProxy=false ---- 2018-06-15 18:19:07 UTC - Sijie Guo: I thik you need to enable tls ---- 2018-06-15 18:36:07 UTC - Eric Henry: @Eric Henry has joined the channel ---- 2018-06-15 18:37:05 UTC - Eric Henry: Hi I'm setting up a new cluster and getting a ServiceUnitNotReadyException...how can I fix this? ---- 2018-06-15 18:52:27 UTC - Sanjeev Kulkarni: @Eric Henry is this standalone cluster? Whats your environment? ---- 2018-06-15 18:54:06 UTC - Eric Henry: yep, its a standalone cluster. Using incubator version 1.21.0. I have a three node cluster and a three node zookeeper cluster. Everything came up fine except I keep getting "Namespace not served by this instance" ---- 2018-06-15 18:56:02 UTC - Eric Henry: or rather a "single cluster"...deployed using the instructions here: <https://pulsar.incubator.apache.org/docs/latest/deployment/cluster/> ---- 2018-06-15 19:01:44 UTC - Eric Henry: @Sanjeev Kulkarni What does this error mean? ---- 2018-06-15 19:06:34 UTC - Sanjeev Kulkarni: @Eric Henry any specific reason for using 1.21.0? Any reason not to use 1.22 or even better 2.0? ---- 2018-06-15 19:07:29 UTC - Eric Henry: @Sanjeev Kulkarni It's just the version we have deployed currently. We've been using it in production for several months....first time I've encountered this error setting up a new cluster. ---- 2018-06-15 20:26:11 UTC - Sijie Guo: @Eric Henry it is probably related to a namespace not being owned by one worker and the request being redirected. do you have any logs at the server side? ---- 2018-06-15 21:17:24 UTC - Matteo Merli: @Eric Henry Apart from the recommendation upgrade to 1.22 (which fixes several issues) or 2.0. Can you past the exception stack trace? Also it would help if you can describe which step of the deployment and when it happens. ---- 2018-06-15 22:03:30 UTC - Eric Henry: @Matteo Merli @Sijie Guo hey thanks so much for responding...I found my issue. I had a typo in my cluster name when I created the metadata in zookeeper. Consequently no instances were serving a namespace with that cluster name. But I'm definitely taking your recommendation and getting my version bumped to 2.0 during my next maintenance window +1 : Sijie Guo, Matteo Merli ---- 2018-06-16 07:07:48 UTC - Idan: @Ivan Kelly but what if I dont have admin installed. using only java client. no way to drop everything via that? i see this api is only for the admin app ----
