2019-01-08 09:57:54 UTC - bossbaby: i found different in library pulsar java and c++. In ack message by consumer, message will schedule and resend but in c++, message will resend when system started again so that message will be lost ---- 2019-01-08 11:17:44 UTC - Yuvaraj Loganathan: @Yuvaraj Loganathan set the channel topic: Pulsar graduated as a Top-Level project :tada: - Pulsar release 2.2.1 - <https://pulsar.apache.org/release-notes/#2.2.1> +1 : Sijie Guo, Guangzhong Yao, Bogdan BUNECI, Edward, Karthik Ramasamy 100 : Sijie Guo tada : Sijie Guo ---- 2019-01-08 11:40:03 UTC - Sijie Guo: Can you explain more about this? ---- 2019-01-08 14:08:54 UTC - eolivelli: Hi, I have a question about getting last message id <https://lists.apache.org/thread.html/1e8ab9ad9718b35944440513eccba7cbbc44ac09df2086669694300d@%3Cusers.pulsar.apache.org%3E>
Can anyone help please? ---- 2019-01-08 14:21:47 UTC - Ivan Kelly: @eolivelli there was an admin command added recently to get it ---- 2019-01-08 14:22:29 UTC - eolivelli: Do you have any pointer? I am still new to Pulsar. Thanks ---- 2019-01-08 14:23:26 UTC - Ivan Kelly: it's only in master, <https://github.com/apache/pulsar/commit/f4936763af1c1dab54d031ae355e40d66d3a602e> ---- 2019-01-08 14:23:46 UTC - Ivan Kelly: on further inspection, it looks like they added the rest endpoint and not the admin api itsel ---- 2019-01-08 14:23:49 UTC - Ivan Kelly: :confused: ---- 2019-01-08 14:27:35 UTC - Sijie Guo: @Ivan Kelly the admin api was added in a subsequent change if I remembered correctly ---- 2019-01-08 14:30:33 UTC - Sijie Guo: <https://github.com/apache/pulsar/commit/94a942384b4d1e07fba4dfb1fe2cc0c06bc0a2ea> ---- 2019-01-08 14:31:26 UTC - Ivan Kelly: ah, wasn't noted on the original issue ---- 2019-01-08 15:52:43 UTC - Grant Wu: Is that a question? No, it uses a more complicated scheme; <https://zookeeper.apache.org/doc/current/zookeeperOver.html> might help? ---- 2019-01-08 15:57:13 UTC - bossbaby: i run my consumer client and get message from topic. In java, im testing by consumer get message "A" and not commit ack to broker then about 30s, consumer receive message "A" again and continue it unless consumer commit ack In c++, same as java but consumer will receive message "A" (not ack to broker) when client reconnect to broker ---- 2019-01-08 19:20:15 UTC - Ryan Samo: Have you guys ever used Digicert as a root CA instead of creating your own certificate authority? ---- 2019-01-08 19:23:01 UTC - Matteo Merli: Mostly used with custom created CA. We had the idea to include a set of default CA certs to ship with client library (eg: LetsEncrypt) to simplify TLS setup on client side ---- 2019-01-08 19:30:16 UTC - Ryan Samo: I guess I’m just wondering if it is more secure by using your own root CA via the Pulsar documentation vs a Digicert since anyone can get a Digicert root and create a client with a CNAME match and their in? Is that right? ---- 2019-01-08 19:31:09 UTC - Matteo Merli: Never used Digicert so not sure how’s that model ---- 2019-01-08 19:33:06 UTC - Ryan Samo: Enterprise security always turns their nose up self signed certs so just wondered if you guys had any experience with that. Pulsar came out of Yahoo right? I guess they were comfortable with self signed :) ---- 2019-01-08 19:33:36 UTC - Matteo Merli: That’s not self-signed though. You have an internal CA that’s created ---- 2019-01-08 19:33:49 UTC - Matteo Merli: all certificates are created and signed with that CA ---- 2019-01-08 19:34:28 UTC - Matteo Merli: Self-signed means no CA is involved (and there’s no way to verify the identity of the peer) ---- 2019-01-08 19:35:11 UTC - Ryan Samo: Ok makes sense, that’s the ammo we need perfect, makes sense. ---- 2019-01-08 19:36:09 UTC - Matteo Merli: The question is only that maintaining that CA and generating TLS certs with OpenSSL is a bit painful.. but that’s how it is ---- 2019-01-08 19:37:30 UTC - Ryan Samo: Yup you’re right, but works like a champ ---- 2019-01-08 20:03:55 UTC - Karthik Palanivelu: Team, I am launching broker on a Kubernetes Platform. Can you please let me know how I can expose hostIP as Advertised address in a Deployment mode? PodStatus.hostIP is not working. ---- 2019-01-08 20:05:11 UTC - Matteo Merli: There are several way to expose brokers outside Kubernetes cluster. The easiest way it use Pulsar proxy and attach a load-balancer to the proxy service. ---- 2019-01-08 20:05:36 UTC - Matteo Merli: Then client can just use the LB url as the Pulsar service URL ---- 2019-01-08 20:09:59 UTC - Karthik Palanivelu: I understand, but I am trying Sync Replication between two K8s clusters in which broker has to be exposed via the discoverable IP. So I am using hostPort, which will have a broker per node on a port 6651. But still it is using PodIp and is connect time out to reach to the other broker. If I expose broker on host IP, I should be able to reach. Is there a way to do in Deployment Kind, it is possible in Daemonsets using status.hostIP. ---- 2019-01-08 20:11:13 UTC - Karthik Palanivelu: In this case I do not need proxy ---- 2019-01-08 20:18:31 UTC - Matteo Merli: Oh, ok. In that case, you need to pass down the hostIP and tell the broker to advertise that address ---- 2019-01-08 20:19:07 UTC - Matteo Merli: This is done by setting `advertisedAddress=1.2.3.4` in `broker.conf`. ---- 2019-01-08 20:19:16 UTC - Matteo Merli: Let me search if I can find an example ---- 2019-01-08 20:34:53 UTC - Karthik Palanivelu: Is there a way to pass this hostIP via the yaml file like env variables from K8s? ---- 2019-01-08 20:58:37 UTC - Matteo Merli: @Karthik Palanivelu Take a look at <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/generic/bookie.yaml#L78> ---- 2019-01-08 20:58:46 UTC - Matteo Merli: It’s for bookies, but it’s the same thing ---- 2019-01-08 21:01:56 UTC - Matteo Merli: to be applied for brokers ---- 2019-01-08 21:29:41 UTC - Karthik Palanivelu: Yes if I change to daemonset it will work but I am trying as deployment ---- 2019-01-08 21:51:35 UTC - Grant Wu: ~Out of curiosity, is there a reason why the default cluster name is called “pulsar” when running Pulsar in the clustered mode, but it’s “standalone” when running Pulsar in standalone mode?~ ---- 2019-01-08 21:52:15 UTC - Grant Wu: ~This is causing some headaches where we’re trying to automatically create some new namespaces; we need to then figure out whether or not the deployment is running in clustered or standalone mode in order to figure out what the correct cluster name is.~ ---- 2019-01-08 21:55:28 UTC - Grant Wu: Er, hold up, it seems like this might be an artifact of some customizations we’re doing. ---- 2019-01-08 22:33:48 UTC - Grant Wu: Can someone using the Helm charts available on GitHub tell me what the default value of `pulsar-admin namespaces get-clusters public/default` is? ---- 2019-01-09 00:04:23 UTC - Matteo Merli: Sure, for brokers it will also work with a “deployment” ---- 2019-01-09 00:04:52 UTC - Matteo Merli: You just need to expose the host IP in the env: ``` env: - name: advertisedAddress valueFrom: fieldRef: fieldPath: status.hostIP ``` ---- 2019-01-09 02:52:06 UTC - bossbaby: i was deploy 2 VMs, each VM is 1 broker, 1 bookie, 1 zookeer. How to setup that 1 cluster with 2 broker use 2 VMs? ---- 2019-01-09 03:29:23 UTC - Vincent Ngan: Is there any way I can explicitly delete a message from a topic so that the message will never appear again in any subscriptions to the topic? ---- 2019-01-09 03:32:35 UTC - Vincent Ngan: The reason I ask this is that the acknowledgement of the consumption of a message only applicable to the subscription the consumer is subscribing. If another consumer subscribes to the same topic using a different subscription, that message will be delivered again to that new consumer. ---- 2019-01-09 03:33:56 UTC - Sijie Guo: Unfortunately, there is no way to do so at this moment. Do you mind sharing more details about your use case? ---- 2019-01-09 03:34:22 UTC - Vincent Ngan: But I have a use case that I need to delete (or acknowledge) a message so that it will never be delivered again to any consumer using any subscriptions. ---- 2019-01-09 03:37:39 UTC - Vincent Ngan: I explained this in another message please check. ---- 2019-01-09 04:08:58 UTC - Matteo Merli: @Vincent Ngan wouldn’t a shared subscription work for your case? Or do you want some of the messages to be delivered to all consumers? ---- 2019-01-09 04:15:03 UTC - Vincent Ngan: The consumers do not share messages. Each of them needs to see all the messages in the topic with the exception that for certain messages, when one consumer acknowledge them, they will not appear to other consumers. ---- 2019-01-09 04:16:27 UTC - Matteo Merli: (Just speculating here) couldn’t you use a different topic for these messages? ---- 2019-01-09 04:17:00 UTC - Matteo Merli: Eg, keep the 2 kind of messages separated ---- 2019-01-09 04:22:33 UTC - Vincent Ngan: Actually, what I mean is that they do share messages but not in the sense Pulsar shares messages across consumers with the same subscription. Pulsar sharing means the messages split to multiple consumers in round-robin way. What I mean is that I want multiple consumers to receive identical messages. I can achieve this by having the consumers subscribing to the topic with different subscription names. ---- 2019-01-09 04:24:59 UTC - Vincent Ngan: But, because the consumers are in different subscriptions, a consumer will not be aware of the acknowledgement made by other consumers. ---- 2019-01-09 04:25:38 UTC - Matteo Merli: Correct, the “model” is decided at the subscription level and not per each single message bananadance : Yuvaraj Loganathan ---- 2019-01-09 04:26:40 UTC - Matteo Merli: In your case you want to have multiple subscriptions for some messages and 1 single subscription for some others ---- 2019-01-09 04:28:14 UTC - Matteo Merli: That’s why I was suggesting if it’s possible to isolate the different types of messages between 2 different topics. On one you can use 1 subscription per consumer and in the other 1 single shared subscription across all consumers ---- 2019-01-09 04:32:53 UTC - Vincent Ngan: No, the messages can not be classified into different types. Any message can be acknowledged and that message once been acknowledged by a particular consumer, it should not appear again to other consumers. But for those messages which have not been acknowledged will be able to be delivered ( or redelivered) to other consumers. ---- 2019-01-09 04:33:57 UTC - Matteo Merli: ...delivered to 1 other consumer or to all other consumers? ---- 2019-01-09 04:36:38 UTC - Matteo Merli: In the first case it is exactly how a shared subscription works. You can specify an ”ack timeout” after which if a message was not acked it will be resent to another consumer ---- 2019-01-09 04:37:52 UTC - Matteo Merli: If the message is acked by one consumer, it will not be resent again ---- 2019-01-09 04:47:23 UTC - xuyi: @xuyi has joined the channel ---- 2019-01-09 05:16:32 UTC - Vincent Ngan: So, can I set ack timeout to zero? ---- 2019-01-09 05:19:22 UTC - Matteo Merli: That would mean to disable the ack timeout ---- 2019-01-09 05:19:38 UTC - Matteo Merli: You can set to a small time ---- 2019-01-09 05:21:49 UTC - Vincent Ngan: It this is that case, then it almost solves my problem. I will do some tests to verify it. Thanks a lot for your help. ---- 2019-01-09 05:23:04 UTC - Matteo Merli: :+1: ----
