2018-06-20 10:02:12 UTC - Goloshchapov Egor: Collegues, could you advise me please, how can I get topic name when I'm using wildcard subscription? ---- 2018-06-20 10:09:24 UTC - Ali Ahmed: @Goloshchapov Egor are you trying to get topic name from a message in a wildcard subscription ? ---- 2018-06-20 10:25:47 UTC - Ivan Kelly: @Goloshchapov Egor You have to cast the message id to TopicMessageIdImpl ---- 2018-06-20 11:05:07 UTC - Goloshchapov Egor: I didn't try it, but will try it now. I just checked the API and found out that there is no 'topic' there , and was confused ---- 2018-06-20 11:05:34 UTC - Goloshchapov Egor: Do you think it would make sense to have that information in separate method? (to avoid casting) ---- 2018-06-20 11:16:28 UTC - Ivan Kelly: I think so, you're the second person to ask this week ---- 2018-06-20 11:16:38 UTC - Ivan Kelly: Could you create an issue on github? ---- 2018-06-20 14:48:43 UTC - Sijie Guo: @Idan you would use similar approach as 1) when using #receive ---- 2018-06-20 16:51:54 UTC - Daniel Ferreira Jorge: Hi, I'm trying to achieve some sort of atomicity when publishing to 2 or more different topics (a consumer will read a message, split it into 2 different messages and publish them into separate topics "atomically"). Is there a way I can achieve that? Maybe not a pulsar feature by itself, but a pattern of some sort... ---- 2018-06-20 16:53:21 UTC - Matteo Merli: One option could be to “not split the message” but have 2 subscription (and 2 consumers) to pick it up, each performing a different action on the message ---- 2018-06-20 16:58:43 UTC - Daniel Ferreira Jorge: Yes, but the whole task is a very complex one, with various steps (imagine a binary tree). To do that in this manner, I would need many (over 20, instead of just 2) consumers that are there just there to produce to new topics. Are there any other options? ---- 2018-06-20 17:00:45 UTC - Daniel Ferreira Jorge: I have to enrich and split the message into 9 different topics then the consumer of one of this topics will enrich and split the message in further 13 topics ---- 2018-06-20 17:03:53 UTC - Matteo Merli: not sure if it really applies to your case, but I was referring to avoid the republish for the splitting. eg: * you get 1 message (that contains multiple parts) * You can enrich the message and republish a single, new enriched message * Multiple subscribers will get the message and process a subset of it ---- 2018-06-20 17:05:03 UTC - Matteo Merli: Regarding “atomicity”, what’s the main goal? Prevent half-committed data to be visible to consumers or just making sure the message is published on all the topics? ---- 2018-06-20 17:07:40 UTC - Daniel Ferreira Jorge: Making sure that the message is published in all topics ---- 2018-06-20 17:09:18 UTC - durga: Hi @Alex Bradbury - We did not do the prototype yet. It is in the pipeline but other things came up that needed attention first. If you do then please let me know. ---- 2018-06-20 17:09:57 UTC - Matteo Merli: In this case, if you’re consuming from a topic, delaying the acknowledgement on the 1st topic will guarantee that message will be replayed unless it was successfully published on all downstream topics.
To avoid duplicates, you can also enable de-deduplication and derive the sequenceId from the received `MessageId` ---- 2018-06-20 17:13:52 UTC - Daniel Ferreira Jorge: Oh, that is great! So simple... ---- 2018-06-20 17:14:00 UTC - Daniel Ferreira Jorge: thank you @Matteo Merli +1 : Matteo Merli, Ali Ahmed ---- 2018-06-20 17:53:27 UTC - Karthik Palanivelu: Team, I found the below exception when I start the autorecovery in k8s based on the bookie.yaml. Please advice how to fix it. ---- 2018-06-20 17:53:42 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FBASDSK9S/-.txt|Untitled> ---- 2018-06-20 18:25:09 UTC - Sijie Guo: @Karthikeyan Palanivelu this is info logging, so nothing is wrong there. it just means client (autorecovery) doesnt configure a network topology script for using rackware placement policy. it raises a warning message but fallback to use default placement policy. rackware is useful when you deploy to a on-prem cluster. but in k8s environment, there is no real *rack* information that you can use. so that is a fine warning logging. you can ignore it. I will improve the logging ---- 2018-06-20 18:45:30 UTC - Karthik Palanivelu: Cool Thank You @Sijie Guo ---- 2018-06-20 19:50:56 UTC - Karthik Palanivelu: @Ali Ahmed @Sijie Guo @Matteo Merli As recommended yesterday, I was able to get the pulsar send and consume messages without SSL in K8s. Now I am going to enable SSL. I have the config to turn on in proxy and broker. Both Proxy and Broker is configured with same broker cert. Please confirm that proxy would be a pass through to broker to forward the same client cert or will it terminate the client and sends cert configured in proxy in the request to broker? ---- 2018-06-20 19:52:09 UTC - Matteo Merli: you can configure it in either way, both having TLS terminated at proxy or having proxy to re-encrypt when talking to broker ---- 2018-06-20 20:15:41 UTC - Karthik Palanivelu: @Matteo Merli so broker container will have TLS disabled in broker.conf if I use tlsEnabledWithBroker=false in proxy.conf? ---- 2018-06-20 21:23:29 UTC - Yuwei Jiang: Hi guys, I’ve encountered some issue when trying pulsar 2.0 admin REST APIs. I’ve deployed a pulsar 2.0 cluster on K8s, with docker image `apachepulsar/pulsar:2.0.1-incubating` . I have another pod in my K8s that is making requests to broker REST APIs. When trying to create a tenant, I received a 404 error: ``` 21:02:38.805 [pulsar-web-27-10] INFO org.eclipse.jetty.server.RequestLog - 100.96.3.6 - - [20/Jun/2018:21:02:38 +0000] "PUT //broker.c8.svc.cluster.local:8080/admin/tenants/_mm HTTP/1.1" 404 334 "-" "python-requests/2.18.4" 0 ``` But if I change the API to use `admin/properties` , I was able to create the tenant: ``` 21:03:23.307 [pulsar-web-27-21] INFO org.eclipse.jetty.server.RequestLog - 100.96.3.6 - - [20/Jun/2018:21:03:23 +0000] "PUT //broker.c8.svc.cluster.local:8080/admin/properties/_mm HTTP/1.1" 204 0 "-" "python-requests/2.18.4" 6 21:03:23.317 [pulsar-web-27-3] INFO org.eclipse.jetty.server.RequestLog - 100.96.3.6 - - [20/Jun/2018:21:03:23 +0000] "GET //broker.c8.svc.cluster.local:8080/admin/namespaces/_mm HTTP/1.1" 200 2 "-" "python-requests/2.18.4" 3 ``` However, when try to create namespace using `PUT /admin/namespaces/:tenant/:namespace` I received a 405 error ``` 21:03:23.326 [pulsar-web-27-10] INFO org.eclipse.jetty.server.RequestLog - 100.96.3.6 - - [20/Jun/2018:21:03:23 +0000] "PUT //broker.c8.svc.cluster.local:8080/admin/namespaces/_mm/c8local HTTP/1.1" 405 354 "-" "python-requests/2.18.4" 1 ``` I can create namespace via CLI: ``` $ kubectl --kubeconfig=us-west-1 -n c8 exec -it pulsar-admin -- bin/pulsar-admin namespaces create _mm/c8local $ kubectl --kubeconfig=us-west-1 -n c8 exec -it pulsar-admin -- bin/pulsar-admin namespaces list _mm _mm/c8local ``` Could you point out what I’m missing? Thanks. ---- 2018-06-20 21:44:04 UTC - Matteo Merli: @Yuwei Jiang the `properties` were renamed into `tenants` in Pulsar 2.0 . The correct path for the REST API would be `/admin/v2/tenants/_mm` When creating a namespace, the namespace needs to contain the tenant name as well. Eg. the REST path would be : `/admin/v2/namespaces/_mm/c8local` ---- 2018-06-20 21:55:55 UTC - Karthik Palanivelu: Hi, Can you please layout how TLS work in k8s from Client to Proxy to Broker with Certificates CN to understand better to troubleshoot my issue of getting Connection Closed? I saw super user is missing in proxy. Please help me here. ---- 2018-06-20 22:19:55 UTC - Yuwei Jiang: @Matteo Merli Thanks for the response! Adding the missing `v2` resolved this issue. ---- 2018-06-21 00:25:57 UTC - Karthik Palanivelu: Team, Adding to the above question, how can I configure proxy as just a pass through? ---- 2018-06-21 00:32:54 UTC - Sijie Guo: @Karthik Palanivelu If I understand this correctly, you want proxy to pass through tls handshake stuffs, right? I am not sure currently it is supported. what I understand, proxy you can enable tls with connections connecting to the proxy, enable tls on connections going to the broker, but I don’t think there is any tls pass through here. ---- 2018-06-21 00:33:03 UTC - Sijie Guo: (@Matteo Merli can correct me if I am wrong) ---- 2018-06-21 00:44:43 UTC - Matteo Merli: Correct, proxy has to terminate the TLS connection in any case, since it has to take part in the initial handshake (in Pulsar protocol) between client and broker. After that it just becomes a simple TCP proxy. ---- 2018-06-21 00:58:10 UTC - Karthik Palanivelu: Thanks @Sijie Guo @Matteo Merli If I understand correctly I need to perform handshake at Proxy level and execute the connection on proxy to broker as non-TLS. If Can I able to authorize the client at proxy level for the topic/namespace permission? Based on your answer, If I need TLS between Proxy and Broker I need a separate Client Certificate? If so how do I pass the CN from client to authorize at broker level? ---- 2018-06-21 01:00:35 UTC - Matteo Merli: the proxy will perform the authorization and will also pass the client crendential to the broker who will authenticate both client and proxy ---- 2018-06-21 01:04:10 UTC - Karthik Palanivelu: So I need a Client Cert for Proxy and Pass Authorization object to broker? ---- 2018-06-21 01:04:34 UTC - Matteo Merli: correct ---- 2018-06-21 01:06:44 UTC - Karthik Palanivelu: Can you please help fill in the details for the below params? And Please help me know which param I need to enable for the Proxy-Broker. ``` # Authentication settings of the proxy itself. Used to connect to brokers brokerClientAuthenticationPlugin= brokerClientAuthenticationParameters= brokerClientTrustCertsFilePath= ``` ---- 2018-06-21 01:08:43 UTC - Karthik Palanivelu: Or is it similar to Client.conf? ---- 2018-06-21 07:17:52 UTC - Goloshchapov Egor: @Ivan Kelly Hello Ivan. Issue regarding topic name in message is created. Thank you in advance. <https://github.com/apache/incubator-pulsar/issues/2007> ---- 2018-06-21 09:00:00 UTC - Ivan Kelly: @Goloshchapov Egor thanks ---- 2018-06-21 09:03:13 UTC - Ivan Kelly: @Karthikeyan Palanivelu proxy -> broker TLS currently does not work (at least i've not been able to make it work). We have an issue open (<https://github.com/apache/incubator-pulsar/issues/1991>). Someone should look into it this week (it may work if you explicitly specify the broker url, but I haven't tried that) ----
