2019-03-05 10:10:02 UTC - Darragh: I've deployed pulsar following the deploy on aws guide and it seems that this does not create a management node of any kind. Is there a way for me to run the pulsar-admin commands and see the prometheus/grafana dashboards as such ---- 2019-03-05 10:20:42 UTC - Darragh: ok nvm I've found it ---- 2019-03-05 10:31:30 UTC - Darragh: hm I'm missing something though it seems. the python example at the bottom of the deployment on aws guide, gives me an error `2019-03-05 11:24:58.526 ERROR ClientImpl:179 | Error Checking/Getting Partition Metadata while creating producer on <persistent://public/default/test-topic> -- 5` ---- 2019-03-05 10:50:57 UTC - Darragh: when I ssh into a broker and try to run a pulsar-admin command I also get `Reason: javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)`. So something with the setup is clearly wrong, any clues as to what I should do ? ---- 2019-03-05 11:07:27 UTC - Maarten Tielemans: Working with Darragh on this. He deployed according to the Terraform/Ansible guide. It seems the Elastic Load Balancer sees the proxy service as being offline. ---- 2019-03-05 12:22:14 UTC - Maarten Tielemans: *the proxy service seems offline/does not expose port 8080 ---- 2019-03-05 13:21:45 UTC - Darragh: It also creates 9 instances instead of 6 as mentioned in the article. 1 for each bookie, 1 for each zk, 1 for each broker and one for a proxy. Not sure if this is intended and the documentation is out of date, or whether this is related to my problems ---- 2019-03-05 14:07:36 UTC - Maarten Tielemans: When the proxy service gets started, the following error is returned. Not immediately clear what causes this.
``` org.apache.pulsar.proxy.server.BrokerDiscoveryProvider - Failed to start Zookkeeper java.lang.IllegalArgumentException: A HostProvider may not be empty! ``` ---- 2019-03-05 16:52:04 UTC - Sébastien de Melo: Hi, we are trying to enable the functions worker on our broker deployed to EKS. We have set the following variables in the configuration: functionsWorkerEnabled: "true" PF_pulsarFunctionsCluster: pulsar and added bin/gen-yml-from-env.py conf/functions_worker.yml before running the broker but there is still an error message: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: Namespace missing local cluster name in clusters list: local_cluster=pulsar ns=public/functions clusters=[standalone] ---- 2019-03-05 16:54:23 UTC - Guy Feldman: are you using the helm chart or standard kubernetes files ---- 2019-03-05 16:54:44 UTC - Guy Feldman: also, did you assign a cluster to the public/functions namespace ---- 2019-03-05 17:01:43 UTC - Sébastien de Melo: standard kubernetes files ---- 2019-03-05 17:02:30 UTC - Sébastien de Melo: and I guess I have assigned a cluster, by adding PF_pulsarFunctionsCluster: pulsar to the broker config ---- 2019-03-05 17:02:49 UTC - Sébastien de Melo: or is there anything else to do? ---- 2019-03-05 17:03:37 UTC - Sébastien de Melo: in the pod conf/functions_worker.yml contains pulsarFunctionsCluster: pulsar as expected ---- 2019-03-05 17:46:06 UTC - Byron: @Matteo Merli Let me know if you need any other info ---- 2019-03-05 17:46:41 UTC - Matteo Merli: No thanks, I’m going to get back to the go tests today ---- 2019-03-05 19:06:42 UTC - Alexandre DUVAL: Hey, are messages properties or message key available as filters on consume or something like this? ---- 2019-03-05 19:09:41 UTC - Ali Ahmed: @Alexandre DUVAL pulsar has concept of properties which are defined as a key value pair of tags ---- 2019-03-05 19:13:34 UTC - Guy Feldman: hmm. there shouldn't be ---- 2019-03-05 19:15:18 UTC - Guy Feldman: I'm assuming the clusterName: is set to pulsar ---- 2019-03-05 19:15:44 UTC - Alexandre DUVAL: @Ali Ahmed yes, i defined them in my producer, but dunno how use them as consumer or as Pulsar fonction input. ---- 2019-03-05 19:24:37 UTC - Matteo Merli: when you get a message from the consumer, you can access the properties set by the producer ---- 2019-03-05 22:20:41 UTC - Chris DiGiovanni: I want to setup Tiered Storage using our Ceph Rados GW (S3 Compatible). This has worked for most 3rd party tools. From the docs I would assume I would populate `s3ManagedLedgerOffloadServiceEndpoint=` with our Radow Gateway and leave `s3ManagedLedgerOffloadRegion=` blank. Though I'm thinking the Broker is going to react poorly to our internally signed certs. How do I get around this problem? Looking at the jclouds docs it looks like it is possible to just trust the certs <https://jclouds.apache.org/guides/aws-s3/> Though how does this setting translate to the broker.conf? ---- 2019-03-05 22:46:41 UTC - Vikas: @Vikas has joined the channel ---- 2019-03-05 22:52:56 UTC - Vikas: <!here> hi Folks, I am trying to connect NiFi's PublishPulsar processor to the SSL authenticated Pulsar topic but getting an error as below. I am able to connect and publish messages to the Pulsar 2.3 that I deployed on my local machine. ---- 2019-03-05 22:53:21 UTC - Grant Wu: Please refrain from using @ here unnecessarily ---- 2019-03-05 22:53:32 UTC - David Kjerrumgaard: @Vikas Have you configured the SSL_CONTEXT_SERVICE inside the pulsar client service? ---- 2019-03-05 22:53:35 UTC - Vikas: ---- 2019-03-05 22:54:01 UTC - Vikas: thanks @David Kjerrumgaard for the quick reply, yes I have ---- 2019-03-05 22:55:03 UTC - Vikas: ---- 2019-03-05 22:55:33 UTC - David Kjerrumgaard: @Vikas I meant inside NiFi. :smiley: ---- 2019-03-05 22:56:25 UTC - Vikas: oh ok, you mean inside of StandardPulsarClientService? +1 : David Kjerrumgaard ---- 2019-03-05 22:56:51 UTC - David Kjerrumgaard: so the processors require a PulsarClientService. and inside that you can attach a NiFi ssl context service which will use your certificates to make the connection to the Pulsar broker ---- 2019-03-05 22:57:03 UTC - Vikas: sure, sorry. pray : Grant Wu ---- 2019-03-05 22:58:12 UTC - Vikas: oh ok, thanks David. let me give a try :+1: ---- 2019-03-05 22:58:21 UTC - David Kjerrumgaard: no problem ---- 2019-03-05 23:00:34 UTC - David Kjerrumgaard: Add the context service to the StandardPulsarClientService as shown ---- 2019-03-05 23:07:59 UTC - Vikas: sure, thank you so much. I will fill in the necessary details in the StandardRestrictedSSLContextService and will give a try. :party-parrot: ---- 2019-03-06 02:11:57 UTC - Matteo Merli: @Byron I have created a PR at <https://github.com/apache/pulsar/pull/3764> ---- 2019-03-06 02:14:14 UTC - Byron: Great, thanks for digging into it ---- 2019-03-06 02:14:25 UTC - Matteo Merli: Thanks for reporting ! ---- 2019-03-06 02:47:35 UTC - Byron: I have a deployment/arch question. Given that a pulsar “instance” may contain many “clusters”, I presume it is common to have clusters designated to specific environments (e.g. prod, stage, dev)? ---- 2019-03-06 02:48:13 UTC - Byron: That seems like the most obvious approach, but was curious of any insight for those who are doing this in practice. ---- 2019-03-06 02:48:29 UTC - crisis: @crisis has joined the channel ---- 2019-03-06 04:16:06 UTC - Vincent Ngan: The exploration of Pulsar for use in my applications leads me to know about Zookeeper and I discover that there could be many good use cases of Zookeeper for my applications. So, if I use Pulsar for my applications, is it okay to also use the underlying Zookeeper directly for other purposes in my applications? If it is so, is there anything I need to be careful about to avoid affecting the normal functioning of Pulsar? ---- 2019-03-06 04:17:02 UTC - Ali Ahmed: @Vincent Ngan you can make sure you plan for enough zk capacity and failover ---- 2019-03-06 05:09:06 UTC - Matteo Merli: Suggestion typically is to separate completely staging and prod, as no data should be flowing between these 2 clusters for any reason ---- 2019-03-06 05:12:03 UTC - Matteo Merli: You can watch the ZK stats in terms of writes/s and dataset size. If these are limited or anyway are not overloading the zk machines, there should be no major problem. What you want to avoid is to have other applications that you don’t control directly having access to zk cluster, because there you wouldn’t have any way to stop the clients ---- 2019-03-06 06:09:51 UTC - Vincent Ngan: Thanks for your advice. ---- 2019-03-06 06:10:19 UTC - Shivji Kumar Jha: @Sijie Guo Is this package scoped on purpose? <https://github.com/apache/pulsar/blob/3c36705f233c73cef96572eba3a2f3ec69fc5bea/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroSchema.java> ---- 2019-03-06 06:11:30 UTC - Sijie Guo: yes. ---- 2019-03-06 06:12:37 UTC - Sijie Guo: currently GenericSchema is not directly exposed. They are used by AUTO_CONSUME schema. where the client will detect the schema and using the schema to deserialize the messages. ---- 2019-03-06 06:13:58 UTC - Shivji Kumar Jha: I actually need something like this: <https://github.com/apache/pulsar/commit/3c36705f233c73cef96572eba3a2f3ec69fc5bea#diff-b4407166a9949a76e0c2453ddc2cf82f> Is this available in a snapshot binary? 2.4.0.-SNAPSHOT? ---- 2019-03-06 06:14:03 UTC - Sijie Guo: there are pull requests around this area such as multi-versioning, schema builder, generic record for fully completing the story around generic schema. @Penghui Li @CongBo and me are working on that area. ---- 2019-03-06 06:14:29 UTC - Sijie Guo: AUTO_CONSUME is available in 2.3.0. ---- 2019-03-06 06:14:59 UTC - Sijie Guo: but it is only using the latest schema to deserialize the events. ---- 2019-03-06 06:15:05 UTC - Shivji Kumar Jha: AUTO_CONSUME doesnt work with multi version schema on a topic though.. Maybe i need to compile the latest client code and use that ---- 2019-03-06 06:15:09 UTC - Sijie Guo: so there are some limitations. ---- 2019-03-06 06:15:12 UTC - Shivji Kumar Jha: exactly ---- 2019-03-06 06:16:44 UTC - Shivji Kumar Jha: I wish I could fetch the schema registry by passing the version and topic, perhaps. That not doable right? ---- 2019-03-06 06:16:51 UTC - Sijie Guo: @Shivji Kumar Jha I would suggest you wait for a few more days. @Penghui Li just had a multi-version provider for generic schema and he need to wire that up with AUTO_CONSUME. There are also a task to improve current POJO schema to leverage this multi versioning. ---- 2019-03-06 06:18:51 UTC - Shivji Kumar Jha: ack ---- 2019-03-06 08:05:42 UTC - Yuvaraj Loganathan: Also there are predefined zk paths pulsar . Make sure you don't overwrite any of those data. ---- 2019-03-06 08:16:23 UTC - Sébastien de Melo: Yes it is ----
