2018-09-10 09:54:18 UTC - Matti-Pekka Laaksonen: The same thing happens when listing clusters. Only one in three requests returns my cluster name, others nothing ---- 2018-09-10 09:57:55 UTC - Matti-Pekka Laaksonen: Is the broker advertised address supposed to be the private subnet IP address the other nodes in the cluster can locate the host on? ---- 2018-09-10 09:58:21 UTC - Matti-Pekka Laaksonen: So in my case 10.60.1.5, 10.60.1.6, 10.60.1.7 for brokers ---- 2018-09-10 11:16:25 UTC - Matti-Pekka Laaksonen: Ok, seems to be that the main problem is with Zookeeper. When I run "sudo bin/pulsar zookeeper-shell -server <zk-instance-ip> ls /admin/clusters", only one zk instance returns the cluster name, others return an empty array ---- 2018-09-10 11:49:45 UTC - Matti-Pekka Laaksonen: Alright. There has been some confusion if the Zookeeper configuration file is "zoo.cfg" or "zookeeper.conf". I had it named "zoo.cfg", but a "zookeeper.conf" already exists in the /opt/pulsar/conf folder alongside the zoo.cfg. This caused Zookeeper to use the default configs ---- 2018-09-10 11:50:07 UTC - Matti-Pekka Laaksonen: I now corrected the config file name and everything works perfectly ---- 2018-09-10 15:26:49 UTC - Jon Bock: @Matti-Pekka Laaksonen Glad to see it’s now working for you. Let us know if there’s any additional help that you need. ---- 2018-09-10 18:41:46 UTC - Sravan Tatikonda: @Sravan Tatikonda has joined the channel ---- 2018-09-10 21:27:53 UTC - Natty: <!here> I’m trying to do a little bit of performance base-lining. I’m using a standalone cluster, which I know is going to be less than optimal, but was just trying to get some super basic numbers on what I might expect for throughput performance across a topic ---- 2018-09-10 21:28:03 UTC - Natty: and not sure what the fastest way is to just dump messages into a pulsar topic ---- 2018-09-10 21:28:25 UTC - Natty: i was trying the `pulsar-client produce` cli tool ---- 2018-09-10 21:28:31 UTC - Natty: but it seems to be very slow ---- 2018-09-10 21:28:48 UTC - Natty: taking ~4 seconds to write 1000 records ---- 2018-09-10 21:29:08 UTC - Sijie Guo: @Natty try pulsar-perf ---- 2018-09-10 21:29:26 UTC - Natty: ah! I missed that one ---- 2018-09-11 02:40:34 UTC - annette: @annette has joined the channel ---- 2018-09-11 06:17:56 UTC - Inilien: @Inilien has joined the channel ---- 2018-09-11 07:01:23 UTC - Inilien: Hi! After reading the docummentation and <http://streaml.io|streaml.io> posts I've got very excited about Apacha Pulsar. But from what I've gathered there is no easy way to use it with existing open source applications (such as Elasticsearch Filebeat for example). Mostly they provide support for Kafka but not for Pulsar. I read about Kafka Wrapper <https://pulsar.apache.org/docs/en/adaptors-kafka/> but, if only I am not very much mistaken, one is supposed to recompile each application by himself which is very cumbersome. Is it possible to add support for Kafka API in the Pulsar itself (maybe with some limitations)? I imagine this would ease the switch from Kafka to Pulsar quite a lot. ---- 2018-09-11 07:17:50 UTC - Sijie Guo: @Inilien
> But from what I’ve gathered there is no easy way to use it with existing open source applications (such as Elasticsearch Filebeat for example). We are concurrently growing the community. Expect to see more and more systems integrated with Pulsar soon. Regarding ElasticSearch, we have an elastic search connector just checked in, will be released as part of upcoming 2.2 release. > one is supposed to recompile each application by himself which is very cumbersome. yeah, current it requires the application to replace the kafka jar with the pulsar-kafka-wrapper jar. > Is it possible to add support for Kafka API in the Pulsar itself (maybe with some limitations)? There is actually a discussion of providing some sort of messaging gateway/proxy to support Kafka protocol. so Kafka applications can easily migrate to Pulsar without code change. The effort is still under discussion. We are not sure what is the community demand for this feature. If the community/user demand is high, we will prioritize it. ---- 2018-09-11 07:30:04 UTC - Inilien: > We are not sure what is the community demand for this feature. Is there an issue request on the github people can upvote? I coudn't find one. At least for me, this feature is the last barrier preventing me from choosing Pulsar over Kafka. ---- 2018-09-11 07:55:50 UTC - Sijie Guo: @Inilien the discussion was offline. let me create an issue and share it here. so people can comment on it. +1 : Inilien ---- 2018-09-11 08:06:54 UTC - Sijie Guo: @Inilien <https://github.com/apache/incubator-pulsar/issues/2556> +1 : Inilien ---- 2018-09-11 08:09:06 UTC - Sijie Guo: Here, if you have any ideas or requirements for Kafka support in Pulsar, it would be great that you can share your thoughts in the issue. so we can collect all the feedback. ----
