2018-05-18 14:41:20 UTC - Igor Zubchenok: *MEMORY LEAKS?* Hello. We have released our server with pulsar v1.22 client and we got new memory leaks. Do you have any reports on memory leaks in 1.22?
At production our java app used 30GB of server memory, but memory dump of our java app was only 4GB. FYI: I once noticed a WARN/ERROR in console (from some kind of LeakDetector) that some Netty buffer is not correctly refcounted. Unfortunately I did not copied the message. ---- 2018-05-18 14:56:21 UTC - Igor Zubchenok: We've added Pulsar logo at the bottom of page: <https://taxistartup.com/> :laughing: +1 : Matteo Merli, Sijie Guo, Nicolas Ha, Ali Ahmed ---- 2018-05-18 15:12:16 UTC - Artem Shaban: @Artem Shaban has joined the channel ---- 2018-05-18 15:19:14 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FASC2AABC/-.txt|Untitled> and commented: Hi All, I am running a zk container using 2181, a bookie container with host HostName:3181. When I start a broker, I am getting the above exception. If I query the container for bookies, there is one bookie available by using cmd `./bookkeeper shell listbookies -rw`. Can you please help? ``` 11:12:57.577 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. ReadWrite Bookies : localhost(XX.XX.XX.XX):3181 11:12:57.642 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x16373b4222a0009 closed 11:12:57.648 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x16373b4222a0009 ``` ---- 2018-05-18 15:25:23 UTC - Karthik Palanivelu: Below are my bookies config in broker.conf ``` # Enable rack-aware bookie selection policy. BK will chose bookies from different racks when # forming a new bookie ensemble bookkeeperClientRackawarePolicyEnabled=true # Enable bookie isolation by specifying a list of bookie groups to choose from. Any bookie # outside the specified groups will not be used by the broker bookkeeperClientIsolationGroups= ### --- Managed Ledger --- ### # Number of bookies to use when creating a ledger managedLedgerDefaultEnsembleSize=1 # Number of copies to store for each message managedLedgerDefaultWriteQuorum=1 # Number of guaranteed copies (acks to wait before write is complete) managedLedgerDefaultAckQuorum=1 ``` ---- 2018-05-18 15:53:40 UTC - Karthik Palanivelu: Any help Team, I am stuck here.. ---- 2018-05-18 15:56:08 UTC - Sijie Guo: @Karthikeyan Palanivelu this looks like failed to bind to an IP address. What command are you running with the docker? ---- 2018-05-18 15:59:02 UTC - Karthik Palanivelu: I have 3 docker for ZK, Bookie and Broker. This is a custom image I am creating based on my company policy. Bookie and Broker advertisedAddress is same as my HostName. I turned on the Property on bookie `useHostNameAsBookieID=true` and advertisedAddress as hostname passing through command line. Do I need to make any property update to broker.conf? ---- 2018-05-18 16:01:22 UTC - Karthik Palanivelu: I have issue only with broker, any property to concentrate on? ---- 2018-05-18 16:03:40 UTC - Sijie Guo: Are you using advertisedAddress for both bookie and broker? Or only broker and bookie is using useHostNameAsBookieID ---- 2018-05-18 16:10:07 UTC - Karthik Palanivelu: I used for Both First and I removed it from Broker ---- 2018-05-18 16:11:47 UTC - Karthik Palanivelu: Reason for Bookie is to have multiple bookies on same host with different port as we discussed yesterday ---- 2018-05-18 16:16:24 UTC - Karthik Palanivelu: Now I reomoved advertisedAddress from both and still getting same issue... ---- 2018-05-18 16:17:51 UTC - Matteo Merli: @Karthikeyan Palanivelu one question, it appears the broker is failing to start because of a port conflict. Are you running the broker in a different container or just in the host machine? ---- 2018-05-18 16:18:03 UTC - Matteo Merli: Is ZK also running on the host machine? ---- 2018-05-18 16:18:25 UTC - Karthik Palanivelu: Yes, I am starting ZK, Bookie and Broker on my Mac ---- 2018-05-18 16:18:56 UTC - Matteo Merli: Problem could be related to port 8080. That’s what is being used also for ZK metrics ---- 2018-05-18 16:19:16 UTC - Matteo Merli: we have added Prometheus exporter in the ZK server ---- 2018-05-18 16:19:27 UTC - Karthik Palanivelu: hmm..Where would I change it ---- 2018-05-18 16:19:32 UTC - Matteo Merli: broker also tries to use that port. ---- 2018-05-18 16:19:33 UTC - Matteo Merli: one sec ---- 2018-05-18 16:20:03 UTC - Karthik Palanivelu: but I am not exposing that in ZK Docker Run ---- 2018-05-18 16:20:29 UTC - Matteo Merli: ok, so ZK is running inside docker? and broker outside? ---- 2018-05-18 16:21:18 UTC - Karthik Palanivelu: All are running as separate containers in my Mac ---- 2018-05-18 16:21:22 UTC - Karthik Palanivelu: Same Host ---- 2018-05-18 16:21:26 UTC - Vasily Yanov: @Karthikeyan Palanivelu I had the same. ---- 2018-05-18 16:21:48 UTC - Vasily Yanov: it was fixed with listeningInterface=enp3s0 in bookkeeper.conf ---- 2018-05-18 16:22:06 UTC - Karthik Palanivelu: Let me Try it ---- 2018-05-18 16:22:16 UTC - Matteo Merli: ok, then I don’t understand why broker should get port binding error, inside the container ---- 2018-05-18 16:22:58 UTC - Vasily Yanov: but I'm not sure about docker deployment. We are using pulsar on bare-metal without docker ---- 2018-05-18 16:24:26 UTC - Vasily Yanov: as a result in ZK (with bookkeeper shell listbookies -rw) you have to see your non localhost addresses ---- 2018-05-18 16:24:58 UTC - Karthik Palanivelu: It did not solve it for me.. ---- 2018-05-18 16:25:02 UTC - Karthik Palanivelu: Same error ---- 2018-05-18 16:25:40 UTC - Karthik Palanivelu: Let me post my docker cmds ---- 2018-05-18 16:31:57 UTC - Matteo Merli: sure, in any case, for reference, the HTTP port for ZK metrics can be changed by adding `-Dstats_server_port=1234` to the JVM options ---- 2018-05-18 16:36:02 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FATF6EG79/-.sh|Untitled> and commented: My docker run for ZK, Bookie and Broker differentiated via NodeType. I included Cluster Metadata as well... ---- 2018-05-18 16:36:17 UTC - Karthik Palanivelu: I am running from my Mac ---- 2018-05-18 16:42:07 UTC - Matteo Merli: I don’t see anything obvious. Indeed if the port is not exposed in the ZK container, it shouldn’t affect the broker ---- 2018-05-18 16:42:58 UTC - Matteo Merli: do you have a custom script to start the broker inside the container? ---- 2018-05-18 16:43:25 UTC - Matteo Merli: is it possible there something else running on 8080 inside container? ---- 2018-05-18 16:52:24 UTC - Martin Kunev: @Martin Kunev has joined the channel ---- 2018-05-18 17:14:07 UTC - Sijie Guo: @Matteo Merli I am not sure “Cannot assign requested address” is port conflict. It seems more be can’t listen on or bound to the ip address / hostname ---- 2018-05-18 17:14:53 UTC - Matteo Merli: unfortunately from netty exception we don’t have more details :confused: ---- 2018-05-18 17:15:09 UTC - Karthik Palanivelu: I checked nothing is running on port 8080 and moreover I changed the Ports as well to check it. It fails all the time on same issue ---- 2018-05-18 17:16:36 UTC - Sijie Guo: @Karthikeyan Palanivelu: is the goal to run the cluster in your laptop, do you need to let it being access outside? ---- 2018-05-18 17:16:42 UTC - Matteo Merli: last think I can think of: there’s a `bindAddress` parameter for broker. Default is `0.0.0.0` ---- 2018-05-18 17:18:19 UTC - Ali Ahmed: @Karthikeyan Palanivelu my understanding is you can’t get this to work in mac with out a segregated docker compose network. ---- 2018-05-18 17:18:49 UTC - Ali Ahmed: docker containers are not aware of osx host , they are running inside a linux vm on osx ---- 2018-05-18 17:19:25 UTC - Ali Ahmed: you need to bind the containers to a docker network and link them ---- 2018-05-18 17:20:50 UTC - Sijie Guo: @Ali Ahmed I think network is a separated issue though. the problem here is its broker container can’t bind to an address ---- 2018-05-18 17:24:36 UTC - Karthik Palanivelu: @Sijie Guo Not exposing to outside world but would like to test the image I am creating and test whether it works ---- 2018-05-18 17:24:45 UTC - Sijie Guo: @Karthikeyan Palanivelu I am trying similar commands in my laptop now ---- 2018-05-18 17:25:20 UTC - Karthik Palanivelu: I just Packaged the 2.0-rc1 within the docker on RHEL base Image along with jdk1.8. ---- 2018-05-18 17:31:28 UTC - Karthik Palanivelu: All, Just found the issue. I was assigning the bindaddress with HostName. I will test further and will keep posted. Thanks So Much - @Sijie Guo @Matteo Merli @Ali Ahmed ---- 2018-05-18 17:31:52 UTC - Matteo Merli: :+1: ---- 2018-05-18 17:31:53 UTC - Sijie Guo: great ---- 2018-05-18 17:32:48 UTC - Ali Ahmed: I think from pulsar’s side we could improve logging ---- 2018-05-18 17:36:51 UTC - Sijie Guo: I don’t think pulsar has enough knowledge to logging. the knowledge is mostly aware only by netty. so we might consider improving netty logging :slightly_smiling_face: ---- 2018-05-18 17:38:05 UTC - Matteo Merli: we should be able to attach the address context when calling async bind on netty, though we would have to do that for each call ---- 2018-05-18 18:25:26 UTC - Matteo Merli: @Karthikeyan Palanivelu @Sijie Guo Added a bit of context there: <https://github.com/apache/incubator-pulsar/pull/1811> +1 : Karthik Palanivelu ---- 2018-05-18 18:26:50 UTC - Ali Ahmed: I think it’s a problem of netty logging level Host and port should be printed by netty ``` 15:36:51 [WARNING] Could not bind to host /91.134.252.235:25565io.netty.channel.unix.Errors$NativeIoException: bind() failed: Cannot assign requested address at io.netty.channel.unix.Errors.newIOException(Errors.java:117) at io.netty.channel.unix.Socket.bind(Socket.java:234) at io.netty.channel.epoll.EpollServerSocketChannel.doBind(EpollServerSocketChannel.java:91) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:554) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1258) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:512) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:497) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:980) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:250) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:363) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:312) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) at java.lang.Thread.run(Thread.java:745) ``` ---- 2018-05-18 18:27:51 UTC - Matteo Merli: they should but they are not. Since the workaround is easy, we should do it ---- 2018-05-18 18:27:56 UTC - Ali Ahmed: ok ---- 2018-05-18 18:32:24 UTC - Karthik Palanivelu: Good News, I am able to test now the Multi-Tenant using Docker Containers. +1 : Ali Ahmed, Sanjeev Kulkarni ---- 2018-05-18 18:32:28 UTC - Karthik Palanivelu: Thanks All ---- 2018-05-18 18:41:58 UTC - Karthik Palanivelu: @Sijie Guo @Matteo Merli Referring to yesterday's discussion on running multiple bookies on same host, I passed the hostname of host as advertisedAddress with different port numbers of 3181 and 3182 to docker run cmd. But even after that I am getting the InvalidCookieException on that hostname. Can you please help? ---- 2018-05-18 18:46:11 UTC - Matteo Merli: Did you configured the 2 bookie instances to use different paths on the disk? ---- 2018-05-18 18:48:04 UTC - Karthik Palanivelu: Good Point, Let me check that ---- 2018-05-18 18:50:41 UTC - Karthik Palanivelu: Got it Working.. +1 : Sanjeev Kulkarni ---- 2018-05-18 19:18:13 UTC - Martin Kunev: Hi people, I'm new here and I'm using pulsar for a project I'm working on. I have a couple of questions regarding pulsar topics. * I wanted to use non-persistent topics with C++, but I see that the pulsar-cpp-client doesn't allow them (TopicName::validate). Is there any particular reason for this? How easy would it be to change this? * When trying to delete a topic in the global namespace, I get 403 "Delete forbidden on global namespace". However in the latest commit there seems to be a force flag to override this. The name "force" implies that it's dangerous to do so. What issues can arise when deleting a topic in the global namespace with force? ---- 2018-05-18 19:22:23 UTC - Matteo Merli: > * I wanted to use non-persistent topics with C++, but I see that the pulsar-cpp-client doesn’t allow them (TopicName::validate). Is there any particular reason for this? How easy would it be to change this? Yes, that was an oversight, C++ lib was not updated after non-persistent topics were introduced. Only the validation needs to be fixed, there is no other change required in client library. ---- 2018-05-18 19:25:34 UTC - Matteo Merli: > When trying to delete a topic in the global namespace, I get 403 “Delete forbidden on global namespace”. However in the latest commit there seems to be a force flag to override this. The name “force” implies that it’s dangerous to do so. What issues can arise when deleting a topic in the global namespace with force? The delete on a replicated topic is “tricky” because it involved deleting the topic in multiple data centers and if there are ongoing network partitions it might get funky. There are 2 ways to get rid of a replicated topic: * Remove the replication clusters and reduce it to a single cluster. At that point the delete will succeed * Just stop using the topic and “unsubscribe” all subscription. There is a mechanism that will make sure the topic is eventually deleted ---- 2018-05-18 19:29:14 UTC - Martin Kunev: Thank you very much! ---- 2018-05-18 19:46:40 UTC - Ali Ahmed: @Igor Zubchenok is your memory issue resolved ? ---- 2018-05-18 19:59:13 UTC - Martin Kunev: One more thing - if I want to delete a global namespace, I have to first either force-delete all topics in it or wait for them to be deleted? ---- 2018-05-18 20:00:22 UTC - Matteo Merli: Yes, it doesn’t do the overall cleanup yet (it would just complain that namespace is not empty) ---- 2018-05-18 20:08:44 UTC - Adam Williams: Hello - I'd appreciate an assist with enabling TLS on the standalone Pulsar (1.22). I've followed the steps in the docs to create the required certs and modified the `standalone.conf` to enable TLS. When running, I get the following error: ``` 2018-05-18 14:04:03,133 - ERROR - [main:PulsarService@350] - Private key must be accompanied by certificate chain java.lang.IllegalArgumentException: Private key must be accompanied by certificate chain at java.security.KeyStore.setKeyEntry(KeyStore.java:1136) at org.apache.pulsar.common.util.KeyStoreHolder.setPrivateKey(KeyStoreHolder.java:55) at org.apache.pulsar.common.util.SecurityUtility.createSslContext(SecurityUtility.java:144) at org.apache.pulsar.common.util.SecurityUtility.createSslContext(SecurityUtility.java:71) at org.apache.pulsar.broker.web.WebService.<init>(WebService.java:95) at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:276) at org.apache.pulsar.PulsarStandaloneStarter.start(PulsarStandaloneStarter.java:164) at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:209) ``` ---- 2018-05-18 21:57:16 UTC - Ali Ahmed: @Adam Williams can you summarize the steps you have done ---- 2018-05-18 21:58:36 UTC - Adam Williams: Hi @Ali Ahmed - Thanks, but I was able to get it working in the end. It required generating the root CA in a different way than the documentation (I'm on OSX). I can followup once I'm sure everything works. ---- 2018-05-18 21:59:02 UTC - Matteo Merli: :+1: ---- 2018-05-18 22:26:30 UTC - m.ali.r: @m.ali.r has joined the channel ---- 2018-05-19 07:56:53 UTC - Poule: is it possible to publish a message that will be treated in the future? ---- 2018-05-19 07:57:12 UTC - Poule: for retry stuff ---- 2018-05-19 07:58:56 UTC - Ali Ahmed: Hi @Poule can you clarify what you mean ? ---- 2018-05-19 07:59:28 UTC - Ali Ahmed: pulsar persists message you can consume previous messages ---- 2018-05-19 08:05:56 UTC - Poule: something like publish(event, delay=30) that returns immediately but the event is delivered into pulsar only 30 seconds later ---- 2018-05-19 08:06:54 UTC - Ali Ahmed: you can publish events async into pulsar , we have a future’s based api ---- 2018-05-19 08:07:13 UTC - Poule: pulsar would queue the event during 30 sec. before actually processing it ---- 2018-05-19 08:08:09 UTC - Ali Ahmed: it should be doable, essentially you would adding a delay to the completable future ---- 2018-05-19 08:08:23 UTC - Ali Ahmed: but this would be client side ---- 2018-05-19 08:11:41 UTC - Ali Ahmed: may I ask what exactly is the use case for the delay ? ---- 2018-05-19 08:12:54 UTC - Poule: i call a Rest API when I read an event. if the REst API fails, I'd like to reschedule the event to be reprocessed in 30 seconds ---- 2018-05-19 08:15:13 UTC - Ali Ahmed: I would model it differently, depending on the exact use I think a dedicated retry topic might be a better choice, this should be handled on the consumer side ---- 2018-05-19 08:15:29 UTC - Poule: ok ---- 2018-05-19 08:16:23 UTC - Ali Ahmed: in my mind consumer reads events as fast as possible tries api call , if fails commits to retry topic ---- 2018-05-19 08:17:02 UTC - Ali Ahmed: retry consumer is separate gets event waits a while and then tries again ---- 2018-05-19 08:19:29 UTC - Poule: ok i will take that idea in consideration ----
