2018-12-09 09:15:14 UTC - Christophe Bornet: @Beast in Black As @Matteo Merli pointed out, try `my-pulsar-broker-.*`. This should work. ---- 2018-12-09 13:05:04 UTC - Samuel Sun: one question for the pulsar proxy and pulsar service_discovery , technically proxy has the whole feature for the service_discovery ? ---- 2018-12-09 13:08:11 UTC - legendtkl: Hello, when I follow this guide <https://pulsar.apache.org/docs/en/client-libraries-python/> to *Installing from source* , I got the error: error: [Errno 2] No such file or directory: ‘_pulsar.so’ . Any suggestion? ---- 2018-12-09 13:35:20 UTC - Mike Card: In my current application I have two threads (Java Runnable) that run in parallel which each have a producer they call using the synchronous send API. A stress test application calls my application at an aggregate write rate of about 15KHz, and during these writes the producers send messages into a topic which get truncated at 64 bytes in length. This ultimately results in a downstream buffer underflow exception that kills the consumer tasks. If I stop everything and restart the application, upon restart it will resume fetching messages from the topic and immediately die with the buffer underflow exception, indicating that the messages in the topic really are only 64 bytes in length. So after a stress test run like this I delete the topics as well so they will be re-created and no truncated messages will be present. ---- 2018-12-09 16:59:32 UTC - Matteo Merli: You have the RPM installed? ---- 2018-12-09 17:26:29 UTC - David Kjerrumgaard: @Matteo Merli If you can give me the specifics, I will work on updating the documentation on this setting. We should also publicize this issue/ resolution on SO and similar for those who don't use this slack channel ---- 2018-12-09 23:45:50 UTC - Christophe Bornet: @Beast in Black @Matteo Merli So I tested namespace isolation and the failover to secondary works fine for me. But when the primary brokers come back, the secondary are still used. Any idea on how to fallback to the normal state ? ---- 2018-12-10 01:20:49 UTC - Beast in Black: @Christophe Bornet interesting question. Since you've apparently managed to get this working, what happens if you bounce one of the secondaries? Does the primary then take over? ---- 2018-12-10 01:32:16 UTC - Vincent Ngan: I am new to pulsar. I found there are two pulsar client maven packages. One is called pulsar-client and one is called pulsar-client-original. What is their difference and which one should I use? ---- 2018-12-10 01:42:59 UTC - David Kjerrumgaard: @Vincent Ngan I would recommend using the latest client, pulsar-client. <https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-client> ---- 2018-12-10 01:45:01 UTC - Matteo Merli: @Christophe Bornet @Beast in Black no need to bounce the brokers, you can force the reassignment with `pulsar-admin namespaces unload $NS` +1 : Beast in Black ---- 2018-12-10 02:14:19 UTC - legendtkl: yeah ---- 2018-12-10 02:27:18 UTC - Vincent Ngan: Thanks ---- 2018-12-10 02:28:35 UTC - Matteo Merli: @Vincent Ngan `pulsar-client` is shaded, while `pulsar-client-original` is not ---- 2018-12-10 02:30:01 UTC - Vincent Ngan: what does it mean by “shaded”? ---- 2018-12-10 02:30:42 UTC - Matteo Merli: Most dependencies are bundled with classes renamed to avoid conflicts ---- 2018-12-10 02:31:05 UTC - Matteo Merli: Take a look at maven shade plugin ---- 2018-12-10 02:33:32 UTC - Vincent Ngan: One problem with pulsar-client is that it doesn’t come with the sources.jar making it inconvenient to be used in Intellij when I want to navigate to see the source code. ---- 2018-12-10 02:38:06 UTC - Matteo Merli: Yes, that is a problem with the shade plugin ---- 2018-12-10 02:38:48 UTC - Matteo Merli: You can try to use -original and check if you don’t have dependencies conflicts ---- 2018-12-10 03:13:28 UTC - Vincent Ngan: I have just tried the pulsar-client-original package and it doesn’t seem to have any dependencies conflicts with my project. But, in view of the possibility of conflicts in the future, I think it’s better to use the shaded pulsar-client. ---- 2018-12-10 03:14:13 UTC - Matteo Merli: typically problems are around different versions of protocol buffers ---- 2018-12-10 03:14:27 UTC - Matteo Merli: (or Netty, Jackson) ---- 2018-12-10 03:15:27 UTC - Vincent Ngan: Thanks a lot for your advice. ---- 2018-12-10 03:21:53 UTC - Lingzhi Yang: @Lingzhi Yang has joined the channel ---- 2018-12-10 04:11:27 UTC - Samuel Sun: Hi , I have a question about pulsar proxy, most users use it as the gw in front of brokers, or they use their own gw(proxy) ? ---- 2018-12-10 04:12:08 UTC - Matteo Merli: The proxy is meant for deployments where the brokers are not directly reachable from the clients ---- 2018-12-10 04:22:14 UTC - Samuel Sun: ic, can I run it in background ? ---- 2018-12-10 04:25:19 UTC - Samuel Sun: and can I use normal proxy like Nginx/haproxy as an alternative ? ---- 2018-12-10 04:33:43 UTC - Matteo Merli: No, the proxy needs to be aware of Pulsar protocol, to redirect internal connection to the right broker ---- 2018-12-10 04:34:12 UTC - Matteo Merli: What do you mean by running in background? ---- 2018-12-10 04:35:15 UTC - Samuel Sun: like a daemon ---- 2018-12-10 04:35:48 UTC - Matteo Merli: Oh, sure. With pulsar-daemon command, like all other components ---- 2018-12-10 04:37:11 UTC - Samuel Sun: got it. thanks. and one more question pulsar proxy support what kind of protocal of Pulsar right now ? Does it support to send function to the right broker also ? ---- 2018-12-10 04:40:51 UTC - Matteo Merli: Yes, both data protocol and HTTP admin requests (including functions operations) are supported by the proxy ---- 2018-12-10 04:43:30 UTC - Samuel Sun: sweet, thanks ---- 2018-12-10 05:16:10 UTC - Samuel Sun: Usage: pulsar-daemon (start|stop) <command> <args...> where command is one of: broker Run a broker server bookie Run a bookie server zookeeper Run a zookeeper server configuration-store Run a configuration-store server discovery Run a discovery server websocket Run a websocket proxy server standalone Run a standalone Pulsar service ---- 2018-12-10 05:16:41 UTC - Samuel Sun: @Matteo Merli tried daemon start proxy, not working ---- 2018-12-10 05:29:48 UTC - Matteo Merli: Ouch, probably we forgot to add that. Can you please open an issue so that I don’t forget to fix? ---- 2018-12-10 05:33:17 UTC - Samuel Sun: <https://github.com/apache/pulsar/pull/3151/files> ---- 2018-12-10 05:33:21 UTC - Samuel Sun: @Matteo Merli ---- 2018-12-10 05:33:25 UTC - Samuel Sun: yeah, found it. ---- 2018-12-10 05:33:33 UTC - Samuel Sun: just made a small change ---- 2018-12-10 05:33:36 UTC - Samuel Sun: : ) ---- 2018-12-10 05:33:43 UTC - Matteo Merli: :+1: ---- 2018-12-10 07:00:17 UTC - richardliu: @richardliu has joined the channel ---- 2018-12-10 07:02:15 UTC - Samuel Sun: hi, can anyone make a “PIP” proposal on the wiki page ? <https://github.com/apache/pulsar.wiki.git> ---- 2018-12-10 07:02:36 UTC - Samuel Sun: looks like the fork not contain this wiki permission ? ---- 2018-12-10 07:02:44 UTC - richardliu: standalone ---- 2018-12-10 07:02:50 UTC - Sijie Guo: @Samuel Sun: currently only committers have permissions to do so. ---- 2018-12-10 07:03:10 UTC - Samuel Sun: a little sad ~ ---- 2018-12-10 07:03:12 UTC - Sijie Guo: you can write the PIP proposal in your gist, and start an email thread in dev@ mailing list for discussions. ---- 2018-12-10 07:03:22 UTC - Samuel Sun: got it. ---- 2018-12-10 07:03:33 UTC - richardliu: can standalone use in production ---- 2018-12-10 07:03:47 UTC - richardliu: cause we only one server ---- 2018-12-10 07:03:53 UTC - Sijie Guo: any of the committers can help you moving your gist to pulsar wiki. ---- 2018-12-10 07:04:41 UTC - Samuel Sun: one more question, how to be a committer ? ---- 2018-12-10 07:04:43 UTC - Samuel Sun: : ) ---- 2018-12-10 07:05:20 UTC - Sijie Guo: @richardliu - typically standalone is not recommended to be used in production. a better approach is to start with running one zk, bookie and broker on one server. so if you want to expand in future, you can expand. ---- 2018-12-10 07:05:24 UTC - Samuel Sun: find it in the doc, thanks ---- 2018-12-10 07:06:18 UTC - richardliu: To run Pulsar on bare metal, you will need:
At least 6 Linux machines or VMs 3 running ZooKeeper 3 running a Pulsar broker, and a BookKeeper bookie A single DNS name covering all of the Pulsar broker hosts ---- 2018-12-10 07:06:22 UTC - richardliu: i did ---- 2018-12-10 07:06:51 UTC - richardliu: <https://pulsar.apache.org/docs/en/deploy-bare-metal/> ---- 2018-12-10 07:06:57 UTC - richardliu: from this link ---- 2018-12-10 07:07:29 UTC - Sijie Guo: @Samuel Sun: when you contributes the project more (code changes, documentations, discussions), some one from the existing committers will make a proposal to the PMC to promote you as a committer ---- 2018-12-10 07:09:21 UTC - Sijie Guo: @richardliu: yeah. that’s typically setup. but you said you have only one server for now :slightly_smiling_face: if you wanna expand to more servers in future, I will suggest you to start with 1 zk, 1 bk and 1 broker. however if you only want to run pulsar on one server forever, you can use standalone for sure. ---- 2018-12-10 07:10:36 UTC - richardliu: ye. thanks.but if i want to start with 1 zk, 1 bk and 1 broker.where i can find the docs…… ---- 2018-12-10 07:10:41 UTC - Samuel Sun: trying, any new sharing would be in Singapore ? :slightly_smiling_face: ---- 2018-12-10 07:11:24 UTC - richardliu: only find deploy-bare-metal-multi-cluster ---- 2018-12-10 07:12:17 UTC - richardliu: and deploy-bare-metal ---- 2018-12-10 07:12:25 UTC - richardliu: <https://pulsar.apache.org/docs/en/deploy-bare-metal/> ---- 2018-12-10 07:12:32 UTC - richardliu: <https://pulsar.apache.org/docs/en/deploy-bare-metal-multi-cluster/> ---- 2018-12-10 07:13:26 UTC - Sijie Guo: @richardliu: <https://pulsar.apache.org/docs/en/deploy-bare-metal/> should work. change 3 to 1. that steps will be the same. the only thing you need to change is to modify the `conf/broker.conf` file, to make sure following these settings are set to 1 ``` # Number of bookies to use when creating a ledger managedLedgerDefaultEnsembleSize=1 # Number of copies to store for each message managedLedgerDefaultWriteQuorum=1 # Number of guaranteed copies (acks to wait before write is complete) managedLedgerDefaultAckQuorum=1 ``` ---- 2018-12-10 07:14:19 UTC - richardliu: ok thanks ---- 2018-12-10 07:22:04 UTC - Sijie Guo: I am not aware of any pulsar events in Singapore. but we are happy to help organize/sponsor meetups if there are interests in Singapore :slightly_smiling_face: ---- 2018-12-10 07:24:28 UTC - richardliu: you can clarify this in doc for anyone else need. my two cents ---- 2018-12-10 07:24:54 UTC - Sijie Guo: @richardliu yes. working on a PR to update the doc ---- 2018-12-10 07:27:21 UTC - Samuel Sun: cool, I am working in Apple Siri, just interested in puslar, although we used a lot of kafka right now. if need some places for event in Singapore, we could contribute a bit. ---- 2018-12-10 07:29:04 UTC - richardliu: let me know if you need any help ---- 2018-12-10 07:34:00 UTC - Sijie Guo: awesome. glad to hear about that. will definitely consider that :slightly_smiling_face: ---- 2018-12-10 07:42:24 UTC - Samuel Sun: welcome ---- 2018-12-10 07:59:38 UTC - Sijie Guo: @richardliu: I have a PR to update the deployment doc with instructions for one-node cluster. would like to hear from you if the information is good enough. feel free to comment in the PR about your thoughts. I will address them. <https://github.com/apache/pulsar/pull/3152/files> ---- 2018-12-10 08:03:33 UTC - richardliu: looks good to me. ---- 2018-12-10 08:05:05 UTC - richardliu: but i run pulsar in windows subsystem for linux(wsl).given that Pulsar is currently only available for MacOS and Linux ----
