2018-09-26 09:33:16 UTC - Matti-Pekka Laaksonen: I have problems using the pulsar-admin cli to get stats for topics (bin/pulsar-admin persistent stats <topic-name-here>) ---- 2018-09-26 09:33:38 UTC - Matti-Pekka Laaksonen: Sometimes I get an answer right away, and sometimes the request times out ---- 2018-09-26 09:34:37 UTC - Matti-Pekka Laaksonen: I can see the GET request in the proxy logs and broker logs, but I never get a response ---- 2018-09-26 09:36:34 UTC - Matti-Pekka Laaksonen: Could this be a performance problem with the broker node? The memory and CPU is very limited at the moment (I will increase it next time I restart the stack), but CPU utilization is in the 2% range for every broker node ---- 2018-09-26 10:50:25 UTC - Ivan Kelly: are you using TLS? ---- 2018-09-26 10:54:54 UTC - Ivan Kelly: have you verified whether it blocks at the broker or the proxy? ---- 2018-09-26 11:21:36 UTC - Matti-Pekka Laaksonen: No TLS ---- 2018-09-26 11:22:08 UTC - Matti-Pekka Laaksonen: I can see the request in the proxy logs, and then in the logs of one broker node ---- 2018-09-26 11:22:20 UTC - Matti-Pekka Laaksonen: So it seems to go through from proxy to broker ---- 2018-09-26 13:09:16 UTC - Ivan Kelly: but does the response get back to the proxy? ---- 2018-09-26 13:09:41 UTC - Ivan Kelly: could you post logs from broker? ---- 2018-09-26 13:28:05 UTC - Matti-Pekka Laaksonen: ---- 2018-09-26 13:28:45 UTC - Matteo Merli: That’s a 307 redirection ---- 2018-09-26 13:28:57 UTC - Matti-Pekka Laaksonen: Those are the only lines that are related to the request I made. Others concern just zookeeper connections ---- 2018-09-26 13:29:00 UTC - Matteo Merli: you need to follow that ---- 2018-09-26 13:29:20 UTC - Matti-Pekka Laaksonen: But isn't that intentional? The proxy redirects to broker ---- 2018-09-26 13:30:04 UTC - Matti-Pekka Laaksonen: Oh, does the broker redirect to another broker if the topic is not found at the first one? So the proxy doesn't know that information? ---- 2018-09-26 13:30:22 UTC - Matteo Merli: proxy -> broker is internally handled. brokers will redirect to specific broker to handle some of the calls. eg. for stats, only the broker running a topic will have the current stats ---- 2018-09-26 13:30:45 UTC - Matteo Merli: if you use curl, just pass the `-L` flag ---- 2018-09-26 13:30:52 UTC - Matti-Pekka Laaksonen: So the pulsar-admin cli tool doesn't follow the second redirect by default? ---- 2018-09-26 13:30:56 UTC - Matti-Pekka Laaksonen: I'm using the cli ---- 2018-09-26 13:31:16 UTC - Matteo Merli: oh, CLI will follow that ---- 2018-09-26 13:32:20 UTC - Matteo Merli: uhm, I suspect it might be something related to going through the proxy for HTTP requests ---- 2018-09-26 13:32:40 UTC - Matteo Merli: can you just point that to the brokers service URL (still HTTP) ? ---- 2018-09-26 13:34:01 UTC - Matti-Pekka Laaksonen: Instead of the proxy? Can't do, I don't have visibility there from outside network ---- 2018-09-26 13:37:34 UTC - Matteo Merli: Ok. So in theory the proxy should actually handle the redirect by itself, otherwise it would be redirecting to an IP unreachable to the client… From the logs though, it looks it sends back the 307 to client.. ---- 2018-09-26 13:37:36 UTC - Matti-Pekka Laaksonen: With curl testing this is faster: every third request returns the correct info immediately, two out of third time out immediately. So the proxy directs the connections to the brokers on a round robin basis ---- 2018-09-26 13:37:47 UTC - Matteo Merli: correct ---- 2018-09-26 13:38:27 UTC - Matti-Pekka Laaksonen: curl is actually insanely faster than the pulsar-admin tool ---- 2018-09-26 13:39:32 UTC - Matteo Merli: I know.. the pulsar-admin is Java based… it takes a second at least to start the JVM.. BTW We’re working on getting Go CLI tools! That would be as fast as curl ---- 2018-09-26 13:40:02 UTC - Matti-Pekka Laaksonen: Oh, nice! ---- 2018-09-26 13:40:47 UTC - Matti-Pekka Laaksonen: So do you think this is a bug on the broker side? ---- 2018-09-26 13:41:25 UTC - Matti-Pekka Laaksonen: curl -L <api-address> just hangs indefinitely ---- 2018-09-26 13:43:59 UTC - Matti-Pekka Laaksonen: Oh, curl gets redirected to the internal IP address and naturally can't connect there ---- 2018-09-26 13:44:05 UTC - Matteo Merli: On proxy side. It shouldn’t send back the 307 to client, but rather following up itself and returning the 200 to client ---- 2018-09-26 13:45:09 UTC - Matti-Pekka Laaksonen: Could there be something wrong with my cluster setup? ---- 2018-09-26 13:46:41 UTC - Matteo Merli: Let me try to reproduce this in an integration test. Can you open an issue in any case? ---- 2018-09-26 13:47:18 UTC - Matti-Pekka Laaksonen: Sure thing ---- 2018-09-26 13:49:22 UTC - Matti-Pekka Laaksonen: Thank you again for the help ---- 2018-09-26 14:28:39 UTC - Matti-Pekka Laaksonen: Are inactive topics those that have no subscriptions or producers? So are topics that contain retained messages but which have no subscriptions or producers deleted if brokerDeleteInactiveTopics is set to true? ---- 2018-09-26 14:41:13 UTC - Matteo Merli: If the time based retention is set, the topic won’t be deleted until that time expired (if there are no subscriptions) ---- 2018-09-26 15:02:34 UTC - Grant Wu: Well, one thing I’ve been interested in looking into is Zulip ---- 2018-09-26 15:03:00 UTC - Grant Wu: <https://zulipchat.com/for/open-source/> they have free commercial-tier hosting for open source projects ---- 2018-09-26 15:03:38 UTC - Grant Wu: Oh, actually, they don’t have open core, everything is open ---- 2018-09-26 15:03:56 UTC - Grant Wu: What I mean is that they have no-quota hosting for open source projects ---- 2018-09-26 15:15:25 UTC - Grant Wu: <https://zulipchat.com/new/> after you make an org here, email <mailto:[email protected]|[email protected]> ---- 2018-09-26 15:15:46 UTC - Grant Wu: There’s also a Slack import tool ---- 2018-09-26 15:15:46 UTC - Marko: @Marko has joined the channel ---- 2018-09-26 18:24:24 UTC - Grant Wu: @Jon Bock Found a dead link: <https://streaml.io/blog/pulsar-effectively-once-end-to-end> ---- 2018-09-26 18:24:44 UTC - Grant Wu: Near the end - it links to <https://streaml.io/pulsar-effectively-once> ---- 2018-09-26 18:31:37 UTC - Jon Bock: Thanks for pointing that out, will fix! ---- 2018-09-26 18:50:35 UTC - Robert Morgan: @Robert Morgan has joined the channel ---- 2018-09-26 18:56:22 UTC - Robert Morgan: The 'Pulsar Functions - Overview' section of the docs mentions "Pulsar Functions can currently be written in Java and Python. Support for additional languages is coming soon." Is Golang support on the Functions roadmap? Thanks ---- 2018-09-26 18:56:40 UTC - Sijie Guo: yes golang is in the roadmap ---- 2018-09-26 19:02:13 UTC - Robert Morgan: Thank you @Sijie Guo That's really exciting- the team is really doing some incredible work! ---- 2018-09-26 19:08:54 UTC - Robert Morgan: I missed that @Sanjeev Kulkarni talked about that on Sept 25, sorry about that - don't know how I missed it on scanning prior conversations ---- 2018-09-26 19:12:26 UTC - Sijie Guo: although it might be in 2.4, we have quite a lot of other features going in 2.3. however if anyone from community is interested in helping with the development of go functions for pulsar, let us know and we’d happy to help to expedite that. ---- 2018-09-26 19:42:23 UTC - Yuwei Jiang: Hi guys, I’d like to get a better understanding on how pulsar does geo-replication. Can you please point me to which part in the code base I should be looking into? ---- 2018-09-26 19:49:01 UTC - Matteo Merli: If you want the nitty-gritty details, start taking a look at <https://github.com/apache/pulsar/blob/83e31579f5df941ad49fb54ec269c1d917821504/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java> ---- 2018-09-26 19:54:59 UTC - Yuwei Jiang: @Matteo Merli Thanks! ---- 2018-09-26 22:47:54 UTC - Nathanial Murphy: Hi guys! Is there an issue I can follow for the go support of Pulsar Functions? ---- 2018-09-26 22:48:50 UTC - Sijie Guo: @Nathanial Murphy let me create one ---- 2018-09-26 22:50:54 UTC - Sijie Guo: @Nathanial Murphy here you go - <https://github.com/apache/pulsar/issues/2662> ---- 2018-09-26 22:51:57 UTC - Nathanial Murphy: Thanks for that! Let me know if there's anything I can do to help with the efforts. :slightly_smiling_face: ---- 2018-09-26 22:53:45 UTC - Sijie Guo: Sure. we will probably come up with a plan and break it down with smaller tasks and update that to the issue. feel free to watch on that issue. when we start the efforts, you are welcome to work with us on go functions ---- 2018-09-27 04:10:23 UTC - Nathanial Murphy: From what I'm reading on <http://pulsar.apache.org/docs/latest/advanced/PartitionedTopics/>, would it be correct to say a single partitioned topic is still linearised when using a single consumer? ---- 2018-09-27 04:11:25 UTC - Ali Ahmed: yes ---- 2018-09-27 04:12:19 UTC - Matteo Merli: What do you mean with “linearised” ? ---- 2018-09-27 04:12:38 UTC - Matteo Merli: Ordering will be only guaranteed at the partition level ---- 2018-09-27 04:13:39 UTC - Nathanial Murphy: As in can-consume-in-same-order-as-published ---- 2018-09-27 04:15:13 UTC - Nathanial Murphy: I'm trying to publish messages that need to be strict, total ordered and trying to figure out if I am able to use `RoundRobinPartition` as the `MessageRoutingMode` ---- 2018-09-27 04:18:46 UTC - Nathanial Murphy: sounds like I can't, which is fine. Thank you for your help so far :slightly_smiling_face: ---- 2018-09-27 04:23:05 UTC - Matteo Merli: You can have per-key ordering if you set keys on the messages. Each message with same key will be hashed on the same partition ---- 2018-09-27 04:23:43 UTC - Matteo Merli: If you need total ordering across all messages, then there’s no other way than use a single partition ---- 2018-09-27 04:26:50 UTC - Nathanial Murphy: Figured that would be the case, but was hopeful otherwise. I'm assuming that the bookies' ability to ship older segments of a given partition has little impact to my partitioning strategies on the messaging level? ---- 2018-09-27 04:30:32 UTC - Matteo Merli: Yes, that’s orthogonal to the partitioning ---- 2018-09-27 04:30:53 UTC - Matteo Merli: In any case, what kind of throughput are you expecting? ---- 2018-09-27 04:54:50 UTC - Paul Wills: @Paul Wills has joined the channel ----
