2018-02-16 10:20:17 UTC - Goloshchapov Egor: @Goloshchapov Egor has joined the channel ---- 2018-02-16 10:23:30 UTC - Goloshchapov Egor: Hi all. Could you advise me please. I have 3 brokers, 1 partitioned topic (10 partitions), and round robin message route mode in producer. I can see that 10 virtual topics were created and load distributed correctly within topics, but all topics are in 1 broker. I'm expecting that virtual topics are distributed between all brokers, but they are not. Do you have any idea what I'm doing wrong? Thank you very much! ---- 2018-02-16 15:27:24 UTC - Goloshchapov Egor: I think it has something to do with bundles, but there is no enough information about that in documentation. I cannot figure out what is this and how to use it.. ---- 2018-02-16 15:37:41 UTC - Adam Williams: Hello - I was wondering if there's a way to see how many times a message was delivered to a consumer? ---- 2018-02-16 19:55:53 UTC - Matteo Merli: Hi @Goloshchapov Egor. Yes, the documentation on how to do the balancing across brokers is not exaustive. I’ve tried to address it here on Slack earlier but the we haven’t got to polish it and add it to documentation properly yet. I’m reposting here, since it was buried in the channel history.
======== In Pulsar, “namespaces” are the administrative unit: you can configure most options on a namespace and they will be applied on the topics contained on the namespace. It gives the convenience of doing settings and operations on a group of topics rather than doing it once per topic. In general, the pattern is to use a namespace for each user application. So a single user/tenant, can create multiple namespaces to manage its own applications. When it comes to topics, we need a way to assign topics to brokers, control the load and move them if a broker becomes overloaded. Rather that doing this operations per each single topic (ownership, load-monitoring, assigning), we do it in bundles, or “groups of topics”. In practical words, the number of bundles determines “into how many brokers can I spread the topics for a given namespace”. From the client API or implementation, there’s no concept of bundles, clients will lookup the topics that want to publish/consumer individually. On the broker side, the namespace is broken down into multiple bundles, and each bundle can be assigned to a different broker. Effectively, bundles are the “unit of assignment” for topics into brokers and this is what the load-manager uses to track the traffic and decide where to place “bundles” and whether to offload them to other brokers. A bundle is represented by a hash-range. The 32-bit hash space is initially divided equally into the requested bundles. Topics are matched to a bundle by hashing on the topic name. Default number of bundles is configured in `broker.conf`: `defaultNumberOfNamespaceBundles=4` When the traffic increases on a given bundle, it will be split in 2 and reassigned to a different broker. Enable auto-split: `loadBalancerAutoBundleSplitEnable=true` trigger unload and reassignment after splitting: `loadBalancerAutoUnloadSplitsEnable=true`. If is expected to have a high traffic on a particular namespace, it’s a good practice to specify a higher number of bundles when creating the namespace: `bin/pulsar-admin namespaces create $NS --bundles 64`. This will avoid the initial auto-adjustment phase. All the thresholds for the auto-splitting can be configured in `broker.conf`, eg: number of topics/partitions, messages in/out, bytes in/out, etc... ---- 2018-02-16 19:56:04 UTC - Matteo Merli: One other addition to quickly address the unbalance is to “force” a reassignment. You can do it by unloading a particular bundle, or the whole namespaces. ``` bin/pulsar-admin namespaces unload $MY_NAMESPACE ``` After unloading, the brokers load will be taken into consideration to decide where to reassign the “bundles” of topics. ---- 2018-02-16 19:57:49 UTC - Matteo Merli: > Hello - I was wondering if there’s a way to see how many times a message was delivered to a consumer? @Adam Williams the broker is not keeping track of that information currently. What is the intention for that? Is that to put a limit on how many times a message can be delivered and discard after that? ---- 2018-02-16 20:12:16 UTC - Goloshchapov Egor: Thanks @Matteo Merli. I will read one more time carefully. Right now I didn't catch how to make distribution between brokers correctly using bundles feature ---- 2018-02-16 20:15:01 UTC - Matteo Merli: Yes, there are a a lot of details, I would suggest to use the quick ways to correct the unbalance by forcing the reassignment: ``` bin/pulsar-admin namespaces unload $MY_NAMESPACE ``` ---- 2018-02-16 20:31:45 UTC - Goloshchapov Egor: @Matteo Merli, I'm interesting in non-persistent topics, standalone installation in windows. for me it looks like I don't need rocksDB at all. But even when I disable in in config file (enablePersistentTopics=false) - rocksdb still trying to start and fails (becuase of lz4 link absence in windows). is it expected behavior? is it possible not to use bookeeper in case of enablePersistentTopics=false configuration? ---- 2018-02-16 20:37:39 UTC - Matteo Merli: I see, can you give me the details for the RocksDb link error. We’ll open an issue on RocksDB to see whether that can be fixed quickly. For starting without bookie, you’re correct.. in standalone mode, it should not start the bookie component if we don’t have persistent topics.. Anyway, I just double-checked that you workaround with asking for 0 bookies: ``` bin/pulsar standalone --num-bookies 0 ``` Hopefully that should be working on windows ---- 2018-02-16 20:52:50 UTC - Goloshchapov Egor: @Goloshchapov Egor uploaded a file: <https://apache-pulsar.slack.com/files/U9A9WLDUM/F9AE0DE5R/1.txt|1.txt> ---- 2018-02-16 20:53:10 UTC - Goloshchapov Egor: even with --num-bookies 0 parameter ---- 2018-02-16 20:54:54 UTC - Matteo Merli: I see, we should really disable the LZ4 compression there. That’s a quick fix ---- 2018-02-16 20:56:48 UTC - Goloshchapov Egor: is it possible to disable lz4 comporession with configuration? ---- 2018-02-16 20:56:56 UTC - Goloshchapov Egor: or code recompilation required? ---- 2018-02-16 20:57:07 UTC - Matteo Merli: Unfortunately not :confused: ---- 2018-02-16 20:58:50 UTC - Goloshchapov Egor: that fix would be great, but I think it would be better not to start bookie if number of bookies is zero. in that case users like me would be able to start pulsar even in windows ---- 2018-02-16 20:59:04 UTC - Matteo Merli: but I’m not understaning why it’s starting the bookie ---- 2018-02-16 20:59:31 UTC - Goloshchapov Egor: may be I did a mistake with bat file ---- 2018-02-16 20:59:34 UTC - Matteo Merli: looking at the code, it shouldn’t start :thinking_face: ----
