2020-07-07 09:15:51 UTC - aloyszhang: Yes, produce performance problem has been fixed right now and you can try it by checkout the master . ---- 2020-07-07 09:36:58 UTC - Rahul Vashishth: i am exploring an option to access borker binary protocol over 80 port instead of using the default 6550. ---- 2020-07-07 09:47:26 UTC - Penghui Li: You can try following steps: ```mvn clean install -DskipTests -pl pulsar-broker -am cd distribution/server mvn clean install -DskipTests``` ---- 2020-07-07 09:48:09 UTC - Sijie Guo: You don’t need HTTP port to be exposed if you only use Pulsar client. If you need to use Pulsar Admin client or use `pulsar-admin` tool, you need to expose HTTP port. ---- 2020-07-07 09:48:30 UTC - Sijie Guo: You can create two different DNS entries for different ports. That’s totally fine. ---- 2020-07-07 09:56:02 UTC - Hiroyuki Yamada: Thank you ! ---- 2020-07-07 10:41:39 UTC - Hiroyuki Yamada: I posed what’s going and some questions. Please take a look when you get a chance. <https://github.com/apache/pulsar/issues/7455> ---- 2020-07-07 10:42:56 UTC - Rahul Vashishth: @Sijie Guo i assume http port is required for broker lookup? how does broker lookup will work if we expose binary and http on different ports?
do we need to do any config change for broker lookup? ---- 2020-07-07 12:49:12 UTC - Ming: @Devin G. Bost For cgo based lib, C++ library is required to be installed as prerequisite. Please see this example <https://github.com/kafkaesque-io/pulsar-beam/blob/5add413be2d6d730ca1fa11b4942feddbfad329a/Dockerfile#L29> But we have moved all go client to native pulsar go-client library. ---- 2020-07-07 13:03:05 UTC - David Lanouette: Sorry, I don't know the answer to that off the top of my head. ---- 2020-07-07 14:29:26 UTC - Vaibhav Aiyar: Hi, we want to setup pulsar with 2 bookkeeper racks each rack having 3 rookies each. Ensemble size is 4, with write quorum as 4 and read quorum as 3. We have enabled Rack Aware policy. When we create topics will pulsar the 4 bookies be selected evenly from two racks. So that we have 2 bookies selected from one rack and 2 from the other? ---- 2020-07-07 15:27:26 UTC - Sijie Guo: HTTP lookup is not required. The binary endpoint also can do the lookup. The workflow doesn’t change event you expose the binary and http on different endpoints. ---- 2020-07-07 15:28:20 UTC - Sijie Guo: Correct ---- 2020-07-07 15:37:12 UTC - Viktor: No worries. let me repost ---- 2020-07-07 15:38:17 UTC - David Lanouette: I'd suggest pinging the <#C5Z4T36F7|general> channel. You'll likely get people that are knowledgeable over there :slightly_smiling_face: +1 : Viktor ---- 2020-07-07 15:50:33 UTC - Viktor: Hello is there a way to control the number of partitions for `produce` test the `pulsar-perf` tool? ---- 2020-07-07 15:56:10 UTC - Sijie Guo: You can pre-create the partitions topic with `bin/pulsar-admin topics create-partitioned-topic` ---- 2020-07-07 16:06:12 UTC - Joshua Dunham: Hey Everyone, I'm trying to replicate a postgres DB from one site to another. Does anyone have experience using the debezium connector over distance to sync one postgres set of tables with another were both are considered masters (but the DB's have no awareness of each other)? ---- 2020-07-07 16:09:45 UTC - Joshua Dunham: I don't think special config of postgres besides CDC is needed. I just want one DB to have as clients users/apps and the other to have as client pulsar. If 1 fails I could change the URL leading to 1 over to 2 for failover. ---- 2020-07-07 16:20:07 UTC - David Lanouette: I don't have an answer to your specific question. But this doesn't sound like a great solution to me. (Note: I don't know you exact needs/situation). Trying to recreate Postgres replication and failover using Pulsar (or, any message broker), is likely going to give you a bad experience. There are just too many weird edge cases to deal with. ---- 2020-07-07 16:20:11 UTC - David Lanouette: Is there a reason you don't use the built in <https://www.postgresql.org/docs/current/warm-standby.html|Warm Standby> feature of Postgres? ---- 2020-07-07 16:22:36 UTC - Joshua Dunham: I don't think I can use warm-standby to host other live tables. ---- 2020-07-07 16:23:26 UTC - Joshua Dunham: My exact use case is that I have three instances of an app in three geographic locations. ---- 2020-07-07 16:23:36 UTC - David Lanouette: Ah, I missed that both were "master". ---- 2020-07-07 16:24:09 UTC - Joshua Dunham: I wanted to have each DB instance have a RW section and two that were backups (one per site). ---- 2020-07-07 16:24:51 UTC - Joshua Dunham: In the case of DR I could get the app to the other site and have a ready to go instance with no config needed (since it's not a war standby that needs to be promoted). ---- 2020-07-07 16:25:49 UTC - David Lanouette: So, you are writing to all 3 instances of the db at the same time? (with different data) ---- 2020-07-07 16:26:32 UTC - Joshua Dunham: One pulsar bridge would have the three DB's as clients. Writes come into the system from one and multiplexed to the other two. ---- 2020-07-07 16:27:16 UTC - Joshua Dunham: The actual client is very low throughput so I don't think it's a stressful config. ---- 2020-07-07 16:30:12 UTC - David Lanouette: It _might_ work, but I'd be very hesitant. If you can afford it, I'd consider a multi-master cluster, like the [PostgresPro MultiMaster](<https://postgrespro.com/docs/enterprise/12/multimaster>), or [Postgres XL](<https://www.postgres-xl.org/>). ---- 2020-07-07 16:31:34 UTC - David Lanouette: Note: I don't have any experience with these. For a "real" suggestion, consider checking in a Postgres specific forum. ---- 2020-07-07 16:31:39 UTC - David Lanouette: Good luck. ---- 2020-07-07 16:47:21 UTC - Joshua Dunham: thx for your tips on this! ---- 2020-07-07 17:13:39 UTC - Vil: When will Pulsar transactions be available? ---- 2020-07-07 18:11:23 UTC - Sijie Guo: The Pulsar-Flink connector is being contributed to Flink as part of FLIP-72 ---- 2020-07-07 18:11:53 UTC - Sijie Guo: Those two pull requests were actually contributed by us. ---- 2020-07-07 18:13:37 UTC - Sijie Guo: The Pulsar Transaction is planned for 2.7 release and will be released in September +1 : Kirill Kosenko ---- 2020-07-07 18:14:03 UTC - Devin G. Bost: Thanks ---- 2020-07-07 20:48:10 UTC - Viktor: @Sijie Guo Thanks. ended up doing something similar.. Able to get it up and running now.. ---- 2020-07-07 21:00:55 UTC - Viktor: Reposting my questions based on a run using `pulsar-perf produce` (instead of omb as suggested). Actual command on a 16 partition topic. ```bin/pulsar-perf produce -bb 1048576 -bm 10000 -b 500 -ef -c 16 -o 10000 -p 2000000000 -n 4 -t 16 -threads 4 -r 500000 -s 1024 -i 10 --time 300 --warmup-time 10 <persistent://benchmark/ns-DPgpF10/test-OBLzePM-0000>``` I still see produce perf capped at 100-150 MB/s. I have turned off `journalSyncData=false` for throughput. Otherwise, it's all default settings. There is plenty of network/cpu. ---- 2020-07-07 21:10:19 UTC - Chris Hansen: How are folks integrating Vault and Pulsar? I’d like to set it up such that Pulsar functions, producers, and consumers can have distinct tokens that rotate periodically. It seems like you could set up a custom Vault backend that calls out to Pulsar admin to manage tokens, or you could write a custom auth plugin that lives in Pulsar and calls out to Vault for auth. Anyone have informed opinions about this or experience with it? ---- 2020-07-07 21:11:43 UTC - Matteo Merli: @chris talked about it in this talk: <https://www.youtube.com/watch?v=PIT8PilhHV4&list=PLqRma1oIkcWjVlPfaWlf3VO9W-XWsF_4-&index=7&t=0s> ---- 2020-07-07 21:13:24 UTC - Chris Hansen: I did watch the talk, and the code on his <https://github.com/cckellogg/pulsar-vault|github repo> w/ the Pulsar plugin approach. Though he hasn’t implemented rotation (yet). ---- 2020-07-07 21:15:46 UTC - Chris Hansen: or maybe Vault is handling the rotation but I’m not sure how he’s configured Vault ---- 2020-07-07 22:11:55 UTC - chris: the vault agent should automatically refresh/renew the tokens bases on the expiration of the token/secret ---- 2020-07-07 22:12:36 UTC - Chris Hansen: ah, makes sense to delegate to the agent ---- 2020-07-07 22:12:48 UTC - Chris Hansen: thanks @chris ---- 2020-07-07 22:14:03 UTC - chris: it would be fairly easy to write a plugin as well but if you are running in kubernetes the vault agent does it for you and makes it seamless ---- 2020-07-07 23:42:18 UTC - Addison Higham: @Chris Hansen also, for another approach: <https://www.youtube.com/watch?v=ViX1wJTmRmk> I talk on how I did that at Instructure that didn't require us to do any pulsar changes, we just wrote some minimal tooling to help bootstrap pulsar by fetching a token ---- 2020-07-08 01:26:08 UTC - victornicolaslizarrag: @victornicolaslizarrag has joined the channel ---- 2020-07-08 01:34:01 UTC - victornicolaslizarrag: Hi everybody! I'm new here and a beginner in Apache Pulsar wave : Addison Higham, Ali Ahmed ---- 2020-07-08 01:45:04 UTC - victornicolaslizarrag: I'm currently looking for something to make *event sourcing/CQRS systems* and pulsar looks very good, but currently I have the question about how to split my topics, Im considering two options right now 1. entity type per topic and then make multiple partitions -> `<persistent://tenant/namespace/accounts>` 2. entity per topic -> `<persistent://tenant/namespace/accounts-{accountId}>` What you think is the best way to split them? are you think pulsar could support entity per topic split? ---- 2020-07-08 03:50:20 UTC - Addison Higham: How many entities do you plan on having? Pulsar can support hundreds of thousand of topics fairly easily and the cost of topics is fairly low, but still non-zero. The more common problem is if you have use cases that require you to subscribe to all of the instances of an entity at once. If you have 10,000 account entities, now you need to have a consumer which subscribes to 10,000 topics, which is doable, but does stress Pulsar in some unique ways. If the number of entities really is well bound, then it can be workable, but generally I would say go for the first option and using message keys to make sure messages route to the correct partitions (if you are using partitioned topics). A key_shared subscription can be pretty useful if you go that direction :slightly_smiling_face: +1 : victornicolaslizarrag ---- 2020-07-08 04:05:50 UTC - victornicolaslizarrag: Oh I see, thanks! I wouldn't like to be concern about the number of entities so I guess I'm going to take the first option but I have a doubt about what happend if I added a new partition, will I loose my keys sequence like what happens in kafka? or its gonna keep the sequence since Pulsar has virtual tpoics ---- 2020-07-08 06:20:24 UTC - Hiroyuki Yamada: Hi, I’m using standalone mode for quick checking and updated standalone.conf for `journalDirectories` and `ledgerDirectories` like this. ```journalDirectories=data/bookkeeper/journal ledgerDirectories=data/bookkeeper/ledger``` But it seems like it’s not reflected and the default values are used for both. ```"ledgerDirectories" : "data/standalone/bookkeeper0", "journalDirectories" : "data/standalone/bookkeeper0",``` Am I missing something ? It would be great if anyone can let me know how to update those values in standalone mode. ---- 2020-07-08 07:27:49 UTC - Vil: Thank you @Sijie Guo ---- 2020-07-08 08:20:47 UTC - Josué: alright thanks ----
