2020-05-15 09:26:52 UTC - Konstantinos Papalias: it should work yes, make sure you define your namespace as well ---- 2020-05-15 09:28:07 UTC - Konstantinos Papalias: and or tenant if they are not on `public/default` ---- 2020-05-15 10:09:04 UTC - Patrik Kleindl: I have questions regarding the comparison of Pulsar and Kafka: For the simple case of a non-replicated topic (=only 1 Kafka Broker or Bookkeeper has each segment) the write latency should be in favor of Kafka because the single broker could directly acknowledge the write while on Pulsar the latency between the Pulsar broker and the bookkeeper would be added to the processing? For replicated topics it would be more similar because for Kafka the latency of writes to the replicas is added and for Pulsar multiple bookkeepers have to acknowledge. Is my understanding correct or did I miss something? ---- 2020-05-15 10:12:41 UTC - Ali Ahmed: @Patrik Kleindl no it’s not correct, if the latency is primary you can run bookkeeper and pulsar broker in one node like standalone mode (your non replicated case) ---- 2020-05-15 10:13:38 UTC - Ali Ahmed: for replicated we do quorum writes they happen in parallel not serially, overall pulsar can sustain much higher write throughput with lower latency then kafka ---- 2020-05-15 10:20:46 UTC - Patrik Kleindl: Sure, but that would be a custom setup for a single topic. If you run a normal cluster with separate instances then for a single non-replicated topic this would still be true, right? ---- 2020-05-15 10:26:06 UTC - Rakshit Menpara: @Rakshit Menpara has joined the channel ---- 2020-05-15 10:27:24 UTC - Vincent Ngan: I have a question. Is it possible to run Pulsar server on ARM based Linux machine such as Raspberry Pi? ---- 2020-05-15 10:57:30 UTC - Ali Ahmed: @Vincent Ngan yes it can ok_hand : Vincent Ngan ---- 2020-05-15 11:10:19 UTC - Vladimir Shchur: Hi! I want to announce that Pulsar.Client for .net now supports schemas! (<https://github.com/fsharplang-ru/pulsar-client-dotnet|Repository>) Also kudos to@Ebere Abanonu who worked on AvroSchemaGenerator library which was used to complete this task. +1 : Ali Ahmed, Fred, Penghui Li, Konstantinos Papalias, David Kjerrumgaard ---- 2020-05-15 11:18:55 UTC - Patrik Kleindl: That is interesting, thanks for the information. One small question regarding the bookkeeper writes, does it write the data once or twice locally? One thing I read is the data is first appended to one logfile and then written to the actual ledger, or is that the same? ---- 2020-05-15 11:26:52 UTC - Patrik Kleindl: @Ali Ahmed And just to verify because I am working out the differences, the serial replication was not meant regarding Kafka but in general or did I understand you wrong? ---- 2020-05-15 13:47:02 UTC - Rattanjot Singh: Hi, getting this error when using transaction Which version will it get released? ```ava.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException: Not Implemented Yet at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at com.intuit.platform.integration.messaging.client.TransactionClient.main(TransactionClient.java:73) Caused by: java.lang.UnsupportedOperationException: Not Implemented Yet at org.apache.pulsar.client.impl.transaction.TransactionImpl.commit(TransactionImpl.java:122) ... 1 more``` ---- 2020-05-15 14:04:41 UTC - Peter Nordstrom: @Peter Nordstrom has joined the channel ---- 2020-05-15 14:29:36 UTC - Penghui Li: Transaction is not ready now. There is a master tracker <https://github.com/apache/pulsar/issues/2664> for transaction. ---- 2020-05-15 14:44:03 UTC - Tanner Nilsson: Is there a way to change the name of the `standalone` cluster that is automatically created when starting pulsar standalone? ---- 2020-05-15 15:26:48 UTC - Vincent Ngan: I have just tried to run pulsar 2.5.1 on my raspberry pi 4 with ubuntu 20.04 server, and I get the following error: 15:22:04.020 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service. <http://java.io|java.io>.IOException: Failed to load RocksDB JNI library at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:91) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:58) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:161) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:149) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:129) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:775) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:105) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:278) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.5.1.jar:2.5.1] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:403) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.5.1.jar:2.5.1] at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:260) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1] at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:119) [org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1] Caused by: java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni7982149528101845860.so: /tmp/librocksdbjni7982149528101845860.so: cannot open shared object file: No such file or directory (Possible cause: can’t load AMD 64-bit .so on a AARCH64-bit platform) at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_252] at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934) ~[?:1.8.0_252] at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817) ~[?:1.8.0_252] at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_252] at java.lang.System.load(System.java:1088) ~[?:1.8.0_252] at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?] at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?] at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?] at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] ... 13 more 15:22:04.033 [Thread-1] ERROR org.apache.pulsar.PulsarStandaloneStarter - Shutdown failed: null ---- 2020-05-15 15:29:18 UTC - Andres Riofrio: Is the messageId guaranteed to be unique? Even when there are multiple partitions in a topic? ---- 2020-05-15 15:33:08 UTC - Vincent Ngan: It seems that the org.rocksdb-rocksdbjni-5.13.3.jar jar requires a ARM version of native library. ---- 2020-05-15 15:50:25 UTC - Ming: Have you tried to change `clusterName` in the standalone.conf? ---- 2020-05-15 16:00:58 UTC - Tanner Nilsson: ya, that just seems to make standalone fail because it's trying to connect to a cluster that doesn't exist. `standalone` seems to be the name of the existing cluster still...but I could be doing something wrong? ---- 2020-05-15 17:12:48 UTC - Ruian: Hi, I run into a weird problem that message retention are quite unbalanced between partitions of a topic (6GB vs 3xxMB). Could someone help me to identify the cause? ---- 2020-05-15 17:17:26 UTC - Ruian: I have one subscription on this partition topic and it seems normal. ---- 2020-05-15 17:24:07 UTC - Ruian: How could this happened? The only thing I noticed that is the cursors of subscription on different partitions are quite different:
The cursor on the 6GB partition: ---- 2020-05-15 17:24:35 UTC - Ruian: The cursor on the 3xxMB partition: ---- 2020-05-15 17:26:08 UTC - Ruian: The logs of my consumer, no error ---- 2020-05-15 17:32:24 UTC - Ming: conf/functions_worker.yml has two occurances `standalone` that should be updated as well. I tested with a clean install from a tar ball. It worked. I suspected my old folder had some data conflicting with the new cluster. ---- 2020-05-15 17:38:19 UTC - Tanner Nilsson: that was it! Thanks @Ming! ---- 2020-05-15 18:43:05 UTC - Sijie Guo: it is unique in the entire pulsar cluster ---- 2020-05-15 18:46:38 UTC - Sijie Guo: Can you run `pulsar-admin topics stats-internal` to get the internral stats for the 6GB partition? ---- 2020-05-15 18:56:59 UTC - Sijie Guo: you use it in output topic? Can you describe your test sequence? ---- 2020-05-15 20:12:44 UTC - Raman Gupta: I know this is OT, but just to close the loop here, I created this issue against Kafka as I can see no reason for this to have happened other than a bug: <https://issues.apache.org/jira/browse/KAFKA-10007>. ---- 2020-05-15 20:14:12 UTC - JG: I have a consumer as output topic and the function is the input topic ---- 2020-05-15 20:14:34 UTC - JG: I also have a producer who produces messages on input topic ( the function ) ---- 2020-05-15 22:19:39 UTC - ckdarby: When configuring Pulsar for cloud block storage is there any specific settings I should look at tweaking. We're getting pretty garbage performance with ebs g2 with ledger reads. Pulsar is getting ~50-70 mbyte/s vs we're able to read directly off EBS at around 250 mbyte/s Perf reader gets around ~44k msg/s at ~ 500 mbit/s Went through all Grafana, we're not seeing any bottlenecks for CPU, gc pause, etc We've got the following configuration at least set: dbStorage_writeCacheMaxSizeMb: "2700" dbStorage_readAheadCacheMaxSizeMb: "2700" dbStorage_rocksDB_blockCacheSize: "1073741824" ---- 2020-05-15 22:38:27 UTC - David Kjerrumgaard: @Patrik Kleindl In your scenario, the latency on Kafka would be less because of the cost of the network jump to the BookKeeper layer. However, in the more traditional (default) replicated scenario, the opposite is true because the data is sent from the Pulsar Broker to the 3 Bookies simultaneously, so the writes on those bookies can occur in parallel. In Kafka, the sequence of events is more serial in nature. First the topic leader gets the data and writes to its local disk, then the other replicas get the data sent to them in a serial fashion, where they write the data to their local disk. HTH. +1 : Patrik Kleindl ---- 2020-05-16 02:15:33 UTC - Ruian: Thank you sijieg, here is the stats of one 6GB partition: ---- 2020-05-16 02:16:54 UTC - Ruian: And the stats of one 3xx partition: ----
