2018-11-28 09:11:22 UTC - chenaschen: @chenaschen has joined the channel ---- 2018-11-28 11:11:34 UTC - Ivan Kelly: ya, I'm just talking from the point of view of what the word 'bastion' actually means. it's like a defensive outpost, which would be more like what the proxy is ---- 2018-11-28 11:17:24 UTC - Yifan: :disappointed_relieved: pulsar client only supports python 2.7? ---- 2018-11-28 11:20:47 UTC - Yifan: misled by the pypi doc… looks will support python3 … phew ---- 2018-11-28 12:21:59 UTC - Yifan: is anyone using Pulsar Python client? how is namespace used with python client? is topic “<persistent://tenant/namespace/topic>” ? ---- 2018-11-28 12:22:31 UTC - Ivan Kelly: yes ---- 2018-11-28 12:23:24 UTC - Yifan: thanks @Ivan Kelly I am also getting pulsar-client==2.2.0 not found error when using pip, are you using 2.2.0? ---- 2018-11-28 12:24:47 UTC - Ivan Kelly: <https://pypi.org/project/pulsar-client/2.2.0/#files> <- it exists, perhaps the problem is your config. What OS/python version are you using? ---- 2018-11-28 12:25:40 UTC - Yifan: both my docker and osx failed. I saw on documentation that only 3.7 is supported on osx ---- 2018-11-28 12:27:08 UTC - Yifan: trying on my linux box now ---- 2018-11-28 12:27:24 UTC - Ivan Kelly: for python 3, only 3.7 ---- 2018-11-28 12:31:02 UTC - Yifan: Documentation says 3.6 is supported on Linux, because I have other dependencies will only work on python 3.6. I hope it will work … ---- 2018-11-28 13:28:38 UTC - Samuel Sun: I face this error when I ran ./bin/pulsar standalone, ``` 21:26:39.821 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir data/standalone/zookeeper/version-2 snapdir data/standalone/zookeeper/version-2 21:26:39.876 [main] INFO org.apache.zookeeper.server.NIOServerCnxnFactory - binding to port 0.0.0.0/0.0.0.0:2181 Exception in thread "main" java.lang.NoSuchMethodError: org.apache.pulsar.zookeeper.SerializeUtilsAspect.aspectOf()Lorg/apache/pulsar/zookeeper/SerializeUtilsAspect; at org.apache.zookeeper.server.util.SerializeUtils.deserializeTxn(SerializeUtils.java:56) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:641) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:553) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:531) at org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:358) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:193) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:176) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:217) at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:284) at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:407) at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118) at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:190) at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:383) at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:259) at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:102) 21:26:39.914 [Thread-1] ERROR org.apache.pulsar.PulsarStandaloneStarter - Shutdown failed: null ``` ---- 2018-11-28 13:28:48 UTC - Samuel Sun: any suggestion ? ---- 2018-11-28 13:32:07 UTC - Ivan Kelly: version? ---- 2018-11-28 13:32:38 UTC - Samuel Sun: git master, java 1.8.0_192 ---- 2018-11-28 13:35:46 UTC - Ivan Kelly: you've done a maven clean? ---- 2018-11-28 13:36:02 UTC - Samuel Sun: trying ---- 2018-11-28 13:39:09 UTC - Yifan: quick question, what is subscriptionRolePrefix in reader configuration? ---- 2018-11-28 13:43:50 UTC - Ivan Kelly: each subscription is given a name in pulsar, whether it's durable or not ---- 2018-11-28 13:44:06 UTC - Ivan Kelly: the prefix is prepended onto the subscription name used by the reader ---- 2018-11-28 13:46:10 UTC - Yifan: so reader will generate its own subscription name? and will be different every time I create a new reader? ---- 2018-11-28 13:49:13 UTC - Ivan Kelly: yes ---- 2018-11-28 13:49:37 UTC - Ivan Kelly: String subscription = "reader-" + DigestUtils.sha1Hex(UUID.randomUUID().toString()).substring(0, 10); ---- 2018-11-28 13:50:54 UTC - Yifan: thanks. I was hoping to use reader to replace subscribe, given the option to seek to earlest message or latest message, looks like they are not interchangeable ---- 2018-11-28 13:56:59 UTC - Ivan Kelly: not really. Consumer has seek though, if that's what you need ---- 2018-11-28 13:59:05 UTC - Yifan: I see, got it. Thanks a lot. ---- 2018-11-28 13:59:26 UTC - Ivan Kelly: np ---- 2018-11-28 14:32:38 UTC - Sanjeev Kulkarni: @Yifan which configuration if not set is leading to crash? ---- 2018-11-28 14:33:11 UTC - Sanjeev Kulkarni: @Yifan 3.6 should be supported on linux. Did that work for you? ---- 2018-11-28 14:48:54 UTC - Yifan: @Sanjeev Kulkarni I basically had to set all optional configuration with default value null to something not null. But I am using docker image, not master in github. The code in KafkaAbstractSource are different. I assume that it has been fixed in master ---- 2018-11-28 14:50:20 UTC - Yifan: as for python 3.6 on Linux, it did work for me on my Linux box. ---- 2018-11-28 14:50:55 UTC - Yifan: I am using a slim down python 3.6 in my docker image, it didn’t work there. I am upgrading to use 3.7 now ---- 2018-11-28 14:55:55 UTC - Yifan: I think that the documentation can be improved. can you point me where I can fork the documentation? ---- 2018-11-28 14:57:59 UTC - Samuel Sun: @Ivan Kelly, fixed after mvn clean, thanks ---- 2018-11-28 14:58:24 UTC - Ivan Kelly: @Samuel Sun good to hear ---- 2018-11-28 14:59:44 UTC - Samuel Sun: one more question for this class NoopLoadManager, the function in this class is for which scenario ? ---- 2018-11-28 15:02:02 UTC - Samuel Sun: <https://github.com/apache/pulsar/blob/03d1302c91506df1dace73a60689e78ea589505f/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/NoopLoadManager.java> ---- 2018-11-28 15:03:08 UTC - Samuel Sun: got it, it's only for the standalone ---- 2018-11-28 15:23:20 UTC - Stepan: @Stepan has joined the channel ---- 2018-11-28 15:32:36 UTC - Stepan: Hi, I'm having a problem with pulsar functions. When using custom `SerDe` that serializes and deserializes particular protobuf schema it crashes with following exception: ```15:25:05.090 [public/default/Router-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - Failed to close source java.lang.NullPointerException: null at org.apache.pulsar.functions.source.PulsarSource.close(PulsarSource.java:145) ~[java-instance.jar:2.2.0] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.close(JavaInstanceRunnable.java:360) [java-instance.jar:2.2.0] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:231) [java-instance.jar:2.2.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]``` I'm using pulsar 2.2.0 and specify the serde with ```--custom-serde-inputs '{"<non-persistent://tenant/namespace/topic>": "com.example.CustomSerde"}'``` this error doesn't happen when using String or even when using custom `SerDe` but only specifying `--inputs` parameter (it complains about not being able to convert String to my class). Any idea what do I do wrong? ---- 2018-11-28 17:21:43 UTC - Christophe Bornet: Hi, it would be great if consumers of a cross-datacenter stretch cluster could read in priority on brokers of their own region. This would prevent expensive cross-dc network (it's actually a blocking point for me). FYI there's a discussion for it in Kafka : <https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica> . Would be great to also consider this feature in Pulsar. WDYT ? ---- 2018-11-28 18:42:32 UTC - Matteo Merli: @Christophe Bornet That is already possible :wink: ---- 2018-11-28 18:43:38 UTC - Matteo Merli: <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#priorityLevel-int->
It works on shared subscriptions. From 2.3 it will also work on failover subscriptions ---- 2018-11-28 18:51:51 UTC - Sijie Guo: do you mind filing a github issue for us? ---- 2018-11-28 18:52:45 UTC - Sijie Guo: @Samuel Sun that is used for running pulsar in standalone ---- 2018-11-28 18:57:54 UTC - Yifan: Hi, I have a subscription on a topic, I tried to do `pulsar-admin topics unsubscribe -s tenant/namespace/function <persistent://tenant/namespace/topic>`, it says 404, subscription not found. When I list subscriptions with `pulsar-admin topics subscriptions`, it is listed together with other subscriptions… not sure what’s wrong ---- 2018-11-28 18:59:28 UTC - Matteo Merli: there is a bug in CLI tool with subscription names with `/`. There is a fix in an open PR <https://github.com/apache/pulsar/pull/2910>. this is scheduled for 2.3 (or 2.2.1) ---- 2018-11-28 19:40:33 UTC - Byron: @Matteo Merli Just curious, but is there any inherent technical reason why a reader cannot be created for a partitioned topic? ---- 2018-11-28 19:41:53 UTC - Matteo Merli: The main reason is that a reader always takes a MessageId, to specify where to start reading from. In a partitioned topic, there would be a list of them ---- 2018-11-28 19:43:13 UTC - Matteo Merli: For 2.3, we’ve added a method in `PulsarClient` to retrieve the list of partition. The intended usage is to get the list and then create readers on the individual partitions ---- 2018-11-28 19:47:49 UTC - Byron: I see. Since a message ID includes a partition, specifying one would be exclusive to the partition and not the topic. ---- 2018-11-28 19:50:20 UTC - Byron: For 2.3 that sounds like a reasonable solution. The client can then subscribe to each partition and fan-in the messages. The only issue would be if additional partitions are added and the client would need to be aware of this. ---- 2018-11-28 19:51:31 UTC - Matteo Merli: Yes, though the current behavior is a bit “raw” even when you subscribe to the partitioned topic. Currently you have to bounce producers/consumers to have them pick up the new partitions :confused: ---- 2018-11-28 19:55:20 UTC - Byron: Right. This is true for changing the number of partitions in general? ---- 2018-11-28 19:55:34 UTC - Matteo Merli: Changing the partitions only ---- 2018-11-28 19:55:53 UTC - Byron: Ok. I was thinking about the _safe_ way to do this the other day ---- 2018-11-28 19:56:25 UTC - Byron: If the producers are bounced first, they will publish to new partitions. The consumers can be bounced and they will just pick up these new partitions. ---- 2018-11-28 19:57:24 UTC - Byron: The concern I have is splitting a stream of messages (with the same key) across different partitions and having parallel consumers process the messages ---- 2018-11-28 19:57:43 UTC - Byron: But if the producers are bounced first, then that shouldn’t happen ---- 2018-11-28 19:57:56 UTC - Byron: (assuming all messages in the previous partitions are processed already) ---- 2018-11-28 20:00:17 UTC - Matteo Merli: When you increase number of partitions, it will automatically ensure the subscriptions are created on the new partitions, so you don’t lose data in the process ---- 2018-11-28 20:00:53 UTC - Matteo Merli: In any case, there will be a brief out-of-order scenario possible, since the keys will be hashing to different partitions ---- 2018-11-28 20:03:55 UTC - Byron: > there will be a brief out-of-order scenario possible, since the keys will be hashing to different partitions Yes, understood ---- 2018-11-28 20:41:02 UTC - Christophe Bornet: @Matteo Merli I don't see how the consumer priorityLevel can bring the feature. To my understanding, at one point in time one broker is the partition owner and all reads go to that broker. Reads from consumers of the other DC will result in cross-dc bandwidth consumption. What would be needed is that these consumer read from a broker on their dc (so not the partition owner but another read-only one). ---- 2018-11-28 20:51:51 UTC - Beast in Black: Hi @Matteo Merli and team, I'm seeing some broker policy errors on a namespace (possibly related to partitioned topics, but I'm not using those) when creating producers (and also consumer) to a nonpersistent topic in a namespace. I'm using pulsar-2.2 for brokers and the 2.1 CPP client. The cpp client library shows this error when trying to create a subscriber to the topic: ``` 07:06:03.430 [0x7fa4cd197700] ERROR /myapp/3rdParty/pulsar-client-cpp/lib/ClientConnection.cc(699) - [100.96.3.9:35232 -> 100.96.3.5:6650] Failed partition-metadata lookup req_id: 33 error: 1 07:06:03.430 [0x7fa4cd197700] ERROR /myapp/3rdParty/pulsar-client-cpp/lib/ClientImpl.cc(138) - Error Checking/Getting Partition Metadata while creating producer on <non-persistent://diagnostics/local.diagnostics.guestnamespace/diagnostics_nonpersist_stream> -- 5 ``` Broker log (one of the brokers, I have 3) shows this *repeatedly*: ``` 22:26:28.416 [ForkJoinPool.commonPool-worker-0] ERROR org.apache.pulsar.broker.web.PulsarWebResource - Policies not found for diagnostics/local.diagnostics.guestnamespace namespace 22:26:28.416 [ForkJoinPool.commonPool-worker-0] WARN org.apache.pulsar.broker.service.ServerCnx - Failed to get Partitioned Metadata [/100.96.2.13:41676] <non-persistent://diagnostics/local.diagnostics.guestnamespace/_routing>: Policies not found for diagnostics/local.diagnostics.guestnamespace namespace org.apache.pulsar.broker.web.RestException: Policies not found for diagnostics/local.diagnostics.guestnamespace namespace at org.apache.pulsar.broker.web.PulsarWebResource.lambda$4(PulsarWebResource.java:657) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) ~[?:1.8.0_171] at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_171] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_171] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_171] at org.apache.pulsar.zookeeper.ZooKeeperDataCache.lambda$0(ZooKeeperDataCache.java:66) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.2.0.jar:2.2.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) ~[?:1.8.0_171] at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_171] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_171] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_171] at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$7(ZooKeeperCache.java:349) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.2.0.jar:2.2.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) ~[?:1.8.0_171] at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_171] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_171] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_171] at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$13(ZooKeeperCache.java:334) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.2.0.jar:2.2.0] at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) [?:1.8.0_171] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_171] at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_171] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_171] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) [?:1.8.0_171] ``` Any idea how I can fix this `Policies not found` issue? I checked the pulsar documentation but perhaps I missed something related to this... ---- 2018-11-28 20:52:03 UTC - Stepan: sure, will do ---- 2018-11-28 20:53:01 UTC - Beast in Black: Again, I'm not using parititioned topics - AFAIK the default config is to use non-partitioned topics, and I have not set anything specific for this. ---- 2018-11-28 21:08:19 UTC - Sanjeev Kulkarni: @Stepan this bug has been fixed in the master and will be released in 2.3 ---- 2018-11-28 21:08:55 UTC - Stepan: ah, can you please point me to the commit/github issue? ---- 2018-11-28 21:11:08 UTC - Sanjeev Kulkarni: @Yifan the documentation of python client is at <https://github.com/apache/pulsar/blob/master/site2/docs/client-libraries-python.md> ---- 2018-11-28 21:11:35 UTC - Sanjeev Kulkarni: You can fork and submit a pr. Looking forward to seeing it :) +1 : jia zhai, Yifan ---- 2018-11-28 21:15:56 UTC - Christophe Bornet: What would be nice would be to have region aware election of partition owners. So that we could configure for a given topic or partition on which DC the owner broker shall be chosen preferably. ---- 2018-11-29 01:22:39 UTC - Sanjeev Kulkarni: <https://github.com/apache/pulsar/pull/3064> ---- 2018-11-29 06:11:57 UTC - Shaun: @Shaun has joined the channel ----
