2019-10-09 11:22:15 UTC - Retardust: Hi, I have questions about default functions behaviour and functions configuration <https://pulsar.apache.org/docs/en/functions-cli/> 1) retain-ordering option - what's the default behaviour? 2) auto-ack - what's the behaviour when enabled and when disabled? ---- 2019-10-09 11:26:25 UTC - Retardust: if function does not retain order order by default then it's very default unsafe behaviour ---- 2019-10-09 13:00:45 UTC - Retardust: Pulsar couldn't find deployed SerDe class. But it exists in my fat jar any idea? pulslar 2.4.1 ``` 12:59:00.903 [t1/n1/JournalBatchParseFunction-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - [t1/n1/JournalBatchParseFunction:0] Uncaught exception in Java Instance java.lang.RuntimeException: User class must be in class path at org.apache.pulsar.functions.utils.Reflections.createInstance(Reflections.java:67) ~[org.apache.pulsar-pulsar-functions-utils-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.instance.InstanceUtils.createInstance(InstanceUtils.java:84) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:?] at org.apache.pulsar.functions.instance.InstanceUtils.initializeSerDe(InstanceUtils.java:43) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:?] at org.apache.pulsar.functions.source.TopicSchema.newSchemaInstance(TopicSchema.java:197) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.source.TopicSchema.lambda$getSchema$0(TopicSchema.java:66) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_212] at org.apache.pulsar.functions.source.TopicSchema.getSchema(TopicSchema.java:66) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.sink.PulsarSink.initializeSchema(PulsarSink.java:330) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.sink.PulsarSink.open(PulsarSink.java:255) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupOutput(JavaInstanceRunnable.java:769) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:?] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupJavaInstance(JavaInstanceRunnable.java:203) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:?] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:234) [org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.lang.ClassNotFoundException: ru.alfabank.api.utils.JournalBatchSerde at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_212] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_212] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_212] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_212] at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_212] at org.apache.pulsar.functions.utils.Reflections.createInstance(Reflections.java:65) ~[org.apache.pulsar-pulsar-functions-utils-2.4.1.jar:2.4.1] ... 12 more
``` ---- 2019-10-09 13:02:27 UTC - Retardust: ---- 2019-10-09 13:04:55 UTC - Sijie Guo: Can you show the command that you used for running your function? ---- 2019-10-09 13:07:11 UTC - Retardust: ``` bin/pulsar-admin --admin-url <http://corpint4:6680> functions create --retain-ordering --jar <http://binary/artifactory/libs-release-local/ru/alfabank/api/utils/pulsar-util-function-api-pack/0.10.2/pulsar-util-function-api-pack-0.10.2-all.jar> --classname ru.alfabank.api.utils.JournalBatchParseFunction --inputs <persistent://t1/n1/queue_1> --output <persistent://t1/n1/queue_journal> --output-serde-classname ru.alfabank.api.utils.JournalBatchSerde --tenant t1 --namespace n1 ``` ---- 2019-10-09 13:29:08 UTC - Cartik: @Cartik has joined the channel ---- 2019-10-09 13:33:05 UTC - Sijie Guo: is the class packaged in the jar `<http://binary/artifactory/libs-release-local/ru/alfabank/api/utils/pulsar-util-function-api-pack/0.10.2/pulsar-util-function-api-pack-0.10.2-all.jar>`? ---- 2019-10-09 13:35:18 UTC - Retardust: yes, content of the jar file is on screenshot ---- 2019-10-09 13:39:44 UTC - Retardust: And if the class will not exists in jar then function will not be even deployed ---- 2019-10-09 13:58:34 UTC - Sijie Guo: Okay can you try with submitting with the jar file first? ---- 2019-10-09 13:59:01 UTC - Sijie Guo: I am not sure if it is related to http package handling or not ---- 2019-10-09 13:59:06 UTC - Retardust: ok ---- 2019-10-09 14:25:01 UTC - Raman Gupta: I'm still interested in answers to this one. Interestingly I just saw exactly the same message (with the exact same message count of 3600 unacked messages) in a totally different environment. ---- 2019-10-09 14:33:12 UTC - Retardust: @Sijie Guo same result:( ---- 2019-10-09 15:11:51 UTC - Naby: Does anybody else have a problem running the python pulsar-client on the new macOS (Catalina)? ---- 2019-10-09 15:59:14 UTC - Xiaolei Zhu: I couldn’t get 2.4 to run even before the update. The latest version that worked on my Mac was 2.2 ---- 2019-10-09 17:39:32 UTC - Naby: 2.4.1 worked for me on macOS 10.14 (Mojave). But, it does not work on macOS 10.15 (Catalina). I don’t know if it’s the Python or the pulsar-client. ---- 2019-10-09 17:41:37 UTC - thms.hmm: @thms.hmm has joined the channel ---- 2019-10-09 18:45:00 UTC - Nicolas Ha: I am trying to make an interceptor. I have a test that creates a producer/consumer and then calls `consumer.negativeAcknowledge(msg)` But for some reason `onNegativeAcksSend` always gets an emtpy set of messageIds - is this expected? ---- 2019-10-09 19:05:40 UTC - Sijie Guo: Can you get the function log for me so that I can take a look at the function log to see if loads the function jar correctly? ---- 2019-10-09 19:07:20 UTC - Raman Gupta: And now the same 3600 messages have timed out in a 3rd environment. ---- 2019-10-09 19:12:54 UTC - Sijie Guo: Nvm. I think I found the problem ---- 2019-10-09 19:13:36 UTC - Sijie Guo: It seems that you are using a customized serde and it seems recent changes made in Pulsar Function used a wrong class loader to load the serde class. +1 : Retardust ---- 2019-10-09 19:14:08 UTC - Raman Gupta: I created <https://github.com/apache/pulsar/issues/5349>. ---- 2019-10-09 19:15:16 UTC - Sijie Guo: I don’t think that is expected. It seems that a problem. @Penghui Li might have a better insight on this part. ---- 2019-10-09 19:18:05 UTC - Sijie Guo: Currently the python client is built on top a c++ client. We don’t provide the python client on some version of mac. For the full list of mac version, you can checkout the list of python client on PyPi. <https://pypi.org/project/pulsar-client/#files> If the macos version is not there, please create a github issue, we will try to address that. ---- 2019-10-09 19:56:21 UTC - Retardust: So my option is to downgrade pulsar and wait fix? Maybe there is another way to register custom jackson module avoiding custom serde usage? ---- 2019-10-09 19:59:42 UTC - Sijie Guo: Currently I am not sure which version has introduced the problem. tag @xiaolong.ran to help take a look at the fix. > Maybe there is another way to register custom jackson module avoiding custom serde usage? I am not sure what your Serde does. But if you have a POJO class, the function will use a JSON schema for serialization and deserialization. Using an existing Schema can get around this problem. ---- 2019-10-09 20:02:54 UTC - Sijie Guo: I just commented in the issue. ---- 2019-10-09 20:23:27 UTC - b_rousseau: @b_rousseau has joined the channel ---- 2019-10-09 20:29:31 UTC - Sijie Guo: hmm I can’t comment on github. I just repasted it here ---- 2019-10-09 20:29:36 UTC - Sijie Guo: ``` @rocketraman For the consumers, there is a ackTimeout setting. If you don't ack messages, the consumers will attempt redelivering message after ack timeout. You can set ackTimeout to -1 to disable it. ``` ---- 2019-10-09 20:30:56 UTC - Raman Gupta: I thought that was for if a consumer read a message, but then never acked it? I don't believe that is the situation here. ---- 2019-10-09 20:31:23 UTC - Raman Gupta: @Sijie Guo ---- 2019-10-09 20:39:35 UTC - Poule: @Naby you can build it <https://github.com/apache/pulsar/issues/5248#issuecomment-533939386> ---- 2019-10-09 20:42:34 UTC - Pete Tanski: For anyone looking at this thread, the problem is caused by the connection from the function workers and disappears if standalone is run with the option `-nfw` or `--no-functions-worker` ---- 2019-10-09 20:44:54 UTC - Raman Gupta: Based on what you just said, I realized the UnAckedMessageTracker does something different than I thought it did. However, the behavior is still unexplained. I've updated the issue report. ---- 2019-10-09 20:59:53 UTC - Retardust: I have bitset in my pojo and jackson have no default serializator for that. But I will fix that on my side:) Am I need to open issue or you will? ---- 2019-10-09 21:29:03 UTC - Addison Higham: huh... so... this is weird. My zookeeper quorum shows as being stable and clients can establish connections, but they never are successfully making a session... ---- 2019-10-09 22:35:52 UTC - Sandeep Kotagiri: Hello folks, I have trouble getting tiered storage to work with standalone pulsar 2.4.1. I am following instructions per the cookbook at <https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/> ---- 2019-10-09 22:37:10 UTC - Sandeep Kotagiri: I am seeing watning messages in the logs such as WARN org.apache.pulsar.broker.admin.impl.ClustersBase - [null] Failure-domain is not configured for cluster standalone org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /admin/clusters/standalone/failureDomain ---- 2019-10-09 22:37:37 UTC - Sandeep Kotagiri: And these warnings seem to be coming only when I configure tiered storage. ---- 2019-10-09 22:37:46 UTC - David Kjerrumgaard: @Sandeep Kotagiri Are you making your changes in the broker.conf file or the standalone.conf file. For standalone operation, you need to update the latter. ---- 2019-10-09 22:38:19 UTC - Sandeep Kotagiri: @David Kjerrumgaard, thank you for the pointer. I am making changes in the broker.conf ---- 2019-10-09 22:38:29 UTC - Sandeep Kotagiri: my mistake. I will change this out. ---- 2019-10-09 22:38:46 UTC - David Kjerrumgaard: @Sandeep Kotagiri No worries. Hopefully it was as simple as that! ---- 2019-10-09 22:38:50 UTC - Sandeep Kotagiri: and thank you. ---- 2019-10-10 02:43:47 UTC - Penghui Li: @Nicolas Ha It is not a expected behavior, can you help add an issue to describe how to reproduce the problem and the version of the client you are using? ---- 2019-10-10 02:59:58 UTC - Penghui Li: Seems no way to find the non-acked messages currently. ---- 2019-10-10 03:01:10 UTC - Penghui Li: Sorry, not clearly written in the java doc, it should be PulsarClientException. ---- 2019-10-10 03:04:55 UTC - xiaolong.ran: Hello @Retardust <https://github.com/apache/pulsar/issues/5350> and i will fix it. heart : Retardust ---- 2019-10-10 03:07:07 UTC - Penghui Li: About a “good” acknowledge, here is the document to explain the difference of timeout and nack <http://pulsar.apache.org/docs/en/concepts-messaging/#negative-acknowledgement>, nack is more flexible ---- 2019-10-10 04:56:00 UTC - Vivek Binjola: @Vivek Binjola has joined the channel ---- 2019-10-10 05:01:04 UTC - Sijie Guo: These are just warnings. They are fine ---- 2019-10-10 06:40:32 UTC - leonidv: @leonidv has joined the channel ---- 2019-10-10 07:15:01 UTC - leonidv: Hello! I'm preparing the internal report about Apache Pulsar. Could you please give advice when I should use exclusive subscription mode? Any use cases, articles? ---- 2019-10-10 08:12:55 UTC - Ali Ahmed: @leonidv it all depends on your communication flow, exclusive subscription is analog to single reader queue or point to point communication. Anywhere you need strong coupling between producer and consumer code exclusive subscription is preferred. ---- 2019-10-10 08:16:50 UTC - leonidv: Thank you, I'll research point-to-point communication pattern. ---- 2019-10-10 08:51:21 UTC - Retardust: @Sijie Guo sorry, but there is function+json scheme usage info in documentation? couldn't find ---- 2019-10-10 08:53:37 UTC - xiaolong.ran: @Retardust ---- 2019-10-10 08:53:43 UTC - xiaolong.ran: ---- 2019-10-10 08:54:03 UTC - Retardust: yes, it creates, but fails in runtime ---- 2019-10-10 08:54:38 UTC - Retardust: it will not create if class doesn't exists on creation time in the jar. If it's exists then it's fails then function runs ---- 2019-10-10 08:55:02 UTC - Retardust: you could see that in function logs on yours side I think ---- 2019-10-10 08:57:18 UTC - xiaolong.ran: cool, I have encountered the same error locally. +1 : Retardust ---- 2019-10-10 09:02:59 UTC - Retardust: ``` Caused by: org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException: Incompatible schema used at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:253) ~[java-instance.jar:?] at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:88) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.sink.PulsarSink$PulsarSinkProcessorBase.createProducer(PulsarSink.java:104) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] at org.apache.pulsar.functions.sink.PulsarSink$PulsarSinkAtMostOnceProcessor.<init>(PulsarSink.java:172) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:2.4.1] ``` when I try to run with ``` -st json ``` :disappointed: default jackson mapper serialize object correctly, but fails on broker( ----
