2020-04-28 09:56:27 UTC - Ebere Abanonu: @Sijie Guo something broke in consumer in 2.5.1? The issue I have with function consumer failing avro schema validation is happening with client consumer, even previous schemas which worked in 2.5.0 is now failing validation in 2.5.1. Meanwhile presto can consume successfully without issue. I will downgrade to 2.5.0 to verify +1 : Konstantinos Papalias ---- 2020-04-28 10:13:09 UTC - Sijie Guo: original is supported +1 : Lari Hotari ---- 2020-04-28 10:13:21 UTC - Sijie Guo: it includes all unshaded dependencies ---- 2020-04-28 10:13:57 UTC - Sijie Guo: which version are you running? ---- 2020-04-28 10:14:37 UTC - Sijie Guo: Do you have the code sample? @Penghui Li can help check that. ---- 2020-04-28 10:17:28 UTC - Ebere Abanonu: For the function <https://github.com/eaba/PulsarTestFunction> ---- 2020-04-28 10:18:34 UTC - Lari Hotari: Thanks. I'm just running in an issue with the circe checksum dependency. At least on Gradle, it resolves to circe-checksum-4.10.0.nar file. This works when the file is directly the classpath. However, I'm using <https://github.com/GoogleContainerTools/jib> to create a Docker container. That defaults to use -cp "/app/resources:/app/classes:/app/libs/*" , where the last part relies on JVM's wildcard classpath. That only accepts files that end with .jar or .JAR and that's why circe-checksum-4.10.0.nar won't get loaded. I did find a workaround, to explicitly add the .nar file to the classpath for the container entrypoint command by adding "jib.container.extraClasspath = ['/app/libs/circe-checksum-4.10.0.nar']" to the gradle build file.
It seems that circe checksum also has a jar packaging besides the nar packaging in maven central (<https://repo1.maven.org/maven2/org/apache/bookkeeper/circe-checksum/4.10.0/>). However, the maven pom doesn't reference that. I wonder why the default packaging for circe checksum is using "nar"? ---- 2020-04-28 10:18:36 UTC - Ebere Abanonu: I did helm upgrade from 2.5.0 to 2.5.1 ---- 2020-04-28 10:25:26 UTC - Sankaran Srinivasan: @Sankaran Srinivasan has joined the channel ---- 2020-04-28 10:25:32 UTC - Thamarai Kannan: @Thamarai Kannan has joined the channel ---- 2020-04-28 10:39:17 UTC - Ebere Abanonu: for the client side cmd 45 to register function <https://github.com/eaba/SharpPulsar/blob/master/Sample/Program.cs> ---- 2020-04-28 11:06:35 UTC - Lari Hotari: UPDATE: I found a workaround for Gradle and pulsar-client-original issue about circe-checksum-4.10.0.nar file. By adding this dependency, it will also add the circe-checksum-4.10.0.jar file to the classpath and it works with the Jib built container ```implementation 'org.apache.bookkeeper:circe-checksum:4.10.0@jar'``` ---- 2020-04-28 13:09:49 UTC - Subash Kunjupillai: Hi, We are planning for an application that will poll messages from Pulsar and do some processing and will store the information in HBase. We see pulsar (2.5.0) is using Guava 25 but HBase (1.1) is using Guava 12. So as the Guava 25 jar loads first, we are facing issue with the HBase call. As we don't directly access Guava libraries, it is very little we could do over here, it is either Pulsar or HBase should support over here. Our Application --> Apache Pulsar --> Guava 25 Our Application --> Apache HBase --> Guava 12 I would like to understand is there any feasibility of providing a shaded jar by Pulsar for guava to handle these kind of dependency issues like it was done in <https://gite.lirmm.fr/yagoubi/spark/commit/c9f743957fa963bc1dbed7a44a346ffce1a45cf2|Apache Spark>? Or do we foresee any other way to handle it. ---- 2020-04-28 13:09:49 UTC - Subash Kunjupillai: ```Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604) at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588) at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:567) at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1252) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1218) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1192) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1398) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1221) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1221) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) ... 20 more - Notifications will be logged with default definition [Loaded sun.reflect.GeneratedConstructorAccessor54 from __JVM_DefineClass__] Error in retriving notification definition meta data - javax.jdo.JDODataStoreException [Loaded org.apache.pulsar.common.api.proto.PulsarApi$CommandPing$Builder from file:/home/admin/mari/pulsarlib/org.apache.pulsar-pulsar-common-2.5.0.jar] [Loaded org.apache.pulsar.common.api.proto.PulsarApi$CommandPing$Builder$1 from file:/home/admin/mari/pulsarlib/org.apache.pulsar-pulsar-common-2.5.0.jar] at com.ericsson.bss.platform.oam.notification.dao.NotificationDefinitionDAO.getAllNotificationDefinitions(NotificationDefinitionDAO.java:35) at com.ericsson.bss.platform.oam.notification.NotificationConfig.refreshNotificationDefinition(NotificationConfig.java:137) at com.ericsson.bss.platform.oam.notification.NotificationConfig$2.run(NotificationConfig.java:47) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604) at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588) at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:567) at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1252) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1218) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1192) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1398) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1221) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)``` ---- 2020-04-28 13:18:48 UTC - Ebere Abanonu: @Sijie Guo it works fine on a fresh clean kubernetes cluster. Am suspecting schema version differences. Now I need help in understanding this error: ---- 2020-04-28 13:18:48 UTC - Ebere Abanonu: ``` 2020-04-28T13:11:34.118649226Z 13:11:34.118 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xcb0840a4, L:/10.244.0.9:56450 - R:pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/10.244.4.6:6650]] Connected to server 2020-04-28T13:11:34.122824067Z 13:11:34.119 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [<persistent://public/default/covid-19-mobile][test-function-sub]> Subscribing to topic on cnx [id: 0xcb0840a4, L:/10.244.0.9:56450 - R:pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/10.244.4.6:6650] 2020-04-28T13:11:34.140431506Z 13:11:34.140 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [<persistent://public/default/covid-19-mobile][test-function-sub]> Subscribed to topic on pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/10.244.4.6:6650 -- consumer: 0 2020-04-28T13:11:34.170694879Z 13:11:34.170 [pulsar-client-io-1-1] INFO com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized 2020-04-28T13:11:34.229850313Z 13:11:34.225 [public/default/Covid19-function-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - [public/default/Covid19-function:0] Uncaught exception in Java Instance 2020-04-28T13:11:34.229878615Z java.lang.NullPointerException: null 2020-04-28T13:11:34.229884116Z at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:877) ~[java-instance.jar:?] 2020-04-28T13:11:34.229889216Z at com.google.common.cache.LocalCache.get(LocalCache.java:3950) ~[java-instance.jar:?] 2020-04-28T13:11:34.229893516Z at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3973) ~[java-instance.jar:?] 2020-04-28T13:11:34.229897917Z at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4957) ~[java-instance.jar:?] 2020-04-28T13:11:34.229902417Z at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:100) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1] 2020-04-28T13:11:34.229906718Z at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:96) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1] 2020-04-28T13:11:34.229911018Z at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:39) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1] 2020-04-28T13:11:34.229915518Z at org.apache.pulsar.client.api.Schema.decode(Schema.java:95) ~[java-instance.jar:?] 2020-04-28T13:11:34.229919719Z at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:274) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1] 2020-04-28T13:11:34.229924119Z at org.apache.pulsar.functions.source.PulsarRecord.getValue(PulsarRecord.java:74) ~[org.apache.pulsar-pulsar-functions-instance-2.5.1.jar:2.5.1] 2020-04-28T13:11:34.22993982Z at org.apache.pulsar.functions.instance.JavaInstanceRunnable.readInput(JavaInstanceRunnable.java:472) ~[org.apache.pulsar-pulsar-functions-instance-2.5.1.jar:?] 2020-04-28T13:11:34.229945121Z at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:246) [org.apache.pulsar-pulsar-functions-instance-2.5.1.jar:?] 2020-04-28T13:11:34.229950521Z at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] 2020-04-28T13:11:34.237697754Z 13:11:34.237 [public/default/Covid19-function-0] INFO org.apache.pulsar.functions.instance.JavaInstanceRunnable - Closing instance 2020-04-28T13:11:34.24364264Z 13:11:34.243 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [<persistent://public/default/covid-19-mobile>] [test-function-sub] Closed consumer 2020-04-28T13:11:34.24779788Z 13:11:34.247 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [<persistent://public/default/covid-19>] [pulsar-3-8] Closed Producer 2020-04-28T13:11:34.248366326Z 13:11:34.248 [public/default/Covid19-function-0] INFO org.apache.pulsar.functions.instance.JavaInstanceRunnable - Unloading JAR files for function InstanceConfig(instanceId=0, functionId=01106ad5-f00d-4b82-9bb4-d57ebfb1b643, functionVersion=228e71cb-ef8b-4da3-89ca-a45722ddeb18, functionDetails=tenant: "public" 2020-04-28T13:11:34.248380727Z``` ---- 2020-04-28 13:42:39 UTC - Ebere Abanonu: @Sijie Guo it is all green now and am now a happy man! I got the above error resolved by setting the schemaType to avro. Successfully consuming and producing ---- 2020-04-28 13:43:11 UTC - Ebere Abanonu: Am using 2.5.1 ---- 2020-04-28 13:56:58 UTC - Frank Kelly: If I specify PULSAR_PID_DIR - can someone tell me the name of the file that will be created? I am setting things as follows in my docker container but see no "obvious" file in /tmp (using Apache Pulsar 2.5.1) ```ENV PULSAR_PID_DIR="/tmp" RUN export PULSAR_PID_DIR``` ---- 2020-04-28 15:45:03 UTC - Gilles Barbier: @Ebere Abanonu you had the issue with a Json schema? ---- 2020-04-28 16:22:12 UTC - Ebere Abanonu: With an Avro Schema ---- 2020-04-28 16:22:49 UTC - Ebere Abanonu: May likely occur with a Json Schema ---- 2020-04-28 18:39:07 UTC - Barak Bloch: @Barak Bloch has joined the channel ---- 2020-04-28 18:41:00 UTC - Devin G. Bost: If anyone missed my presentation today on Real-World Architectural Patterns with Apache Pulsar, you can access the recording here: <https://youtu.be/pmaCG1SHAW8> thanks : Konstantinos Papalias, Yuvaraj Loganathan, Ali Ahmed +1 : Konstantinos Papalias, Frank Kelly, David Kjerrumgaard, Sijie Guo, Oleg Kozlov, Ali Ahmed, wuYin, Yuvaraj Loganathan bananadance : Karthik Ramasamy ---- 2020-04-28 20:38:00 UTC - Sijie Guo: by default pulsar-client shade every dependencies ---- 2020-04-28 20:38:29 UTC - Sijie Guo: Which version of pulsar-client are you using? ---- 2020-04-28 20:40:06 UTC - Sijie Guo: @Ebere Abanonu In most of the case, functions will infer the schema type when the topic exists. If the topic doesn’t exist, you might need to specify the schemaType when submitting functions. ---- 2020-04-28 20:40:40 UTC - Sijie Guo: Glad that you already figured it out. ---- 2020-04-28 21:21:57 UTC - Oleg Kozlov: Hello, i have a question regarding negative acknowledgements support in binary protocol. For context: we developed our own in-house pulsar client lib for Erlang, and need to support neg-acks. Based on this: <http://pulsar.apache.org/docs/en/concepts-messaging/#negative-acknowledgement> there is a way to send negative ack from client to broker, however we can't find a NegativeAcknowledgement command in <http://pulsar.apache.org/docs/en/develop-binary-protocol/> . Is there a separate message type for this, or how can this be implemented. We prefer to rely on explicit neg ack's as opposed to "redeliver unacknowledged messages" and acknowledgement timeout. ---- 2020-04-29 01:14:33 UTC - Tymm: pulsar 2.4.2 ---- 2020-04-29 02:00:41 UTC - Slackbot: This message was deleted. -1 : Jared Mackey, Greg Methvin, Steven Duncan, Chris ---- 2020-04-29 02:05:43 UTC - plsr: please do not use `@channel`, especially in channels that every user is in (e.g. <#C5Z4T36F7|general>). you just sent a notification to literally everyone in this workspace. i think you'll find that people do not appreciate that. ---- 2020-04-29 02:56:55 UTC - Sajith Sebastian: make sense. Going to repost without a broadcast ---- 2020-04-29 02:57:38 UTC - Sajith Sebastian: Hello ,Want to enable AuthenticationProviderBasic in standalone mode. Made changes in standalone.conf & functions_worker.yml to enable authentication. failing with error "ERROR org.apache.pulsar.functions.worker.WorkerService - Error Starting up in worker org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException: HTTP 401 Unauthorized" Can someone share the instructions to follow. ---- 2020-04-29 06:16:11 UTC - Ebere Abanonu: If the topic exists without sschema, function registers a schema for it. That was the start of my problems ---- 2020-04-29 06:52:26 UTC - Xavier Levaux: Hi @Frank Kelly. You better use a dedicated instance of Pulsar (and anything else) when doing tests. Tests should be predictable and reproduceable. ---- 2020-04-29 07:41:43 UTC - Tymm: Hello, how to enable state? I am using pulsar 2.4.2, running in standalone mode, trying to access state in a pulsar function using the context object, but exception: "State is not enabled" is thrown... Please advice ---- 2020-04-29 08:24:38 UTC - Pierre-Yves Lebecq: I noticed the same issue on 2.5.0. I did not find any solution for this. ---- 2020-04-29 08:33:18 UTC - Pierre-Yves Lebecq: Hey. How are you running your function? I once had this issue when using the command “pulsar-admin functions localrun”. When switching to “pulsar-admin functions create” instead, it worked fine. Hope this helps. ---- 2020-04-29 08:48:44 UTC - Gilles Barbier: Hi everyone, on node.js, when a create a consumer with a “Shared” or “Failover” subscription ``` const consumer = await client.subscribe({ topic: 'my-topic', subscription: 'my-sub', subscriptionType: 'Shared', ackTimeoutMs: 10000, });``` I can launch 2 consumers. But when I use a “KeyShared” subscriptionType, the second consumer is rejected with “Failed to create consumer: ConsumerBusy”. Do I miss something? (using pulsar-client 1.1.0 and pulsar 2.5.1) ----
