2019-03-26 10:33:21 UTC - Ganga Lakshmanasamy: @Chris Bartholomew Not able to execute command in docker based setup. Can you please share the steps to do that ---- 2019-03-26 13:21:37 UTC - Matteo Merli: I don’t think there’s a direct example, but you should just need to pass the token in the Autorizaton header. ---- 2019-03-26 13:22:34 UTC - Matteo Merli: Eg: `Authorization: Bearer anassaasaassaasss` ---- 2019-03-26 13:32:31 UTC - Chris DiGiovanni: I was wondering if anyone has seen a issue with the s3 offloader stalling? I can seem to get it going again by bouncing all my brokers, though it always ends up stalling. Not much to by in the logs though I do see a couple issues in these logs. Example of the errors I see are attached. ---- 2019-03-26 13:35:12 UTC - Chris DiGiovanni: You might notice I am using 2.4.0 snapshot that @jia zhai provided for another issue. Though I'm seeing the same behavior on the 2.3.0 offloader as well. ---- 2019-03-26 15:01:00 UTC - Sriram KS: @Sriram KS has joined the channel ---- 2019-03-26 15:25:31 UTC - Sriram KS: Hi all , May i know if Pulsar IO balances workload for a COnnector Instance like Kafka Connect? I mean if a Task has parallelism as 2 and number of brokers = 2 will it be balanced across the two brokers? ---- 2019-03-26 16:22:34 UTC - Sanjeev Kulkarni: @Ganga Lakshmanasamy one option would be to login to a docker container and run the command. also attach a --admin-url=<pulsar-admin-url> to redirect it to the right broker ---- 2019-03-26 16:24:41 UTC - Sanjeev Kulkarni: @Sriram KS yes. The instances will be spread across all available nodes able to run sources/sinks/functions ---- 2019-03-26 16:47:39 UTC - Sree Vaddi: <https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/73968> ---- 2019-03-26 17:06:14 UTC - Sree Vaddi: Welcome, Remco. Team, Remco created LinkedIn Group for us and building an awesome product based on Apache Pulsar. Let's hear more from @Remco Verhoef. ---- 2019-03-26 17:14:29 UTC - Chris Bartholomew: @Ganga Lakshmanasamy, if you are using Kubernetes, you should be able to run the command on the broker pod (or and admin/bastion pod if you have one) like this: ```kubectl exec <broker-pod-name> bin/pulsar-admin topics skip <topic> -s <subscription> -n 1``` ---- 2019-03-26 18:41:45 UTC - Sriram KS: @Sanjeev Kulkarni Does Pulsar IO has internal topics like kafka connect for managing configs, offsets and status? ---- 2019-03-26 18:47:01 UTC - Sanjeev Kulkarni: Internally pulsar io maps to a function. It has its own metadata to manage which functions are running what’s their status, distribution etc ---- 2019-03-26 19:20:18 UTC - Jean-Bernard van Zuylen: Hi all, we are currently having an issue on some our topics. We are unable to connect to them: broker is throwing `Incorrect parameter input` ---- 2019-03-26 19:20:52 UTC - Jean-Bernard van Zuylen: ---- 2019-03-26 19:20:58 UTC - Grant Wu: ~Revert to Pulsar 2.2.1~ ---- 2019-03-26 19:21:07 UTC - Grant Wu: wait no ---- 2019-03-26 19:21:11 UTC - Grant Wu: Sorry, this is the _other_ issue ---- 2019-03-26 19:21:29 UTC - Grant Wu: Let me copy paste what I have in my company slack: ---- 2019-03-26 19:21:34 UTC - Grant Wu: Just to recap - if the Pulsar broker crashloop issue happens again - these are the steps to fix the brokers:
1. Enter a Zookeeper shell, by SSH/kubectl exec/k8s dashboard “exec” ’ing into a Zookeeper pod, and then running `zkCli.sh` (It’s in the PATH already). 2. Enter the command `rmr /managed-ledgers/public/functions/persistent/coordinate` in a Zookeeper shell 3. To speed up the recovery process, delete the crashlooping Pulsar pods ---- 2019-03-26 19:21:45 UTC - Grant Wu: We deploy in k8s - you may need to adapt the process if you deploy differently ---- 2019-03-26 19:22:27 UTC - Grant Wu: Alternatively, you may upgrade to Pulsar 2.3.0 ---- 2019-03-26 19:22:32 UTC - Grant Wu: Which has this fix: <https://github.com/apache/pulsar/pull/3487> ---- 2019-03-26 19:23:24 UTC - Jean-Bernard van Zuylen: @Grant Wu thanks! ---- 2019-03-26 19:54:00 UTC - Sriram KS: @Sanjeev Kulkarni Thanks for the info ---- 2019-03-26 19:55:26 UTC - Jean-Bernard van Zuylen: @Grant Wu to be sure, I guess that in our case the command to be executed is `rmr /managed-ledgers/proxyclick-qa/core/persistent/events`, correct? ---- 2019-03-26 19:55:55 UTC - Grant Wu: Wait ---- 2019-03-26 19:56:00 UTC - Grant Wu: Sorry, I jumped to conclusions ---- 2019-03-26 19:56:04 UTC - Grant Wu: I am not sure if that’s safe to run ---- 2019-03-26 19:56:28 UTC - Grant Wu: @Matteo Merli can you weigh in here? It looks like a similar issue to what I was dealing with with Pulsar Functions, but in something else entirely ---- 2019-03-26 19:57:00 UTC - Jean-Bernard van Zuylen: Here it's a topic ---- 2019-03-26 20:00:47 UTC - Matteo Merli: Yes, the procedure reported by @Grant Wu is correct. The fix is in that PR (and 2.3.0). Doing `rmr /managed-ledgers/proxyclick-qa/core/persistent/events` will manually drop the subscription in this case. If the subscription has valid data, you can recreate it and position that to either a message id or a timestamp ---- 2019-03-26 20:03:59 UTC - Jean-Bernard van Zuylen: Thanks for the info! ---- 2019-03-26 21:37:18 UTC - Prakash Saraf: @Prakash Saraf has joined the channel ---- 2019-03-26 21:38:17 UTC - Prakash Saraf: hi, can we subscribe to multiple topics belonging to different tenant and namespace in multi-topic subscription? ---- 2019-03-26 21:38:55 UTC - Sree Vaddi: welcome @Prakash Saraf ---- 2019-03-26 22:08:19 UTC - Prakash Saraf: hi @Sree Vaddi ---- 2019-03-26 22:09:50 UTC - Matteo Merli: Yes, you can pass an arbitrary list of topics (as long as you have permissions to consume to these topics) ---- 2019-03-27 03:20:47 UTC - bossbaby: ``` 19:45:08.121 [SessionTracker] INFO org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x100a44704030002, timeout of 30000ms exceeded 19:45:08.203 [ProcessThread(sid:0 cport:2181):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x100a44704030002 19:45:08.313 [SyncThread:0] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /localhost:44986 which had sessionid 0x100a44704030002 19:45:29.610 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /localhost:56016 19:45:29.638 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.ZooKeeperServer - Client attempting to renew session 0x100a44704030002 at /localhost:56016 19:45:29.655 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.ZooKeeperServer - Invalid session 0x100a44704030002 for client /localhost:56016, probably expired 19:45:29.737 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /localhost:56016 which had sessionid 0x100a44704030002 ``` Why pulsar does not reconnect zookeeper? ---- 2019-03-27 05:13:23 UTC - Shivji Kumar Jha: I am trying to enable tsl on broker 1. Added key and certs to all brokers 2. Added ca key to /usr/local/share/ca-certificates and 3. changed broker.conf to paths to these key files on both the brokers ``` brokerServicePortTls=6651 tlsEnabled=true # Path for the TLS certificate file tlsCertificateFilePath=/etc/pulsar/certs/pulsar-node-3.bm.infra.crt # Path for the TLS private key file tlsKeyFilePath=/etc/pulsar/certs/pulsar-node-3.bm.infra.key # Path for the trusted TLS certificate file. # This cert is used to verify that any certs presented by connecting clients # are signed by a certificate authority. If this verification # fails, then the certs are untrusted and the connections are dropped. tlsTrustCertsFilePath=/usr/local/share/ca-certificates/pulsar-pki-ca.pem``` 4. changed client to use pulsar+ssl:// … 5. opened 6651 port for inbound traffic on brokers ---- 2019-03-27 05:14:01 UTC - Shivji Kumar Jha: ERROR: org.apache.pulsar.client.api.PulsarClientException: Connection already closed ---- 2019-03-27 05:15:20 UTC - Matteo Merli: did you set `brokerServicePortTls=6651` ? ---- 2019-03-27 05:15:35 UTC - Shivji Kumar Jha: Yes ---- 2019-03-27 05:16:04 UTC - Matteo Merli: where is the client connecting then? ---- 2019-03-27 05:18:14 UTC - Shivji Kumar Jha: ---- 2019-03-27 05:18:35 UTC - Shivji Kumar Jha: This is my client instance on java client - from my debugger ---- 2019-03-27 05:19:15 UTC - Shivji Kumar Jha: By the way I get this error on producer create() method ---- 2019-03-27 05:22:33 UTC - Matteo Merli: Yes, that’s when the connection will be attempted the first time (if there’s not already one connection available) ---- 2019-03-27 05:23:05 UTC - Matteo Merli: if you try to connect to that host:port, does it work? ---- 2019-03-27 05:23:10 UTC - Matteo Merli: like, with telnet ---- 2019-03-27 05:24:04 UTC - Matteo Merli: also, can you paste the client log. that will also tell which broker specifically it’s trying to connect ---- 2019-03-27 05:26:15 UTC - Shivji Kumar Jha: telnet is fine telnet pulsar.stage.bm.infra 6651 Trying 10.10.58.7... Connected to pulsar.stage.bm.infra. Escape character is ‘^]’. ---- 2019-03-27 05:27:59 UTC - Shivji Kumar Jha: stacktrace ---- 2019-03-27 05:30:14 UTC - Matteo Merli: what about the logs on console? :slightly_smiling_face: ---- 2019-03-27 05:30:51 UTC - Shivji Kumar Jha: I will have to configure slf4j on the client, Will do in a moment SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”. SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details. ---- 2019-03-27 05:31:08 UTC - Matteo Merli: Ok ---- 2019-03-27 05:31:11 UTC - Shivji Kumar Jha: Is there a way to redirect this to stdout maybe? ---- 2019-03-27 05:31:28 UTC - Matteo Merli: You need to have an implementation of Slf4j ---- 2019-03-27 05:31:38 UTC - Shivji Kumar Jha: OK i will do that. ---- 2019-03-27 05:32:02 UTC - Matteo Merli: you can start by adding ``` <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> <version>1.7.26</version> </dependency> ``` +1 : Shivji Kumar Jha ---- 2019-03-27 05:36:20 UTC - Shivji Kumar Jha: [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x3ada4e76, L:/10.8.0.65:56749 - R:pulsar.stage.bm.infra/10.10.58.6:6651]] Connected to server [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - Error during handshake javax.net.ssl.SSLException: SSLEngine closed already at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [pulsar.stage.bm.infra/10.10.58.6:6651] Got exception DecoderException : javax.net.ssl.SSLHandshakeException: General SSLEngine problem org.apache.pulsar.shade.io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) at org.apache.pulsar.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) at org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591) at org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) at org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ... 16 more ---- 2019-03-27 05:38:29 UTC - Shivji Kumar Jha: @Yuvaraj Loganathan ^ ---- 2019-03-27 05:39:21 UTC - Matteo Merli: > javax.net.ssl.SSLHandshakeException: General SSLEngine problem ---- 2019-03-27 05:39:33 UTC - Matteo Merli: seems an error using the keys ---- 2019-03-27 05:39:57 UTC - Matteo Merli: either the path is incorrect or the format ---- 2019-03-27 05:40:09 UTC - Shivji Kumar Jha: the path is fine, i have verified that ---- 2019-03-27 05:40:38 UTC - Matteo Merli: does it print any error like this on broker side? ---- 2019-03-27 05:40:51 UTC - Shivji Kumar Jha: Quoting docs ```The broker expects the key to be in PKCS 8 format, so convert it.``` This must be the issue i guess. I have a pem file ---- 2019-03-27 05:41:12 UTC - Shivji Kumar Jha: broker.log has nothing ---- 2019-03-27 05:48:23 UTC - Shivji Kumar Jha: Thank you for help @Matteo Merli I will fix that and check ---- 2019-03-27 05:52:58 UTC - Matteo Merli: :+1: ---- 2019-03-27 05:53:01 UTC - Zarrar: Hi guys, When I enable authentication and authorization in broker, what properties do I need to change in functions_worker.yml I have tried PF_authenticationEnabled, PF_clientAuthenticationPlugin and PF_clientAuthenticationParameters but no luck ---- 2019-03-27 05:53:33 UTC - Zarrar: the functions worker is running inside the broker ---- 2019-03-27 07:29:30 UTC - bhagesharora: Hello guys, I am trying to implemtnt "Pulsar adaptor for Apache Kafka" through java, and I am following same document from pulsar apache website <https://pulsar.apache.org/docs/en/adaptors-kafka/> I am following github example for implementation <https://github.com/apache/pulsar/tree/master/pulsar-client-kafka-compat/pulsar-client-kafka-tests/src/test/java/org/apache/pulsar/client/kafka/compat/examples> I have added the maven dependecny in my pom.xml file <dependency> <groupId>org.apache.pulsar</groupId> <artifactId>pulsar-client-kafka</artifactId> <version>2.3.0</version> </dependency> After the maven build success, same file I put into the target location and running ConsumerExample.java and ProducerExample.java in parallel window When I am running ProducerExample.java class, I am facing Exeption. see below Exception in thread "main" java.lang.NoSuchMethodError: org.apache.pulsar.client.api.PulsarClient.getPartitionsForTopic(Ljava/lang/String;)Ljava/util/concurrent/CompletableFuture; at org.apache.kafka.clients.producer.KafkaProducer.readPartitionsInfo(KafkaProducer.java:237) at org.apache.kafka.clients.producer.KafkaProducer.createNewProducer(KafkaProducer.java:229) at org.apache.kafka.clients.producer.KafkaProducer.lambda$send$1(KafkaProducer.java:161) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:161) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:153) at pulsar_producer_consumer.pulsar_producer_consumer.ProducerExample.main(ProducerExample.java:26) What would be the solution ?? ---- 2019-03-27 08:06:11 UTC - jia zhai: @Chris DiGiovanni There seems be 2 issues here： 1， java.lang.NoClassDefFoundError: com/jamesmurty/utils/XMLBuilder This seems like a leak of related jar. 2, bk read index error, this is related to this issue: <https://github.com/apache/bookkeeper/issues/1970> I will fix the first issue, and make a try. ----
