2019-12-03 09:32:04 UTC - Nuno Ferreira: @Nuno Ferreira has joined the channel
----
2019-12-03 10:45:17 UTC - Nuno Ferreira: Hi all. I'm trying to develop a pulsar
function but currently the function only works in localrun mode. When I run the
function *on localrun mode it works as intended*. But when I run it in cluster
mode I get the following stack trace:
10:21:10.463 [function-timer-thread-86-1] ERROR
org.apache.pulsar.functions.runtime.ProcessRuntime - Health check failed for
ContextFunction-0
java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException:
UNAVAILABLE: io exception
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
~[?:1.8.0_222]
(....)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:530)
~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
(....)
... 1 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: /127.0.0.1:56999
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
~[?:1.8.0_222]
(....)
... 2 more
Caused by: <http://java.net|java.net>.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
~[?:1.8.0_222]
(....)
... 2 more
10:21:10.471 [function-timer-thread-86-1] ERROR
org.apache.pulsar.functions.runtime.ProcessRuntime - Extracted Process death
exception
java.lang.RuntimeException:
at
org.apache.pulsar.functions.runtime.ProcessRuntime.tryExtractingDeathException(ProcessRuntime.java:380)
~[org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:2.4.1]
(.....)
10:21:10.472 [function-timer-thread-86-1] ERROR
org.apache.pulsar.functions.runtime.RuntimeSpawner -
public/default/ContextFunction-java.lang.RuntimeException: Function Container
is dead with exception.. restarting
I'm willing to provide any aditional details that you find necessary. Thanks in
advance!
----
2019-12-03 11:06:47 UTC - jia zhai: @Nuno Ferreira what is your Pulsar version
and environment, and what are the steps to reproduce this?
----
2019-12-03 11:09:59 UTC - Nuno Ferreira: pulsar version 2.4.1
running on ubuntu
steps to reproduce:
1 - create the function: in my case
public class ContextFunction implements Function<User, Void>
@Override
public Void process(User user, Context context) throws Exception {
Attachment attachment = new Attachment(user.getName() + "from
user", user.getAge());
context.newOutputMessage("attachment-topic",
JSONSchema.of(Attachment.class)).value(attachment).sendAsync();
return null;
}
2 - mvn clean install to generate JAR file
3 - run the command to create pulsar function:
./pulsar-admin functions create --jar
/mnt/d/FDA/functions/target/functions-0.0.1-SNAPSHOT.jar --classname
ksa.jawwy.fda.pulsar.functions.mapper.ContextFunction --tenant public
--namespace default --inputs <persistent://public/default/topic> --output
<persistent://public/default/attachment-topic> --output-serde-classname
ksa.jawwy.fda.pulsar.functions.mapper.AttachmentSerDe
----
2019-12-03 11:29:41 UTC - tuteng: You can try check log file:
logs/functions/tenant-name/namespace-name/function-name/
----
2019-12-03 11:47:40 UTC - Nuno Ferreira: an interesting exception appeared:
java.lang.NoClassDefFoundError: org/apache/pulsar/client/impl/schema/JSONSchema
at
ksa.jawwy.fda.pulsar.functions.mapper.ContextFunction.process(ContextFunction.java:14)
~[functions-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at
ksa.jawwy.fda.pulsar.functions.mapper.ContextFunction.process(ContextFunction.java:9)
~[functions-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at
org.apache.pulsar.functions.instance.JavaInstance.handleMessage(JavaInstance.java:63)
~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:?]
at
org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:259)
[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: java.lang.ClassNotFoundException:
org.apache.pulsar.client.impl.schema.JSONSchema
at
<http://java.net|java.net>.URLClassLoader.findClass(URLClassLoader.java:382)
~[?:1.8.0_222]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_222]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_222]
... 5 more
----
2019-12-03 11:48:25 UTC - Nuno Ferreira: im using the JSONSchema class provided
by Pulsar
import org.apache.pulsar.client.impl.schema.JSONSchema;
----
2019-12-03 12:06:56 UTC - Nuno Ferreira: any idea of whats happening here?
----
2019-12-03 14:51:57 UTC - Nuno Ferreira: @tuteng @jia zhai
----
2019-12-03 15:09:36 UTC - jia zhai: @Nuno Ferreira It is strange. Not sure it
is related to your env or your configs.
Have you tried to run the examples provided by pulsar?
<https://github.com/apache/pulsar/wiki/Release-Candidate-Validation#validate-pubsub-and-java-functions>
----
2019-12-03 15:10:22 UTC - jia zhai: If the example runs success, then you could
compare and get what is the different for your own function.
----
2019-12-03 15:27:31 UTC - Nuno Ferreira: i found the solution. The JSONSchema
class was not packaged into my jar. It runs in localrun mode because my pc has
maven .m2 folder and (i guess xD) retrieve the class from there but in cluster
mode there's no maven repo. Thanks for your help @jia zhai
----
2019-12-03 15:28:10 UTC - Nuno Ferreira: and @tuteng :stuck_out_tongue:
----
2019-12-03 15:30:20 UTC - Eduardo Caceres: @David Kjerrumgaard I meant does
pulsar can ingest data from Kafka version 0.7.2. I have a legacy kafka cluster
that i need to ingest data from , but nothing seams to be compatible with it.
so I was wondering if Pulsar can ingest data from kafka topics 0.7.2
----
2019-12-03 15:53:18 UTC - jia zhai: :thumbsup_all:
----
2019-12-03 16:18:09 UTC - David Kjerrumgaard: @Eduardo Caceres There isn't a
tool that moves the data, but it should be simple enough to write a program
that reads the data from Kafka and publishes it to Pulsar.
----
2019-12-03 16:32:01 UTC - Eduardo Caceres: ok. thk.
----
2019-12-03 16:33:17 UTC - Sijie Guo: It seems to me that you need a connector
(most likely a KafkaSourceConnector) that reads data from your Kafka 0.7.0 and
writes data into pulsar topics.
----
2019-12-03 16:39:18 UTC - David Kjerrumgaard: Yes, if you want to continue to
forward data from Kafka 0.7.2 into Pulsar that would be a good approach. If
this is a one-time process, then any simple client should suffice. :smiley:
----
2019-12-03 16:59:55 UTC - Igor Zubchenok: In my cluster with huge number of
persistent topics, getting list of persistent topics works ok, but getting list
of non-persistent topics overloads brokers and gets the whole pulsar cluster
into an instable state.
----
2019-12-03 17:04:33 UTC - David Kjerrumgaard: Approximately how many
non-persistent topics do you have?
----
2019-12-03 20:15:24 UTC - Igor Zubchenok: zero, I don't use non-persistent.
----
2019-12-03 21:40:07 UTC - juraj: @juraj has joined the channel
----
2019-12-03 21:42:02 UTC - juraj: hi guys!:) the 2.4.2 release seems to linger
a little longer than usual, is there any estimation for availability pls?
----
2019-12-03 21:57:33 UTC - Jasdeep Singh: @Jasdeep Singh has joined the channel
----
2019-12-03 23:00:17 UTC - Sijie Guo: It should be done today or tomorrow. The
vote has run over 3 candidates. We need to make sure the release binaries
following the ASF policy. So sorry about the delay.
----
2019-12-03 23:16:53 UTC - juraj: thank you Sijie !
----
2019-12-04 01:28:44 UTC - NG: @NG has joined the channel
----
2019-12-04 09:05:17 UTC - Brian Doran: @tuteng Thanks Yes I had configured
prometheus.yml but still can't get the bookie stats and when I used the
template I can get Overview and Messaging stats but not bookies.
----
2019-12-04 09:07:41 UTC - tuteng: I'm not sure if you have turned on bookie's
http metrics service, you can test it.
```curl -v <http://bookie-ip:8000/metrics/>```
----
