2020-01-07 09:14:38 UTC - Lykant: @Lykant has joined the channel
----
2020-01-07 14:17:33 UTC - Alexandre DUVAL: There is a simple way to run pulsar
standalone without function worker ?
----
2020-01-07 15:11:13 UTC - Sijie Guo: `bin/pulsar standalone -nfw`
----
2020-01-07 15:12:05 UTC - vikash: Hello All,
i am using Apache Pulsar v2.4.2. and running as Standalone.
i am getting the NullPointerException checkNotNull exception in custom
connector and connector write() method not getting trigger while sending
message to pulsar.
1.I have taken refrence of pulsar-io jdbc built-in connector code and changed
driver connection to Apache Ignite JDBC.
2.Created AVRO schema for the pulsar topic and uploaded.
schema like ./connectors/schema_example.conf:
{
"type": "AVRO",
"schema":
"{\"type\":\"record\",\"name\":\"Test\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"int\"]},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}",
"properties": {}
}
Uploaded Schema:
/apache-pulsar-2.4.2/bin#./pulsar-admin schemas upload myjdbc --filename
./connectors/schema_example.conf
3.created connector on pulsar:
/apache-pulsar-2.4.2/bin#./pulsar-admin sinks localrun --archive
./connectors/pulsar-io-jdbc-2.4.2.nar --inputs myjdbc --name myjdbc
--sink-config-file ./connectors/pulsar-io-jdbc-ignite-sink.yaml
4.Open() method triggering and insert statement preparing successfully.
5.If topic already not exist and first time create command executing on pulsar
no exception.
but onces sending message to pulsar from producer.sendAsync(byteArrayMessage)
getting the NullPointerException checkNotNull exception
and Write() method not getting call/triggering
here is the exception and log message:
=================================================================================
root@biz-VBox:/opt/pulsar/apache-pulsar-2.4.2/bin#
root@biz-VBox:/opt/pulsar/apache-pulsar-2.4.2/bin#
root@biz-VBox:/opt/pulsar/apache-pulsar-2.4.2/bin# ./pulsar-admin sinks
localrun --archive ./connectors/pulsar-io-jdbc-2.4.2.nar --inputs myjdbc --name
myjdbc --sink-config-file ./connectors/pulsar-io-jdbc-ignite-sink.yaml
19:48:28.849 [main] INFO
<http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils
- Searching for connectors in /opt/pulsar/apache-pulsar-2.4.2/connectors
19:48:29.003 [main] INFO
<http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils
- Found connector ConnectorDefinition(name=ignite, description=Ignite sink
connector, sourceClass=null,
sinkClass=<http://org.apache.pulsar.io|org.apache.pulsar.io>.ignite.IgniteBytesSink)
from /opt/pulsar/apache-pulsar-2.4.2/connectors/pulsar-io-ignite-2.4.1.nar
19:48:29.612 [main] INFO
<http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils
- Found connector ConnectorDefinition(name=ignitetopictest, description=Jdbc
Ignite sink, sourceClass=null,
sinkClass=<http://org.apache.pulsar.io|org.apache.pulsar.io>.jdbc.ignite.JdbcAutoSchemaSink)
from /opt/pulsar/apache-pulsar-2.4.2/connectors/pulsar-io-jdbc-ignite-2.4.1.nar
19:48:30.113 [main] INFO
<http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils
- Found connector ConnectorDefinition(name=jdbc, description=Jdbc sink,
sourceClass=null,
sinkClass=<http://org.apache.pulsar.io|org.apache.pulsar.io>.jdbc.JdbcAutoSchemaSink)
from /opt/pulsar/apache-pulsar-2.4.2/connectors/pulsar-io-jdbc-2.4.1.nar
19:48:30.520 [main] INFO
<http://org.apache.pulsar.functions.utils.io|org.apache.pulsar.functions.utils.io>.ConnectorUtils
- Found connector ConnectorDefinition(name=myjdbc, description=Jdbc sink,
sourceClass=null,
sinkClass=<http://org.apache.pulsar.io|org.apache.pulsar.io>.jdbc.JdbcAutoSchemaSink)
from /opt/pulsar/apache-pulsar-2.4.2/connectors/pulsar-io-jdbc-2.4.2.nar
19:48:33.026 [main] INFO org.apache.pulsar.functions.runtime.RuntimeSpawner -
public/default/myjdbc-0 RuntimeSpawner starting function
19:48:33.029 [main] INFO org.apache.pulsar.functions.runtime.ThreadRuntime -
ThreadContainer starting function with instance config
InstanceConfig(instanceId=0, functionId=aa384dfc-da74-4e1a-9739-355f5f57aac4,
functionVersion=6c9f3846-7a21-4722-a453-a5f563d101f4, functionDetails=tenant:
"public"
namespace: "default"
name: "myjdbc"
className: "org.apache.pulsar.functions.api.utils.IdentityFunction"
autoAck: true
parallelism: 1
source {
typeClassName: "org.apache.pulsar.client.api.schema.GenericRecord"
inputSpecs {
key: "myjdbc"
value {
}
}
cleanupSubscription: true
}
sink {
className:
"<http://org.apache.pulsar.io|org.apache.pulsar.io>.jdbc.JdbcAutoSchemaSink"
configs:
"{\"jdbcUrl\":\"jdbc:ignite:<thin://localhost/>\",\"tableName\":\"ENTITIESTABLEINFO\"}"
typeClassName: "org.apache.pulsar.client.api.schema.GenericRecord"
}
resources {
cpu: 1.0
ram: 1073741824
disk: 10737418240
}
componentType: SINK
, maxBufferedTuples=1024, functionAuthenticationSpec=null, port=43629,
clusterName=local)
2020-01-07 19:48:33,952 - INFO - [public/default/myjdbc-0:JdbcAbstractSink@80]
- OPEN INVOKE>>>>>>>>
19:48:36.332 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x10072908,
L:/127.0.0.1:39486 - R:localhost/127.0.0.1:6650]] Connected to server
19:48:36.500 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.PulsarClientImpl - Configuring schema for topic
myjdbc : {
"name": "public/default/myjdbc",
"schema": {
"type": "record",
"name": "Test",
"fields": [
{
"name": "id",
"type": [
"null",
"int"
]
},
{
"name": "name",
"type": [
"null",
"string"
]
}
]
},
"type": "AVRO",
"properties": {}
}
19:48:37.141 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.schema.AutoConsumeSchema - Configure topic schema
for topic myjdbc :
{"type":"record","name":"Test","fields":[{"name":"id","type":["null","int"]},{"name":"name","type":["null","string"]}]}
19:48:37.319 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar
consumer perf with config: {
"topicNames" : [ "myjdbc" ],
"topicsPattern" : null,
"subscriptionName" : "public/default/myjdbc",
"subscriptionType" : "Shared",
"receiverQueueSize" : 1000,
"acknowledgementsGroupTimeMicros" : 100000,
"negativeAckRedeliveryDelayMicros" : 60000000,
"maxTotalReceiverQueueSizeAcrossPartitions" : 50000,
"consumerName" : null,
"ackTimeoutMillis" : 0,
"tickDurationMillis" : 1000,
"priorityLevel" : 0,
"cryptoFailureAction" : "CONSUME",
"properties" : {
"application" : "pulsar-sink",
"id" : "public/default/myjdbc",
"instance_id" : "0"
},
"readCompacted" : false,
"subscriptionInitialPosition" : "Latest",
"patternAutoDiscoveryPeriod" : 1,
"regexSubscriptionMode" : "PersistentOnly",
"deadLetterPolicy" : null,
"autoUpdatePartitions" : true,
"replicateSubscriptionState" : false,
"resetIncludeHead" : false
}
19:48:37.343 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config:
{
"serviceUrl" : "<pulsar://localhost:6650>",
"authPluginClassName" : null,
"authParams" : null,
"operationTimeoutMs" : 30000,
"statsIntervalSeconds" : 60,
"numIoThreads" : 1,
"numListenerThreads" : 1,
"connectionsPerBroker" : 1,
"useTcpNoDelay" : true,
"useTls" : false,
"tlsTrustCertsFilePath" : null,
"tlsAllowInsecureConnection" : false,
"tlsHostnameVerificationEnable" : false,
"concurrentLookupRequest" : 5000,
"maxLookupRequest" : 50000,
"maxNumberOfRejectedRequestPerConnection" : 50,
"keepAliveIntervalSeconds" : 30,
"connectionTimeoutMs" : 10000,
"requestTimeoutMs" : 60000,
"initialBackoffIntervalNanos" : 100000000,
"maxBackoffIntervalNanos" : 60000000000
}
19:48:37.434 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x557fb218,
L:/127.0.0.1:39488 - R:localhost/127.0.0.1:6650]] Connected to server
19:48:37.437 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ClientCnx - [id: 0x557fb218, L:/127.0.0.1:39488 -
R:localhost/127.0.0.1:6650] Connected through proxy to target broker at
192.168.1.97:6650
19:48:37.445 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ConsumerImpl - [myjdbc][public/default/myjdbc]
Subscribing to topic on cnx [id: 0x557fb218, L:/127.0.0.1:39488 -
R:localhost/127.0.0.1:6650]
19:48:37.503 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ConsumerImpl - [myjdbc][public/default/myjdbc]
Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0
19:48:37.557 [pulsar-client-io-1-1] INFO
com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider
initialized
19:48:37.715 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ConsumerImpl - [myjdbc] [public/default/myjdbc]
Closed consumer
19:48:37.942 [main] INFO org.apache.pulsar.functions.LocalRunner -
RuntimeSpawner quit because of
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:877)
~[com.google.guava-guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3950)
~[com.google.guava-guava-25.1-jre.jar:?]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3973)
~[com.google.guava-guava-25.1-jre.jar:?]
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4957)
~[com.google.guava-guava-25.1-jre.jar:?]
at
org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:94)
~[org.apache.pulsar-pulsar-client-original-2.4.2.jar:2.4.2]
at
org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:72)
~[org.apache.pulsar-pulsar-client-original-2.4.2.jar:2.4.2]
at
org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:36)
~[org.apache.pulsar-pulsar-client-original-2.4.2.jar:2.4.2]
at org.apache.pulsar.client.api.Schema.decode(Schema.java:97)
~[org.apache.pulsar-pulsar-client-api-2.4.2.jar:2.4.2]
at
org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:268)
~[org.apache.pulsar-pulsar-client-original-2.4.2.jar:2.4.2]
at
org.apache.pulsar.functions.source.PulsarRecord.getValue(PulsarRecord.java:74)
~[org.apache.pulsar-pulsar-functions-instance-2.4.2.jar:2.4.2]
at
org.apache.pulsar.functions.instance.JavaInstanceRunnable.readInput(JavaInstanceRunnable.java:473)
~[org.apache.pulsar-pulsar-functions-instance-2.4.2.jar:2.4.2]
at
org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:246)
~[org.apache.pulsar-pulsar-functions-instance-2.4.2.jar:2.4.2]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_232]
19:49:03.105 [function-timer-thread-5-1] ERROR
org.apache.pulsar.functions.runtime.RuntimeSpawner -
public/default/myjdbc-java.lang.NullPointerException Function Container is dead
with exception.. restarting
----
2020-01-07 15:13:23 UTC - Alexandre DUVAL: Ty!
----
2020-01-07 15:45:57 UTC - Roman Popenov: Looks like some of the defined fields
are null when the object is being deserialized and sinked as a record
----
2020-01-07 15:50:21 UTC - Roman Popenov: When you serialize the object from the
source and write it to a topic, it looks like the value is `null`
----
2020-01-07 16:31:29 UTC - Jocelyne Bergey: @Jocelyne Bergey has joined the
channel
----
2020-01-07 16:39:06 UTC - Lukas Chripko: did anybody succesfully build
install/build pulsar-client on alpine? dockerfile example working with alpine?
----
2020-01-07 16:47:42 UTC - juraj: u can create a "Snippet" post here on slack to
avoid flooding the channel
----
2020-01-07 16:50:52 UTC - juraj: seach for the keyword `build` on this slack, u
will find some stuff
----
2020-01-07 19:55:03 UTC - Martin Garnier: @Martin Garnier has joined the channel
----
2020-01-07 20:19:18 UTC - Mathieu Druart: Hi, we try to use the State storage
inside a Pulsar Function but we always have this error :
`java.lang.IllegalStateException: State is not enabled.`
----
2020-01-07 20:20:28 UTC - Mathieu Druart: The Function is run in a k8s cluster
deployed with the mini Helm configuration
----
2020-01-07 20:22:30 UTC - Mathieu Druart: @Sijie Guo we changed our code to use
a JSONSchema instead of the Avro Schema and now it works
----
2020-01-07 20:24:36 UTC - Mathieu Druart: Is there something to do to enable
State ?
----
2020-01-07 21:02:42 UTC - Pedro Cardoso: Yes, you need to add
```extraServerComponents:
"org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent"```
Under the bookkeeper.configData section, see
<https://github.com/apache/pulsar/issues/5687>
+1 : Mathieu Druart
----
2020-01-07 21:03:51 UTC - Pedro Cardoso: I believe is fixed on the 2.5.0
release candidate
----
2020-01-07 21:36:08 UTC - Martin Garnier: awesome, thanks Pedro :)
----
2020-01-07 22:08:44 UTC - Eron Wright: Would it make sense to have Pulsar SQL
listed as a connector in Presto documentation? Just curious.
<https://prestodb.io/docs/current/connector.html>
----
2020-01-07 22:41:10 UTC - Roman Popenov: So the functions are packaged in a
`jar`, but the connectors are in a `nar` ?
----
2020-01-07 22:42:03 UTC - Roman Popenov: Can I specify a `--jar` for Pulsar IO?
----
2020-01-07 22:42:49 UTC - Roman Popenov: or point it to a jar with the `-a`?
----
2020-01-07 22:43:27 UTC - Roman Popenov: ```Usage: pulsar-admin sources
[options] [command] [command options]
Commands:
create Submit a Pulsar IO source connector to run in a Pulsar cluster
Usage: create [options]
Options:
-a, --archive
The path to the NAR archive for the Source. It also supports
url-path [http/https/file (file protocol assumes that file already
exists on
worker host)] from which worker can download the package.```
----
2020-01-07 23:52:54 UTC - Roman Popenov: JAR also seems to be working just fine
as well
----
2020-01-08 00:01:24 UTC - Roman Popenov: I keep getting:
```00:00:22.491 [function-timer-thread-5-1] ERROR
org.apache.pulsar.functions.runtime.RuntimeSpawner -
test-tenant/test-ns/s3-org.apache.avro.AvroRuntimeException:
avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.avro.AvroRuntimeException: Can't find element type of Collection
Function Container is dead with exception.. restarting```
----
2020-01-08 00:17:50 UTC - Addison Higham: :thinking_face: has anyone tried to
write to s3 using the pulsar IO hdfs sink and s3a library? Quick read of the
code makes me think it should work is likely to produce duplicates and also
would likely cache to local disk... also not sure how in S3 it would chunk files
----
2020-01-08 00:21:13 UTC - Roman Popenov: s3a library works like a charm, I used
it in a Pulsar Function
----
2020-01-08 00:52:26 UTC - Roman Popenov: Guess this is related to
<https://github.com/streamnative/pulsar/issues/222>
----
2020-01-08 01:31:07 UTC - juraj: how far back does the broker search for a
messageid to fulfill a cumulative acknowledgement ? i'm trying after ~25
minutes (~1k msgs), but the ack call is basically ignored (persistent topic,
deduplication enabled)
----
2020-01-08 05:53:19 UTC - vikash: Hello ,I have struck in issue where i
want to know whats going on from server side
----
2020-01-08 05:53:31 UTC - vikash: how to start pulsar standalone as debug
mode
----
2020-01-08 05:53:47 UTC - vikash: bin/pulsar standalone --debug not working
----
2020-01-08 08:01:21 UTC - xiaolong.ran: You can configure it in `conf /
log4j.yml`, as follows:
``` - name: org.apache.pulsar.broker.service
level: debug
additivity: false
AppenderRef:
- ref: Console```
----
