[jira] [Commented] (DRILL-6111) NullPointerException with Kafka Storage Plugin
[
https://issues.apache.org/jira/browse/DRILL-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346983#comment-16346983
]
Arina Ielchiieva commented on DRILL-6111:
-
[~akumarb2010] & [~kam_iitkgp] could you please take a look?
> NullPointerException with Kafka Storage Plugin
> --
>
> Key: DRILL-6111
> URL: https://issues.apache.org/jira/browse/DRILL-6111
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Affects Versions: 1.12.0
>Reporter: Jared Stehler
>Priority: Major
>
> I'm unable to query using the kafka storage plugin; queries are failing with
> a NPE which *seems* like a json typo:
> {code:java}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment 1:2
> [Error Id: 49d5f72f-0187-480b-8b29-6eeeb5adc88f on 10.80.53.16:31820]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
> ~[drill-common-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298)
> [drill-java-exec-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> [drill-java-exec-1.12.0.jar:1.12.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
> [drill-java-exec-1.12.0.jar:1.12.0]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.12.0.jar:1.12.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation
> of [simple type, class org.apache.drill.exec.store.kafka.KafkaSubScan] value
> failed (java.lang.NullPointerException): null
> at [Source: {
> "pop" : "single-sender",
> "@id" : 0,
> "receiver-major-fragment" : 0,
> "receiver-minor-fragment" : 0,
> "child" : {
> "pop" : "selection-vector-remover",
> "@id" : 1,
> "child" : {
> "pop" : "limit",
> "@id" : 2,
> "child" : {
> "pop" : "kafka-partition-scan",
> "@id" : 3,
> "userName" : "",
> "columns" : [ "`*`" ],
> "partitionSubScanSpecList" : [ {
> "topicName" : "ingest-prime",
> "partitionId" : 5,
> "startOffset" : 8824294,
> "endOffset" : 8874172
> }, {
> "topicName" : "ingest-prime",
> "partitionId" : 1,
> "startOffset" : 8826346,
> "endOffset" : 8874623
> }, {
> "topicName" : "ingest-prime",
> "partitionId" : 6,
> "startOffset" : 8824744,
> "endOffset" : 8874617
> } ],
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "KafkaStoragePluginConfig" : {
> "type" : "kafka",
> "kafkaConsumerProps" : {
> "key.deserializer" :
> "org.apache.kafka.common.serialization.ByteArrayDeserializer",
> "auto.offset.reset" : "earliest",
> "bootstrap.servers" :
> "kafkas.dev3.master.us-west-2.prod.aws.intellify.io:9092",
> "enable.auto.commit" : "true",
> "group.id" : "drill-query-consumer-1",
> "value.deserializer" :
> "org.apache.kafka.common.serialization.ByteArrayDeserializer",
> "session.timeout.ms" : "3"
> },
> "enabled" : true
> },
> "cost" : 0.0
> },
> "first" : 0,
> "last" : 2,
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 2.0
> },
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 2.0
> },
> "destination" : "CgsxMC44MC41My4xNhDM+AEYzfgBIM74ATIGMS4xMi4wOAA=",
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 2.0
> }; line: 49, column: 7] (through reference chain:
> org.apache.drill.exec.physical.config.SingleSender["child"]->org.apache.drill.exec.physical.config.SelectionVectorRemover["child"]->org.apache.drill.exec.physical.config.Limit["child"])
> at
> com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:263)
> ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
> at
> com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:453)
> ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
> at
> com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:472)
> ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
> at
> com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:258)
> ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
> at
> com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:135)
> ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:444)
>
[jira] [Commented] (DRILL-6111) NullPointerException with Kafka Storage Plugin
[
https://issues.apache.org/jira/browse/DRILL-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345479#comment-16345479
]
Anton Gozhiy commented on DRILL-6111:
-
Reproduced with the Drill version 1.13.0-SNAPSHOT (commit id:
a9ea4ec1c5645ddab4b7aef9ac060ff5f109b696)
*Prerequisites:*
*1.* Kafka topic was created with 5 partitions:
{code:java}
bin/kafka-topics.sh --create --zookeeper --replication-factor 1
--partitions 5 --topic test
{code}
*2.* Test data - json file with 10 key-value records:
{code:java}
{"key": "value_1"}
{"key": "value_2"}
{"key": "value_10"}
{code}
*3.* The data is added to the topic:
{code:java}
cat test_kafka_big_file.json | bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic test
{code}
*4.* Kafka storage plugin is enabled in Drill with the following configuration:
{code:java}
{
"type": "kafka",
"kafkaConsumerProps": {
"key.deserializer":
"org.apache.kafka.common.serialization.ByteArrayDeserializer",
"auto.offset.reset": "earliest",
"bootstrap.servers": "localhost:9092",
"enable.auto.commit": "true",
"group.id": "drill-query-consumer-1",
"value.deserializer":
"org.apache.kafka.common.serialization.ByteArrayDeserializer",
"session.timeout.ms": "3"
},
"enabled": true
}
{code}
*Steps:*
*1*. Run the following query from Drill:
{code:java}
select * from kafka.test limit 5;
{code}
*Expected result:*
Query should be completed successfully.
*Actual result*
NPE happens:
{noformat}
Error: SYSTEM ERROR: NullPointerException
Fragment 1:0
[Error Id: 1e62c2ac-1f41-4284-86ef-af262169bcce on node1:31010]
(com.fasterxml.jackson.databind.JsonMappingException) Instantiation of
[simple type, class org.apache.drill.exec.store.kafka.KafkaSubScan] value
failed (java.lang.NullPointerException): null
at [Source: {
"pop" : "single-sender",
"@id" : 0,
"receiver-major-fragment" : 0,
"receiver-minor-fragment" : 0,
"child" : {
"pop" : "selection-vector-remover",
"@id" : 1,
"child" : {
"pop" : "limit",
"@id" : 2,
"child" : {
"pop" : "project",
"@id" : 3,
"exprs" : [ {
"ref" : "`**`",
"expr" : "`**`"
} ],
"child" : {
"pop" : "kafka-partition-scan",
"@id" : 4,
"userName" : "",
"columns" : [ "`*`" ],
"partitionSubScanSpecList" : [ {
"topicName" : "test",
"partitionId" : 2,
"startOffset" : 0,
"endOffset" : 2
}, {
"topicName" : "test",
"partitionId" : 4,
"startOffset" : 0,
"endOffset" : 2
}, {
"topicName" : "test",
"partitionId" : 1,
"startOffset" : 0,
"endOffset" : 2
}, {
"topicName" : "test",
"partitionId" : 3,
"startOffset" : 0,
"endOffset" : 2
}, {
"topicName" : "test",
"partitionId" : 0,
"startOffset" : 0,
"endOffset" : 2
} ],
"initialAllocation" : 100,
"maxAllocation" : 100,
"KafkaStoragePluginConfig" : {
"type" : "kafka",
"kafkaConsumerProps" : {
"key.deserializer" :
"org.apache.kafka.common.serialization.ByteArrayDeserializer",
"auto.offset.reset" : "earliest",
"bootstrap.servers" : "localhost:9092",
"enable.auto.commit" : "true",
"group.id" : "drill-query-consumer-1",
"value.deserializer" :
"org.apache.kafka.common.serialization.ByteArrayDeserializer",
"session.timeout.ms" : "3"
},
"enabled" : true
},
"cost" : 0.0
},
"outputProj" : false,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 10.0
},
"first" : 0,
"last" : 5,
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 5.0
},
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 5.0
},
"destination" : "CgVub2RlMRCi8gEYo/IBIKTyATIPMS4xMy4wLVNOQVBTSE9UOAA=",
"initialAllocation" : 100,
"maxAllocation" : 100,
"cost" : 5.0
}; line: 66, column: 9] (through reference chain:
org.apache.drill.exec.physical.config.SingleSender["child"]->org.apache.drill.exec.physical.config.SelectionVectorRemover["child"]->org.apache.drill.exec.physical.config.Limit["child"]->org.apache.drill.exec.physical.config.Project["child"])
com.fasterxml.jackson.databind.JsonMappingException.from():263
com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException():453
com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem():472
