Grigory Turunov created SPARK-9780:
--------------------------------------
Summary: In case of invalid initialization of KafkaDirectStream,
NPE is thrown
Key: SPARK-9780
URL: https://issues.apache.org/jira/browse/SPARK-9780
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.4.1, 1.3.1
Reporter: Grigory Turunov
Priority: Minor
[o.a.s.streaming.kafka.KafkaRDD.scala#L143|https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala#L143]
In initialization of KafkaRDDIterator, there is an addition of
TaskCompletionListener to the context, which calls close() to the consumer,
which is not initialized yet (and will be initialized 12 lines after that).
If something happens in this 12 lines (in my case there was a private
constructor for valueDecoder), an Exception, which is thrown, triggers
context.markTaskCompleted() in
[o.a.s.scheduler.Task.scala#L90|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/Task.scala#L90]
which throws NullPointerException, when tries to call close() for
non-initialized consumer.
This masks original exception - so it is very hard to understand, what is
happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
