Li Yuanjian created SPARK-22956:
-----------------------------------
Summary: Union Stream Failover Cause `IllegalStateException`
Key: SPARK-22956
URL: https://issues.apache.org/jira/browse/SPARK-22956
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.1.0
Reporter: Li Yuanjian
When we union 2 streams from kafka or other sources, while one of them have no
continues data coming and in the same time task restart, this will cause an
`IllegalStateException`. This mainly cause because the code in
[MicroBatchExecution|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L190]
, while one stream has no continues data, its comittedOffset same with
availableOffset during `populateStartOffsets`, and `currentPartitionOffsets`
not properly handled in KafkaSource. Also, maybe we should also consider this
scenario in other Source.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
