[ https://issues.apache.org/jira/browse/SPARK-24404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liangchang Zhu closed SPARK-24404. ---------------------------------- > Increase currentEpoch when meet a EpochMarker in > ContinuousQueuedDataReader.next() in CP mode based on PR #21353 #21332 > #21293 and the latest master > ------------------------------------------------------------------------------------------------------------------------------------------------------ > > Key: SPARK-24404 > URL: https://issues.apache.org/jira/browse/SPARK-24404 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.3.0 > Reporter: Liangchang Zhu > Priority: Major > > In CP mode, based on PR #21353 #21332 #21293 and the latest master > ContinuousQueuedDataReader.next() will be invoked by > ContinuousDataSourceRDD.compute to return UnsafeRow. When currentEntry polled > from ArrayBlockingQueue is a EpochMarker, ContinuousQueuedDataReader will > send `ReportPartitionOffset` message to epochCoordinator with currentEpoch of > EpochTracker. The currentEpoch is a ThreadLocal variable, but now no place > invoke `incrementCurrentEpoch` to increase currentEpoch in its thread, so > `getCurrentEpoch` will return `None` all the time(because currentEpoch is > -1). This will cause exception when invoke `None.get`. At the same time, in > order to make the `ReportPartitionOffset` have correct semantics, we need > increase currentEpoch before send this message -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org