Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/20675 > it just means that for very long-running streams task restarts will eventually run out. Ah, I know your means. Yeah, if we support task level retry we should also set the task retry number unlimited. > But if you're worried that the current implementation of task restart will become incorrect as more complex scenarios are supported, I'd definitely lean towards deferring it until continuous processing is more feature-complete. Yep, the "complex scenarios" I mentioned mainly including shuffle and aggregation scenario like comments in https://issues.apache.org/jira/browse/SPARK-20928?focusedCommentId=16245556&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16245556, in those scenario maybe task level retry should consider epoch align, but current implementation of task restart is completed for map-only continuous processing I think. Agree with you about deferring it, so I just leave a comment in SPARK-23033 and close this or you think this should reviewed by others? > Do you want to spin that off into a separate PR? (I can handle it otherwise.) Of cause, #20689 added a new interface `ContinuousDataReaderFactory` as our comments.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org