[ https://issues.apache.org/jira/browse/SPARK-28597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-28597: ---------------------------------- Affects Version/s: (was: 2.4.3) 3.0.0 > Spark streaming terminated when close meta data log error > ---------------------------------------------------------- > > Key: SPARK-28597 > URL: https://issues.apache.org/jira/browse/SPARK-28597 > Project: Spark > Issue Type: Improvement > Components: DStreams, Structured Streaming > Affects Versions: 3.0.0 > Reporter: ulysses you > Priority: Major > > Now, HDFSMetaDataLog is the only implement of MetaDataLog. If > HDFSMetaDataLog.writeBatchToFile() execute error, then total app will be > terminated. > The meta data log is very important for spark, we should retry some times > when exception, and add a config to control retry times. > For example HDFS has a bug that close error if has no enough replicas detail > HDFS-11486. > -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org