[ https://issues.apache.org/jira/browse/SPARK-29217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-29217. ---------------------------------- Resolution: Invalid Please ask questions to mailing list or stackoverflow. > How to read streaming output path by ignoring metadata log files > ---------------------------------------------------------------- > > Key: SPARK-29217 > URL: https://issues.apache.org/jira/browse/SPARK-29217 > Project: Spark > Issue Type: Question > Components: Spark Core > Affects Versions: 2.4.3 > Reporter: Thanida > Priority: Minor > > As the output path of spark streaming contains `_spark_metadata` directory, > reading by > {code:java} > spark.read.format("parquet").load(filepath) > {code} > always depend on files listing in metadata log. > Moving some files in the output while streaming caused reading data failed. > So, how to read data in the streaming output path by ignoring metadata log > files? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org