[ https://issues.apache.org/jira/browse/SPARK-29217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936186#comment-16936186 ]
holdenk commented on SPARK-29217: --------------------------------- Can you clarify what you mean by "Moving some files in the output while streaming"? > How to read streaming output path by ignoring metadata log files > ---------------------------------------------------------------- > > Key: SPARK-29217 > URL: https://issues.apache.org/jira/browse/SPARK-29217 > Project: Spark > Issue Type: Question > Components: Spark Core > Affects Versions: 2.4.3 > Reporter: Thanida > Priority: Minor > > As the output path of spark streaming contains `_spark_metadata` directory, > reading by > {code:java} > spark.read.format("parquet").load(filepath) > {code} > always depend on files listing in metadata log. > Moving some files in the output while streaming caused reading data failed. > So, how to read data in the streaming output path by ignoring metadata log > files? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org