viirya commented on pull request #32702: URL: https://github.com/apache/spark/pull/32702#issuecomment-851065018
> And one more, I think let file stream sink to ignore metadata directory on reading existing metadata but write to the metadata directory is odd and error-prone. The metadata is no longer valid when Spark starts to write a new metadata on the same directory, and the option must be set to true for such directory to read properly despite Spark writes the metadata. There's no indication and end users have to memorize it. I don't know if it is a typo, but this doesn't let file stream sink but actually lets file stream source (and batch read path) to ignore metadata directory when reading the output of file stream sink. It doesn't change how file stream sink reads or writes to the metadata directory. Is it possible we are talking two different things? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
