shrutig opened a new pull request #25514: [SPARK-28784]StreamExecution and StreamingQueryManager should utilize CheckpointFileManager to interact with checkpoint directories URL: https://github.com/apache/spark/pull/25514 ### What changes were proposed in this pull request? After PR https://github.com/apache/spark/pull/21048, the CheckpointFileManager interface was created to handle all structured streaming checkpointing operations and helps users to choose how they wish to write checkpointing files atomically. StreamExecution and StreamingQueryManager still uses some FileSystem operations without using the CheckpointFileManager. For instance, https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L137 https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L392 Instead, StreamExecution and StreamingQueryManager should use CheckpointFileManager for these operations. ### Why are the changes needed? This change will allow users to use CheckpointFileManager for structured streaming checkpointing files without need for a separate FileSystem implementation for the same. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing tests
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
