steveloughran commented on code in PR #37474: URL: https://github.com/apache/spark/pull/37474#discussion_r953592317
########## docs/cloud-integration.md: ########## @@ -231,9 +231,15 @@ The size of the window needs to be set to handle this. is no need for a workflow of write-then-rename to ensure that files aren't picked up while they are still being written. Applications can write straight to the monitored directory. -1. Streams should only be checkpointed to a store implementing a fast and -atomic `rename()` operation. -Otherwise the checkpointing may be slow and potentially unreliable. +1. In case of the default checkpoint file manager called `FileContextBasedCheckpointFileManager` +streams should only be checkpointed to a store implementing a fast and +atomic `rename()` operation. Otherwise the checkpointing may be slow and potentially unreliable. +On AWS S3 with Hadoop 3.3.1 or later the abortable stream based checkpoint file manager Review Comment: need to be specific about s3a connector, as EMR doesn't have it (yet). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
