phoerious commented on issue #29663: URL: https://github.com/apache/beam/issues/29663#issuecomment-1846100339
Spark local dir is pretty much your writable tmp scratch space. That's where all the temporary output files and shuffle spills land. If you run Spark in a container, it's usually a semi-persistent storage mount, such as a Kubernetes emptyDir. I think Flink has something similar to this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
