tvalentyn commented on issue #29663: URL: https://github.com/apache/beam/issues/29663#issuecomment-1849679440
It sounds reasonable to use `spark.local.dir` for Spark runner. > Unfortunately, this caused some Dataflow tests to fail (iirc) ... > it's usually a semi-persistent storage mount, such as a Kubernetes emptyDir Note that the rootcause of Dataflow's issue was that semi-persistent dir was backed by /var directory, which is often configured as `noexec` partition, presumably as an additional security measure. Per https://stackoverflow.com/questions/46525081/does-kubernetes-mount-an-emtpydir-volume-on-the-host , emptyDir also might live in /var. I wonder if you might encounter same frictions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
