andygrove opened a new issue, #2767: URL: https://github.com/apache/datafusion-ballista/issues/2767
**Describe the bug** When `work_dir` isn't set, `start_executor_process` (`ballista/executor/src/executor_process.rs`) takes the path of a new `TempDir` and drops the `TempDir`: ```rust } else if let Some(temp_dir) = TempDir::new()?.path().to_str().map(ToOwned::to_owned) ``` Dropping it deletes the directory, so the logged `Executor working directory` doesn't exist. The first task creates it again, through DataFusion's `DiskManager` and the shuffle writer, but: - with the process's default permissions rather than `TempDir`'s `0700`: under a `022` umask the directory is `drwxr-xr-x` and its shuffle files `-rw-r--r--`, so other users can read them where the temporary directory is shared, such as `/tmp` on Linux; - and nothing removes it when the executor exits, since the guard that would have is gone, so its shuffle files stay behind unless `job_data_clean_up_interval_seconds` is set. **To Reproduce** Start an executor without `--work-dir`. The directory it logs as its working directory doesn't exist, even after it registers with the scheduler. Run a query and the directory reappears as `drwxr-xr-x`, holding `<job id>/<stage>/<partition>/data-0.arrow` files with `-rw-r--r--`, and it stays after the executor stops. **Expected behavior** The executor holds the `TempDir` for as long as it runs, so the directory exists with `0700` permissions from startup and is removed when the executor exits. **Additional context** Seen with 55.0.0, through Apache DataFusion Comet's executor binary, which uses this default when `COMET_EXECUTOR_WORK_DIR` isn't set; `main` at 70a074322 has the same code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
