andygrove opened a new issue, #7060: URL: https://github.com/apache/datafusion-comet/issues/7060
### What is the problem the feature request solves? Every native plan builds its own `RuntimeEnv` in [`prepare_datafusion_session_context`](https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/core/src/execution/jni_api.rs#L994-L1011). That `RuntimeEnv` has a `DiskManager` in `DiskManagerMode::Directories` over the executor's local disk directories, which [`CometExecIterator`](https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/spark/src/main/scala/org/apache/comet/CometExecIterator.scala#L115) takes from `SparkEnv.get.blockManager.getLocalDiskDirs`. DataFusion 55.2 creates that mode's directories as soon as the `DiskManager` is built. `create_local_dirs` makes a `datafusion-*` temp directory in every configured directory, and each is removed recursively when the plan's `RuntimeEnv` is dropped. So every native plan does a `stat` and a `mkdir` for each local disk when it is created, and a recursive delete for each local disk when it is released, whether or not it ever spills. DataFusion's `OsTmpDirectory` mode, by contrast, creates its directory only on the first `create_tmp_file`. I measured this with a CPU profile of `SELECT count(*) FROM lineitem WHERE l_quantity > 49` on TPC-H SF2, using `spark.sql.files.maxPartitionBytes=1m` and `spark.sql.files.openCostInBytes=0`. That gives 470 map tasks with one native plan each. The run used `local[8]`, a release build, and macOS on an Apple M3 Max with one local directory. The directory syscalls were 4.7% of task and Tokio thread samples: `mkdir` 2.1%, the `unlinkat` and `openat` calls of `remove_dir_all` 2.4%, and `stat` 0.2%. These syscalls are cheaper on Linux than on macOS. But their number grows with the number of local disks, and YARN and Kubernetes nodes often configure several. ### Describe the potential solution Create the directories on first use, the way `OsTmpDirectory` already does. That is a small change to DataFusion's `DiskManager`, which Comet would pick up on upgrade. Until then, Comet could share one `DiskManager` per executor for each set of local directories. That would turn `spark.comet.maxTempDirectorySize` from a per-plan limit into a per-executor one, which may be acceptable or may need its own accounting. ### Additional context Found while profiling small-task workloads for per-call costs like the one #6805 removed. #4994 covers the other per-task setup cost on the same path, the per-task Parquet metadata cache. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
