andygrove commented on PR #7082: URL: https://github.com/apache/datafusion-comet/pull/7082#issuecomment-6334870154
Thanks @comphead, @viirya and @sunchao. I merged main, which fixes the conflicts in `jni_api.rs` and `CometExecSuite.scala`. As viirya noted, the prefetch from #7094 loads through the reader factory's cache, which is the store-scoped view, so the two share entries and keep the store scope. Each inline thread has a reply. The main changes: - A stage's cache now holds the 50 MiB per-task limit for each of its running tasks, counted through a per-task handle, and shrinks to 50 MiB when none is running. The handle ignores the limit a task's `CacheManager` sets, so a new task can't reset it. sunchao showed that sizing it by executor cores overshoots on standalone executors, so it counts tasks instead. - Sharing stays on by default. The numbers are in the `CometConf` thread: at the default split size, footer reads drop by 40 to 68% when files have many splits, and the time saved shows on object stores rather than on local disk. - The config doc and `memory_management.md` describe what the cache holds and that the retained caches outlive their tasks. @comphead, I added `run-delta-tests` so the Delta contrib jobs give a verdict on the `delta_dv.rs` and `delta_spark_scan.rs` changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
