sunchao commented on code in PR #7082:
URL: https://github.com/apache/datafusion-comet/pull/7082#discussion_r4345260430
##########
native/core/src/execution/jni_api.rs:
##########
@@ -1009,6 +1020,13 @@ fn prepare_datafusion_session_context(
.with_object_store_registry(Arc::new(CometObjectStoreRegistry::default()));
rt_config = rt_config.with_memory_pool(memory_pool);
+ // Let the tasks of one stage on this executor share the Parquet footer
and page index cache
+ // instead of each fetching and parsing the footers of the files it reads.
+ let task_slots = spark_config.get_usize(SPARK_EXECUTOR_CORES, 1) /
task_cpus.max(1);
Review Comment:
[P2] Use actual executor slots when sizing the shared cache. On a standalone
host reporting 12 processors, with `spark.cores.max=2`, `spark.task.cpus=1`,
and no explicit `spark.executor.cores`, Spark can allocate a two-core executor
while `numDriverOrExecutorCores` returns 12. The new use of that estimate here
gives the stage a 600 MiB cache instead of the intended 100 MiB. A bounded
reproduction retained 147,254,940 bytes after task handles were dropped, versus
98,169,960 bytes with the correct two-slot limit. This introduces excess native
retention outside Spark's accounting, and two such caches can remain idle.
Please pass authoritative executor capacity for cache sizing, or use a
conservative bound when it is unknown.
Evidence: Spark 4.1.3 source: `StandaloneSchedulerBackend` passes allocated
cores through `--cores`, `Master.scheduleExecutorsOnWorkers` caps allocation by
`app.coresLeft`, and `CoarseGrainedExecutorBackend` copies cores into
`spark.executor.cores` only for non-default resource profiles.
`SparkEnv.createExecutorEnv` receives the allocated count separately. Comet's
existing helper at `CometExecIterator.scala:1016` returns `availableProcessors`
for standalone when that config is absent. Disposable test
`review_executor_core_estimate_cache_retention` loaded 12 local Parquet files,
each with 128 Int64 columns and 128 two-row row groups, through the current
scoped cache. After dropping task handles, results were `slots=2
limit=104857600 idle_entries=8 idle_metadata_bytes=98169960` and `slots=12
limit=629145600 idle_entries=12 idle_metadata_bytes=147254940`. The probe
passed with `cargo test -p datafusion-comet --lib --no-default-features
--features delta review_executor_core_estimate_c
ache_retention -- --nocapture`. Probe source and output are retained at
`/tmp/comet7082-d142-core-estimate-probe.rs` and
`/tmp/comet7082-d142-core-estimate.log`. No full standalone application was run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
