mitchsw opened a new issue, #23644: URL: https://github.com/apache/datafusion/issues/23644
### Is your feature request related to a problem or challenge? I am benchmarking a query plan with many partitions and a scan over >12,000 Parquet files. Each file opened by the default `ParquetSource` registers 45 metrics into the one `ExecutionPlanMetricsSet` shared by the whole `DataSourceExec`. This results in half a million synchronous mutex acquisitions and metric registrations. Dial9 schedule/off-CPU profiling of high-file-count scans consistently identified this mutex as the dominant application-level blocking source. The captured stacks ran from Parquet file opening through `MetricBuilder::build` and `ExecutionPlanMetricsSet::register`, where workers entered `futex_wait` while acquiring the shared `parking_lot::Mutex`. Because registration uses a synchronous mutex on Tokio runtime threads, concurrent file opens create a lock convoy: worker threads park, runnable tasks accumulate in the Tokio queue, and CPU remains underutilized. This per-file detail is not used by normal queries or `EXPLAIN ANALYZE`, which aggregates metrics by name and discards filename labels. The filename-labelled metrics are only exposed through `EXPLAIN ANALYZE VERBOSE` or direct inspection of unaggregated metrics. ### Describe the solution you'd like Add a metrics cardinality `ConfigOption`: ```text datafusion.execution.metrics_cardinality = compact | verbose ``` Default to `compact`. With `compact` cardinality, the `ParquetSource` would: - Register one metric set per execution partition - Omit filename labels - Share the handles across files and replacement readers With `verbose` cardinality, the `ParquetSource` would preserve per-file filename-labelled metrics. This changes metric registration growth from `O(files)` to `O(partitions)`. To maintain existing behavior, `AnalyzeExec` will handle `EXPLAIN ANALYZE VERBOSE` by mutating the child `TaskContext` enabling verbose cardinality (no other `ExecutionPlan` edits its `ConfigOptions`, but it felt appropriate here). Other use cases that directly consume per-file metrics could also explicitly configure it. ### Describe alternatives you've considered A larger metrics-system redesign could introduce lock-free or sharded registration, typed metric scopes, cardinality limits, or execution-local registries. That may be worthwhile, but it is substantially broader than the Parquet contention problem. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
