gengliangwang opened a new pull request, #59164: URL: https://github.com/apache/spark/pull/59164
### What changes were proposed in this pull request? Prototype an opt-in compact Spark UI store for applications with many retained tasks, jobs and SQL executions. Enable it with `spark.ui.store.compact.enabled=true`; the default remains `false`. - Add an encoded in-memory KVStore with primitive numeric keys, parent membership, deferred decoding and bounded selection for small pages. - Pack completed tasks into stage-attempt column blocks with exact bit-packed numbers and string dictionaries. Keep error/accumulator details separate and preserve running-task updates, retries and iterator snapshots. - Separate job/SQL list summaries from cold details, load job details after pagination, and stream task-search matches while retaining only the requested page. - Use exact completion bitmaps, fold completed SQL sums, pack other SQL metrics losslessly, release completed task-ID mappings, deduplicate AQE metric metadata, and aggregate one metric at a time. - Support live UI, memory-only History Server replay and HybridStore. Transfer hybrid records in batches of 1,024; fix bulk index counts and repeated-key handling in the disk stores, which the handoff regression exposed. Persistent SQL schemas remain unchanged. Design, limitations and reproducible benchmarks: [docs/ui-store-prototype.md](https://github.com/gengliangwang/spark/blob/codex/compact-ui-store-prototype/docs/ui-store-prototype.md). ### Why are the changes needed? Retained UI history keeps large JVM object graphs and metric arrays. List requests can also decode or format records outside the requested page. This prototype tests how much memory can be recovered while preserving retention counts and exact values. Local synthetic benchmarks on OpenJDK 17.0.15 / Xeon 6975P-C produced these full-graph layout estimates: | State | Baseline | Compact | Reduction | | --- | ---: | ---: | ---: | | 100,000 completed tasks and 1,000 jobs, above empty store | 46.66 MiB | 11.77 MiB | 74.8% | | Completed stage: 100,000 tasks, 30 SQL metrics | 25.94 MiB | 1.57 MiB | 93.9% | | 1,000 SQL executions and graphs, 128 plan lines each | 8.18 MiB | 1.01 MiB | 87.7% | These use a new internal, non-sampling SizeEstimator traversal to count shared blocks once. They are object-layout estimates, not measured post-GC retained heap or RSS. The workloads favor compression: sparse successful tasks, repeated strings, small SQL metric values and repeated plan text. This is a draft for evaluating the design. There are material CPU tradeoffs: for a 10,000-task stage, deep paging changed from 1.13 to 4.90 ms and uncached exact quantiles from 33.46 to 290.18 ms. Listing 1,000 SQL summaries changed from 0.020 to 2.167 ms. Task/job population changed from 207 to 246 ms; SQL metric updates from 82 to 120 ms. The first task page improved from 1.12 to 0.52 ms. Full timing results are in the design document. There is no hard total-memory budget. Large/unbounded views still sort O(N) compact references, full REST responses materialize their requested data, and exact/custom metric aggregation uses temporary arrays. Many unfinished sum contributions can exceed dense-array memory. Representative event-log replays, retained/peak heap, GC, listener delay and concurrent UI latency remain required before considering default enablement. ### Does this PR introduce _any_ user-facing change? Yes: adds the experimental `spark.ui.store.compact.enabled` configuration, disabled by default. A configured disk UI store keeps precedence. Retention settings, REST schemas and metric results are preserved. Common bitmap, pagination and disk batch correctness changes also apply with the option disabled. ### How was this patch tested? Added round-trip/index parity, randomized primitive-map, iterator lifecycle, concurrent snapshot, exact quantile/search, summary-only pagination, SQL metric/retention, queued-write shutdown and bounded disk-handoff regressions. The following focused suites passed locally (commands group suites that were run across several invocations): ```sh build/sbt 'kvstore/testOnly *CompactInMemoryStoreSuite *CompactInMemoryIteratorSuite *CompactRecordMapSuite *RocksDBSuite *LevelDBSuite' build/sbt 'core/testOnly *CompactTaskDataCodecSuite *CompactJobDataCodecSuite *CompactLongArraySuite *AppStatusStoreSuite *AppStatusListenerWithCompactStoreSuite *AppStatusListenerWithInMemoryStoreSuite *LiveEntitySuite *JobDataSourceSuite *CompactRocksDBHybridStoreSuite *RocksDBHybridStoreSuite *SizeEstimatorSuite' build/sbt 'sql/testOnly *CompactSQLExecutionDataSuite *SQLAppStatusListenerWithCompactStoreSuite *SQLAppStatusListenerWithInMemoryStoreSuite' ``` Core and SQL test compilation, KVStore main/test Checkstyle, core/SQL main/test Scalastyle, and `git diff --check` passed. Both local benchmarks completed: ```sh build/sbt 'core/Test/runMain org.apache.spark.status.CompactUIStoreBenchmark 100000 10000 1000' build/sbt 'sql/Test/runMain org.apache.spark.sql.execution.ui.CompactSQLStatusBenchmark 100000 30 1000 128' ``` The full Spark test suite and GitHub Actions benchmark workflow have not been run. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
