andygrove opened a new issue, #7517: URL: https://github.com/apache/datafusion-comet/issues/7517
## Summary Spark 4.3 adds an opt-in Arrow cache format for Dataset caching: `org.apache.spark.sql.execution.columnar.ArrowCachedBatchSerializer`, selected with the static config `spark.sql.cache.serializer` (default stays `DefaultCachedBatchSerializer`). Comet already has an experimental cache of its own (`spark.comet.exec.inMemoryCache.enabled`, see the In-Memory Cache user guide page) that stores cached data as Arrow and scans it with `CometInMemoryTableScan`. This issue is to research how the two relate and whether Comet should interoperate with, defer to, or stay independent of Spark's serializer on 4.3+. This is research only. No code change is proposed yet. ## What Spark 4.3.0 ships Verified against the `v4.3.0` tag (`docs/sql-arrow-cache-format.md`, `ArrowCachedBatch.scala`, `ArrowCachedBatchSerializer.scala`): - `ArrowCachedBatch(numRows, arrowData: Array[Byte], stats: InternalRow)` extends `SimpleMetricsCachedBatch`. `arrowData` is one schema-less encapsulated Arrow RecordBatch message; the schema is rebuilt from the relation's attributes on read. - It accepts both row and columnar input, collects per-column min/max/null-count statistics for batch pruning, and supports `none`, `zstd` and `lz4` Arrow compression (`spark.sql.execution.arrow.compression.codec`). - Reads can produce `ArrowColumnVector`-backed `ColumnarBatch`es, with a background prefetch option (`spark.sql.execution.arrow.cache.prefetch.enabled`). - It rejects any schema containing a type Arrow cannot represent. There is no per-relation fallback to the default serializer, since `spark.sql.cache.serializer` is a single static class for the whole session. ## How this overlaps with Comet's cache Comet's `org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer` (note the same simple name) is installed by `CometDriverPlugin` as `spark.sql.cache.serializer` when the cache is enabled. Its `CometCachedBatch` is also a schema-less encapsulated Arrow IPC RecordBatch message with per-buffer compression and `SimpleMetricsCachedBatch` stats, so the two formats look close. Differences already visible: Comet delegates unsupported schemas per relation to Spark's default format, scans through `CometInMemoryTableScan` into native operators, and projects columns by copying only the selected buffers' byte ranges. ## Questions to answer 1. **Selection conflict.** Both are chosen via the same single static config. What does the plugin do today if a user sets Spark's `ArrowCachedBatchSerializer` explicitly on 4.3? Should Comet keep overriding, respect the user's choice, or refuse? 2. **Can Comet read Spark's format?** If the payloads are layout-compatible (buffer order, compression framing, dictionary and nested-type handling, `stats` layout), `CometInMemoryTableScan` could scan relations cached by Spark's serializer natively, with no need to replace the serializer. What would it take, and is the compatibility stable enough to depend on given Spark calls it an internal format? 3. **Can Comet reuse Spark's serializer for writes?** Would delegating to Spark's serializer on 4.3+ and keeping only the scan side simplify Comet's cache code (`CometCachedBatch`, `CachedBatchIpc`), or lose features we rely on (per-relation fallback, column projection by buffer range, per-batch codec recorded in the message)? 4. **Type coverage.** Compare supported types with Comet's writer (intervals, TIME, nanosecond timestamps, variant, nested types). Spark throws on unsupported types where Comet falls back per relation. 5. **Performance.** Using the existing cache benchmarks, compare write cost, read cost into native operators, and memory footprint for Spark's Arrow cache + a Comet scan against Comet's own format, on filter-heavy, projection-heavy and wide-table workloads. 6. **Operational.** Kryo registration, memory accounting (Spark's serializer holds `Array[Byte]`; Comet uses `ChunkedByteBuffer` to pass the 2 GiB array limit), and what changes in the checks that read `spark.sql.cache.serializer` (`CometCacheBuildRule`, `CometExecRule`). ## Deliverable A short write-up (comment on this issue or a doc under the contributor guide) with a recommendation: keep independent, interoperate on read, or converge on Spark's format on 4.3+, plus follow-up issues for whichever path is chosen. ## Context Found while adding the Spark 4.3 build profile (Stage 1 of the new-version bring-up). Nothing in that PR touches the cache. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
