sezruby commented on issue #10215: URL: https://github.com/apache/gluten/issues/10215#issuecomment-5120401565
**Follow-up / backlog: Parquet page index (ColumnIndex/OffsetIndex) not written by the Velox backend** Flagging a Velox-writer gap that affects native Delta (and plain Parquet) writes. Velox's Parquet writer wraps Arrow (`velox/dwio/parquet/writer/Writer.cpp` → `getArrowParquetWriterOptions`) and never calls Arrow's `enable_write_page_index()`; `ParquetWriterOptions` exposes no option for it. Arrow defaults page-index writing to off, so files written through the Velox backend carry no page index — unlike vanilla Spark/parquet-mr, which writes it by default (SPARK-26345). **Impact:** Delta/Parquet files produced by native write can't be page-pruned by readers relying on the page index — including Velox's own reader-side page-level pruning effort (facebookincubator/velox#14195, facebookincubator/velox#18079). **Proposed split:** - *Velox:* add a page-index write option to `ParquetWriterOptions` and call `enable_write_page_index()` on the Arrow builder (self-contained; no Arrow change needed). Filed upstream: facebookincubator/velox#18323. - *Gluten:* plumb the Spark config → Velox writer options, default-on to match vanilla Spark. Tracking as a backlog item (Gluten side blocked on the Velox change). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
