malinjawi opened a new pull request, #13387: URL: https://github.com/apache/gluten/pull/13387
## What changes are proposed in this pull request? Follow-up to #12218. Gluten kept a change data feed read on Spark whenever its version range touched a file with a deletion vector: `DeltaCDFScanRule` declined to expand the relation, and `DeltaScanTransformer` failed validation for CDF scans of such files. With deletion vectors on, DELETE writes no `_change_data` files, so every CDF read over a DELETE took that path. On Spark 3.5+ (Delta 3.3+), Delta's CDF file indexes put each file's deletion vector and row index filter type into its `PartitionedFile` metadata, and the native Delta scan already applies both filter types: `IF_CONTAINED` drops the marked rows, `IF_NOT_CONTAINED` keeps only them, which is how Delta picks the rows a deletion vector newly deleted or re-added. So this PR: - expands CDF ranges with deletion vectors on Spark 3.5+; - removes the CDF guard in `DeltaScanTransformer`. It couldn't fire on Spark 3.4 anyway, since `OffloadDeltaScan` keeps every 3.4 deletion vector scan on Spark; - updates the change data feed rows in the docs. Spark 3.4 (Delta 2.4) is unchanged. Delta 2.4 hands deletion vectors to its reader through a broadcast map rather than file metadata, so those ranges still fall back. One rare case differs from Delta: a CDF stream that started before deletion vectors were enabled on the table. Delta's planner skips the deletion vector there and reports every row of the file as deleted; Gluten returns only the rows the deletion vector marks. ## How was this patch tested? New and updated tests in `DeltaSuite` (through `VeloxDeltaSuite`, Spark 3.4 to 4.2), each compared with vanilla Spark: - `change data feed read with deletion vectors` now checks the read is native on 3.5+ and still falls back on 3.4. - `change data feed read over deletion vector changes` (3.5+) walks one file through a DELETE, a second DELETE on the same file (the add-side difference is empty and must return nothing), a RESTORE, and a metadata-only DELETE. It checks every Delta file scan is a `DeltaScanTransformer`, and repeats with three-row batches and with `useMetadataRowIndex=false`. - `streaming change data feed read with deletion vectors` (3.5+) checks the micro-batch scans are native. The Delta UT gate also runs Delta's own CDF suites with deletion vectors; no baseline change is expected. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: IBM Bob -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
