felipepessoto opened a new pull request, #13042: URL: https://github.com/apache/gluten/pull/13042
## What changes are proposed in this pull request? Fixes #12743. Fall back `FileSourceScanExec` to Delta's JVM reader when `spark.databricks.delta.deletionVectors.useMetadataRowIndex=false` and the scan requests JVM-generated Delta deletion-vector metadata: - `__delta_internal_row_index` - `__delta_internal_is_row_deleted` - Spark's temporary top-level row-index field The fallback checks both scan output and required schema. It no longer depends on the table containing a deletion vector, so DV-free scans requesting these synthetic fields receive the JVM-generated values instead of native NULL/incorrect values. Ordinary Delta scans remain eligible for native offload, and the existing metadata-row-index and DV/DML behavior remains scoped as before. Adds focused Delta 3.3 and Delta 4.0 coverage for non-null generated row indexes, zero deleted-row flags on DV-free files, and continued native offload for ordinary scans. This PR is independent of #13036 and does not change the scheduled known-failures baseline or row-group fixtures. ## How was this patch tested? - `./build/mvn -pl backends-velox -am test-compile -Pdelta,backends-velox,spark-3.5,scala-2.12,fast-build -DskipTests` (passed; compiles the Delta 3.3 focused tests) - `./build/mvn -pl gluten-delta -am test-compile -Pdelta,backends-velox,spark-3.5,scala-2.12,fast-build -DskipTests` (passed) - `./build/mvn -pl gluten-delta,backends-velox -am test-compile -Pdelta,backends-velox,spark-4.0,scala-2.13,java-17,fast-build -DskipTests` (the `gluten-delta` production module passed; `backends-velox` is blocked on Windows because repository symlink placeholders under `src-delta40` are parsed as Scala source) - Scoped Spotless apply passed for all three touched Scala files. - `git diff --check` passed. - Focused ScalaTest execution was attempted, but suite startup is blocked on Windows by the unavailable native artifact `linux/amd64/gluten.dll`. - All touched files retain the ASF license header. The Python header checker could not run because Python is not installed in this environment. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot GPT-5.6 Sol -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
