malinjawi opened a new pull request, #13383: URL: https://github.com/apache/gluten/pull/13383
## What changes are proposed in this pull request? On Spark 3.5, native Delta write has no fallback: every write goes to the Velox writer. For column-mapped tables that writes wrong files. Delta writes Parquet field IDs in both name and id mode, but the Velox writer drops `StructField` metadata on the way to Arrow, so the files carry none. Spark's own reader then rejects id-mode files (`Spark read schema expects field Ids, but Parquet file schema doesn't contain any field Ids`), and it is the reader used whenever a scan falls back or Gluten is off. UniForm tables are affected too, since Iceberg compatibility requires column mapping. Spark 4.x already sends these writes to Delta's writer (`GlutenDeltaNativeWriteGate`, from #13271). This does the same on Spark 3.5: `GlutenOptimisticTransaction.writeFiles` returns to Delta's `writeFiles` when the table uses column mapping. Native write is experimental and off by default, so only tables written with it on are affected. The docs listed these writes as ResultMismatch and Exception on Spark 3.5; they now say Fallback. Reading id-mode tables by field ID (#12241) depends on this, since that read needs the files to carry field IDs. ## How was this patch tested? New tests in `DeltaNativeWriteSuite` (Spark 3.5): a table without column mapping is still written by Velox, and name- and id-mode tables are written by parquet-mr, carry field IDs on every column, and read back correctly through Spark's reader. They check the footers rather than the plan, because the `GlutenDelta` command node shows whichever writer runs. Compiled main and test code with `-Pbackends-velox,spark-3.5,delta,spark-ut`. The suite loads the native library, so it runs in CI's Spark 3.5 Delta job. cc @felipepessoto @zhouyuan ## Was this patch authored or co-authored using generative AI tooling? Generated-by: IBM Bob -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
