rishi-rana commented on issue #18131: URL: https://github.com/apache/iceberg/issues/18131#issuecomment-5722478173
I'd like to work on this, unless someone is already on it. I traced this against `main` (code reading, not yet a local Spark run). Spark 3.5 is the only version that overrides `readSchema()` in `SparkCopyOnWriteScan` to strip `__metadata_col` from `_row_id` / `_last_updated_sequence_number` (`rowLineageAsDataCols`, added in #12736), because 3.5's row-level rules would otherwise drop the lineage columns from the output. Spark 4.x has no such override and relies on the native metadata-column semantics from SPARK-50820, which matches the version split reported here. The invalid plan is intermediate rather than final: once those fields lose the marker, `ReplaceData.outputResolved` counts them as data columns, so the plan produced by `GroupBasedRowLevelOperationScanPlanning` is briefly unresolved. Normal runs never notice because per-rule plan validation only happens when the `spark.testing` system property is present. Before writing a patch I'd like to check the intended direction, since there seem to be two options: 1. Keep the scan schema as it is and make the lineage columns acceptable to `ReplaceData` resolution, e.g. by handling them in Iceberg's own `RemoveRowLineageOutputFromOriginalTable` rule so the plan stays resolved through the batch. 2. Stop stripping `__metadata_col` in `SparkCopyOnWriteScan.readSchema()` and preserve the lineage columns some other way in the 3.5 rewrite path. @amogh-jahagirdar, as the author of the 3.5 row-lineage path, do you have a preference? Happy to go either way, and I'll include a regression test that runs a v3 copy-on-write `DELETE` with `spark.testing` enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
