Mola-maker opened a new pull request, #973: URL: https://github.com/apache/incubator-xtable/pull/973
Fixes #972 ## What is the purpose of the pull request With `delta.columnMapping.mode=name`, Delta writes the `partitionValues` of add/remove actions keyed by each column's physical name (`col-<uuid>`), while XTable's internal partition fields carry the logical name. `DeltaKernelPartitionExtractor` and `DeltaPartitionExtractor` looked values up by the logical name only, so every identity partition value came out `null` — the sync reported SUCCESS while the converted Iceberg table had NULL partition columns (e.g. `WHERE country='USA'` matched 0 rows instead of 2). Two gaps combined into the bug on the Kernel path: `DeltaKernelSchemaExtractor` computed each field's physical (storage) name from `delta.columnMapping.physicalName` but never stored it on the built `InternalField` (the Spark-based `DeltaSchemaExtractor` does set it), and neither partition extractor consulted `InternalField.getStorageName()` when resolving partition values. ## Brief change log - `DeltaKernelSchemaExtractor`: set the already-computed storage name on the built `InternalField`, mirroring `DeltaSchemaExtractor`. - `DeltaKernelPartitionExtractor` / `DeltaPartitionExtractor`: in `getSerializedPartitionValue`, resolve identity partition values by the field's storage name first when one is present, falling back to the logical name. - Tests: added `testPartitionValueExtractionWithColumnMappingPhysicalName` to `TestDeltaKernelPartitionExtractor` and `TestDeltaPartitionExtractor`, feeding a partition-values map keyed by the physical name. ## Verify this pull request This change added tests and can be verified as follows: - Before the fix, the new extraction test fails: expected partition range `USA`, actual `null`. Verified in CI (fork run of this branch without the fix): xtable-core ran 550 tests with this single failure. - After the fix, the same CI build (`Maven CI Build` on the fork PR) passes: xtable-core `Tests run: 550, Failures: 0, Errors: 0, Skipped: 3`, BUILD SUCCESS. Red/green runs: https://github.com/Mola-maker/incubator-xtable/pull/1 - `./mvnw -pl xtable-core -Dtest=TestDeltaKernelPartitionExtractor,TestDeltaPartitionExtractor test` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
