vinishjail97 opened a new pull request, #957: URL: https://github.com/apache/incubator-xtable/pull/957
## What is the purpose of the pull request Both Delta targets (`DeltaDataFileUpdatesExtractor` and `DeltaKernelDataFileUpdatesExtractor`) pass every data file to `PathUtils.getRelativePath`, which assumes the file is under the table base path. For a file outside the base path it removes characters by length, so the Delta log gets a broken path such as `nch.db/spans_c3_glue_144053/data/<file>.parquet`, and the sync still reports success. This happens when the target base path differs from the location of the source data files, for example: - an Iceberg table with `write.data.path` set to another prefix, - an Iceberg table with files added by `add_files` or `snapshot`, which still point to their original location, - an Iceberg table loaded from a catalog with the target base path set to another prefix. ## Brief change log - Add `DeltaPathUtils.getPathForDeltaLog`, the write-side counterpart of `getFullPathToFile` from #932. A file under the base path is recorded relative to it, comparing the paths without the scheme to handle `s3` against `s3a`. Any other file is recorded with its absolute path, which the Delta protocol allows and which `getFullPathToFile` already reads. - Call it from both Delta targets in place of `PathUtils.getRelativePath`. `PathUtils` is unchanged. - A sibling directory that shares the base path as a prefix (`.../tab2` next to `.../tab`) is no longer treated as part of the table. ## Verify this pull request - New cases in `TestDeltaPathUtils`. `./mvnw -pl xtable-api,xtable-core install -DskipITs` passes. - Synced a Glue Iceberg table (66 files, 179,200 rows) to a Delta target at a separate S3 prefix with the bundled jar. Before the change, all 66 `add` paths were truncated. After it, all 66 are absolute, and Spark Delta reads 179,200 rows with the same distinct key count and column sum as the source Iceberg table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
