vinishjail97 commented on PR #935: URL: https://github.com/apache/incubator-xtable/pull/935#issuecomment-6426551758
Validated this end to end on S3 with an AWS Glue catalog (Iceberg source with `S3FileIO`), using the PR head merged with current main. TPC-DS 1TB `customer` (12M rows) was loaded into two Iceberg tables in Glue, one unpartitioned and one partitioned by `c_birth_month`. Each table was synced by `ConversionController` to a Hudi target at the data path with `xtable.hudi.target.secondary.index.column=c_email_address` (a non-unique key), and read back with `HudiBackedSecondaryIndexReader` loading the table through the Glue catalog config. After the initial sync, a no-op sync, an append, an update and a delete (copy-on-write), the check was a multiset equality between: - the reader's lookup of every key: `(c_email_address, _file, _pos)` - every row of the snapshot's data files read with Iceberg's Parquet reader (`_pos` from `MetadataColumns.ROW_POSITION`), for rows whose key is not null Both tables passed in every phase. Both differences had 0 rows, the last synced source identifier matched the current snapshot, and the keys removed by the update and the delete (115,727 each) returned no rows. The full build with unit and integration tests also passes on the merged branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
