vinishjail97 commented on PR #935: URL: https://github.com/apache/incubator-xtable/pull/935#issuecomment-5943004484
Tested this on S3 against the TPC-DS 1 TB Iceberg tables, with Spark 3.4.2, Hudi 1.2.1 and `iceberg-spark-runtime` 1.9.2. Each table was copied, indexed with `syncIndex`, and every distinct key was looked up and compared with Iceberg's `_file`, `_pos` and `_partition` in both directions. | Table | Rows | Index column | Distinct keys | Index size | Full lookup | 1000-key lookup | Mismatches | |---|---|---|---|---|---|---|---| | `customer` (unpartitioned) | 12.0M | `c_email_address` | 11.6M | 293 MB | 12.7 s | ~2.2 s | 0 | | `web_returns` (partitioned on `wr_returned_date_sk`) | 68.8M | `wr_order_number` | 41.4M | 1.25 GB | 31.6 s | ~2.7 s | 0 | On both tables, 1000 keys that are not in the table returned nothing, and keys shared by several rows resolved to every row. The partitioned run hit a `ClassCastException` in `IcebergPartitionValueConverter`: `iceberg-spark-runtime` relocates Avro, so its `PartitionData` is not an `org.apache.avro.generic.IndexedRecord`. This is fixed in 76437821. The same cast is also on `main` and `branch-0.4`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
