ipatriciahonorato commented on issue #17596: URL: https://github.com/apache/iceberg/issues/17596#issuecomment-6526679593
Hi, we tested the #17597 draft on a real workload and it helps a lot. On our wide scans, Iceberg went from 2.0-2.7x slower than reading the same Parquet files directly with Spark to 2.5-3x faster. Setup: one day of time-series data (about 1 TB of Parquet, rows with wide structs of arrays), Spark 3.5.8, Java 17, HDFS, default 128 MiB row groups. We built Iceberg main (commit 6938d219) with the #17597 draft, the fix suggested in its review, and parquet-java 1.17.1 with #3748. We also added a switch to turn pruning on and off, and support for IN predicates. Most of our queries filter on a list of ids, so for us IN support would be needed in the first version, not later. The queries read all columns for a set of ids in a 1 minute window. Two runs with pruning off and two with it on: | | pruning off | pruning on | |---|---|---| | Iceberg time / Parquet time (3 wide-scan queries) | 2.0-2.7 | 0.33-0.40 | | Iceberg data read (one query) | 1593 MB | 107 MB | | Iceberg rows read (same query) | 28.5 M | 1.88 M | A query that reads array columns read 44% less data and got 20% faster. Results matched direct Parquet in all our queries, but we only checked our own scenarios. Two things we noticed: - Queries that only read primitive columns go through the vectorized reader and don't change, so vectorized support (#17663) would matter for us too. - In our table with the widest rows, a row group has only about 3,500 rows, and the filter columns have one page per row group, so nothing gets pruned there. A smaller page row limit should help, but we haven't tested it yet. We'd really like to see this merged and are happy to run more tests or help review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
