kinolaev opened a new pull request, #16956: URL: https://github.com/apache/iceberg/pull/16956
Equality delete files can be referenced by many data files. This PR allows to accumulate all equality deletes of a table snapshot in the structure `spec id -> partition -> equality field ids -> equality key -> max data sequence number` that can be reused for many data files. If passed to `DeleteFilter`, it fills the structure lazily. I added shared equality deletes to Spark's `RowDataReader` and `BatchDataReader` by default, because the size of the shared structure is bounded by split size (multiplied by compression rate). To `GenericReader` I added a boolean parameter `shareEqDeletes` which is false by default, because the size is unbounded, and switching the parameter to true might cause OOMs. And I don't use Flink, so I hesitated to propose changing its behavior. The non-shared `DeleteFilter` path is unchanged, so Flink and other engines are byte-for-byte identical. The shared path produces identical results (equivalence covered by tests). Additionally, this PR changes task weight calculation algorithm - each delete file is accounted only once per bin even if it is referenced by several data files in that bin. This change can be moved into a separate PR if preferred. In our case these changes reduced Spark memory consumption by 30-50% and, more important, prevented Spark from hanging for hours on building and evicting huge equality delete sets for each data file. For frequently updated tables (several thousands of equality delete files per day) ingested with [Debezium Iceberg Sink](https://github.com/memiiso/debezium-server-iceberg) `rewrite_data_files` execution time is reduced from several hours to several minutes. Btw, I found that Trino uses a similar technique: [EqualityDeleteFilter](https://github.com/trinodb/trino/blob/481/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/delete/EqualityDeleteFilter.java), [DeleteManager](https://github.com/trinodb/trino/blob/481/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/delete/DeleteManager.java). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
