bryanck opened a new pull request #200: Add SnapshotProducer that can rewrite manifests URL: https://github.com/apache/incubator-iceberg/pull/200 This PR adds a SnapshotProducer implementation that allows for rewriting the manifests for a table. This can be used to reorganize the manifests for more optimal query planning. We have a situation where we are frequently appending to a table, and each append covers a wide range of partitions. The resulting lower/upper bound range for the partitions covered by the append manifests is thus very broad. As the manifests accumulate, query planning became slow as Iceberg is not able to effectively prune partitions at the manifest level. We plan to use this new API to rewrite the manifests and organize them by partition so Iceberg will be able to more efficiently prune partitions during query planning.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
