bryanck commented on issue #200: Add SnapshotProducer that can rewrite manifests URL: https://github.com/apache/incubator-iceberg/pull/200#issuecomment-498414249 This will create entirely new manifests (with a new snapshot ID and with new contents) that replace existing manifests, so in essence, yes, the manifests and contents will be reordered. We were planning to apply this API on a regular basis on new manifests (e.g. hourly or daily). Also, `commit.manifest.target-size-bytes` will not be respected thus this could generate large manifest files. I first thought of adding this functionality to `MergingSnapshotProducer` but this would either generate far too many manifests (trading long file scans for long manifest scans) or generate manifests with wide partition ranges as it does now. Respecting `commit.manifest.target-size-bytes` should be straightforward if that is desired. I'm not sure how respecting manifest order is possible though.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
