rdblue commented on issue #1438: URL: https://github.com/apache/iceberg/issues/1438#issuecomment-693653398
While we are considering the state problem, why not go ahead and update the sink to store all the `DataFile` objects in a `ManifestFile` and use @stevenzwu's code to serialize the manifests? Then we have more confidence because we've been running this in prod for a long time now. That avoids breaking state later to move to storing manifest files, fixes the current problem, and should make storing checkpoint state a bit faster because it would be smaller. What do you think? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
