linhao1990 opened a new pull request, #5533: URL: https://github.com/apache/iceberg/pull/5533
We want to consume an iceberg table as a stream but it throws `UnsupportedOperationException` if table contains overwrite snapshots. I find issue #315 has already commented this problem: > I think this needs an `else` case that validates we aren't skipping over `overwrite` commits. I think it is okay to skip `rewrite` and `delete`, but `overwrite` adds data so I'm not sure about skipping it. I could be convinced to add a config option to do this later, but it seems dangerous to me to skip it automatically. We can always allow more later, but it's hard to disallow something that always worked. > > _Originally posted by @rdblue in https://github.com/apache/iceberg/pull/315#discussion_r373632112_ So I implement this pr according to the comment. I add a new table property `read.incremental-pull-ignore-overwrite` which default to `false` to keep the same behavior, it can ignore overwrite snapshot when it set to `true`. We have verified this in our production environment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
