JingsongLi commented on issue #1383: URL: https://github.com/apache/iceberg/issues/1383#issuecomment-706998170
Hi @stevenzwu , about `TableScan.appendsBetween`, we can limit the snapshot number of scan, even scan only one at a time. Because `TableScan.appendsBetween` seems to be just a combination of single incremental snapshots, we can handle only one snapshot at a time. - For FLIP-27 Source, it is easy to do, because tasks came to ask for splits, the only problem is that within the coordinator, the coordinator can completely control how the splits are generated. - For the old API, the downstream reading operator should has a max split queue size to back pressure the Enumerator. And do above thing. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
