kbendick commented on issue #2632: URL: https://github.com/apache/iceberg/issues/2632#issuecomment-853489322
If you have partitioning by 8 buckets, then every snapshot will produce 8 data files (at most... assuming there's data for each output partition / bucket). So if you took a snapshot, the snapshot would commit the data. And so therefore, your files will be less than 512 mb if the job needs to commit and you don't have 512 * 8 mb (as Flink jobs always commit data on snapshot iirc for correctness guarantees). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
