chenjunjiedada opened a new pull request #2273: URL: https://github.com/apache/iceberg/pull/2273
In our production environment, the Flink iceberg sink could produce a large number of small files, especially when the checkpoint interval is set to a small value, for example, one minute. Or when online business is spare. When rewriting the small files, we found the filter is not enough for collecting small files especially when the table is partitioned by the bucket. This adds a predicate to rewrite action so that we could filter out data files according to data file properties, such as record count, or file size. It collects small files in the specified size range and thus mitigates the overhead of the compaction service. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
