leaves12138 opened a new pull request, #1680: URL: https://github.com/apache/incubator-paimon/pull/1680
The RollingFileWriter judge if need rolling by: file(which is written).getPos + bytes(in memory) * judgedCompressionRate The first part is the real size, but the second part is not accurate. When use zstd compression, the output file size is smaller than target file size, and the output file will be regarded as small files, which cause all the files in compaction all the way. We filter middle size file to avoid this, the file which is larger than targetFileSize * 0.7 will not participate in compaction anymore. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
