hbgstc123 commented on PR #9660: URL: https://github.com/apache/paimon/pull/9660#issuecomment-6096028672
> Consider splitting the PR into smaller PRs. Then, for small Parquet files, this should be detrimental. <meta charset="utf-8"><p>Thanks @JingsongLi for raising this. The benchmark confirms that small inputs can make row-group copy slower than rewrite. For the default <code>preserve-page-index=false</code> path, I propose a configurable <strong>8MiB minimum average compressed row-group size</strong>:</p> <pre><code class="language-text">sum(input row-group compressed bytes) / total input row-group count >= 8MiB </code></pre> <p>The size would come directly from footer metadata. This measures the actual compressed size, rather than the configured <code>parquet.block.size</code>.</p> <p>I compared copy against rewrite using narrow numeric and wide string schemas, with zstd, snappy, and no compression. The runs used local storage with either zero delay or an injected 10ms delay per input open/seek, serial footer reads, two warmup rounds, and five to seven measured rounds. Rewrite used a 128MiB output block size.</p> <p>To select a conservative default, I required a median rewrite/copy speedup of at least <strong>1.10×</strong>, with copy faster in every paired measured round. The limiting boundary case was the wide-string schema without compression:</p> Average compressed RG size | Local speedup | Speedup with 10ms injected delay -- | -- | -- ~4.02MiB | 3.712× | 1.096× ~8.00MiB | 3.026× | 1.206× <p>The ~4MiB case already benefits from copy, but falls slightly below the chosen 10% margin. Rounding above that measured boundary to the next power of two gives <strong>8MiB</strong>; the confirmation run near 8MiB passed the criterion.</p> <p>The break-even point also depends on how many row groups share each file’s fixed overhead. A file containing many row groups can benefit at smaller RG sizes than a file containing just one. Therefore, 8MiB is a conservative initial default for the tested conditions, and should remain configurable.</p> <p>We could reuse the same threshold for a cheap early rejection using manifest metadata: for non-empty input files, if the average file size is below 8MiB, the average compressed RG size cannot reach 8MiB either. That avoids reading footers for obviously small batches without introducing a separate file-size option.</p> <p>These measurements use synthetic latency rather than real HDFS/S3, and the proposed value applies specifically to the path that does not preserve page indexes.</p> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
