zhangyue19921010 opened a new issue, #11212: URL: https://github.com/apache/paimon/issues/11212
### Search before asking - [x] I searched in the issues and found nothing similar. ### Motivation Ordinary append clustering copies each generated order-key row before serializing it into the sort buffer. It also allocates an output projection view for every sorted row. Both views are consumed synchronously before processing the next row. On `2.2-SNAPSHOT`, base `00b4d05a6cc118091a38eeca23549519cfb775f4`, the following real clustering rewrite shows avoidable temporary allocation: | Metric (median) | Base | Patch | | --- | ---: | ---: | | Calling-thread allocation (MiB) | 113.10 | 85.64 | | Rewrite wall time (ms) | 176.09 | 166.84 | | Calling-thread CPU (ms) | 174.51 | 165.05 | | Output bytes | 1,040,848 | 1,040,848 | Local Java `clusterRewrite` measurements on JDK 11.0.18 with G1, `-Xms512m -Xmx2g`: 200,000 rows `(a INT, b INT, c STRING)`, ORDER clustering, Parquet, a 64 MiB buffer and synchronous file output. Each version ran in two fresh JVMs, with three warmups and seven measured rewrites per JVM (14 samples each), in base/patch/patch/base order. Input files were produced by real table writes and committed once; every output row and its order were checked after each rewrite. Fixture writes and validation reads are outside the measured region. Allocation standard deviations were 0.042/0.042 MiB (base/patch); wall-time standard deviations were 17.50/16.28 ms. Allocation fell 24.3%. Timing and whole-JVM RSS samples overlap. ### Solution Use the generated order-key projection until the sort buffer has serialized it, and reuse one output projection view per sorter. Document the view lifetime and clear its backing row at close. ### Anything else? Eight Java table controls cover two clustering keys, NULLs and long strings with a 1 MiB buffer, all three clustering curves, and synchronous/asynchronous output. Two cases use nullable long strings in non-prefix ORDER keys `c,a` and check the clustering-key order. The controls verify the real write, compaction, commit and read path and pass on the unchanged base. AI-generated with TRAE CLI (GPT-6.1-Sol). ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
