zhangyue19921010 opened a new issue, #11212:
URL: https://github.com/apache/paimon/issues/11212

   ### Search before asking
   
   - [x] I searched in the issues and found nothing similar.
   
   ### Motivation
   
   Ordinary append clustering copies each generated order-key row before 
serializing it into the sort buffer. It also allocates an output projection 
view for every sorted row. Both views are consumed synchronously before 
processing the next row.
   
   On `2.2-SNAPSHOT`, base `00b4d05a6cc118091a38eeca23549519cfb775f4`, the 
following real clustering rewrite shows avoidable temporary allocation:
   
   | Metric (median) | Base | Patch |
   | --- | ---: | ---: |
   | Calling-thread allocation (MiB) | 113.10 | 85.64 |
   | Rewrite wall time (ms) | 176.09 | 166.84 |
   | Calling-thread CPU (ms) | 174.51 | 165.05 |
   | Output bytes | 1,040,848 | 1,040,848 |
   
   Local Java `clusterRewrite` measurements on JDK 11.0.18 with G1, `-Xms512m 
-Xmx2g`: 200,000 rows `(a INT, b INT, c STRING)`, ORDER clustering, Parquet, a 
64 MiB buffer and synchronous file output. Each version ran in two fresh JVMs, 
with three warmups and seven measured rewrites per JVM (14 samples each), in 
base/patch/patch/base order. Input files were produced by real table writes and 
committed once; every output row and its order were checked after each rewrite. 
Fixture writes and validation reads are outside the measured region.
   
   Allocation standard deviations were 0.042/0.042 MiB (base/patch); wall-time 
standard deviations were 17.50/16.28 ms. Allocation fell 24.3%. Timing and 
whole-JVM RSS samples overlap.
   
   ### Solution
   
   Use the generated order-key projection until the sort buffer has serialized 
it, and reuse one output projection view per sorter. Document the view lifetime 
and clear its backing row at close.
   
   ### Anything else?
   
   Eight Java table controls cover two clustering keys, NULLs and long strings 
with a 1 MiB buffer, all three clustering curves, and synchronous/asynchronous 
output. Two cases use nullable long strings in non-prefix ORDER keys `c,a` and 
check the clustering-key order. The controls verify the real write, compaction, 
commit and read path and pass on the unchanged base.
   
   AI-generated with TRAE CLI (GPT-6.1-Sol).
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to