JeonDaehong commented on code in PR #1440:
URL: https://github.com/apache/paimon-rust/pull/1440#discussion_r4650886449


##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1973,6 +2060,27 @@ fn build_output(
                         picked_rows.len()
                     ]))
                 }
+                _ if picked_rows.iter().any(|row| {
+                    row.values
+                        .iter()
+                        .any(|(col_idx, _)| *col_idx == output_col_idx)
+                }) =>
+                {
+                    let nulls = new_null_array(field.data_type(), 1);
+                    let parts: Vec<&dyn arrow_array::Array> = picked_rows
+                        .iter()
+                        .map(|row| {
+                            row.values
+                                .iter()
+                                .find(|(col_idx, _)| *col_idx == 
output_col_idx)

Review Comment:
   Thanks for running the wide case, that was a real problem. Picked rows now 
keep the aggregated values indexed by output column, so assembly is one lookup 
per column (e1b7c40d, with main merged in).
   
   With your setup (8,192 keys, two overlapping runs, 256 INT `sum` columns, 
batch size 1,024, release build), 7 reads each, two rounds:
   - before the fix: picked 1106–1195 ms, materialized 743–846 ms
   - after: picked 750–830 ms, materialized 821–911 ms
   
   Table reads, `merge_picked` on vs off in one binary, three alternating 
rounds:
   - 256 columns, 100k keys: 9124–9314 vs 9509–9933 ms
   - 32 columns, 500k keys: 4514–4591 vs 4720–4850 ms
   - the narrow case from the description: 1523–1544 vs 1908–1932 ms
   
   I added the wide cases to the description.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to