JingsongLi commented on code in PR #1440:
URL: https://github.com/apache/paimon-rust/pull/1440#discussion_r4643364164


##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1973,6 +2060,27 @@ fn build_output(
                         picked_rows.len()
                     ]))
                 }
+                _ if picked_rows.iter().any(|row| {
+                    row.values
+                        .iter()
+                        .any(|(col_idx, _)| *col_idx == output_col_idx)
+                }) =>
+                {
+                    let nulls = new_null_array(field.data_type(), 1);
+                    let parts: Vec<&dyn arrow_array::Array> = picked_rows
+                        .iter()
+                        .map(|row| {
+                            row.values
+                                .iter()
+                                .find(|(col_idx, _)| *col_idx == 
output_col_idx)

Review Comment:
   **[P2] Look up aggregate results directly by output column**
   
   The per-column `.find()` traverses each picked row's aggregate results. For 
a projection with N aggregate columns this makes assembly perform roughly N²/2 
column comparisons per merged row, in addition to concatenation and interleave. 
I exercised the actual sort-merge reader with 8,192 keys, two overlapping runs 
and 256 INT `sum` columns at batch size 1,024. Using the PR's `Materializing` 
wrapper as the same-binary baseline, four alternating runs measured a 597 ms 
median with materialized rows and 729 ms with picked rows on the repeat run, 
about 22% slower. Paimon was compiled with test-profile `opt-level=2`. Both 
paths returned the same row count and aggregate check value.
   
   Index the aggregate result by output column, or collect it in column-major 
buffers, so output assembly scales linearly with projection width. Please keep 
a wide case in the performance comparison.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to