jayzhan211 commented on code in PR #12059:
URL: https://github.com/apache/arrow-rs/pull/12059#discussion_r4655766126


##########
arrow-select/src/interleave.rs:
##########
@@ -312,6 +342,78 @@ fn interleave_dictionaries<K: ArrowDictionaryKeyType>(
     Ok(Arc::new(array))
 }
 
+/// Returns the values array shared by every referenced, non-null dictionary 
key.
+///
+/// Inputs with no selected rows or only selected null keys are ignored 
because their
+/// values cannot affect the output. Returns `None` if two referenced sources 
use
+/// different values arrays. If every selected key is null, returns the values 
array
+/// for the first selected input.
+fn referenced_shared_dictionary_values<'a, K: ArrowDictionaryKeyType>(

Review Comment:
   I agree with @alamb: measured, this scan makes `interleave` up to 55% slower 
when a distinct mapping is referenced, and none of the PR's benchmarks reach 
it. For example, slices A and B sharing one values `Arc` plus a distinct C used 
only in the last 8 of 2,048 rows go from 2.01 µs to 3.12 µs; the scan only wins 
for an unused distinct input with ~600 values (3.16 → 2.20 µs). Could you drop 
it and keep only the all-shared check? That keeps the two-slice win (2.04 → 
1.08 µs) and the merge-path gain.
   
   ```diff
            let first_values = dictionaries[0].values();
   -        let shared_values = if dictionaries
   +        if dictionaries
                .iter()
                .skip(1)
                .all(|dictionary| Arc::ptr_eq(first_values, 
dictionary.values()))
            {
   -            Some(first_values)
   -        } else if dictionaries.len() > 2 {
   -            // For larger interleaves, distinct mappings may belong only to
   -            // unused or all-null inputs and can be ignored.
   -            referenced_shared_dictionary_values(&dictionaries, indices)
   -        } else {
   -            None
   -        };
   -        if let Some(shared_values) = shared_values {
   -            return interleave_shared_dictionary(&dictionaries, indices, 
shared_values);
   +            return interleave_shared_dictionary(&dictionaries, indices, 
first_values);
            }
   ```
   
   and delete `referenced_shared_dictionary_values` and 
`test_interleave_dictionary_reuses_referenced_shared_values`, which only tests 
it. With that applied, the crate's tests and clippy pass.
   
   <details>
   <summary>Benchmark: µs per call, mean of two interleaved runs (base 4c03e90, 
PR 931cbf8, PR without the scan); 2,048 output rows, 10 values per dictionary 
unless noted</summary>
   
   | Inputs | base | PR | PR without scan |
   |---|---|---|---|
   | A, B slices of one dictionary | 2.04 | 1.09 | 1.08 |
   | A, B shared + distinct C used in last 8 rows | 2.01 | 3.12 | 2.08 |
   | same, 600 values each | 3.17 | 4.19 | 3.13 |
   | same, 50% null keys | 4.55 | 5.71 | 4.58 |
   | 3 distinct, first one supplies 2,040 rows | 1.99 | 2.30 | 2.07 |
   | 3 distinct, random rows | 2.07 | 2.09 | 2.06 |
   | A, B shared + unused distinct C | 2.05 | 2.16 | 2.05 |
   | same, 600 values each | 3.16 | 2.20 | 3.11 |
   | merge path: 16 distinct × 1,024 values, 1,024 rows | 24.7 | 20.4 | 20.4 |
   
   Null sentinel: `interleave(&[new_null_array(dict_type, 1), batch], ...)` and 
the same with two slices of `batch` both return copied values on this PR, not 
`batch.values()`.
   
   </details>
   



##########
arrow-select/src/interleave.rs:
##########
@@ -265,32 +265,62 @@ fn interleave_dictionaries<K: ArrowDictionaryKeyType>(
     let (should_merge, has_overflow) =
         should_merge_dictionary_values::<K>(&dictionaries, indices.len());
     if !should_merge {
+        let first_values = dictionaries[0].values();
+        let shared_values = if dictionaries
+            .iter()
+            .skip(1)
+            .all(|dictionary| Arc::ptr_eq(first_values, dictionary.values()))

Review Comment:
   No test fails if this `Arc::ptr_eq` check is removed: the two-input shared 
case falls back to copying the values, and the only multi-input reuse test has 
four inputs, so the referenced-input scan catches it instead. Could you add a 
two-input test? It also keeps the null-key coverage of the four-input test.
   
   ```rs
   #[test]
   fn test_interleave_dictionary_reuses_shared_values() {
       let values: ArrayRef = Arc::new(StringArray::from_iter_values(["a", "b", 
"c"]));
       let dict_a = DictionaryArray::<Int32Type>::new(
           Int32Array::from(vec![Some(0), None, Some(2)]),
           Arc::clone(&values),
       );
       let dict_b = DictionaryArray::<Int32Type>::new(
           Int32Array::from(vec![Some(1), Some(0)]),
           Arc::clone(&values),
       );
   
       let result = interleave(
           &[&dict_a, &dict_b],
           &[(0, 2), (1, 0), (0, 1), (1, 1), (0, 0)],
       )
       .unwrap();
       let result = result.as_dictionary::<Int32Type>();
   
       assert!(Arc::ptr_eq(result.values(), &values));
       assert_eq!(
           result.keys(),
           &Int32Array::from(vec![Some(2), Some(1), None, Some(0), Some(0)])
       );
   }
   ```



##########
arrow-select/src/interleave.rs:
##########
@@ -1232,6 +1393,24 @@ mod tests {
         assert_eq!(string_result, vec!["v0", "v0", "v49"]);

Review Comment:
   Optional cleanup: this test also passes on `main`, since it only checks the 
strings. Asserting the reuse makes it test what its new name says.
   
   ```suggestion
           assert_eq!(string_result, vec!["v0", "v0", "v49"]);
           assert!(Arc::ptr_eq(dict_result.values(), &values));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to