alexy opened a new pull request, #26066: URL: https://github.com/apache/datafusion/pull/26066
## Which issue does this PR close? - Closes #26065. ## Rationale for this change Physical planning slows down, faster than the query grows, when projections carry many typed NULL struct literals: a `UNION ALL` of 35 branches with 35 struct columns of 20 fields takes 862 ms in `create_physical_plan` on `main`. `EquivalenceGroup::add_constant` compares each new constant with every existing class, and `ScalarValue::eq` compares nested values with arrow's array equality, which converts both arrays to `ArrayData` before it compares anything, even when the two values have different types. ## What changes are included in this PR? `ScalarValue::eq` compares the nested variants (`List`, `LargeList`, `FixedSizeList`, `ListView`, `LargeListView`, `Struct`, `Map`) through a helper that returns early for the same `Arc`, and compares lengths and data types before arrow's array equality. Results are unchanged; only values of the same type and length reach the `ArrayData` comparison. With the reproducer from the issue (release build, Apple M1 Max), `create_physical_plan`: | struct columns x fields | main | this PR | |---|---:|---:| | 10 x 20 | 31 ms | 17 ms | | 20 x 20 | 179 ms | 47 ms | | 35 x 20 | 862 ms | 147 ms | | 35 x 40 | 1,766 ms | 253 ms | ## What is the testing strategy for this PR? `test_nested_scalar_eq` covers typed NULL structs of the same and of different types, struct values that are equal and unequal, and lists of different element types. The scalar tests in `datafusion-common` pass. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
