FrankChen021 commented on PR #3395:
URL: https://github.com/apache/calcite/pull/3395#issuecomment-5665073662
This change appears to introduce a severe performance regression when
validating very large array constructors containing mixed-width string literals.
We reproduced it using Apache Druid’s
`InPlanningBenchmark.queryStringFunctionInSql`:
```sql
EXPLAIN PLAN FOR
SELECT COUNT(*)
FROM foo
WHERE long1 = 8
OR LOWER(string1) IN ('1', '2', ..., '1000000')
```
Benchmark parameters:
- `inClauseLiteralsCount=1000000`
- `inSubQueryThreshold=2147483647`
- `rowsPerSegment=500000`
Results:
| Calcite version | Result |
|---|---:|
| 1.35.0 | 10.39 s/op, 14.94 GB allocated/op |
| 1.37.0 | No completed operation after 20 minutes |
| 1.41.0 | No completed operation after 20 minutes |
| 1.42.0 | No completed operation after 20 minutes |
The regression first appears after upgrading from Calcite 1.35 to 1.37; this
PR was included in Calcite 1.36.
A JFR profile of the 100,000-string case attributed most allocation pressure
to:
- `ImmutableList.copyOf`: 50.29%
- `Platform.copy`: 41.28%
The dominant stack was:
```text
SqlArrayValueConstructor.inferReturnType
-> SqlValidatorUtil.adjustTypeForArrayConstructor
-> SqlValidatorUtil.adjustTypeForMultisetConstructor
-> SqlBasicCall.setOperand
-> ImmutableNullableList.copyOf
```
`adjustTypeForMultisetConstructor` invokes `setOperand` separately for every
element requiring a cast. `SqlBasicCall.setOperand` copies the complete
immutable operand list each time. For large mixed-width string arrays, this
produces approximately quadratic copying and extreme allocation pressure.
Numeric arrays generally do not reproduce the same degradation because their
elements commonly already have the derived component type.
Related Druid investigation:
https://github.com/apache/druid/issues/20326#issuecomment-5664833172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
