FrankChen021 opened a new pull request, #20367:
URL: https://github.com/apache/druid/pull/20367
Related to #20326.
### Description
Large literal arrays already contain constant Calcite values, but the
existing planning path converts them into Druid expressions, formats them as
text, parses that text, evaluates it, and rebuilds Calcite literals:
```text
DruidRexExecutor.reduce()
Expressions.toDruidExpression(...)
ArrayConstructorOperatorConversion
DirectOperatorConversion
Convert each literal into a child DruidExpression
DruidExpression.ofFunctionCall(..., "array", operands)
Creates the text-generator callback
druidExpression.getExpression()
Formats "array('a','b',...)"
Parser.parse(...)
Builds a new Druid expression tree
expr.eval(...)
Produces the array value
rexBuilder.makeLiteral(...)
Rebuilds the Calcite array and its elements
```
This round trip allocates per-element expression objects, strings, parser
objects, evaluated values, and replacement Calcite nodes.
This change skips that work for already-normalized string and integer
literal arrays. Expressions requiring conversion or evaluation retain the
existing path.
### Benchmark results
| Type | Elements | Before MB/op | After MB/op | Diff MB/op | Reduction |
Before ms/op | After ms/op |
|---|---:|---:|---:|---:|---:|---:|---:|
| LONG | 100,000 | 355.87 | 226.23 | 129.63 | 36.43% | 215.95 | 153.58 |
| LONG | 1,000,000 | 3535.11 | 2216.22 | 1318.89 | 37.31% | 2899.52 |
2324.95 |
| STRING | 100,000 | 1541.08 | 1231.01 | 310.06 | 20.12% | 788.72 | 580.71 |
| STRING | 1,000,000 | 15561.88 | 12307.44 | 3254.44 | 20.91% | 8106.12 |
6150.46 |
MB is decimal; allocation means **total bytes allocated per planning
operation**, not peak or retained heap.
Both variants use Calcite 1.42.0 with only the CALCITE-7782 fix. The
benchmark isolates this Druid change using identical dependencies and harness
settings.
JDK 25.0.4.1; two forks; two warmup and three measurement iterations; GC
profiler. Queries are planned without execution or EXPLAIN serialization.
Timing results are preliminary.
### Tests
Tests cover identity reuse, NULLs, integer boundaries, type mismatches,
charset/collation differences, computed operands, and SQL IN/NOT IN semantics.
- **261 tests passed** with the declared Calcite dependency.
- **186 targeted tests passed** with patched Calcite. The broader patched
run encounters an unrelated exact-version assertion.
#### Release note
Reduced SQL planning allocations for large literal string and integer
arrays, including arrays generated from large IN lists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
