FrankChen021 commented on PR #5284: URL: https://github.com/apache/calcite/pull/5284#issuecomment-6356691319
> Thanks for the measurements. The benchmark shows no measurable change in time, since the confidence intervals overlap, and allocation drops by 1.3–2.5% only on an IN list of 100,000 to 1,000,000 string literals. On ordinary queries the difference would be lost in noise. The price is two new public overloads of `NlsString.asSql` that we would have to maintain, one of them with a `suffix` parameter that has no effect. > You are right to question it. These optimizations were done from the biggest gain to the lowest, and this optimization was the last one I conducted. After previous 4 was done, the allocation dropped from 1,454.93GB to about 400MiB-500MiB, and then this small change saves another 35.73MiB. If we measure this one based on the 400MiB-500MiB, the percentage is not 2.46%, but about 8.9%. Since this optimization is an independant one, I have to measure it again based on the master branch, this makes the improvement 'little gain' if we look at the percentage number. I agree with you that we should focus on the biggest gain first. > Could you also explain why Druid does not pass a large list of IDs to Calcite as a single array value? Even an array bound to a `?` parameter is expanded by Druid's `SqlParameterizerShuttle#createArrayLiteral` into `ARRAY[...]` with one `SqlLiteral` per element, so planning still processes a million literals. A `RexDynamicParam` has a constant-size digest and needs no per-element validation or type inference. For this proposal, I posted comment in [CALCITE-7805](https://issues.apache.org/jira/browse/CALCITE-7805) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
