[
https://issues.apache.org/jira/browse/DRILL-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18122027#comment-18122027
]
ASF GitHub Bot commented on DRILL-8555:
---------------------------------------
letian-jiang commented on PR #3086:
URL: https://github.com/apache/drill/pull/3086#issuecomment-5956870771
@cgivre Thanks for pointing me to #3023. I have reviewed it, and I agree
that both PRs share the same underlying goal: reusing planning work to reduce
the overhead of repeated queries.
Before considering #3023 production-ready, I think we would need to
establish the conditions for safe plan reuse, even for identical SQL. For
example:
1. Time-dependent or non-deterministic expressions, such as date/time and
random functions, need special handling when expressions are evaluated or
folded during planning.
2. Changes to options that affect planning may invalidate a cached plan.
3. Physical plans contain mutable state, such as node assignments, which
should be isolated between executions.
4. Changes to the underlying data files may require rebuilding scan state.
5. Changes to the underlying table schema may invalidate the plan.
There is also a practical consideration: production workloads often repeat
the same query shape with different literals. Restricting reuse to identical
SQL text would substantially limit the benefit for those workloads.
My implementation focuses on these correctness requirements through
eligibility checks, context and table compatibility checks, and reconstruction
of a fresh plan and scan state for each execution. It also preserves typed
parameter slots so queries with different literals can reuse compatible cached
plans.
> Physical plan cache for parameterized SQL queries
> -------------------------------------------------
>
> Key: DRILL-8555
> URL: https://issues.apache.org/jira/browse/DRILL-8555
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
> Reporter: Letian Jiang
> Priority: Major
>
> h2. Motivation
> Repeated queries that differ only in literals are a common workload, yet each
> query currently requires full planning. This adds unnecessary latency and
> wastes CPU time.
> h2. Brief Design
> Add a per-Drillbit, in-memory plan cache. Drill automatically parameterizes
> eligible SQL queries and reuses a cached physical plan with the current
> literal values, without changes to client queries.
> h2. Plan reuse safety
> * *Engine:* matches the SQL template and checks relevant option changes,
> plugin configuration changes, and whether base table versions still match.
> * *Storage plugin:* explicitly opts in, provides a plan-compatible table
> version, and is responsible for rebuilding its predicate-dependent scan work
> from the rebound expressions.
> h2. Expected benefit
> Short, frequently executed queries such as HBase reads can benefit
> substantially. Analytical queries with significant planning cost can benefit
> as well.
> h2. Implementation status
> A working implementation is available, including HBase, Paimon and Iceberg
> integrations. A pull request will follow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
