[
https://issues.apache.org/jira/browse/IMPALA-13454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097926#comment-18097926
]
Michael Smith edited comment on IMPALA-13454 at 8/18/26 8:27 PM:
-----------------------------------------------------------------
Follow-up ideas:
* Limit how many Cells CTEProducers push on LocalExchanger: Limit the number
of batches to be at most N batches ahead of the fastest consumer rather than
accumulating more. This doesn't really bound memory, but it avoids some cases
where the extra memory doesn't help much. (But this can't be used with the
SequenceNode or UnionNode.)
* -Add support for runtime filters pushing to CTEConsumer.-
* Limiting CTEs based on expected result size.
* Spilling in the LocalExchanger for disparate consumers.
was (Author: JIRAUSER288956):
Follow-up ideas:
* Limit how many Cells CTEProducers push on LocalExchanger: Limit the number
of batches to be at most N batches ahead of the fastest consumer rather than
accumulating more. This doesn't really bound memory, but it avoids some cases
where the extra memory doesn't help much. (But this can't be used with the
SequenceNode or UnionNode.)
* Add support for runtime filters pushing to CTEConsumer.
* Limiting CTEs based on expected result size.
* Spilling in the LocalExchanger for disparate consumers.
> Support CTE Materialization in Calcite Planner
> ----------------------------------------------
>
> Key: IMPALA-13454
> URL: https://issues.apache.org/jira/browse/IMPALA-13454
> Project: IMPALA
> Issue Type: Improvement
> Reporter: weihua zhang
> Assignee: Michael Smith
> Priority: Major
>
> Implement support for processing Common Table Expressions identified by the
> Calcite planner integrated in Impala.
> Requirements:
> * Must be able to spill to disk to minimize memory use.
> * Must be able to disable CTEs for a specific query or all queries.
> * Non-goal: CTEs do not need to be usable across multiple queries.
> [Optimization of Common Table Expressions in MPP Database
> Systems|https://www.vldb.org/pvldb/vol8/p1704-elhelw.pdf] presents a model
> for describing CTEs in planning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
