Zoltán Borók-Nagy created IMPALA-15658:
------------------------------------------
Summary: ExecSummary repeats CTE producers under every consumer:
TPC-DS q14_1's profile is 78 MB
Key: IMPALA-15658
URL: https://issues.apache.org/jira/browse/IMPALA-15658
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Zoltán Borók-Nagy
With materialized CTEs (PLANNER=CALCITE, CTE_THRESHOLD=1), the ExecSummary
prints each CTE producer's subtree again under every consumer that reads it,
and CTEs that read other CTEs multiply this.
TPC-DS q14_1 at SF1000 on 8 executors (MT_DOP=8): its CTEs have up to 15
consumers each, and
* the text profile is 78 MB, with 1.5M ExecSummary lines;
* the JSON profile is 214 MB.
Profiles this large are slow to produce, store and render in the web UI, and
they blow up the profile log.
*Proposal:* print each producer once and let its consumers refer to it, as
exchange nodes refer to the fragment that sends to them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
