karuppayya commented on PR #4519:
URL:
https://github.com/apache/datafusion-comet/pull/4519#issuecomment-4597557555
Thanks @comphead.
> However this approach proposes a totally new vision and make decisions in
runtime?
Yes and no. This is still a planning-phase rule (`postColumnarTransitions`).
For non-AQE, it's staticsince the decision is made once during initial plan
optimization. For AQE, it runs per-stage as each stage is planned at every
shuffle boundary during runtime, but it's still a plan based decision.
> Is it considered a heavy weight stage? what is the strategy to fallback?
Currently the heuristic is structural ie it is based on counts the number of
C2R transitions in a stage's plan. It doesn't take into account data volume
etc. Incorporating those kinds of statistics could be a natural next step
toward a more cost-based decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
