karuppayya commented on PR #4519:
URL:
https://github.com/apache/datafusion-comet/pull/4519#issuecomment-4595666716
Thanks @andygrove . I think difference is that
`COMET_EXEC_SHUFFLE_CONVERT_FROM_SPARK_PLAN_ENABLED` only checks the immediate
child of the shuffle to decide whether to convert or not. It doesn't look at
the full stage plan, so you could still end up with costly transitions deeper
in the stage
This changes tries evaluates the full stage plan and makes a per-stage
decision based on the actual transition cost. In future we can extend it to
factor in estimated row counts, operator complexity, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
