mbutrovich opened a new issue, #1995:
URL: https://github.com/apache/datafusion-comet/issues/1995
While working on #1887 I looked at logic related to wrapping nodes with
CopyExec (mostly for unpacking dictionaries. We have logic in planner.rs in
several places that determines when to insert CopyExec nodes in the native
physical plan. We should move that logic to the Spark side and define a new
CopyExec node that we can translate 1:1 on the native side. This has a few
benefits:
- CopyExec nodes will show up in the Spark UI representation of the plan
- More logic about plan translation is consolidated on the Spark side, which
offers benefits of its own:
- Easier to understand what the final native plan will be during Comet
rule execution on the Spark side
- Easier to cache serialized native plans
- Reduce redundant work across Spark workers in native code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
