andygrove commented on issue #1123:
URL: 
https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2510299725

   > If so, it makes some sense because Comet shuffle reader is still JVM 
based. 
   
   This is also an issue for shuffle writes. The child node of 
`ShuffleWriterExec` is always a `ScanNode` reading the output from a native 
plan.
   
   We pay the FFI cost twice - once to import from native plan to JVM, then 
again to export to the shuffle native plan. We have the cost of schema serde in 
both directions. Perhaps there is a way to shortcut this and avoid a full serde 
because we do not need to read the batch in the JVM in this case, just pass it 
from one native plan to another.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org

Reply via email to