andygrove commented on issue #1123: URL: https://github.com/apache/datafusion-comet/issues/1123#issuecomment-2510299725
> If so, it makes some sense because Comet shuffle reader is still JVM based. This is also an issue for shuffle writes. The child node of `ShuffleWriterExec` is always a `ScanNode` reading the output from a native plan. We pay the FFI cost twice - once to import from native plan to JVM, then again to export to the shuffle native plan. We have the cost of schema serde in both directions. Perhaps there is a way to shortcut this and avoid a full serde because we do not need to read the batch in the JVM in this case, just pass it from one native plan to another. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
