advancedxy commented on issue #102: URL: https://github.com/apache/arrow-datafusion-comet/issues/102#issuecomment-1962789502
Thanks for detailed explanation, it helps a lot. I changed the issue title as well.. > I think, is that it has to do extra row-to-columnar conversion regardless of whether the input is columnar batch (from native Comet operator), or rows (e.g., from Spark operator) It seems there's still place to improve. Will native shuffle supports sort-based shuffle in your long-term roadmap? I understand that columnar shuffle has better coverage and all the dirty works are done at the JVM side and prioritize that for now. I can imagine that It would require great engineering effort to support feature parity in the pure native shuffle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
