peterenescu commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6303536126
Thanks @HyukjinKwon, this is really exciting. Wondering if there are future plans to offload more operators to native libraries beyond the table scan, such as filters, projections, aggregations and joins, up to a native boundary at the exchange. Gluten of course helps with this, but a native solution to help run plans in Velox would be amazing. If so, I think @zhouyuan's request for a threading contract on the stream makes sense. The docs say Spark calls the stream from the task's thread, but a native engine would want to consume it directly from its own threads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
