FelixYBW commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6304031890
> Wondering if there are future plans to offload more operators to native libraries beyond the table scan, Compared to more operators, the other missing parts may be more important and key to production: - configurations, key to native performance - memory management, memory allocation in offheap in native library - metrics reported from the native library - introducing refcount-based batch management, instead of current independent batch + reusable batch - automatic batch type detection and transition [SPARK-57468](https://issues.apache.org/jira/browse/SPARK-57468) - relational expressions of pushdown filter and agg; currently, the PR uses plain SQL. Better to reuse Spark Connect, Substrate, or any other 3rd-party protocol. - task cancellation and cleanup of native library and many others. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
