comphead commented on code in PR #5064: URL: https://github.com/apache/datafusion-comet/pull/5064#discussion_r3666975917
########## docs/source/contributor-guide/roadmap.md: ########## @@ -85,25 +88,20 @@ Comet's native hash join currently requires the build side to fit entirely in me support will allow Comet to handle larger joins without falling back to Spark, improving both reliability and performance for memory-intensive workloads. -## Java/Scala Columnar and Arrow UDF Support +## Java/Scala UDF Support -Spark users frequently define custom UDFs in Java or Scala. Comet currently falls back to Spark when a query -contains a JVM UDF. Adding support for calling Java/Scala UDFs that operate on columnar Arrow data directly -from native execution will reduce fallbacks and allow more queries to run end-to-end in Comet. +Spark users frequently define custom UDFs in Java or Scala. Comet now dispatches scalar `ScalaUDF` expressions +through a JVM codegen bridge (`CometScalaUDF`) instead of always falling back to Spark. Aggregate UDFs, table +UDFs/generators, Python/Pandas UDFs, and Hive `GenericUDF`/`SimpleUDF` still fall back to Spark entirely. +Extending the codegen-dispatch approach to cover these remaining categories will reduce fallbacks further and +allow more queries to run end-to-end in Comet. ## Memory Management Improvements Comet coordinates memory between the JVM and native Rust execution through a custom memory pool. Improving memory accounting, reservation strategies, and spill integration will reduce out-of-memory errors and allow Comet to make better use of available resources, especially in multi-query and multi-task environments. -## Prepare for 1.0.0 Release - -The project is working toward a 1.0.0 release. This effort includes finalizing configuration options, -resolving known correctness issues, and improving documentation. Progress is tracked in [#4082]. - -[#4082]: https://github.com/apache/datafusion-comet/issues/4082 - ## Native Parquet Writes Review Comment: Iceberg HDFS reads -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
