andygrove opened a new pull request, #6777: URL: https://github.com/apache/datafusion-comet/pull/6777
## Which issue does this PR close? No issue. These are follow-ups to #4459, split out of #6697 so they can land without its vectorized JVM UDF API. ## Rationale for this change Two gaps in the native UDFs that #4459 added: - `CometNativeUDF.register` accepts a signature with a type Comet has no native representation for, such as a user-defined type. Every call then falls back to Spark, which cannot evaluate a native UDF, so the query fails at run time, far from the registration that caused it. - Spark evaluates a call itself not only in an operator Comet does not take, but also while it plans a query: over local data such as `VALUES`, in a filter on partition columns, and when it samples a global sort's keys to choose range bounds. In those cases Comet runs every operator, yet the error says that Comet did not take the operator holding the call and points to the extended explain output, which shows no fallback. The scaladoc and the Rust UDF guide say the same. ## What changes are included in this PR? - `CometNativeUDF.register` refuses a signature with an argument or return type that `QueryPlanSerde.serializeDataType` cannot serialize, before it loads the library, and the error names the type. - `CometUdfNotEvaluatedException`'s message gives both causes: an operator Comet did not take, whose reason is in the extended explain output, or Spark evaluating the call while it plans the query, in the three cases above. - The scaladoc of `CometNativeUDF.register` and `NativeUdfCall` says the same. - The Rust UDF guide has a new section, "When Spark evaluates the call", and a limitation that links to it. It also says that registration checks the types, and that a query should sort on a column holding a UDF's result rather than on the call itself. ## How are these changes tested? Three new tests in `CometNativeUdfSuite`: - a native UDF over `VALUES` fails while Spark plans the query, with the error naming the planning-time cases - a global sort on a column holding a native UDF's result runs, which is the workaround the guide gives - registering a UDF with an `ObjectType` argument is refused and installs no function Removing the type check fails the registration test, and taking the planning-time cases out of the error fails the `VALUES` test. `CometNativeUdfSuite` passes on Spark 4.1 (71 tests). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
