marin-ma commented on issue #6914: URL: https://github.com/apache/incubator-gluten/issues/6914#issuecomment-2296064889
@ccat3z The partition filter is computed in Rule `PruneFileSourcePartitions` against the logical plan. It's not in the execution phase so Gluten cannot take over the evaluation of the partition filters. Therefore, the udf in partition filter must be the Java-based udf. With #6829, user can register the native implementation for a hive udf while also registering a Java-based hive udf via `create temporary function`. This allows Spark to locate and use the Java-based udf when calling `Expression.eval`. With this patch, SQL execution will be offloaded if validation passes; otherwise, it will fall back to the Java-based implementation. However, for partition filters, it directly uses the java-based implementation. Here's an example https://github.com/apache/incubator-gluten/pull/6829/commits/41ec4c72787c7a9f8d8cbf9c1844e988fba2db59 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
