alamb commented on issue #11413: URL: https://github.com/apache/datafusion/issues/11413#issuecomment-2232921403
> Sorry it takes longer than I expected to make this works end-to-end. I plan to make an ScalarUDF with arrow-udf as the completion of the prototype work. Thank you so much @xinlifoobar -- this is really helpful and a great analysis (I think the pros/cons you identified make a lot of sense to me) From what I can see, if we wanted to proceed with using `arrow-udf` in DataFusion we would need to address the shortcomings you identified above. Here are some additional discussions > By default all udf are private, lack of a way to reference the udf that could be used in e.g., ExprPlanner. I think this is part of the same concept as discussed on https://lists.apache.org/thread/x8wvlkfr0osl15o52rw85wom0p4v05x6 -- basically the arrow-udf library's scope is large enough to encompass things like a function registry that DataFusion already has > Lack of support for operations against array and scalar. I do think being able to special case scalar value is a critical requirement for performance. I will post about your findings on the mailing lists and let's see what the authors of arrow-udf have to say cc @wangrunji0408 @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org