gstvg commented on code in PR #18921: URL: https://github.com/apache/datafusion/pull/18921#discussion_r3006017610
########## datafusion/functions-nested/src/array_transform.rs: ########## Review Comment: @LiaCastaneda Hmm `clean_null_values` sound good, implemented at https://github.com/apache/datafusion/pull/18921/changes/a9d0e6cbf4b7ddd0e3d70a4076193f93073af62a. For variable sized lists we can also check if any null is non-empty before moving the data. The cleanup is indeed the same everytime, so we can do it transparently Maybe in the future we can add a `null_safe` method to `PhysicalExpr` and only cleanup if any expr of the lambda body is non null safe? Yes, vectorized execution is a must, and if the cleanup proves to be too expensive maybe we can use it only when there are no nulls, and otherwise use a loop like Trino does -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
