jizezhang commented on issue #17964: URL: https://github.com/apache/datafusion/issues/17964#issuecomment-3536203735
Ah when working on the change I realized one more difference between the two: when the array is empty, `make_array` sets data type to be `DataType::Null` while `spark_array` sets it to be `DataType::Int32`. This logic is used both in determining return types and in `make_array_inner` method. Specifically, for `make_array`, see https://github.com/apache/datafusion/blob/899653f692433cb47f43c8e43cf2c2e874158f41/datafusion/functions-nested/src/make_array.rs#L106-L107 and https://github.com/apache/datafusion/blob/899653f692433cb47f43c8e43cf2c2e874158f41/datafusion/functions-nested/src/make_array.rs#L161 And for `spark_array`, see https://github.com/apache/datafusion/blob/899653f692433cb47f43c8e43cf2c2e874158f41/datafusion/spark/src/function/array/spark_array.rs#L95-L97 and https://github.com/apache/datafusion/blob/899653f692433cb47f43c8e43cf2c2e874158f41/datafusion/spark/src/function/array/spark_array.rs#L163 I think the difference in return types may also explains why `make_array` uses default `return_field_from_args` while `spark_array` overrrides so that it can set nullability to be false? https://github.com/apache/datafusion/blob/899653f692433cb47f43c8e43cf2c2e874158f41/datafusion/spark/src/function/array/spark_array.rs#L105-L109 Also as mentioned previously, `make_array_inner` method of the two has the difference in using `Field::new_list_field` with default element name "item" and using `Field::new` with specified element name "element", which further makes it a bit tricky to share `make_array_inner` method. Please let me know your thoughts. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
