divyankshah commented on code in PR #5235: URL: https://github.com/apache/datafusion-comet/pull/5235#discussion_r3761775980
########## native/spark-expr/src/array_funcs/nested_float_normalize.rs: ########## @@ -0,0 +1,208 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::math_funcs::internal::normalize_float; +use arrow::array::{ + Array, ArrayRef, AsArray, FixedSizeListArray, Float32Array, Float64Array, LargeListArray, + ListArray, StructArray, +}; +use arrow::datatypes::{DataType, Float32Type, Float64Type}; +use std::sync::Arc; + +pub(super) fn has_float_leaf(dt: &DataType) -> bool { + match dt { + DataType::Float32 | DataType::Float64 => true, + DataType::List(field) | DataType::LargeList(field) | DataType::FixedSizeList(field, _) => { + has_float_leaf(field.data_type()) + } + DataType::Struct(fields) => fields.iter().any(|f| has_float_leaf(f.data_type())), + _ => false, + } +} + +/// Recursively rebuilds nested arrays with `-0.0` normalized to `0.0` and NaN canonicalized +/// in any Float32/Float64 leaves. +pub(super) fn normalize_nested_floats(array: &ArrayRef) -> ArrayRef { Review Comment: Hi @peterxcli, thanks for digging into this and for finding the #5166 thread, that's genuinely useful context. I think the two pieces you're describing are different sizes though. Moving the helper is a small refactor, but making NormalizeNaNAndZero actually recurse into List/Struct doesn't do much on its own, since nothing on the JVM side constructs it with a nested data_type today. The part that actually delivers the perf win needs contraintExpressions.scala to detect the ArrayTransform(arr, x -> NormalizeNaNAndZero(x)) shape and collapse it, and that's real serde work with its own tests, on a node Spark inserts broadly for joins, grouping and sort, not just collect_set or arrays_overlap. Does that split sound reasonable? I'd rather keep it out of this PR and open it as its own issue instead, referencing this thread and the one andygrove raised on #5166. I'll also add all the deatils to it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
