mbutrovich commented on code in PR #70:
URL: https://github.com/apache/datafusion-spark/pull/70#discussion_r4602374291


##########
crates/spark/src/function/array/array_insert.rs:
##########
@@ -0,0 +1,457 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+use std::sync::Arc;
+
+use crate::function::error_utils::unsupported_data_type_exec_err;
+use arrow::array::{
+    Array, ArrayRef, AsArray, Capacities, GenericListArray, Int32Array, 
MutableArrayData,
+    OffsetSizeTrait, make_array,
+};
+use arrow::buffer::{NullBuffer, OffsetBuffer};
+use arrow::datatypes::{DataType, Field, FieldRef};
+use datafusion_common::utils::take_function_args;
+use datafusion_common::{Result, exec_err, internal_err};
+use datafusion_expr::type_coercion::binary::binary_numeric_coercion;
+use datafusion_expr::{
+    ColumnarValue, ReturnFieldArgs, ScalarFunctionArgs, ScalarUDFImpl, 
Signature,
+    Volatility,
+};
+
+/// `ByteArrayUtils.MAX_ROUNDED_ARRAY_LENGTH` (`Integer.MAX_VALUE - 15`) in 
Spark.
+const MAX_ROUNDED_ARRAY_LENGTH: i64 = 2_147_483_632;
+
+/// Spark-compatible `array_insert(array, pos, item)`.
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#array_insert>
+///
+/// - Indexes are 1-based. A negative index counts from the end, where `-1`
+///   inserts after the last element.
+/// - An index past the end pads with NULLs. A negative index before the start
+///   places the item first, then NULL padding, then the original elements.
+/// - The result is NULL when the array or the index is NULL. A NULL item is
+///   inserted as a NULL element.
+/// - Index `0` is an error.
+/// - The index may be any integer type and is cast to INT, because DataFusion
+///   types bare integer literals as BIGINT.
+///
+/// Spark's `spark.sql.legacy.negativeIndexInArrayInsert` (default `false`) 
makes
+/// `-1` insert before the last element. Use 
[`Self::with_legacy_negative_index`].
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkArrayInsert {
+    signature: Signature,
+    legacy_negative_index: bool,
+}
+
+impl Default for SparkArrayInsert {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkArrayInsert {
+    pub fn new() -> Self {
+        Self::with_legacy_negative_index(false)
+    }
+
+    pub fn with_legacy_negative_index(legacy_negative_index: bool) -> Self {
+        Self {
+            signature: Signature::user_defined(Volatility::Immutable),
+            legacy_negative_index,
+        }
+    }
+}
+
+/// The element field of a list type.
+fn list_element_field(data_type: &DataType) -> Option<&FieldRef> {
+    match data_type {
+        DataType::List(f) | DataType::LargeList(f) | 
DataType::FixedSizeList(f, _) => {
+            Some(f)
+        }
+        _ => None,
+    }
+}
+
+impl ScalarUDFImpl for SparkArrayInsert {
+    fn name(&self) -> &str {
+        "array_insert"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+        internal_err!("return_field_from_args should be used instead")
+    }
+
+    fn return_field_from_args(&self, args: ReturnFieldArgs) -> 
Result<FieldRef> {
+        let [array, pos, _item] = take_function_args(self.name(), 
args.arg_fields)?;
+        let Some(elem) = list_element_field(array.data_type()) else {
+            return internal_err!(
+                "array_insert expects a list, got {}",
+                array.data_type()
+            );
+        };
+        // Padding can introduce NULLs even when neither input has any, so the
+        // element field is always nullable.
+        let elem = Arc::new(Field::new(elem.name(), elem.data_type().clone(), 
true));
+        let data_type = match array.data_type() {
+            DataType::LargeList(_) => DataType::LargeList(elem),
+            _ => DataType::List(elem),
+        };
+        Ok(Arc::new(Field::new(
+            self.name(),
+            data_type,
+            array.is_nullable() || pos.is_nullable(),
+        )))
+    }
+
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+        let [array, pos, item] = take_function_args(self.name(), arg_types)?;
+
+        let (elem, wrap): (&FieldRef, fn(FieldRef) -> DataType) = match array {
+            DataType::List(f) | DataType::FixedSizeList(f, _) => (f, 
DataType::List),
+            DataType::LargeList(f) => (f, DataType::LargeList),
+            _ => {
+                return Err(unsupported_data_type_exec_err(
+                    self.name(),
+                    "an array as first argument",
+                    array,
+                ));
+            }
+        };
+
+        // Spark takes an INT index. DataFusion types bare integer literals as
+        // BIGINT, so accept every integer type and cast to INT; a value 
outside
+        // the INT range then fails the cast instead of being silently 
truncated.
+        if !pos.is_integer() {
+            return Err(unsupported_data_type_exec_err(
+                self.name(),
+                "an INT index as second argument",
+                pos,
+            ));
+        }
+
+        // Spark uses the tightest common type of the element and the item.
+        let elem_type = elem.data_type();
+        let common = if elem_type == item || *item == DataType::Null {
+            elem_type.clone()
+        } else if *elem_type == DataType::Null {
+            item.clone()
+        } else if let Some(common) = binary_numeric_coercion(elem_type, item) {
+            common
+        } else {
+            return exec_err!(

Review Comment:
   > The fallback now runs after the numeric case, with a check that rejects 
any pair mixing string or binary with a different kind, at any nesting depth.
   
   The fallback also accepts structs that Spark rejects, because 
`type_union_resolution` matches struct fields by name. With this PR, 
`array_insert(make_array(named_struct('a', 1::INT, 'b', 2::INT)), 1, 
named_struct('b', 3::INT, 'a', 4::INT))` returns `[{b: 3, a: 4}, {b: 2, a: 
1}]`. Spark 4.2.0 raises `ARRAY_FUNCTION_DIFF_TYPES` for the same query in both 
ANSI modes, because 
[`findTypeForComplex`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala#L164-L177)
 pairs struct fields by position. Struct field types also widen here: 
`named_struct('a', 1::INT)` with `named_struct('a', 2::BIGINT)` gives 
`List(Struct("a": Int64))`. Spark widens them only with ANSI on, and 
`array_position` on `main` rejects them to [follow ANSI 
off](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/spark/src/function/array/array_position.rs#L245-L24
 8).
   
   Could we drop `spark_rejects_pair` and the DataFusion fallback and call 
[`tightest_common_type`](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/spark/src/function/array/array_position.rs#L203-L276)
 from `array_position.rs`, as #330 does? It already implements 
`findTightestCommonType` for these types and pairs struct fields by position, 
so the two functions would accept the same types. My review of #332 proposes 
moving it, with the Spark type names that `spark_type_name` here renders, into 
one shared module. It needs one fix first. Its `Map` arm [recurses into the 
entries 
struct](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/spark/src/function/array/array_position.rs#L267-L273),
 whose arm requires identical field types, so `map<int, int>` and `map<bigint, 
bigint>` have no common type. Spark [widens the key and the 
value](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e
 
084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala#L149-L163)
 in both ANSI modes, and the existing "Maps whose key and value types widen" 
case in `array_insert.slt` checks it. A `.slt` case with reordered struct 
fields would cover the struct point.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to