mbutrovich commented on code in PR #345:
URL: https://github.com/apache/datafusion-spark/pull/345#discussion_r4605672194


##########
crates/spark/src/function/type_coercion.rs:
##########
@@ -0,0 +1,427 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Spark's rules for finding a common type of several arguments, from its 
`TypeCoercion`
+//! and `AnsiTypeCoercion`, on the Arrow types that Spark's types map to. Also 
the checks
+//! and type names that the errors for those arguments use.
+//!
+//! These are for functions that need one type for several arguments, such as
+//! `array_position`. They are not Spark's implicit casts to a function's 
declared input
+//! types, which the functions do through their [`datafusion_expr::Signature`].
+
+use std::sync::Arc;
+
+use arrow::datatypes::{DataType, Field, Fields, IntervalUnit};
+
+/// Spark's `findTightestCommonType`. `ansi_mode` picks between 
`AnsiTypeCoercion` and
+/// `TypeCoercion`, which differ only in that ANSI widens an integer and a 
FLOAT to DOUBLE.
+/// Arrays, maps and structs are widened element by element with this function.
+pub(crate) fn tightest_common_type(
+    left: &DataType,
+    right: &DataType,
+    ansi_mode: bool,
+) -> Option<DataType> {
+    use DataType::*;
+    if left == right {
+        return Some(left.clone());
+    }
+    // Spark has no dictionary types. A dictionary-encoded argument, such as a
+    // Parquet string column, has the type of its values.
+    if let Dictionary(_, value) = left {
+        return tightest_common_type(value, right, ansi_mode);
+    }
+    if let Dictionary(_, value) = right {
+        return tightest_common_type(left, value, ansi_mode);
+    }

Review Comment:
   When both arguments have the same dictionary type, the `left == right` check 
returns the dictionary before the unwrapping below it runs, so the dictionary 
reaches the function as it is. `normalize_array` doesn't look inside 
dictionaries, so `-0.0` and `0.0` stay unequal. `SELECT 
array_position(make_array(-0.0, 1.0), 0.0)` returns `1`, as Spark does, but the 
same query with both sides cast to `Dictionary(Int32, Float64)` 
(`arrow_cast(make_array(-0.0, 1.0), 'List(Dictionary(Int32, Float64))')` and 
`arrow_cast(0.0, 'Dictionary(Int32, Float64)')`) returns `0`. The new `.slt` 
case at `array_position.slt` passes because its strings don't need normalizing. 
Could the dictionary checks come first, with a float case for two dictionaries?
   
   ```suggestion
       // Spark has no dictionary types. A dictionary-encoded argument, such as 
a
       // Parquet string column, has the type of its values.
       if let Dictionary(_, value) = left {
           return tightest_common_type(value, right, ansi_mode);
       }
       if let Dictionary(_, value) = right {
           return tightest_common_type(left, value, ansi_mode);
       }
       if left == right {
           return Some(left.clone());
       }
   ```



##########
crates/spark/src/function/type_coercion.rs:
##########
@@ -0,0 +1,427 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Spark's rules for finding a common type of several arguments, from its 
`TypeCoercion`
+//! and `AnsiTypeCoercion`, on the Arrow types that Spark's types map to. Also 
the checks
+//! and type names that the errors for those arguments use.
+//!
+//! These are for functions that need one type for several arguments, such as
+//! `array_position`. They are not Spark's implicit casts to a function's 
declared input
+//! types, which the functions do through their [`datafusion_expr::Signature`].
+
+use std::sync::Arc;
+
+use arrow::datatypes::{DataType, Field, Fields, IntervalUnit};
+
+/// Spark's `findTightestCommonType`. `ansi_mode` picks between 
`AnsiTypeCoercion` and
+/// `TypeCoercion`, which differ only in that ANSI widens an integer and a 
FLOAT to DOUBLE.
+/// Arrays, maps and structs are widened element by element with this function.
+pub(crate) fn tightest_common_type(
+    left: &DataType,
+    right: &DataType,
+    ansi_mode: bool,
+) -> Option<DataType> {
+    use DataType::*;
+    if left == right {
+        return Some(left.clone());
+    }
+    // Spark has no dictionary types. A dictionary-encoded argument, such as a
+    // Parquet string column, has the type of its values.
+    if let Dictionary(_, value) = left {
+        return tightest_common_type(value, right, ansi_mode);
+    }
+    if let Dictionary(_, value) = right {
+        return tightest_common_type(left, value, ansi_mode);
+    }
+    if let (Some(l), Some(r)) = (numeric_precedence(left), 
numeric_precedence(right)) {
+        let wider = if l >= r { left } else { right };
+        // With ANSI mode on, an integer and a FLOAT widen to DOUBLE, which 
holds every INT
+        // exactly.
+        return Some(if ansi_mode && *wider == Float32 {
+            Float64
+        } else {
+            wider.clone()
+        });
+    }
+    match (left, right) {
+        (Null, other) | (other, Null) => Some(other.clone()),
+        // Spark has one string type and one binary type.
+        (l, r) if l.is_string() && r.is_string() => 
Some(wider_physical_type(l, r)),
+        (l, r) if is_binary(l) && is_binary(r) => Some(wider_physical_type(l, 
r)),
+        // The decimal must hold every value of the integral type.
+        (int, Decimal128(p, s)) | (Decimal128(p, s), int) => {
+            let digits = integral_digits(int)?;
+            (*p as i16 - *s as i16 >= digits).then_some(Decimal128(*p, *s))
+        }
+        (Date32, ts @ Timestamp(_, _)) | (ts @ Timestamp(_, _), Date32) => {
+            Some(ts.clone())
+        }
+        // TIMESTAMP_NTZ and TIMESTAMP widen to TIMESTAMP.
+        (Timestamp(lu, ltz), Timestamp(ru, rtz)) => Some(Timestamp(
+            // Arrow orders the units from Second to Nanosecond.
+            (*lu).max(*ru),
+            ltz.clone().or_else(|| rtz.clone()),
+        )),
+        _ => complex_type(left, right, &|l, r| tightest_common_type(l, r, 
ansi_mode)),
+    }
+}
+
+/// Spark's `findTypeForComplex`: widens the elements of arrays, the keys and 
values of maps,
+/// and the fields of structs with `find_type`. A nested field is nullable if 
it is on either
+/// side. Struct fields must have the same names in the same order. Spark 
compares them
+/// ignoring case, but DataFusion cannot cast between structs whose field 
names differ only
+/// in case, so this requires an exact match.
+fn complex_type(
+    left: &DataType,
+    right: &DataType,
+    find_type: &dyn Fn(&DataType, &DataType) -> Option<DataType>,
+) -> Option<DataType> {
+    use DataType::*;
+    match (left, right) {
+        (
+            List(lf) | LargeList(lf) | FixedSizeList(lf, _),
+            List(rf) | LargeList(rf) | FixedSizeList(rf, _),
+        ) => {
+            let item = find_type(lf.data_type(), rf.data_type())?;
+            let field = Arc::new(Field::new(
+                lf.name(),
+                item,
+                lf.is_nullable() || rf.is_nullable(),
+            ));
+            Some(match (left, right) {
+                (FixedSizeList(_, ln), FixedSizeList(_, rn)) if ln == rn => {
+                    FixedSizeList(field, *ln)
+                }
+                (LargeList(_), _) | (_, LargeList(_)) => LargeList(field),
+                _ => List(field),
+            })
+        }
+        (Struct(lf), Struct(rf)) if lf.len() == rf.len() => {
+            let fields = lf
+                .iter()
+                .zip(rf.iter())
+                .map(|(a, b)| {
+                    if a.name() != b.name() {
+                        return None;
+                    }
+                    let data_type = find_type(a.data_type(), b.data_type())?;
+                    Some(Field::new(
+                        a.name(),
+                        data_type,
+                        a.is_nullable() || b.is_nullable(),
+                    ))
+                })
+                .collect::<Option<Vec<_>>>()?;
+            Some(Struct(Fields::from(fields)))
+        }
+        // The entries are a struct of the key and the value. Spark's 
`MapType` has no field
+        // names, so the key and the value are matched by position.
+        (Map(lf, sorted), Map(rf, _)) => {
+            let (Struct(lkv), Struct(rkv)) = (lf.data_type(), rf.data_type()) 
else {
+                return None;
+            };
+            let ([lk, lv], [rk, rv]) = (&lkv[..], &rkv[..]) else {
+                return None;
+            };
+            let key = find_type(lk.data_type(), rk.data_type())?;

Review Comment:
   Spark's `findTypeForComplex` drops a widened map key type when 
[`Cast.forceNullable`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L427-L448)
 says the cast to it can return NULL 
([`TypeCoercionHelper.scala`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala#L149-L163)).
 Of the widenings `tightest_common_type` does, only DATE to TIMESTAMP_NTZ hits 
that ([`case (DateType, _) => 
true`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L440-L442)).
 So Spark finds no common type for `MAP<DATE, INT>` and `MAP<TIMESTAMP_NTZ, 
INT>` in either ANSI mode, while this returns `MAP<TIMESTAMP_NTZ, INT>`. Values 
aren't filtered, so `MAP<INT, DATE>` and `MAP<IN
 T, TIMESTAMP_NTZ>` do widen in Spark.
   
   This changes `array_position` from `main`, where the `Map` arm [required 
identical key and value 
types](https://github.com/apache/datafusion-spark/blob/48326999dfc8b631085a118970ceaab94e42fd23/crates/spark/src/function/array/array_position.rs#L249-L273).
 In Spark 4.2.0, in both ANSI modes:
   
   ```
   > SELECT array_position(array(map(DATE '2020-01-01', 1)), 
map(CAST('2020-01-01' AS TIMESTAMP_NTZ), 1));
   [DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES] ... but it's ["ARRAY<MAP<DATE, 
INT>>", "MAP<TIMESTAMP_NTZ, INT>"]. SQLSTATE: 42K09
   ```
   
   With this PR, the same query (with `arrow_cast(..., 'Timestamp(Microsecond, 
None)')` for the TIMESTAMP_NTZ) fails with `INVALID_ORDERING_TYPE` on 
`"MAP<TIMESTAMP_NTZ, BIGINT>"`. `array_insert` in #70 doesn't check 
orderability, so it would accept arguments that Spark's `array_insert` rejects 
with `ARRAY_FUNCTION_DIFF_TYPES`.
   
   Could we filter the key the way Spark does? With a helper next to 
`is_orderable`:
   
   ```rust
   /// Spark's `Cast.forceNullable`, for the casts that these rules produce. Of 
those, Spark
   /// marks only DATE to TIMESTAMP_NTZ as one that can return NULL.
   fn force_nullable(from: &DataType, to: &DataType) -> bool {
       matches!((from, to), (DataType::Date32, DataType::Timestamp(_, None)))
   }
   ```
   
   ```suggestion
               // Spark rejects a key type that the cast can make NULL 
(`Cast.forceNullable`).
               let key = find_type(lk.data_type(), rk.data_type()).filter(|key| 
{
                   !force_nullable(lk.data_type(), key)
                       && !force_nullable(rk.data_type(), key)
               })?;
   ```
   
   A unit test for both orders of the two maps, and an `.slt` record for the 
`array_position` query above, would cover it.



##########
crates/spark/src/function/type_coercion.rs:
##########
@@ -0,0 +1,427 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Spark's rules for finding a common type of several arguments, from its 
`TypeCoercion`
+//! and `AnsiTypeCoercion`, on the Arrow types that Spark's types map to. Also 
the checks
+//! and type names that the errors for those arguments use.
+//!
+//! These are for functions that need one type for several arguments, such as
+//! `array_position`. They are not Spark's implicit casts to a function's 
declared input
+//! types, which the functions do through their [`datafusion_expr::Signature`].
+
+use std::sync::Arc;
+
+use arrow::datatypes::{DataType, Field, Fields, IntervalUnit};
+
+/// Spark's `findTightestCommonType`. `ansi_mode` picks between 
`AnsiTypeCoercion` and
+/// `TypeCoercion`, which differ only in that ANSI widens an integer and a 
FLOAT to DOUBLE.
+/// Arrays, maps and structs are widened element by element with this function.
+pub(crate) fn tightest_common_type(
+    left: &DataType,
+    right: &DataType,
+    ansi_mode: bool,
+) -> Option<DataType> {
+    use DataType::*;
+    if left == right {
+        return Some(left.clone());
+    }
+    // Spark has no dictionary types. A dictionary-encoded argument, such as a
+    // Parquet string column, has the type of its values.
+    if let Dictionary(_, value) = left {
+        return tightest_common_type(value, right, ansi_mode);
+    }
+    if let Dictionary(_, value) = right {
+        return tightest_common_type(left, value, ansi_mode);
+    }
+    if let (Some(l), Some(r)) = (numeric_precedence(left), 
numeric_precedence(right)) {
+        let wider = if l >= r { left } else { right };
+        // With ANSI mode on, an integer and a FLOAT widen to DOUBLE, which 
holds every INT
+        // exactly.
+        return Some(if ansi_mode && *wider == Float32 {
+            Float64
+        } else {
+            wider.clone()
+        });
+    }
+    match (left, right) {
+        (Null, other) | (other, Null) => Some(other.clone()),
+        // Spark has one string type and one binary type.
+        (l, r) if l.is_string() && r.is_string() => 
Some(wider_physical_type(l, r)),
+        (l, r) if is_binary(l) && is_binary(r) => Some(wider_physical_type(l, 
r)),
+        // The decimal must hold every value of the integral type.
+        (int, Decimal128(p, s)) | (Decimal128(p, s), int) => {
+            let digits = integral_digits(int)?;
+            (*p as i16 - *s as i16 >= digits).then_some(Decimal128(*p, *s))
+        }
+        (Date32, ts @ Timestamp(_, _)) | (ts @ Timestamp(_, _), Date32) => {
+            Some(ts.clone())
+        }
+        // TIMESTAMP_NTZ and TIMESTAMP widen to TIMESTAMP.
+        (Timestamp(lu, ltz), Timestamp(ru, rtz)) => Some(Timestamp(
+            // Arrow orders the units from Second to Nanosecond.
+            (*lu).max(*ru),
+            ltz.clone().or_else(|| rtz.clone()),
+        )),
+        _ => complex_type(left, right, &|l, r| tightest_common_type(l, r, 
ansi_mode)),
+    }
+}
+
+/// Spark's `findTypeForComplex`: widens the elements of arrays, the keys and 
values of maps,
+/// and the fields of structs with `find_type`. A nested field is nullable if 
it is on either
+/// side. Struct fields must have the same names in the same order. Spark 
compares them
+/// ignoring case, but DataFusion cannot cast between structs whose field 
names differ only
+/// in case, so this requires an exact match.
+fn complex_type(
+    left: &DataType,
+    right: &DataType,
+    find_type: &dyn Fn(&DataType, &DataType) -> Option<DataType>,
+) -> Option<DataType> {
+    use DataType::*;
+    match (left, right) {
+        (
+            List(lf) | LargeList(lf) | FixedSizeList(lf, _),
+            List(rf) | LargeList(rf) | FixedSizeList(rf, _),
+        ) => {
+            let item = find_type(lf.data_type(), rf.data_type())?;
+            let field = Arc::new(Field::new(
+                lf.name(),
+                item,
+                lf.is_nullable() || rf.is_nullable(),
+            ));
+            Some(match (left, right) {
+                (FixedSizeList(_, ln), FixedSizeList(_, rn)) if ln == rn => {
+                    FixedSizeList(field, *ln)
+                }
+                (LargeList(_), _) | (_, LargeList(_)) => LargeList(field),
+                _ => List(field),
+            })
+        }
+        (Struct(lf), Struct(rf)) if lf.len() == rf.len() => {
+            let fields = lf
+                .iter()
+                .zip(rf.iter())
+                .map(|(a, b)| {
+                    if a.name() != b.name() {
+                        return None;
+                    }
+                    let data_type = find_type(a.data_type(), b.data_type())?;
+                    Some(Field::new(
+                        a.name(),
+                        data_type,
+                        a.is_nullable() || b.is_nullable(),
+                    ))
+                })
+                .collect::<Option<Vec<_>>>()?;
+            Some(Struct(Fields::from(fields)))
+        }
+        // The entries are a struct of the key and the value. Spark's 
`MapType` has no field
+        // names, so the key and the value are matched by position.
+        (Map(lf, sorted), Map(rf, _)) => {
+            let (Struct(lkv), Struct(rkv)) = (lf.data_type(), rf.data_type()) 
else {
+                return None;
+            };
+            let ([lk, lv], [rk, rv]) = (&lkv[..], &rkv[..]) else {
+                return None;
+            };
+            let key = find_type(lk.data_type(), rk.data_type())?;
+            let value = find_type(lv.data_type(), rv.data_type())?;
+            let entries = Struct(Fields::from(vec![
+                Field::new(lk.name(), key, false),
+                Field::new(lv.name(), value, lv.is_nullable() || 
rv.is_nullable()),
+            ]));
+            Some(Map(
+                Arc::new(Field::new(lf.name(), entries, false)),
+                *sorted,
+            ))
+        }
+        _ => None,
+    }
+}
+
+/// Spark's `OrderUtils.isOrderable`. Spark cannot order maps or 
`CalendarIntervalType`, which
+/// is `Interval(MonthDayNano)` in Arrow, at any depth.
+pub(crate) fn is_orderable(data_type: &DataType) -> bool {
+    match data_type {
+        DataType::Map(_, _)
+        | DataType::Union(_, _)
+        | DataType::Interval(IntervalUnit::MonthDayNano) => false,
+        DataType::List(field)
+        | DataType::LargeList(field)
+        | DataType::ListView(field)
+        | DataType::LargeListView(field)
+        | DataType::FixedSizeList(field, _) => is_orderable(field.data_type()),
+        DataType::Struct(fields) => fields.iter().all(|f| 
is_orderable(f.data_type())),
+        DataType::Dictionary(_, value) => is_orderable(value),
+        _ => true,
+    }
+}
+
+/// Whether the structs in two types, at any depth, have the same field types. 
With ANSI mode
+/// off, Spark's implicit cast cannot change the field types of a struct, so a 
function that
+/// casts its arguments to their tightest common type rejects such structs, as
+/// `array_position` does. With ANSI mode on, Spark widens the field types.
+pub(crate) fn struct_field_types_match(left: &DataType, right: &DataType) -> 
bool {
+    use DataType::*;
+    match (left, right) {
+        (Dictionary(_, l), r) | (r, Dictionary(_, l)) => 
struct_field_types_match(l, r),
+        (
+            List(l) | LargeList(l) | FixedSizeList(l, _),
+            List(r) | LargeList(r) | FixedSizeList(r, _),
+        ) => struct_field_types_match(l.data_type(), r.data_type()),
+        // The entries are a struct of the key and the value, which are 
checked on their own.
+        (Map(l, _), Map(r, _)) => match (l.data_type(), r.data_type()) {
+            (Struct(l), Struct(r)) => l
+                .iter()
+                .zip(r.iter())
+                .all(|(a, b)| struct_field_types_match(a.data_type(), 
b.data_type())),
+            _ => true,
+        },
+        (Struct(_), Struct(_)) => same_type(left, right),
+        _ => true,
+    }
+}
+
+/// Whether two types are equal when the nullability of nested fields is 
ignored, like
+/// Spark's `DataTypeUtils.sameType`.
+fn same_type(left: &DataType, right: &DataType) -> bool {
+    use DataType::*;
+    match (left, right) {
+        (Dictionary(_, l), r) | (r, Dictionary(_, l)) => same_type(l, r),
+        (List(l), List(r)) | (LargeList(l), LargeList(r)) | (Map(l, _), Map(r, 
_)) => {
+            same_type(l.data_type(), r.data_type())
+        }
+        (FixedSizeList(l, ln), FixedSizeList(r, rn)) => {
+            ln == rn && same_type(l.data_type(), r.data_type())
+        }
+        (Struct(l), Struct(r)) => {
+            l.len() == r.len()
+                && l.iter().zip(r.iter()).all(|(a, b)| {
+                    a.name() == b.name() && same_type(a.data_type(), 
b.data_type())
+                })
+        }
+        _ => left == right,
+    }
+}
+
+/// The name Spark's `toSQLType` gives the Spark type that an Arrow type maps 
to, for error
+/// messages. Arrow types without a Spark counterpart use their Arrow name.
+pub(crate) fn spark_sql_type(data_type: &DataType) -> String {
+    use DataType::*;
+    match data_type {
+        Null => "VOID".to_string(),
+        Boolean => "BOOLEAN".to_string(),
+        Int8 => "TINYINT".to_string(),
+        Int16 => "SMALLINT".to_string(),
+        Int32 => "INT".to_string(),
+        Int64 => "BIGINT".to_string(),
+        Float32 => "FLOAT".to_string(),
+        Float64 => "DOUBLE".to_string(),
+        Decimal128(p, s) => format!("DECIMAL({p},{s})"),
+        Utf8 | LargeUtf8 | Utf8View => "STRING".to_string(),
+        Binary | LargeBinary | BinaryView => "BINARY".to_string(),
+        Date32 => "DATE".to_string(),
+        Timestamp(_, Some(_)) => "TIMESTAMP".to_string(),
+        Timestamp(_, None) => "TIMESTAMP_NTZ".to_string(),
+        Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(),
+        List(field) | LargeList(field) | FixedSizeList(field, _) => {
+            format!("ARRAY<{}>", spark_sql_type(field.data_type()))
+        }
+        Struct(fields) => format!(
+            "STRUCT<{}>",
+            fields
+                .iter()
+                .map(|f| format!(
+                    "{}: {}{}",
+                    f.name(),
+                    spark_sql_type(f.data_type()),
+                    if f.is_nullable() { "" } else { " NOT NULL" }

Review Comment:
   `spark_sql_type` prints Arrow names for three Spark types that Comet [maps 
to 
Arrow](https://github.com/apache/datafusion-comet/blob/0fec45f232eac64aea65971c97956a64811af1dd/native/core/src/execution/serde.rs#L100-L105)
 and this crate produces (`make_dt_interval` returns `Duration(Microsecond)`, 
`make_time` returns `Time64(Nanosecond)`). It also doesn't quote struct field 
names the way 
[`StructField.sql`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala#L300-L301)
 does with 
[`quoteIfNeeded`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/QuotingUtils.scala#L36-L54).
 All four reach `array_position`'s error text:
   
   | Query | This PR | Spark 4.2.0 |
   |---|---|---|
   | `array_position(array(make_dt_interval(1)), 'x')` | 
`"ARRAY<Duration(µs)>"` | `"ARRAY<INTERVAL DAY TO SECOND>"` |
   | `array_position(array(make_ym_interval(1)), 'x')` | 
`"ARRAY<Interval(YearMonth)>"` | `"ARRAY<INTERVAL YEAR TO MONTH>"` |
   | `array_position(array(make_time(1, 2, 3)), 'x')` | `"ARRAY<Time64(ns)>"` | 
`"ARRAY<TIME(6)>"` (with the TIME type enabled) |
   | `array_position(array(named_struct('a b', 1)), 'x')` | `"ARRAY<STRUCT<a b: 
BIGINT>>"` | ``"ARRAY<STRUCT<`a b`: INT NOT NULL>>"`` |
   
   The last row also differs in nullability and the literal's type, which this 
PR doesn't control. The first column for `make_ym_interval` used `arrow_cast(1, 
'Interval(YearMonth)')`, since the crate has no `make_ym_interval`. Could we 
add these arms, and a helper for the quoting?
   
   ```rust
   /// Spark's `QuotingUtils.quoteIfNeeded`.
   fn quote_if_needed(name: &str) -> String {
       let mut chars = name.chars();
       let plain = chars
           .next()
           .is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
           && chars.all(|c| c.is_ascii_alphanumeric() || c == '_');
       if plain {
           name.to_string()
       } else {
           format!("`{}`", name.replace('`', "``"))
       }
   }
   ```
   
   ```suggestion
           Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(),
           Interval(IntervalUnit::YearMonth) => "INTERVAL YEAR TO 
MONTH".to_string(),
           Duration(_) => "INTERVAL DAY TO SECOND".to_string(),
           // Arrow has no TIME precision. Spark's `make_time` returns TIME(6).
           Time64(_) => "TIME(6)".to_string(),
           List(field) | LargeList(field) | FixedSizeList(field, _) => {
               format!("ARRAY<{}>", spark_sql_type(field.data_type()))
           }
           Struct(fields) => format!(
               "STRUCT<{}>",
               fields
                   .iter()
                   .map(|f| format!(
                       "{}: {}{}",
                       quote_if_needed(f.name()),
                       spark_sql_type(f.data_type()),
                       if f.is_nullable() { "" } else { " NOT NULL" }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to