This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new 7d72e66  [SPARK-58490] Support collection functions in 
`CollectionFunctions`
7d72e66 is described below

commit 7d72e669bea89f2ef87b5eaac1ae5b80802a499c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 31 18:26:18 2026 -0700

    [SPARK-58490] Support collection functions in `CollectionFunctions`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to add collection functions (array, map, struct, and generator 
functions) in a new file, `CollectionFunctions.swift`. The function names and 
signatures follow Scala's `org.apache.spark.sql.functions`.
    
    | Category | Functions |
    | --- | --- |
    | Array | `array`, `array_append`, `array_compact`, `array_contains`, 
`array_distinct`, `array_except`, `array_insert`, `array_intersect`, 
`array_join`, `array_max`, `array_min`, `array_position`, `array_prepend`, 
`array_remove`, `array_repeat`, `array_size`, `array_sort`, `arrays_overlap`, 
`arrays_zip`, `flatten`, `get`, `sequence`, `shuffle`, `slice`, `sort_array` |
    | Map | `map`, `map_concat`, `map_contains_key`, `map_entries`, 
`map_from_arrays`, `map_from_entries`, `map_keys`, `map_values`, `str_to_map` |
    | Struct | `named_struct`, `struct` (declared as `` `struct` `` because it 
is a Swift keyword) |
    | Generator | `explode`, `explode_outer`, `inline`, `inline_outer`, 
`posexplode`, `posexplode_outer`, `stack` |
    | Common collection | `cardinality`, `concat`, `element_at`, `reverse`, 
`size`, `try_element_at` |
    
    Following the existing `SparkLiteral` convention, functions whose Scala 
signatures accept `Any` (`array_append`, `array_contains`, `array_position`, 
`array_prepend`, `array_remove`, `element_at`, `map_contains_key`) are provided 
as both `Column` and `some SparkLiteral` overloads. Scala's `Int` parameters 
(`slice`, `array_repeat`) use Swift `Int32`.
    
    The following functions are excluded intentionally.
    
    | Functions | Reason |
    | --- | --- |
    | `transform`, `filter`, `exists`, `forall`, `aggregate`, `reduce`, 
`zip_with`, `map_zip_with`, `map_filter`, `transform_keys`, `transform_values`, 
`array_sort(e, comparator)` | Requires lambda expression support 
(`LambdaFunction` / `UnresolvedNamedLambdaVariable` proto), which this client 
does not have yet |
    | `array(colName: String, ...)`, `struct(colName: String, ...)` | 
String-based convenience overloads; only Column-based signatures are provided |
    
    ### Why are the changes needed?
    
    To improve the usability of the column-based `DataFrame` API by providing 
collection functions in the same shape as Scala/PySpark, so users can write 
queries without falling back to `selectExpr` or raw SQL strings.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, this adds new public functions. There is no behavior change to 
existing APIs.
    
    ### How was this patch tested?
    
    Pass the CIs with a newly added test suite, `CollectionFunctionsTests`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Fable 5
    
    Closes #463 from dongjoon-hyun/dongjoon/bold-cray-8ce033.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 Sources/SparkConnect/CollectionFunctions.swift     | 598 +++++++++++++++++++++
 .../CollectionFunctionsTests.swift                 | 272 ++++++++++
 2 files changed, 870 insertions(+)

diff --git a/Sources/SparkConnect/CollectionFunctions.swift 
b/Sources/SparkConnect/CollectionFunctions.swift
new file mode 100644
index 0000000..2dd1382
--- /dev/null
+++ b/Sources/SparkConnect/CollectionFunctions.swift
@@ -0,0 +1,598 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//  http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+// MARK: - Collection functions
+
+/// Creates a new array column. The input columns must all have the same data 
type.
+/// - Parameter cols: ``Column``s to be combined into an array.
+/// - Returns: A ``Column``.
+public func array(_ cols: Column...) -> Column {
+  return fn("array", cols)
+}
+
+/// Appends the element to the end of the array.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - element: An element ``Column`` to be appended.
+/// - Returns: A ``Column``.
+public func array_append(_ column: Column, _ element: Column) -> Column {
+  return fn("array_append", column, element)
+}
+
+/// Appends the element to the end of the array.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - element: A literal element to be appended.
+/// - Returns: A ``Column``.
+public func array_append(_ column: Column, _ element: some SparkLiteral) -> 
Column {
+  return array_append(column, element.toLiteralColumn)
+}
+
+/// Removes null values from the array.
+/// - Parameter column: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_compact(_ column: Column) -> Column {
+  return fn("array_compact", column)
+}
+
+/// Returns null if the array is null, true if the array contains `value`, and 
false otherwise.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - value: A value ``Column`` to check.
+/// - Returns: A ``Column``.
+public func array_contains(_ column: Column, _ value: Column) -> Column {
+  return fn("array_contains", column, value)
+}
+
+/// Returns null if the array is null, true if the array contains `value`, and 
false otherwise.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - value: A literal value to check.
+/// - Returns: A ``Column``.
+public func array_contains(_ column: Column, _ value: some SparkLiteral) -> 
Column {
+  return array_contains(column, value.toLiteralColumn)
+}
+
+/// Removes duplicate values from the array.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_distinct(_ col: Column) -> Column {
+  return fn("array_distinct", col)
+}
+
+/// Returns an array of the elements in the first array but not in the second 
array, without
+/// duplicates. The order of elements in the result is not determined.
+/// - Parameters:
+///   - col1: An array ``Column``.
+///   - col2: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_except(_ col1: Column, _ col2: Column) -> Column {
+  return fn("array_except", col1, col2)
+}
+
+/// Adds an item into a given array at a specified position.
+/// - Parameters:
+///   - arr: An array ``Column``.
+///   - pos: A position ``Column``.
+///   - value: A value ``Column`` to be inserted.
+/// - Returns: A ``Column``.
+public func array_insert(_ arr: Column, _ pos: Column, _ value: Column) -> 
Column {
+  return fn("array_insert", arr, pos, value)
+}
+
+/// Returns an array of the elements in the intersection of the given two 
arrays, without
+/// duplicates.
+/// - Parameters:
+///   - col1: An array ``Column``.
+///   - col2: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_intersect(_ col1: Column, _ col2: Column) -> Column {
+  return fn("array_intersect", col1, col2)
+}
+
+/// Concatenates the elements of the array column using the delimiter.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - delimiter: A delimiter string.
+/// - Returns: A ``Column``.
+public func array_join(_ column: Column, _ delimiter: String) -> Column {
+  return fn("array_join", column, lit(delimiter))
+}
+
+/// Concatenates the elements of the array column using the delimiter.
+/// Null values are replaced with `nullReplacement`.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - delimiter: A delimiter string.
+///   - nullReplacement: A string to replace null values.
+/// - Returns: A ``Column``.
+public func array_join(_ column: Column, _ delimiter: String, _ 
nullReplacement: String) -> Column
+{
+  return fn("array_join", column, lit(delimiter), lit(nullReplacement))
+}
+
+/// Returns the maximum value in the array. NaN is greater than any non-NaN 
elements for
+/// double/float type. Null elements will be ignored.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_max(_ col: Column) -> Column {
+  return fn("array_max", col)
+}
+
+/// Returns the minimum value in the array. NaN is greater than any non-NaN 
elements for
+/// double/float type. Null elements will be ignored.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_min(_ col: Column) -> Column {
+  return fn("array_min", col)
+}
+
+/// Locates the position of the first occurrence of the value in the given 
array as long.
+/// Returns null if either of the arguments are null.
+/// The position is not zero based, but 1 based index. Returns 0 if the value 
could not be
+/// found in the array.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - value: A value ``Column`` to locate.
+/// - Returns: A ``Column``.
+public func array_position(_ column: Column, _ value: Column) -> Column {
+  return fn("array_position", column, value)
+}
+
+/// Locates the position of the first occurrence of the value in the given 
array as long.
+/// Returns null if either of the arguments are null.
+/// The position is not zero based, but 1 based index. Returns 0 if the value 
could not be
+/// found in the array.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - value: A literal value to locate.
+/// - Returns: A ``Column``.
+public func array_position(_ column: Column, _ value: some SparkLiteral) -> 
Column {
+  return array_position(column, value.toLiteralColumn)
+}
+
+/// Prepends the element to the beginning of the array.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - element: An element ``Column`` to be prepended.
+/// - Returns: A ``Column``.
+public func array_prepend(_ column: Column, _ element: Column) -> Column {
+  return fn("array_prepend", column, element)
+}
+
+/// Prepends the element to the beginning of the array.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - element: A literal element to be prepended.
+/// - Returns: A ``Column``.
+public func array_prepend(_ column: Column, _ element: some SparkLiteral) -> 
Column {
+  return array_prepend(column, element.toLiteralColumn)
+}
+
+/// Removes all elements that equal to the element from the given array.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - element: An element ``Column`` to be removed.
+/// - Returns: A ``Column``.
+public func array_remove(_ column: Column, _ element: Column) -> Column {
+  return fn("array_remove", column, element)
+}
+
+/// Removes all elements that equal to the element from the given array.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - element: A literal element to be removed.
+/// - Returns: A ``Column``.
+public func array_remove(_ column: Column, _ element: some SparkLiteral) -> 
Column {
+  return array_remove(column, element.toLiteralColumn)
+}
+
+/// Creates an array containing the left argument repeated the number of times 
given by the
+/// right argument.
+/// - Parameters:
+///   - left: A value ``Column`` to be repeated.
+///   - right: A ``Column`` for the number of times.
+/// - Returns: A ``Column``.
+public func array_repeat(_ left: Column, _ right: Column) -> Column {
+  return fn("array_repeat", left, right)
+}
+
+/// Creates an array containing the left argument repeated the number of times 
given by the
+/// right argument.
+/// - Parameters:
+///   - col: A value ``Column`` to be repeated.
+///   - count: The number of times.
+/// - Returns: A ``Column``.
+public func array_repeat(_ col: Column, _ count: Int32) -> Column {
+  return array_repeat(col, lit(count))
+}
+
+/// Returns the total number of elements in the array. The function returns 
null for null input.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_size(_ col: Column) -> Column {
+  return fn("array_size", col)
+}
+
+/// Sorts the input array in ascending order. The elements of the input array 
must be orderable.
+/// NaN is greater than any non-NaN elements for double/float type. Null 
elements will be placed
+/// at the end of the returned array.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_sort(_ col: Column) -> Column {
+  return fn("array_sort", col)
+}
+
+/// Returns an array of the elements in the union of the given two arrays, 
without duplicates.
+/// - Parameters:
+///   - col1: An array ``Column``.
+///   - col2: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_union(_ col1: Column, _ col2: Column) -> Column {
+  return fn("array_union", col1, col2)
+}
+
+/// Returns true if `a1` and `a2` have at least one non-null element in 
common. If not and both
+/// the arrays are non-empty and any of them contains a null, it returns null. 
It returns false
+/// otherwise.
+/// - Parameters:
+///   - a1: An array ``Column``.
+///   - a2: An array ``Column``.
+/// - Returns: A ``Column``.
+public func arrays_overlap(_ a1: Column, _ a2: Column) -> Column {
+  return fn("arrays_overlap", a1, a2)
+}
+
+/// Returns a merged array of structs in which the N-th struct contains all 
N-th values of input
+/// arrays.
+/// - Parameter cols: Array ``Column``s to be merged.
+/// - Returns: A ``Column``.
+public func arrays_zip(_ cols: Column...) -> Column {
+  return fn("arrays_zip", cols)
+}
+
+/// Returns the length of the array or map. This is an alias of ``size(_:)``.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func cardinality(_ col: Column) -> Column {
+  return fn("cardinality", col)
+}
+
+/// Concatenates multiple input columns together into a single column. The 
function works with
+/// strings, numeric, binary and compatible array columns.
+/// - Parameter exprs: ``Column``s to be concatenated.
+/// - Returns: A ``Column``.
+public func concat(_ exprs: Column...) -> Column {
+  return fn("concat", exprs)
+}
+
+/// Returns element of array at given index in `value` if `column` is array. 
Returns value for
+/// the given key in `value` if `column` is map.
+/// - Parameters:
+///   - column: An array or map ``Column``.
+///   - value: An index or key ``Column``.
+/// - Returns: A ``Column``.
+public func element_at(_ column: Column, _ value: Column) -> Column {
+  return fn("element_at", column, value)
+}
+
+/// Returns element of array at given index in `value` if `column` is array. 
Returns value for
+/// the given key in `value` if `column` is map.
+/// - Parameters:
+///   - column: An array or map ``Column``.
+///   - value: A literal index or key.
+/// - Returns: A ``Column``.
+public func element_at(_ column: Column, _ value: some SparkLiteral) -> Column 
{
+  return element_at(column, value.toLiteralColumn)
+}
+
+/// Creates a new row for each element in the given array or map column. Uses 
the default column
+/// name `col` for elements in the array and `key` and `value` for elements in 
the map unless
+/// specified otherwise.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func explode(_ col: Column) -> Column {
+  return fn("explode", col)
+}
+
+/// Creates a new row for each element in the given array or map column. Uses 
the default column
+/// name `col` for elements in the array and `key` and `value` for elements in 
the map unless
+/// specified otherwise. Unlike ``explode(_:)``, if the array/map is null or 
empty then null is
+/// produced.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func explode_outer(_ col: Column) -> Column {
+  return fn("explode_outer", col)
+}
+
+/// Creates a single array from an array of arrays. If a structure of nested 
arrays is deeper
+/// than two levels, only one level of nesting is removed.
+/// - Parameter col: An array of arrays ``Column``.
+/// - Returns: A ``Column``.
+public func flatten(_ col: Column) -> Column {
+  return fn("flatten", col)
+}
+
+/// Returns element of array at given (0-based) index. If the index points 
outside of the array
+/// boundaries, then this function returns NULL.
+/// - Parameters:
+///   - column: An array ``Column``.
+///   - index: An index ``Column``.
+/// - Returns: A ``Column``.
+public func get(_ column: Column, _ index: Column) -> Column {
+  return fn("get", column, index)
+}
+
+/// Creates a new row for each element in the given array of structs.
+/// - Parameter col: An array of structs ``Column``.
+/// - Returns: A ``Column``.
+public func inline(_ col: Column) -> Column {
+  return fn("inline", col)
+}
+
+/// Creates a new row for each element in the given array of structs. Unlike 
``inline(_:)``,
+/// if the array is null or empty then null is produced for each nested column.
+/// - Parameter col: An array of structs ``Column``.
+/// - Returns: A ``Column``.
+public func inline_outer(_ col: Column) -> Column {
+  return fn("inline_outer", col)
+}
+
+/// Creates a new map column. The input columns must be grouped as key-value 
pairs, e.g.
+/// (key1, value1, key2, value2, ...). The key columns must all have the same 
data type, and
+/// can't be null. The value columns must all have the same data type.
+/// - Parameter cols: ``Column``s grouped as key-value pairs.
+/// - Returns: A ``Column``.
+public func map(_ cols: Column...) -> Column {
+  return fn("map", cols)
+}
+
+/// Returns the union of all the given maps.
+/// - Parameter cols: Map ``Column``s to be unioned.
+/// - Returns: A ``Column``.
+public func map_concat(_ cols: Column...) -> Column {
+  return fn("map_concat", cols)
+}
+
+/// Returns true if the map contains the key.
+/// - Parameters:
+///   - column: A map ``Column``.
+///   - key: A key ``Column`` to check.
+/// - Returns: A ``Column``.
+public func map_contains_key(_ column: Column, _ key: Column) -> Column {
+  return fn("map_contains_key", column, key)
+}
+
+/// Returns true if the map contains the key.
+/// - Parameters:
+///   - column: A map ``Column``.
+///   - key: A literal key to check.
+/// - Returns: A ``Column``.
+public func map_contains_key(_ column: Column, _ key: some SparkLiteral) -> 
Column {
+  return map_contains_key(column, key.toLiteralColumn)
+}
+
+/// Returns an unordered array of all entries in the given map.
+/// - Parameter col: A map ``Column``.
+/// - Returns: A ``Column``.
+public func map_entries(_ col: Column) -> Column {
+  return fn("map_entries", col)
+}
+
+/// Creates a new map column. The array in the first column is used for keys. 
The array in the
+/// second column is used for values. All elements in the array for key should 
not be null.
+/// - Parameters:
+///   - keys: An array ``Column`` for keys.
+///   - values: An array ``Column`` for values.
+/// - Returns: A ``Column``.
+public func map_from_arrays(_ keys: Column, _ values: Column) -> Column {
+  return fn("map_from_arrays", keys, values)
+}
+
+/// Returns a map created from the given array of entries.
+/// - Parameter col: An array of structs ``Column``.
+/// - Returns: A ``Column``.
+public func map_from_entries(_ col: Column) -> Column {
+  return fn("map_from_entries", col)
+}
+
+/// Returns an unordered array containing the keys of the map.
+/// - Parameter col: A map ``Column``.
+/// - Returns: A ``Column``.
+public func map_keys(_ col: Column) -> Column {
+  return fn("map_keys", col)
+}
+
+/// Returns an unordered array containing the values of the map.
+/// - Parameter col: A map ``Column``.
+/// - Returns: A ``Column``.
+public func map_values(_ col: Column) -> Column {
+  return fn("map_values", col)
+}
+
+/// Creates a struct with the given field names and values.
+/// - Parameter cols: ``Column``s grouped as name-value pairs, e.g.
+///   (name1, val1, name2, val2, ...).
+/// - Returns: A ``Column``.
+public func named_struct(_ cols: Column...) -> Column {
+  return fn("named_struct", cols)
+}
+
+/// Creates a new row for each element with position in the given array or map 
column. Uses the
+/// default column name `pos` for position, and `col` for elements in the 
array and `key` and
+/// `value` for elements in the map unless specified otherwise.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func posexplode(_ col: Column) -> Column {
+  return fn("posexplode", col)
+}
+
+/// Creates a new row for each element with position in the given array or map 
column. Uses the
+/// default column name `pos` for position, and `col` for elements in the 
array and `key` and
+/// `value` for elements in the map unless specified otherwise. Unlike 
``posexplode(_:)``, if
+/// the array/map is null or empty then the row (null, null) is produced.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func posexplode_outer(_ col: Column) -> Column {
+  return fn("posexplode_outer", col)
+}
+
+/// Returns a reversed string or an array with reverse order of elements.
+/// - Parameter col: A string or array ``Column``.
+/// - Returns: A ``Column``.
+public func reverse(_ col: Column) -> Column {
+  return fn("reverse", col)
+}
+
+/// Generates a sequence of integers from `start` to `stop`, incrementing by 1 
if `start` is
+/// less than or equal to `stop`, otherwise -1.
+/// - Parameters:
+///   - start: A start ``Column``.
+///   - stop: A stop ``Column``.
+/// - Returns: A ``Column``.
+public func sequence(_ start: Column, _ stop: Column) -> Column {
+  return fn("sequence", start, stop)
+}
+
+/// Generates a sequence of integers from `start` to `stop`, incrementing by 
`step`.
+/// - Parameters:
+///   - start: A start ``Column``.
+///   - stop: A stop ``Column``.
+///   - step: A step ``Column``.
+/// - Returns: A ``Column``.
+public func sequence(_ start: Column, _ stop: Column, _ step: Column) -> 
Column {
+  return fn("sequence", start, stop, step)
+}
+
+/// Returns a random permutation of the given array.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func shuffle(_ col: Column) -> Column {
+  return fn("shuffle", col)
+}
+
+/// Returns a random permutation of the given array with the given seed.
+/// - Parameters:
+///   - col: An array ``Column``.
+///   - seed: A seed ``Column``.
+/// - Returns: A ``Column``.
+public func shuffle(_ col: Column, _ seed: Column) -> Column {
+  return fn("shuffle", col, seed)
+}
+
+/// Returns the length of the array or map.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func size(_ col: Column) -> Column {
+  return fn("size", col)
+}
+
+/// Returns an array containing all the elements in `x` from index `start` (or 
starting from the
+/// end if `start` is negative) with the specified `length`.
+/// - Parameters:
+///   - x: An array ``Column``.
+///   - start: A start index ``Column`` (1-based).
+///   - length: A length ``Column``.
+/// - Returns: A ``Column``.
+public func slice(_ x: Column, _ start: Column, _ length: Column) -> Column {
+  return fn("slice", x, start, length)
+}
+
+/// Returns an array containing all the elements in `x` from index `start` (or 
starting from the
+/// end if `start` is negative) with the specified `length`.
+/// - Parameters:
+///   - x: An array ``Column``.
+///   - start: A start index (1-based).
+///   - length: A length.
+/// - Returns: A ``Column``.
+public func slice(_ x: Column, _ start: Int32, _ length: Int32) -> Column {
+  return slice(x, lit(start), lit(length))
+}
+
+/// Sorts the input array in ascending order according to the natural ordering 
of the array
+/// elements. Null elements will be placed at the beginning of the returned 
array.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func sort_array(_ col: Column) -> Column {
+  return sort_array(col, true)
+}
+
+/// Sorts the input array in ascending or descending order according to the 
natural ordering of
+/// the array elements. Null elements will be placed at the beginning of the 
returned array in
+/// ascending order or at the end of the returned array in descending order.
+/// - Parameters:
+///   - col: An array ``Column``.
+///   - asc: True for the ascending order.
+/// - Returns: A ``Column``.
+public func sort_array(_ col: Column, _ asc: Bool) -> Column {
+  return fn("sort_array", col, lit(asc))
+}
+
+/// Separates the columns into the specified number of rows.
+/// - Parameter cols: ``Column``s where the first is the number of rows, 
followed by values.
+/// - Returns: A ``Column``.
+public func stack(_ cols: Column...) -> Column {
+  return fn("stack", cols)
+}
+
+/// Creates a map after splitting the text into key/value pairs using 
delimiters. The default
+/// delimiters are `,` for `pairDelim` and `:` for `keyValueDelim`.
+/// - Parameter text: A text ``Column``.
+/// - Returns: A ``Column``.
+public func str_to_map(_ text: Column) -> Column {
+  return fn("str_to_map", text)
+}
+
+/// Creates a map after splitting the text into key/value pairs using 
delimiters. The default
+/// delimiter is `:` for `keyValueDelim`.
+/// - Parameters:
+///   - text: A text ``Column``.
+///   - pairDelim: A pair delimiter ``Column``.
+/// - Returns: A ``Column``.
+public func str_to_map(_ text: Column, _ pairDelim: Column) -> Column {
+  return fn("str_to_map", text, pairDelim)
+}
+
+/// Creates a map after splitting the text into key/value pairs using 
delimiters.
+/// - Parameters:
+///   - text: A text ``Column``.
+///   - pairDelim: A pair delimiter ``Column``.
+///   - keyValueDelim: A key/value delimiter ``Column``.
+/// - Returns: A ``Column``.
+public func str_to_map(_ text: Column, _ pairDelim: Column, _ keyValueDelim: 
Column) -> Column {
+  return fn("str_to_map", text, pairDelim, keyValueDelim)
+}
+
+/// Creates a new struct column.
+/// - Parameter cols: ``Column``s to be combined into a struct.
+/// - Returns: A ``Column``.
+public func `struct`(_ cols: Column...) -> Column {
+  return fn("struct", cols)
+}
+
+/// Returns element of array at given (1-based) index. If the index points 
outside of the array
+/// boundaries, then this function returns NULL. Returns value for the given 
key in map, or NULL
+/// if the key is not contained in the map.
+/// - Parameters:
+///   - column: An array or map ``Column``.
+///   - value: An index or key ``Column``.
+/// - Returns: A ``Column``.
+public func try_element_at(_ column: Column, _ value: Column) -> Column {
+  return fn("try_element_at", column, value)
+}
diff --git a/Tests/SparkConnectTests/CollectionFunctionsTests.swift 
b/Tests/SparkConnectTests/CollectionFunctionsTests.swift
new file mode 100644
index 0000000..357fe5b
--- /dev/null
+++ b/Tests/SparkConnectTests/CollectionFunctionsTests.swift
@@ -0,0 +1,272 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//  http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+import Testing
+
+@testable import SparkConnect
+
+/// A test suite for `CollectionFunctions`
+@Suite(.serialized)
+struct CollectionFunctionsTests {
+
+  @Test
+  func collectionFunctions() throws {
+    for (column, name) in [
+      (array_compact(col("a")), "array_compact"),
+      (array_distinct(col("a")), "array_distinct"),
+      (array_max(col("a")), "array_max"),
+      (array_min(col("a")), "array_min"),
+      (array_size(col("a")), "array_size"),
+      (array_sort(col("a")), "array_sort"),
+      (cardinality(col("a")), "cardinality"),
+      (explode(col("a")), "explode"),
+      (explode_outer(col("a")), "explode_outer"),
+      (flatten(col("a")), "flatten"),
+      (inline(col("a")), "inline"),
+      (inline_outer(col("a")), "inline_outer"),
+      (map_entries(col("a")), "map_entries"),
+      (map_from_entries(col("a")), "map_from_entries"),
+      (map_keys(col("a")), "map_keys"),
+      (map_values(col("a")), "map_values"),
+      (posexplode(col("a")), "posexplode"),
+      (posexplode_outer(col("a")), "posexplode_outer"),
+      (reverse(col("a")), "reverse"),
+      (shuffle(col("a")), "shuffle"),
+      (size(col("a")), "size"),
+      (str_to_map(col("a")), "str_to_map"),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
 == "a")
+    }
+  }
+
+  @Test
+  func collectionFunctionArguments() throws {
+    for (column, name) in [
+      (array_append(col("a"), col("b")), "array_append"),
+      (array_contains(col("a"), col("b")), "array_contains"),
+      (array_except(col("a"), col("b")), "array_except"),
+      (array_intersect(col("a"), col("b")), "array_intersect"),
+      (array_position(col("a"), col("b")), "array_position"),
+      (array_prepend(col("a"), col("b")), "array_prepend"),
+      (array_remove(col("a"), col("b")), "array_remove"),
+      (array_repeat(col("a"), col("b")), "array_repeat"),
+      (array_union(col("a"), col("b")), "array_union"),
+      (arrays_overlap(col("a"), col("b")), "arrays_overlap"),
+      (element_at(col("a"), col("b")), "element_at"),
+      (get(col("a"), col("b")), "get"),
+      (map_contains_key(col("a"), col("b")), "map_contains_key"),
+      (map_from_arrays(col("a"), col("b")), "map_from_arrays"),
+      (sequence(col("a"), col("b")), "sequence"),
+      (shuffle(col("a"), col("b")), "shuffle"),
+      (str_to_map(col("a"), col("b")), "str_to_map"),
+      (try_element_at(col("a"), col("b")), "try_element_at"),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      #expect(expr.unresolvedFunction.arguments.count == 2)
+    }
+
+    for (column, name) in [
+      (array_insert(col("a"), col("b"), col("c")), "array_insert"),
+      (sequence(col("a"), col("b"), col("c")), "sequence"),
+      (slice(col("a"), col("b"), col("c")), "slice"),
+      (str_to_map(col("a"), col("b"), col("c")), "str_to_map"),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      #expect(expr.unresolvedFunction.arguments.count == 3)
+    }
+  }
+
+  @Test
+  func collectionFunctionLiteralArguments() throws {
+    for (column, name) in [
+      (array_append(col("a"), "x"), "array_append"),
+      (array_contains(col("a"), "x"), "array_contains"),
+      (array_position(col("a"), "x"), "array_position"),
+      (array_prepend(col("a"), "x"), "array_prepend"),
+      (array_remove(col("a"), "x"), "array_remove"),
+      (element_at(col("a"), "x"), "element_at"),
+      (map_contains_key(col("a"), "x"), "map_contains_key"),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      #expect(expr.unresolvedFunction.arguments[1].literal.string == "x")
+    }
+
+    #expect(array_join(col("a"), 
",").expr.unresolvedFunction.arguments[1].literal.string == ",")
+    #expect(array_join(col("a"), ",", 
"*").expr.unresolvedFunction.arguments.count == 3)
+    #expect(array_repeat(col("a"), 
2).expr.unresolvedFunction.arguments[1].literal.integer == 2)
+    
#expect(sort_array(col("a")).expr.unresolvedFunction.arguments[1].literal.boolean
 == true)
+    #expect(sort_array(col("a"), 
false).expr.unresolvedFunction.arguments[1].literal.boolean == false)
+
+    let sliced = slice(col("a"), 2, 3).expr
+    #expect(sliced.unresolvedFunction.arguments[1].literal.integer == 2)
+    #expect(sliced.unresolvedFunction.arguments[2].literal.integer == 3)
+  }
+
+  @Test
+  func collectionFunctionVariadicArguments() throws {
+    for (column, name, count) in [
+      (array(col("a"), col("b")), "array", 2),
+      (arrays_zip(col("a"), col("b")), "arrays_zip", 2),
+      (concat(col("a"), col("b"), col("c")), "concat", 3),
+      (map(col("a"), col("b"), col("c"), col("d")), "map", 4),
+      (map_concat(col("a"), col("b")), "map_concat", 2),
+      (named_struct(col("a"), col("b"), col("c"), col("d")), "named_struct", 
4),
+      (stack(col("a"), col("b"), col("c")), "stack", 3),
+      (`struct`(col("a"), col("b")), "struct", 2),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      #expect(expr.unresolvedFunction.arguments.count == count)
+    }
+  }
+
+  @Test
+  func selectArrayFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.range(1)
+    let arr = array(lit(1), lit(2), lit(3))
+
+    let modified = try await df.select(
+      arr.cast("string"), array_append(arr, 4).cast("string"),
+      array_prepend(arr, 0).cast("string"), array_remove(arr, 
2).cast("string"),
+      array_insert(arr, lit(Int32(1)), lit(0)).cast("string")
+    ).collect()
+    #expect(modified == [Row("[1, 2, 3]", "[1, 2, 3, 4]", "[0, 1, 2, 3]", "[1, 
3]", "[0, 1, 2, 3]")])
+
+    let setOps = try await df.select(
+      array_distinct(array(lit(1), lit(1), lit(2))).cast("string"),
+      array_except(arr, array(lit(2))).cast("string"),
+      array_intersect(arr, array(lit(2), lit(4))).cast("string"),
+      array_union(array(lit(1)), array(lit(2))).cast("string")
+    ).collect()
+    #expect(setOps == [Row("[1, 2]", "[1, 3]", "[2]", "[1, 2]")])
+
+    let lookups = try await df.select(
+      array_contains(arr, 2), array_position(arr, 2), element_at(arr, 
Int32(1)),
+      try_element_at(arr, lit(Int32(1))), get(arr, lit(Int32(0)))
+    ).collect()
+    #expect(lookups == [Row(true, 2, 1, 1, 1)])
+
+    let ordered = try await df.select(
+      sort_array(array(lit(3), lit(1), lit(2))).cast("string"),
+      sort_array(arr, false).cast("string"), reverse(arr).cast("string"),
+      slice(arr, 2, 2).cast("string"), flatten(array(array(lit(1)), 
array(lit(2)))).cast("string")
+    ).collect()
+    #expect(ordered == [Row("[1, 2, 3]", "[3, 2, 1]", "[3, 2, 1]", "[2, 3]", 
"[1, 2]")])
+
+    let sizes = try await df.select(
+      size(arr), cardinality(arr), array_size(arr), array_min(arr), 
array_max(arr),
+      size(shuffle(arr)), size(shuffle(arr, lit(42)))
+    ).collect()
+    #expect(sizes == [Row(3, 3, 3, 1, 3, 3, 3)])
+
+    let combined = try await df.select(
+      array_repeat(lit(1), 2).cast("string"), sequence(lit(1), 
lit(3)).cast("string"),
+      sequence(lit(5), lit(1), lit(-2)).cast("string"),
+      arrays_zip(array(lit(1)), array(lit("a"))).cast("string"),
+      arrays_overlap(arr, array(lit(3), lit(9)))
+    ).collect()
+    #expect(combined == [Row("[1, 1]", "[1, 2, 3]", "[5, 3, 1]", "[{1, a}]", 
true)])
+
+    let dfWithNull = try await spark.sql("SELECT array(1, null, 3) AS a")
+    let compacted = try await dfWithNull.select(
+      array_compact(col("a")).cast("string"), array_join(col("a"), ","),
+      array_join(col("a"), ",", "*")
+    ).collect()
+    #expect(compacted == [Row("[1, 3]", "1,3", "1,*,3")])
+    await spark.stop()
+  }
+
+  @Test
+  func selectMapFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.range(1)
+    let m = map(lit("a"), lit(1), lit("b"), lit(2))
+
+    let rows = try await df.select(
+      m.cast("string"), map_keys(m).cast("string"), 
map_values(m).cast("string"),
+      map_entries(m).cast("string")
+    ).collect()
+    #expect(rows == [Row("{a -> 1, b -> 2}", "[a, b]", "[1, 2]", "[{a, 1}, {b, 
2}]")])
+
+    let lookups = try await df.select(
+      map_contains_key(m, "a"), element_at(m, "a"), try_element_at(m, lit("b"))
+    ).collect()
+    #expect(lookups == [Row(true, 1, 2)])
+
+    let created = try await df.select(
+      map_concat(map(lit("a"), lit(1)), map(lit("b"), lit(2))).cast("string"),
+      map_from_arrays(array(lit("a")), array(lit(1))).cast("string"),
+      map_from_entries(array(`struct`(lit("a"), lit(1)))).cast("string")
+    ).collect()
+    #expect(created == [Row("{a -> 1, b -> 2}", "{a -> 1}", "{a -> 1}")])
+
+    let parsed = try await df.select(
+      str_to_map(lit("a:1,b:2")).cast("string"),
+      str_to_map(lit("a:1;b:2"), lit(";")).cast("string"),
+      str_to_map(lit("a=1;b=2"), lit(";"), lit("=")).cast("string")
+    ).collect()
+    #expect(
+      parsed == [Row("{a -> 1, b -> 2}", "{a -> 1, b -> 2}", "{a -> 1, b -> 
2}")])
+    await spark.stop()
+  }
+
+  @Test
+  func selectStructAndConcatFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.range(1)
+    let rows = try await df.select(
+      `struct`(lit(1), lit("a")).cast("string"),
+      named_struct(lit("x"), lit(1), lit("y"), lit("a")).cast("string"),
+      concat(lit("a"), lit("b"), lit("c")),
+      concat(array(lit(1)), array(lit(2))).cast("string")
+    ).collect()
+    #expect(rows == [Row("{1, a}", "{1, a}", "abc", "[1, 2]")])
+    await spark.stop()
+  }
+
+  @Test
+  func selectGeneratorFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.range(1)
+    #expect(try await df.select(explode(array(lit(1), lit(2)))).collect() == 
[Row(1), Row(2)])
+    #expect(
+      try await df.select(explode_outer(array(lit(1), lit(2)))).collect() == 
[Row(1), Row(2)])
+    #expect(try await df.select(explode(map(lit("a"), lit(1)))).collect() == 
[Row("a", 1)])
+    #expect(
+      try await df.select(posexplode(array(lit(10), lit(20)))).collect()
+        == [Row(0, 10), Row(1, 20)])
+    #expect(
+      try await df.select(posexplode_outer(array(lit(10), lit(20)))).collect()
+        == [Row(0, 10), Row(1, 20)])
+
+    let structArray = array(named_struct(lit("a"), lit(1), lit("b"), lit(2)))
+    #expect(try await df.select(inline(structArray)).collect() == [Row(1, 2)])
+    #expect(try await df.select(inline_outer(structArray)).collect() == 
[Row(1, 2)])
+    #expect(
+      try await df.select(stack(lit(Int32(2)), lit(1), lit(2), lit(3), 
lit(4))).collect()
+        == [Row(1, 2), Row(3, 4)])
+    await spark.stop()
+  }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to