This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 7d72e66 [SPARK-58490] Support collection functions in
`CollectionFunctions`
7d72e66 is described below
commit 7d72e669bea89f2ef87b5eaac1ae5b80802a499c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 31 18:26:18 2026 -0700
[SPARK-58490] Support collection functions in `CollectionFunctions`
### What changes were proposed in this pull request?
This PR aims to add collection functions (array, map, struct, and generator
functions) in a new file, `CollectionFunctions.swift`. The function names and
signatures follow Scala's `org.apache.spark.sql.functions`.
| Category | Functions |
| --- | --- |
| Array | `array`, `array_append`, `array_compact`, `array_contains`,
`array_distinct`, `array_except`, `array_insert`, `array_intersect`,
`array_join`, `array_max`, `array_min`, `array_position`, `array_prepend`,
`array_remove`, `array_repeat`, `array_size`, `array_sort`, `arrays_overlap`,
`arrays_zip`, `flatten`, `get`, `sequence`, `shuffle`, `slice`, `sort_array` |
| Map | `map`, `map_concat`, `map_contains_key`, `map_entries`,
`map_from_arrays`, `map_from_entries`, `map_keys`, `map_values`, `str_to_map` |
| Struct | `named_struct`, `struct` (declared as `` `struct` `` because it
is a Swift keyword) |
| Generator | `explode`, `explode_outer`, `inline`, `inline_outer`,
`posexplode`, `posexplode_outer`, `stack` |
| Common collection | `cardinality`, `concat`, `element_at`, `reverse`,
`size`, `try_element_at` |
Following the existing `SparkLiteral` convention, functions whose Scala
signatures accept `Any` (`array_append`, `array_contains`, `array_position`,
`array_prepend`, `array_remove`, `element_at`, `map_contains_key`) are provided
as both `Column` and `some SparkLiteral` overloads. Scala's `Int` parameters
(`slice`, `array_repeat`) use Swift `Int32`.
The following functions are excluded intentionally.
| Functions | Reason |
| --- | --- |
| `transform`, `filter`, `exists`, `forall`, `aggregate`, `reduce`,
`zip_with`, `map_zip_with`, `map_filter`, `transform_keys`, `transform_values`,
`array_sort(e, comparator)` | Requires lambda expression support
(`LambdaFunction` / `UnresolvedNamedLambdaVariable` proto), which this client
does not have yet |
| `array(colName: String, ...)`, `struct(colName: String, ...)` |
String-based convenience overloads; only Column-based signatures are provided |
### Why are the changes needed?
To improve the usability of the column-based `DataFrame` API by providing
collection functions in the same shape as Scala/PySpark, so users can write
queries without falling back to `selectExpr` or raw SQL strings.
### Does this PR introduce _any_ user-facing change?
Yes, this adds new public functions. There is no behavior change to
existing APIs.
### How was this patch tested?
Pass the CIs with a newly added test suite, `CollectionFunctionsTests`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #463 from dongjoon-hyun/dongjoon/bold-cray-8ce033.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/CollectionFunctions.swift | 598 +++++++++++++++++++++
.../CollectionFunctionsTests.swift | 272 ++++++++++
2 files changed, 870 insertions(+)
diff --git a/Sources/SparkConnect/CollectionFunctions.swift
b/Sources/SparkConnect/CollectionFunctions.swift
new file mode 100644
index 0000000..2dd1382
--- /dev/null
+++ b/Sources/SparkConnect/CollectionFunctions.swift
@@ -0,0 +1,598 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+// MARK: - Collection functions
+
+/// Creates a new array column. The input columns must all have the same data
type.
+/// - Parameter cols: ``Column``s to be combined into an array.
+/// - Returns: A ``Column``.
+public func array(_ cols: Column...) -> Column {
+ return fn("array", cols)
+}
+
+/// Appends the element to the end of the array.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - element: An element ``Column`` to be appended.
+/// - Returns: A ``Column``.
+public func array_append(_ column: Column, _ element: Column) -> Column {
+ return fn("array_append", column, element)
+}
+
+/// Appends the element to the end of the array.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - element: A literal element to be appended.
+/// - Returns: A ``Column``.
+public func array_append(_ column: Column, _ element: some SparkLiteral) ->
Column {
+ return array_append(column, element.toLiteralColumn)
+}
+
+/// Removes null values from the array.
+/// - Parameter column: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_compact(_ column: Column) -> Column {
+ return fn("array_compact", column)
+}
+
+/// Returns null if the array is null, true if the array contains `value`, and
false otherwise.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - value: A value ``Column`` to check.
+/// - Returns: A ``Column``.
+public func array_contains(_ column: Column, _ value: Column) -> Column {
+ return fn("array_contains", column, value)
+}
+
+/// Returns null if the array is null, true if the array contains `value`, and
false otherwise.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - value: A literal value to check.
+/// - Returns: A ``Column``.
+public func array_contains(_ column: Column, _ value: some SparkLiteral) ->
Column {
+ return array_contains(column, value.toLiteralColumn)
+}
+
+/// Removes duplicate values from the array.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_distinct(_ col: Column) -> Column {
+ return fn("array_distinct", col)
+}
+
+/// Returns an array of the elements in the first array but not in the second
array, without
+/// duplicates. The order of elements in the result is not determined.
+/// - Parameters:
+/// - col1: An array ``Column``.
+/// - col2: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_except(_ col1: Column, _ col2: Column) -> Column {
+ return fn("array_except", col1, col2)
+}
+
+/// Adds an item into a given array at a specified position.
+/// - Parameters:
+/// - arr: An array ``Column``.
+/// - pos: A position ``Column``.
+/// - value: A value ``Column`` to be inserted.
+/// - Returns: A ``Column``.
+public func array_insert(_ arr: Column, _ pos: Column, _ value: Column) ->
Column {
+ return fn("array_insert", arr, pos, value)
+}
+
+/// Returns an array of the elements in the intersection of the given two
arrays, without
+/// duplicates.
+/// - Parameters:
+/// - col1: An array ``Column``.
+/// - col2: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_intersect(_ col1: Column, _ col2: Column) -> Column {
+ return fn("array_intersect", col1, col2)
+}
+
+/// Concatenates the elements of the array column using the delimiter.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - delimiter: A delimiter string.
+/// - Returns: A ``Column``.
+public func array_join(_ column: Column, _ delimiter: String) -> Column {
+ return fn("array_join", column, lit(delimiter))
+}
+
+/// Concatenates the elements of the array column using the delimiter.
+/// Null values are replaced with `nullReplacement`.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - delimiter: A delimiter string.
+/// - nullReplacement: A string to replace null values.
+/// - Returns: A ``Column``.
+public func array_join(_ column: Column, _ delimiter: String, _
nullReplacement: String) -> Column
+{
+ return fn("array_join", column, lit(delimiter), lit(nullReplacement))
+}
+
+/// Returns the maximum value in the array. NaN is greater than any non-NaN
elements for
+/// double/float type. Null elements will be ignored.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_max(_ col: Column) -> Column {
+ return fn("array_max", col)
+}
+
+/// Returns the minimum value in the array. NaN is greater than any non-NaN
elements for
+/// double/float type. Null elements will be ignored.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_min(_ col: Column) -> Column {
+ return fn("array_min", col)
+}
+
+/// Locates the position of the first occurrence of the value in the given
array as long.
+/// Returns null if either of the arguments are null.
+/// The position is not zero based, but 1 based index. Returns 0 if the value
could not be
+/// found in the array.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - value: A value ``Column`` to locate.
+/// - Returns: A ``Column``.
+public func array_position(_ column: Column, _ value: Column) -> Column {
+ return fn("array_position", column, value)
+}
+
+/// Locates the position of the first occurrence of the value in the given
array as long.
+/// Returns null if either of the arguments are null.
+/// The position is not zero based, but 1 based index. Returns 0 if the value
could not be
+/// found in the array.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - value: A literal value to locate.
+/// - Returns: A ``Column``.
+public func array_position(_ column: Column, _ value: some SparkLiteral) ->
Column {
+ return array_position(column, value.toLiteralColumn)
+}
+
+/// Prepends the element to the beginning of the array.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - element: An element ``Column`` to be prepended.
+/// - Returns: A ``Column``.
+public func array_prepend(_ column: Column, _ element: Column) -> Column {
+ return fn("array_prepend", column, element)
+}
+
+/// Prepends the element to the beginning of the array.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - element: A literal element to be prepended.
+/// - Returns: A ``Column``.
+public func array_prepend(_ column: Column, _ element: some SparkLiteral) ->
Column {
+ return array_prepend(column, element.toLiteralColumn)
+}
+
+/// Removes all elements that equal to the element from the given array.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - element: An element ``Column`` to be removed.
+/// - Returns: A ``Column``.
+public func array_remove(_ column: Column, _ element: Column) -> Column {
+ return fn("array_remove", column, element)
+}
+
+/// Removes all elements that equal to the element from the given array.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - element: A literal element to be removed.
+/// - Returns: A ``Column``.
+public func array_remove(_ column: Column, _ element: some SparkLiteral) ->
Column {
+ return array_remove(column, element.toLiteralColumn)
+}
+
+/// Creates an array containing the left argument repeated the number of times
given by the
+/// right argument.
+/// - Parameters:
+/// - left: A value ``Column`` to be repeated.
+/// - right: A ``Column`` for the number of times.
+/// - Returns: A ``Column``.
+public func array_repeat(_ left: Column, _ right: Column) -> Column {
+ return fn("array_repeat", left, right)
+}
+
+/// Creates an array containing the left argument repeated the number of times
given by the
+/// right argument.
+/// - Parameters:
+/// - col: A value ``Column`` to be repeated.
+/// - count: The number of times.
+/// - Returns: A ``Column``.
+public func array_repeat(_ col: Column, _ count: Int32) -> Column {
+ return array_repeat(col, lit(count))
+}
+
+/// Returns the total number of elements in the array. The function returns
null for null input.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_size(_ col: Column) -> Column {
+ return fn("array_size", col)
+}
+
+/// Sorts the input array in ascending order. The elements of the input array
must be orderable.
+/// NaN is greater than any non-NaN elements for double/float type. Null
elements will be placed
+/// at the end of the returned array.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_sort(_ col: Column) -> Column {
+ return fn("array_sort", col)
+}
+
+/// Returns an array of the elements in the union of the given two arrays,
without duplicates.
+/// - Parameters:
+/// - col1: An array ``Column``.
+/// - col2: An array ``Column``.
+/// - Returns: A ``Column``.
+public func array_union(_ col1: Column, _ col2: Column) -> Column {
+ return fn("array_union", col1, col2)
+}
+
+/// Returns true if `a1` and `a2` have at least one non-null element in
common. If not and both
+/// the arrays are non-empty and any of them contains a null, it returns null.
It returns false
+/// otherwise.
+/// - Parameters:
+/// - a1: An array ``Column``.
+/// - a2: An array ``Column``.
+/// - Returns: A ``Column``.
+public func arrays_overlap(_ a1: Column, _ a2: Column) -> Column {
+ return fn("arrays_overlap", a1, a2)
+}
+
+/// Returns a merged array of structs in which the N-th struct contains all
N-th values of input
+/// arrays.
+/// - Parameter cols: Array ``Column``s to be merged.
+/// - Returns: A ``Column``.
+public func arrays_zip(_ cols: Column...) -> Column {
+ return fn("arrays_zip", cols)
+}
+
+/// Returns the length of the array or map. This is an alias of ``size(_:)``.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func cardinality(_ col: Column) -> Column {
+ return fn("cardinality", col)
+}
+
+/// Concatenates multiple input columns together into a single column. The
function works with
+/// strings, numeric, binary and compatible array columns.
+/// - Parameter exprs: ``Column``s to be concatenated.
+/// - Returns: A ``Column``.
+public func concat(_ exprs: Column...) -> Column {
+ return fn("concat", exprs)
+}
+
+/// Returns element of array at given index in `value` if `column` is array.
Returns value for
+/// the given key in `value` if `column` is map.
+/// - Parameters:
+/// - column: An array or map ``Column``.
+/// - value: An index or key ``Column``.
+/// - Returns: A ``Column``.
+public func element_at(_ column: Column, _ value: Column) -> Column {
+ return fn("element_at", column, value)
+}
+
+/// Returns element of array at given index in `value` if `column` is array.
Returns value for
+/// the given key in `value` if `column` is map.
+/// - Parameters:
+/// - column: An array or map ``Column``.
+/// - value: A literal index or key.
+/// - Returns: A ``Column``.
+public func element_at(_ column: Column, _ value: some SparkLiteral) -> Column
{
+ return element_at(column, value.toLiteralColumn)
+}
+
+/// Creates a new row for each element in the given array or map column. Uses
the default column
+/// name `col` for elements in the array and `key` and `value` for elements in
the map unless
+/// specified otherwise.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func explode(_ col: Column) -> Column {
+ return fn("explode", col)
+}
+
+/// Creates a new row for each element in the given array or map column. Uses
the default column
+/// name `col` for elements in the array and `key` and `value` for elements in
the map unless
+/// specified otherwise. Unlike ``explode(_:)``, if the array/map is null or
empty then null is
+/// produced.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func explode_outer(_ col: Column) -> Column {
+ return fn("explode_outer", col)
+}
+
+/// Creates a single array from an array of arrays. If a structure of nested
arrays is deeper
+/// than two levels, only one level of nesting is removed.
+/// - Parameter col: An array of arrays ``Column``.
+/// - Returns: A ``Column``.
+public func flatten(_ col: Column) -> Column {
+ return fn("flatten", col)
+}
+
+/// Returns element of array at given (0-based) index. If the index points
outside of the array
+/// boundaries, then this function returns NULL.
+/// - Parameters:
+/// - column: An array ``Column``.
+/// - index: An index ``Column``.
+/// - Returns: A ``Column``.
+public func get(_ column: Column, _ index: Column) -> Column {
+ return fn("get", column, index)
+}
+
+/// Creates a new row for each element in the given array of structs.
+/// - Parameter col: An array of structs ``Column``.
+/// - Returns: A ``Column``.
+public func inline(_ col: Column) -> Column {
+ return fn("inline", col)
+}
+
+/// Creates a new row for each element in the given array of structs. Unlike
``inline(_:)``,
+/// if the array is null or empty then null is produced for each nested column.
+/// - Parameter col: An array of structs ``Column``.
+/// - Returns: A ``Column``.
+public func inline_outer(_ col: Column) -> Column {
+ return fn("inline_outer", col)
+}
+
+/// Creates a new map column. The input columns must be grouped as key-value
pairs, e.g.
+/// (key1, value1, key2, value2, ...). The key columns must all have the same
data type, and
+/// can't be null. The value columns must all have the same data type.
+/// - Parameter cols: ``Column``s grouped as key-value pairs.
+/// - Returns: A ``Column``.
+public func map(_ cols: Column...) -> Column {
+ return fn("map", cols)
+}
+
+/// Returns the union of all the given maps.
+/// - Parameter cols: Map ``Column``s to be unioned.
+/// - Returns: A ``Column``.
+public func map_concat(_ cols: Column...) -> Column {
+ return fn("map_concat", cols)
+}
+
+/// Returns true if the map contains the key.
+/// - Parameters:
+/// - column: A map ``Column``.
+/// - key: A key ``Column`` to check.
+/// - Returns: A ``Column``.
+public func map_contains_key(_ column: Column, _ key: Column) -> Column {
+ return fn("map_contains_key", column, key)
+}
+
+/// Returns true if the map contains the key.
+/// - Parameters:
+/// - column: A map ``Column``.
+/// - key: A literal key to check.
+/// - Returns: A ``Column``.
+public func map_contains_key(_ column: Column, _ key: some SparkLiteral) ->
Column {
+ return map_contains_key(column, key.toLiteralColumn)
+}
+
+/// Returns an unordered array of all entries in the given map.
+/// - Parameter col: A map ``Column``.
+/// - Returns: A ``Column``.
+public func map_entries(_ col: Column) -> Column {
+ return fn("map_entries", col)
+}
+
+/// Creates a new map column. The array in the first column is used for keys.
The array in the
+/// second column is used for values. All elements in the array for key should
not be null.
+/// - Parameters:
+/// - keys: An array ``Column`` for keys.
+/// - values: An array ``Column`` for values.
+/// - Returns: A ``Column``.
+public func map_from_arrays(_ keys: Column, _ values: Column) -> Column {
+ return fn("map_from_arrays", keys, values)
+}
+
+/// Returns a map created from the given array of entries.
+/// - Parameter col: An array of structs ``Column``.
+/// - Returns: A ``Column``.
+public func map_from_entries(_ col: Column) -> Column {
+ return fn("map_from_entries", col)
+}
+
+/// Returns an unordered array containing the keys of the map.
+/// - Parameter col: A map ``Column``.
+/// - Returns: A ``Column``.
+public func map_keys(_ col: Column) -> Column {
+ return fn("map_keys", col)
+}
+
+/// Returns an unordered array containing the values of the map.
+/// - Parameter col: A map ``Column``.
+/// - Returns: A ``Column``.
+public func map_values(_ col: Column) -> Column {
+ return fn("map_values", col)
+}
+
+/// Creates a struct with the given field names and values.
+/// - Parameter cols: ``Column``s grouped as name-value pairs, e.g.
+/// (name1, val1, name2, val2, ...).
+/// - Returns: A ``Column``.
+public func named_struct(_ cols: Column...) -> Column {
+ return fn("named_struct", cols)
+}
+
+/// Creates a new row for each element with position in the given array or map
column. Uses the
+/// default column name `pos` for position, and `col` for elements in the
array and `key` and
+/// `value` for elements in the map unless specified otherwise.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func posexplode(_ col: Column) -> Column {
+ return fn("posexplode", col)
+}
+
+/// Creates a new row for each element with position in the given array or map
column. Uses the
+/// default column name `pos` for position, and `col` for elements in the
array and `key` and
+/// `value` for elements in the map unless specified otherwise. Unlike
``posexplode(_:)``, if
+/// the array/map is null or empty then the row (null, null) is produced.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func posexplode_outer(_ col: Column) -> Column {
+ return fn("posexplode_outer", col)
+}
+
+/// Returns a reversed string or an array with reverse order of elements.
+/// - Parameter col: A string or array ``Column``.
+/// - Returns: A ``Column``.
+public func reverse(_ col: Column) -> Column {
+ return fn("reverse", col)
+}
+
+/// Generates a sequence of integers from `start` to `stop`, incrementing by 1
if `start` is
+/// less than or equal to `stop`, otherwise -1.
+/// - Parameters:
+/// - start: A start ``Column``.
+/// - stop: A stop ``Column``.
+/// - Returns: A ``Column``.
+public func sequence(_ start: Column, _ stop: Column) -> Column {
+ return fn("sequence", start, stop)
+}
+
+/// Generates a sequence of integers from `start` to `stop`, incrementing by
`step`.
+/// - Parameters:
+/// - start: A start ``Column``.
+/// - stop: A stop ``Column``.
+/// - step: A step ``Column``.
+/// - Returns: A ``Column``.
+public func sequence(_ start: Column, _ stop: Column, _ step: Column) ->
Column {
+ return fn("sequence", start, stop, step)
+}
+
+/// Returns a random permutation of the given array.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func shuffle(_ col: Column) -> Column {
+ return fn("shuffle", col)
+}
+
+/// Returns a random permutation of the given array with the given seed.
+/// - Parameters:
+/// - col: An array ``Column``.
+/// - seed: A seed ``Column``.
+/// - Returns: A ``Column``.
+public func shuffle(_ col: Column, _ seed: Column) -> Column {
+ return fn("shuffle", col, seed)
+}
+
+/// Returns the length of the array or map.
+/// - Parameter col: An array or map ``Column``.
+/// - Returns: A ``Column``.
+public func size(_ col: Column) -> Column {
+ return fn("size", col)
+}
+
+/// Returns an array containing all the elements in `x` from index `start` (or
starting from the
+/// end if `start` is negative) with the specified `length`.
+/// - Parameters:
+/// - x: An array ``Column``.
+/// - start: A start index ``Column`` (1-based).
+/// - length: A length ``Column``.
+/// - Returns: A ``Column``.
+public func slice(_ x: Column, _ start: Column, _ length: Column) -> Column {
+ return fn("slice", x, start, length)
+}
+
+/// Returns an array containing all the elements in `x` from index `start` (or
starting from the
+/// end if `start` is negative) with the specified `length`.
+/// - Parameters:
+/// - x: An array ``Column``.
+/// - start: A start index (1-based).
+/// - length: A length.
+/// - Returns: A ``Column``.
+public func slice(_ x: Column, _ start: Int32, _ length: Int32) -> Column {
+ return slice(x, lit(start), lit(length))
+}
+
+/// Sorts the input array in ascending order according to the natural ordering
of the array
+/// elements. Null elements will be placed at the beginning of the returned
array.
+/// - Parameter col: An array ``Column``.
+/// - Returns: A ``Column``.
+public func sort_array(_ col: Column) -> Column {
+ return sort_array(col, true)
+}
+
+/// Sorts the input array in ascending or descending order according to the
natural ordering of
+/// the array elements. Null elements will be placed at the beginning of the
returned array in
+/// ascending order or at the end of the returned array in descending order.
+/// - Parameters:
+/// - col: An array ``Column``.
+/// - asc: True for the ascending order.
+/// - Returns: A ``Column``.
+public func sort_array(_ col: Column, _ asc: Bool) -> Column {
+ return fn("sort_array", col, lit(asc))
+}
+
+/// Separates the columns into the specified number of rows.
+/// - Parameter cols: ``Column``s where the first is the number of rows,
followed by values.
+/// - Returns: A ``Column``.
+public func stack(_ cols: Column...) -> Column {
+ return fn("stack", cols)
+}
+
+/// Creates a map after splitting the text into key/value pairs using
delimiters. The default
+/// delimiters are `,` for `pairDelim` and `:` for `keyValueDelim`.
+/// - Parameter text: A text ``Column``.
+/// - Returns: A ``Column``.
+public func str_to_map(_ text: Column) -> Column {
+ return fn("str_to_map", text)
+}
+
+/// Creates a map after splitting the text into key/value pairs using
delimiters. The default
+/// delimiter is `:` for `keyValueDelim`.
+/// - Parameters:
+/// - text: A text ``Column``.
+/// - pairDelim: A pair delimiter ``Column``.
+/// - Returns: A ``Column``.
+public func str_to_map(_ text: Column, _ pairDelim: Column) -> Column {
+ return fn("str_to_map", text, pairDelim)
+}
+
+/// Creates a map after splitting the text into key/value pairs using
delimiters.
+/// - Parameters:
+/// - text: A text ``Column``.
+/// - pairDelim: A pair delimiter ``Column``.
+/// - keyValueDelim: A key/value delimiter ``Column``.
+/// - Returns: A ``Column``.
+public func str_to_map(_ text: Column, _ pairDelim: Column, _ keyValueDelim:
Column) -> Column {
+ return fn("str_to_map", text, pairDelim, keyValueDelim)
+}
+
+/// Creates a new struct column.
+/// - Parameter cols: ``Column``s to be combined into a struct.
+/// - Returns: A ``Column``.
+public func `struct`(_ cols: Column...) -> Column {
+ return fn("struct", cols)
+}
+
+/// Returns element of array at given (1-based) index. If the index points
outside of the array
+/// boundaries, then this function returns NULL. Returns value for the given
key in map, or NULL
+/// if the key is not contained in the map.
+/// - Parameters:
+/// - column: An array or map ``Column``.
+/// - value: An index or key ``Column``.
+/// - Returns: A ``Column``.
+public func try_element_at(_ column: Column, _ value: Column) -> Column {
+ return fn("try_element_at", column, value)
+}
diff --git a/Tests/SparkConnectTests/CollectionFunctionsTests.swift
b/Tests/SparkConnectTests/CollectionFunctionsTests.swift
new file mode 100644
index 0000000..357fe5b
--- /dev/null
+++ b/Tests/SparkConnectTests/CollectionFunctionsTests.swift
@@ -0,0 +1,272 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+import Testing
+
+@testable import SparkConnect
+
+/// A test suite for `CollectionFunctions`
+@Suite(.serialized)
+struct CollectionFunctionsTests {
+
+ @Test
+ func collectionFunctions() throws {
+ for (column, name) in [
+ (array_compact(col("a")), "array_compact"),
+ (array_distinct(col("a")), "array_distinct"),
+ (array_max(col("a")), "array_max"),
+ (array_min(col("a")), "array_min"),
+ (array_size(col("a")), "array_size"),
+ (array_sort(col("a")), "array_sort"),
+ (cardinality(col("a")), "cardinality"),
+ (explode(col("a")), "explode"),
+ (explode_outer(col("a")), "explode_outer"),
+ (flatten(col("a")), "flatten"),
+ (inline(col("a")), "inline"),
+ (inline_outer(col("a")), "inline_outer"),
+ (map_entries(col("a")), "map_entries"),
+ (map_from_entries(col("a")), "map_from_entries"),
+ (map_keys(col("a")), "map_keys"),
+ (map_values(col("a")), "map_values"),
+ (posexplode(col("a")), "posexplode"),
+ (posexplode_outer(col("a")), "posexplode_outer"),
+ (reverse(col("a")), "reverse"),
+ (shuffle(col("a")), "shuffle"),
+ (size(col("a")), "size"),
+ (str_to_map(col("a")), "str_to_map"),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
== "a")
+ }
+ }
+
+ @Test
+ func collectionFunctionArguments() throws {
+ for (column, name) in [
+ (array_append(col("a"), col("b")), "array_append"),
+ (array_contains(col("a"), col("b")), "array_contains"),
+ (array_except(col("a"), col("b")), "array_except"),
+ (array_intersect(col("a"), col("b")), "array_intersect"),
+ (array_position(col("a"), col("b")), "array_position"),
+ (array_prepend(col("a"), col("b")), "array_prepend"),
+ (array_remove(col("a"), col("b")), "array_remove"),
+ (array_repeat(col("a"), col("b")), "array_repeat"),
+ (array_union(col("a"), col("b")), "array_union"),
+ (arrays_overlap(col("a"), col("b")), "arrays_overlap"),
+ (element_at(col("a"), col("b")), "element_at"),
+ (get(col("a"), col("b")), "get"),
+ (map_contains_key(col("a"), col("b")), "map_contains_key"),
+ (map_from_arrays(col("a"), col("b")), "map_from_arrays"),
+ (sequence(col("a"), col("b")), "sequence"),
+ (shuffle(col("a"), col("b")), "shuffle"),
+ (str_to_map(col("a"), col("b")), "str_to_map"),
+ (try_element_at(col("a"), col("b")), "try_element_at"),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments.count == 2)
+ }
+
+ for (column, name) in [
+ (array_insert(col("a"), col("b"), col("c")), "array_insert"),
+ (sequence(col("a"), col("b"), col("c")), "sequence"),
+ (slice(col("a"), col("b"), col("c")), "slice"),
+ (str_to_map(col("a"), col("b"), col("c")), "str_to_map"),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments.count == 3)
+ }
+ }
+
+ @Test
+ func collectionFunctionLiteralArguments() throws {
+ for (column, name) in [
+ (array_append(col("a"), "x"), "array_append"),
+ (array_contains(col("a"), "x"), "array_contains"),
+ (array_position(col("a"), "x"), "array_position"),
+ (array_prepend(col("a"), "x"), "array_prepend"),
+ (array_remove(col("a"), "x"), "array_remove"),
+ (element_at(col("a"), "x"), "element_at"),
+ (map_contains_key(col("a"), "x"), "map_contains_key"),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments[1].literal.string == "x")
+ }
+
+ #expect(array_join(col("a"),
",").expr.unresolvedFunction.arguments[1].literal.string == ",")
+ #expect(array_join(col("a"), ",",
"*").expr.unresolvedFunction.arguments.count == 3)
+ #expect(array_repeat(col("a"),
2).expr.unresolvedFunction.arguments[1].literal.integer == 2)
+
#expect(sort_array(col("a")).expr.unresolvedFunction.arguments[1].literal.boolean
== true)
+ #expect(sort_array(col("a"),
false).expr.unresolvedFunction.arguments[1].literal.boolean == false)
+
+ let sliced = slice(col("a"), 2, 3).expr
+ #expect(sliced.unresolvedFunction.arguments[1].literal.integer == 2)
+ #expect(sliced.unresolvedFunction.arguments[2].literal.integer == 3)
+ }
+
+ @Test
+ func collectionFunctionVariadicArguments() throws {
+ for (column, name, count) in [
+ (array(col("a"), col("b")), "array", 2),
+ (arrays_zip(col("a"), col("b")), "arrays_zip", 2),
+ (concat(col("a"), col("b"), col("c")), "concat", 3),
+ (map(col("a"), col("b"), col("c"), col("d")), "map", 4),
+ (map_concat(col("a"), col("b")), "map_concat", 2),
+ (named_struct(col("a"), col("b"), col("c"), col("d")), "named_struct",
4),
+ (stack(col("a"), col("b"), col("c")), "stack", 3),
+ (`struct`(col("a"), col("b")), "struct", 2),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments.count == count)
+ }
+ }
+
+ @Test
+ func selectArrayFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ let arr = array(lit(1), lit(2), lit(3))
+
+ let modified = try await df.select(
+ arr.cast("string"), array_append(arr, 4).cast("string"),
+ array_prepend(arr, 0).cast("string"), array_remove(arr,
2).cast("string"),
+ array_insert(arr, lit(Int32(1)), lit(0)).cast("string")
+ ).collect()
+ #expect(modified == [Row("[1, 2, 3]", "[1, 2, 3, 4]", "[0, 1, 2, 3]", "[1,
3]", "[0, 1, 2, 3]")])
+
+ let setOps = try await df.select(
+ array_distinct(array(lit(1), lit(1), lit(2))).cast("string"),
+ array_except(arr, array(lit(2))).cast("string"),
+ array_intersect(arr, array(lit(2), lit(4))).cast("string"),
+ array_union(array(lit(1)), array(lit(2))).cast("string")
+ ).collect()
+ #expect(setOps == [Row("[1, 2]", "[1, 3]", "[2]", "[1, 2]")])
+
+ let lookups = try await df.select(
+ array_contains(arr, 2), array_position(arr, 2), element_at(arr,
Int32(1)),
+ try_element_at(arr, lit(Int32(1))), get(arr, lit(Int32(0)))
+ ).collect()
+ #expect(lookups == [Row(true, 2, 1, 1, 1)])
+
+ let ordered = try await df.select(
+ sort_array(array(lit(3), lit(1), lit(2))).cast("string"),
+ sort_array(arr, false).cast("string"), reverse(arr).cast("string"),
+ slice(arr, 2, 2).cast("string"), flatten(array(array(lit(1)),
array(lit(2)))).cast("string")
+ ).collect()
+ #expect(ordered == [Row("[1, 2, 3]", "[3, 2, 1]", "[3, 2, 1]", "[2, 3]",
"[1, 2]")])
+
+ let sizes = try await df.select(
+ size(arr), cardinality(arr), array_size(arr), array_min(arr),
array_max(arr),
+ size(shuffle(arr)), size(shuffle(arr, lit(42)))
+ ).collect()
+ #expect(sizes == [Row(3, 3, 3, 1, 3, 3, 3)])
+
+ let combined = try await df.select(
+ array_repeat(lit(1), 2).cast("string"), sequence(lit(1),
lit(3)).cast("string"),
+ sequence(lit(5), lit(1), lit(-2)).cast("string"),
+ arrays_zip(array(lit(1)), array(lit("a"))).cast("string"),
+ arrays_overlap(arr, array(lit(3), lit(9)))
+ ).collect()
+ #expect(combined == [Row("[1, 1]", "[1, 2, 3]", "[5, 3, 1]", "[{1, a}]",
true)])
+
+ let dfWithNull = try await spark.sql("SELECT array(1, null, 3) AS a")
+ let compacted = try await dfWithNull.select(
+ array_compact(col("a")).cast("string"), array_join(col("a"), ","),
+ array_join(col("a"), ",", "*")
+ ).collect()
+ #expect(compacted == [Row("[1, 3]", "1,3", "1,*,3")])
+ await spark.stop()
+ }
+
+ @Test
+ func selectMapFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ let m = map(lit("a"), lit(1), lit("b"), lit(2))
+
+ let rows = try await df.select(
+ m.cast("string"), map_keys(m).cast("string"),
map_values(m).cast("string"),
+ map_entries(m).cast("string")
+ ).collect()
+ #expect(rows == [Row("{a -> 1, b -> 2}", "[a, b]", "[1, 2]", "[{a, 1}, {b,
2}]")])
+
+ let lookups = try await df.select(
+ map_contains_key(m, "a"), element_at(m, "a"), try_element_at(m, lit("b"))
+ ).collect()
+ #expect(lookups == [Row(true, 1, 2)])
+
+ let created = try await df.select(
+ map_concat(map(lit("a"), lit(1)), map(lit("b"), lit(2))).cast("string"),
+ map_from_arrays(array(lit("a")), array(lit(1))).cast("string"),
+ map_from_entries(array(`struct`(lit("a"), lit(1)))).cast("string")
+ ).collect()
+ #expect(created == [Row("{a -> 1, b -> 2}", "{a -> 1}", "{a -> 1}")])
+
+ let parsed = try await df.select(
+ str_to_map(lit("a:1,b:2")).cast("string"),
+ str_to_map(lit("a:1;b:2"), lit(";")).cast("string"),
+ str_to_map(lit("a=1;b=2"), lit(";"), lit("=")).cast("string")
+ ).collect()
+ #expect(
+ parsed == [Row("{a -> 1, b -> 2}", "{a -> 1, b -> 2}", "{a -> 1, b ->
2}")])
+ await spark.stop()
+ }
+
+ @Test
+ func selectStructAndConcatFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ let rows = try await df.select(
+ `struct`(lit(1), lit("a")).cast("string"),
+ named_struct(lit("x"), lit(1), lit("y"), lit("a")).cast("string"),
+ concat(lit("a"), lit("b"), lit("c")),
+ concat(array(lit(1)), array(lit(2))).cast("string")
+ ).collect()
+ #expect(rows == [Row("{1, a}", "{1, a}", "abc", "[1, 2]")])
+ await spark.stop()
+ }
+
+ @Test
+ func selectGeneratorFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ #expect(try await df.select(explode(array(lit(1), lit(2)))).collect() ==
[Row(1), Row(2)])
+ #expect(
+ try await df.select(explode_outer(array(lit(1), lit(2)))).collect() ==
[Row(1), Row(2)])
+ #expect(try await df.select(explode(map(lit("a"), lit(1)))).collect() ==
[Row("a", 1)])
+ #expect(
+ try await df.select(posexplode(array(lit(10), lit(20)))).collect()
+ == [Row(0, 10), Row(1, 20)])
+ #expect(
+ try await df.select(posexplode_outer(array(lit(10), lit(20)))).collect()
+ == [Row(0, 10), Row(1, 20)])
+
+ let structArray = array(named_struct(lit("a"), lit(1), lit("b"), lit(2)))
+ #expect(try await df.select(inline(structArray)).collect() == [Row(1, 2)])
+ #expect(try await df.select(inline_outer(structArray)).collect() ==
[Row(1, 2)])
+ #expect(
+ try await df.select(stack(lit(Int32(2)), lit(1), lit(2), lit(3),
lit(4))).collect()
+ == [Row(1, 2), Row(3, 4)])
+ await spark.stop()
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]