dongjoon-hyun opened a new pull request, #463: URL: https://github.com/apache/spark-connect-swift/pull/463
### What changes were proposed in this pull request? This PR aims to add collection functions (array, map, struct, and generator functions) in a new file, `CollectionFunctions.swift`. The function names and signatures follow Scala's `org.apache.spark.sql.functions`. | Category | Functions | | --- | --- | | Array | `array`, `array_append`, `array_compact`, `array_contains`, `array_distinct`, `array_except`, `array_insert`, `array_intersect`, `array_join`, `array_max`, `array_min`, `array_position`, `array_prepend`, `array_remove`, `array_repeat`, `array_size`, `array_sort`, `arrays_overlap`, `arrays_zip`, `flatten`, `get`, `sequence`, `shuffle`, `slice`, `sort_array` | | Map | `map`, `map_concat`, `map_contains_key`, `map_entries`, `map_from_arrays`, `map_from_entries`, `map_keys`, `map_values`, `str_to_map` | | Struct | `named_struct`, `struct` (declared as `` `struct` `` because it is a Swift keyword) | | Generator | `explode`, `explode_outer`, `inline`, `inline_outer`, `posexplode`, `posexplode_outer`, `stack` | | Common collection | `cardinality`, `concat`, `element_at`, `reverse`, `size`, `try_element_at` | Following the existing `SparkLiteral` convention, functions whose Scala signatures accept `Any` (`array_append`, `array_contains`, `array_position`, `array_prepend`, `array_remove`, `element_at`, `map_contains_key`) are provided as both `Column` and `some SparkLiteral` overloads. Scala's `Int` parameters (`slice`, `array_repeat`) use Swift `Int32`. The following functions are excluded intentionally. | Functions | Reason | | --- | --- | | `transform`, `filter`, `exists`, `forall`, `aggregate`, `reduce`, `zip_with`, `map_zip_with`, `map_filter`, `transform_keys`, `transform_values`, `array_sort(e, comparator)` | Requires lambda expression support (`LambdaFunction` / `UnresolvedNamedLambdaVariable` proto), which this client does not have yet | | `array(colName: String, ...)`, `struct(colName: String, ...)` | String-based convenience overloads; only Column-based signatures are provided | ### Why are the changes needed? To improve the usability of the column-based `DataFrame` API by providing collection functions in the same shape as Scala/PySpark, so users can write queries without falling back to `selectExpr` or raw SQL strings. ### Does this PR introduce _any_ user-facing change? Yes, this adds new public functions. There is no behavior change to existing APIs. ### How was this patch tested? Pass the CIs with a newly added test suite, `CollectionFunctionsTests`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
