This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new a5829e4 [SPARK-58483] Support string functions in `StringFunctions`
a5829e4 is described below
commit a5829e4c726517aa62cc6565fe0c45539be98976
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 31 16:01:05 2026 -0700
[SPARK-58483] Support string functions in `StringFunctions`
### What changes were proposed in this pull request?
This PR aims to add string functions in a new file,
`StringFunctions.swift`. The function names and signatures follow Scala's
`org.apache.spark.sql.functions` (`group string_funcs`), covering 72 functions
(111 overloads including `Column`-based and literal-based signatures).
| Category | Functions |
| --- | --- |
| Case and basic conversion | `lower`/`lcase`, `upper`/`ucase`, `initcap`,
`ascii`, `char`, `chr`, `soundex`, `quote` |
| Length | `length`, `len`, `char_length`, `character_length`,
`bit_length`, `octet_length` |
| Trim and pad | `trim`, `ltrim`, `rtrim` (each with trim-string
overloads), `btrim`, `lpad`, `rpad` |
| Search and comparison | `instr`, `locate`, `position`, `contains`,
`startswith`, `endswith`, `find_in_set`, `levenshtein` (with a `threshold`
overload), `jaro_winkler_similarity` |
| Transformation | `concat_ws`, `format_number`, `format_string`, `printf`,
`elt`, `repeat`, `replace`, `translate`, `overlay`, `left`, `right`, `substr`,
`substring`, `substring_index`, `split_part`, `mask`, `sentences`, `randstr` |
| Regular expression | `regexp_count`, `regexp_extract`,
`regexp_extract_all`, `regexp_instr`, `regexp_replace`, `regexp_substr`,
`split` |
| Encoding and conversion | `base64`, `unbase64`, `encode`, `decode`,
`to_binary`, `try_to_binary`, `to_char`, `to_varchar`, `to_number`,
`try_to_number` |
| UTF-8 validation | `is_valid_utf8`, `make_valid_utf8`, `validate_utf8`,
`try_validate_utf8` |
| Collation | `collate`, `collation` |
The following are excluded intentionally.
| Excluded | Reason |
| --- | --- |
| `concat` | It belongs to `collection_funcs` in Scala and will be handled
by a separate PR. |
| `lpad`/`rpad` with `Array[Byte]` pad | Binary literals are not supported
by `lit()` in this client yet. |
### Why are the changes needed?
To improve the usability of the column-based `DataFrame` API by providing
string functions in the same shape as Scala/PySpark, so users can write queries
without falling back to `selectExpr` or raw SQL strings.
### Does this PR introduce _any_ user-facing change?
Yes, this adds new public functions. There is no behavior change to
existing APIs.
### How was this patch tested?
Pass the CIs with a newly added test suite.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #461 from dongjoon-hyun/SPARK-58483.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/Functions.swift | 10 +
Sources/SparkConnect/StringFunctions.swift | 1080 ++++++++++++++++++++
Tests/SparkConnectTests/StringFunctionsTests.swift | 393 +++++++
3 files changed, 1483 insertions(+)
diff --git a/Sources/SparkConnect/Functions.swift
b/Sources/SparkConnect/Functions.swift
index 59e1fcf..4943bd2 100644
--- a/Sources/SparkConnect/Functions.swift
+++ b/Sources/SparkConnect/Functions.swift
@@ -252,6 +252,16 @@ func fn(_ name: String, _ args: Column...) -> Column {
return Column(expr)
}
+func fn(_ name: String, _ args: [Column], isDistinct: Bool = false) -> Column {
+ var function = Spark_Connect_Expression.UnresolvedFunction()
+ function.functionName = name
+ function.arguments = args.map { $0.expr }
+ function.isDistinct = isDistinct
+ var expr = Spark_Connect_Expression()
+ expr.unresolvedFunction = function
+ return Column(expr)
+}
+
private func litColumn(_ literal: ExpressionLiteral) -> Column {
var expr = Spark_Connect_Expression()
expr.literal = literal
diff --git a/Sources/SparkConnect/StringFunctions.swift
b/Sources/SparkConnect/StringFunctions.swift
new file mode 100644
index 0000000..fbdb7b8
--- /dev/null
+++ b/Sources/SparkConnect/StringFunctions.swift
@@ -0,0 +1,1080 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+/// Computes the numeric value of the first character of the string column.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func ascii(_ col: Column) -> Column {
+ return fn("ascii", col)
+}
+
+/// Computes the BASE64 encoding of a binary column and returns it as a string
column.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func base64(_ col: Column) -> Column {
+ return fn("base64", col)
+}
+
+/// Calculates the bit length for the specified string column.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func bit_length(_ col: Column) -> Column {
+ return fn("bit_length", col)
+}
+
+/// Removes the leading and trailing space characters from `str`.
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func btrim(_ str: Column) -> Column {
+ return fn("btrim", str)
+}
+
+/// Removes the leading and trailing `trim` characters from `str`.
+/// - Parameters:
+/// - str: A ``Column`` to trim.
+/// - trim: A ``Column`` of the trim string characters to trim.
+/// - Returns: A ``Column``.
+public func btrim(_ str: Column, _ trim: Column) -> Column {
+ return fn("btrim", str, trim)
+}
+
+/// Returns the ASCII character having the binary equivalent to `n`. If n is
larger than 256 the
+/// result is equivalent to char(n % 256).
+/// - Parameter n: A ``Column``.
+/// - Returns: A ``Column``.
+public func char(_ n: Column) -> Column {
+ return fn("char", n)
+}
+
+/// Returns the character length of string data or number of bytes of binary
data.
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func char_length(_ str: Column) -> Column {
+ return fn("char_length", str)
+}
+
+/// Returns the character length of string data or number of bytes of binary
data.
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func character_length(_ str: Column) -> Column {
+ return fn("character_length", str)
+}
+
+/// Returns the ASCII character having the binary equivalent to `n`. If n is
larger than 256 the
+/// result is equivalent to chr(n % 256).
+/// - Parameter n: A ``Column``.
+/// - Returns: A ``Column``.
+public func chr(_ n: Column) -> Column {
+ return fn("chr", n)
+}
+
+/// Marks a given column with specified collation.
+/// - Parameters:
+/// - col: A ``Column``.
+/// - collation: A collation name.
+/// - Returns: A ``Column``.
+public func collate(_ col: Column, _ collation: String) -> Column {
+ return fn("collate", col, lit(collation))
+}
+
+/// Returns the collation name of a given column.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func collation(_ col: Column) -> Column {
+ return fn("collation", col)
+}
+
+/// Concatenates multiple input string columns together into a single string
column, using the
+/// given separator.
+/// - Parameters:
+/// - sep: A separator string.
+/// - exprs: ``Column``s to concatenate.
+/// - Returns: A ``Column``.
+public func concat_ws(_ sep: String, _ exprs: Column...) -> Column {
+ return fn("concat_ws", [lit(sep)] + exprs)
+}
+
+/// Returns a boolean. The value is true if `right` is found inside `left`.
Returns NULL if
+/// either input expression is NULL. Otherwise, returns false.
+/// - Parameters:
+/// - left: A ``Column`` to search in.
+/// - right: A ``Column`` to search for.
+/// - Returns: A ``Column``.
+public func contains(_ left: Column, _ right: Column) -> Column {
+ return fn("contains", left, right)
+}
+
+/// Computes the first argument into a string from a binary using the provided
character set
+/// (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE',
'UTF-16').
+/// - Parameters:
+/// - value: A ``Column`` to decode.
+/// - charset: A character set name.
+/// - Returns: A ``Column``.
+public func decode(_ value: Column, _ charset: String) -> Column {
+ return fn("decode", value, lit(charset))
+}
+
+/// Returns the `n`-th input, e.g., returns the second input when `n` is 2.
+/// - Parameter inputs: An index ``Column`` followed by input ``Column``s.
+/// - Returns: A ``Column``.
+public func elt(_ inputs: Column...) -> Column {
+ return fn("elt", inputs)
+}
+
+/// Computes the first argument into a binary from a string using the provided
character set
+/// (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE',
'UTF-16').
+/// - Parameters:
+/// - value: A ``Column`` to encode.
+/// - charset: A character set name.
+/// - Returns: A ``Column``.
+public func encode(_ value: Column, _ charset: String) -> Column {
+ return fn("encode", value, lit(charset))
+}
+
+/// Returns a boolean. The value is true if `str` ends with `suffix`. Returns
NULL if either
+/// input expression is NULL. Otherwise, returns false.
+/// - Parameters:
+/// - str: A ``Column`` to search in.
+/// - suffix: A suffix ``Column``.
+/// - Returns: A ``Column``.
+public func endswith(_ str: Column, _ suffix: Column) -> Column {
+ return fn("endswith", str, suffix)
+}
+
+/// Returns the index (1-based) of the given string `str` in the
comma-delimited list `strArray`.
+/// Returns 0 if the given string was not found or if it contains a comma.
+/// - Parameters:
+/// - str: A ``Column`` to search for.
+/// - strArray: A comma-delimited list ``Column``.
+/// - Returns: A ``Column``.
+public func find_in_set(_ str: Column, _ strArray: Column) -> Column {
+ return fn("find_in_set", str, strArray)
+}
+
+/// Formats numeric column `x` to a format like '#,###,###.##', rounded to `d`
decimal places
+/// with HALF_EVEN round mode, and returns the result as a string column.
+/// - Parameters:
+/// - x: A numeric ``Column``.
+/// - d: The number of decimal places.
+/// - Returns: A ``Column``.
+public func format_number(_ x: Column, _ d: Int32) -> Column {
+ return fn("format_number", x, lit(d))
+}
+
+/// Formats the arguments in printf-style and returns the result as a string
column.
+/// - Parameters:
+/// - format: A printf-style format string.
+/// - arguments: ``Column``s to format.
+/// - Returns: A ``Column``.
+public func format_string(_ format: String, _ arguments: Column...) -> Column {
+ return fn("format_string", [lit(format)] + arguments)
+}
+
+/// Returns a new string column by converting the first letter of each word to
uppercase.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func initcap(_ col: Column) -> Column {
+ return fn("initcap", col)
+}
+
+/// Locate the position of the first occurrence of `substring` in `str`. The
position is not
+/// zero based, but 1 based index. Returns 0 if `substring` could not be found
in `str`.
+/// - Parameters:
+/// - str: A ``Column`` to search in.
+/// - substring: A substring to search for.
+/// - Returns: A ``Column``.
+public func instr(_ str: Column, _ substring: String) -> Column {
+ return instr(str, lit(substring))
+}
+
+/// Locate the position of the first occurrence of `substring` in `str`. The
position is not
+/// zero based, but 1 based index. Returns 0 if `substring` could not be found
in `str`.
+/// - Parameters:
+/// - str: A ``Column`` to search in.
+/// - substring: A substring ``Column`` to search for.
+/// - Returns: A ``Column``.
+public func instr(_ str: Column, _ substring: Column) -> Column {
+ return fn("instr", str, substring)
+}
+
+/// Locate the position of the first occurrence of `substring` in `str`,
starting the search
+/// from the `start` position.
+/// - Parameters:
+/// - str: A ``Column`` to search in.
+/// - substring: A substring ``Column`` to search for.
+/// - start: A start position of the search.
+/// - Returns: A ``Column``.
+public func instr(_ str: Column, _ substring: Column, _ start: Int32) ->
Column {
+ return fn("instr", str, substring, lit(start))
+}
+
+/// Locate the position of the first occurrence of `substring` in `str`,
starting the search
+/// from the `start` position.
+/// - Parameters:
+/// - str: A ``Column`` to search in.
+/// - substring: A substring ``Column`` to search for.
+/// - start: A start position ``Column`` of the search.
+/// - Returns: A ``Column``.
+public func instr(_ str: Column, _ substring: Column, _ start: Column) ->
Column {
+ return fn("instr", str, substring, start)
+}
+
+/// Locate the position of the `occurrence`-th occurrence of `substring` in
`str`, starting the
+/// search from the `start` position.
+/// - Parameters:
+/// - str: A ``Column`` to search in.
+/// - substring: A substring ``Column`` to search for.
+/// - start: A start position of the search.
+/// - occurrence: The occurrence to find.
+/// - Returns: A ``Column``.
+public func instr(_ str: Column, _ substring: Column, _ start: Int32, _
occurrence: Int32)
+ -> Column
+{
+ return fn("instr", str, substring, lit(start), lit(occurrence))
+}
+
+/// Locate the position of the `occurrence`-th occurrence of `substring` in
`str`, starting the
+/// search from the `start` position.
+/// - Parameters:
+/// - str: A ``Column`` to search in.
+/// - substring: A substring ``Column`` to search for.
+/// - start: A start position ``Column`` of the search.
+/// - occurrence: An occurrence ``Column`` to find.
+/// - Returns: A ``Column``.
+public func instr(_ str: Column, _ substring: Column, _ start: Column, _
occurrence: Column)
+ -> Column
+{
+ return fn("instr", str, substring, start, occurrence)
+}
+
+/// Returns true if the input is a valid UTF-8 string, otherwise returns false.
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func is_valid_utf8(_ str: Column) -> Column {
+ return fn("is_valid_utf8", str)
+}
+
+/// Computes the Jaro-Winkler similarity of the two given string columns.
+/// - Parameters:
+/// - l: A ``Column``.
+/// - r: A ``Column``.
+/// - Returns: A ``Column``.
+public func jaro_winkler_similarity(_ l: Column, _ r: Column) -> Column {
+ return fn("jaro_winkler_similarity", l, r)
+}
+
+/// Converts a string column to lowercase.
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func lcase(_ str: Column) -> Column {
+ return fn("lcase", str)
+}
+
+/// Returns the leftmost `len` characters from the string `str`. If `len` is
less or equal than
+/// 0 the result is an empty string.
+/// - Parameters:
+/// - str: A ``Column``.
+/// - len: A length ``Column``.
+/// - Returns: A ``Column``.
+public func left(_ str: Column, _ len: Column) -> Column {
+ return fn("left", str, len)
+}
+
+/// Computes the character length of a given string or number of bytes of a
binary string.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func len(_ col: Column) -> Column {
+ return fn("len", col)
+}
+
+/// Computes the character length of a given string or number of bytes of a
binary string.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func length(_ col: Column) -> Column {
+ return fn("length", col)
+}
+
+/// Computes the Levenshtein distance of the two given string columns.
+/// - Parameters:
+/// - l: A ``Column``.
+/// - r: A ``Column``.
+/// - Returns: A ``Column``.
+public func levenshtein(_ l: Column, _ r: Column) -> Column {
+ return fn("levenshtein", l, r)
+}
+
+/// Computes the Levenshtein distance of the two given string columns if it's
less than or
+/// equal to a given threshold. Returns -1 if the distance is larger than the
threshold.
+/// - Parameters:
+/// - l: A ``Column``.
+/// - r: A ``Column``.
+/// - threshold: A threshold of the distance.
+/// - Returns: A ``Column``.
+public func levenshtein(_ l: Column, _ r: Column, _ threshold: Int32) ->
Column {
+ return fn("levenshtein", l, r, lit(threshold))
+}
+
+/// Locate the position of the first occurrence of `substr` in `str`. The
position is not zero
+/// based, but 1 based index. Returns 0 if `substr` could not be found in
`str`.
+/// - Parameters:
+/// - substr: A substring to search for.
+/// - str: A ``Column`` to search in.
+/// - Returns: A ``Column``.
+public func locate(_ substr: String, _ str: Column) -> Column {
+ return fn("locate", lit(substr), str)
+}
+
+/// Locate the position of the first occurrence of `substr` in `str`, starting
from the `pos`
+/// position. The position is not zero based, but 1 based index. Returns 0 if
`substr` could
+/// not be found in `str`.
+/// - Parameters:
+/// - substr: A substring to search for.
+/// - str: A ``Column`` to search in.
+/// - pos: A start position of the search.
+/// - Returns: A ``Column``.
+public func locate(_ substr: String, _ str: Column, _ pos: Int32) -> Column {
+ return fn("locate", lit(substr), str, lit(pos))
+}
+
+/// Converts a string column to lowercase.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func lower(_ col: Column) -> Column {
+ return fn("lower", col)
+}
+
+/// Left-pad the string column with `pad` to a length of `len`. If the string
column is longer
+/// than `len`, the return value is shortened to `len` characters.
+/// - Parameters:
+/// - str: A ``Column`` to pad.
+/// - len: A length to pad to.
+/// - pad: A pad string.
+/// - Returns: A ``Column``.
+public func lpad(_ str: Column, _ len: Int32, _ pad: String) -> Column {
+ return lpad(str, lit(len), lit(pad))
+}
+
+/// Left-pad the string column with `pad` to a length of `len`. If the string
column is longer
+/// than `len`, the return value is shortened to `len` characters.
+/// - Parameters:
+/// - str: A ``Column`` to pad.
+/// - len: A length ``Column`` to pad to.
+/// - pad: A pad ``Column``.
+/// - Returns: A ``Column``.
+public func lpad(_ str: Column, _ len: Column, _ pad: Column) -> Column {
+ return fn("lpad", str, len, pad)
+}
+
+/// Trim the spaces from left end for the specified string value.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func ltrim(_ col: Column) -> Column {
+ return fn("ltrim", col)
+}
+
+/// Trim the specified character string from left end for the specified string
column.
+/// - Parameters:
+/// - col: A ``Column`` to trim.
+/// - trimString: The trim string characters to trim.
+/// - Returns: A ``Column``.
+public func ltrim(_ col: Column, _ trimString: String) -> Column {
+ return ltrim(col, lit(trimString))
+}
+
+/// Trim the specified character string from left end for the specified string
column.
+/// - Parameters:
+/// - col: A ``Column`` to trim.
+/// - trim: A ``Column`` of the trim string characters to trim.
+/// - Returns: A ``Column``.
+public func ltrim(_ col: Column, _ trim: Column) -> Column {
+ return fn("ltrim", trim, col)
+}
+
+/// Returns a new string in which all invalid UTF-8 byte sequences, if any,
are replaced by the
+/// Unicode replacement character (U+FFFD).
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func make_valid_utf8(_ str: Column) -> Column {
+ return fn("make_valid_utf8", str)
+}
+
+/// Masks the given string value. The function replaces upper-case characters
with 'X',
+/// lower-case characters with 'x', and numbers with 'n'.
+/// - Parameter input: A string ``Column``.
+/// - Returns: A ``Column``.
+public func mask(_ input: Column) -> Column {
+ return fn("mask", input)
+}
+
+/// Masks the given string value. The function replaces upper-case characters
with the
+/// specific character, lower-case characters with 'x', and numbers with 'n'.
+/// - Parameters:
+/// - input: A string ``Column``.
+/// - upperChar: A character ``Column`` to replace upper-case characters
with.
+/// - Returns: A ``Column``.
+public func mask(_ input: Column, _ upperChar: Column) -> Column {
+ return fn("mask", input, upperChar)
+}
+
+/// Masks the given string value. The function replaces upper-case and
lower-case characters
+/// with the characters specified respectively, and numbers with 'n'.
+/// - Parameters:
+/// - input: A string ``Column``.
+/// - upperChar: A character ``Column`` to replace upper-case characters
with.
+/// - lowerChar: A character ``Column`` to replace lower-case characters
with.
+/// - Returns: A ``Column``.
+public func mask(_ input: Column, _ upperChar: Column, _ lowerChar: Column) ->
Column {
+ return fn("mask", input, upperChar, lowerChar)
+}
+
+/// Masks the given string value. The function replaces upper-case, lower-case
characters and
+/// numbers with the characters specified respectively.
+/// - Parameters:
+/// - input: A string ``Column``.
+/// - upperChar: A character ``Column`` to replace upper-case characters
with.
+/// - lowerChar: A character ``Column`` to replace lower-case characters
with.
+/// - digitChar: A character ``Column`` to replace digit characters with.
+/// - Returns: A ``Column``.
+public func mask(_ input: Column, _ upperChar: Column, _ lowerChar: Column, _
digitChar: Column)
+ -> Column
+{
+ return fn("mask", input, upperChar, lowerChar, digitChar)
+}
+
+/// Masks the given string value. The function replaces upper-case, lower-case
characters,
+/// numbers and other characters with the characters specified respectively.
+/// - Parameters:
+/// - input: A string ``Column``.
+/// - upperChar: A character ``Column`` to replace upper-case characters
with.
+/// - lowerChar: A character ``Column`` to replace lower-case characters
with.
+/// - digitChar: A character ``Column`` to replace digit characters with.
+/// - otherChar: A character ``Column`` to replace all other characters with.
+/// - Returns: A ``Column``.
+public func mask(
+ _ input: Column, _ upperChar: Column, _ lowerChar: Column, _ digitChar:
Column,
+ _ otherChar: Column
+) -> Column {
+ return fn("mask", input, upperChar, lowerChar, digitChar, otherChar)
+}
+
+/// Calculates the byte length for the specified string column.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func octet_length(_ col: Column) -> Column {
+ return fn("octet_length", col)
+}
+
+/// Overlay the specified portion of `src` with `replace`, starting from byte
position `pos`
+/// of `src`.
+/// - Parameters:
+/// - src: A ``Column`` to overlay.
+/// - replace: A replacement ``Column``.
+/// - pos: A position ``Column``.
+/// - Returns: A ``Column``.
+public func overlay(_ src: Column, _ replace: Column, _ pos: Column) -> Column
{
+ return fn("overlay", src, replace, pos)
+}
+
+/// Overlay the specified portion of `src` with `replace`, starting from byte
position `pos`
+/// of `src` and proceeding for `len` bytes.
+/// - Parameters:
+/// - src: A ``Column`` to overlay.
+/// - replace: A replacement ``Column``.
+/// - pos: A position ``Column``.
+/// - len: A length ``Column``.
+/// - Returns: A ``Column``.
+public func overlay(_ src: Column, _ replace: Column, _ pos: Column, _ len:
Column) -> Column {
+ return fn("overlay", src, replace, pos, len)
+}
+
+/// Returns the position of the first occurrence of `substr` in `str` after
position `1`. The
+/// return value are 1-based.
+/// - Parameters:
+/// - substr: A substring ``Column`` to search for.
+/// - str: A ``Column`` to search in.
+/// - Returns: A ``Column``.
+public func position(_ substr: Column, _ str: Column) -> Column {
+ return fn("position", substr, str)
+}
+
+/// Returns the position of the first occurrence of `substr` in `str` after
position `start`.
+/// The given `start` and return value are 1-based.
+/// - Parameters:
+/// - substr: A substring ``Column`` to search for.
+/// - str: A ``Column`` to search in.
+/// - start: A start position ``Column`` of the search.
+/// - Returns: A ``Column``.
+public func position(_ substr: Column, _ str: Column, _ start: Column) ->
Column {
+ return fn("position", substr, str, start)
+}
+
+/// Formats the arguments in printf-style and returns the result as a string
column.
+/// - Parameters:
+/// - format: A printf-style format ``Column``.
+/// - arguments: ``Column``s to format.
+/// - Returns: A ``Column``.
+public func printf(_ format: Column, _ arguments: Column...) -> Column {
+ return fn("printf", [format] + arguments)
+}
+
+/// Returns `str` enclosed by single quotes and each instance of single quote
in it is
+/// preceded by a backslash.
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func quote(_ str: Column) -> Column {
+ return fn("quote", str)
+}
+
+/// Returns a string of the specified length whose characters are chosen
uniformly at random
+/// from the following pool of characters: 0-9, a-z, A-Z.
+/// - Parameter length: A length ``Column``.
+/// - Returns: A ``Column``.
+public func randstr(_ length: Column) -> Column {
+ return fn("randstr", length)
+}
+
+/// Returns a string of the specified length whose characters are chosen
uniformly at random
+/// from the following pool of characters: 0-9, a-z, A-Z, with the chosen
random seed.
+/// - Parameters:
+/// - length: A length ``Column``.
+/// - seed: A random seed ``Column``.
+/// - Returns: A ``Column``.
+public func randstr(_ length: Column, _ seed: Column) -> Column {
+ return fn("randstr", length, seed)
+}
+
+/// Returns a count of the number of times that the regular expression pattern
`regexp` is
+/// matched in the string `str`.
+/// - Parameters:
+/// - str: A ``Column`` to match.
+/// - regexp: A regular expression ``Column``.
+/// - Returns: A ``Column``.
+public func regexp_count(_ str: Column, _ regexp: Column) -> Column {
+ return fn("regexp_count", str, regexp)
+}
+
+/// Extract a specific group matched by a Java regex, from the specified
string column. If the
+/// regex did not match, or the specified group did not match, an empty string
is returned.
+/// - Parameters:
+/// - col: A ``Column`` to match.
+/// - exp: A regular expression.
+/// - groupIdx: A group index.
+/// - Returns: A ``Column``.
+public func regexp_extract(_ col: Column, _ exp: String, _ groupIdx: Int32) ->
Column {
+ return fn("regexp_extract", col, lit(exp), lit(groupIdx))
+}
+
+/// Extract all strings in the `str` that match the `regexp` expression and
corresponding to
+/// the first regex group index.
+/// - Parameters:
+/// - str: A ``Column`` to match.
+/// - regexp: A regular expression ``Column``.
+/// - Returns: A ``Column``.
+public func regexp_extract_all(_ str: Column, _ regexp: Column) -> Column {
+ return fn("regexp_extract_all", str, regexp)
+}
+
+/// Extract all strings in the `str` that match the `regexp` expression and
corresponding to
+/// the regex group index.
+/// - Parameters:
+/// - str: A ``Column`` to match.
+/// - regexp: A regular expression ``Column``.
+/// - idx: A group index ``Column``.
+/// - Returns: A ``Column``.
+public func regexp_extract_all(_ str: Column, _ regexp: Column, _ idx: Column)
-> Column {
+ return fn("regexp_extract_all", str, regexp, idx)
+}
+
+/// Searches a string for a regular expression and returns an integer that
indicates the
+/// beginning position of the matched substring. Positions are 1-based, not
0-based. If no
+/// match is found, returns 0.
+/// - Parameters:
+/// - str: A ``Column`` to match.
+/// - regexp: A regular expression ``Column``.
+/// - Returns: A ``Column``.
+public func regexp_instr(_ str: Column, _ regexp: Column) -> Column {
+ return fn("regexp_instr", str, regexp)
+}
+
+/// Searches a string for a regular expression and returns an integer that
indicates the
+/// beginning position of the matched substring. Positions are 1-based, not
0-based. If no
+/// match is found, returns 0.
+/// - Parameters:
+/// - str: A ``Column`` to match.
+/// - regexp: A regular expression ``Column``.
+/// - idx: A group index ``Column``.
+/// - Returns: A ``Column``.
+public func regexp_instr(_ str: Column, _ regexp: Column, _ idx: Column) ->
Column {
+ return fn("regexp_instr", str, regexp, idx)
+}
+
+/// Replace all substrings of the specified string value that match regexp
with rep.
+/// - Parameters:
+/// - col: A ``Column`` to match.
+/// - pattern: A regular expression.
+/// - replacement: A replacement string.
+/// - Returns: A ``Column``.
+public func regexp_replace(_ col: Column, _ pattern: String, _ replacement:
String) -> Column {
+ return regexp_replace(col, lit(pattern), lit(replacement))
+}
+
+/// Replace all substrings of the specified string value that match regexp
with rep, starting
+/// at the specified 1-based position `pos`.
+/// - Parameters:
+/// - col: A ``Column`` to match.
+/// - pattern: A regular expression.
+/// - replacement: A replacement string.
+/// - pos: A start position of the search.
+/// - Returns: A ``Column``.
+public func regexp_replace(
+ _ col: Column, _ pattern: String, _ replacement: String, _ pos: Int32
+) -> Column {
+ return regexp_replace(col, lit(pattern), lit(replacement), lit(pos))
+}
+
+/// Replace all substrings of the specified string value that match regexp
with rep.
+/// - Parameters:
+/// - col: A ``Column`` to match.
+/// - pattern: A regular expression ``Column``.
+/// - replacement: A replacement ``Column``.
+/// - Returns: A ``Column``.
+public func regexp_replace(_ col: Column, _ pattern: Column, _ replacement:
Column) -> Column {
+ return fn("regexp_replace", col, pattern, replacement)
+}
+
+/// Replace all substrings of the specified string value that match regexp
with rep, starting
+/// at the specified 1-based position `pos`.
+/// - Parameters:
+/// - col: A ``Column`` to match.
+/// - pattern: A regular expression ``Column``.
+/// - replacement: A replacement ``Column``.
+/// - pos: A start position ``Column`` of the search.
+/// - Returns: A ``Column``.
+public func regexp_replace(
+ _ col: Column, _ pattern: Column, _ replacement: Column, _ pos: Column
+) -> Column {
+ return fn("regexp_replace", col, pattern, replacement, pos)
+}
+
+/// Returns the substring that matches the regular expression `regexp` within
the string `str`.
+/// If the regular expression is not found, the result is null.
+/// - Parameters:
+/// - str: A ``Column`` to match.
+/// - regexp: A regular expression ``Column``.
+/// - Returns: A ``Column``.
+public func regexp_substr(_ str: Column, _ regexp: Column) -> Column {
+ return fn("regexp_substr", str, regexp)
+}
+
+/// Repeats a string column `n` times, and returns it as a new string column.
+/// - Parameters:
+/// - str: A ``Column`` to repeat.
+/// - n: The number of times to repeat.
+/// - Returns: A ``Column``.
+public func `repeat`(_ str: Column, _ n: Int32) -> Column {
+ return fn("repeat", str, lit(n))
+}
+
+/// Repeats a string column `n` times, and returns it as a new string column.
+/// - Parameters:
+/// - str: A ``Column`` to repeat.
+/// - n: A ``Column`` for the number of times to repeat.
+/// - Returns: A ``Column``.
+public func `repeat`(_ str: Column, _ n: Column) -> Column {
+ return fn("repeat", str, n)
+}
+
+/// Removes all occurrences of `search` from `src`.
+/// - Parameters:
+/// - src: A ``Column`` to replace.
+/// - search: A ``Column`` to search for.
+/// - Returns: A ``Column``.
+public func replace(_ src: Column, _ search: Column) -> Column {
+ return fn("replace", src, search)
+}
+
+/// Replaces all occurrences of `search` with `replace`.
+/// - Parameters:
+/// - src: A ``Column`` to replace.
+/// - search: A ``Column`` to search for.
+/// - replace: A replacement ``Column``.
+/// - Returns: A ``Column``.
+public func replace(_ src: Column, _ search: Column, _ replace: Column) ->
Column {
+ return fn("replace", src, search, replace)
+}
+
+/// Returns the rightmost `len` characters from the string `str`. If `len` is
less or equal
+/// than 0 the result is an empty string.
+/// - Parameters:
+/// - str: A ``Column``.
+/// - len: A length ``Column``.
+/// - Returns: A ``Column``.
+public func right(_ str: Column, _ len: Column) -> Column {
+ return fn("right", str, len)
+}
+
+/// Right-pad the string column with `pad` to a length of `len`. If the string
column is longer
+/// than `len`, the return value is shortened to `len` characters.
+/// - Parameters:
+/// - str: A ``Column`` to pad.
+/// - len: A length to pad to.
+/// - pad: A pad string.
+/// - Returns: A ``Column``.
+public func rpad(_ str: Column, _ len: Int32, _ pad: String) -> Column {
+ return rpad(str, lit(len), lit(pad))
+}
+
+/// Right-pad the string column with `pad` to a length of `len`. If the string
column is longer
+/// than `len`, the return value is shortened to `len` characters.
+/// - Parameters:
+/// - str: A ``Column`` to pad.
+/// - len: A length ``Column`` to pad to.
+/// - pad: A pad ``Column``.
+/// - Returns: A ``Column``.
+public func rpad(_ str: Column, _ len: Column, _ pad: Column) -> Column {
+ return fn("rpad", str, len, pad)
+}
+
+/// Trim the spaces from right end for the specified string value.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func rtrim(_ col: Column) -> Column {
+ return fn("rtrim", col)
+}
+
+/// Trim the specified character string from right end for the specified
string column.
+/// - Parameters:
+/// - col: A ``Column`` to trim.
+/// - trimString: The trim string characters to trim.
+/// - Returns: A ``Column``.
+public func rtrim(_ col: Column, _ trimString: String) -> Column {
+ return rtrim(col, lit(trimString))
+}
+
+/// Trim the specified character string from right end for the specified
string column.
+/// - Parameters:
+/// - col: A ``Column`` to trim.
+/// - trim: A ``Column`` of the trim string characters to trim.
+/// - Returns: A ``Column``.
+public func rtrim(_ col: Column, _ trim: Column) -> Column {
+ return fn("rtrim", trim, col)
+}
+
+/// Splits a string into arrays of sentences, where each sentence is an array
of words. The
+/// default locale is used.
+/// - Parameter string: A string ``Column`` to be split.
+/// - Returns: A ``Column``.
+public func sentences(_ string: Column) -> Column {
+ return fn("sentences", string)
+}
+
+/// Splits a string into arrays of sentences, where each sentence is an array
of words. The
+/// default `country`('') is used.
+/// - Parameters:
+/// - string: A string ``Column`` to be split.
+/// - language: A language ``Column`` of the locale.
+/// - Returns: A ``Column``.
+public func sentences(_ string: Column, _ language: Column) -> Column {
+ return fn("sentences", string, language)
+}
+
+/// Splits a string into arrays of sentences, where each sentence is an array
of words.
+/// - Parameters:
+/// - string: A string ``Column`` to be split.
+/// - language: A language ``Column`` of the locale.
+/// - country: A country ``Column`` of the locale.
+/// - Returns: A ``Column``.
+public func sentences(_ string: Column, _ language: Column, _ country: Column)
-> Column {
+ return fn("sentences", string, language, country)
+}
+
+/// Returns the soundex code for the specified expression.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func soundex(_ col: Column) -> Column {
+ return fn("soundex", col)
+}
+
+/// Splits str around matches of the given pattern.
+/// - Parameters:
+/// - str: A string ``Column`` to split.
+/// - pattern: A regular expression to split.
+/// - Returns: A ``Column``.
+public func split(_ str: Column, _ pattern: String) -> Column {
+ return fn("split", str, lit(pattern))
+}
+
+/// Splits str around matches of the given pattern.
+/// - Parameters:
+/// - str: A string ``Column`` to split.
+/// - pattern: A regular expression ``Column`` to split.
+/// - Returns: A ``Column``.
+public func split(_ str: Column, _ pattern: Column) -> Column {
+ return fn("split", str, pattern)
+}
+
+/// Splits str around matches of the given pattern up to the `limit`. If
`limit` is positive,
+/// the resulting array's length will not be more than `limit`. If `limit` is
negative, the
+/// resulting array can be of any size.
+/// - Parameters:
+/// - str: A string ``Column`` to split.
+/// - pattern: A regular expression to split.
+/// - limit: A limit of the split.
+/// - Returns: A ``Column``.
+public func split(_ str: Column, _ pattern: String, _ limit: Int32) -> Column {
+ return fn("split", str, lit(pattern), lit(limit))
+}
+
+/// Splits str around matches of the given pattern up to the `limit`. If
`limit` is positive,
+/// the resulting array's length will not be more than `limit`. If `limit` is
negative, the
+/// resulting array can be of any size.
+/// - Parameters:
+/// - str: A string ``Column`` to split.
+/// - pattern: A regular expression ``Column`` to split.
+/// - limit: A limit ``Column`` of the split.
+/// - Returns: A ``Column``.
+public func split(_ str: Column, _ pattern: Column, _ limit: Column) -> Column
{
+ return fn("split", str, pattern, limit)
+}
+
+/// Splits `str` by delimiter and return requested part of the split
(1-based). If any input is
+/// null, returns null. If the index is out of range of split parts, returns
empty string.
+/// - Parameters:
+/// - str: A string ``Column`` to split.
+/// - delimiter: A delimiter ``Column``.
+/// - partNum: A part number ``Column``.
+/// - Returns: A ``Column``.
+public func split_part(_ str: Column, _ delimiter: Column, _ partNum: Column)
-> Column {
+ return fn("split_part", str, delimiter, partNum)
+}
+
+/// Returns a boolean. The value is true if `str` starts with `prefix`.
Returns NULL if either
+/// input expression is NULL. Otherwise, returns false.
+/// - Parameters:
+/// - str: A ``Column`` to search in.
+/// - prefix: A prefix ``Column``.
+/// - Returns: A ``Column``.
+public func startswith(_ str: Column, _ prefix: Column) -> Column {
+ return fn("startswith", str, prefix)
+}
+
+/// Returns the substring of `str` that starts at `pos`, or the slice of byte
array that starts
+/// at `pos`.
+/// - Parameters:
+/// - str: A ``Column``.
+/// - pos: A position ``Column``.
+/// - Returns: A ``Column``.
+public func substr(_ str: Column, _ pos: Column) -> Column {
+ return fn("substr", str, pos)
+}
+
+/// Returns the substring of `str` that starts at `pos` and is of length
`len`, or the slice of
+/// byte array that starts at `pos` and is of length `len`.
+/// - Parameters:
+/// - str: A ``Column``.
+/// - pos: A position ``Column``.
+/// - len: A length ``Column``.
+/// - Returns: A ``Column``.
+public func substr(_ str: Column, _ pos: Column, _ len: Column) -> Column {
+ return fn("substr", str, pos, len)
+}
+
+/// Substring starts at `pos` and is of length `len` when str is String type
or returns the
+/// slice of byte array that starts at `pos` in byte and is of length `len`
when str is Binary
+/// type.
+/// - Parameters:
+/// - str: A ``Column``.
+/// - pos: A position.
+/// - len: A length.
+/// - Returns: A ``Column``.
+public func substring(_ str: Column, _ pos: Int32, _ len: Int32) -> Column {
+ return fn("substring", str, lit(pos), lit(len))
+}
+
+/// Substring starts at `pos` and is of length `len` when str is String type
or returns the
+/// slice of byte array that starts at `pos` in byte and is of length `len`
when str is Binary
+/// type.
+/// - Parameters:
+/// - str: A ``Column``.
+/// - pos: A position ``Column``.
+/// - len: A length ``Column``.
+/// - Returns: A ``Column``.
+public func substring(_ str: Column, _ pos: Column, _ len: Column) -> Column {
+ return fn("substring", str, pos, len)
+}
+
+/// Returns the substring from string str before count occurrences of the
delimiter delim. If
+/// count is positive, everything the left of the final delimiter (counting
from left) is
+/// returned. If count is negative, every to the right of the final delimiter
(counting from
+/// the right) is returned.
+/// - Parameters:
+/// - str: A ``Column``.
+/// - delim: A delimiter string.
+/// - count: The number of occurrences.
+/// - Returns: A ``Column``.
+public func substring_index(_ str: Column, _ delim: String, _ count: Int32) ->
Column {
+ return fn("substring_index", str, lit(delim), lit(count))
+}
+
+/// Converts the input `col` to a binary value based on the default format
'hex'.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func to_binary(_ col: Column) -> Column {
+ return fn("to_binary", col)
+}
+
+/// Converts the input `col` to a binary value based on the supplied `format`.
The `format` can
+/// be a case-insensitive string literal of 'hex', 'utf-8', 'utf8', or
'base64'.
+/// - Parameters:
+/// - col: A ``Column``.
+/// - format: A format ``Column``.
+/// - Returns: A ``Column``.
+public func to_binary(_ col: Column, _ format: Column) -> Column {
+ return fn("to_binary", col, format)
+}
+
+/// Convert `col` to a string based on the `format`.
+/// - Parameters:
+/// - col: A ``Column``.
+/// - format: A format ``Column``.
+/// - Returns: A ``Column``.
+public func to_char(_ col: Column, _ format: Column) -> Column {
+ return fn("to_char", col, format)
+}
+
+/// Convert string `col` to a number based on the string format `format`.
+/// - Parameters:
+/// - col: A ``Column``.
+/// - format: A format ``Column``.
+/// - Returns: A ``Column``.
+public func to_number(_ col: Column, _ format: Column) -> Column {
+ return fn("to_number", col, format)
+}
+
+/// Convert `col` to a string based on the `format`.
+/// - Parameters:
+/// - col: A ``Column``.
+/// - format: A format ``Column``.
+/// - Returns: A ``Column``.
+public func to_varchar(_ col: Column, _ format: Column) -> Column {
+ return fn("to_varchar", col, format)
+}
+
+/// Translate any character in the src by a character in replaceString. The
characters in
+/// replaceString correspond to the characters in matchingString. The
translate will happen
+/// when any character in the string matches the character in the
`matchingString`.
+/// - Parameters:
+/// - src: A ``Column`` to translate.
+/// - matchingString: A matching string.
+/// - replaceString: A replacement string.
+/// - Returns: A ``Column``.
+public func translate(_ src: Column, _ matchingString: String, _
replaceString: String) -> Column
+{
+ return fn("translate", src, lit(matchingString), lit(replaceString))
+}
+
+/// Trim the spaces from both ends for the specified string column.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func trim(_ col: Column) -> Column {
+ return fn("trim", col)
+}
+
+/// Trim the specified character from both ends for the specified string
column.
+/// - Parameters:
+/// - col: A ``Column`` to trim.
+/// - trimString: The trim string characters to trim.
+/// - Returns: A ``Column``.
+public func trim(_ col: Column, _ trimString: String) -> Column {
+ return trim(col, lit(trimString))
+}
+
+/// Trim the specified character from both ends for the specified string
column.
+/// - Parameters:
+/// - col: A ``Column`` to trim.
+/// - trim: A ``Column`` of the trim string characters to trim.
+/// - Returns: A ``Column``.
+public func trim(_ col: Column, _ trim: Column) -> Column {
+ return fn("trim", trim, col)
+}
+
+/// Converts the input `col` to a binary value based on the default format
'hex'. The function
+/// returns NULL if at least one of the input parameters is NULL.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func try_to_binary(_ col: Column) -> Column {
+ return fn("try_to_binary", col)
+}
+
+/// Converts the input `col` to a binary value based on the supplied `format`.
The `format` can
+/// be a case-insensitive string literal of 'hex', 'utf-8', 'utf8', or
'base64'. The function
+/// returns NULL if at least one of the input parameters is NULL.
+/// - Parameters:
+/// - col: A ``Column``.
+/// - format: A format ``Column``.
+/// - Returns: A ``Column``.
+public func try_to_binary(_ col: Column, _ format: Column) -> Column {
+ return fn("try_to_binary", col, format)
+}
+
+/// Convert string `col` to a number based on the string format `format`.
Returns NULL if the
+/// string `col` does not match the expected format.
+/// - Parameters:
+/// - col: A ``Column``.
+/// - format: A format ``Column``.
+/// - Returns: A ``Column``.
+public func try_to_number(_ col: Column, _ format: Column) -> Column {
+ return fn("try_to_number", col, format)
+}
+
+/// Returns the input value if it corresponds to a valid UTF-8 string, or NULL
otherwise.
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func try_validate_utf8(_ str: Column) -> Column {
+ return fn("try_validate_utf8", str)
+}
+
+/// Converts a string column to uppercase.
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func ucase(_ str: Column) -> Column {
+ return fn("ucase", str)
+}
+
+/// Decodes a BASE64 encoded string column and returns it as a binary column.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func unbase64(_ col: Column) -> Column {
+ return fn("unbase64", col)
+}
+
+/// Converts a string column to uppercase.
+/// - Parameter col: A ``Column``.
+/// - Returns: A ``Column``.
+public func upper(_ col: Column) -> Column {
+ return fn("upper", col)
+}
+
+/// Returns the input value if it corresponds to a valid UTF-8 string, or
emits an error
+/// otherwise.
+/// - Parameter str: A ``Column``.
+/// - Returns: A ``Column``.
+public func validate_utf8(_ str: Column) -> Column {
+ return fn("validate_utf8", str)
+}
diff --git a/Tests/SparkConnectTests/StringFunctionsTests.swift
b/Tests/SparkConnectTests/StringFunctionsTests.swift
new file mode 100644
index 0000000..7865f67
--- /dev/null
+++ b/Tests/SparkConnectTests/StringFunctionsTests.swift
@@ -0,0 +1,393 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+import Testing
+
+@testable import SparkConnect
+
+/// A test suite for `StringFunctions`
+@Suite(.serialized)
+struct StringFunctionsTests {
+
+ @Test
+ func stringFunctions() throws {
+ for (column, name) in [
+ (ascii(col("a")), "ascii"),
+ (base64(col("a")), "base64"),
+ (bit_length(col("a")), "bit_length"),
+ (btrim(col("a")), "btrim"),
+ (char(col("a")), "char"),
+ (char_length(col("a")), "char_length"),
+ (character_length(col("a")), "character_length"),
+ (chr(col("a")), "chr"),
+ (collation(col("a")), "collation"),
+ (initcap(col("a")), "initcap"),
+ (is_valid_utf8(col("a")), "is_valid_utf8"),
+ (lcase(col("a")), "lcase"),
+ (len(col("a")), "len"),
+ (length(col("a")), "length"),
+ (lower(col("a")), "lower"),
+ (ltrim(col("a")), "ltrim"),
+ (make_valid_utf8(col("a")), "make_valid_utf8"),
+ (mask(col("a")), "mask"),
+ (octet_length(col("a")), "octet_length"),
+ (quote(col("a")), "quote"),
+ (randstr(col("a")), "randstr"),
+ (rtrim(col("a")), "rtrim"),
+ (sentences(col("a")), "sentences"),
+ (soundex(col("a")), "soundex"),
+ (to_binary(col("a")), "to_binary"),
+ (trim(col("a")), "trim"),
+ (try_to_binary(col("a")), "try_to_binary"),
+ (try_validate_utf8(col("a")), "try_validate_utf8"),
+ (ucase(col("a")), "ucase"),
+ (unbase64(col("a")), "unbase64"),
+ (upper(col("a")), "upper"),
+ (validate_utf8(col("a")), "validate_utf8"),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+
#expect(expr.unresolvedFunction.arguments[0].unresolvedAttribute.unparsedIdentifier
== "a")
+ }
+ }
+
+ @Test
+ func stringFunctionArguments() throws {
+ for (column, name) in [
+ (btrim(col("a"), col("b")), "btrim"),
+ (contains(col("a"), col("b")), "contains"),
+ (endswith(col("a"), col("b")), "endswith"),
+ (find_in_set(col("a"), col("b")), "find_in_set"),
+ (instr(col("a"), col("b")), "instr"),
+ (jaro_winkler_similarity(col("a"), col("b")), "jaro_winkler_similarity"),
+ (left(col("a"), col("b")), "left"),
+ (levenshtein(col("a"), col("b")), "levenshtein"),
+ (mask(col("a"), col("b")), "mask"),
+ (position(col("a"), col("b")), "position"),
+ (randstr(col("a"), col("b")), "randstr"),
+ (regexp_count(col("a"), col("b")), "regexp_count"),
+ (regexp_extract_all(col("a"), col("b")), "regexp_extract_all"),
+ (regexp_instr(col("a"), col("b")), "regexp_instr"),
+ (regexp_substr(col("a"), col("b")), "regexp_substr"),
+ (`repeat`(col("a"), col("b")), "repeat"),
+ (replace(col("a"), col("b")), "replace"),
+ (right(col("a"), col("b")), "right"),
+ (sentences(col("a"), col("b")), "sentences"),
+ (split(col("a"), col("b")), "split"),
+ (startswith(col("a"), col("b")), "startswith"),
+ (substr(col("a"), col("b")), "substr"),
+ (to_binary(col("a"), col("b")), "to_binary"),
+ (to_char(col("a"), col("b")), "to_char"),
+ (to_number(col("a"), col("b")), "to_number"),
+ (to_varchar(col("a"), col("b")), "to_varchar"),
+ (try_to_binary(col("a"), col("b")), "try_to_binary"),
+ (try_to_number(col("a"), col("b")), "try_to_number"),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments.count == 2)
+ }
+
+ for (column, name, count) in [
+ (instr(col("a"), col("b"), 2), "instr", 3),
+ (instr(col("a"), col("b"), lit(2)), "instr", 3),
+ (instr(col("a"), col("b"), 2, 2), "instr", 4),
+ (instr(col("a"), col("b"), lit(2), lit(2)), "instr", 4),
+ (mask(col("a"), lit("X"), lit("x")), "mask", 3),
+ (mask(col("a"), lit("X"), lit("x"), lit("n")), "mask", 4),
+ (mask(col("a"), lit("X"), lit("x"), lit("n"), lit("o")), "mask", 5),
+ (overlay(col("a"), col("b"), lit(1)), "overlay", 3),
+ (overlay(col("a"), col("b"), lit(1), lit(2)), "overlay", 4),
+ (position(col("a"), col("b"), lit(1)), "position", 3),
+ (regexp_extract_all(col("a"), col("b"), lit(1)), "regexp_extract_all",
3),
+ (regexp_instr(col("a"), col("b"), lit(1)), "regexp_instr", 3),
+ (regexp_replace(col("a"), col("b"), col("c")), "regexp_replace", 3),
+ (regexp_replace(col("a"), col("b"), col("c"), lit(1)), "regexp_replace",
4),
+ (sentences(col("a"), col("b"), col("c")), "sentences", 3),
+ (split(col("a"), col("b"), lit(2)), "split", 3),
+ (split_part(col("a"), col("b"), lit(1)), "split_part", 3),
+ (substr(col("a"), col("b"), col("c")), "substr", 3),
+ (substring(col("a"), col("b"), col("c")), "substring", 3),
+ (elt(lit(1), col("a"), col("b")), "elt", 3),
+ (printf(lit("%s"), col("a")), "printf", 2),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments.count == count)
+ }
+ }
+
+ @Test
+ func stringFunctionLiteralArguments() throws {
+ for (column, name, literal) in [
+ (collate(col("a"), "UTF8_BINARY"), "collate", "UTF8_BINARY"),
+ (decode(col("a"), "UTF-8"), "decode", "UTF-8"),
+ (encode(col("a"), "UTF-8"), "encode", "UTF-8"),
+ (instr(col("a"), "b"), "instr", "b"),
+ (split(col("a"), ","), "split", ","),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments[1].literal.string == literal)
+ }
+
+ let concatenated = concat_ws("|", col("a"), col("b")).expr
+ #expect(concatenated.unresolvedFunction.functionName == "concat_ws")
+ #expect(concatenated.unresolvedFunction.arguments[0].literal.string == "|")
+ #expect(concatenated.unresolvedFunction.arguments.count == 3)
+
+ let formatted = format_string("%s-%s", col("a"), col("b")).expr
+ #expect(formatted.unresolvedFunction.functionName == "format_string")
+ #expect(formatted.unresolvedFunction.arguments[0].literal.string ==
"%s-%s")
+ #expect(formatted.unresolvedFunction.arguments.count == 3)
+
+ let located = locate("b", col("a")).expr
+ #expect(located.unresolvedFunction.functionName == "locate")
+ #expect(located.unresolvedFunction.arguments[0].literal.string == "b")
+
+ // The trim string argument comes first for `ltrim`, `rtrim` and `trim`.
+ for (column, name) in [
+ (ltrim(col("a"), "x"), "ltrim"),
+ (ltrim(col("a"), lit("x")), "ltrim"),
+ (rtrim(col("a"), "x"), "rtrim"),
+ (rtrim(col("a"), lit("x")), "rtrim"),
+ (trim(col("a"), "x"), "trim"),
+ (trim(col("a"), lit("x")), "trim"),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments[0].literal.string == "x")
+
#expect(expr.unresolvedFunction.arguments[1].unresolvedAttribute.unparsedIdentifier
== "a")
+ }
+
+ for (column, name, literal) in [
+ (format_number(col("a"), 2), "format_number", Int32(2)),
+ (levenshtein(col("a"), col("b"), 3), "levenshtein", Int32(3)),
+ (locate("b", col("a"), 2), "locate", Int32(2)),
+ (`repeat`(col("a"), 2), "repeat", Int32(2)),
+ (split(col("a"), ",", 2), "split", Int32(2)),
+ (substring_index(col("a"), ".", 2), "substring_index", Int32(2)),
+ ] {
+ let expr = column.expr
+ #expect(expr.unresolvedFunction.functionName == name)
+ #expect(expr.unresolvedFunction.arguments.last!.literal.integer ==
literal)
+ }
+
+ let padded = lpad(col("a"), 5, "*").expr
+ #expect(padded.unresolvedFunction.functionName == "lpad")
+ #expect(padded.unresolvedFunction.arguments[1].literal.integer == 5)
+ #expect(padded.unresolvedFunction.arguments[2].literal.string == "*")
+ #expect(rpad(col("a"), 5, "*").expr.unresolvedFunction.functionName ==
"rpad")
+
+ let extracted = regexp_extract(col("a"), "(\\d+)", 1).expr
+ #expect(extracted.unresolvedFunction.functionName == "regexp_extract")
+ #expect(extracted.unresolvedFunction.arguments[1].literal.string ==
"(\\d+)")
+ #expect(extracted.unresolvedFunction.arguments[2].literal.integer == 1)
+
+ let replaced = regexp_replace(col("a"), "(\\d+)", "num").expr
+ #expect(replaced.unresolvedFunction.functionName == "regexp_replace")
+ #expect(replaced.unresolvedFunction.arguments[2].literal.string == "num")
+ #expect(
+ regexp_replace(col("a"), "(\\d+)", "num",
5).expr.unresolvedFunction.arguments.count == 4)
+
+ let translated = translate(col("a"), "abc", "123").expr
+ #expect(translated.unresolvedFunction.functionName == "translate")
+ #expect(translated.unresolvedFunction.arguments[1].literal.string == "abc")
+ #expect(translated.unresolvedFunction.arguments[2].literal.string == "123")
+
+ let substringed = substring(col("a"), 2, 3).expr
+ #expect(substringed.unresolvedFunction.functionName == "substring")
+ #expect(substringed.unresolvedFunction.arguments[1].literal.integer == 2)
+ #expect(substringed.unresolvedFunction.arguments[2].literal.integer == 3)
+ }
+
+ @Test
+ func selectBasicStringFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ let cases = try await df.select(
+ lower(lit("ABC")), lcase(lit("ABC")), upper(lit("abc")),
ucase(lit("abc")),
+ initcap(lit("spark sql"))
+ ).collect()
+ #expect(cases == [Row("abc", "abc", "ABC", "ABC", "Spark Sql")])
+
+ let lengths = try await df.select(
+ length(lit("Spark")), len(lit("Spark")), char_length(lit("Spark")),
+ character_length(lit("Spark")), bit_length(lit("abc")),
octet_length(lit("abc"))
+ ).collect()
+ #expect(lengths == [Row(5, 5, 5, 5, 24, 3)])
+
+ let characters = try await df.select(
+ ascii(lit("A")), char(lit(65)), chr(lit(65)), soundex(lit("Miller"))
+ ).collect()
+ #expect(characters == [Row(65, "A", "A", "M460")])
+ await spark.stop()
+ }
+
+ @Test
+ func selectTrimAndPadFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ let spaces = try await df.select(
+ trim(lit(" abc ")), ltrim(lit(" abc")), rtrim(lit("abc ")),
btrim(lit(" abc "))
+ ).collect()
+ #expect(spaces == [Row("abc", "abc", "abc", "abc")])
+
+ let trimmed = try await df.select(
+ trim(lit("xxabcxx"), "x"), ltrim(lit("xxabc"), "x"), rtrim(lit("abcxx"),
"x"),
+ btrim(lit("xxabcxx"), lit("x")), trim(lit("yabcy"), lit("y"))
+ ).collect()
+ #expect(trimmed == [Row("abc", "abc", "abc", "abc", "abc")])
+
+ let padded = try await df.select(
+ lpad(lit("hi"), 5, "??"), rpad(lit("hi"), 5, "??"),
+ lpad(lit("hi"), lit(5), lit("*")), rpad(lit("hi"), lit(5), lit("*"))
+ ).collect()
+ #expect(padded == [Row("???hi", "hi???", "***hi", "hi***")])
+ await spark.stop()
+ }
+
+ @Test
+ func selectSearchFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ let positions = try await df.select(
+ instr(lit("SparkSQL"), "SQL"), instr(lit("SparkSQL"), lit("SQL")),
+ locate("SQL", lit("SparkSQL")), locate("SQL", lit("SparkSQLSQL"), 7),
+ position(lit("SQL"), lit("SparkSQL")), position(lit("SQL"),
lit("SparkSQLSQL"), lit(7))
+ ).collect()
+ #expect(positions == [Row(6, 6, 6, 9, 6, 9)])
+
+ let predicates = try await df.select(
+ contains(lit("Spark"), lit("par")), startswith(lit("Spark"), lit("Spa")),
+ endswith(lit("Spark"), lit("ark")), find_in_set(lit("ab"),
lit("abc,b,ab,c,def"))
+ ).collect()
+ #expect(predicates == [Row(true, true, true, 3)])
+
+ let distances = try await df.select(
+ levenshtein(lit("kitten"), lit("sitting")), levenshtein(lit("kitten"),
lit("sitting"), 3)
+ ).collect()
+ #expect(distances == [Row(3, 3)])
+ await spark.stop()
+ }
+
+ @Test
+ func selectTransformFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ let combined = try await df.select(
+ concat_ws("|", lit("a"), lit("b")), format_string("%s-%s", lit("a"),
lit("b")),
+ printf(lit("%s-%s"), lit("a"), lit("b")), format_number(lit(12345.678),
2),
+ elt(lit(2), lit("scala"), lit("java"))
+ ).collect()
+ #expect(combined == [Row("a|b", "a-b", "a-b", "12,345.68", "java")])
+
+ let substrings = try await df.select(
+ substring(lit("Spark SQL"), 5, 1), substring(lit("Spark SQL"), lit(7),
lit(3)),
+ substr(lit("Spark SQL"), lit(5)), substr(lit("Spark SQL"), lit(5),
lit(1)),
+ substring_index(lit("a.b.c"), ".", 2), left(lit("Spark SQL"), lit(3)),
+ right(lit("Spark SQL"), lit(3))
+ ).collect()
+ #expect(substrings == [Row("k", "SQL", "k SQL", "k", "a.b", "Spa", "SQL")])
+
+ let replaced = try await df.select(
+ replace(lit("ABCabc"), lit("abc"), lit("DEF")), replace(lit("ABCabc"),
lit("abc")),
+ translate(lit("AaBbCc"), "abc", "123"), `repeat`(lit("ab"), 2),
`repeat`(lit("ab"), lit(3)),
+ overlay(lit("SPARK_SQL"), lit("CORE"), lit(7)),
+ overlay(lit("SPARK_SQL"), lit("ANSI "), lit(7), lit(0))
+ ).collect()
+ #expect(replaced == [Row("ABCDEF", "ABC", "A1B2C3", "abab", "ababab",
"SPARK_CORE", "SPARK_ANSI SQL")])
+
+ let masked = try await df.select(
+ mask(lit("AbCD123-@$#")), mask(lit("AbCD123-@$#"), lit("Q")),
+ mask(lit("AbCD123-@$#"), lit("Q"), lit("q"), lit("d"), lit("o"))
+ ).collect()
+ #expect(masked == [Row("XxXXnnn-@$#", "QxQQnnn-@$#", "QqQQdddoooo")])
+ await spark.stop()
+ }
+
+ @Test
+ func selectRegexpFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ let extracted = try await df.select(
+ regexp_count(lit("banana"), lit("a")), regexp_extract(lit("100-200"),
"(\\d+)-(\\d+)", 1),
+ regexp_extract_all(lit("100-200"), lit("(\\d+)")).cast("string"),
+ regexp_instr(lit("[email protected]"), lit("@[^.]*")),
+ regexp_substr(lit("100-200"), lit("(\\d+)"))
+ ).collect()
+ #expect(extracted == [Row(3, "100", "[100, 200]", 5, "100")])
+
+ let replaced = try await df.select(
+ regexp_replace(lit("100-200"), "(\\d+)", "num"),
+ regexp_replace(lit("100-200"), lit("(\\d+)"), lit("num")),
+ regexp_replace(lit("100-200"), "(\\d+)", "num", 5)
+ ).collect()
+ #expect(replaced == [Row("num-num", "num-num", "100-num")])
+
+ let separated = try await df.select(
+ split(lit("oneAtwoBthree"), "[AB]").cast("string"),
+ split(lit("oneAtwoBthree"), lit("[AB]")).cast("string"),
+ split(lit("oneAtwoBthree"), "[AB]", 2).cast("string"),
+ split_part(lit("11.12.13"), lit("."), lit(3)),
+ sentences(lit("Hi there! Good morning.")).cast("string")
+ ).collect()
+ #expect(
+ separated == [
+ Row(
+ "[one, two, three]", "[one, two, three]", "[one, twoBthree]", "13",
+ "[[Hi, there], [Good, morning]]")
+ ])
+ await spark.stop()
+ }
+
+ @Test
+ func selectConversionFunctions() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.range(1)
+ let encoded = try await df.select(
+ base64(lit("abc")), unbase64(lit("YWJj")).cast("string"),
+ decode(encode(lit("abc"), "UTF-8"), "UTF-8"), encode(lit("abc"),
"UTF-8").cast("string")
+ ).collect()
+ #expect(encoded == [Row("YWJj", "abc", "abc", "abc")])
+
+ let converted = try await df.select(
+ to_binary(lit("616263")).cast("string"), to_binary(lit("abc"),
lit("utf-8")).cast("string"),
+ try_to_binary(lit("616263")).cast("string"),
+ try_to_binary(lit("abc"), lit("utf-8")).cast("string"),
+ to_char(lit(454), lit("999")), to_varchar(lit(454), lit("999")),
+ to_number(lit("454"), lit("999")).cast("int"),
+ try_to_number(lit("454"), lit("999")).cast("int")
+ ).collect()
+ #expect(converted == [Row("abc", "abc", "abc", "abc", "454", "454", 454,
454)])
+
+ let utf8 = try await df.select(
+ is_valid_utf8(lit("abc")), make_valid_utf8(lit("abc")),
validate_utf8(lit("abc")),
+ try_validate_utf8(lit("abc"))
+ ).collect()
+ #expect(utf8 == [Row(true, "abc", "abc", "abc")])
+
+ let others = try await df.select(
+ collate(lit("abc"), "UTF8_BINARY"), char_length(randstr(lit(5))),
+ char_length(randstr(lit(5), lit(0)))
+ ).collect()
+ #expect(others == [Row("abc", 5, 5)])
+ await spark.stop()
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]