This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new bebd5e1 [SPARK-51572] Support `binary` type in `show` and `collect`
bebd5e1 is described below
commit bebd5e1abc1b504a0640a905afccb0eee7ebcc15
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Mar 20 20:49:53 2025 -0700
[SPARK-51572] Support `binary` type in `show` and `collect`
### What changes were proposed in this pull request?
This PR aims to support `binary` type in `show` and `collect`.
### Why are the changes needed?
Like the other client, binary type values shows in an array style.
```python
$ bin/pyspark --remote sc://localhost:15002
>>> sql("SELECT binary('abc')").show()
+----------+
| abc|
+----------+
|[61 62 63]|
+----------+
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #26 from dongjoon-hyun/SPARK-51572.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/DataFrame.swift | 6 ++++++
Tests/SparkConnectTests/Resources/queries/binary.sql | 1 +
Tests/SparkConnectTests/Resources/queries/binary.sql.json | 1 +
3 files changed, 8 insertions(+)
diff --git a/Sources/SparkConnect/DataFrame.swift
b/Sources/SparkConnect/DataFrame.swift
index 81c92e4..ad833de 100644
--- a/Sources/SparkConnect/DataFrame.swift
+++ b/Sources/SparkConnect/DataFrame.swift
@@ -173,6 +173,9 @@ public actor DataFrame: Sendable {
let str = column.array as! AsString
if column.data.isNull(i) {
values.append(nil)
+ } else if column.data.type.info == ArrowType.ArrowBinary {
+ let binary = str.asString(i).utf8.map { String(format: "%02x", $0)
}.joined(separator: " ")
+ values.append("[\(binary)]")
} else {
values.append(str.asString(i))
}
@@ -201,6 +204,9 @@ public actor DataFrame: Sendable {
let str = column.array as! AsString
if column.data.isNull(i) {
values.append("NULL")
+ } else if column.data.type.info == ArrowType.ArrowBinary {
+ let binary = str.asString(i).utf8.map { String(format: "%02x",
$0) }.joined(separator: " ")
+ values.append("[\(binary)]")
} else {
values.append(str.asString(i))
}
diff --git a/Tests/SparkConnectTests/Resources/queries/binary.sql
b/Tests/SparkConnectTests/Resources/queries/binary.sql
new file mode 100644
index 0000000..9d5e14f
--- /dev/null
+++ b/Tests/SparkConnectTests/Resources/queries/binary.sql
@@ -0,0 +1 @@
+SELECT binary('abc')
diff --git a/Tests/SparkConnectTests/Resources/queries/binary.sql.json
b/Tests/SparkConnectTests/Resources/queries/binary.sql.json
new file mode 100644
index 0000000..08434e6
--- /dev/null
+++ b/Tests/SparkConnectTests/Resources/queries/binary.sql.json
@@ -0,0 +1 @@
+[["[61 62 63]"]]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
