dongjoon-hyun opened a new pull request, #445:
URL: https://github.com/apache/spark-connect-swift/pull/445
### What changes were proposed in this pull request?
This PR adds Arrow `Null` type support to the vendored Arrow reader so that
collecting a result with a NULL-typed (`void`) column returns `nil` values
instead of crashing.
- `ArrowType.swift`: add `ArrowType.ArrowNull` and handle `.null` in
`getStride()` (previously `fatalError`).
- `ArrowReaderHelper.swift`: map the flatbuffer `.null` field type in
`findArrowType` and add `makeNullHolder` which builds an `ArrowData` with no
buffers.
- `ArrowReader.swift`: add a `.null` branch in `loadField` (`loadNullData`)
that consumes only the field node, since Arrow `Null` arrays carry no validity
or data buffers.
- `ArrowArray.swift`: add `NullArray` (always returns `nil`) and a `.null`
case in `ArrowArrayHolderImpl.loadArray` for null columns nested inside structs.
- `ArrowData.swift`: `isNull` returns `true` for the null type instead of
accessing the non-existent validity buffer.
### Why are the changes needed?
Any query whose result schema contains a `NullType` column crashes the
client process with `SIGTRAP`.
```swift
let spark = try await SparkSession.builder.getOrCreate()
_ = try await spark.sql("SELECT null").collect()
// Fatal error: Unexpectedly found nil while unwrapping an Optional value
```
The reader did not recognize the Arrow `Null` type, so `loadSchema` failed
and `DataFrame.execute()` proceeded with a `nil` `messageSchema`, crashing on a
force-unwrap in `ArrowReader.fromMessage`. This is easy to hit, e.g. `SELECT
:x` with a null named parameter.
### Does this PR introduce _any_ user-facing change?
Yes, this is a bug fix. Previously, `collect()` on a result containing a
`void` column crashed the process; now it returns `nil` for those values (e.g.
`SELECT null` returns `[Row(nil)]`).
### How was this patch tested?
Pass the CIs with a new regression test case, `DataFrameTests.collectNull`,
covering a single null column, a null column mixed with a non-null column,
multiple rows, and `struct(null)`. Also verified the existing `DataFrameTests`
suite (78 tests) passes against a local Spark Connect server
(`apache/spark:4.2.0`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
