dongjoon-hyun opened a new pull request, #471:
URL: https://github.com/apache/spark-connect-swift/pull/471
### What changes were proposed in this pull request?
This PR aims to fix two bugs in the vendored Arrow `VariableBufferBuilder`
(used for `STRING`/`BINARY` serialization in `SparkSession.createDataFrame`).
1. **Heap buffer overflow in `append()`**: the values-buffer capacity check
was inside the `if index > 0` branch, so the very first value was written
without any capacity check. When the first string/binary value is longer than
the initial 64-byte buffer, the write overflows the heap allocation
(intermittent segfaults or data corruption). The fix moves the capacity check
out of the `if index > 0` branch so it applies to the first value as well.
2. **Truncated offsets buffer in `finish()`**: the Arrow format requires
`length + 1` offset entries because value `i` is read from the range
`offsets[i]..<offsets[i + 1]`, but `finish()` allocated only `length` entries,
dropping the final offset. The fix allocates `(length + 1) * 4` bytes for the
offsets buffer while keeping its `.length` as `length`, because `ArrowData`
uses the offsets buffer's `.length` as the array length.
The same bugs exist in upstream `apache/arrow-swift`; they will be reported
there separately.
### Why are the changes needed?
Both bugs corrupt Arrow IPC data produced by `createDataFrame`:
- Bug 1 causes intermittent crashes or corrupted values when the first
string exceeds 64 bytes.
- Bug 2 truncates the last offsets entry whenever `4 * rowCount` is a
multiple of 64 (e.g., 16 rows), making the server fail with `Cannot grow
BufferHolder by size <negative>`. In other cases, the 64-byte alignment padding
happens to preserve the final offset, which is why the existing small-scale
tests passed.
Reproducer (fails or crashes before this fix, passes after):
```swift
let spark = try await SparkSession.builder.getOrCreate()
let value = String(repeating: "x", count: 200)
let data: [[Sendable?]] = (0..<16).map { (i: Int) in [i, value + String(i)] }
let rows = try await spark.createDataFrame(data, "id INT, value
STRING").collect()
```
### Does this PR introduce _any_ user-facing change?
No, this is a bug fix. `createDataFrame` now works correctly with
string/binary data that previously crashed or was rejected by the server.
### How was this patch tested?
Pass the CIs with a newly added test case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
