deepak-2605 opened a new pull request, #1851:
URL: https://github.com/apache/iceberg-go/pull/1851
## What
When a `RollingDataWriter`'s `stream()` goroutine exits early on a write
error, batches already queued in its `recordCh` (sent via `Add` before the
error surfaced) were never released, leaking their Arrow buffers.
Closes #1825.
## Root cause
`stream()` deregisters itself from `writerFactory.writers` via a deferred
`CompareAndDelete` the moment it returns. On an early error return, that
runs while `recordCh` may still hold queued batches. The cleanup path that
is supposed to drain those batches, `abortAll()`, walks the writer registry
— but the writer has already removed itself, so `abortAll()` never sees it
and the buffered batches are stranded.
This is independent of any partitioning race: any write error partway
through a rolling writer's stream strands whatever is still buffered in
`recordCh`. (An earlier revision of #1825 described a Windows-specific
trigger for the same symptom; that turned out to be a separate path-parsing
bug, tracked in #1262.)
## Fix
- `stream()`'s cleanup now cancels its context and drains `recordCh`
(non-blocking) before the `CompareAndDelete` defer deregisters the writer,
so cleanup no longer depends on `abortAll()` finding it in the registry.
- `Add()` checks `ctx.Done()` up front (non-blocking) so that once a writer
has died, in-flight sends from other fanout workers back off instead of
queuing into a channel nobody will drain.
- `recordCh` is never closed from inside `stream()` — that stays with the
external `closeAll()`/`abortAll()` paths, which only run after all `Add`
callers have returned, avoiding a send-on-closed-channel panic.
## Test plan
- Added `TestStreamErrorDrainsBufferedRecords`: queues several batches,
forces
the file open to fail, and asserts the checked allocator returns to zero.
Fails on `main`, passes with this change.
- `go test ./table/ -run TestRollingDataWriter -race` passes.
- 50x `-race` stress run of the fanout/rolling-writer tests: no panics, no
data races.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
