zeroshade opened a new pull request, #1346:
URL: https://github.com/apache/arrow-go/pull/1346
### Rationale for this change
Fixes #1325. Some callers decode one logical IPC stream that arrives as a
sequence of blobs, where only the first blob carries the schema. Through
v18.7.0 they could reuse one `ipc.Reader` by swapping the underlying
`io.Reader`. Since v18.8.0, `done` stays set once a blob reaches EOF and can't
be reset, so every blob after the first decodes as zero rows with a nil error.
### What changes are included in this PR?
- `(*Reader).ContinueFrom(io.Reader) error` and
`(*Reader).ContinueFromMessageReader(MessageReader) error` resume a reader on a
new source. The schema and dictionaries already read are kept. Both methods
also:
- discard the current record batch, release the previous message reader,
and clear `done`;
- refuse to continue a reader that has already failed, so an earlier error
can't be hidden;
- refuse to continue a reader that hasn't read its schema yet.
- As requested in
https://github.com/apache/arrow-go/issues/1325#issuecomment-5878340460, a
schema message where a record batch is expected now fails with `arrow/ipc:
unexpected schema message: …`. That marks the reader done, and the error is
sticky. Before, this gave the generic invalid-message-type error. The other
invalid-type errors now also have their closing parenthesis back.
### Are these changes tested?
Yes. `TestReaderContinueFrom` splits a single writer's output, with an int64
column and a dictionary column, into three blobs and checks:
- all batches round-trip through both `Next` and `Read`, and dictionaries
carry across blobs;
- a continuation that starts with a schema message fails loudly and stays
failed;
- `ContinueFrom` on a reader whose schema hasn't been read yet returns an
error and leaves the reader usable.
The test uses a checked allocator. Removing the `done` reset makes it fail.
### Are there any user-facing changes?
Yes: two new methods on `ipc.Reader`, and a clearer error when a schema
message appears mid-stream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
