[
https://issues.apache.org/jira/browse/ARROW-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799169#comment-16799169
]
Eric Erhardt commented on ARROW-4997:
-------------------------------------
[~wesmckinn], [~xhochy], [~kou] - this is a pretty major issue in the C#
library that is blocking usage of the Arrow C# library over a network stream. I
have a [PR|https://github.com/apache/arrow/pull/4017] up to fix it that has
been approved by Stephen Toub. I'm hoping this PR can be merged for the 0.13
release, as it is super important for our usages.
> [C#] ArrowStreamReader doesn't consume whole stream and doesn't implement
> sync read
> -----------------------------------------------------------------------------------
>
> Key: ARROW-4997
> URL: https://issues.apache.org/jira/browse/ARROW-4997
> Project: Apache Arrow
> Issue Type: Bug
> Components: C#
> Reporter: Eric Erhardt
> Assignee: Eric Erhardt
> Priority: Blocker
> Labels: pull-request-available
> Original Estimate: 4h
> Time Spent: 1h 10m
> Remaining Estimate: 2h 50m
>
> There are 2 major issues with the ArrowStreamReader that are blocking me from
> using it.
> # When it reads a batch from a .NET Stream that doesn't return the whole
> chunk of memory in one "Read" call (like a socket/network stream), it only
> calls Read once, and then continues on. This is an issue because it has
> "garbage" at the end of its buffer (which was never written to by the
> stream), and when attempting to read the next batch, it is in the middle of
> the previous batch from the .NET Stream. This causes all sorts of issues
> because it assumes the next 4 bytes are the message length, which it
> obviously isn't. See [the reading
> code|https://github.com/apache/arrow/blob/13fd813445b4738cbebbd137490fe3c02071c04b/csharp/src/Apache.Arrow/Ipc/ArrowStreamReaderImplementation.cs#L90-L97]
> for where it only calls Read once - it should be in a loop.
> # ArrowStreamReader has a synchronous ReadNextRecordBatch() method - but it
> throws NotImplementedException. This is necessary when a caller isn't in an
> async method, they can't/shouldn't call the async API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
