[
https://issues.apache.org/jira/browse/ARROW-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765332#comment-16765332
]
Eric Erhardt commented on ARROW-4503:
-------------------------------------
Thanks for the info, [~cshutchinson]. I was thinking the same approach that you
describe above, so that's good.
As for the moving to .NET Standard 2.0, I don't think that is necessary at this
time. We can make all the changes described above and still target 1.3, as far
as I can tell.
Side note: We may want to multi-target for `netstandard1.3`, `netstandard2.0`
and `net4x`, following the OSS Library guidance here:
[https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/cross-platform-targeting.]
But for now we can keep just `netstandard1.3`, until we get more users asking
for `netstandard2.0` or `net4x` targets. (They are mainly just convenience.)
> [C#] ArrowStreamReader allocates and copies data excessively
> ------------------------------------------------------------
>
> Key: ARROW-4503
> URL: https://issues.apache.org/jira/browse/ARROW-4503
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C#
> Reporter: Eric Erhardt
> Assignee: Eric Erhardt
> Priority: Major
> Labels: performance
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When reading `RecordBatch` instances using the `ArrowStreamReader` class, it
> is currently allocating and copying memory 3 times for the data.
> # It is allocating memory in order to [read the data from the
> Stream|https://github.com/apache/arrow/blob/044b418fa108a57f0b4e2e887546cc3e68271397/csharp/src/Apache.Arrow/Ipc/ArrowStreamReader.cs#L72-L74],
> and then reading from the Stream. (This should be the only allocation that
> is necessary.)
> # It then [creates a new
> `ArrowBuffer.Builder`|https://github.com/apache/arrow/blob/044b418fa108a57f0b4e2e887546cc3e68271397/csharp/src/Apache.Arrow/Ipc/ArrowStreamReader.cs#L227-L228],
> which allocates another `byte[]`, and calls `Append` on it, which copies the
> values to the new `byte[]`.
> # Finally, it then calls `.Build()` on the `ArrowBuffer.Builder`, which
> [allocates memory from the MemoryPool, and then copies the intermediate
> buffer|https://github.com/apache/arrow/blob/044b418fa108a57f0b4e2e887546cc3e68271397/csharp/src/Apache.Arrow/ArrowBuffer.Builder.cs#L112-L121]
> into it.
>
> We should reduce this overhead to only allocating a single time (from the
> MemoryPool), and not copying the data more times than necessary.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
