[
https://issues.apache.org/jira/browse/ARROW-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765331#comment-16765331
]
Eric Erhardt commented on ARROW-4502:
-------------------------------------
I think the first thing that needs to be done is update to the latest Google
FlatBuffers code. I see they have [C# support for directly reading and writing
to memory other than byte|https://github.com/google/flatbuffers/pull/4886] now.
I will get this done today and send a PR. Does that need a separate Jira issue
[~wesmckinn]? Or can I just reference this issue in the PR, and say the PR is
"working towards ARROW-4502"?
[~cshutchinson] - Some thoughts about your reply above:
# I don't think generating the code during the build is ideal. My main
reasoning is for things like [Source
Link|https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/sourcelink]
to work correctly, it needs to point to source code contained in a repository.
It also becomes a pain for people who are just browsing the code on GitHub or
in a code editor. The arrow schema doesn't change that often does it, to the
point where it needs to be re-generated on every build?
# See my comment above - I will update the FlatBuffers code to the latest
version, which appears to meet my needs.
# Yes! I will include the benchmarks (using
[BenchmarkDotNet|https://github.com/dotnet/benchmarkdotnet] a great tool). I
will have to modify the current version a bit, since I am currently using a
checked-in Arrow Stream file, which I assume we don't want to do.
# I will give the changes a look and provide what feedback I can.
> [C#] Add support for zero-copy reads
> ------------------------------------
>
> Key: ARROW-4502
> URL: https://issues.apache.org/jira/browse/ARROW-4502
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C#
> Reporter: Eric Erhardt
> Assignee: Eric Erhardt
> Priority: Major
> Labels: performance
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> In the Python (and C++) API, you can create a `RecordBatchStreamReader`, and
> if you give it an `InputStream` that supports zero-copy reads, you can get
> back `RecordBatch` objects without allocating new memory and copying all the
> data.
> There is currently no way to read Arrow RecordBatch instances without
> allocating new memory and copying all the data. We should enable this
> scenario in the C# API.
>
> My proposal is to create a new `class ArrowRecordBatchReader : IArrowReader`.
> It's constructor will take a `ReadOnlyMemory<byte> data` parameter, and it
> will be able to read `RecordBatch` instances just like the existing
> `ArrowStreamReader`. As part of this new class, we will refactor any common
> code out of `ArrowStreamReader` in order for the parsing logic to be shared,
> where necessary.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
