[ 
https://issues.apache.org/jira/browse/ARROW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009556#comment-16009556
 ] 

Wes McKinney edited comment on ARROW-249 at 5/14/17 12:15 AM:
--------------------------------------------------------------

I'd like to prototype a GRPC-based client and server that sends components of 
an Arrow stream as a series of RPC messages according to the Arrow streaming 
format. See this discussion about some issues around this

https://github.com/grpc/grpc-java/issues/1054#issuecomment-301282237

In ARROW-1008, I've refactored the "stream reader" and "stream writer" concepts 
to be abstract interfaces, so now we could define {{GrpcStreamWriter}} and 
{{GrpcStreamReader}} that would be a middleman to the underlying GRPC 
client/server machinery, presenting a standard stream iterator to the client 
side. I figure we could prototype this using protobuf messages that hold each 
piece of the Arrow stream (first RPC message is the schema) and see how the 
perf compares with a "raw" stream. 

cc [~nongli]


was (Author: wesmckinn):
I'd like to prototype a GRPC-based client and serve that sends components of an 
Arrow stream as a series of RPC messages according to the Arrow streaming 
format. See this discussion about some issues around this

https://github.com/grpc/grpc-java/issues/1054#issuecomment-301282237

In ARROW-1008, I've refactored the "stream reader" and "stream writer" concepts 
to be abstract interfaces, so now we could define {{GrpcStreamWriter}} and 
{{GrpcStreamReader}} that would be a middleman to the underlying GRPC 
client/server machinery, presenting a standard stream iterator to the client 
side. I figure we could prototype this using protobuf messages that hold each 
piece of the Arrow stream (first RPC message is the schema) and see how the 
perf compares with a "raw" stream. 

cc [~nongli]

> [Format] Define GRPC IDL / wire protocol for messaging with Arrow data
> ----------------------------------------------------------------------
>
>                 Key: ARROW-249
>                 URL: https://issues.apache.org/jira/browse/ARROW-249
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: Format
>            Reporter: Wes McKinney
>             Fix For: 0.4.0
>
>
> In addition to memory maps / shared memory, we should be able to assemble one 
> or more Arrow arrays in a memory block suitable for sending over the wire 
> with GRPC (http://www.grpc.io/). This can be similarly adapted to other 
> messaging / RPC frameworks. 
> We can continue to use Flatbuffers for the metadata serialization. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to