[jira] [Commented] (ARROW-5331) [C++] FlightDataStream should be higher-level
[
https://issues.apache.org/jira/browse/ARROW-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855776#comment-16855776
]
Antoine Pitrou commented on ARROW-5331:
---
I had overlooked that the {{FlightDescriptor}} is unused where
{{FlightDataStream}} is concerned. {{FlightDataStream}} is used for the
server's {{DoGet}} implementation only. So {{RecordBatchStream}} should already
be sufficient in all cases where a record batches-only data stream is desired.
This leaves the question of heterogenous Flight streams. They should be handled
at the IPC layer first before adapting Flight to work with them. We probably
need some kind of IPC {{Datum}} that can represent several different kinds of
data (record batch, tensor...).
> [C++] FlightDataStream should be higher-level
> -
>
> Key: ARROW-5331
> URL: https://issues.apache.org/jira/browse/ARROW-5331
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, FlightRPC
>Affects Versions: 0.13.0
>Reporter: Antoine Pitrou
>Priority: Major
>
> Currently, {{FlightDataStream}} is expected to provide {{FlightPayload}}
> objects. This requires the user to handle IPC serialization themselves.
> Instead, it could provide higher-level {{FlightData}} objects (perhaps a
> simple struct containing a {{FlightDescriptor}} and a {{RecordBatch}}),
> letting Flight handle IPC encoding.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5331) [C++] FlightDataStream should be higher-level
[
https://issues.apache.org/jira/browse/ARROW-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841372#comment-16841372
]
Wes McKinney commented on ARROW-5331:
-
It might be worth putting in some thought now about how to handle generic
streams of encapsulated IPC messages in the Flight protocol. Of course, the use
case of a record batch stream (schema, then dictionaries, then record batches)
needs some special handling.
> [C++] FlightDataStream should be higher-level
> -
>
> Key: ARROW-5331
> URL: https://issues.apache.org/jira/browse/ARROW-5331
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, FlightRPC
>Affects Versions: 0.13.0
>Reporter: Antoine Pitrou
>Priority: Major
>
> Currently, {{FlightDataStream}} is expected to provide {{FlightPayload}}
> objects. This requires the user to handle IPC serialization themselves.
> Instead, it could provide higher-level {{FlightData}} objects (perhaps a
> simple struct containing a {{FlightDescriptor}} and a {{RecordBatch}}),
> letting Flight handle IPC encoding.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5331) [C++] FlightDataStream should be higher-level
[
https://issues.apache.org/jira/browse/ARROW-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841254#comment-16841254
]
David Li commented on ARROW-5331:
-
I think that complicates the mental model, and at least for tensors, since
they're treated entirely separately, they should have their own method (if we
want to support them).
> [C++] FlightDataStream should be higher-level
> -
>
> Key: ARROW-5331
> URL: https://issues.apache.org/jira/browse/ARROW-5331
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, FlightRPC
>Affects Versions: 0.13.0
>Reporter: Antoine Pitrou
>Priority: Major
>
> Currently, {{FlightDataStream}} is expected to provide {{FlightPayload}}
> objects. This requires the user to handle IPC serialization themselves.
> Instead, it could provide higher-level {{FlightData}} objects (perhaps a
> simple struct containing a {{FlightDescriptor}} and a {{RecordBatch}}),
> letting Flight handle IPC encoding.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5331) [C++] FlightDataStream should be higher-level
[
https://issues.apache.org/jira/browse/ARROW-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841050#comment-16841050
]
Antoine Pitrou commented on ARROW-5331:
---
One counter-argument against this is that people might want to send something
else than record batches (e.g. tensors), or perhaps even a mixed stream of
different things. I'm not sure whether we care about supporting this.
[~wesmckinn] [~lidavidm]
> [C++] FlightDataStream should be higher-level
> -
>
> Key: ARROW-5331
> URL: https://issues.apache.org/jira/browse/ARROW-5331
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, FlightRPC
>Affects Versions: 0.13.0
>Reporter: Antoine Pitrou
>Priority: Major
>
> Currently, {{FlightDataStream}} is expected to provide {{FlightPayload}}
> objects. This requires the user to handle IPC serialization themselves.
> Instead, it could provide higher-level {{FlightData}} objects (perhaps a
> simple struct containing a {{FlightDescriptor}} and a {{RecordBatch}}),
> letting Flight handle IPC encoding.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
