Hi guys, 1. Regarding IPC vs Flight: in fact my initial suggestion was to add this feature starting from the IPC(I moved initial write up steps to the bottom of the doc). Afterwards David suggested focusing on Flight and that's how we ended up with the protobufs change in the proposal. This being said I do think that the place where this should be impemented is a good question on its own. Maybe it makes sense to have this kind of a feature in IPC and somehow use it in Flight, maybe not. 2. The point about dictionaries deserves a dedicated section in the proposal. Nate and David brought it up and shared some insights. I'll try to aggregate them and we can continue the discussion form there.
Cheers, Gosh On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <natebauernfe...@deephaven.io> wrote: > > > > > > makes it more difficult to bring schema evolution back into the > > > > IPC Stream format (i.e. it would live only in flight) > > > > > > Gosh's proposal extends the flatbuffer structures not the protobufs. > Can > > > you help me understand how difficult it would be to bring the > `schema_id` > > > approach to the IPC stream format? > > > > I thought we were talking solely about the Flight Protobuf definitions - > > not the Flatbuffers (and the Google doc at least only talks about the > > Protobufs). > > > > I somehow missed that schema_id is being added to protobuf in the document. > It feels to me that the schema_id is a property that would ideally only > apply to the RecordBatch. I better understand Micah's dictionary concerns, > now, too. > > > Side Question: Why isn't the IPC stream format a series of the flight > > > protobufs? It's a real shame that there is no standard way to > > > capture/replay a stream with app_metadata. (Obviously ignoring the > > > annoyances around protobuf wrapping flatbuffers.) > > > > The IPC format was defined long before Flight, and Flight's app_metadata > > was added after Flight's initial definition. Note an IPC message does > have > > a provision for key-value metadata, though I think APIs for that are not > > fully exposed. (See ARROW-6940: > > https://issues.apache.org/jira/browse/ARROW-6940 and despite my comments > > there perhaps we need to unify or at least consider how Flight's > > app_metadata relates to the IPC message custom_metadata. Also perhaps see > > ARROW-1059.) > > > > KeyValue unfortunately is string to string. In flatbuffer strings are only > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is opaque bytes. > The latter is a bit more useful. > > -- >
