GitHub user jskowronski-factset created a discussion: Flight DoGet multiple batch DictionaryArray deltas
I have a FlightServer written in c++ using subclasses of flight::RecordBatchStream and arrow::RecordBatchReader. For efficiency, I'm trying to convert some StringArray fields to DictionaryArrays. It all seems to work if all dictionary values are known with the first RecordBatch streamed back, but if new values occur in subsequent RecordBatches the client doesn't see them. As far as I can tell the client only ever sees the dictionary associated with the first RecordBatch streamed back. I checked both c++ and python clients. It seems like the IPC protocol itself supports this, but flight doesn't seem to support it automatically, and I don't see any obvious way to enable it. Does anybody know if this is even possible, and if yes how I can accomplish it? I already tried using the same Dictionary32Builder between batches, but it didn't help. GitHub link: https://github.com/apache/arrow/discussions/47718 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
