kflansburg commented on pull request #8968: URL: https://github.com/apache/arrow/pull/8968#issuecomment-748504940
I definitely want to support subscribing to multiple topics, its often the case that multiple topics share the same schema. My concern is that the full Schema may not be possible to infer from a single message. Its possible we can have the user supply the full schema but that would be cumbersome. I think I was planning to have a `BinaryArray` (maybe cast as a `JSONArray`, that is a good idea) be able to be parsed to a `StructArray`, but I haven't verified that that all makes sense. Now I'm thinking though that if the parsing happens after the `JSONArray` has been formed, then you have access to all of the messages in the `RecordBatch`, and can determine the schema. You just can't guarantee that the schema is consistent across `RecordBatch`es, but I think that is fine. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
