nevi-me commented on pull request #8968:
URL: https://github.com/apache/arrow/pull/8968#issuecomment-748503983
> The stretch goal here could be support for integration with a schema
registry, but I haven't worked much with that.
An interesting challenge would be how to then structure the data when
subscribing to multiple topics. Perhaps it'd then make sense to convert the
`payload` into a `StructArray` or `RecordBatch`, with the other fields
remaining as part of `KafkaBatch`?
If you have multiple topics, you'd end up with different schemas for the
`StructArray`, which isn't in the spirit of Arrow, as all arrays should be
homogenous. A `UnionArray` could address that issue, but it's too much effort,
in my opinion.
Converting `Field::new("payload", DataType::Binary, true)` to `StructArray`
will also require the data to be the same.
___
More rabbit-hole kinds of ideas:
- We could support a JSONArray that is an extension of `BinaryArray` but
holds JSON data. I believe it's already a thing in Parquet, but I haven't
looked at it in detail. There's `ExtensionArray` as part of the Arrow spec, but
we haven't implemented it in Rust as yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
