kflansburg commented on pull request #8968: URL: https://github.com/apache/arrow/pull/8968#issuecomment-748495686
Hey @andygrove, this is definitely not a great use-case for Arrow since the format is not columnar, but I'm hoping to implement micro-batch style processing (possibly in DataFusion / Ballista), similar to Spark Structured Streaming. I really like your idea of using Kafka as a transport layer for Arrow Flight messages. I was planning to try to implement some sort of JSON parsing -> Arrow StructArray for the Kafka payload field, but parsing it as Arrow flight would be very cool as well. FYI I'm working now to resolve the CI issues related to compiling and/or linking against `librdkafka`, but it may not be possible to get this to build without including this dependency. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
