GitHub user numinnex added a comment to the discussion: Arrow as zero-serialization target
Hello @infogulch I've took a quick peek at Apache Arrow and from what I understand it's an in memory zero-copy columnar data format. They've built some serialization infrastructure that can be used for IPC/Network transportation, but nevertheless they aren't natively a network format, rather an in-memory one, which would make it a bit problemtic, since we end up both on network and on-disk (they seem to support Parquet tho, which could help with the latter). On top of that them being strictly columnar data format makes it less desirable as we are schemaless general purpose message streaming platform. How I see that in the future, when we would support some sort of schema registry (like Avro), we could potentially create integrations with the Apache Arrow eco system that would allow to create arrows from avro and use that for streaming analytics. GitHub link: https://github.com/apache/iggy/discussions/1602#discussioncomment-12438476 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
