GitHub user numinnex added a comment to the discussion: Arrow as zero-serialization target
> In that case maybe you should reconsider Arrow; having all the metadata > arranged in dense arrays separate from message content could be beneficial > for cache locality and simd vectorization. Yeah this is something that we are considering as well, in fact @hubcio currently is rewriting my batch based implementation to a more fine grained one, but grouping the metadata together and storing it separately from the messages data is something that we are still considering as an option, maybe not in the form of `Arrow` format, but rather using other zero copy layouts, for example from the already mentioned `rkyv` or other crates such as `musli`, maybe even if we'd made those arrays fixed size (which is totally doable), we could get away with some pointer cast from the struct -> bytes, either using rust native apis or `bytemuck` crate. GitHub link: https://github.com/apache/iggy/discussions/1602#discussioncomment-12480692 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
