GitHub user v1gnesh added a comment to the discussion: Arrow as zero-serialization target
I've also asked the Arrow question to buf (https://github.com/bufbuild/buf/issues/4141). I get that Arrow (columnar) in a streaming context means needing to wait for a chunk of data, but Arrow will be fully serde free, all the way from client-side python dataframes, over the wire (say Arrow Flight), onto query engines (Datafusion), and databases (Snowflake, etc.), data lakes (of parquet files). In [spice.ai](https://spice.ai), there is even use of [moka-rs](https://github.com/moka-rs/moka) as an in-line cache that caches Arrow RecordBatches. GitHub link: https://github.com/apache/iggy/discussions/1602#discussioncomment-14882683 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
