binary-signal commented on code in PR #59:
URL: https://github.com/apache/fluss-rust/pull/59#discussion_r2557915600
##########
Cargo.toml:
##########
@@ -34,5 +34,5 @@ members = ["crates/fluss", "crates/examples",
"bindings/python"]
fluss = { version = "0.1.0", path = "./crates/fluss" }
tokio = { version = "1.44.2", features = ["full"] }
clap = { version = "4.5.37", features = ["derive"] }
-arrow = "57.0.0"
+arrow = { version = "57.0.0", features = ["ipc_compression"] }
Review Comment:
@luoyuxia
1 → Yes, Arrow automatically decompresses data when reading into a
`RecordBatch`.
From the Arrow IPC docs: https://arrow.apache.org/rust/arrow_ipc/index.html
```
The Arrow IPC format defines how to read/write RecordBatches to/from files
or byte streams. It handles serialization and deserialization.
```
2 → Arrow IPC supports **LZ4** and **ZSTD**.
In the Fluss docs, I've also seen support for **LZ4** and **ZSTD**:
`table.log.arrow.compression.type` can be `NONE`, `LZ4_FRAME`, or `ZSTD`
(default: `ZSTD`).
Decompression happens at a lower layer than the Fluss logical types. For the
non-standard types like **ltz** (I assume you mean `TIMESTAMP_LTZ`), this must
be handled in the Rust binding code since you need to parse the metadata stored
in the arrow timestamp which has information about the precision and the
timezone. I already have another PR (not submitted) adding support for `ltz` +
timestamps.
But also found PR #53 , which implements *all* types including ltz
timestamp. Since this is more holistic PR than what I was planning to submit,
it’s worth reviewing and merging this PR instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
