wuchong commented on issue #2256: URL: https://github.com/apache/fluss/issues/2256#issuecomment-3699296367
Besides, I’d like to discuss the documentation structure and design here first. Directly adding documentation for the `COMPACTED` table format under the current "Table Design" section may not be the best approach, especially since `COMPACTED` is just one of several data encodings we support. We also have `ARROW`, `INDEXED`, and will likely introduce more in the future. To better organize this information, I suggest creating a dedicated documentation page titled **“Data Encodings”** (between "Data Distribution" and "Data Types"). This page would provide: - An **overview** of the encoding landscape in Fluss, - A clear distinction between **log data encoding** and **KV data encoding**, - Guidance on **when to choose which encoding**, including key trade-offs. Under this page, we can then introduce each encoding with consistent subsections covering: main characteristics, primary use cases, design trade-offs, and APIs how to use. - **ARROW**: A columnar encoding. Can only be used as a *log encoding*. Ideal for streaming analytical workloads due to columnar efficiency and interoperability with Arrow-based systems. Like projection pushdown for streaming read. - **COMPACTED**: A row-oriented encoding that leverages [variable-length](https://lucene.apache.org/java/3_5_0/fileformats.html#VInt) coding to have a smallest binary data size. Can be used as both *log encoding* and *KV encoding*. Introduce when it is combined with `table.changelog.image=WAL`, it enables an **efficient, lightweight KV store**, perfect for system tables or use cases that don’t require full changelog reads or column projections. - **INDEXED**: A row-oriented encoding but is deprecated. Supports both *log* and *KV encoding*. This structure would make the documentation more scalable, discoverable, and aligned with Fluss’s evolving architecture. What do you think? cc @polyzos , what do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
