Hi all, Semi-structured data (JSON, event payloads, metadata bags) is ubiquitous in modern data pipelines. In the current Fluss implementation, users are forced to store such data in STRING columns and parse it at query time. This approach suffers from several fundamental problems: no compact binary encoding, no field-level access, and no ability to leverage Fluss's existing columnar optimizations (column pruning, predicate pushdown).
Meanwhile, the industry is converging on the Parquet Variant Binary Encoding specification as the standard for semi-structured data — Apache Spark 4.0, Flink 2.1 (FLIP-521), Paimon, and Parquet itself are all adopting it. As a streaming storage layer bridging Flink and Paimon in stream-lake architectures, Fluss needs native Variant support to enable seamless data flow across the ecosystem. So, I'd like to propose FIP-36: Support Variant Type and Shredding[1]. This proposal introduces a three-layer design: 1. Variant Binary Encoding: Adopt the Parquet Variant spec as the canonical binary representation, enabling O(log n) field lookup and zero-copy nested extraction. 2. Arrow Columnar Storage: Store Variant values in Arrow StructVector with decomposed metadata/value children, integrating with Fluss's existing zero-copy RecordBatch infrastructure. 3. Shredding for Query Optimization: Automatically extract frequently accessed fields into typed sub-columns within the Variant StructVector, enabling server-side sub-column pruning on semi-structured data — all transparent to users. Any feedback and suggestions on this proposal are welcome! [1]: https://cwiki.apache.org/confluence/display/FLUSS/FIP-36%3A+Support+Variant+Type+and+Shredding Regards, Yunhong (Swuferhong)
