Thanks for driving this FIP, Yunhong.
+1(binding) from my side.
I agree with the overall direction of introducing a native VARIANT type in
Fluss, aligning the binary format with Parquet Variant, and using the Arrow
arrow.parquet.variant layout as the foundation for future sub-field pruning
and stream-lake interoperability. This is an important step for supporting
semi-structured data in Fluss and for better alignment with Flink, Spark and
Paimon.
I have a few non-blocking comments that I hope we can clarify in the FIP or
during the implementation PRs:
1. Public Variant API and immutability
The proposed Variant API exposes metadata() and value() as byte arrays. Could
we clarify whether these methods return defensive copies or zero-copy views?
If we want zero-copy behavior for performance, it may be better to make the
copy/wrap semantics explicit in the API.
In comparison, Flink's FLIP-521 mainly exposes a higher-level Variant
interface, while BinaryVariant is an implementation detail. Therefore Flink
does not expose the byte-array mutability issue as directly at the public API
level. Since Fluss exposes metadata/value more explicitly, I think we should
document the immutability contract carefully.
2. Null semantics
Could the FIP explicitly distinguish the following cases?
- SQL column null
- Variant null, e.g. JSON null
- shredded field missing
- shredded field exists but its value is null
- shredded field exists but does not match the inferred typed_value type
Flink's Variant design makes a useful distinction here: a missing field or
out-of-bound array access returns Java null, while an explicit JSON null is
represented as a Variant whose isNull() returns true. I think Fluss should
define similarly clear semantics, especially because shredding introduces
both value and typed_value paths.
3. Projection path syntax and escaping
The project("payload:field::TYPE") syntax is convenient, but we should define
escaping rules for field names containing special characters such as ':',
'::', '.', spaces, etc.
Flink's SQL functions use JSON path expressions and support quoted field
names, for example $['a b']. Since Fluss v1 only supports top-level field
projection, it would also be useful to clarify whether "a.b" is rejected as a
nested path or treated as a literal top-level field name.
4. Type mismatch and casting semantics
The FIP currently says that project("payload:field::TYPE") returns NULL on
cast/type mismatch. I think this is a reasonable default and is consistent
with the lenient behavior in Flink/Spark-style Variant access. Could we make
the exact rule explicit?
For example:
- without ::TYPE, return the selected field as VARIANT and preserve fallback
values;
- with ::TYPE, return the typed scalar and return NULL when the value cannot
be cast to the requested type.
5. Shredding controllability and observability
FIP-36 v1 uses writer-side autonomous inference and does not expose a table
property or DDL hint for user-controlled shredding. This keeps v1 simple, but
it also means users cannot force a critical field to be shredded or predict
the exact layout.
Paimon seems to provide both inference and explicit configuration, such as
variant.inferShreddingSchema and variant.shreddingSchema. I do not think
Fluss has to support user-controlled shredding in v1, but it would be helpful
to explicitly call out this gap and possibly mention what metrics or system
views could be added later to expose the active shredded layout and hit/miss
ratio.
6. Writer-side inference lifecycle
Could we clarify whether a writer's inferred shredding schema is fixed after
MIN_SAMPLE_SIZE, or whether it can evolve as more records are observed?
Related questions:
- how are null values counted in presence/type-consistency statistics?
- is candidate ranking deterministic when scores tie?
- are there limits on field depth, field name length, or number of observed
fields before MAX_SHREDDED_FIELDS is applied?
- do we plan to expose metrics for selected fields, inference sample size,
and projection hit/miss?
This is especially important because Fluss allows different writers to
produce different shredded layouts for the same Variant column.
7. Column index vs column id in the wire protocol
It is good that column_index in PbVariantFieldProjection has been changed to
optional to leave room for column_id in the future. Since Fluss already has
column IDs and column indices may shift after schema evolution, should we add
column_id to the initial message shape, or at least clearly bind the request
to a schema id?
This issue is more specific to Fluss than to Flink or Paimon, because Fluss
has a client-server fetch protocol and existing projected_fields are
index-based.
8. Arrow extension metadata
The FIP says the storage layout follows arrow.parquet.variant. Could we
clarify whether the Arrow Field will also carry the canonical extension name
"arrow.parquet.variant", not only the StructVector shape with metadata/value
/ typed_value children?
Since Fluss relies heavily on Arrow IPC and server-side sub-column pruning,
being precise about the Arrow extension contract would help future
interoperability.
9. Scope of predicate pushdown
The motivation mentions predicate pushdown on nested fields, but v1 seems to
focus on Variant sub-field projection pushdown. This is fine, but I suggest
explicitly listing Variant predicate pushdown as future work if it is out of
scope for v1.
Flink FLIP-521 also mentions shredding/projection/filter pushdown as a future
optimization rather than part of the initial Variant type integration, so
being explicit about the boundary would help set user expectations.
These comments do not change my support for the proposal. I think FIP-36 is
well aligned with the direction of Flink/Spark/Paimon, and the current scope
is a reasonable foundation for future nested shredding, user-controlled
shredding, predicate pushdown and lake integrations.
Best,
Leonard
> 2026 7月 8 12:26 上午，Prajwal Banakar <[email protected]> 写道：
>
> +1 (non-binding)
>
> Best,
> Prajwal
>
> On Tue, 7 Jul, 2026, 7:47 pm Giannis Polyzos, <[email protected]> wrote:
>
>> + 1 (binding)
>>
>> Best,
>> Giannis
>>
>> On Tue, 7 Jul 2026 at 4:36 PM, Zhe Wang <[email protected]> wrote:
>>
>>> +1(non-binding)
>>>
>>> Best regards,
>>> Zhe Wang
>>>
>>>
>>> yunhong Zheng <[email protected]> 于2026年7月6日周一 11:57写道：
>>>
>>>> Hi, all.
>>>>
>>>> I'd like to start a vote on FIP-36: Support Variant Type and Shredding
>>>> [1]. You can find the discussion on it in here [2]. The vote will last
>>>> for at least 72 hours unless there is objection or insufficient votes.
>>>>
>>>> [1]
>>>>
>>>
>> https://cwiki.apache.org/confluence/spaces/FLUSS/pages/406623649/FIP-36+Support+Variant+Type+and+Shredding
>>>> [2] https://lists.apache.org/thread/m4wrtxoxrkg0zbgoy7p1yr3o15onylcm
>>>>
>>>>
>>>> Best regards,
>>>> Yunhong
>>>>
>>>
>>
