github-actions[bot] commented on code in PR #68822:
URL: https://github.com/apache/doris/pull/68822#discussion_r4230622858
##########
gensrc/thrift/DataSinks.thrift:
##########
@@ -494,12 +494,14 @@ struct TIcebergTableSink {
17: optional TIcebergWriteType write_type = TIcebergWriteType.INSERT;
// Unset keeps collection enabled for rolling upgrades with older FEs.
18: optional bool collect_column_stats;
+ // Thrift map values cannot be null. These keys distinguish SQL NULL from
text and empty bytes.
+ 19: optional set<string> static_partition_null_keys;
// Iceberg field ids of the FLOAT/DOUBLE fields whose NaN count would
survive the table's metrics policy
// (effective mode != none). Counting a NaN is an extra pass over the data
-- unlike the other statistics,
// which the parquet footer already carries -- so BE must not pay it for a
field FE would then drop.
// Unset or empty means count nothing: an older FE does not read
nan_value_counts back, so counting for it
// would be pure waste, and a table whose float fields are all
metrics-disabled has nothing to report.
- 19: optional list<i32> nan_count_field_ids;
+ 20: optional list<i32> nan_count_field_ids;
Review Comment:
[P2] Keep NaN counts when an older FE writes through a new BE. During a
BE-first rolling upgrade, the older FE sends `nan_count_field_ids` as LIST/19;
this schema makes field 19 a SET, so Thrift skips that request.
`VIcebergPartitionWriter` then writes no `nan_value_counts` for NaN-free
Parquet/ORC files. FE records unknown counts, so float range queries must keep
scanning those files after the upgrade until they are rewritten. Please provide
a fallback when field 20 is absent, or explicitly document and test this
persistent compatibility cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
