github-actions[bot] commented on code in PR #67135:
URL: https://github.com/apache/doris/pull/67135#discussion_r3869483684
##########
gensrc/proto/snii.proto:
##########
@@ -81,25 +81,35 @@ message SniiCoreMetadataPB {
optional uint32 common_grams_posting_policy = 5;
}
+// FIELD NUMBERS ARE ON DISK. The SNII format shipped before this message
reached
+// upstream, and those segments put null_count at 4. Upstream later inserted
+// sum_total_term_freq at 4 and pushed null_count to 5, so the two layouts
+// disagree on what tag 4 means -- and both are uint64, so a mismatched read
+// returns a plausible wrong number instead of failing. Numbering here follows
+// the shipped layout; append new fields, never insert.
message SniiStatsPB {
optional uint64 doc_count = 1;
optional uint64 indexed_doc_count = 2;
optional uint64 term_count = 3;
- optional uint64 sum_total_term_freq = 4;
- optional uint64 null_count = 5;
+ optional uint64 null_count = 4;
+ optional uint64 sum_total_term_freq = 5;
Review Comment:
[P1] Preserve fields absent from shipped metadata
The compatibility table says shipped `SniiStatsPB` had no tag 5, so those
proto2 bytes cannot satisfy `has_sum_total_term_freq()`. After this
renumbering, `decode_core_pb()` still rejects any Stats without it
(`core_metadata.cpp:169-172`), and likewise requires `has_norms()` even though
shipped SectionRefs also ended at tag 4 (`core_metadata.cpp:180-183`).
Consequently the segments this PR is meant to recover still fail as
`INVERTED_INDEX_FILE_CORRUPTED`. A shipped CommonGrams record with only tags
1-7 has the same problem because `decode_common_grams()` requires appended tags
8-12. Please accept/default these appended fields for the legacy capability set
(or add an explicit compatibility discriminator) and cover it with a frozen
shipped-byte decode test; descriptor checks and new-writer digests cannot
exercise absent legacy fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
