eldenmoon opened a new pull request, #70820:
URL: https://github.com/apache/doris/pull/70820

   ### Background
   
   V3 segments keep Variant subcolumn metadata outside the footer and index it 
by relative path. Query planning probes that index to distinguish an object 
path from an absent field. Extended DESCRIBE aggregates extracted subcolumns 
across sampled tablets.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #70723 (retry after memory limit errors)
   PR Type: Bugfix
   
   Problem Summary:
   
   An ordinary materialized field containing an internal marker can lose its 
value when written to V3. Metadata I/O failures can also produce a false 
negative for a prefix, an incomplete extended schema, or a cached failure after 
the disk read can succeed. The patch identifies generated binary columns by 
their complete paths and carries failures through the planner, schema RPC, and 
FE.
   
   ### Details
   
   - **Change map:** Production adds 263 lines and removes 226. Tests add 370 
lines and remove 7. BE changes cover binary path classification, prefix and 
schema status propagation, and retry state. The FE change reports exhausted 
coordinator retries. Unit tests use physical V3 rowsets, scoped read failures 
and mocked coordinator responses.
   - **Frequency:** Always under the covered triggers. The writer can drop an 
ordinary field with an overlapping marker. An index or metadata read error 
triggers the corresponding failure path.
   - **Root cause:** The writer used substring matching for the internal 
columns. Prefix and schema collection interpreted returned errors as successful 
data. The one-time bulk load retained an I/O failure on the reader.
   - **Why existing tests missed it:** Storage external metadata tests covered 
ordinary field names and successful reads. The Variant external metadata and 
DESCRIBE SQL suites exercise healthy reads. The new unit cases use an 
overlapping name and returned read errors.
   - **Fix:** Require the generated marker and bucket path for internal 
columns, preserve the root as a single path part, and distinguish ordinary 
metadata from binary maps. Return prefix and extended-schema errors to their 
callers and report exhausted coordinator attempts in FE. Permit metadata reads 
to retry after an I/O error while keeping fully loaded paths.
   - **Other paths and branches:** The shared recognizer also covers DocValue 
buckets and inline metadata readers. Validation targets Apache master. 
Previously misclassified segments require separate recovery.
   - **Behavior change:** Extended DESCRIBE reports a failed fetch after 
coordinator retries are exhausted. An unavailable prefix is returned as an 
error. Newly written Variant segments preserve ordinary fields with overlapping 
names.
   
   ### Testing
   
   On Apache master at 5bc6ed62f9, the test-only run failed on all four target 
assertions and passed three existing controls. The observed failures were an 
empty object for the marker-bearing row, a false prefix result, a missing 
subcolumn in a returned schema, and a repeated IO_ERROR.
   
   The four cases pass on the final source. BE ASAN passed 245 tests and 
skipped four NestedGroup cases because the write path is unavailable in this 
build. FE ran both coordinator tests successfully. Formatting, header hygiene 
and changed-line clang-tidy passed. The ASAN FE and BE runtime packages were 
built and an isolated cluster was started for SQL regression.
   
   SQL regression passed 57 of the 60 original suites, including V3 external 
metadata, extended DESCRIBE, DocValue compaction, and Variant SEARCH. Two 
dotted-key suites passed on temporary copies after changing only how they read 
a hidden session variable. The remaining OUTFILE suite reached the external OSS 
write and received HTTP 403 with InvalidAccessKeyId.
   
   ### Release note
   
   Variant V3 preserves ordinary field names that contain internal markers. 
Query planning and extended DESCRIBE report metadata read errors, and a 
transient I/O error can be retried.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Partially verified: the remote CSV OUTFILE case 
requires a valid OSS test fixture
   
   ### AI Disclosure
   
   - Tool: Codex
   - Generated: Code, tests, and description
   - Human review: Pending
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to