eldenmoon opened a new pull request, #70820: URL: https://github.com/apache/doris/pull/70820
### Background V3 segments keep Variant subcolumn metadata outside the footer and index it by relative path. Query planning probes that index to distinguish an object path from an absent field. Extended DESCRIBE aggregates extracted subcolumns across sampled tablets. ### What problem does this PR solve? Issue Number: None Related PR: #70723 (retry after memory limit errors) PR Type: Bugfix Problem Summary: An ordinary materialized field containing an internal marker can lose its value when written to V3. Metadata I/O failures can also produce a false negative for a prefix, an incomplete extended schema, or a cached failure after the disk read can succeed. The patch identifies generated binary columns by their complete paths and carries failures through the planner, schema RPC, and FE. ### Details - **Change map:** Production adds 263 lines and removes 226. Tests add 370 lines and remove 7. BE changes cover binary path classification, prefix and schema status propagation, and retry state. The FE change reports exhausted coordinator retries. Unit tests use physical V3 rowsets, scoped read failures and mocked coordinator responses. - **Frequency:** Always under the covered triggers. The writer can drop an ordinary field with an overlapping marker. An index or metadata read error triggers the corresponding failure path. - **Root cause:** The writer used substring matching for the internal columns. Prefix and schema collection interpreted returned errors as successful data. The one-time bulk load retained an I/O failure on the reader. - **Why existing tests missed it:** Storage external metadata tests covered ordinary field names and successful reads. The Variant external metadata and DESCRIBE SQL suites exercise healthy reads. The new unit cases use an overlapping name and returned read errors. - **Fix:** Require the generated marker and bucket path for internal columns, preserve the root as a single path part, and distinguish ordinary metadata from binary maps. Return prefix and extended-schema errors to their callers and report exhausted coordinator attempts in FE. Permit metadata reads to retry after an I/O error while keeping fully loaded paths. - **Other paths and branches:** The shared recognizer also covers DocValue buckets and inline metadata readers. Validation targets Apache master. Previously misclassified segments require separate recovery. - **Behavior change:** Extended DESCRIBE reports a failed fetch after coordinator retries are exhausted. An unavailable prefix is returned as an error. Newly written Variant segments preserve ordinary fields with overlapping names. ### Testing On Apache master at 5bc6ed62f9, the test-only run failed on all four target assertions and passed three existing controls. The observed failures were an empty object for the marker-bearing row, a false prefix result, a missing subcolumn in a returned schema, and a repeated IO_ERROR. The four cases pass on the final source. BE ASAN passed 245 tests and skipped four NestedGroup cases because the write path is unavailable in this build. FE ran both coordinator tests successfully. Formatting, header hygiene and changed-line clang-tidy passed. The ASAN FE and BE runtime packages were built and an isolated cluster was started for SQL regression. SQL regression passed 57 of the 60 original suites, including V3 external metadata, extended DESCRIBE, DocValue compaction, and Variant SEARCH. Two dotted-key suites passed on temporary copies after changing only how they read a hidden session variable. The remaining OUTFILE suite reached the external OSS write and received HTTP 403 with InvalidAccessKeyId. ### Release note Variant V3 preserves ordinary field names that contain internal markers. Query planning and extended DESCRIBE report metadata read errors, and a transient I/O error can be retried. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: the remote CSV OUTFILE case requires a valid OSS test fixture ### AI Disclosure - Tool: Codex - Generated: Code, tests, and description - Human review: Pending -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
