eldenmoon opened a new pull request, #70730:
URL: https://github.com/apache/doris/pull/70730

   ### Background
   
   Extended DESCRIBE queries fetch inferred VARIANT subcolumns from sampled 
tablets on the backends. This applies to both the table and partition forms.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Bugfix
   
   Problem Summary: A slow schema response can hold the DESCRIBE read lock long 
enough for a Stream Load commit on the same table to fail. Capture the catalog 
metadata under the read lock and fetch the remote schema after releasing it.
   
   On a shared-nothing cluster, set `describe_extend_variant_column = true`, 
pause the schema response for `DESC t` or `DESC t PARTITION (p1)`, and run 
Stream Load against `t` from a separate session. On the baseline both loads 
returned `get tableList write lock timeout`. With this fix both loads finished 
while their respective DESCRIBE statements were still waiting.
   
   ### Details
   
   - **Frequency:** Intermittent in practice. The controlled reproduction held 
the DESCRIBE read lock past the commit deadline and reproduced the timeout for 
both statement forms. Production frequency is unknown.
   - **Root cause:** The outer DescribeCommand read lock encloses both forms of 
the remote schema fetch.
   - **Why existing tests missed it:** `ShowExecutorTest` and 
`IndexSchemaProcNodeTest` cover statement behavior and schema rendering. 
`regression_test_variant_desc` checks result contents. None of these tests 
pause a backend schema response while a Stream Load commits to the same table.
   - **Fix:** Snapshot the base schema rows and tablet replica locations under 
the read lock. Fetch and append the inferred columns outside it.
   - **Other paths and branches:** The master fix covers both forms of DESCRIBE 
and shares the snapshot logic with SHOW PROC. The same outer lock is present on 
branch-4.1. Other release branches and cloud runtime were outside this 
validation.
   
   - **Behavior change:** Stream Load can commit while extended DESCRIBE is 
waiting for backend schema metadata.
   
   ### Testing
   
   On dev-x, the ASAN master baseline a5d5a52b3f reproduced the commit 
write-lock timeout with both forms of extended DESCRIBE. Both runs spent 10,001 
ms in commit and publish. The candidate passed the same regression with 
successful loads while DESCRIBE was still paused. The commit and publish times 
were 29 ms for plain DESCRIBE and 18 ms for the partition form. The existing 
regression_test_variant_desc suite passed. IndexSchemaProcNodeTest and 
ShowExecutorTest finished with 19 passes and one skip.
   
   The affected-suite sweep covered 26 regression suites. Twenty-one passed on 
the first run. Three passed unchanged after enabling BE debug points. The two 
dotted-key suites passed with temporary test copies that read the invisible 
session variable through `SELECT @@default_variant_enable_nested_group`. The 
existing DESCRIBE assertions and goldens ran unchanged in those copies.
   
   ### Release note
   
   Extended DESCRIBE no longer holds the table lock while waiting for backend 
schemas, allowing concurrent Stream Load commits to proceed.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Ready
   
   ### AI Disclosure
   
   - Tool: Codex
   - Generated: Implementation, regression suite, and verification notes.
   - Human review: No author review statement was provided.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to