eldenmoon opened a new pull request, #70825:
URL: https://github.com/apache/doris/pull/70825

   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #70730
   PR Type: Performance
   
   Problem Summary: Extended DESCRIBE opens segment metadata for every rowset 
in the sampled tablets. A tablet with many small rowsets can spend most of its 
schema discovery time rebuilding the same physical VARIANT paths and types.
   
   ### Details
   
   - **Change map:** The Variant module builds, deduplicates, and merges 
physical subcolumn summaries. The segment and rowset writers pass the optional 
summary through local and Cloud metadata. Tablet captures the base schema and 
visible rowsets together.
   - **Why it was slow:** The BE repeatedly collected the same paths from 
segment metadata and compared the structures of repeated paths pairwise.
   - **Optimization:** Persist the path and type observations from finalized 
segment metadata. Merge each distinct summary once, reuse the existing type 
rules and schema cache, and compare each path structure with one 
representative. Rowsets without a complete summary use the segment reader and 
cache successful reads.
   - **Results:** RELEASE on an isolated shared-nothing cluster, one tablet 
with one replica, 24 VARIANT paths, and 20 repeated DESCRIBE calls per case.
   
   | Data rowsets | First call, baseline → candidate | p50, baseline → 
candidate | p95, baseline → candidate |
   | --- | ---: | ---: | ---: |
   | 1 | 65.07 → 60.07 ms | 11.92 → 11.93 ms | 14.39 → 14.39 ms |
   | 100 with the same layout | 16.16 → 5.32 ms | 13.00 → 3.76 ms | 18.59 → 
4.51 ms |
   | 1,000 with the same layout | 177.15 → 6.01 ms | 154.79 → 3.65 ms | 227.71 
→ 4.57 ms |
   | 1,001 with a new path in the last rowset | 152.38 → 6.44 ms | 156.32 → 
3.55 ms | 196.28 → 4.72 ms |
   
   - **Applicability and cost:** The summary is optional and size bounded. With 
1,000 rowsets written before the change, the candidate took 91.8 ms on the 
first read and 18.0 ms p50 after caching. Runtime benchmarks cover 
shared-nothing.
   
   ### Testing
   
   - The four RELEASE comparison cases returned identical DESCRIBE rows. The 
candidate covered 44,142 rowset contributions using persisted summaries, with 
zero segment fallbacks. A BE restart preserved the fast path and the new path.
   - Seven focused ASAN BE unit tests passed, including a comparison of 
persisted, mixed, and segment-reader results with a type change.
   - `regression_test_variant_desc`, `test_variant_bool_path`, and 
`test_compaction_nokey_variant` passed. The `test_variant_desc_rpc_lock` suite 
from #70730 also passed.
   - The runtime checks used this patch on a #70730-based checkout. The PR 
reapplies the same patch on master at `0a6b84b5a88` as a single commit.
   
   ### Release note
   
   Extended DESCRIBE can discover VARIANT subcolumns from per-rowset metadata.
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: No
   - Pre-submission status: Partially verified: Shared-nothing runtime covered. 
Cloud runtime coverage is pending.
   
   ### AI Disclosure
   
   - Tool: Codex
   - Generated: Implementation, tests, and performance measurements.
   - Human review: No author review statement was provided.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to