morningman opened a new pull request, #436:
URL: https://github.com/apache/doris-thirdparty/pull/436

   ### Background
   
   Independent ORC range readers have separate streams, schemas and predicates, 
but need the same file footer and absolute stripe row positions. Doris physical 
splitting creates multiple such readers for one immutable file. Shared stripe 
statistics were added in #435; serialized file tails still require each child 
to parse the footer and rebuild the row-prefix directory.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #435 (shared stripe statistics), apache/doris#70664 (downstream 
physical stripe readers)
   PR Type: Performance
   
   Problem Summary:
   
   Physical stripe children repeat file-wide tail parsing, prefix construction 
and range-directory traversal. Share an immutable parsed tail and one lazily 
initialized row-prefix buffer, then locate each reader's byte and row 
boundaries with binary searches. Metadata-only opens remain lazy, while 
streams, mutable schemas, predicates and metrics stay private. Existing 
serialized-tail callers remain supported.
   
   ### Details
   
   - **Change map:** Five files, +318/-70: production code +169/-70 for the 
shared handle, lazy prefix initialization, bounded range/seek lookup and 
counters; tests +149 for concurrency, lifetime and edge cases. The second 
commit preserves lazy metadata-only opens.
   - **Why it was slow:** `createReader` reparses the serialized tail for each 
child. Each `RowReader` also rebuilds all absolute stripe prefixes and scans 
the file-wide stripe directory, so setup grows with both child count and stripe 
count.
   - **Optimization:** Reuse immutable Footer/PostScript data through 
`SharedFileTail`. Build the source-pool-backed prefix once on the first row 
read, publish it only after successful initialization, and permit retries after 
exceptions. Binary-search range boundaries and accumulate only the owned 
interval. Validate stripe ordering before using binary search. Add counters for 
tail parsing, prefix entries and range-accumulation visits.
   - **Results:** In a local macOS ASAN Doris fixture with eight stripes, a 
planning reader and eight one-stripe children, the candidate measured the 
counts below. The comparison temporarily restored repeated setup in the same 
candidate; it was not an execution of an untouched historical commit. The 
restored path failed nine expected counter assertions while row-value 
assertions still passed. These are operation counts; no elapsed-time speedup is 
claimed.
   
     | Work | Restored repeated setup | Candidate |
     | --- | ---: | ---: |
     | File-tail parses | 9 | 1 |
     | Stripe-prefix entries built | 72 | 8 |
     | Range-accumulation stripe visits | 72 | 16 |
   
   - **Applicability and cost:** Sharing helps sibling readers of the same 
immutable file. The first row reader builds one file-wide prefix; metadata-only 
opens build none. Binary-search comparisons are additional logarithmic work 
beyond the counted visits. The source memory pool must outlive all shared 
handles. The added reader interface requires native consumers to rebuild; file 
formats remain unchanged. No RELEASE timing benchmark was run.
   
   ### Testing
   
   Locally, Doris `run-be-ut.sh` passed 255 selected tests under ASAN, with 
zero failures, errors or skips, using the exact SDK source in this PR and the 
pending Doris integration. This includes 58 SDK tests across reader, 
row-reader, read-intent, predicate and stripe-statistics suites, plus 197 Doris 
ORC tests.
   
   New SDK coverage checks cold concurrent readers after parent destruction, 
single/coalesced intervals, one shared prefix build, zero metadata-only prefix 
work, warm reuse, values and absolute row positions, seeks, serialized-tail 
compatibility, empty files/ranges and repeated malformed-directory failures. 
The downstream counter test detects the controlled restoration described above; 
the candidate was restored and its source hashes verified before the passing 
run.
   
   ### Notes for Reviewers
   
   The published head of apache/doris#70664 does not yet consume 
`SharedFileTail`. Its local follow-up passes the parsed handle to children and 
supplies the integration counters used above. This SDK PR contains only the two 
commits after merged #435 (`4ec2590..89c98c2`).
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: No; SDK lifetime and sharing contracts are 
documented in the header.
   - Pre-submission status: Partially verified: local ASAN tests passed; no 
RELEASE timing benchmark or external cluster regression was run.
   
   ### AI Disclosure
   
   - Tool: Codex; GPT-6.1-Sol for review
   - Generated: SDK implementation, tests and this description.
   - Human review: not stated
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to