morningman opened a new pull request, #68885:
URL: https://github.com/apache/doris/pull/68885
### What problem does this PR solve?
Issue Number: None
Related PR: #65868, #66297, #66247
Problem Summary:
**In short.** This ports the read batch default of branch-4.1 #65868, in the
form #66297 left it, to master's Paimon JNI scanner. A JNI read of a Paimon
table that does not set `read.batch-size` now reads in batches of the session
`batch_size` instead of Paimon's 1024 rows; a configured value still wins.
**Background.** The Paimon JNI scanner reads Paimon in batches of
`read.batch-size` rows and hands them to Doris in blocks of the session
`batch_size` (8160 by default). branch-4.1 #65868 set `read.batch-size` to the
Doris batch size. #66297, the branch-4.1 backport of master #66247, narrowed
that to readers whose table, catalog and relation options leave it unset.
Master took only the reader-option validation of #66247, so its JNI reads use
Paimon's default.
**The problem, and what it cost.** On master, a JNI read without
`read.batch-size` fills each Doris block from about eight Paimon batches of
1024 rows, where branch-4.x reads one batch per block. It changes how many rows
Paimon decodes per call and how much it buffers, not what the query returns.
**How this PR fixes it.**
- `PaimonJniScanner.initTable` calls the new `applyDefaultReadBatchSize`
after the serialized options are validated. Every reader without
`read.batch-size` gets the Doris batch size; a reader with one keeps it.
- The readers are reached through the traversal that already bounds the
manifest parallelism, now named `adjustReaders`. Each branch of a fallback pair
is set on its own and the pair is rebuilt with its read order. The source
behind a read-only system wrapper is set and the wrapper is rebuilt around it.
A table that pins an older snapshot is copied with `copyWithoutTimeTravel`, so
it keeps its latest schema.
- This differs from branch-4.x in two ways. branch-4.x leaves the whole
table alone when any reader is configured; here each reader decides for itself.
branch-4.x also copies the table as a whole, which for a fallback pair loses
the branch order that the rebuild keeps.
**Results.**
| Read | before | after |
|---|---|---|
| no `read.batch-size` anywhere | Paimon batches of 1024 rows | batches of
the session `batch_size` |
| `read.batch-size` on the table, catalog or relation | that value | that
value |
| fallback pair, only the main branch configured | main: its value;
fallback: 1024 | main: its value; fallback: `batch_size` |
| system table (`$ro`, `$audit_log`, `$files`, ...) | source reads 1024-row
batches | source reads `batch_size` batches; same wrapper class |
Original author: @suxiaogang223 (#65868), @Gabriel39 (#66297)
### Release note
Paimon tables read through the JNI scanner now read in batches of the
session `batch_size` when `read.batch-size` is not set on the table, the
catalog or the query.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `PaimonJniScannerTest`: an unconfigured reader gets the Doris
batch size through `initTable`; a configured value is kept (ported from
branch-4.x); each fallback branch is set on its own; every system table of the
bundled Paimon that loads for a primary-key table keeps its wrapper class and
gets the batch size on its source. Dropping the call from `initTable`, setting
the batch size unconditionally, or copying the table as a whole the branch-4.x
way each fails the matching new case.
- The paimon-scanner tests (117), including the existing
manifest-parallelism cases that now go through `adjustReaders`, pass.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. JNI reads without `read.batch-size` use the session
`batch_size` as the Paimon read batch.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
