re20052 opened a new pull request, #68042:
URL: https://github.com/apache/doris/pull/68042
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Concurrent queries against the same Remote Doris Catalog table with
`use_arrow_flight=false` can stall during FE planning.
Two factors compound the problem:
1. **Lock design:** `makeSureInitialized()` and metadata refresh share the
same table monitor, which the refresh holds throughout RPC, deserialization,
partition reconstruction, and cache updates. Arriving requests block at
initialization before reaching the reuse branch. After a refresh completes, a
thread leaving initialization may immediately reacquire the monitor for another
refresh while others remain blocked, limiting result reuse.
2. **Large partition counts:** More partitions increase metadata processing
work and extend the monitor hold time, amplifying contention and request
buildup.
This PR uses a `FutureTask<RemoteOlapTable>` to represent one metadata
refresh:
- Refresh coordination holds the monitor only to create or select a task;
initialization remains synchronized.
- The initiating request runs the refresh outside the monitor, without an
executor or background thread.
- Requests selecting an unfinished task on the same table instance share it.
Each caller keeps its task reference, so a later refresh cannot replace the
result it awaits.
- If the current task has completed, successfully or exceptionally, the next
task selection creates a new refresh. No TTL cache is added.
- Refresh failures are wrapped in `AnalysisException` with the original
cause. An interrupted waiter restores its interrupt status without cancelling
the shared task.
The scope is one `RemoteDorisExternalTable` instance within one FE process.
The metadata RPC parameters, partition synchronization, and cache update logic
remain unchanged.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Manual test:
Concurrent-query validation was performed on a downstream build with the
same refresh-coordination change, using temporary diagnostic logs excluded from
this PR. It has not been rerun on this upstream branch.
Steps to reproduce the validation:
1. Create a Remote Doris Catalog with `use_arrow_flight=false`.
2. Disable SQL Cache in every query connection:
```sql
SET enable_sql_cache = false;
```
3. Run concurrent queries against the same remote table.
4. With temporary instrumentation around task selection and execution, group
logs by FE and table instance. Verify that requests selecting an unfinished
task share its task identifier and only its creator runs the refresh.
- Behavior changed:
- [ ] No.
- [x] Yes.
Requests selecting an in-progress refresh now share that task. After
completion, the next task selection starts a new refresh. Refresh failures are
consistently wrapped in `AnalysisException`, and interrupted waiters restore
their interrupt status.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
`https://github.com/apache/doris-website/pull/1214` -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
