morningman opened a new pull request, #66400:
URL: https://github.com/apache/doris/pull/66400
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
A cold full BE build spends most of its wall clock re-parsing a handful of
hub headers in nearly every TU. `runtime/exec_env.h`,
`runtime/runtime_state.h`
and `runtime/thread_context.h` each reached ~1000 of the ~1400 be/src TUs and
dragged multi-MB include subtrees (thrift/protobuf type universes, the AWS
SDK
surface, the workload_management family) into every one of them.
This PR adds a measurement harness and then cuts the high-payload include
edges from those hub headers, in four prepare+cut commit pairs. Measured with
the included bench mode (cold, cache-free, `-j5`, Apple M5 Pro, clang 20):
| wave | build wall | delta |
|---|---|---|
| baseline | 38m54s | |
| workload_group isolation | 35m33s | -8.4% |
| file cache / frontend info / cluster info out of exec_env.h | 32m34s |
-8.3% |
| olap_file.pb / threadpool / options out of exec_env.h | 32m40s | neutral
(incremental blast-radius + guards) |
| s3_file_system / resource_context / data.pb out of runtime_state.h +
thread_context.h | 30m21s | -7.1% |
**Total: 39m21s -> 30m48s wall (-21.7%).** Per-file compare shows 672-1065
files faster per wave and zero per-file compile-time regressions.
Key touch-count reductions (TUs that stop parsing a header):
`FrontendService_types.h` 1086->459, `BackendService_types.h` ->225,
`io/fs/s3_file_system.h` + AWS SDK 769->58, `gen_cpp/cloud.pb.h` ->192,
`workload_management/resource_context.h` 1015->214, `gen_cpp/data.pb.h`
->443,
`util/threadpool.h` -515 TUs, `storage/options.h` -837 TUs.
What is in the series (10 commits, meant to be read pairwise):
1. `build.sh --compile-bench`: cold cache-free BE bench with per-TU
wall/user/maxrss capture, report and A/B compare tooling
(`build-support/compile-bench/`)
2. include-edge analysis tools: `cut_impact.py` (blast-radius and reseed
estimation from `ninja -t deps` + parsed include graph) and
`syntax_sweep.py` (parallel `-fsyntax-only` replay of all TUs)
3. four prepare+cut pairs. Every "prepare" commit is pure-additive (forward
declarations, out-of-line sinks of cold method bodies, direct includes for
TUs that free-rode the edge about to be cut); every "cut" commit only
removes the edges and locks them with new rules in
`build-support/check-header-deps.py` (15 rules total), so none of the
cut edges can silently return.
The only code moved out of line: assigning ThreadPool setters,
`ThreadContext::attach_task`, the orphan `ResourceContext` fallback of
`resource_ctx()` (hot attached path stays inline), `ScanTask` ctor/dtor and
`DataTypeTimeStampTz::to_pb_column_meta` -- all cold paths.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- `build-support/compile-bench/syntax_sweep.py`: 1358/1358 be/src TUs
pass `-fsyntax-only` after every cut wave
- `build-support/check-header-deps.py`: 15/15 layering rules pass
- full `sh build.sh --compile-bench` builds succeed after every wave
(five complete cold builds); per-file A/B compare via
`build-support/compile-bench/report.py compare <old> <new>` shows
zero compile-time regressions
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
