This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7933a82bddd [opt](build) Enforce BE compile-time hygiene at configure
time (#66901)
7933a82bddd is described below
commit 7933a82bddd09d1e4667665706ecf833a1153ac8
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Aug 19 10:07:58 2026 +0800
[opt](build) Enforce BE compile-time hygiene at configure time (#66901)
### What problem does this PR solve?
Related PR: #66510 (the BE build-time series, #66615..#66826, that this
PR locks in)
Problem Summary:
The build-time series cut BE cold builds by ~49% and took
`olap_common.h` / `uid_util.h` off the everything-rebuilds line. Every
one of those wins can regress **silently**: a stray include, a lost
`extern template`, a header added to the PCH — the code still compiles,
only every build afterwards is slower, and nothing reports it. `#66052`
already showed the failure mode: the layering rules in
`check-header-deps.py` only worked because the author happened to
maintain them by hand; nothing in CI runs them.
This PR turns each known regression vector into a **fail-loud,
seconds-level gate that runs at configure time**, so a violation is a
build error on every path that builds the BE (CI, `build.sh`, the UT
build) with zero extra CI wiring — the same mounting pattern the FE side
already uses for its architecture gates (`exec-maven-plugin` running
`check-fe-core-metadata-funnel.sh` at `validate`).
**Checks** (pure text scans, ~1.3s combined, zero build dependency, zero
false positives on current master):
| Gate | Regression it stops |
|---|---|
| Header layering rules (30, +6 new) | a hub header (e.g. `exec_env.h`,
`function.h`, `pch.h`) re-growing an include edge into a subsystem that
was deliberately cut |
| Third-party ban table (new) |
`fmt`/`boost`/`concurrentqueue.h`/`<ranges>` returning to headers whose
bodies were moved out of line to contain template-instantiation cost
(`<ranges>` in a src header also breaks the `-fno-access-control` UT
build on libc++) |
| Two-axis budgets (new) | the edges nobody names: light hubs must stay
light (include-closure budget, zero slack), heavy payloads must not
spread (transitive-reach baseline +10%); `--closure` / `--reach`
subcommands rank the offending edges for the fix |
| pch whitelist (new) | any quoted include added to `pch/pch.h` — the
single most leveraged regression in the repo (touch one such header and
the whole backend plus the PCH rebuild) |
| `check-extern-template-pairing.py` (new) | the one **silent**
extern-template failure: a `.cpp` keeps the explicit instantiation but
the header loses the `extern` declaration, so every TU quietly
re-instantiates; also the loud direction (declaration with no
definition), reported at once instead of at link |
| `check-unity-skip-coverage.py` (new) | a test `#include`-ing a src
`.cpp` that is not opted out of unity batching — reported at configure
with the exact skip entry to add, instead of as duplicate symbols at the
end of the BE UT link |
`build-support/check-build-hygiene.sh` runs all three scripts;
`be/CMakeLists.txt` executes it at configure time. Escape hatches, by
design:
- `-DENABLE_BUILD_HYGIENE=OFF` (default ON; a missing `python3` is a
loud FATAL naming this switch, never a silent skip);
- every budget/whitelist/rule table lives in the scripts themselves, so
the legal way past a gate is a **one-line reviewed diff in the same PR**
— the gate is never a dead end. Every failure message carries the
mechanism (why this edge is expensive) and the concrete fix path.
**Found by the new gates on day one, fixed here:** `core/types.h` has
carried five `extern template struct fmt::formatter<Decimal*>;`
declarations with **no matching definition anywhere** since #53483 —
every member is defined in-class, so the linker never noticed; deleted.
Also made the FE funnel self-test's `sed -i` invocation portable to
BSD/macOS.
Blind spot, by design: instantiations expanded from macros (e.g.
`DECLARE_OPERATOR` in `operator.cpp`) are invisible to the pairing scan
on both sides — macro bodies are skipped; a comment at the site says so.
`be/README.md` (new) documents the gates, what to do when each fires,
and the `*_fwd.h` convention as the sanctioned way through a layering
barrier.
The same rules are encoded for AI-assisted development and review:
`be/AGENTS.md` (new — include discipline while writing BE code + review
checkpoints), `build-support/AGENTS.md` (new — maintenance discipline
for the gate scripts themselves), an include-src-cpp checkpoint in
`be/test/AGENTS.md`, and a Header Hygiene entry in the root `AGENTS.md`,
so coding agents and the review pipeline apply them automatically.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
- `build-support/check-build-hygiene.sh` on master: all green, ~1.3s,
zero false positives.
- Self-tests `bash build-support/tests/run.sh` (7/7 PASS): red/green
injection per gate family — each seeded violation turns red with the
mechanism + fix in the message, restoring the file turns it green again.
- Real configure on macOS arm64: green path passes the gate section;
with an injected violation, configure dies at the gate in under a second
with the full message; `-DENABLE_BUILD_HYGIENE=OFF` bypasses; a missing
`python3` is a loud exit-2 FATAL naming the switch.
- This PR's own Linux CI exercises the configure mount on the second
platform.
- Behavior changed:
- [x] Yes. BE configure now fails fast (with mechanism + fix in the
message) on header-hygiene violations; emergency bypass via
`-DENABLE_BUILD_HYGIENE=OFF`. No runtime behavior change — the only
source-code changes are comments and the deletion of five never-defined
`extern template` declarations.
- Does this need documentation?
- [x] No. Developer-facing docs are included in-repo (`be/README.md`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01EyZL631YTeuNXa9WLjvk7s
---------
Co-authored-by: Claude Fable 5 <[email protected]>
---
AGENTS.md | 2 +
be/AGENTS.md | 70 +++
be/CMakeLists.txt | 26 ++
be/README.md | 85 ++++
be/src/core/column/column_decimal.cpp | 2 +
be/src/core/column/column_string.cpp | 2 +
be/src/core/column/column_vector.cpp | 4 +
be/src/core/types.h | 6 -
be/src/exec/operator/operator.cpp | 5 +
be/test/AGENTS.md | 4 +
build-support/AGENTS.md | 49 ++
build-support/check-build-hygiene.sh | 118 +++++
build-support/check-extern-template-pairing.py | 271 ++++++++++++
build-support/check-header-deps.py | 491 ++++++++++++++++++++-
build-support/check-unity-skip-coverage.py | 226 ++++++++++
build-support/tests/test-build-hygiene-entry.sh | 101 +++++
.../tests/test-build-hygiene-extern-pairing.sh | 91 ++++
.../tests/test-build-hygiene-header-deps.sh | 136 ++++++
.../tests/test-build-hygiene-unity-skip.sh | 92 ++++
.../tests/test-fe-core-metadata-funnel.sh | 4 +-
20 files changed, 1761 insertions(+), 24 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index 34617013274..5d551bb1c3e 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -30,6 +30,8 @@ All code must pass style checks before committing. Use the
corresponding skill f
**BE (C++) Static Analysis**: After building BE (which generates
`compile_commands.json`), run `build-support/run-clang-tidy.sh` to check
modified C++ files against the `.clang-tidy` config. The script parses `git
diff` to filter warnings to changed lines where possible, reducing noise from
pre-existing code (diagnostics from included headers may still appear). For
Cloud C++ files, pass `--build-dir` pointing to the Cloud compilation database
(e.g., `cloud/build_ASAN`). Try to fix all re [...]
+**BE (C++) Header Hygiene**: BE configure enforces compile-time hygiene gates
(header layering rules, include-closure/reach budgets, pch whitelist,
extern-template pairing, unity-skip coverage) via
`build-support/check-build-hygiene.sh` (~1s, pure text). Run it directly after
touching BE headers, includes, template instantiation lists, or unity skip
lists — do not wait for CI. Violations are build errors whose messages carry
the mechanism and fix; deliberate budget/whitelist changes are [...]
+
**FE (Java) Style**: Checkstyle is integrated into the Maven build
(`maven-checkstyle-plugin`). Running `build.sh --fe` automatically validates
style via `mvn validate`. If checkstyle fails, fix the reported issues
according to `fe/check/checkstyle/checkstyle.xml`. See the `fe-code-style`
skill for details.
## Code Review
diff --git a/be/AGENTS.md b/be/AGENTS.md
new file mode 100644
index 00000000000..497d8a10eda
--- /dev/null
+++ b/be/AGENTS.md
@@ -0,0 +1,70 @@
+# AGENTS.md — Doris BE
+
+Scope: `be/`. This file is operational — what to do while developing or
+reviewing BE C++ code so the compile-time invariants hold. The gate mechanics
+and the `*_fwd.h` convention are documented in `be/README.md` (read it when a
+gate fires). Repository-wide rules (commit format, build/test commands,
+clang-format/clang-tidy) live in the root `AGENTS.md`; nothing here overrides
+them.
+
+## Machine-Checked Obligations (configure-time gates)
+
+BE configure runs `build-support/check-build-hygiene.sh` (option
+`ENABLE_BUILD_HYGIENE`, default ON; scripts and self-tests live in
+`build-support/` and `build-support/tests/`). It is pure text and takes about
+a second — **run it directly after any change touching BE headers, includes,
+template instantiations, unity skip lists, or test files that include src
+sources**; do not wait for configure or CI to tell you.
+
+Every failure message carries the mechanism and the fix path. The legal way
+past a gate is never to bypass it, but one of:
+
+- fix the edge as the message says (forward-declare + include in the `.cpp`,
+ or route declarations through a `*_fwd.h`);
+- when the change is deliberate, edit the corresponding table
+ (rule exception set / `ANGLE_BANS` / `FORWARD_CLOSURE_BUDGETS` /
+ `REVERSE_REACH_BASELINES` / `PCH_QUOTED_WHITELIST` / `ALLOW`) **in the same
+ commit** and justify it in the commit message — the table diff is the
+ review signal.
+
+## Rules while writing BE code
+
+- **Adding an `#include` to a widely-included (hub) header is a design
+ decision, not a convenience.** Everything a hub includes is reparsed by
+ every TU behind it (~1000 TUs for `exec_env.h`, `thread_context.h`,
+ `runtime_state.h`, `function.h`, `dependency.h`, `column.h`, ...). Prefer
+ forward declarations; put the real include in the `.cpp`. If many files
+ need the declarations, use a `*_fwd.h` (declarations and lightweight
+ aliases only — never bodies or non-fwd project includes).
+- **Never add a quoted include to `pch/pch.h`.** Every header on the PCH
+ rebuilds the whole backend (plus the PCH itself) when touched; it is the
+ single most leveraged regression surface in the repo.
+- **Keep `extern template` families paired.** An explicit instantiation in a
+ `.cpp` needs the matching `extern template` in the header, spelled with the
+ same template arguments — a missing extern compiles and links fine and just
+ silently re-instantiates in every TU. Instantiations expanded from macros
+ (e.g. `DECLARE_OPERATOR` in `operator.cpp`) are invisible to the pairing
+ gate on both sides: keep those in sync by hand.
+- **A test that `#include`s a be/src `.cpp`** needs that file opted out of
+ unity batching via `doris_skip_unity_inclusion` in the owning
+ `be/src/.../CMakeLists.txt`; otherwise the BE UT link fails with duplicate
+ symbols an hour later. The gate error names the exact entry to add.
+- **Do not re-add banned third-party includes** (`fmt`/`boost`/
+ `concurrentqueue.h`/`<ranges>`) to the headers listed in `ANGLE_BANS`:
+ their bodies were deliberately moved out of line, and `<ranges>` in a src
+ header additionally breaks the `-fno-access-control` UT build on libc++.
+
+## Review checkpoints (AI review and self-review)
+
+- [ ] New includes in hub headers: could a forward declaration or `*_fwd.h`
+ carry this instead? Does the PR pay a closure/reach budget bump — and
+ if so, does the commit message justify it?
+- [ ] Any edit to a gate table (`RULES` exceptions, budgets, whitelist,
+ `ALLOW`) must be deliberate, minimal, and explained in the same
+ commit; an unexplained table edit is a red flag, not a fix.
+- [ ] Any change to `pch/pch.h` is near-always wrong; demand the reasoning.
+- [ ] New explicit instantiation lists or `extern template` blocks: both
+ sides present, same spelling? New test `#include` of a src `.cpp`:
+ skip entry present?
+- [ ] `git grep` for a deleted/renamed header in skip lists and gate tables:
+ stale entries fail configure loudly — fix them in the same PR.
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index b2f1b8129ce..093a787cd9f 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -159,6 +159,32 @@ endif()
set(THIRDPARTY_DIR "$ENV{DORIS_THIRDPARTY}/installed")
message(STATUS "THIRDPARTY_DIR is ${THIRDPARTY_DIR}")
+# Build-hygiene gates: seconds-level, zero-build-dependency text checks that
+# keep the backend's compile-time invariants from regressing (header layering
+# rules and budgets, extern-template pairing, unity-skip coverage for
+# test-included sources). Running them at configure time makes a violation a
+# build error on every path that builds the BE -- CI, build.sh, the UT build --
+# with no separate wiring, and it fails in the first seconds instead of
+# somewhere inside a long compile. Every failure message carries the mechanism
+# and the fix path; see build-support/check-build-hygiene.sh.
+option(ENABLE_BUILD_HYGIENE "Run the build-hygiene gates at configure time" ON)
+message(STATUS "ENABLE_BUILD_HYGIENE is ${ENABLE_BUILD_HYGIENE}")
+if (ENABLE_BUILD_HYGIENE)
+ execute_process(
+ COMMAND bash "${BASE_DIR}/../build-support/check-build-hygiene.sh"
+ ERROR_VARIABLE BUILD_HYGIENE_ERRORS
+ RESULT_VARIABLE BUILD_HYGIENE_RESULT
+ )
+ if (NOT BUILD_HYGIENE_RESULT EQUAL 0)
+ message(FATAL_ERROR
+ "build-hygiene gates failed (exit ${BUILD_HYGIENE_RESULT}):\n"
+ "${BUILD_HYGIENE_ERRORS}"
+ "Fix the violation (each message ends with its fix path), or "
+ "configure with -DENABLE_BUILD_HYGIENE=OFF to bypass in an "
+ "emergency.")
+ endif()
+endif()
+
option(MAKE_TEST "ON for make unit test or OFF for not" OFF)
message(STATUS "make test: ${MAKE_TEST}")
option(BUILD_BENCHMARK "ON for make google benchmark or OFF for not" OFF)
diff --git a/be/README.md b/be/README.md
new file mode 100644
index 00000000000..758e6407a9b
--- /dev/null
+++ b/be/README.md
@@ -0,0 +1,85 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Doris BE development notes
+
+## Compile-time hygiene gates
+
+BE configure runs `build-support/check-build-hygiene.sh` (option
+`ENABLE_BUILD_HYGIENE`, default `ON`): a set of seconds-level text checks that
+keep the backend's compile-time invariants from regressing. They exist because
+include-graph regressions are silent -- the code still compiles, only every
+build afterwards is slower. The gates make them loud, in the first seconds of
+configure, with the mechanism and the fix in the message.
+
+What each gate guards, and what to do when it fires:
+
+* **Header layering rules** (`check-header-deps.py`). A hub header
+ (`exec_env.h`, `thread_context.h`, ...) must not reach a named subsystem,
+ because everything a hub includes is reparsed by the ~1000 TUs behind it.
+ Fix: forward-declare the type in the header and include the real header in
+ the `.cpp`; if only declarations are needed by many files, route them
+ through a `*_fwd.h` (see below). A genuinely leaf-like header can be added
+ to the rule's exception set -- in the same PR, with the reasoning in the
+ commit message.
+* **Third-party bans**. A few headers deliberately moved their `fmt`/`boost`/
+ `<ranges>`-using bodies out of line; the ban keeps the template machinery
+ from coming back. Fix: put the code that needs the library into the matching
+ `.cpp`.
+* **Closure / reach budgets**. The safety net for edges no rule names: light
+ hubs must stay light (`--closure <header>` lists what grew), heavy payloads
+ must not spread (`--reach <header>` ranks the spreading edges). Fix: cut the
+ new edge, or -- when the growth is intended -- bump the number in the budget
+ table in the same PR and say why in the commit message. The diff of the
+ budget table is the review signal; the gate is never a dead end.
+* **PCH whitelist**. `pch/pch.h`'s quoted includes are pinned exactly: every
+ header on the PCH rebuilds the whole backend when touched. Adding one is
+ almost never right; if it is, change the whitelist in the same PR.
+* **extern template pairing** (`check-extern-template-pairing.py`). Every
+ `extern template` declaration in a header must have its explicit
+ instantiation definition in a `.cpp`, and -- within a family that uses
+ externs -- every definition must have its declaration. The missing-extern
+ direction is the one silent case: it compiles and links, every TU just
+ quietly re-instantiates the specialization. Fix: keep both sides in sync,
+ spelling the template arguments the same way.
+* **Unity-skip coverage** (`check-unity-skip-coverage.py`). A test that
+ `#include`s a be/src `.cpp` needs that file opted out of unity batching
+ (`doris_skip_unity_inclusion`), otherwise the BE UT link fails with
+ duplicate symbols an hour later. The error names the exact entry to add.
+
+The gates are python3 sources and resolve their own interpreter (`python3`,
+then `python`, then `/usr/libexec/platform-python`, the RHEL8/AlmaLinux8 system
+Python that the build-env image ships in place of `/usr/bin/python3`). They
+deliberately ignore `PYTHON`, which `env.sh` exports as the interpreter the
+*build* uses for code generation and which is python2 by default. Set
+`BUILD_HYGIENE_PYTHON` to pin a specific interpreter.
+
+Escape hatch for emergencies: configure with `-DENABLE_BUILD_HYGIENE=OFF`.
+Self-tests live in `build-support/tests/` (`run.sh`).
+
+## The `*_fwd.h` convention
+
+`*_fwd.h` headers are the sanctioned way through a layering barrier: they
+carry forward declarations (and lightweight aliases) only, so they cost
+nothing to include and are exempt from the layering rules by suffix. When a
+hub header needs a subsystem's type names but not its definitions, put the
+declarations in `<subsystem>_fwd.h` (precedent:
+`exec/common/hash_table/phmap_fwd_decl.h`) and include the real headers only
+in `.cpp` files. Do not put function bodies, class bodies, or includes of
+non-fwd project headers into a `*_fwd.h`.
diff --git a/be/src/core/column/column_decimal.cpp
b/be/src/core/column/column_decimal.cpp
index fed2fe04c3a..314bccd5d54 100644
--- a/be/src/core/column/column_decimal.cpp
+++ b/be/src/core/column/column_decimal.cpp
@@ -591,6 +591,8 @@ typename ColumnDecimal<T>::CppNativeType
ColumnDecimal<T>::get_fractional_part(s
}
}
+// A new instantiation here needs the matching 'extern template' declaration
+// in column_decimal.h (enforced by check-extern-template-pairing.py).
template class ColumnDecimal<TYPE_DECIMAL32>;
template class ColumnDecimal<TYPE_DECIMAL64>;
template class ColumnDecimal<TYPE_DECIMALV2>;
diff --git a/be/src/core/column/column_string.cpp
b/be/src/core/column/column_string.cpp
index b8b7a6c451e..1ea2f24c1e7 100644
--- a/be/src/core/column/column_string.cpp
+++ b/be/src/core/column/column_string.cpp
@@ -774,6 +774,8 @@ bool ColumnStr<T>::is_valid_utf8() const {
return true;
}
+// A new instantiation here needs the matching 'extern template' declaration
+// in column_string.h (enforced by check-extern-template-pairing.py).
template class ColumnStr<uint32_t>;
template class ColumnStr<uint64_t>;
} // namespace doris
diff --git a/be/src/core/column/column_vector.cpp
b/be/src/core/column/column_vector.cpp
index 31e4862187f..00de7a17ba7 100644
--- a/be/src/core/column/column_vector.cpp
+++ b/be/src/core/column/column_vector.cpp
@@ -545,6 +545,10 @@ void ColumnVector<T>::replace_float_special_values() {
}
/// Explicit template instantiations - to avoid code bloat in headers.
+/// A new instantiation here needs the matching 'extern template' declaration
+/// in column_vector.h: without it, every TU that uses the specialization
+/// silently instantiates its own copy again (enforced by
+/// build-support/check-extern-template-pairing.py at configure time).
template class ColumnVector<TYPE_BOOLEAN>;
template class ColumnVector<TYPE_TINYINT>;
template class ColumnVector<TYPE_SMALLINT>;
diff --git a/be/src/core/types.h b/be/src/core/types.h
index 0ea2a81c3bf..e732c31d3e8 100644
--- a/be/src/core/types.h
+++ b/be/src/core/types.h
@@ -626,9 +626,3 @@ struct fmt::formatter<doris::Decimal<T>> {
return fmt::format_to(ctx.out(), "{}", to_string(value.value));
}
};
-
-extern template struct fmt::formatter<doris::Decimal32>;
-extern template struct fmt::formatter<doris::Decimal64>;
-extern template struct fmt::formatter<doris::Decimal128V2>;
-extern template struct fmt::formatter<doris::Decimal128V3>;
-extern template struct fmt::formatter<doris::Decimal256>;
diff --git a/be/src/exec/operator/operator.cpp
b/be/src/exec/operator/operator.cpp
index cfc9d664511..2eeb27d2970 100644
--- a/be/src/exec/operator/operator.cpp
+++ b/be/src/exec/operator/operator.cpp
@@ -819,6 +819,11 @@ Status AsyncWriterSink<Writer,
Parent>::close(RuntimeState* state, Status exec_s
return Base::close(state, exec_status);
}
+// An instantiation added outside the macros below needs the matching
+// 'extern template' declaration in the operator's header (enforced by
+// build-support/check-extern-template-pairing.py). Instantiations expanded
+// through DECLARE_OPERATOR are invisible to that check -- macro bodies are
+// skipped -- so their extern pairing has to be kept in sync by hand.
#define DECLARE_OPERATOR(LOCAL_STATE) template class
DataSinkOperatorX<LOCAL_STATE>;
DECLARE_OPERATOR(HashJoinBuildSinkLocalState)
DECLARE_OPERATOR(ResultSinkLocalState)
diff --git a/be/test/AGENTS.md b/be/test/AGENTS.md
index 3b0e9fed7a6..ce6d84d22f1 100644
--- a/be/test/AGENTS.md
+++ b/be/test/AGENTS.md
@@ -3,3 +3,7 @@
## Access Control
BE-UT has actually been configured to ignore access control, so it can access
all private interfaces.
+
+## Including src .cpp files
+
+- [ ] A test that `#include`s a be/src `.cpp` (to reach file-static helpers)
must have that file opted out of unity batching via
`doris_skip_unity_inclusion` in the owning `be/src/.../CMakeLists.txt` —
otherwise the batch object's copy of the definitions collides with the test's
inlined copy and the BE UT link fails with duplicate symbols. Enforced at
configure time by `build-support/check-unity-skip-coverage.py`; its error
message names the exact entry to add. Prefer not including `.cpp [...]
diff --git a/build-support/AGENTS.md b/build-support/AGENTS.md
new file mode 100644
index 00000000000..8f140275ee5
--- /dev/null
+++ b/build-support/AGENTS.md
@@ -0,0 +1,49 @@
+# AGENTS.md — build-support
+
+Operational rules for the architecture/hygiene gates in this directory (the
+`check-*.sh` / `check-*.py` scripts wired into FE `mvn validate` and BE
+configure). These scripts fail other people's builds — treat them as
+production code with a stricter bar than the code they guard.
+
+## Gate maintenance discipline
+
+- **Zero false positives, non-negotiable.** A per-commit gate must be
+ deterministic, pure-text, and second-level. Anything heuristic
+ (nm audits, closure sweeps, wall-clock trends) belongs in offline reports,
+ never in the configure/validate path. One false positive burns more trust
+ than ten missed regressions.
+- **Fail loud, never silently skip.** A missing tool, a table entry pointing
+ at a renamed file, a sentinel nobody references any more — each is an
+ error with a message, not a silent pass (precedents:
+ `doris_skip_unity_inclusion` in `be/CMakeLists.txt`, the missing-python3
+ branch of `check-build-hygiene.sh`, the missing-header errors in
+ `check-header-deps.py`).
+- **Every failure message carries three parts**: the violation, the
+ *mechanism* (why this edge/entry is expensive — model:
+ `check-header-deps.py`'s reason/chain/fix form), and a concrete fix path
+ the reader can act on without opening the script.
+- **Escape hatches are tables in the script, not flags.** Budgets,
+ whitelists, exception sets and `ALLOW` lists live next to the rules so a
+ deliberate change is a one-line reviewed diff in the same commit. Do not
+ add bypass environment variables or config files.
+- **Rebaselining budgets**: forward closure budgets carry zero slack (bump =
+ explicit, justified edit); reverse reach baselines carry +10% and are
+ re-measured on the audit cadence with `check-header-deps.py --budget`.
+
+## Changing a gate script
+
+- Any behavior change to a `check-*` script requires updating its self-test
+ in `build-support/tests/` (red/green injection form: every seeded
+ violation turns red with the fix path in the message, restoring turns it
+ green) and running `bash build-support/tests/run.sh` — all of it, since
+ the entry scripts aggregate.
+- The BE hygiene self-tests briefly mutate working-tree files (backed up and
+ restored by EXIT traps): do not run them concurrently with a
+ build/configure of the same tree, and do not "fix" them by pointing at
+ fixtures — the gate tables name real headers on purpose.
+- Keep scripts portable: bash 3.2 (macOS), BSD *and* GNU userland — in
+ particular `sed -i` needs the `-i.bak` + `rm` form. Python: stdlib only,
+ no third-party imports.
+- Performance envelope: the combined configure gate is ~1s today; keep any
+ addition within a low single-digit second budget, with zero build
+ dependency (no compiler, no compile_commands.json).
diff --git a/build-support/check-build-hygiene.sh
b/build-support/check-build-hygiene.sh
new file mode 100755
index 00000000000..5f0a51619a8
--- /dev/null
+++ b/build-support/check-build-hygiene.sh
@@ -0,0 +1,118 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Unified entry for the BE build-hygiene gates. Runs every seconds-level,
+# zero-build-dependency check that protects the backend's compile-time
+# invariants:
+#
+# check-header-deps.py header layering rules, third-party bans,
+# the two-axis closure/reach budgets and the
+# pch include lock
+# check-extern-template-pairing.py extern template declarations and explicit
+# instantiation definitions stay paired in
+# both directions
+# check-unity-skip-coverage.py every src .cpp a test #includes is opted
+# out of unity batching
+#
+# All checks are pure text scans and run in about a second combined. Every
+# failure message carries the mechanism (why the edge/entry is expensive) and
+# the fix path. All checks run even when an early one fails, so one pass shows
+# everything there is to fix.
+#
+# Mounted at configure time by be/CMakeLists.txt (ENABLE_BUILD_HYGIENE, default
+# ON), which is what makes a violation a build error rather than advice. To run
+# by hand:
+#
+# build-support/check-build-hygiene.sh
+#
+# Exit code: 0 -- all checks passed; 1 -- at least one violation (details on
+# stderr); 2 -- environment problem (no Python 3 interpreter).
+#
+# Self-tests: build-support/tests/test-build-hygiene-*.sh (run.sh runs them).
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# The gates are python3 sources, so they need a Python 3 interpreter -- which
is
+# not the interpreter the build itself uses. env.sh exports PYTHON as
+# DORIS_BUILD_PYTHON_VERSION, defaulting to `python`; on the official build-env
+# image (FROM almalinux:8) that is python2.7, under which every gate dies with
a
+# SyntaxError on its first f-string. So resolve an interpreter here and never
+# read PYTHON.
+#
+# The probe order covers the two shapes that exist in practice: `python3` on
+# developer boxes and most distros, and /usr/libexec/platform-python -- the
+# RHEL8/AlmaLinux8 system interpreter (3.6), which the build-env image ships
+# while carrying no /usr/bin/python3 at all.
+#
+# BUILD_HYGIENE_PYTHON names an interpreter explicitly (a venv, a non-standard
+# prefix). It selects which python runs the gates; it cannot switch them off --
+# that is -DENABLE_BUILD_HYGIENE=OFF.
+is_python3() {
+ "$1" -c 'import sys; sys.exit(0 if sys.version_info >= (3, 6) else 1)' \
+ >/dev/null 2>&1
+}
+
+PYTHON=""
+if [ -n "${BUILD_HYGIENE_PYTHON:-}" ]; then
+ # An explicit choice is honoured strictly: if it is unusable, say so rather
+ # than quietly running the gates under some other interpreter.
+ if is_python3 "${BUILD_HYGIENE_PYTHON}"; then
+ PYTHON="${BUILD_HYGIENE_PYTHON}"
+ else
+ echo "check-build-hygiene:
BUILD_HYGIENE_PYTHON=${BUILD_HYGIENE_PYTHON} is not found, or is not Python >=
3.6." >&2
+ echo "Point it at a Python 3 interpreter, or re-run with
-DENABLE_BUILD_HYGIENE=OFF to skip the gates." >&2
+ exit 2
+ fi
+else
+ for candidate in python3 python /usr/libexec/platform-python; do
+ if is_python3 "${candidate}"; then
+ PYTHON="${candidate}"
+ break
+ fi
+ done
+ if [ -z "${PYTHON}" ]; then
+ echo "check-build-hygiene: no Python >= 3.6 found (tried python3,
python, /usr/libexec/platform-python); the build-hygiene gates need one." >&2
+ echo "Install python3 or set BUILD_HYGIENE_PYTHON, or re-run with
-DENABLE_BUILD_HYGIENE=OFF to skip the gates." >&2
+ exit 2
+ fi
+fi
+
+FAILURES=0
+for check in \
+ check-header-deps.py \
+ check-extern-template-pairing.py \
+ check-unity-skip-coverage.py; do
+ if ! "${PYTHON}" "${SCRIPT_DIR}/${check}"; then
+ FAILURES=$((FAILURES + 1))
+ fi
+done
+
+if [ "${FAILURES}" -ne 0 ]; then
+ echo "" >&2
+ echo "build hygiene: ${FAILURES} check(s) failed (details above)." >&2
+ echo "Each message ends with its fix path; the budget/whitelist tables
live in the" >&2
+ echo "check scripts themselves, so a deliberate change is a one-line,
reviewed diff." >&2
+ exit 1
+fi
+# Name the resolved interpreter: it is the one thing about this run that varies
+# per environment, and the configure log is where you look when a gate behaves
+# differently in CI than it did locally.
+echo "build hygiene: all checks passed (python: ${PYTHON} $("${PYTHON}" -c
'import sys; print("%d.%d.%d" % sys.version_info[:3])'))"
diff --git a/build-support/check-extern-template-pairing.py
b/build-support/check-extern-template-pairing.py
new file mode 100755
index 00000000000..602c99d2cad
--- /dev/null
+++ b/build-support/check-extern-template-pairing.py
@@ -0,0 +1,271 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Pairing guard for `extern template` families.
+
+The BE uses extern-template families (ColumnVector, ColumnDecimal, the operator
+and serde stacks, ...) to instantiate hot template specializations once instead
+of in every translation unit. Of all the ways such a pairing can rot, exactly
+one is silent: a .cpp carries the explicit instantiation definition but the
+header lost (or never gained) the matching `extern template` declaration. The
+code still compiles and links -- every TU just quietly goes back to implicitly
+instantiating that specialization, and the build slows down with no diagnostic
+anywhere. This script turns both pairing directions into an error:
+
+ forward every `extern template` declaration in a be/src header must have a
+ matching explicit instantiation definition in some be/src .cpp
+ (missing ones eventually break the link, but only when a TU
odr-uses
+ a non-inline member -- this reports them at once, with the file the
+ definition should go in).
+ reverse within a COVERED family -- a template name that already has at
least
+ one extern declaration -- every explicit instantiation definition
+ must have a matching extern declaration (the silent case). Families
+ with no externs at all are intentionally out of scope:
instantiation
+ without extern is a legitimate TU-local idiom there.
+
+Matching is textual, on a normalized signature: whitespace folded, comments
+stripped, namespace qualifiers dropped, known type aliases canonicalized, and
+multi-line statements joined. Zero build dependency; runs in about half a
+second.
+
+Usage:
+ build-support/check-extern-template-pairing.py # enforce
+ build-support/check-extern-template-pairing.py --list # dump the
pairing tables
+"""
+
+import argparse
+import collections
+import os
+import re
+import sys
+
+SRC_ROOT = "be/src"
+
+# Spellings that legitimately differ between a declaration site and its
+# definition site are rewritten to one canonical form ON BOTH SIDES before
+# matching. Because the rewrite is uniform, a pair that already spells its
+# arguments consistently can never be broken by an entry here; entries are only
+# ever needed when the header and the .cpp spell the same type differently.
+# Keep this table small -- the better fix for a new mismatch is to spell both
+# sides identically.
+TYPE_ALIASES = {
+ # core/types.h vectorized aliases vs the underlying fixed-width types
+ # (column_string.h declares ColumnStr<UInt32>, column_string.cpp
+ # instantiates ColumnStr<uint32_t>).
+ "UInt32": "uint32_t",
+ "UInt64": "uint64_t",
+ # wide:: aliases vs the underlying wide::integer specializations
+ # (wide_integer_to_string.h declares to_string(const Int128&), the .cpp
+ # instantiates to_string(const integer<128, signed>&)).
+ "Int128": "integer<128,signed>",
+ "UInt128": "integer<128,unsigned>",
+ "Int256": "integer<256,signed>",
+ "UInt256": "integer<256,unsigned>",
+}
+
+# Escape hatch, deliberately empty. An entry is
+# ("be/src/path/to/file.ext", "<normalized signature>")
+# as printed by a failure (or by --list), and silences that one declaration or
+# definition. Adding one must be a reviewed decision with a comment saying why
+# the pairing is intentionally broken.
+ALLOW = set()
+
+EXTERN_START = re.compile(r"^\s*extern\s+template\b")
+# An explicit instantiation begins `template` followed by anything but `<`
+# (`template <...>` opens a template definition, not an instantiation).
+DEF_START = re.compile(r"^\s*template\s+[A-Za-z_:]")
+LINE_COMMENT = re.compile(r"//.*$")
+BLOCK_COMMENT = re.compile(r"/\*.*?\*/")
+
+
+def statements(path, start):
+ """Yields (lineno, statement) for namespace-scope statements matching
+ `start`, joining continuation lines up to the terminating semicolon and
+ skipping preprocessor directives and macro bodies (an instantiation-shaped
+ line inside a #define is not an instantiation)."""
+ with open(path, encoding="utf-8", errors="ignore") as handle:
+ lines = handle.read().splitlines()
+ in_macro = False
+ i = 0
+ while i < len(lines):
+ raw = lines[i]
+ if in_macro or raw.lstrip().startswith("#"):
+ in_macro = raw.rstrip().endswith("\\")
+ i += 1
+ continue
+ if not start.match(raw):
+ i += 1
+ continue
+ stmt, j = raw, i
+ while ";" not in stmt and j + 1 < len(lines) and j - i < 20:
+ j += 1
+ stmt += " " + lines[j]
+ if ";" in stmt and "\\" not in stmt:
+ yield i + 1, stmt.split(";")[0].strip()
+ i = j + 1
+
+
+def normalize(sig):
+ """Collapses a declaration/definition to the comparable skeleton."""
+ s = BLOCK_COMMENT.sub(" ", sig)
+ s = LINE_COMMENT.sub(" ", s)
+ s = re.sub(r"\s+", " ", s).strip()
+ s = re.sub(r"^extern\s+", "", s)
+ s = re.sub(r"^template\s+", "", s)
+ s = re.sub(r"^(class|struct)\s+", "", s)
+ s = re.sub(r"\b[A-Za-z_]\w*::", "", s)
+ for alias, canonical in TYPE_ALIASES.items():
+ s = re.sub(rf"\b{alias}\b", canonical, s)
+ return s.replace(" ", "")
+
+
+def family(norm):
+ """The template name a normalized signature instantiates: the identifier
+ before the argument list for a type, the function name for a function."""
+ paren = norm.find("(")
+ if paren != -1:
+ m = re.search(r"([A-Za-z_]\w*)(?:<[^()]*>)?\($", norm[: paren + 1])
+ if m:
+ return m.group(1)
+ m = re.match(r"([A-Za-z_]\w*)<", norm)
+ return m.group(1) if m else norm
+
+
+def walk(suffixes):
+ for directory, _, names in os.walk(SRC_ROOT):
+ for name in names:
+ if name.endswith(suffixes):
+ yield os.path.join(directory, name)
+
+
+def collect():
+ externs, defs = [], []
+ for path in walk((".h", ".hpp")):
+ with open(path, encoding="utf-8", errors="ignore") as handle:
+ if "extern template" not in handle.read():
+ continue
+ for lineno, stmt in statements(path, EXTERN_START):
+ externs.append((path, lineno, stmt, normalize(stmt)))
+ for path in walk((".cpp", ".cc")):
+ for lineno, stmt in statements(path, DEF_START):
+ if re.match(r"^\s*extern\b", stmt):
+ continue
+ defs.append((path, lineno, stmt, normalize(stmt)))
+ return externs, defs
+
+
+def enforce(externs, defs):
+ def_index = collections.defaultdict(list)
+ for path, lineno, stmt, norm in defs:
+ def_index[norm].append((path, lineno))
+ extern_index = collections.defaultdict(list)
+ covered = collections.defaultdict(set) # family -> headers declaring it
+ for path, lineno, stmt, norm in externs:
+ extern_index[norm].append((path, lineno))
+ covered[family(norm)].add(path)
+
+ failures = 0
+ for path, lineno, stmt, norm in externs:
+ if norm in def_index or (path, norm) in ALLOW:
+ continue
+ failures += 1
+ print(
+ "error: extern template declaration pairs with no explicit "
+ "instantiation definition",
+ file=sys.stderr,
+ )
+ print(f" decl: {path}:{lineno}: {stmt};", file=sys.stderr)
+ print(f" norm: {norm}", file=sys.stderr)
+ print(
+ " reason: 'extern template' promises the specialization is "
+ "instantiated in some .cpp; without one, any TU that odr-uses a "
+ "non-inline member fails to link -- and if every member is inline,
"
+ "the declaration is dead weight",
+ file=sys.stderr,
+ )
+ print(
+ " fix: add the matching 'template class/struct/... ;' to the "
+ ".cpp that owns this family, or delete the declaration; an "
+ "intentionally unpaired declaration goes in ALLOW with a comment",
+ file=sys.stderr,
+ )
+ for path, lineno, stmt, norm in defs:
+ fam = family(norm)
+ if fam not in covered or norm in extern_index or (path, norm) in ALLOW:
+ continue
+ failures += 1
+ headers = ", ".join(sorted(covered[fam]))
+ print(
+ "error: explicit instantiation lacks the matching 'extern "
+ "template' declaration",
+ file=sys.stderr,
+ )
+ print(f" def: {path}:{lineno}: {stmt};", file=sys.stderr)
+ print(f" norm: {norm}", file=sys.stderr)
+ print(f" family: {fam} is extern-covered in {headers}",
file=sys.stderr)
+ print(
+ " reason: every extern-declared specialization of this family is "
+ "instantiated once, but this one re-instantiates in every "
+ "including TU -- the one pairing mistake that is silent: it "
+ "compiles, links, and only slows the build down",
+ file=sys.stderr,
+ )
+ print(
+ " fix: add 'extern template ...;' next to the family's other "
+ "externs in the header above, spelling the template arguments the "
+ "same way as the definition, or list the definition in ALLOW with "
+ "a comment",
+ file=sys.stderr,
+ )
+ return failures
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--list",
+ action="store_true",
+ help="dump every declaration and definition with its normalized form",
+ )
+ args = parser.parse_args()
+
+ root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ os.chdir(root)
+ externs, defs = collect()
+
+ if args.list:
+ for path, lineno, stmt, norm in externs:
+ print(f"decl {path}:{lineno}\n {norm}")
+ for path, lineno, stmt, norm in defs:
+ print(f"def {path}:{lineno}\n {norm}")
+ return 0
+
+ failures = enforce(externs, defs)
+ if failures:
+ print(f"\n{failures} extern/instantiation pairing violation(s)",
file=sys.stderr)
+ return 1
+ families = {family(n) for _, _, _, n in externs}
+ print(
+ f"extern/instantiation pairing: {len(externs)} declaration(s), "
+ f"{len(families)} covered family(s), all paired"
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/build-support/check-header-deps.py
b/build-support/check-header-deps.py
index 2e22721f390..5c5783991e6 100755
--- a/build-support/check-header-deps.py
+++ b/build-support/check-header-deps.py
@@ -24,12 +24,30 @@ every translation unit, so one stray include can multiply
rebuild cost by an ord
of magnitude and does so silently: the code still compiles, only the build
slows
down. This script turns that into a build error instead.
-Each rule names a hub header and a directory prefix it must not reach. Keeping
a
-hub out of a subsystem is what lets that subsystem's headers be edited cheaply.
+Four kinds of guard, all on the same text include graph (be/src + be/test, all
+ifdef branches -- not comparable to ninja-measured radii, each baseline
compares
+only with itself):
+
+ RULES a hub header must not reach a subsystem. Keeping a
+ hub out of a subsystem is what lets that subsystem's
+ headers be edited cheaply.
+ ANGLE_BANS a header must not include a named third-party header
+ whose template machinery was deliberately moved out
+ of line.
+ FORWARD/REVERSE budgets the two-axis safety net for edges no rule names:
+ light hubs must stay light (closure budget, zero
+ slack), heavy payloads must not spread (reach
+ baseline +10%).
+ PCH whitelist pch/pch.h's quoted project includes are pinned
+ exactly; the PCH is the single most leveraged
+ regression surface in the repo.
Usage:
- build-support/check-header-deps.py # enforce the rules
- build-support/check-header-deps.py --report # rank headers by rebuild
cost
+ build-support/check-header-deps.py # enforce everything
+ build-support/check-header-deps.py --report # rank headers by
rebuild cost
+ build-support/check-header-deps.py --budget # budgets only
(rebaselining)
+ build-support/check-header-deps.py --closure HEADER # list a hub's
closure, with chains
+ build-support/check-header-deps.py --reach HEADER # rank an includer's
edge weights
"""
import argparse
@@ -288,19 +306,213 @@ RULES = [
"rec_cte_shared_state.cpp; the brpc client stack must not ride a "
"SharedState header",
),
+ (
+ "exprs/function/function.h",
+ "storage/",
+ {
+ # Plain result struct (<cstdint> only); the sanctioned carrier left
+ # behind when the zonemap machinery edge was cut.
+ "storage/index/zone_map/zonemap_filter_result.h",
+ # Plain id struct, methods defined in rowset_id.cpp; rides in via
+ # core/column/column.h.
+ "storage/rowset_id.h",
+ },
+ "function.h is the base header of every scalar function; cutting its "
+ "three storage edges (expr_zonemap_filter, inverted_index_iterator, "
+ "inverted_index_parser) removed 122,885 preprocessed lines from the "
+ "exprs TUs, and one edge flowing back re-couples the whole expression "
+ "layer to the storage stack",
+ ),
+ (
+ "core/field.h",
+ "util/json/path_in_data.h",
+ set(),
+ "a single using-alias here used to drag path_in_data.h and with it "
+ "gen_cpp/segment_v2.pb.h (11.6k lines) into every TU that sees a "
+ "Field; the alias users include path_in_data.h themselves now",
+ ),
+ (
+ "core/data_type/primitive_type.h",
+ "util/json/path_in_data.h",
+ set(),
+ "primitive_type.h is included by ~44 project headers transitively; "
+ "the path_in_data edge would put gen_cpp/segment_v2.pb.h behind the "
+ "most basic type-enum header in the backend",
+ ),
+ (
+ "core/value/variant/variant_field.h",
+ "util/json/path_in_data.h",
+ set(),
+ "the field.h cut moved the variant alias down here, so this header "
+ "is where the path_in_data edge would most naturally regrow; variant "
+ "consumers that need PathInData include it directly",
+ ),
+ (
+ "core/types.h",
+ "storage/olap_common.h",
+ set(),
+ "the int128/uint128 typedefs moved to core/extended_types.h precisely "
+ "so core/types.h stops paying for the storage domain; the binary_cast "
+ "include chain used to put all of olap_common behind two lines of "
+ "typedef",
+ ),
+ (
+ "pch/pch.h",
+ "storage/olap_common.h",
+ set(),
+ "every header on the PCH is on the everything-rebuilds line: touch it "
+ "and the whole backend plus the precompiled header itself rebuild; "
+ "olap_common.h alone used to carry 22 project headers onto that line",
+ ),
+]
+
+# Third-party bans, (header, banned include, why). The include graph above only
+# follows quoted project includes and <gen_cpp/...>, so forbidden *third-party*
+# edges get their own single-file check: the named header must not include the
+# banned spelling (a trailing '/' bans the whole directory, otherwise the match
+# is exact; angle and quoted forms are both caught). These are headers whose
+# formatting/queueing bodies were deliberately moved out of line -- the ban
keeps
+# the heavy third-party template machinery from re-entering every includer.
+ANGLE_BANS = [
+ (
+ "core/uint24.h",
+ "fmt/",
+ "to_string/to_buffer bodies live in uint24.cpp precisely so the "
+ "FMT_COMPILE formatter templates (53.5 CPU s over ~1150 TUs for the "
+ "date format alone) are instantiated once instead of in every
includer",
+ ),
+ (
+ "core/value/large_int_value.h",
+ "fmt/",
+ "the fmt formatting bodies moved to large_int_value.cpp under the same
"
+ "contract as core/uint24.h: the formatter templates must be "
+ "instantiated once, not in every includer",
+ ),
+ (
+ "exec/pipeline/dependency.h",
+ "concurrentqueue.h",
+ "was a dead 152 KB third-party include here; the moodycamel users "
+ "(local_exchanger.h, scanner_context.h, async_result_writer.h) "
+ "include it themselves",
+ ),
+ (
+ "util/pretty_printer.h",
+ "boost/",
+ "one boost::algorithm::join dragged ~60k preprocessed lines into every
"
+ "TU that sees runtime_profile.h; the join is a plain loop in "
+ "pretty_printer.cpp now",
+ ),
+ # <ranges> is both a heavy header (object_pool.h alone reaches most of the
+ # backend) and a build breaker: libc++'s <ranges> rejects the
+ # -fno-access-control flag that doris_be_test builds with, so a <ranges>
+ # include in a src header can take the whole UT build down (#66615).
+ (
+ "common/object_pool.h",
+ "ranges",
+ "widely-included header; <ranges> is heavy for every includer and "
+ "breaks the -fno-access-control UT build on libc++",
+ ),
+ (
+ "exprs/lambda_function/lambda_execution_context.h",
+ "ranges",
+ "rides into every lambda-capable expression TU; <ranges> is heavy and "
+ "breaks the -fno-access-control UT build on libc++",
+ ),
+ (
+ "format/table/iceberg_reader_mixin.h",
+ "ranges",
+ "rides the table-reader stack; <ranges> is heavy and breaks the "
+ "-fno-access-control UT build on libc++",
+ ),
+ (
+ "storage/index/inverted/query_v2/collect/top_k_collector.h",
+ "ranges",
+ "rides the inverted-index query stack; <ranges> is heavy and breaks "
+ "the -fno-access-control UT build on libc++",
+ ),
+ (
+ "storage/index/inverted/query_v2/composite_reader.h",
+ "ranges",
+ "rides the inverted-index query stack; <ranges> is heavy and breaks "
+ "the -fno-access-control UT build on libc++",
+ ),
+ (
+ "storage/index/inverted/query_v2/wand/block_wand.h",
+ "ranges",
+ "rides the inverted-index query stack; <ranges> is heavy and breaks "
+ "the -fno-access-control UT build on libc++",
+ ),
]
-# Not expressible as RULES entries (the scanner only follows quoted project
-# includes and <gen_cpp/...>): core/uint24.h and core/value/large_int_value.h
-# must not regain <fmt/compile.h> / <fmt/format.h>. Their to_string/to_buffer
-# bodies live in the matching .cpp files precisely so the FMT_COMPILE formatter
-# templates (53.5 CPU s over ~1150 TUs for the uint24 date format alone) are
-# instantiated once instead of in every includer.
+# Budgets: the edge rules and bans above pin the edges someone has already
+# thought about; the two budget axes below catch the ones nobody thought about.
+# Both read the same text include graph (be/src + be/test, all ifdef branches);
+# neither is comparable to ninja-measured rebuild radii -- each compares only
+# against its own baseline.
#
-# Likewise <concurrentqueue.h> must not return to exec/pipeline/dependency.h:
-# it was a dead 152 KB third-party include there; the moodycamel users
-# (local_exchanger.h, scanner_context.h, async_result_writer.h) include it
-# themselves.
+# Forward axis, "this header must stay light": the number of project headers
+# transitively reachable from the hub (the hub itself excluded). Slack is ZERO
+# by design -- growing a hub's closure must be a visible, deliberate act, so
the
+# legal way over the budget is to bump the number here in the same PR and say
+# why in the commit message.
+# Baselines: master 9a48f8120c0, 2026-08-18.
+FORWARD_CLOSURE_BUDGETS = {
+ "common/logging.h": 0, # a leaf on purpose: logging must not drag project
headers
+ "util/uid_util.h": 1,
+ "common/status.h": 6,
+ "runtime/exec_env.h": 10,
+ "core/pod_array.h": 18,
+ "util/pretty_printer.h": 31,
+ "storage/olap_common.h": 32,
+ "core/types.h": 35,
+ "runtime/runtime_state.h": 43,
+ "core/data_type/primitive_type.h": 44,
+ "runtime/thread_context.h": 55,
+ "core/field.h": 60,
+ "core/column/column.h": 66,
+ "exprs/function/function.h": 106,
+ "exec/pipeline/dependency.h": 357,
+ "pch/pch.h": 8,
+}
+
+# Reverse axis, "this heavy payload must not spread": how many TUs (src + test)
+# transitively include the header. New TUs legitimately reference these, so the
+# limit is baseline +10% (rounded up); rebaseline when the audit cadence
+# re-measures, or in the same PR with justification when a real growth burst is
+# intended.
+# Baselines: master 9a48f8120c0, 2026-08-18.
+REVERSE_REACH_BASELINES = {
+ "gen_cpp/segment_v2.pb.h": 1234,
+ "gen_cpp/PaloInternalService_types.h": 1133,
+ "util/threadpool.h": 988,
+ # parsed_page.h holds RleDecoder<bool> by value and constructs it in the
+ # header, so the whole storage read stack sees the RLE machinery; a live
+ # dependency, budgeted as-is (trimming it is a refactor, not a gate).
+ "util/rle_encoding.h": 823,
+ "gen_cpp/internal_service.pb.h": 741,
+ "gen_cpp/FrontendService_types.h": 576,
+ "storage/options.h": 464,
+ "gen_cpp/cloud.pb.h": 342,
+ "gen_cpp/BackendService_types.h": 222,
+ "gen_cpp/data.pb.h": 220,
+ "io/fs/s3_file_system.h": 109, # the AWS SDK surface
+ "util/brpc_closure.h": 61,
+ "runtime/workload_group/workload_group.h": 42, # thrift type universe
carrier
+}
+REVERSE_SLACK = 0.10
+
+# The PCH is the single most leveraged file in the repo: every header on it is
+# on the everything-rebuilds line (touch one and the whole backend plus the
+# precompiled header itself rebuild). Its quoted project includes are therefore
+# pinned exactly; the transitive closure is capped by the pch/pch.h entry in
+# FORWARD_CLOSURE_BUDGETS. Generated <gen_cpp/...> includes are the PCH's whole
+# point and stay out of this lock.
+PCH_HEADER = "pch/pch.h"
+PCH_QUOTED_WHITELIST = {
+ "common/config.h",
+ "common/status.h",
+ "common/version_internal.h",
+}
# Forward-declaration headers are the sanctioned way through a barrier: they
carry
# declarations only, so they cost nothing to include.
@@ -398,6 +610,213 @@ def enforce(includes):
return failures
+def forward_closure(hub, includes):
+ """Project headers transitively reachable from `hub`, hub excluded."""
+ chains = reachable(hub, includes)
+ return sorted(
+ h for h in chains if h != hub and resolve(h, includes) is not None
+ )
+
+
+ANY_INCLUDE = re.compile(r'^\s*#\s*include\s+[<"]([^>"]+)[>"]')
+
+
+def enforce_angle_bans():
+ failures = 0
+ for header, banned, why in ANGLE_BANS:
+ path = os.path.join(INCLUDE_ROOT, header)
+ if not os.path.exists(path):
+ print(f"error: angle ban names a missing header: {header}",
file=sys.stderr)
+ failures += 1
+ continue
+ with open(path, encoding="utf-8", errors="ignore") as handle:
+ spellings = [m.group(1) for m in map(ANY_INCLUDE.match, handle) if
m]
+ hits = [
+ s
+ for s in spellings
+ if s == banned or (banned.endswith("/") and s.startswith(banned))
+ ]
+ for hit in hits:
+ failures += 1
+ print(f"error: {header} must not include <{hit}>", file=sys.stderr)
+ print(f" reason: {why}", file=sys.stderr)
+ print(
+ " fix: move the code that needs it into the matching .cpp "
+ "(that is where the previous cut put it), or take this ban out
"
+ "of ANGLE_BANS in the same PR and justify it in the commit "
+ "message",
+ file=sys.stderr,
+ )
+ return failures
+
+
+def enforce_budgets(includes):
+ failures = 0
+ for hub, budget in FORWARD_CLOSURE_BUDGETS.items():
+ if resolve(hub, includes) is None:
+ print(
+ f"error: forward budget names a missing header: {hub} "
+ "(renamed or moved? update FORWARD_CLOSURE_BUDGETS)",
+ file=sys.stderr,
+ )
+ failures += 1
+ continue
+ actual = len(forward_closure(hub, includes))
+ if actual > budget:
+ failures += 1
+ print(
+ f"error: {hub} include closure grew to {actual} project "
+ f"headers (budget {budget})",
+ file=sys.stderr,
+ )
+ print(
+ " reason: everything this hub includes is reparsed by every "
+ "TU that includes the hub, so closure growth is a rebuild-cost
"
+ "multiplier nobody sees in a diff; the budget makes it
visible",
+ file=sys.stderr,
+ )
+ print(
+ f" list: build-support/check-header-deps.py --closure
{hub}",
+ file=sys.stderr,
+ )
+ print(
+ " fix: cut the new edge (forward-declare the type, or "
+ "route it through a *_fwd.h), or bump the budget in "
+ "FORWARD_CLOSURE_BUDGETS in the same PR and justify it in the "
+ "commit message",
+ file=sys.stderr,
+ )
+ counts = translation_units_affected(includes)
+ for header, baseline in REVERSE_REACH_BASELINES.items():
+ limit = -(-baseline * (100 + int(REVERSE_SLACK * 100)) // 100)
+ actual = counts.get(header, 0)
+ is_project = resolve(header, includes) is not None
+ if not is_project and not header.startswith("gen_cpp/"):
+ print(
+ f"error: reverse budget names a missing header: {header} "
+ "(renamed or moved? update REVERSE_REACH_BASELINES)",
+ file=sys.stderr,
+ )
+ failures += 1
+ continue
+ if actual == 0:
+ # A watched header nobody includes any more is either renamed
+ # (the budget is watching nothing) or genuinely dead -- both mean
+ # the table must be updated, loudly.
+ print(
+ f"error: reverse budget sentinel is no longer referenced: "
+ f"{header} (renamed, or truly unused? update "
+ "REVERSE_REACH_BASELINES)",
+ file=sys.stderr,
+ )
+ failures += 1
+ continue
+ if actual > limit:
+ failures += 1
+ print(
+ f"error: {header} now reaches {actual} TUs "
+ f"(baseline {baseline} +{int(REVERSE_SLACK * 100)}% =
{limit})",
+ file=sys.stderr,
+ )
+ print(
+ " reason: this is a heavy payload (generated code / SDK "
+ "surface / template machinery); some header on a popular path "
+ "gained an include of it, taxing every TU downstream",
+ file=sys.stderr,
+ )
+ print(
+ f" list: build-support/check-header-deps.py --reach
{header}",
+ file=sys.stderr,
+ )
+ print(
+ " fix: cut the spreading edge (forward-declare, or move "
+ "the include into the .cpp), or rebaseline in "
+ "REVERSE_REACH_BASELINES in the same PR and justify it in the "
+ "commit message",
+ file=sys.stderr,
+ )
+ return failures
+
+
+def enforce_pch(includes):
+ path = resolve(PCH_HEADER, includes)
+ if path is None:
+ print(f"error: {PCH_HEADER} not found", file=sys.stderr)
+ return 1
+ with open(path, encoding="utf-8", errors="ignore") as handle:
+ quoted = {m.group(1) for m in map(INCLUDE.match, handle) if m}
+ extra = sorted(quoted - PCH_QUOTED_WHITELIST)
+ missing = sorted(PCH_QUOTED_WHITELIST - quoted)
+ if not extra and not missing:
+ return 0
+ print("error: pch/pch.h quoted includes diverged from the whitelist",
file=sys.stderr)
+ for header in extra:
+ print(f" added: \"{header}\"", file=sys.stderr)
+ for header in missing:
+ print(f" removed: \"{header}\"", file=sys.stderr)
+ print(
+ " reason: every header on the PCH is on the everything-rebuilds "
+ "line -- touch it and the whole backend plus the precompiled header "
+ "itself rebuild; adding one is the single most leveraged regression "
+ "in the repo",
+ file=sys.stderr,
+ )
+ print(
+ " fix: include the header in the TUs that need it instead, or "
+ "change PCH_QUOTED_WHITELIST in the same PR and justify it in the "
+ "commit message",
+ file=sys.stderr,
+ )
+ return 1
+
+
+def closure_listing(hub, includes):
+ if resolve(hub, includes) is None:
+ print(f"error: no such project header: {hub}", file=sys.stderr)
+ return 1
+ chains = reachable(hub, includes)
+ members = forward_closure(hub, includes)
+ print(f"{hub}: {len(members)} project header(s) in the include closure")
+ for member in members:
+ print(" " + " -> ".join(chains[member]))
+ return 0
+
+
+def reach_listing(header, includes):
+ """Direct includers of `header` ranked by how many TUs each edge
carries."""
+ users = collections.defaultdict(set)
+ for path, headers in includes.items():
+ for h in headers:
+ users[h].add(path)
+ if not users.get(header):
+ print(f"error: nothing includes {header}", file=sys.stderr)
+ return 1
+
+ def tus(banned_edge=None):
+ seen, frontier = set(), [header]
+ while frontier:
+ current = frontier.pop()
+ for user in users.get(current, ()):
+ if banned_edge and (user, current) == banned_edge:
+ continue
+ if user in seen:
+ continue
+ seen.add(user)
+ if user.startswith(INCLUDE_ROOT + "/"):
+ frontier.append(user[len(INCLUDE_ROOT) + 1:])
+ return sum(1 for f in seen if f.endswith((".cpp", ".cc")))
+
+ total = tus()
+ print(f"{header}: reaches {total} TU(s); direct includers by edge weight")
+ ranked = sorted(
+ ((total - tus((user, header)), user) for user in users[header]),
+ reverse=True,
+ )
+ for marginal, user in ranked:
+ print(f" {marginal:>5} via this edge alone {user}")
+ return 0
+
+
def report(includes):
counts = translation_units_affected(includes)
ranked = sorted(counts.items(), key=lambda kv: kv[1], reverse=True)[:30]
@@ -413,6 +832,21 @@ def main():
action="store_true",
help="rank headers by how many translation units they force a rebuild
of",
)
+ parser.add_argument(
+ "--budget",
+ action="store_true",
+ help="run only the closure/reach budget checks (used when
rebaselining)",
+ )
+ parser.add_argument(
+ "--closure",
+ metavar="HEADER",
+ help="list the project headers in HEADER's include closure, with
chains",
+ )
+ parser.add_argument(
+ "--reach",
+ metavar="HEADER",
+ help="list HEADER's direct includers ranked by how many TUs each edge
carries",
+ )
args = parser.parse_args()
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -422,12 +856,35 @@ def main():
if args.report:
report(includes)
return 0
+ if args.closure:
+ return closure_listing(args.closure, includes)
+ if args.reach:
+ return reach_listing(args.reach, includes)
- failures = enforce(includes)
+ if args.budget:
+ failures = enforce_budgets(includes)
+ else:
+ failures = (
+ enforce(includes)
+ + enforce_angle_bans()
+ + enforce_budgets(includes)
+ + enforce_pch(includes)
+ )
if failures:
- print(f"\n{failures} header layering violation(s)", file=sys.stderr)
+ print(f"\n{failures} header hygiene violation(s)", file=sys.stderr)
return 1
- print(f"header layering: {len(RULES)} rule(s) satisfied")
+ if args.budget:
+ print(
+ f"header budgets: {len(FORWARD_CLOSURE_BUDGETS)} forward + "
+ f"{len(REVERSE_REACH_BASELINES)} reverse within budget"
+ )
+ else:
+ print(
+ f"header hygiene: {len(RULES)} layering rule(s), "
+ f"{len(ANGLE_BANS)} third-party ban(s), "
+ f"{len(FORWARD_CLOSURE_BUDGETS)}+{len(REVERSE_REACH_BASELINES)} "
+ "budget(s), pch whitelist: all satisfied"
+ )
return 0
diff --git a/build-support/check-unity-skip-coverage.py
b/build-support/check-unity-skip-coverage.py
new file mode 100755
index 00000000000..c72c18bd63e
--- /dev/null
+++ b/build-support/check-unity-skip-coverage.py
@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Unity-skip coverage for test-included source files.
+
+A few tests `#include` a be/src .cpp directly (to reach file-static helpers or
+to instantiate a template with test types). That only links because the source
+file is opted out of unity batching: if it stays in a batch, the batch object
is
+pulled into the doris_be_test link for a *sibling's* symbol and its copy of the
+file's definitions collides with the test's inlined copy -- duplicate strong
+symbols, reported at the end of the BE UT build, an hour after the mistake.
+This script reports the mistake at configure time instead, with the exact skip
+entry to add.
+
+It is the mirror of the fail-loud check inside doris_skip_unity_inclusion()
+(be/CMakeLists.txt): that one catches skip entries pointing at files that no
+longer exist; this one catches files that need an entry and do not have one.
+
+A source file counts as covered when its owning CMakeLists (nearest ancestor
+with a CMakeLists.txt, else any CMakeLists naming it) does one of:
+
+ literal entry the file's path appears in a list fed to
+ doris_skip_unity_inclusion(), via set()/list(APPEND)/direct
+ arguments;
+ whole-target the skip list is initialized from the target's own source
+ opt-out list (`set(<VAR> ${SRC_FILES})`), minus any
+ `list(FILTER <VAR> EXCLUDE REGEX ...)` the file matches;
+ no unity the owning CMakeLists never sets UNITY_BUILD, so there are
+ no batches to collide with (archive link rules protect
+ plain object files).
+
+Any other CMake idiom is unknown to this script on purpose: it fails loudly so
+the idiom is either replaced with a literal entry or taught here, instead of
+being silently guessed wrong.
+
+Usage:
+ build-support/check-unity-skip-coverage.py
+"""
+
+import os
+import re
+import sys
+
+TEST_ROOT = "be/test"
+SRC_ROOT = "be/src"
+
+TEST_INCLUDE = re.compile(r'^\s*#\s*include\s+"([^"]+\.(?:cpp|cc))"')
+SKIP_CALL = re.compile(r"doris_skip_unity_inclusion\s*\(([^)]*)\)", re.S)
+SET_BLOCK = re.compile(r"\bset\s*\(\s*(\w+)([^)]*)\)", re.S)
+APPEND_BLOCK = re.compile(r"\blist\s*\(\s*APPEND\s+(\w+)([^)]*)\)", re.S)
+FILTER_BLOCK = re.compile(
+ r'\blist\s*\(\s*FILTER\s+(\w+)\s+EXCLUDE\s+REGEX\s+"([^"]*)"\s*\)'
+)
+VAR_REF = re.compile(r"\$\{(\w+)\}")
+
+
+def strip_comments(text):
+ return "\n".join(line.split("#", 1)[0] for line in text.splitlines())
+
+
+def cpp_tokens(blob):
+ """Path-shaped tokens in a CMake argument blob, ${...} prefixes
stripped."""
+ tokens = []
+ for token in blob.split():
+ if not token.endswith((".cpp", ".cc")):
+ continue
+ tokens.append(re.sub(r"\$\{\w+\}", "", token).lstrip("/"))
+ return tokens
+
+
+class CMakeFile:
+ def __init__(self, path):
+ self.path = path
+ self.directory = os.path.dirname(path)
+ with open(path, encoding="utf-8", errors="ignore") as handle:
+ text = strip_comments(handle.read())
+ self.sets_unity = "UNITY_BUILD" in text
+
+ sets = {}
+ for name, blob in SET_BLOCK.findall(text):
+ sets.setdefault(name, []).append(blob)
+ for name, blob in APPEND_BLOCK.findall(text):
+ sets.setdefault(name, []).append(blob)
+ filters = {}
+ for name, regex in FILTER_BLOCK.findall(text):
+ filters.setdefault(name, []).append(regex)
+
+ # Literal skip entries and whether the skip list is seeded from the
+ # target's whole source list.
+ self.entries = []
+ self.whole_target = False
+ self.whole_target_excludes = []
+ for blob in SKIP_CALL.findall(text):
+ self.entries.extend(cpp_tokens(blob))
+ for var in VAR_REF.findall(blob):
+ for var_blob in sets.get(var, ()):
+ self.entries.extend(cpp_tokens(var_blob))
+ if re.search(r"\$\{\w*SRC_FILES\}", var_blob):
+ self.whole_target = True
+ self.whole_target_excludes.extend(filters.get(var, ()))
+
+ def resolved_entries(self):
+ """Skip entries as repo-relative paths (handles ../ hops)."""
+ for entry in self.entries:
+ yield os.path.normpath(os.path.join(self.directory, entry))
+
+ def covers(self, repo_path):
+ """Does this CMakeLists skip-list `repo_path` (repo-relative .cpp)?"""
+ for resolved in self.resolved_entries():
+ if resolved == repo_path or resolved.endswith("/" + repo_path):
+ return True
+ if self.whole_target and repo_path.startswith(self.directory + "/"):
+ # cmake applies FILTER regexes to absolute paths; approximate with
+ # the repo-relative one, which shares every path component that the
+ # tree's regexes (".*/format_v2/.*") anchor on.
+ return not any(
+ re.search(regex, repo_path)
+ for regex in self.whole_target_excludes
+ )
+ return False
+
+
+def owning_cmakelists(repo_path, cmake_files):
+ directory = os.path.dirname(repo_path)
+ while directory.startswith(SRC_ROOT):
+ candidate = os.path.join(directory, "CMakeLists.txt")
+ if candidate in cmake_files:
+ return cmake_files[candidate]
+ directory = os.path.dirname(directory)
+ return None
+
+
+def test_included_sources():
+ """(test file, line, be/src-relative include) for every src .cpp a test
+ includes."""
+ found = []
+ for directory, _, names in os.walk(TEST_ROOT):
+ for name in names:
+ if not name.endswith((".cpp", ".cc", ".h", ".hpp")):
+ continue
+ path = os.path.join(directory, name)
+ with open(path, encoding="utf-8", errors="ignore") as handle:
+ for lineno, line in enumerate(handle, 1):
+ match = TEST_INCLUDE.match(line)
+ if not match:
+ continue
+ include = match.group(1)
+ if os.path.exists(os.path.join(SRC_ROOT, include)):
+ found.append((path, lineno, include))
+ return found
+
+
+def main():
+ root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ os.chdir(root)
+
+ cmake_files = {}
+ for directory, _, names in os.walk(SRC_ROOT):
+ if "CMakeLists.txt" in names:
+ path = os.path.join(directory, "CMakeLists.txt")
+ cmake_files[path] = CMakeFile(path)
+
+ failures = 0
+ checked = 0
+ for test_path, lineno, include in test_included_sources():
+ checked += 1
+ repo_path = os.path.join(SRC_ROOT, include)
+ owner = owning_cmakelists(repo_path, cmake_files)
+ if owner is not None and not owner.sets_unity:
+ continue # no unity batches in this target, nothing to collide
with
+ if owner is not None and owner.covers(repo_path):
+ continue
+ if any(f.covers(repo_path) for f in cmake_files.values()):
+ continue # cross-directory entry (e.g. appended into a sibling's
list)
+ failures += 1
+ print(
+ f"error: {test_path}:{lineno} includes {repo_path}, which is not "
+ "opted out of unity batching",
+ file=sys.stderr,
+ )
+ print(
+ " reason: the unity batch object containing this file gets pulled
"
+ "into the doris_be_test link for a sibling's symbol, and its "
+ "definitions collide with the test's inlined copy -- duplicate "
+ "strong symbols at the end of the BE UT build",
+ file=sys.stderr,
+ )
+ if owner is not None:
+ entry = os.path.relpath(repo_path, owner.directory)
+ print(
+ f" fix: add ${{CMAKE_CURRENT_SOURCE_DIR}}/{entry} to the "
+ f"doris_skip_unity_inclusion list in {owner.path}",
+ file=sys.stderr,
+ )
+ else:
+ print(
+ " fix: add the file to the doris_skip_unity_inclusion "
+ "list of the CMakeLists whose source glob compiles it; for "
+ "the cross-directory entry form, see the adbc_reader.cpp "
+ "entry in be/src/format/CMakeLists.txt",
+ file=sys.stderr,
+ )
+ if failures:
+ print(f"\n{failures} unity-skip coverage violation(s)",
file=sys.stderr)
+ return 1
+ print(f"unity-skip coverage: {checked} test-included source file(s) all
covered")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/build-support/tests/test-build-hygiene-entry.sh
b/build-support/tests/test-build-hygiene-entry.sh
new file mode 100755
index 00000000000..6bfc157a635
--- /dev/null
+++ b/build-support/tests/test-build-hygiene-entry.sh
@@ -0,0 +1,101 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Self-test for the check-build-hygiene.sh unified entry:
+# - clean tree exits 0 with the summary line
+# - with violations in TWO different checks, BOTH are reported (the entry
+# keeps running after a failing check) and the exit code is 1
+# - a missing python3 is a loud error (exit 2) naming the OFF switch, never
+# a silent skip
+# - env.sh's PYTHON (python2 on the build-env image) is not consulted
+#
+# Injects into working-tree files, restores via EXIT trap. Do not run
+# concurrently with a build/configure of the same tree.
+#
+# Usage: bash build-support/tests/test-build-hygiene-entry.sh
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+ENTRY="${ROOT}/build-support/check-build-hygiene.sh"
+
+UINT24="be/src/core/uint24.h"
+COLVEC="be/src/core/column/column_vector.h"
+BACKUP="$(mktemp -d)"
+cp "${ROOT}/${UINT24}" "${BACKUP}/uint24.h"
+cp "${ROOT}/${COLVEC}" "${BACKUP}/column_vector.h"
+restore() {
+ cp "${BACKUP}/uint24.h" "${ROOT}/${UINT24}"
+ cp "${BACKUP}/column_vector.h" "${ROOT}/${COLVEC}"
+ rm -rf "${BACKUP}"
+}
+trap restore EXIT
+
+FAILED=0
+fail() { echo "FAIL: $1"; FAILED=1; }
+
+# ---- clean tree: exit 0, summary line present ----
+OUT="$(bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "clean tree expected exit 0, got ${EC}:"$'\n'"${OUT}"
+echo "${OUT}" | grep -q "build hygiene: all checks passed" \
+ || fail "clean run lacks the all-passed summary"
+
+# ---- two violations in two different checks: both reported, exit 1 ----
+printf '#include <fmt/format.h>\n' >> "${ROOT}/${UINT24}"
+printf 'extern template class ColumnVector<TYPE_FAKE_ENTRY_TEST>;\n' >>
"${ROOT}/${COLVEC}"
+OUT="$(bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "violations expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "core/uint24.h must not include <fmt/format.h>" \
+ || fail "header-deps violation missing from aggregated output"
+echo "${OUT}" | grep -q "TYPE_FAKE_ENTRY_TEST" \
+ || fail "pairing violation missing from aggregated output (entry must run
every check)"
+echo "${OUT}" | grep -q "2 check(s) failed" \
+ || fail "aggregated failure count missing or wrong"
+restore
+trap - EXIT
+BACKUP="$(mktemp -d)" # restore() removed it; re-arm for the paths below
+cp "${ROOT}/${UINT24}" "${BACKUP}/uint24.h"
+cp "${ROOT}/${COLVEC}" "${BACKUP}/column_vector.h"
+trap restore EXIT
+
+# ---- unusable BUILD_HYGIENE_PYTHON: loud exit 2 with the OFF-switch hint ----
+OUT="$(BUILD_HYGIENE_PYTHON=/nonexistent/python3 bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 2 ] || fail "missing python expected exit 2, got ${EC}"
+echo "${OUT}" | grep -q "not found" || fail "missing-python error not reported"
+echo "${OUT}" | grep -q "ENABLE_BUILD_HYGIENE=OFF" \
+ || fail "missing-python error lacks the OFF-switch hint"
+
+# ---- env.sh's PYTHON is not consulted ----
+# env.sh exports PYTHON=${DORIS_BUILD_PYTHON_VERSION:-python}, which is python2
+# on the build-env image; honouring it made every gate die with a SyntaxError
on
+# its first f-string. The gates must resolve their own Python 3 regardless.
+OUT="$(PYTHON=/nonexistent/python2 bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] \
+ || fail "the gates must resolve their own python3, not env.sh's PYTHON
(exit ${EC}):"$'\n'"${OUT}"
+
+# ---- green again ----
+OUT="$(bash "${ENTRY}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "restored tree expected exit 0, got
${EC}:"$'\n'"${OUT}"
+
+if [ "${FAILED}" -eq 0 ]; then
+ echo "PASS: entry aggregates all checks, resolves its own python3, fails
loud without one, and is green when clean."
+ exit 0
+fi
+exit 1
diff --git a/build-support/tests/test-build-hygiene-extern-pairing.sh
b/build-support/tests/test-build-hygiene-extern-pairing.sh
new file mode 100755
index 00000000000..c9ff755e8e7
--- /dev/null
+++ b/build-support/tests/test-build-hygiene-extern-pairing.sh
@@ -0,0 +1,91 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Self-test for check-extern-template-pairing.py: both pairing directions must
+# fire on injection and the clean tree must stay green (which also proves the
+# normalizer -- namespace dropping, alias table, comment stripping -- because
+# the real tree pairs ColumnStr<UInt32> with ColumnStr<uint32_t> and the wide
+# to_string declarations with their integer<...> definitions).
+#
+# Injects into working-tree files, restores via EXIT trap. Do not run
+# concurrently with a build/configure of the same tree.
+#
+# Usage: bash build-support/tests/test-build-hygiene-extern-pairing.sh
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+GATE="${ROOT}/build-support/check-extern-template-pairing.py"
+# Not ${PYTHON}: env.sh exports that as the build's interpreter (python2 on
+# the build-env image). Same selector the gates themselves use.
+PYTHON="${BUILD_HYGIENE_PYTHON:-python3}"
+
+HEADER="be/src/core/column/column_vector.h"
+BACKUP="$(mktemp -d)"
+cp "${ROOT}/${HEADER}" "${BACKUP}/column_vector.h"
+restore() { cp "${BACKUP}/column_vector.h" "${ROOT}/${HEADER}"; rm -rf
"${BACKUP}"; }
+trap restore EXIT
+
+FAILED=0
+fail() { echo "FAIL: $1"; FAILED=1; }
+
+# ---- baseline: clean tree green (normalizer handles the real alias pairs)
----
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "clean tree expected green, got ${EC}:"$'\n'"${OUT}"
+echo "${OUT}" | grep -q "all paired" || fail "clean run lacks the summary line"
+
+# ---- RED (the silent case): drop one extern, its definition loses its pair
----
+# Commenting the declaration out is exactly the "header forgot the extern"
+# regression the reverse direction exists for.
+sed -i.bak \
+ 's|^extern template class ColumnVector<TYPE_BOOLEAN>;|// injected-out:
extern template class ColumnVector<TYPE_BOOLEAN>;|' \
+ "${ROOT}/${HEADER}" && rm -f "${ROOT}/${HEADER}.bak"
+grep -q "injected-out" "${ROOT}/${HEADER}" || fail "reverse injection did not
apply"
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "reverse injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "explicit instantiation lacks the matching 'extern
template'" \
+ || fail "reverse violation not reported"
+echo "${OUT}" | grep -q "ColumnVector<TYPE_BOOLEAN>" \
+ || fail "reverse violation does not name the specialization"
+echo "${OUT}" | grep -q "extern-covered in be/src/core/column/column_vector.h"
\
+ || fail "reverse violation does not point at the family's header"
+echo "${OUT}" | grep -q "fix:" || fail "reverse violation lacks a fix path"
+cp "${BACKUP}/column_vector.h" "${ROOT}/${HEADER}"
+
+# ---- RED: an extern declaration with no definition anywhere ----
+printf 'extern template class ColumnVector<TYPE_DATETIMEV2_FAKE_INJECTED>;\n' \
+ >> "${ROOT}/${HEADER}"
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "forward injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "pairs with no explicit instantiation definition" \
+ || fail "forward violation not reported"
+echo "${OUT}" | grep -q "TYPE_DATETIMEV2_FAKE_INJECTED" \
+ || fail "forward violation does not name the declaration"
+cp "${BACKUP}/column_vector.h" "${ROOT}/${HEADER}"
+
+# ---- GREEN again ----
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "restored tree expected green, got
${EC}:"$'\n'"${OUT}"
+
+if [ "${FAILED}" -eq 0 ]; then
+ echo "PASS: both pairing directions fire on injection and restore to
green."
+ exit 0
+fi
+exit 1
diff --git a/build-support/tests/test-build-hygiene-header-deps.sh
b/build-support/tests/test-build-hygiene-header-deps.sh
new file mode 100755
index 00000000000..7b9c3cfeed7
--- /dev/null
+++ b/build-support/tests/test-build-hygiene-header-deps.sh
@@ -0,0 +1,136 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Self-test for the check-header-deps.py gate families: layering rules,
+# third-party bans, forward/reverse budgets, pch whitelist.
+#
+# The gate runs against the real tree (its rule and budget tables name real
+# headers), so RED cases are proven by injecting one violation per family into
+# a working-tree file and GREEN by restoring it. Injected files are backed up
+# first and restored by an EXIT trap. Do not run concurrently with a
+# build/configure of the same tree.
+#
+# Usage: bash build-support/tests/test-build-hygiene-header-deps.sh
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+GATE="${ROOT}/build-support/check-header-deps.py"
+# Not ${PYTHON}: env.sh exports that as the build's interpreter (python2 on
+# the build-env image). Same selector the gates themselves use.
+PYTHON="${BUILD_HYGIENE_PYTHON:-python3}"
+
+BACKUP="$(mktemp -d)"
+TARGETS=(
+ "be/src/core/types.h"
+ "be/src/core/uint24.h"
+ "be/src/common/logging.h"
+ "be/src/util/pretty_printer.h"
+ "be/src/pch/pch.h"
+)
+for t in "${TARGETS[@]}"; do
+ mkdir -p "${BACKUP}/$(dirname "${t}")"
+ cp "${ROOT}/${t}" "${BACKUP}/${t}"
+done
+restore() {
+ for t in "${TARGETS[@]}"; do
+ cp "${BACKUP}/${t}" "${ROOT}/${t}"
+ done
+ rm -rf "${BACKUP}"
+}
+trap restore EXIT
+
+FAILED=0
+fail() { echo "FAIL: $1"; FAILED=1; }
+
+run_gate() { "${PYTHON}" "${GATE}" 2>&1; }
+
+# ---- baseline: the clean tree is green ----
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 0 ] || { fail "clean tree expected green, got
${EC}:"$'\n'"${OUT}"; }
+
+# ---- RED: layering rule (types.h must not reach olap_common.h) ----
+printf '#include "storage/olap_common.h"\n' >> "${ROOT}/be/src/core/types.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "rule injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "core/types.h must not reach storage/olap_common.h" \
+ || fail "rule violation not reported for types.h -> olap_common.h"
+echo "${OUT}" | grep -q "chain:" || fail "rule failure lacks the include chain"
+echo "${OUT}" | grep -q "fix:" || fail "rule failure lacks a fix path"
+cp "${BACKUP}/be/src/core/types.h" "${ROOT}/be/src/core/types.h"
+
+# ---- RED: third-party ban (uint24.h must not include fmt) ----
+printf '#include <fmt/format.h>\n' >> "${ROOT}/be/src/core/uint24.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "angle-ban injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "core/uint24.h must not include <fmt/format.h>" \
+ || fail "angle-ban violation not reported"
+cp "${BACKUP}/be/src/core/uint24.h" "${ROOT}/be/src/core/uint24.h"
+
+# ---- RED: forward closure budget (logging.h is budgeted at 0) ----
+printf '#include "util/uid_util.h"\n' >> "${ROOT}/be/src/common/logging.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "forward-budget injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "common/logging.h include closure grew" \
+ || fail "forward budget breach not reported"
+echo "${OUT}" | grep -q -- "--closure common/logging.h" \
+ || fail "forward budget failure lacks the --closure fix pointer"
+cp "${BACKUP}/be/src/common/logging.h" "${ROOT}/be/src/common/logging.h"
+
+# ---- RED: reverse reach budget (workload_group.h must not spread) ----
+printf '#include "runtime/workload_group/workload_group.h"\n' \
+ >> "${ROOT}/be/src/util/pretty_printer.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "reverse-budget injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "runtime/workload_group/workload_group.h now reaches" \
+ || fail "reverse budget breach not reported"
+echo "${OUT}" | grep -q -- "--reach runtime/workload_group/workload_group.h" \
+ || fail "reverse budget failure lacks the --reach fix pointer"
+cp "${BACKUP}/be/src/util/pretty_printer.h"
"${ROOT}/be/src/util/pretty_printer.h"
+
+# ---- RED: pch whitelist (quoted include added to pch.h) ----
+printf '#include "common/logging.h"\n' >> "${ROOT}/be/src/pch/pch.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "pch injection expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q "pch/pch.h quoted includes diverged from the
whitelist" \
+ || fail "pch whitelist divergence not reported"
+echo "${OUT}" | grep -q 'added: "common/logging.h"' \
+ || fail "pch failure does not name the added include"
+cp "${BACKUP}/be/src/pch/pch.h" "${ROOT}/be/src/pch/pch.h"
+
+# ---- diagnostics: --closure and --reach stay usable ----
+OUT="$("${PYTHON}" "${GATE}" --closure common/status.h 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "--closure expected exit 0, got ${EC}"
+echo "${OUT}" | grep -q "common/status.h: .* project header(s)" \
+ || fail "--closure output missing summary line"
+OUT="$("${PYTHON}" "${GATE}" --reach io/fs/s3_file_system.h 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "--reach expected exit 0, got ${EC}"
+echo "${OUT}" | grep -q "via this edge alone" \
+ || fail "--reach output missing edge ranking"
+
+# ---- GREEN again after all restores ----
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "restored tree expected green, got
${EC}:"$'\n'"${OUT}"
+
+if [ "${FAILED}" -eq 0 ]; then
+ echo "PASS: rules/bans/budgets/pch-lock all fire on injection, restore to
green, diagnostics work."
+ exit 0
+fi
+exit 1
diff --git a/build-support/tests/test-build-hygiene-unity-skip.sh
b/build-support/tests/test-build-hygiene-unity-skip.sh
new file mode 100755
index 00000000000..d49fdf92c1a
--- /dev/null
+++ b/build-support/tests/test-build-hygiene-unity-skip.sh
@@ -0,0 +1,92 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Self-test for check-unity-skip-coverage.py. Seeds throwaway files (a src
+# .cpp that is in no skip list, plus probe tests under be/test) to prove:
+# RED a test including an unlisted, unity-batched src .cpp is flagged,
+# with the exact skip entry to add
+# SILENT the literal-entry idiom (cross-directory APPEND included)
+# SILENT the whole-target opt-out idiom (set(<VAR> ${SRC_FILES}))
+# SILENT a commented-out include
+# and that the clean tree is green before and after.
+#
+# Creates files only (never edits tracked ones); removes them via EXIT trap.
+#
+# Usage: bash build-support/tests/test-build-hygiene-unity-skip.sh
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+GATE="${ROOT}/build-support/check-unity-skip-coverage.py"
+# Not ${PYTHON}: env.sh exports that as the build's interpreter (python2 on
+# the build-env image). Same selector the gates themselves use.
+PYTHON="${BUILD_HYGIENE_PYTHON:-python3}"
+
+PROBE_SRC="${ROOT}/be/src/util/tmp_hygiene_probe.cpp"
+PROBE_DIR="${ROOT}/be/test/tmp_hygiene_probe"
+cleanup() { rm -f "${PROBE_SRC}"; rm -rf "${PROBE_DIR}"; }
+trap cleanup EXIT
+
+FAILED=0
+fail() { echo "FAIL: $1"; FAILED=1; }
+
+# ---- baseline: clean tree green ----
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "clean tree expected green, got ${EC}:"$'\n'"${OUT}"
+
+# ---- fixtures ----
+echo "namespace doris { int tmp_hygiene_probe_symbol = 0; }" > "${PROBE_SRC}"
+mkdir -p "${PROBE_DIR}"
+# RED: util sets UNITY_BUILD and lists skips literally; the probe src is new,
+# so it is in no list.
+printf '#include "util/tmp_hygiene_probe.cpp"\n' >
"${PROBE_DIR}/red_probe_test.cpp"
+# SILENT: cross-directory literal entry (appended into format's skip list).
+printf '#include "format_v2/table/adbc_reader.cpp"\n' >
"${PROBE_DIR}/literal_probe_test.cpp"
+# SILENT: whole-target opt-out (format skip list is seeded from SRC_FILES).
+printf '#include "format/orc/orc_file_reader.cpp"\n' >
"${PROBE_DIR}/wholetarget_probe_test.cpp"
+# SILENT: commented include is not an include.
+printf '//#include "format_v2/parquet/reader/native/decoder.cpp"\n' >
"${PROBE_DIR}/comment_probe_test.cpp"
+
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "expected exit 1 with the red probe present, got
${EC}"
+N="$(echo "${OUT}" | grep -c '^error:')"
+[ "${N}" -eq 1 ] || fail "expected exactly 1 violation (silent probes must
stay silent), got ${N}:"$'\n'"${OUT}"
+echo "${OUT}" | grep -q "red_probe_test.cpp:1 includes
be/src/util/tmp_hygiene_probe.cpp" \
+ || fail "red probe not reported"
+echo "${OUT}" | grep -q 'add ${CMAKE_CURRENT_SOURCE_DIR}/tmp_hygiene_probe.cpp
to the doris_skip_unity_inclusion list in be/src/util/CMakeLists.txt' \
+ || fail "fix path does not name the exact entry and CMakeLists"
+echo "${OUT}" | grep -q "duplicate strong symbols" \
+ || fail "failure lacks the mechanism explanation"
+
+# ---- GREEN again once the red probe is gone ----
+rm -f "${PROBE_DIR}/red_probe_test.cpp"
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "expected green with only silent probes, got
${EC}:"$'\n'"${OUT}"
+
+cleanup
+trap - EXIT
+OUT="$("${PYTHON}" "${GATE}" 2>&1)"; EC=$?
+[ "${EC}" -eq 0 ] || fail "restored tree expected green, got
${EC}:"$'\n'"${OUT}"
+
+if [ "${FAILED}" -eq 0 ]; then
+ echo "PASS: unlisted include flagged with exact fix;
literal/whole-target/comment cases stay silent."
+ exit 0
+fi
+exit 1
diff --git a/build-support/tests/test-fe-core-metadata-funnel.sh
b/build-support/tests/test-fe-core-metadata-funnel.sh
index 4736f9ab003..c3a1a3b449f 100755
--- a/build-support/tests/test-fe-core-metadata-funnel.sh
+++ b/build-support/tests/test-fe-core-metadata-funnel.sh
@@ -142,8 +142,10 @@ bash "${GATE}" "${FX}" >/dev/null 2>&1 && CLEAN_EC=0 ||
CLEAN_EC=$?
[ "${CLEAN_EC}" -eq 0 ] || fail "expected exit 0 on clean tree (only
funnel/marked/no-arg/comment), got ${CLEAN_EC}"
# ---- run 3: the marker is load-bearing -> strip it and both write calls are
flagged ----
-sed -i 's/getMetadata-funnel-exempt/marker-removed-here/' \
+# (-i.bak + rm: the in-place flag's argument form differs between GNU and BSD
sed)
+sed -i.bak 's/getMetadata-funnel-exempt/marker-removed-here/' \
"${SRC}/write/WriterTrailing.java" "${SRC}/write/WriterAbove.java"
+rm -f "${SRC}/write/WriterTrailing.java.bak"
"${SRC}/write/WriterAbove.java.bak"
OUT3="$(bash "${GATE}" "${FX}" 2>&1)"; EC3=$?
REP3="$(printf '%s\n' "${OUT3}" | grep -E "^${FX}.*:[0-9]+:" || true)"
N3="$(printf '%s\n' "${REP3}" | grep -c 'getMetadata' || true)"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
