englefly opened a new pull request, #67824:
URL: https://github.com/apache/doris/pull/67824
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
A local edit anywhere in `fe-core` made `mvn compile` rebuild the whole
module: 4473 java
sources, about two minutes. Two independent causes were found and fixed.
**1. The pattern hierarchy was regenerated on every build.**
`GeneratedPlanPatterns` /
`GeneratedMemoPatterns` / `GeneratedPlanRelations` /
`GeneratedExpressionRelations` were
produced by a javac annotation processor which, on every build, parsed all
2678 sources of the
nereids tree with antlr (about 9 s) and unconditionally rewrote the four
generated files. The
fresh timestamps kept the whole module stale, and the generated content was
not even
reproducible: the file walk order came from `FileUtils.listFiles`,
`PlanTypeMappingGenerator.findSuperPlan` collected it into a
`LinkedHashSet`, and
`PlanPatternGeneratorAnalyzer` read an `IdentityHashMap`.
Generation is now a plain program (`PatternCodeGenerator`) run by
`exec-maven-plugin` from
`process-sources`. It caches the parsed ast of every source file, keyed by
its content hash, so
only the modified files are re-parsed, and it rewrites an output file only
when its content
really changed, so unchanged generated sources keep their timestamp. The
emit order was made
deterministic. Measured: the generation step drops from about 9 s to 0.8 s,
and a build with no
source change rewrites 0 of the 4 generated files. Two independent cold runs
now produce
byte-identical output.
**2. `maven-compiler-plugin` is all-or-nothing.** As soon as one source is
newer than its class
file the plugin recompiles every source of the module (`Changes detected -
recompiling the
module!`), which is independent of the generated sources: removing the
annotation processor
alone does not make an edit incremental.
An opt-in profile `-Pfast-fe` now adds `IncrementalSourceMarker`. It indexes
the identifiers each
source mentions and the type names it declares, then marks (touches) the
changed sources plus
every source which mentions a type they declare. The compiler plugin's
per-file mode then
compiles exactly that set. The set is a conservative superset, so no class
can be left referring
to a type which changed, moved or disappeared underneath it.
Measured on `fe-core`, editing one file:
| build | javac work | wall clock |
| --- | --- | --- |
| default | `Compiling 4473 source files` | about 2:15 |
| `-Pfast-fe` | `Compiling 72 source files` | 27 s |
The result was validated by comparing every class file against a forced full
rebuild after
changing the inlined constant `StringLikeLiteral.CHINESE_CHAR_BYTE_LENGTH`,
which a broken
dependency propagation would have left stale: all ~6000 class files
byte-identical. Editing a
leaf operator marks 130 of 4527 sources, editing a widely used class such as
`Plan` marks 578.
The default build is unchanged; `-Pfast-fe` is opt-in and intended for the
local edit loop. The
only coupling it cannot see is one which is not spelled out in the source at
all (reflection,
`ServiceLoader`, resource lookups), so a release build should keep using the
plain build.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- The incremental build was compared byte for byte against a forced
full rebuild (all
~6000 class files identical) after changing an inlined constant,
which a broken
dependency propagation would have left stale.
- `build-support/tests/test-incremental-source-marker.sh` covers the
new build tool: no
change marks nothing, editing a type marks its referrers and not
an unrelated file, a
mention inside a comment or a string literal is not a dependency,
a new type marks only
itself, removing a source deletes its own classes and no others.
- `mvn compile -pl fe-core -am` and `mvn -Pfast-fe compile -pl
fe-core -am` both pass with
0 checkstyle violations.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
