This is an automated email from the ASF dual-hosted git repository.
gnodet pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 64bd35a62a3c ci: improve Scalpel shadow mode using 0.4.x report
features (#26534)
64bd35a62a3c is described below
commit 64bd35a62a3cbbab03bdfaaaf0ef960c643b5e00
Author: Guillaume Nodet <[email protected]>
AuthorDate: Fri Sep 18 15:58:46 2026 +0200
ci: improve Scalpel shadow mode using 0.4.x report features (#26534)
* ci: improve Scalpel shadow mode using 0.4.x report features
Bump Scalpel extension3 from 0.4.0 to 0.4.2 and leverage new report
fields introduced in the 0.4.x series to improve the shadow comparison
section of PR comments.
Changes:
- Bump Scalpel 0.4.0 → 0.4.2 (.mvn/extensions.xml)
- 0.4.1: fixes buildAllIfNoChanges handling, adds reactorModuleCount,
emits decision inputs, fixes non-reactor file cascade to root module
- 0.4.2: fixes NPE when Scalpel trims to empty (currentProject restore)
- Add -Dscalpel.explain=true to runScalpelDetection()
Each affected module now carries an evidence[] array naming the exact
file, property, dependency, or graph edge that caused it to be included.
The PR comment surfaces this inline per module (e.g.
`camel-kafka` ← properties/version.kafka).
- Use reactorModuleCount report field for N-of-M framing
The summary line now reads "47 of 1,847 tested" instead of a bare count,
without any extra shell arithmetic — the reactor size comes directly from
the report.
- Use testedModulesCount report field directly
Eliminates the manual grep-c counting of scalpel_would_test list.
Falls back to counting affectedModules when the field is absent (older
report schema).
- Move POM change details above the module lists in the PR comment
Changed properties/deps/plugins are more useful as context before the
per-module breakdown.
- Update CI-ARCHITECTURE.md to document the new features
* ci: address review nits
- Pass report path as explicit argument to writeScalpelComparison
instead of re-declaring a relative path inside the function body;
the caller (running from the GHA workspace root) passes the same
path that runScalpelDetection uses, making the contract explicit and
safe if the script is ever invoked from a subdirectory
- Reserve N-of-M suffix for testedModulesCount only; scalpel_total
(affected = tested + compile-only) now shows a plain count so the
two numbers in the PR comment share the same denominator without
confusion about why they differ
---
.github/CI-ARCHITECTURE.md | 8 +-
.../actions/incremental-build/incremental-build.sh | 94 +++++++++++++---------
.mvn/extensions.xml | 2 +-
3 files changed, 63 insertions(+), 41 deletions(-)
diff --git a/.github/CI-ARCHITECTURE.md b/.github/CI-ARCHITECTURE.md
index 9c95094f0f63..d4a16ccfaf3d 100644
--- a/.github/CI-ARCHITECTURE.md
+++ b/.github/CI-ARCHITECTURE.md
@@ -165,16 +165,20 @@ Scalpel is only invoked when a **subdirectory** `pom.xml`
is changed (e.g. `pare
- **Source-set-aware propagation**: Distinguishes test-jar dependencies from
regular dependencies. A module that depends only on another module's test-jar
(e.g., `camel-core`'s test-jar with test utilities) is propagated through the
`TEST` source set, not the `MAIN` source set. This prevents a change to test
utilities from triggering tests in all ~500 modules that depend on `camel-core`.
- **`skipTestsForDownstreamModules`**: Allows specifying modules whose tests
should be skipped when they appear as downstream dependents (mirrors the
`EXCLUSION_LIST` in `incremental-build.sh`). This gives Scalpel an accurate
picture of what skip-tests mode would actually test.
+- **Explain mode** (`-Dscalpel.explain=true`): Each affected module carries an
`evidence[]` array naming the exact file, property, managed dependency, or
graph edge that caused it to be included. The PR comment surfaces this evidence
inline — e.g. `` `camel-kafka` ← properties/version.kafka `` — making it
immediately clear why each module is in the build set.
+- **`reactorModuleCount`** (report field, 0.4.1+): Total reactor size emitted
directly by Scalpel. Used for N-of-M framing in the PR comment (e.g. "47 of
1,847 tested") without extra shell arithmetic.
+- **`testedModulesCount`** (report field, 0.4.x): Modules whose tests will
actually run, computed by Scalpel. Used directly instead of counting list
elements in the script.
#### Shadow comparison
Scalpel runs in **shadow mode**: it observes what skip-tests mode *would* have
done and reports it in a collapsible section of the PR comment, without
affecting actual test execution. This allows the team to validate Scalpel's
decisions across many PRs before switching to Scalpel-driven test execution.
The shadow comparison section shows:
-- How many modules Scalpel would test (direct + downstream)
+- How many modules Scalpel would test out of the total reactor (N-of-M framing)
- How many downstream modules would have tests skipped (generated code,
meta-modules)
+- POM change details: changed properties, managed dependencies, managed plugins
- Set differences: modules only Scalpel found vs modules only the current
approach found
-- The full list of modules in each category
+- The full list of modules in each category, with per-module evidence (explain
mode)
The comparison is apples-to-apples: the current approach's reactor is filtered
through the `EXCLUSION_LIST` before comparing, so both sides exclude the same
meta/generated modules (catalog, jbang, docs, etc.).
diff --git a/.github/actions/incremental-build/incremental-build.sh
b/.github/actions/incremental-build/incremental-build.sh
index 4a2096d4b7c9..f8e52905e0af 100755
--- a/.github/actions/incremental-build/incremental-build.sh
+++ b/.github/actions/incremental-build/incremental-build.sh
@@ -256,6 +256,7 @@ runScalpelDetection() {
# Base branch is pre-fetched by the CI workflow (fetchBaseBranch=false).
# Run Maven validate with Scalpel in report mode:
# - mode=report: write JSON report without trimming the reactor
+ # - explain=true: add per-module evidence[] to the report (exact
file/property/dep that triggered each module)
# - fullBuildTriggers="": override .mvn/** default (Scalpel lives in
.mvn/extensions.xml)
# - fetchBaseBranch=false: base branch is pre-fetched by the CI workflow
# - skipTestsForDownstreamModules: derived from EXCLUSION_LIST — tells
Scalpel which
@@ -266,7 +267,7 @@ runScalpelDetection() {
# Always pass baseBranch explicitly — relying on Scalpel's
env.GITHUB_BASE_REF
# auto-detection is fragile across Maven wrappers and CI rerun contexts.
local base_branch="origin/${GITHUB_BASE_REF:-main}"
- local scalpel_args="-Dscalpel.enabled=true -Dscalpel.mode=report
-Dscalpel.fullBuildTriggers= -Dscalpel.fetchBaseBranch=false
-Dscalpel.baseBranch=${base_branch} -Dscalpel.excludePaths=.github/**
-Dscalpel.skipTestsForDownstreamModules=${skip_downstream}"
+ local scalpel_args="-Dscalpel.enabled=true -Dscalpel.mode=report
-Dscalpel.explain=true -Dscalpel.fullBuildTriggers=
-Dscalpel.fetchBaseBranch=false -Dscalpel.baseBranch=${base_branch}
-Dscalpel.excludePaths=.github/**
-Dscalpel.skipTestsForDownstreamModules=${skip_downstream}"
# Verify merge base is reachable (pre-fetched by the CI workflow step)
if ! git merge-base HEAD "${base_branch}" >/dev/null 2>&1; then
@@ -314,23 +315,22 @@ runScalpelDetection() {
scalpel_managed_deps=$(jq -r '(.changedManagedDependencies // []) | if
length > 0 then join(", ") else "" end' "$report" 2>/dev/null || true)
scalpel_managed_plugins=$(jq -r '(.changedManagedPlugins // []) | if length
> 0 then join(", ") else "" end' "$report" 2>/dev/null || true)
- # Scalpel shadow comparison data:
- # - Modules Scalpel skip-tests mode would test (testsSkipped != true)
- # - Modules Scalpel would skip (testsSkipped == true, from
skipTestsForDownstreamModules)
- # - Breakdown by category (DIRECT, DOWNSTREAM)
+ # Scalpel shadow comparison data — read directly from report fields (0.4.x):
+ # - testedModulesCount: modules whose tests will run (report field, avoids
manual counting)
+ # - reactorModuleCount: total reactor size for N-of-M display (report field,
0.4.1+)
+ # - Per-module lists derived from affectedModules (for the comment detail
sections)
scalpel_would_test=$(jq -r '[.affectedModules[] | select(.testsSkipped !=
true)] | map(.artifactId) | sort | join(",")' "$report" 2>/dev/null || true)
scalpel_would_skip=$(jq -r '[.affectedModules[] | select(.testsSkipped ==
true)] | map(.artifactId) | sort | join(",")' "$report" 2>/dev/null || true)
scalpel_direct_count=$(jq '[.affectedModules[] | select(.category ==
"DIRECT")] | length' "$report" 2>/dev/null || echo "0")
scalpel_downstream_tested=$(jq '[.affectedModules[] | select(.category ==
"DOWNSTREAM" and .testsSkipped != true)] | length' "$report" 2>/dev/null ||
echo "0")
scalpel_downstream_skipped=$(jq '[.affectedModules[] | select(.category ==
"DOWNSTREAM" and .testsSkipped == true)] | length' "$report" 2>/dev/null ||
echo "0")
+ # testedModulesCount and reactorModuleCount are written directly by Scalpel
(0.4.x report fields)
+ scalpel_tested_count=$(jq '.testedModulesCount // (.affectedModules |
map(select(.testsSkipped != true)) | length)' "$report" 2>/dev/null || echo "0")
+ scalpel_reactor_count=$(jq '.reactorModuleCount // 0' "$report" 2>/dev/null
|| echo "0")
local mod_count
mod_count=$(jq '.affectedModules | length' "$report" 2>/dev/null || echo "0")
- local test_count=0
- if [ -n "$scalpel_would_test" ]; then
- test_count=$(echo "$scalpel_would_test" | tr ',' '\n' | grep -c . || true)
- fi
- echo " Scalpel detected $mod_count affected modules ($test_count would be
tested)"
+ echo " Scalpel detected $mod_count affected modules ($scalpel_tested_count
would be tested, reactor: $scalpel_reactor_count)"
echo " Direct: $scalpel_direct_count, Downstream tested:
$scalpel_downstream_tested, Downstream skipped: $scalpel_downstream_skipped"
if [ -n "$scalpel_props" ]; then
echo " Changed properties: $scalpel_props"
@@ -427,6 +427,7 @@ checkManualItTests() {
writeScalpelComparison() {
local comment_file="$1"
local current_reactor_ids="${2:-}"
+ local report="${3:-target/scalpel-report.json}"
# If Scalpel failed, show why in the PR comment
if [ -n "$scalpel_failure_reason" ]; then
@@ -448,19 +449,16 @@ writeScalpelComparison() {
return
fi
- # Count Scalpel modules
+ # Counts — use report fields directly (0.4.x); fall back to counting lists
for older reports
local scalpel_total=0
- local scalpel_test_count=0
local scalpel_skip_count=0
if [ -n "$scalpel_module_ids" ]; then
scalpel_total=$(echo "$scalpel_module_ids" | tr ',' '\n' | grep -c . ||
true)
fi
- if [ -n "$scalpel_would_test" ]; then
- scalpel_test_count=$(echo "$scalpel_would_test" | tr ',' '\n' | grep -c .
|| true)
- fi
if [ -n "$scalpel_would_skip" ]; then
scalpel_skip_count=$(echo "$scalpel_would_skip" | tr ',' '\n' | grep -c .
|| true)
fi
+ # scalpel_tested_count and scalpel_reactor_count already set from report
fields in runScalpelDetection
# Compare Scalpel vs current reactor (file-path + grep + -amd expansion)
local current_total=0
@@ -490,8 +488,14 @@ writeScalpelComparison() {
only_current_count=$current_total
fi
- # One-line summary: what Scalpel would change
- local summary="Scalpel: ${scalpel_test_count} tested, ${scalpel_skip_count}
compile-only — current: ${current_total} all tested"
+ # N-of-M framing: use reactorModuleCount from report when available
+ local nm_suffix=""
+ if [ "$scalpel_reactor_count" -gt 0 ] 2>/dev/null; then
+ nm_suffix=" of ${scalpel_reactor_count}"
+ fi
+
+ # One-line summary with N-of-M framing
+ local summary="Scalpel: ${scalpel_tested_count}${nm_suffix} tested,
${scalpel_skip_count} compile-only — current: ${current_total} all tested"
echo "" >> "$comment_file"
echo "---" >> "$comment_file"
@@ -502,8 +506,25 @@ writeScalpelComparison() {
echo "[Maveniverse Scalpel](https://github.com/maveniverse/scalpel) detected
**${scalpel_total} affected modules** (current approach: ${current_total})." >>
"$comment_file"
echo "" >> "$comment_file"
- # Show modules only Scalpel found (not in current reactor)
+ # Show Scalpel-detected change details
+ if [ -n "$scalpel_props" ]; then
+ echo "Changed properties: ${scalpel_props}" >> "$comment_file"
+ echo "" >> "$comment_file"
+ fi
+ if [ -n "$scalpel_managed_deps" ]; then
+ echo "Changed managed dependencies: ${scalpel_managed_deps}" >>
"$comment_file"
+ echo "" >> "$comment_file"
+ fi
+ if [ -n "$scalpel_managed_plugins" ]; then
+ echo "Changed managed plugins: ${scalpel_managed_plugins}" >>
"$comment_file"
+ echo "" >> "$comment_file"
+ fi
+
+ echo "**Skip-tests mode would test ${scalpel_tested_count} modules**
(${scalpel_direct_count} direct + ${scalpel_downstream_tested} downstream),
**skip tests for ${scalpel_skip_count}** (generated code, meta-modules)" >>
"$comment_file"
+
+ # Show modules only Scalpel found (not in current reactor) — with explain
evidence
if [ "$only_scalpel_count" -gt 0 ]; then
+ echo "" >> "$comment_file"
echo "<details><summary>:warning: Modules only in Scalpel
(${only_scalpel_count})</summary>" >> "$comment_file"
echo "" >> "$comment_file"
echo "$only_in_scalpel" | while read -r m; do
@@ -516,6 +537,7 @@ writeScalpelComparison() {
# Show modules only current approach found (not in Scalpel)
if [ "$only_current_count" -gt 0 ]; then
+ echo "" >> "$comment_file"
echo "<details><summary>Modules only in current approach
(${only_current_count})</summary>" >> "$comment_file"
echo "" >> "$comment_file"
echo "$only_in_current" | while read -r m; do
@@ -526,29 +548,23 @@ writeScalpelComparison() {
echo "" >> "$comment_file"
fi
- # Show Scalpel-detected change details
- if [ -n "$scalpel_props" ]; then
- echo "Changed properties: ${scalpel_props}" >> "$comment_file"
- echo "" >> "$comment_file"
- fi
- if [ -n "$scalpel_managed_deps" ]; then
- echo "Changed managed dependencies: ${scalpel_managed_deps}" >>
"$comment_file"
- echo "" >> "$comment_file"
- fi
- if [ -n "$scalpel_managed_plugins" ]; then
- echo "Changed managed plugins: ${scalpel_managed_plugins}" >>
"$comment_file"
- echo "" >> "$comment_file"
- fi
-
- echo "**Skip-tests mode would test ${scalpel_test_count} modules**
(${scalpel_direct_count} direct + ${scalpel_downstream_tested} downstream),
**skip tests for ${scalpel_skip_count}** (generated code, meta-modules)" >>
"$comment_file"
-
- # Show which modules Scalpel would test
+ # Show which modules Scalpel would test, with per-module evidence
(explain=true)
if [ -n "$scalpel_would_test" ]; then
echo "" >> "$comment_file"
- echo "<details><summary>Modules Scalpel would test
(${scalpel_test_count})</summary>" >> "$comment_file"
+ echo "<details><summary>Modules Scalpel would test
(${scalpel_tested_count})</summary>" >> "$comment_file"
echo "" >> "$comment_file"
echo "$scalpel_would_test" | tr ',' '\n' | while read -r m; do
- [ -n "$m" ] && echo "- \`$m\`" >> "$comment_file"
+ if [ -n "$m" ]; then
+ # Pull evidence[] for this module from the report (explain=true
populates it)
+ local evidence=""
+ evidence=$(jq -r --arg art "$m" '
+ .affectedModules[]
+ | select(.artifactId == $art and (.testsSkipped != true))
+ | (.evidence // [])
+ | if length > 0 then " ← " + join(", ") else "" end
+ ' "$report" 2>/dev/null | head -1 || true)
+ echo "- \`$m\`${evidence}" >> "$comment_file"
+ fi
done
echo "" >> "$comment_file"
echo "</details>" >> "$comment_file"
@@ -721,6 +737,8 @@ main() {
scalpel_direct_count="0"
scalpel_downstream_tested="0"
scalpel_downstream_skipped="0"
+ scalpel_tested_count="0"
+ scalpel_reactor_count="0"
scalpel_failure_reason=""
# Step 2a: Grep-based detection (existing approach)
@@ -977,7 +995,7 @@ main() {
fi
done)
fi
- writeScalpelComparison "$comment_file" "$tested_reactor_ids"
+ writeScalpelComparison "$comment_file" "$tested_reactor_ids"
"target/scalpel-report.json"
# Check for tests disabled in CI via @DisabledIfSystemProperty(named =
"ci.env.name")
local disabled_tests
diff --git a/.mvn/extensions.xml b/.mvn/extensions.xml
index f437e147b91f..d5e447684d28 100644
--- a/.mvn/extensions.xml
+++ b/.mvn/extensions.xml
@@ -23,6 +23,6 @@
<extension>
<groupId>eu.maveniverse.maven.scalpel</groupId>
<artifactId>extension3</artifactId>
- <version>0.4.1</version>
+ <version>0.4.2</version>
</extension>
</extensions>
