hubcio commented on code in PR #3999: URL: https://github.com/apache/iggy/pull/3999#discussion_r4060504873
########## .config/rail.toml: ########## @@ -15,14 +15,41 @@ # specific language governing permissions and limitations # under the License. -# cargo-rail change detection configuration -# See: https://github.com/loadingalias/cargo-rail - -[change-detection] -# Changes to these paths trigger a full workspace rebuild/retest -infrastructure = [ - "Cargo.lock", - "rust-toolchain.toml", - ".cargo/**", - ".github/**", +[plan.work.runtime-artifacts] +scope = "cargo" +cargo_prerequisites = [ Review Comment: critical: a `gateways/**` change builds no `iggy-server`, so `bridge_iggy_integration_tests.rs` panics - or worse, silently runs a stale master-tip binary from the restored cache. add a `cargo_prerequisites` entry for `iggy-gateway-kafka` requiring the `iggy-server` bin. ########## .github/actions/rust/pre-merge/action.yml: ########## @@ -325,33 +260,30 @@ runs: fi bins_start=$(date +%s) - if [[ -n "$PACKAGE_FLAGS" ]]; then - cargo build --locked $PACKAGE_FLAGS - else - cargo build --locked - fi + cargo build --locked "${BUILD_ARGS[@]}" bins_end=$(date +%s) bins_duration=$((bins_end - bins_start)) echo "::notice::Binaries and libraries built in ${bins_duration}s ($(date -ud @${bins_duration} +'%M:%S'))" compile_start=$(date +%s) - if [[ -n "$PACKAGE_FLAGS" ]]; then - cargo test --locked --no-run $PACKAGE_FLAGS - else - cargo test --locked --no-run - fi + cargo test --locked --no-run "${TEST_CARGO_ARGS[@]}" "${TEST_TARGET_ARGS[@]}" Review Comment: warning: `--test <name>` cancels everything else in the run. `cargo test -p a -p b --test only_in_a` runs one target, skips `b` and all unit tests, exit 0 - so editing `core/server/tests/sdk_e2e.rs` plus a `src/` file drops `integration` and `simulator`. drop `TEST_TARGET_ARGS`. ########## .config/rail.toml: ########## @@ -15,14 +15,41 @@ # specific language governing permissions and limitations # under the License. -# cargo-rail change detection configuration -# See: https://github.com/loadingalias/cargo-rail - -[change-detection] -# Changes to these paths trigger a full workspace rebuild/retest -infrastructure = [ - "Cargo.lock", - "rust-toolchain.toml", - ".cargo/**", - ".github/**", +[plan.work.runtime-artifacts] +scope = "cargo" +cargo_prerequisites = [ + { source_work = "cargo.test", when = [ + { package = "integration" }, + ], require = [ + { package = "server", target = { name = "iggy-server", kind = "bin" } }, + { package = "iggy-cli", target = { name = "iggy", kind = "bin" } }, + { package = "iggy-bench", target = { name = "iggy-bench", kind = "bin" } }, + { package = "iggy-connectors", target = { name = "iggy-connectors", kind = "bin" } }, + { package = "iggy-mcp", target = { name = "iggy-mcp", kind = "bin" } }, + { package = "iggy_connector_clickhouse_sink", target = { name = "iggy_connector_clickhouse_sink", kind = "cdylib" } }, Review Comment: warning: the 21 cdylib entries used to come from `cargo metadata`, now they are hand-written with nothing checking them. a new connector silently misses the build until its suite fails to load a missing `.so`. ########## .config/rail.toml: ########## @@ -15,14 +15,41 @@ # specific language governing permissions and limitations # under the License. -# cargo-rail change detection configuration -# See: https://github.com/loadingalias/cargo-rail - -[change-detection] -# Changes to these paths trigger a full workspace rebuild/retest -infrastructure = [ - "Cargo.lock", - "rust-toolchain.toml", - ".cargo/**", - ".github/**", +[plan.work.runtime-artifacts] Review Comment: warning: dropping `[change-detection] infrastructure` means a `.github/**`, `.config/**`, `Cargo.lock` or `rust-toolchain.toml` change now skips the rust unit and integration suites in `test-1/2/3`. clippy, miri and the bdd fleet still cover it, but declare these paths full-scope anyway. ########## .github/config/edge-image-variants.json: ########## @@ -0,0 +1,117 @@ +{ + "variant_catalog_version": 2, + "work": "edge-images", + "variants": [ + { + "id": "rust-server", + "dimensions": { "component": "rust-server" }, + "cargo_roots": [ + { "package": "server", "target": { "name": "iggy-server", "kind": "bin" } }, + { "package": "iggy-cli", "target": { "name": "iggy", "kind": "bin" } } + ], + "external_paths": [ + ".dockerignore", + ".github/actions/utils/docker-buildx/**", + ".github/actions/utils/docker-login/**", + ".github/config/publish.yml", + ".github/workflows/post-merge.yml", + ".github/workflows/publish.yml", + "LICENSE", + "NOTICE", + "about.toml", + "about.hbs", + "core/server/Dockerfile", Review Comment: warning: these four `dockerfile:` paths are copied from `publish.yml` and nothing compares the two. move a dockerfile and the catalog keeps the old path, so edits at the new one select no row and the image quietly stops refreshing. ########## .github/config/publish.yml: ########## @@ -55,13 +55,6 @@ components: platforms: ["linux/amd64", "linux/arm64"] Review Comment: nit: nothing here points at `.github/config/edge-image-variants.json`, which decides whether this image gets an `:edge` refresh. the removed `gate:` block carried that link - add a comment. ########## .github/actions/rust/pre-merge/action.yml: ########## @@ -272,44 +180,71 @@ runs: # (test-storage-compat) opt out of partitioning rather than producing # an out-of-range hash:N/3. TEST_PARTITIONS=3 - TASK="${{ inputs.task }}" - PARTITION_FLAG="" + PARTITION_ARGS=() if [[ "$TASK" =~ ^test-([0-9]+)$ ]]; then PARTITION_INDEX="${BASH_REMATCH[1]}" - PARTITION_FLAG="--partition hash:${PARTITION_INDEX}/${TEST_PARTITIONS}" + PARTITION_ARGS=(--partition "hash:${PARTITION_INDEX}/${TEST_PARTITIONS}") echo "::notice::Running test partition ${PARTITION_INDEX}/${TEST_PARTITIONS}" fi - # Read DAG-based affected crate filter (computed in earlier step) - NEXTEST_FILTER="" - PACKAGE_FLAGS="" - TEST_PACKAGE_FLAGS="" - TOTAL_CRATES="?" - PLUGIN_SUITES=0 - if [[ -f /tmp/nextest-filter.txt ]]; then - NEXTEST_FILTER=$(cat /tmp/nextest-filter.txt) - fi - if [[ -f /tmp/packages.txt ]]; then - PACKAGE_FLAGS=$(cat /tmp/packages.txt) - fi - if [[ -f /tmp/test-packages.txt ]]; then - TEST_PACKAGE_FLAGS=$(cat /tmp/test-packages.txt) - fi - if [[ -f /tmp/total-crates.txt ]]; then - TOTAL_CRATES=$(cat /tmp/total-crates.txt) + # Read only validated selectors through the v10 runtime. Any planner or + # selector failure restores the previous full-workspace behavior. + PLAN_OUTPUT_DIR="${RUNNER_TEMP:?RUNNER_TEMP is required}/cargo-rail-selectors-${TASK}" + mkdir -p "$PLAN_OUTPUT_DIR" + PLAN_USABLE=false + TEST_SCOPE=workspace + RUNTIME_SCOPE=workspace + TEST_CARGO_ARGS=() + TEST_TARGET_ARGS=() + TEST_PACKAGES=() + RUNTIME_PACKAGES=() + + if [[ "$PLAN_STATUS" == "success" ]] && + TEST_SCOPE=$(cargo-rail-action plan cargo-scope "$PLAN_FILE" cargo.test) && + RUNTIME_SCOPE=$(cargo-rail-action plan cargo-scope "$PLAN_FILE" runtime-artifacts) && + cargo-rail-action plan cargo-args "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-args" && + cargo-rail-action plan target-args "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-target-args" && + cargo-rail-action plan package-names "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-packages" && + cargo-rail-action plan package-names "$PLAN_FILE" runtime-artifacts > "$PLAN_OUTPUT_DIR/runtime-packages"; then + mapfile -d '' -t TEST_CARGO_ARGS < "$PLAN_OUTPUT_DIR/test-args" + mapfile -d '' -t TEST_TARGET_ARGS < "$PLAN_OUTPUT_DIR/test-target-args" + mapfile -d '' -t TEST_PACKAGES < "$PLAN_OUTPUT_DIR/test-packages" + mapfile -d '' -t RUNTIME_PACKAGES < "$PLAN_OUTPUT_DIR/runtime-packages" + PLAN_USABLE=true + else + TEST_SCOPE=workspace + RUNTIME_SCOPE=workspace + TEST_CARGO_ARGS=() Review Comment: simplification: these four array resets are dead - only the `mapfile` calls in the success branch ever fill them. keep the two scope resets, they do matter. ########## .github/actions/rust/pre-merge/action.yml: ########## @@ -272,44 +180,71 @@ runs: # (test-storage-compat) opt out of partitioning rather than producing # an out-of-range hash:N/3. TEST_PARTITIONS=3 - TASK="${{ inputs.task }}" - PARTITION_FLAG="" + PARTITION_ARGS=() if [[ "$TASK" =~ ^test-([0-9]+)$ ]]; then PARTITION_INDEX="${BASH_REMATCH[1]}" - PARTITION_FLAG="--partition hash:${PARTITION_INDEX}/${TEST_PARTITIONS}" + PARTITION_ARGS=(--partition "hash:${PARTITION_INDEX}/${TEST_PARTITIONS}") echo "::notice::Running test partition ${PARTITION_INDEX}/${TEST_PARTITIONS}" fi - # Read DAG-based affected crate filter (computed in earlier step) - NEXTEST_FILTER="" - PACKAGE_FLAGS="" - TEST_PACKAGE_FLAGS="" - TOTAL_CRATES="?" - PLUGIN_SUITES=0 - if [[ -f /tmp/nextest-filter.txt ]]; then - NEXTEST_FILTER=$(cat /tmp/nextest-filter.txt) - fi - if [[ -f /tmp/packages.txt ]]; then - PACKAGE_FLAGS=$(cat /tmp/packages.txt) - fi - if [[ -f /tmp/test-packages.txt ]]; then - TEST_PACKAGE_FLAGS=$(cat /tmp/test-packages.txt) - fi - if [[ -f /tmp/total-crates.txt ]]; then - TOTAL_CRATES=$(cat /tmp/total-crates.txt) + # Read only validated selectors through the v10 runtime. Any planner or + # selector failure restores the previous full-workspace behavior. + PLAN_OUTPUT_DIR="${RUNNER_TEMP:?RUNNER_TEMP is required}/cargo-rail-selectors-${TASK}" + mkdir -p "$PLAN_OUTPUT_DIR" + PLAN_USABLE=false + TEST_SCOPE=workspace + RUNTIME_SCOPE=workspace + TEST_CARGO_ARGS=() + TEST_TARGET_ARGS=() + TEST_PACKAGES=() + RUNTIME_PACKAGES=() + + if [[ "$PLAN_STATUS" == "success" ]] && + TEST_SCOPE=$(cargo-rail-action plan cargo-scope "$PLAN_FILE" cargo.test) && + RUNTIME_SCOPE=$(cargo-rail-action plan cargo-scope "$PLAN_FILE" runtime-artifacts) && + cargo-rail-action plan cargo-args "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-args" && + cargo-rail-action plan target-args "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-target-args" && + cargo-rail-action plan package-names "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-packages" && + cargo-rail-action plan package-names "$PLAN_FILE" runtime-artifacts > "$PLAN_OUTPUT_DIR/runtime-packages"; then + mapfile -d '' -t TEST_CARGO_ARGS < "$PLAN_OUTPUT_DIR/test-args" + mapfile -d '' -t TEST_TARGET_ARGS < "$PLAN_OUTPUT_DIR/test-target-args" + mapfile -d '' -t TEST_PACKAGES < "$PLAN_OUTPUT_DIR/test-packages" + mapfile -d '' -t RUNTIME_PACKAGES < "$PLAN_OUTPUT_DIR/runtime-packages" + PLAN_USABLE=true + else + TEST_SCOPE=workspace + RUNTIME_SCOPE=workspace + TEST_CARGO_ARGS=() + TEST_TARGET_ARGS=() + TEST_PACKAGES=() + RUNTIME_PACKAGES=() + echo "::warning::Cargo-Rail plan unavailable or invalid; running the full workspace" fi - if [[ -f /tmp/plugin-suites.txt ]]; then - PLUGIN_SUITES=$(cat /tmp/plugin-suites.txt) + + if [[ "$PLAN_USABLE" == true && "$TEST_SCOPE" == "skipped" ]]; then + echo "::notice::Cargo-Rail reports no affected Rust tests" + exit 0 Review Comment: nit: this exit skips `cargo llvm-cov report`, so the codecov step logs an error for a missing `codecov.json`. harmless given `carryforward: true` and `fail_ci_if_error: false`, but it is noise on every skipped run. ########## .github/workflows/post-merge.yml: ########## @@ -60,28 +60,56 @@ jobs: chmod +x /usr/local/bin/yq fi - # cargo-rail computes the affected crate set for the Docker :edge gate. - # Metadata-only `cargo rail plan` (no compile), so it runs on the runner's - # preinstalled cargo (version pinned by rust-toolchain.toml) and skips the - # heavyweight build-cache restore. - # Pinned for the same reason as .github/actions/rust/pre-merge/action.yml: - # 0.24 dropped the `-f json` edge-affected-images.sh relies on. - - name: Install cargo-rail - uses: taiki-e/[email protected] + # Planning is an optimization. A failure falls back to refreshing every + # Docker image in the component registry. + - name: Plan affected edge images + id: rail + continue-on-error: true + uses: loadingalias/cargo-rail-action@4a8df181e4f340a164190cf092c9f9239b20edab # v10.0.1 with: - tool: [email protected] + version: 0.29.0 - name: Check all components id: check + env: + PLAN_FILE: ${{ steps.rail.outputs.plan-file }} + PLAN_STATUS: ${{ steps.rail.outcome }} run: | - chmod +x scripts/extract-version.sh + set -euo pipefail + + COMPONENTS_JSON=$(yq -o=json -I=0 '.components' .github/config/publish.yml) + ALL_DOCKER_COMPONENTS=$(jq -r \ + '[to_entries[] | select(.value.registry == "dockerhub") | .key] | join(",")' \ + <<< "$COMPONENTS_JSON") + + IMAGE_MATRIX=all + if [[ "$PLAN_STATUS" == "success" ]] && + IMAGE_MATRIX=$(cargo-rail-action plan matrix "$PLAN_FILE" edge-images); then + echo "::notice::Using validated Cargo-Rail edge image plan" + else + echo "::warning::Cargo-Rail plan unavailable or invalid; refreshing all edge images" + fi + + if [[ "$IMAGE_MATRIX" == "all" ]]; then + DOCKER_COMPONENTS="$ALL_DOCKER_COMPONENTS" + elif DOCKER_COMPONENTS=$(jq -er --argjson configured "$COMPONENTS_JSON" ' Review Comment: warning: this only checks that every selected component exists in `publish.yml`, never that every dockerhub component has a catalog row - so a sixth image never gets an `:edge` refresh. the deleted script was fail-open here via `has("gate")`. ########## .github/actions/rust/pre-merge/action.yml: ########## @@ -272,44 +180,71 @@ runs: # (test-storage-compat) opt out of partitioning rather than producing # an out-of-range hash:N/3. TEST_PARTITIONS=3 - TASK="${{ inputs.task }}" - PARTITION_FLAG="" + PARTITION_ARGS=() if [[ "$TASK" =~ ^test-([0-9]+)$ ]]; then PARTITION_INDEX="${BASH_REMATCH[1]}" - PARTITION_FLAG="--partition hash:${PARTITION_INDEX}/${TEST_PARTITIONS}" + PARTITION_ARGS=(--partition "hash:${PARTITION_INDEX}/${TEST_PARTITIONS}") echo "::notice::Running test partition ${PARTITION_INDEX}/${TEST_PARTITIONS}" fi - # Read DAG-based affected crate filter (computed in earlier step) - NEXTEST_FILTER="" - PACKAGE_FLAGS="" - TEST_PACKAGE_FLAGS="" - TOTAL_CRATES="?" - PLUGIN_SUITES=0 - if [[ -f /tmp/nextest-filter.txt ]]; then - NEXTEST_FILTER=$(cat /tmp/nextest-filter.txt) - fi - if [[ -f /tmp/packages.txt ]]; then - PACKAGE_FLAGS=$(cat /tmp/packages.txt) - fi - if [[ -f /tmp/test-packages.txt ]]; then - TEST_PACKAGE_FLAGS=$(cat /tmp/test-packages.txt) - fi - if [[ -f /tmp/total-crates.txt ]]; then - TOTAL_CRATES=$(cat /tmp/total-crates.txt) + # Read only validated selectors through the v10 runtime. Any planner or + # selector failure restores the previous full-workspace behavior. + PLAN_OUTPUT_DIR="${RUNNER_TEMP:?RUNNER_TEMP is required}/cargo-rail-selectors-${TASK}" + mkdir -p "$PLAN_OUTPUT_DIR" + PLAN_USABLE=false Review Comment: simplification: `PLAN_USABLE` is redundant - the `else` branch resets `TEST_SCOPE=workspace`, so testing `TEST_SCOPE` alone says the same thing, which lines 279 and 343 already do. also at lines 224, 231, 244, 371. ########## .github/actions/rust/pre-merge/action.yml: ########## @@ -272,44 +180,71 @@ runs: # (test-storage-compat) opt out of partitioning rather than producing # an out-of-range hash:N/3. TEST_PARTITIONS=3 - TASK="${{ inputs.task }}" - PARTITION_FLAG="" + PARTITION_ARGS=() if [[ "$TASK" =~ ^test-([0-9]+)$ ]]; then PARTITION_INDEX="${BASH_REMATCH[1]}" - PARTITION_FLAG="--partition hash:${PARTITION_INDEX}/${TEST_PARTITIONS}" + PARTITION_ARGS=(--partition "hash:${PARTITION_INDEX}/${TEST_PARTITIONS}") echo "::notice::Running test partition ${PARTITION_INDEX}/${TEST_PARTITIONS}" fi - # Read DAG-based affected crate filter (computed in earlier step) - NEXTEST_FILTER="" - PACKAGE_FLAGS="" - TEST_PACKAGE_FLAGS="" - TOTAL_CRATES="?" - PLUGIN_SUITES=0 - if [[ -f /tmp/nextest-filter.txt ]]; then - NEXTEST_FILTER=$(cat /tmp/nextest-filter.txt) - fi - if [[ -f /tmp/packages.txt ]]; then - PACKAGE_FLAGS=$(cat /tmp/packages.txt) - fi - if [[ -f /tmp/test-packages.txt ]]; then - TEST_PACKAGE_FLAGS=$(cat /tmp/test-packages.txt) - fi - if [[ -f /tmp/total-crates.txt ]]; then - TOTAL_CRATES=$(cat /tmp/total-crates.txt) + # Read only validated selectors through the v10 runtime. Any planner or + # selector failure restores the previous full-workspace behavior. + PLAN_OUTPUT_DIR="${RUNNER_TEMP:?RUNNER_TEMP is required}/cargo-rail-selectors-${TASK}" + mkdir -p "$PLAN_OUTPUT_DIR" + PLAN_USABLE=false + TEST_SCOPE=workspace + RUNTIME_SCOPE=workspace + TEST_CARGO_ARGS=() + TEST_TARGET_ARGS=() + TEST_PACKAGES=() + RUNTIME_PACKAGES=() + + if [[ "$PLAN_STATUS" == "success" ]] && + TEST_SCOPE=$(cargo-rail-action plan cargo-scope "$PLAN_FILE" cargo.test) && + RUNTIME_SCOPE=$(cargo-rail-action plan cargo-scope "$PLAN_FILE" runtime-artifacts) && + cargo-rail-action plan cargo-args "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-args" && + cargo-rail-action plan target-args "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-target-args" && + cargo-rail-action plan package-names "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-packages" && + cargo-rail-action plan package-names "$PLAN_FILE" runtime-artifacts > "$PLAN_OUTPUT_DIR/runtime-packages"; then + mapfile -d '' -t TEST_CARGO_ARGS < "$PLAN_OUTPUT_DIR/test-args" + mapfile -d '' -t TEST_TARGET_ARGS < "$PLAN_OUTPUT_DIR/test-target-args" Review Comment: nit: `mapfile -d ''` needs NUL-separated selector output, and the comment above names the validation but not the separator. worth a word in that comment. ########## .github/workflows/post-merge.yml: ########## @@ -60,28 +60,56 @@ jobs: chmod +x /usr/local/bin/yq fi - # cargo-rail computes the affected crate set for the Docker :edge gate. - # Metadata-only `cargo rail plan` (no compile), so it runs on the runner's - # preinstalled cargo (version pinned by rust-toolchain.toml) and skips the - # heavyweight build-cache restore. - # Pinned for the same reason as .github/actions/rust/pre-merge/action.yml: - # 0.24 dropped the `-f json` edge-affected-images.sh relies on. - - name: Install cargo-rail - uses: taiki-e/[email protected] + # Planning is an optimization. A failure falls back to refreshing every + # Docker image in the component registry. + - name: Plan affected edge images + id: rail + continue-on-error: true + uses: loadingalias/cargo-rail-action@4a8df181e4f340a164190cf092c9f9239b20edab # v10.0.1 with: - tool: [email protected] + version: 0.29.0 - name: Check all components id: check + env: + PLAN_FILE: ${{ steps.rail.outputs.plan-file }} + PLAN_STATUS: ${{ steps.rail.outcome }} run: | - chmod +x scripts/extract-version.sh + set -euo pipefail + + COMPONENTS_JSON=$(yq -o=json -I=0 '.components' .github/config/publish.yml) + ALL_DOCKER_COMPONENTS=$(jq -r \ + '[to_entries[] | select(.value.registry == "dockerhub") | .key] | join(",")' \ + <<< "$COMPONENTS_JSON") + + IMAGE_MATRIX=all + if [[ "$PLAN_STATUS" == "success" ]] && + IMAGE_MATRIX=$(cargo-rail-action plan matrix "$PLAN_FILE" edge-images); then + echo "::notice::Using validated Cargo-Rail edge image plan" + else + echo "::warning::Cargo-Rail plan unavailable or invalid; refreshing all edge images" + fi + + if [[ "$IMAGE_MATRIX" == "all" ]]; then + DOCKER_COMPONENTS="$ALL_DOCKER_COMPONENTS" + elif DOCKER_COMPONENTS=$(jq -er --argjson configured "$COMPONENTS_JSON" ' + [.include[].component] as $selected + | if ($selected | all(. as $component | $configured[$component].registry == "dockerhub")) + then $selected | join(",") + else error("Cargo-Rail selected an unknown Docker component") Review Comment: nit: this error passes a constant string, so the log names neither the rejected component nor the matrix. interpolate `$component` into it. ########## .github/workflows/post-merge.yml: ########## @@ -60,28 +60,56 @@ jobs: chmod +x /usr/local/bin/yq fi - # cargo-rail computes the affected crate set for the Docker :edge gate. - # Metadata-only `cargo rail plan` (no compile), so it runs on the runner's - # preinstalled cargo (version pinned by rust-toolchain.toml) and skips the - # heavyweight build-cache restore. - # Pinned for the same reason as .github/actions/rust/pre-merge/action.yml: - # 0.24 dropped the `-f json` edge-affected-images.sh relies on. - - name: Install cargo-rail - uses: taiki-e/[email protected] + # Planning is an optimization. A failure falls back to refreshing every + # Docker image in the component registry. + - name: Plan affected edge images + id: rail + continue-on-error: true + uses: loadingalias/cargo-rail-action@4a8df181e4f340a164190cf092c9f9239b20edab # v10.0.1 with: - tool: [email protected] + version: 0.29.0 - name: Check all components id: check + env: + PLAN_FILE: ${{ steps.rail.outputs.plan-file }} + PLAN_STATUS: ${{ steps.rail.outcome }} run: | - chmod +x scripts/extract-version.sh + set -euo pipefail + + COMPONENTS_JSON=$(yq -o=json -I=0 '.components' .github/config/publish.yml) + ALL_DOCKER_COMPONENTS=$(jq -r \ + '[to_entries[] | select(.value.registry == "dockerhub") | .key] | join(",")' \ + <<< "$COMPONENTS_JSON") + + IMAGE_MATRIX=all Review Comment: nit: line 87 overwrites this `all` default with the selector's stdout even when it exits non-zero. fail-open still holds because the empty value fails both gates, but reset `IMAGE_MATRIX=all` in the `else` branch. ########## .github/actions/rust/pre-merge/action.yml: ########## @@ -272,44 +180,71 @@ runs: # (test-storage-compat) opt out of partitioning rather than producing # an out-of-range hash:N/3. TEST_PARTITIONS=3 - TASK="${{ inputs.task }}" - PARTITION_FLAG="" + PARTITION_ARGS=() if [[ "$TASK" =~ ^test-([0-9]+)$ ]]; then PARTITION_INDEX="${BASH_REMATCH[1]}" - PARTITION_FLAG="--partition hash:${PARTITION_INDEX}/${TEST_PARTITIONS}" + PARTITION_ARGS=(--partition "hash:${PARTITION_INDEX}/${TEST_PARTITIONS}") echo "::notice::Running test partition ${PARTITION_INDEX}/${TEST_PARTITIONS}" fi - # Read DAG-based affected crate filter (computed in earlier step) - NEXTEST_FILTER="" - PACKAGE_FLAGS="" - TEST_PACKAGE_FLAGS="" - TOTAL_CRATES="?" - PLUGIN_SUITES=0 - if [[ -f /tmp/nextest-filter.txt ]]; then - NEXTEST_FILTER=$(cat /tmp/nextest-filter.txt) - fi - if [[ -f /tmp/packages.txt ]]; then - PACKAGE_FLAGS=$(cat /tmp/packages.txt) - fi - if [[ -f /tmp/test-packages.txt ]]; then - TEST_PACKAGE_FLAGS=$(cat /tmp/test-packages.txt) - fi - if [[ -f /tmp/total-crates.txt ]]; then - TOTAL_CRATES=$(cat /tmp/total-crates.txt) + # Read only validated selectors through the v10 runtime. Any planner or + # selector failure restores the previous full-workspace behavior. + PLAN_OUTPUT_DIR="${RUNNER_TEMP:?RUNNER_TEMP is required}/cargo-rail-selectors-${TASK}" + mkdir -p "$PLAN_OUTPUT_DIR" + PLAN_USABLE=false + TEST_SCOPE=workspace + RUNTIME_SCOPE=workspace + TEST_CARGO_ARGS=() + TEST_TARGET_ARGS=() + TEST_PACKAGES=() + RUNTIME_PACKAGES=() + + if [[ "$PLAN_STATUS" == "success" ]] && + TEST_SCOPE=$(cargo-rail-action plan cargo-scope "$PLAN_FILE" cargo.test) && + RUNTIME_SCOPE=$(cargo-rail-action plan cargo-scope "$PLAN_FILE" runtime-artifacts) && + cargo-rail-action plan cargo-args "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-args" && + cargo-rail-action plan target-args "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-target-args" && + cargo-rail-action plan package-names "$PLAN_FILE" cargo.test > "$PLAN_OUTPUT_DIR/test-packages" && + cargo-rail-action plan package-names "$PLAN_FILE" runtime-artifacts > "$PLAN_OUTPUT_DIR/runtime-packages"; then + mapfile -d '' -t TEST_CARGO_ARGS < "$PLAN_OUTPUT_DIR/test-args" + mapfile -d '' -t TEST_TARGET_ARGS < "$PLAN_OUTPUT_DIR/test-target-args" + mapfile -d '' -t TEST_PACKAGES < "$PLAN_OUTPUT_DIR/test-packages" + mapfile -d '' -t RUNTIME_PACKAGES < "$PLAN_OUTPUT_DIR/runtime-packages" + PLAN_USABLE=true + else + TEST_SCOPE=workspace + RUNTIME_SCOPE=workspace + TEST_CARGO_ARGS=() + TEST_TARGET_ARGS=() + TEST_PACKAGES=() + RUNTIME_PACKAGES=() + echo "::warning::Cargo-Rail plan unavailable or invalid; running the full workspace" fi - if [[ -f /tmp/plugin-suites.txt ]]; then - PLUGIN_SUITES=$(cat /tmp/plugin-suites.txt) + + if [[ "$PLAN_USABLE" == true && "$TEST_SCOPE" == "skipped" ]]; then + echo "::notice::Cargo-Rail reports no affected Rust tests" + exit 0 fi - if [[ -n "$PACKAGE_FLAGS" ]]; then - # Each connector suite contributes its own `package(integration)`, so - # subtract them to keep the crate count a crate count. - TEST_CRATE_COUNT=$(( $(echo "$NEXTEST_FILTER" | grep -o 'package(' | wc -l) - PLUGIN_SUITES )) - BUILD_CRATE_COUNT=$(echo "$PACKAGE_FLAGS" | grep -o '\-p ' | wc -l) - echo "::notice::DAG-scoped: testing ${TEST_CRATE_COUNT} crates + ${PLUGIN_SUITES} connector suites, building ${BUILD_CRATE_COUNT} (cargo check/clippy cover full workspace separately)" + BUILD_ARGS=() + BUILD_SCOPE=workspace + if [[ "$PLAN_USABLE" == true && "$TEST_SCOPE" == "packages" && + ( "$RUNTIME_SCOPE" == "packages" || "$RUNTIME_SCOPE" == "skipped" ) ]]; then + declare -A SEEN_BUILD_PACKAGES=() Review Comment: simplification: three structures track one package set here. collapse to one `mapfile` with an `awk 'NF && !seen[$0]++'` dedupe - keep the dedupe, `iggy-cli` lands in both lists and line 370 reads the count. ########## .github/workflows/post-merge.yml: ########## @@ -60,28 +60,56 @@ jobs: chmod +x /usr/local/bin/yq fi - # cargo-rail computes the affected crate set for the Docker :edge gate. - # Metadata-only `cargo rail plan` (no compile), so it runs on the runner's - # preinstalled cargo (version pinned by rust-toolchain.toml) and skips the - # heavyweight build-cache restore. - # Pinned for the same reason as .github/actions/rust/pre-merge/action.yml: - # 0.24 dropped the `-f json` edge-affected-images.sh relies on. - - name: Install cargo-rail - uses: taiki-e/[email protected] + # Planning is an optimization. A failure falls back to refreshing every + # Docker image in the component registry. + - name: Plan affected edge images + id: rail + continue-on-error: true + uses: loadingalias/cargo-rail-action@4a8df181e4f340a164190cf092c9f9239b20edab # v10.0.1 with: - tool: [email protected] + version: 0.29.0 - name: Check all components id: check + env: + PLAN_FILE: ${{ steps.rail.outputs.plan-file }} + PLAN_STATUS: ${{ steps.rail.outcome }} run: | - chmod +x scripts/extract-version.sh + set -euo pipefail + + COMPONENTS_JSON=$(yq -o=json -I=0 '.components' .github/config/publish.yml) + ALL_DOCKER_COMPONENTS=$(jq -r \ + '[to_entries[] | select(.value.registry == "dockerhub") | .key] | join(",")' \ + <<< "$COMPONENTS_JSON") + + IMAGE_MATRIX=all + if [[ "$PLAN_STATUS" == "success" ]] && + IMAGE_MATRIX=$(cargo-rail-action plan matrix "$PLAN_FILE" edge-images); then + echo "::notice::Using validated Cargo-Rail edge image plan" + else + echo "::warning::Cargo-Rail plan unavailable or invalid; refreshing all edge images" + fi + + if [[ "$IMAGE_MATRIX" == "all" ]]; then + DOCKER_COMPONENTS="$ALL_DOCKER_COMPONENTS" + elif DOCKER_COMPONENTS=$(jq -er --argjson configured "$COMPONENTS_JSON" ' + [.include[].component] as $selected + | if ($selected | all(. as $component | $configured[$component].registry == "dockerhub")) + then $selected | join(",") + else error("Cargo-Rail selected an unknown Docker component") + end + ' <<< "$IMAGE_MATRIX"); then + : Review Comment: simplification: the `elif` body is a bare `:`. negate it instead - `elif ! DOCKER_COMPONENTS=$(jq -er ...)` - and keep the warning body. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
