zozo123 commented on code in PR #74311:
URL: https://github.com/apache/airflow/pull/74311#discussion_r4308441067
##########
.github/workflows/finalize-tests.yml:
##########
@@ -149,15 +149,54 @@ jobs:
python: ${{ matrix.python-version }}
use-uv: ${{ inputs.use-uv }}
make-mnt-writeable-and-cleanup: true
+ - name: "Save constraints for the independent dependency report"
+ shell: bash
+ if: >-
+ github.ref == 'refs/heads/main' &&
+ (github.event_name == 'schedule' || github.event_name ==
'workflow_dispatch')
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ CONSTRAINTS_BRANCH: ${{ inputs.constraints-branch }}
+ PYTHON_VERSION: ${{ matrix.python-version }}
+ CONSTRAINTS_MODE: ${{ matrix.constraints-mode }}
+ run: |
+ mkdir -p files/dependency-report
+ REF_API="repos/apache/airflow/git/ref/heads/${CONSTRAINTS_BRANCH}"
+ CONSTRAINTS_SHA=$(gh api "${REF_API}" --jq '.object.sha')
+
FILE_API="repos/apache/airflow/contents/${CONSTRAINTS_MODE}-${PYTHON_VERSION}.txt"
+ gh api "${FILE_API}?ref=${CONSTRAINTS_SHA}" \
+ --jq '.content' | base64 --decode >
files/dependency-report/constraints.txt
Review Comment:
Both `gh api` calls are gone.
- On main schedule/dispatch canaries, the Deps step now runs
`constraints-version-check --save-constraints-to files/dependency-report`.
- Breeze resolves the constraints branch to a commit, downloads the file at
that commit through `download_constraints_file`, and saves both.
- A failure to save only warns, and the upload step is `continue-on-error`,
so the report path can't turn the canary red.
On branch tip vs `head_sha`:
- The saved constraints are exactly what the summary checked, and their
commit is recorded.
- The report summary shows both the source revision and the constraints
commit, and says that bounds that changed between the two can show up as
conflicts.
- Generating constraints for `head_sha` would change what the canary summary
itself checks, so I kept this to recording.
`download_constraints_file` still hard-codes `apache/airflow`. That is the
same path the summary already used, so `CONSTRAINTS_GITHUB_REPOSITORY` is still
not honoured there. Happy to fix that separately.
##########
dev/breeze/doc/ci/05_workflows.md:
##########
@@ -368,6 +368,35 @@ unprivileged, which constrains what can go on it:
When adding a job, reach for `ubuntu-slim` if it only shuffles metadata
around, and
`ubuntu-22.04` otherwise.
+### Dependency upgrade diagnostics
+
+Scheduled and manually dispatched AMD canaries on `main` keep the dependency
freshness
+summary in `finalize-tests.yml`, but run it with `--no-explain-why`. Tests,
constraint
+generation and validation, and image publication still run in the canary.
Release-branch
+and other run types retain their existing explanation behavior.
+
+After the canary completes, `dependency-upgrade-report.yml` runs the detailed
explanations
+independently, including when tests failed but the report inputs were saved.
It checks out
+the original source SHA and downloads the original image and saved constraints
by artifact
+ID from that run. It never substitutes the latest branch image or constraints.
PyPI release
+metadata is fetched when the report executes, so the available upgrade targets
can change
+between the summary and the explanations.
+
+The workflow has read-only repository permissions and runs only on
`workflow_run`, which
+cannot write to the default branch's GitHub Actions cache. Reports use the
saved image artifact.
Review Comment:
Reworded. The doc now says the workflow runs in the default branch's context
and could write that branch's cache, that what protects the cache is that no
step saves one, and that no caching should be added to this workflow.
##########
dev/breeze/tests/test_dependency_upgrade_report.py:
##########
@@ -0,0 +1,168 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import json
+import subprocess
+
+import pytest
+import yaml
+
+from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH
+
+NODE_HARNESS = """
+import { readFileSync } from 'node:fs';
+const input = JSON.parse(readFileSync(0, 'utf8'));
+process.env.SOURCE_RUN_ID = input.id;
+process.env.GITHUB_RUN_ATTEMPT = String(input.attempt);
+const outputs = {};
+const requests = [];
+const summary = {
+ addHeading() { return this; }, addLink() { return this; },
+ addRaw() { return this; }, async write() {}
+};
+const github = {
+ rest: { actions: {
+ async getWorkflowRun(args) { requests.push(args); return { data: input.run
}; },
+ listWorkflowRunArtifacts() {}
+ } },
+ async paginate(method, args) { requests.push(args); return input.artifacts; }
+};
+const core = { setOutput(key, value) { outputs[key] = value; }, summary,
notice() {} };
+const context = { repo: { owner: 'apache', repo: 'airflow' } };
+const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor;
+try {
+ await new AsyncFunction('github', 'context', 'core', input.script)(github,
context, core);
+ console.log(JSON.stringify({ outputs, requests }));
+} catch (error) {
+ console.log(JSON.stringify({ error: error.message, requests }));
+}
+"""
+
+
[email protected]
+def source_run():
+ return {
+ "head_repository": {"full_name": "apache/airflow"},
+ "head_branch": "main",
+ "path": ".github/workflows/ci-amd.yml",
+ "event": "schedule",
+ "status": "completed",
+ "head_sha": "a" * 40,
+ "html_url": "https://github.com/apache/airflow/actions/runs/123";,
+ }
+
+
[email protected]
+def source_artifacts():
+ return [
+ {
+ "id": 1,
+ "name":
"dependency-report-inputs-3.14-constraints-source-providers-true",
+ "expired": False,
+ },
+ {"id": 2, "name": "ci-image-save-v3-linux_amd64-3.14-main", "expired":
False},
+ ]
+
+
+def run_plan(source_run, source_artifacts, *, attempt=1, source_run_id="123"):
+ workflow = yaml.safe_load(
+ (AIRFLOW_ROOT_PATH /
".github/workflows/dependency-upgrade-report.yml").read_text()
+ )
+ script = workflow["jobs"]["plan"]["steps"][0]["with"]["script"]
+ result = subprocess.run(
+ ["node", "--input-type=module", "--eval", NODE_HARNESS],
Review Comment:
Added `requires_node = pytest.mark.skipif(shutil.which("node") is None,
...)` to the node-based tests. Without node, 4 tests pass and 27 are skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
