potiuk commented on code in PR #76955:
URL: https://github.com/apache/airflow/pull/76955#discussion_r4577034168


##########
dev/breeze/src/airflow_breeze/utils/fork_compare.py:
##########
@@ -68,18 +179,39 @@ def get_fork_changes(*, fork_owner: str, sha: str, base: 
str) -> ForkChanges:
         capture_output=True,
         text=True,
     )
-    if result.returncode != 0:
+    compare_merge_base = None
+    if result.returncode == 0:
+        compare = json.loads(result.stdout or "{}")
+        entries = compare.get("files", [])
+        files = tuple(path for entry in entries for path in entry)
+        compare_merge_base = compare.get("merge_base")
+        if len(entries) < COMPARE_FILE_CAP:
+            return ForkChanges(files=files, truncated=False, 
merge_base=compare_merge_base)
+        console_print(
+            f"[warning]Compare API returned {len(entries)} files (capped at 
{COMPARE_FILE_CAP}), "
+            "falling back to local git diff to avoid truncation[/]"
+        )
+    else:
+        files = ()
         console_print(
             f"[warning]Could not compare {fork_owner}:{sha} with 
{APACHE_AIRFLOW_GITHUB_REPOSITORY}:{base}, "
-            f"so everything runs: {result.stderr.strip()}[/]"
+            f"trying local git diff: {result.stderr.strip()}[/]"
         )
-        return ForkChanges(files=(), truncated=True)
-    compare = json.loads(result.stdout or "{}")
-    entries = compare.get("files", [])
-    files = tuple(path for entry in entries for path in entry)
-    return ForkChanges(
-        files=files, truncated=len(entries) >= COMPARE_FILE_CAP, 
merge_base=compare.get("merge_base")
+
+    local_changes = _get_local_git_changes(
+        base=base,
+        sha=sha,
+        known_merge_base=compare_merge_base if result.returncode == 0 else 
None,
     )
+    if local_changes is not None:
+        console_print(
+            f"[info]Local git diff resolved {len(local_changes.files)} changed 
files "
+            f"(merge base: {local_changes.merge_base})[/]"
+        )
+        return local_changes

Review Comment:
   **compare-failure fallback can turn a rate-limited compare into a 
selective-checks crash**
   
   Before this change, a failed compare (rate limit, 5xx) returned 
`merge_base=None` and `truncated=True`, so everything ran and 
`SelectiveChecks._get_previous_file_content` read previous file versions with 
`git show`. Now the git fallback returns a `merge_base`, so 
`_get_previous_file_content` switches to `get_upstream_file`, which goes back 
to the same REST API and raises `UpstreamFileError` on anything but a 404. When 
the compare failed because of a rate limit, a change to any `pyproject.toml` 
(`pyproject_toml_changed`, `provider_dependency_bump`) now makes `breeze ci 
selective-check` exit 1 instead of running everything. The merge-base commit is 
already present locally once the fallback succeeds, so read previous versions 
from git in that case (`git show <merge_base>:<path>`), or keep 
`merge_base=None` when the compare call itself failed.
   
   *Flagged by both the primary and adversarial reviewers.*



##########
.github/workflows/ci-arm.yml:
##########
@@ -280,6 +280,46 @@ jobs:
           ref: ${{ github.sha }}
           fetch-depth: 2
           persist-credentials: false
+      - name: "Check clean integration with upstream target branch"

Review Comment:
   **step guarded by a push event in a workflow without a push trigger**
   
   `ci-arm.yml` triggers only on `schedule` and `workflow_dispatch`, so this 
step's `github.event_name == 'push'` condition is never true and the 40 lines 
never run. Fork CI runs `ci-amd.yml` (`Tests (AMD)`) only. Drop the step from 
`ci-arm.yml`; it also removes the duplicated script.



##########
.github/workflows/ci-amd.yml:
##########
@@ -404,6 +404,46 @@ jobs:
           ref: ${{ github.sha }}
           fetch-depth: 2
           persist-credentials: false
+      - name: "Check clean integration with upstream target branch"
+        if: github.event_name == 'push' && github.repository != 
'apache/airflow'
+        shell: bash
+        run: |
+          target_branch=$(grep -E '^AIRFLOW_BRANCH\s*=\s*' 
dev/breeze/src/airflow_breeze/branch_defaults.py | sed -E 's/.*"([^"]+)".*/\1/' 
|| echo "main")
+          target_branch="${target_branch:-main}"
+          fetched=0
+          for i in 1 2 3; do
+            if git fetch --no-tags --depth=100 
https://github.com/apache/airflow.git 
"${target_branch}:refs/remotes/upstream/${target_branch}"; then
+              fetched=1
+              break
+            fi
+            sleep "$i"
+          done
+          if [ "${fetched}" -eq 1 ] && git rev-parse --verify 
"refs/remotes/upstream/${target_branch}" >/dev/null 2>&1; then
+            merge_base=$(git merge-base 
"refs/remotes/upstream/${target_branch}" HEAD 2>/dev/null || true)
+            if [ -z "${merge_base}" ]; then
+              for i in 1 2 3; do
+                if git fetch --deepen=50 origin "${GITHUB_REF_NAME:-HEAD}"; 
then
+                  break
+                fi
+                sleep "$i"
+              done
+              merge_base=$(git merge-base 
"refs/remotes/upstream/${target_branch}" HEAD 2>/dev/null || true)
+            fi
+            if [ -n "${merge_base}" ]; then
+              conflict=0
+              git merge-tree --write-tree --merge-base "${merge_base}" HEAD 
"refs/remotes/upstream/${target_branch}" >/dev/null 2>&1 || conflict=$?
+              if [ "${conflict}" -eq 1 ]; then
+                echo "::error::Your branch has merge conflicts with 
apache/airflow:${target_branch}. Please rebase on upstream/${target_branch} 
before running fork CI."
+                exit 1
+              elif [ "${conflict}" -gt 1 ]; then
+                echo "::warning::Git merge-tree exited with error code 
${conflict} (not a merge conflict). Skipping conflict check."
+              fi
+            else
+              echo "::warning::Could not find merge-base with 
upstream/${target_branch}. Skipping conflict check."

Review Comment:
   **conflict check silently skips branches whose merge base is beyond the 
shallow fetches**
   
   The checkout has 2 commits of the branch, upstream is fetched 100 deep and 
the branch is deepened once by 50. A branch that forked more than ~100 upstream 
commits ago, or has more than ~50 commits of its own, gets no merge base and 
the check only warns. Those are the branches most likely to conflict, so the 
check misses many of the cases it is meant for. If that is an accepted limit, 
say so in the design document; otherwise deepen until a merge base is found (or 
`git fetch --shallow-since` the branch's fork point).
   
   *Flagged by both the primary and adversarial reviewers.*



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to