(airflow) branch disable-pushing-cache-temporarily created (now 58faedb875)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch disable-pushing-cache-temporarily
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at 58faedb875 Temporarily disable cache pushing from CI

No new revisions were added by this update.

(airflow) branch test-image-cache updated (c3666f8336 -> 11d5f27810)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch test-image-cache
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard c3666f8336 Test image check
 add 11d5f27810 Test image check

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (c3666f8336)
\
 N -- N -- N   refs/heads/test-image-cache (11d5f27810)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci.yml   | 24 
 .github/workflows/push-image-cache.yml | 12 ++--
 2 files changed, 18 insertions(+), 18 deletions(-)

(airflow) 01/01: Test image check

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch test-image-cache
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit c3666f8336cc7e49a0e3718329f3876ced877e0c
Author: Jarek Potiuk 
AuthorDate: Mon Mar 18 11:13:18 2024 +0100

Test image check
---
 .github/workflows/ci.yml | 870 +++
 .github/workflows/early-image-checks.yml | 252 -
 2 files changed, 561 insertions(+), 561 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c6c4cfd352..132b74586e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -261,438 +261,438 @@ jobs:
   include-success-outputs: ${{ 
needs.build-info.outputs.include-success-outputs }}
   debug-resources: ${{ needs.build-info.outputs.debug-resources }}
 
-  static-checks:
-timeout-minutes: 45
-name: "Static checks"
-runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
-needs: [build-info, wait-for-ci-images]
-env:
-  RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
-  PYTHON_MAJOR_MINOR_VERSION: 
"${{needs.build-info.outputs.default-python-version}}"
-  UPGRADE_TO_NEWER_DEPENDENCIES: "${{ 
needs.build-info.outputs.upgrade-to-newer-dependencies }}"
-if: >
-  needs.build-info.outputs.basic-checks-only == 'false' &&
-  needs.build-info.outputs.latest-versions-only != 'true'
-steps:
-  - name: "Cleanup repo"
-shell: bash
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
-  - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
-uses: actions/checkout@v4
-with:
-  persist-credentials: false
-  - name: Cleanup docker
-uses: ./.github/actions/cleanup-docker
-  - name: >
-  Prepare breeze & CI image: 
${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
-uses: ./.github/actions/prepare_breeze_and_image
-id: breeze
-  - name: Cache pre-commit envs
-uses: actions/cache@v4
-with:
-  path: ~/.cache/pre-commit
-  # yamllint disable-line rule:line-length
-  key: "pre-commit-${{steps.breeze.outputs.host-python-version}}-${{ 
hashFiles('.pre-commit-config.yaml') }}"
-  restore-keys: |
-pre-commit-${{steps.breeze.outputs.host-python-version}}-
-  - name: "Static checks"
-run: breeze static-checks --all-files --show-diff-on-failure --color 
always --initialize-environment
-env:
-  VERBOSE: "false"
-  SKIP: ${{ needs.build-info.outputs.skip-pre-commits }}
-  COLUMNS: "250"
-  SKIP_GROUP_OUTPUT: "true"
-  DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
-  RUFF_FORMAT: "github"
-
-  mypy:
-name: "MyPy tests"
-uses: ./.github/workflows/mypy.yml
-needs: [build-info, wait-for-ci-images]
-permissions:
-  contents: read
-  packages: read
-secrets: inherit
-with:
-  runs-on: ${{ needs.build-info.outputs.runs-on }}
-  mypy-folders: ${{ needs.build-info.outputs.mypy-folders }}
-  image-tag: ${{ needs.build-info.outputs.image-tag }}
-  default-python-version: ${{ 
needs.build-info.outputs.default-python-version }}
-  needs-mypy: ${{ needs.build-info.outputs.needs-mypy }}
-if: needs.build-info.outputs.latest-versions-only != 'true'
-
-  docs:
-name: "Docs"
-uses: ./.github/workflows/docs.yml
-needs: [build-info, wait-for-ci-images]
-permissions:
-  contents: read
-  packages: read
-secrets: inherit
-if: >
-  needs.build-info.outputs.docs-build == 'true' &&
-  needs.build-info.outputs.latest-versions-only != 'true'
-with:
-  runs-on: ${{ needs.build-info.outputs.runs-on }}
-  image-tag: ${{ needs.build-info.outputs.image-tag }}
-  default-branch: ${{ needs.build-info.outputs.default-branch }}
-  docs-list-as-string: ${{ needs.build-info.outputs.docs-list-as-string }}
-  default-python-version: ${{ 
needs.build-info.outputs.default-python-version }}
-  include-success-outputs: ${{ 
needs.build-info.outputs.include-success-outputs }}
-  canary-run: ${{ needs.build-info.outputs.canary-run }}
-
-  providers:
-name: "Provider checks"
-uses: ./.github/workflows/check-providers.yml
-needs: [build-info, wait-for-ci-images]
-permissions:
-  contents: read
-  packages: read
-secrets: inherit
-if: >
-  needs.build-info.outputs.skip-providers-tests != 'true' &&
-  needs.build-info.outputs.latest-versions-only != 'true'
-with:
-  runs-on: ${{ needs.build-info.outputs.runs-on }}
-  image-tag: ${{ needs.build-

(airflow) branch test-image-cache created (now c3666f8336)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch test-image-cache
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at c3666f8336 Test image check

This branch includes the following new commits:

 new c3666f8336 Test image check

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) branch main updated: Refactoring issue generation to common methods for easier maintainence (#38241)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new c27bc53009 Refactoring issue generation to common methods for easier 
maintainence (#38241)
c27bc53009 is described below

commit c27bc53009602e10d7ab5530a8cbefde7f93670a
Author: Amogh Desai 
AuthorDate: Mon Mar 18 12:41:34 2024 +0530

Refactoring issue generation to common methods for easier maintainence 
(#38241)
---
 .../commands/release_management_commands.py| 224 -
 1 file changed, 86 insertions(+), 138 deletions(-)

diff --git 
a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py 
b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index 12dbda1196..16bdac52b5 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -2253,75 +2253,9 @@ def generate_issue_content_helm_chart(
 excluded_pr_list: str,
 limit_pr_count: int | None,
 ):
-verbose = get_verbose()
-is_helm_chart = True
-from github import Github, Issue, PullRequest, UnknownObjectException
-
-PullRequestOrIssue = Union[PullRequest.PullRequest, Issue.Issue]
-if excluded_pr_list:
-excluded_prs = [int(pr) for pr in excluded_pr_list.split(",")]
-else:
-excluded_prs = []
-changes = get_changes(verbose, previous_release, current_release, 
is_helm_chart)
-change_prs = [change.pr for change in changes]
-prs = [pr for pr in change_prs if pr is not None and pr not in 
excluded_prs]
-
-g = Github(github_token)
-repo = g.get_repo("apache/airflow")
-pull_requests: dict[int, PullRequestOrIssue] = {}
-linked_issues: dict[int, list[Issue.Issue]] = defaultdict(lambda: [])
-users: dict[int, set[str]] = defaultdict(lambda: set())
-count_prs = limit_pr_count or len(prs)
-with Progress(console=get_console()) as progress:
-task = progress.add_task(f"Retrieving {count_prs} PRs ", 
total=count_prs)
-for pr_number in prs[:count_prs]:
-progress.console.print(
-f"Retrieving PR#{pr_number}: 
https://github.com/apache/airflow/pull/{pr_number};
-)
-
-pr: PullRequestOrIssue
-try:
-pr = repo.get_pull(pr_number)
-except UnknownObjectException:
-# Fallback to issue if PR not found
-try:
-pr = repo.get_issue(pr_number)  # (same fields as PR)
-except UnknownObjectException:
-get_console().print(f"[red]The PR #{pr_number} could not 
be found[/]")
-continue
-
-if pr.user.login == "dependabot[bot]":
-get_console().print(f"[yellow]Skipping PR #{pr_number} as it 
was created by dependabot[/]")
-continue
-# Ignore doc-only and skipped PRs
-label_names = [label.name for label in pr.labels]
-if "type:doc-only" in label_names or "changelog:skip" in 
label_names:
-continue
-
-pull_requests[pr_number] = pr
-# GitHub does not have linked issues in PR - but we quite 
rigorously add Fixes/Closes
-# Relate so we can find those from the body
-if pr.body:
-body = " ".join(pr.body.splitlines())
-linked_issue_numbers = {
-int(issue_match.group(1)) for issue_match in 
ISSUE_MATCH_IN_BODY.finditer(body)
-}
-for linked_issue_number in linked_issue_numbers:
-progress.console.print(
-f"Retrieving Linked issue PR#{linked_issue_number}: "
-
f"https://github.com/apache/airflow/issue/{linked_issue_number};
-)
-try:
-
linked_issues[pr_number].append(repo.get_issue(linked_issue_number))
-except UnknownObjectException:
-progress.console.print(
-f"Failed to retrieve linked issue 
#{linked_issue_number}: Unknown Issue"
-)
-users[pr_number].add(pr.user.login)
-for linked_issue in linked_issues[pr_number]:
-users[pr_number].add(linked_issue.user.login)
-progress.advance(task)
-print_issue_content(current_release, pull_requests, linked_issues, users, 
is_helm_chart)
+generate_issue_content(
+github_token, previous_release, current_release, excluded_pr_list, 
limit_pr_count, is_helm_chart=True
+)
 
 
 @release_management.command(
@@ -2365,75 +2299,9 @@ def 

(airflow) branch main updated: Fix typos in SQLAlchemySchema docstrings (#38238)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 5d5af74a61 Fix typos in SQLAlchemySchema docstrings (#38238)
5d5af74a61 is described below

commit 5d5af74a61cfd3e29bfa3b12daea2bd6950022fd
Author: Hussein Awala 
AuthorDate: Mon Mar 18 08:10:08 2024 +0100

Fix typos in SQLAlchemySchema docstrings (#38238)
---
 airflow/api_connexion/schemas/job_schema.py | 2 +-
 airflow/api_connexion/schemas/trigger_schema.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/airflow/api_connexion/schemas/job_schema.py 
b/airflow/api_connexion/schemas/job_schema.py
index d4f77eedf3..810c99e200 100644
--- a/airflow/api_connexion/schemas/job_schema.py
+++ b/airflow/api_connexion/schemas/job_schema.py
@@ -23,7 +23,7 @@ from airflow.jobs.job import Job
 
 
 class JobSchema(SQLAlchemySchema):
-"""Sla Miss Schema."""
+"""Job Schema."""
 
 class Meta:
 """Meta."""
diff --git a/airflow/api_connexion/schemas/trigger_schema.py 
b/airflow/api_connexion/schemas/trigger_schema.py
index 15d180a573..b867a9bb5a 100644
--- a/airflow/api_connexion/schemas/trigger_schema.py
+++ b/airflow/api_connexion/schemas/trigger_schema.py
@@ -23,7 +23,7 @@ from airflow.models import Trigger
 
 
 class TriggerSchema(SQLAlchemySchema):
-"""Sla Miss Schema."""
+"""Trigger Schema."""
 
 class Meta:
 """Meta."""

(airflow) branch main updated (80e60d7303 -> 7d69e27ba4)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 80e60d7303 Create GKEStartKueueJobOperator operator (#37477)
 add 7d69e27ba4 Optimize cache-building workflows (#38234)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci-image-build.yml |  3 +-
 .github/workflows/ci.yml |  2 +
 .github/workflows/early-image-checks.yml | 10 +++-
 .github/workflows/finalize-tests.yml | 11 +++--
 .github/workflows/prod-image-build.yml   |  7 ++-
 .github/workflows/push-image-cache.yml   | 82 +---
 6 files changed, 91 insertions(+), 24 deletions(-)

(airflow) branch main updated (c32d41d94d -> 80e60d7303)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from c32d41d94d Implement wait_until_job_complete parameter for 
KubernetesJobOperator (#37998)
 add 80e60d7303 Create GKEStartKueueJobOperator operator (#37477)

No new revisions were added by this update.

Summary of changes:
 .../google/cloud/operators/kubernetes_engine.py|  33 ++
 .../operators/cloud/kubernetes_engine.rst  |   8 ++
 .../cloud/operators/test_kubernetes_engine.py  | 121 +
 .../example_kubernetes_engine_kueue.py | 100 -
 4 files changed, 261 insertions(+), 1 deletion(-)

(airflow) branch main updated: Implement wait_until_job_complete parameter for KubernetesJobOperator (#37998)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new c32d41d94d Implement wait_until_job_complete parameter for 
KubernetesJobOperator (#37998)
c32d41d94d is described below

commit c32d41d94d428b8f70274a298158b97fac285045
Author: max <42827971+moiseen...@users.noreply.github.com>
AuthorDate: Mon Mar 18 01:12:15 2024 +0100

Implement wait_until_job_complete parameter for KubernetesJobOperator 
(#37998)
---
 .../providers/cncf/kubernetes/hooks/kubernetes.py  |  65 +++-
 airflow/providers/cncf/kubernetes/operators/job.py |  21 
 .../cncf/kubernetes/hooks/test_kubernetes.py   | 116 +
 .../cncf/kubernetes/operators/test_job.py  |  71 -
 4 files changed, 271 insertions(+), 2 deletions(-)

diff --git a/airflow/providers/cncf/kubernetes/hooks/kubernetes.py 
b/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
index c950e35044..349f886071 100644
--- a/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
+++ b/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
@@ -20,6 +20,7 @@ import contextlib
 import json
 import tempfile
 from functools import cached_property
+from time import sleep
 from typing import TYPE_CHECKING, Any, Generator
 
 import aiofiles
@@ -42,6 +43,13 @@ if TYPE_CHECKING:
 
 LOADING_KUBE_CONFIG_FILE_RESOURCE = "Loading Kubernetes configuration file 
kube_config from {}..."
 
+JOB_FINAL_STATUS_CONDITION_TYPES = {
+"Complete",
+"Failed",
+}
+
+JOB_STATUS_CONDITION_TYPES = JOB_FINAL_STATUS_CONDITION_TYPES | {"Suspended"}
+
 
 def _load_body_to_dict(body: str) -> dict:
 try:
@@ -504,7 +512,7 @@ class KubernetesHook(BaseHook, PodOperatorHookProtocol):
 return resp
 
 def get_job(self, job_name: str, namespace: str) -> V1Job:
-"""Get Job of specified name from Google Cloud.
+"""Get Job of specified name and namespace.
 
 :param job_name: Name of Job to fetch.
 :param namespace: Namespace of the Job.
@@ -512,6 +520,33 @@ class KubernetesHook(BaseHook, PodOperatorHookProtocol):
 """
 return self.batch_v1_client.read_namespaced_job(name=job_name, 
namespace=namespace, pretty=True)
 
+def get_job_status(self, job_name: str, namespace: str) -> V1Job:
+"""Get job with status of specified name and namespace.
+
+:param job_name: Name of Job to fetch.
+:param namespace: Namespace of the Job.
+:return: Job object
+"""
+return self.batch_v1_client.read_namespaced_job_status(
+name=job_name, namespace=namespace, pretty=True
+)
+
+def wait_until_job_complete(self, job_name: str, namespace: str, 
job_poll_interval: float = 10) -> V1Job:
+"""Block job of specified name and namespace until it is complete or 
failed.
+
+:param job_name: Name of Job to fetch.
+:param namespace: Namespace of the Job.
+:param job_poll_interval: Interval in seconds between polling the job 
status
+:return: Job object
+"""
+while True:
+self.log.info("Requesting status for the job '%s' ", job_name)
+job: V1Job = self.get_job_status(job_name=job_name, 
namespace=namespace)
+if self.is_job_complete(job=job):
+return job
+self.log.info("The job '%s' is incomplete. Sleeping for %i sec.", 
job_name, job_poll_interval)
+sleep(job_poll_interval)
+
 def list_jobs_all_namespaces(self) -> V1JobList:
 """Get list of Jobs from all namespaces.
 
@@ -527,6 +562,34 @@ class KubernetesHook(BaseHook, PodOperatorHookProtocol):
 """
 return self.batch_v1_client.list_namespaced_job(namespace=namespace, 
pretty=True)
 
+def is_job_complete(self, job: V1Job) -> bool:
+"""Check whether the given job is complete (with success or fail).
+
+:return: Boolean indicating that the given job is complete.
+"""
+if conditions := job.status.conditions:
+if final_condition_types := list(
+c for c in conditions if c.type in 
JOB_FINAL_STATUS_CONDITION_TYPES and c.status
+):
+s = "s" if len(final_condition_types) > 1 else ""
+self.log.info(
+"The job '%s' state%s: %s",
+job.metadata.name,
+s,
+", ".join(f"{c.type} at {c.last_transition_time}" for c in 
final_condition_types),
+)
+return True
+ 

(airflow) branch optimize-caching-workflows updated (c2750050c4 -> 2384d5bfcf)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch optimize-caching-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard c2750050c4 Optimize cache-building workflows
 add 2384d5bfcf Optimize cache-building workflows

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (c2750050c4)
\
 N -- N -- N   refs/heads/optimize-caching-workflows (2384d5bfcf)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci.yml | 2 ++
 .github/workflows/early-image-checks.yml | 5 +
 .github/workflows/finalize-tests.yml | 5 +
 .github/workflows/push-image-cache.yml   | 5 +
 4 files changed, 17 insertions(+)

(airflow) branch optimize-caching-workflows updated (0f976ad780 -> c2750050c4)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch optimize-caching-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 0f976ad780 Optimize cache-building workflows
 add c2750050c4 Optimize cache-building workflows

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (0f976ad780)
\
 N -- N -- N   refs/heads/optimize-caching-workflows (c2750050c4)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci-image-build.yml   | 3 ++-
 .github/workflows/prod-image-build.yml | 7 +--
 2 files changed, 7 insertions(+), 3 deletions(-)

(airflow) branch optimize-caching-workflows updated (dcc8d1c366 -> 0f976ad780)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch optimize-caching-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit dcc8d1c366 Optimize cache-building workflows
 add 0f976ad780 Optimize cache-building workflows

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (dcc8d1c366)
\
 N -- N -- N   refs/heads/optimize-caching-workflows (0f976ad780)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/push-image-cache.yml | 2 ++
 1 file changed, 2 insertions(+)

(airflow) branch fix-docker-cache-input-name updated (6b7ca1d2eb -> a646908798)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch fix-docker-cache-input-name
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 6b7ca1d2eb Fix docker-cache input name
 add a646908798 Fix docker-cache input name

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (6b7ca1d2eb)
\
 N -- N -- N   refs/heads/fix-docker-cache-input-name (a646908798)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci-image-build.yml   | 1 +
 .github/workflows/prod-image-build.yml | 1 +
 2 files changed, 2 insertions(+)

(airflow) 01/01: Fix docker-cache input name

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch fix-docker-cache-input-name
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 6b7ca1d2eb3828dd4ec4b43fcd0d4ec896bd7862
Author: Jarek Potiuk 
AuthorDate: Mon Mar 18 00:00:06 2024 +0100

Fix docker-cache input name

The docker-cache  after separation of workflow has been
wrongly referred to as cache-directive. This causes slower image
builds than could be.
---
 .github/workflows/ci-image-build.yml   | 2 +-
 .github/workflows/prod-image-build.yml | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/ci-image-build.yml 
b/.github/workflows/ci-image-build.yml
index 8d26c6d2c7..c3945251e9 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -139,7 +139,7 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
   --python "${{ matrix.python-version }}"
   --platform "linux/${{ inputs.platform }}"
 env:
-  DOCKER_CACHE: ${{ inputs.cache-directive }}
+  DOCKER_CACHE: ${{ inputs.docker-cache }}
   INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
   UPGRADE_TO_NEWER_DEPENDENCIES: ${{ 
inputs.upgrade-to-newer-dependencies }}
   GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/prod-image-build.yml 
b/.github/workflows/prod-image-build.yml
index 2d59ca45b7..aa4ffc84ce 100644
--- a/.github/workflows/prod-image-build.yml
+++ b/.github/workflows/prod-image-build.yml
@@ -180,7 +180,7 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
   --use-constraints-for-context-packages --python "${{ 
matrix.python-version }}"
 env:
   PUSH: ${{ inputs.push-image }}
-  DOCKER_CACHE: ${{ inputs.cache-directive }}
+  DOCKER_CACHE: ${{ inputs.docker-cache }}
   DEBIAN_VERSION: ${{ inputs.debian-version }}
   INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
   UPGRADE_TO_NEWER_DEPENDENCIES: ${{ 
inputs.upgrade-to-newer-dependencies }}
@@ -196,7 +196,7 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
   --use-constraints-for-context-packages --python "${{ 
matrix.python-version }}"
 env:
   PUSH: ${{ inputs.push-image }}
-  DOCKER_CACHE: ${{ inputs.cache-directive }}
+  DOCKER_CACHE: ${{ inputs.docker-cache }}
   DEBIAN_VERSION: ${{ inputs.debian-version }}
   INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
   UPGRADE_TO_NEWER_DEPENDENCIES: ${{ 
inputs.upgrade-to-newer-dependencies }}

(airflow) branch fix-docker-cache-input-name created (now 6b7ca1d2eb)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch fix-docker-cache-input-name
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at 6b7ca1d2eb Fix docker-cache input name

This branch includes the following new commits:

 new 6b7ca1d2eb Fix docker-cache input name

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) branch optimize-caching-workflows updated (7369a2aead -> dcc8d1c366)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch optimize-caching-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit 7369a2aead Optimize cache-building workflows
 add dcc8d1c366 Optimize cache-building workflows

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (7369a2aead)
\
 N -- N -- N   refs/heads/optimize-caching-workflows (dcc8d1c366)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:

(airflow) 01/01: Optimize cache-building workflows

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch optimize-caching-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 7369a2aeadbdcc71538a54b3751a967628ac8000
Author: Jarek Potiuk 
AuthorDate: Sun Mar 17 23:25:59 2024 +0100

Optimize cache-building workflows

The cache building workflow uses now smaller machines to build
the cache - and if the ARM images are used to build CI and PROD
images one after the other it exceeds the space available on those
machines.

This PR implements a few optimizations to speed up cache building
and related workflows:

It splits cache building to separate CI/PROD jobs - this way CI cache
building does not need airflow/provider package preparation (saves
a few minutes of build and space on the machine.

Also few related jobs could be easily run on public runners, rather than
self-hosted runners, which would save the self-hosted runners for
heavier load
---
 .github/workflows/early-image-checks.yml |  5 ++-
 .github/workflows/finalize-tests.yml |  6 +--
 .github/workflows/push-image-cache.yml   | 75 +---
 3 files changed, 65 insertions(+), 21 deletions(-)

diff --git a/.github/workflows/early-image-checks.yml 
b/.github/workflows/early-image-checks.yml
index 7e62003719..081efd6184 100644
--- a/.github/workflows/early-image-checks.yml
+++ b/.github/workflows/early-image-checks.yml
@@ -89,6 +89,7 @@ jobs:
   packages: write
 secrets: inherit
 with:
+  runs-on: ${{ inputs.runs-on }}
   cache-type: "Early"
   include-prod-images: "false"
   push-latest-images: "false"
@@ -104,7 +105,7 @@ jobs:
   check-that-image-builds-quickly:
 timeout-minutes: 5
 name: Check that image builds quickly
-runs-on: ${{ fromJSON(inputs.runs-on) }}
+runs-on: ["ubuntu-22.04"]
 env:
   UPGRADE_TO_NEWER_DEPENDENCIES: false
   PLATFORM: "linux/amd64"
@@ -135,7 +136,7 @@ jobs:
   contents: read
 timeout-minutes: 70
 name: Generate constraints ${{ inputs.python-versions-list-as-string }}
-runs-on: ${{ fromJSON(inputs.runs-on) }}
+runs-on: ["ubuntu-22.04"]
 env:
   PYTHON_VERSIONS: ${{ inputs.python-versions-list-as-string }}
   DEBUG_RESOURCES: ${{ inputs.debug-resources }}
diff --git a/.github/workflows/finalize-tests.yml 
b/.github/workflows/finalize-tests.yml
index 685849b075..cf6dccb15b 100644
--- a/.github/workflows/finalize-tests.yml
+++ b/.github/workflows/finalize-tests.yml
@@ -67,14 +67,13 @@ on:  # yamllint disable-line rule:truthy
 type: string
 jobs:
   update-constraints:
-runs-on: ${{ fromJSON(inputs.runs-on) }}
+runs-on: ["ubuntu-22.04"]
 timeout-minutes: 80
 name: "Update constraints"
 permissions:
   contents: write
   packages: read
 env:
-  RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
   DEBUG_RESOURCES: ${{ inputs.debug-resources}}
   PYTHON_VERSIONS: ${{ inputs.python-versions-list-as-string }}
   IMAGE_TAG: ${{ inputs.image-tag }}
@@ -130,6 +129,7 @@ jobs:
   packages: write
 secrets: inherit
 with:
+  runs-on: ${{ inputs.runs-on }}
   cache-type: "Regular"
   include-prod-images: "true"
   push-latest-images: "true"
@@ -144,7 +144,7 @@ jobs:
   summarize-warnings:
 timeout-minutes: 15
 name: "Summarize warnings"
-runs-on: ${{ fromJSON(inputs.runs-on) }}
+runs-on: ["ubuntu-22.04"]
 steps:
   - name: "Cleanup repo"
 shell: bash
diff --git a/.github/workflows/push-image-cache.yml 
b/.github/workflows/push-image-cache.yml
index ffcc399889..f086b704dd 100644
--- a/.github/workflows/push-image-cache.yml
+++ b/.github/workflows/push-image-cache.yml
@@ -20,6 +20,11 @@ name: Push image cache
 on:  # yamllint disable-line rule:truthy
   workflow_call:
 inputs:
+  runs-on:
+description: "The array of labels (in json form) determining type of 
the runner to use for the build."
+required: false
+default: '["self-hosted", "Linux", "X64"]'
+type: string
   cache-type:
 description: "Type of cache to push (Early / Regular)."
 required: true
@@ -67,7 +72,7 @@ on:  # yamllint disable-line rule:truthy
 jobs:
   push-ci-image-cache:
 name: "Push CI image cache"
-runs-on: ["self-hosted", "Linux", "X64"]
+runs-on: ${{ fromJSON(inputs.runs-on) }}
 strategy:
   fail-fast: false
   matrix:
@@ -101,27 +106,72 @@ jobs:
   - name: "Cleanup dist and context file"
 run: rm -fv ./dist/* ./docker-context-files/*
 if: inputs.include-prod-images == 'true'
+  - name: "Start ARM 

(airflow) branch optimize-caching-workflows created (now 7369a2aead)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch optimize-caching-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at 7369a2aead Optimize cache-building workflows

This branch includes the following new commits:

 new 7369a2aead Optimize cache-building workflows

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) branch optimize-cache-building-workflow created (now 4a49bd5d3f)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch optimize-cache-building-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at 4a49bd5d3f Fix typo in runs-on in update constraints and build image 
(#38232)

No new revisions were added by this update.

(airflow) branch use-common-image-workflows-in-pull-request-target updated (93046f57c1 -> e0f0044b92)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
use-common-image-workflows-in-pull-request-target
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 93046f57c1 Use common image build workflows in pull-request-target 
workflow
 add 4a49bd5d3f Fix typo in runs-on in update constraints and build image 
(#38232)
 add e0f0044b92 Use common image build workflows in pull-request-target 
workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (93046f57c1)
\
 N -- N -- N   
refs/heads/use-common-image-workflows-in-pull-request-target (e0f0044b92)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci-image-build.yml | 2 +-
 .github/workflows/finalize-tests.yml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

(airflow) branch main updated: Fix typo in runs-on in update constraints and build image (#38232)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 4a49bd5d3f Fix typo in runs-on in update constraints and build image 
(#38232)
4a49bd5d3f is described below

commit 4a49bd5d3fb50554b21d4bcbfb49395f4152af1f
Author: Jarek Potiuk 
AuthorDate: Sun Mar 17 21:37:02 2024 +0100

Fix typo in runs-on in update constraints and build image (#38232)

* Fix typo in runs-on in update constraints and build image

There is a typo that prevents update-constraints job to run because
it asks for string and not array of values.

* Update .github/workflows/ci-image-build.yml

Co-authored-by: Hussein Awala 

-

Co-authored-by: Hussein Awala 
---
 .github/workflows/ci-image-build.yml | 2 +-
 .github/workflows/finalize-tests.yml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/ci-image-build.yml 
b/.github/workflows/ci-image-build.yml
index 1b90e010fc..8d26c6d2c7 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -95,7 +95,7 @@ ${{ inputs.do-build == 'true' && 'Build' || 'Skip building' 
}} \
 CI ${{inputs.platform}} image\
 ${{matrix.python-version}}${{ inputs.do-build == 'true' && ':' || '' }}\
 ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
-runs-on: ${{fromJson(inputs.runs-on)}}
+runs-on: ${{ fromJSON(inputs.runs-on) }}
 env:
   BACKEND: sqlite
   DEFAULT_BRANCH: ${{ inputs.branch }}
diff --git a/.github/workflows/finalize-tests.yml 
b/.github/workflows/finalize-tests.yml
index c7ec9d0789..685849b075 100644
--- a/.github/workflows/finalize-tests.yml
+++ b/.github/workflows/finalize-tests.yml
@@ -67,7 +67,7 @@ on:  # yamllint disable-line rule:truthy
 type: string
 jobs:
   update-constraints:
-runs-on: ${{ inputs.runs-on }}
+runs-on: ${{ fromJSON(inputs.runs-on) }}
 timeout-minutes: 80
 name: "Update constraints"
 permissions:

(airflow) branch use-common-image-workflows-in-pull-request-target created (now 93046f57c1)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
use-common-image-workflows-in-pull-request-target
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at 93046f57c1 Use common image build workflows in pull-request-target 
workflow

This branch includes the following new commits:

 new 93046f57c1 Use common image build workflows in pull-request-target 
workflow

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) 01/01: Use common image build workflows in pull-request-target workflow

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch 
use-common-image-workflows-in-pull-request-target
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 93046f57c1f7f3bd4bd62d0242f37d5b3c4d1828
Author: Jarek Potiuk 
AuthorDate: Sun Mar 17 12:58:33 2024 +0100

Use common image build workflows in pull-request-target workflow

We left a little duplication of the code that has been used to
build images in "Build Images" workflow - that is run as
"Pull request target" workflow - mainly because of the security
concerns and the way how we are replacing the source for
actions, workflows and CI scripts from the incoming PRs with the
one coming from the target branch.

This change moves parts of the code that was used to replace the
scripts to the shared workflows and removes the composite actions
that we used in the past as common code.

As part of that change it turned out that there was a bug in
target-commit-sha usage for PRs coming from the forks where rather
than using the merge commit, we were using the original commit coming
from the fork. This was the reason why often we asked the users
to rebase their PRs where some new breaking changes were added in
the workflows or new dependencies added. With using merge commit,
we should be experiencing the "please rebase to take into account
the latest version of CI or Airflow" far less frequently.
---
 .github/actions/build-ci-images/action.yml   |  55 
 .github/actions/build-prod-images/action.yml | 108 ---
 .github/workflows/build-images.yml   | 192 ++-
 .github/workflows/ci-image-build.yml |  73 --
 .github/workflows/prod-image-build.yml   |  87 ++--
 5 files changed, 182 insertions(+), 333 deletions(-)

diff --git a/.github/actions/build-ci-images/action.yml 
b/.github/actions/build-ci-images/action.yml
deleted file mode 100644
index b743aad41d..00
--- a/.github/actions/build-ci-images/action.yml
+++ /dev/null
@@ -1,55 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#

-name: 'Build CI images'
-description: 'Build CI images'
-inputs:
-  python-version:
-description: 'Python version to use'
-required: true
-runs:
-  using: "composite"
-  steps:
-- name: "Install Breeze"
-  uses: ./.github/actions/breeze
-- name: "Check if dependencies are properly regenerated"
-  shell: bash
-  run: |
-pip install rich>=12.4.4 pyyaml
-python 
scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py
-  if: env.UPGRADE_TO_NEWER_DEPENDENCIES != 'false'
-- name: Login to ghcr.io
-  shell: bash
-  run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ 
github.actor }} --password-stdin
-- name: "Build & Push AMD64 CI images ${{ env.IMAGE_TAG }} ${{ 
inputs.python-version }}"
-  shell: bash
-  run: >
-breeze ci-image build --push --tag-as-latest --upgrade-on-failure
---python "${{ inputs.python-version }}"
-- name: "Source constraints: ${{ inputs.python-version }}"
-  shell: bash
-  run: >
-breeze release-management generate-constraints --python "${{ 
inputs.python-version }}"
---airflow-constraints-mode constraints-source-providers
-- name: "Upload constraint artifacts"
-  uses: actions/upload-artifact@v4
-  with:
-name: source-constraints-${{ inputs.python-version }}
-path: ./files/constraints-*/constraints-source-providers-*.txt
-retention-days: 7
-if-no-files-found: error
diff --git a/.github/actions/build-prod-images/action.yml 
b/.github/actions/build-prod-images/action.yml
deleted file mode 100644
index 38c79e11be..00
--- a/.github/actions/build-prod-images/action.yml
+++ /dev/null
@@ -1,108 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work 

(airflow) branch main updated: Fix wrong & push condition on building CI image (#38224)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new ea6087834e Fix wrong & push condition on building CI image (#38224)
ea6087834e is described below

commit ea6087834e18ce5d8df4d608931a7933dff64177
Author: Jarek Potiuk 
AuthorDate: Sun Mar 17 16:55:18 2024 +0100

Fix wrong & push condition on building CI image (#38224)
---
 .github/workflows/ci-image-build.yml | 19 ---
 1 file changed, 4 insertions(+), 15 deletions(-)

diff --git a/.github/workflows/ci-image-build.yml 
b/.github/workflows/ci-image-build.yml
index e8b54a083c..1b90e010fc 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -130,21 +130,9 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
   - name: Login to ghcr.io
 run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ 
github.actor }} --password-stdin
 if: inputs.do-build == 'true'
-  - name: "Build & Push ${{ inputs.platform }}:${{ inputs.image-tag }} ${{ 
matrix.python-version }}"
-shell: bash
-run: >
-  breeze ci-image build --push --tag-as-latest --image-tag "${{ 
inputs.image-tag }}"
-  --python "${{ matrix.python-version }}"
-  --platform "linux/${{ inputs.platform }}"
-env:
-  DOCKER_CACHE: ${{ inputs.cache-directive }}
-  INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
-  UPGRADE_TO_NEWER_DEPENDENCIES: ${{ 
inputs.upgrade-to-newer-dependencies }}
-  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-  USE_UV: ${{ inputs.use-uv }}
-  BUILDER: ${{ inputs.platform == 'amd64' && 'default' || 
'airflow_cache' }}
-if: inputs.do-build == 'true'
-  - name: "Build ${{ inputs.platform }}:${{ inputs.image-tag }} ${{ 
matrix.python-version }}"
+  - name: >
+  Build ${{ inputs.push-image == 'true' && ' & push ' || '' }}
+  ${{ inputs.platform }}:${{ matrix.python-version }}:${{ 
inputs.image-tag }}
 shell: bash
 run: >
   breeze ci-image build --tag-as-latest --image-tag "${{ 
inputs.image-tag }}"
@@ -157,6 +145,7 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
   GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
   USE_UV: ${{ inputs.use-uv }}
   BUILDER: ${{ inputs.platform == 'amd64' && 'default' || 
'airflow_cache' }}
+  PUSH: ${{ inputs.push-image }}
 if: inputs.do-build == 'true'
   - name: "Stop ARM instance"
 run: ./scripts/ci/images/ci_stop_arm_instance.sh

(airflow) branch main updated: Fix pip command in the local virtual env dag (#38226)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 07bd0891a3 Fix pip command in the local virtual env dag (#38226)
07bd0891a3 is described below

commit 07bd0891a384696ceec1d2251436cc23a546347d
Author: Elad Kalif <45845474+elad...@users.noreply.github.com>
AuthorDate: Sun Mar 17 17:15:24 2024 +0200

Fix pip command in the local virtual env dag (#38226)
---
 contributing-docs/07_local_virtualenv.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contributing-docs/07_local_virtualenv.rst 
b/contributing-docs/07_local_virtualenv.rst
index 1ded48c0fa..a7a984e47c 100644
--- a/contributing-docs/07_local_virtualenv.rst
+++ b/contributing-docs/07_local_virtualenv.rst
@@ -315,7 +315,7 @@ When you install airflow from sources using editable 
install, you can develop to
 of Airflow and providers, which is pretty convenient, because you can use the 
same environment for both.
 
 
-Running ``pipinstall -e .`` will install Airflow in editable mode, but all 
provider code will also be
+Running ``pip install -e .`` will install Airflow in editable mode, but all 
provider code will also be
 available in the same environment. However, most provider need some additional 
dependencies.
 
 You can install the dependencies of the provider you want to develop by 
installing airflow in editable

(airflow) branch fix-missing-condition-on-image-push updated (b5e51f0061 -> 65acde9ad5)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch fix-missing-condition-on-image-push
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit b5e51f0061 Fix wrong & push condition on building CI image
 add a0dd026fd4 Add missing condition in PROD image build (#38228)
 add 8ae9331b49 Remove breeze-python-version input and propagating it 
everywhere (#38227)
 add 65acde9ad5 Fix wrong & push condition on building CI image

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (b5e51f0061)
\
 N -- N -- N   refs/heads/fix-missing-condition-on-image-push 
(65acde9ad5)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/actions/breeze/action.yml  |  7 --
 .../actions/prepare_breeze_and_image/action.yml|  5 
 ...e-tests.yml => additional-prod-image-tests.yml} |  7 --
 .github/workflows/basic-tests.yml  |  9 +--
 .github/workflows/build-images.yml |  7 --
 .github/workflows/check-providers.yml  | 11 -
 .github/workflows/ci-image-build.yml   |  7 --
 .github/workflows/ci.yml   | 28 --
 .github/workflows/early-image-checks.yml   | 12 +++---
 .github/workflows/finalize-tests.yml   |  6 -
 .github/workflows/helm-tests.yml   |  9 ---
 .github/workflows/k8s-tests.yml|  7 --
 .github/workflows/mypy.yml |  7 --
 .github/workflows/prod-image-build.yml |  8 +--
 .github/workflows/prod-image-extra-checks.yml  |  8 ---
 .github/workflows/push-image-cache.yml |  7 --
 .github/workflows/run-unit-tests.yml   |  7 --
 .github/workflows/special-tests.yml| 13 --
 18 files changed, 14 insertions(+), 151 deletions(-)
 rename .github/workflows/{all-prod-image-tests.yml => 
additional-prod-image-tests.yml} (95%)

(airflow) branch main updated: Remove breeze-python-version input and propagating it everywhere (#38227)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 8ae9331b49 Remove breeze-python-version input and propagating it 
everywhere (#38227)
8ae9331b49 is described below

commit 8ae9331b490a75ed9eb32fe91402209035129a5b
Author: Jarek Potiuk 
AuthorDate: Sun Mar 17 16:06:55 2024 +0100

Remove breeze-python-version input and propagating it everywhere (#38227)

The `breeze-python-version` has been used in a number of places
where we wanted to make sure of reproducibility of prepared
artifacts (because Python 3.8 produces different tar files).

However that caused some problems with caching of breeze
environments - where some of them were using Python 3.8 and some
were using Python 3.9.

This PR fixes it by:

* making Python 3.9 default for all breeze installations in CI
* removing the breeze-python-version parameter and all its usages
  from all workflows

The tests of breeze (ib basic-tests.yml) are still run with the
default-python-version not with Python 3.9 (which currently is
Python 3.8) - so the risk that breeze will not work on Python
3.8 is low.

This will also synchronize itself in October, when Python 3.8 will
reach end-of-life and we will move to Python 3.9 as the default
version.
---
 .github/actions/breeze/action.yml   |  7 +--
 .github/actions/prepare_breeze_and_image/action.yml |  5 -
 .github/workflows/additional-prod-image-tests.yml   |  7 ---
 .github/workflows/basic-tests.yml   |  9 +
 .github/workflows/build-images.yml  |  7 ---
 .github/workflows/check-providers.yml   | 11 ---
 .github/workflows/ci-image-build.yml|  7 ---
 .github/workflows/ci.yml| 20 
 .github/workflows/early-image-checks.yml| 12 +++-
 .github/workflows/finalize-tests.yml|  6 --
 .github/workflows/helm-tests.yml|  9 -
 .github/workflows/k8s-tests.yml |  7 ---
 .github/workflows/mypy.yml  |  7 ---
 .github/workflows/prod-image-build.yml  |  7 ---
 .github/workflows/prod-image-extra-checks.yml   |  8 
 .github/workflows/push-image-cache.yml  |  7 ---
 .github/workflows/run-unit-tests.yml|  7 ---
 .github/workflows/special-tests.yml | 13 -
 18 files changed, 9 insertions(+), 147 deletions(-)

diff --git a/.github/actions/breeze/action.yml 
b/.github/actions/breeze/action.yml
index ef8a00fc31..427864e403 100644
--- a/.github/actions/breeze/action.yml
+++ b/.github/actions/breeze/action.yml
@@ -21,7 +21,10 @@ description: 'Sets up Python and Breeze'
 inputs:
   python-version:
 description: 'Python version to use'
-default: "3.8"
+# Version of Python used for reproducibility of the packages built
+# Python 3.8 tarfile produces different tarballs than Python 3.9+ tarfile 
that's why we are forcing
+# Python 3.9 for all release preparation commands to make sure that the 
tarballs are reproducible
+default: "3.9"
 outputs:
   host-python-version:
 description: Python version used in host
@@ -34,7 +37,7 @@ runs:
   with:
 python-version: ${{ inputs.python-version }}
 cache: 'pip'
-cache-dependency-path: ./dev/breeze/setup*
+cache-dependency-path: ./dev/breeze/pyproject.toml
 - name: Cache breeze
   uses: actions/cache@v4
   with:
diff --git a/.github/actions/prepare_breeze_and_image/action.yml 
b/.github/actions/prepare_breeze_and_image/action.yml
index 4288f60ec9..41aa17092d 100644
--- a/.github/actions/prepare_breeze_and_image/action.yml
+++ b/.github/actions/prepare_breeze_and_image/action.yml
@@ -19,9 +19,6 @@
 name: 'Prepare breeze && current python image'
 description: 'Installs breeze and pulls current python image'
 inputs:
-  python-version:
-description: 'Python version to use'
-default: "3.8"
   pull-image-type:
 description: 'Which image to pull'
 default: CI
@@ -35,8 +32,6 @@ runs:
 - name: "Install Breeze"
   uses: ./.github/actions/breeze
   id: breeze
-  with:
-python-version: ${{ inputs.python-version }}
 - name: Login to ghcr.io
   shell: bash
   run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ 
github.actor }} --password-stdin
diff --git a/.github/workflows/additional-prod-image-tests.yml 
b/.github/workflows/additional-prod-image-tests.yml
index b47acd25b4..8eb8699e6c 100644
--- a/.github/workflows/additional-prod-image-tests.yml
+++ b/.github/workflows/addition

(airflow-ci-infra) 01/01: Update to Ubuntu Jammy as base of our GitHub Runner AMI

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch update-to-jammy
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git

commit 7159e004468568360f4345e24af226d7f081704c
Author: Jarek Potiuk 
AuthorDate: Sun Mar 17 16:04:37 2024 +0100

Update to Ubuntu Jammy as base of our GitHub Runner AMI
---
 github-runner-ami/packer/files/docker.sh   |  26 +--
 .../packer/files/source-list-additions.sh  |  12 +-
 github-runner-ami/packer/files/yarn.key| 220 +
 github-runner-ami/packer/ubuntu2004.pkr.hcl|   8 +-
 4 files changed, 244 insertions(+), 22 deletions(-)

diff --git a/github-runner-ami/packer/files/docker.sh 
b/github-runner-ami/packer/files/docker.sh
index 73e4be9..d6ba481 100644
--- a/github-runner-ami/packer/files/docker.sh
+++ b/github-runner-ami/packer/files/docker.sh
@@ -22,26 +22,20 @@ set -exu -o pipefail
 for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; 
do sudo apt-get remove $pkg --assume-yes || true; done
 
 sudo apt-get update
-sudo apt-get install ca-certificates curl gnupg
+sudo apt-get install ca-certificates curl gnupg --assume-yes
 
+sudo apt-get update
+sudo apt-get install ca-certificates curl
 sudo install -m 0755 -d /etc/apt/keyrings
-curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor 
-o /etc/apt/keyrings/docker.gpg
-sudo chmod a+r /etc/apt/keyrings/docker.gpg
+sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o 
/etc/apt/keyrings/docker.asc
+sudo chmod a+r /etc/apt/keyrings/docker.asc
 
 echo \
-  "deb [arch="$(dpkg --print-architecture)" 
signed-by=/etc/apt/keyrings/docker.gpg] 
https://download.docker.com/linux/ubuntu \
-  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
+  "deb [arch=$(dpkg --print-architecture) 
signed-by=/etc/apt/keyrings/docker.asc] 
https://download.docker.com/linux/ubuntu \
+  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
   sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
-
 sudo apt-get update
 
-# Same version as MIN_VERSIONS in Breeze
-DOCKER_VERSION_STRING="5:24.0.7-1~ubuntu.20.04~focal"
-DOCKER_COMPOSE_VERSION_STRING="2.20.2-1~ubuntu.20.04~focal"
-DOCKER_BUILDX_VERSION_STRING="0.11.2-1~ubuntu.20.04~focal"
-sudo apt-get install \
-  "docker-ce=${DOCKER_VERSION_STRING}" \
-  "docker-ce-cli=${DOCKER_VERSION_STRING}" \
-  containerd.io \
-  "docker-buildx-plugin=${DOCKER_BUILDX_VERSION_STRING}" \
-  "docker-compose-plugin=${DOCKER_COMPOSE_VERSION_STRING}" --assume-yes 
--allow-downgrades
+
+# Install latest versions of docker - that gives better security and 
performance
+sudo apt-get install docker-ce docker-ce-cli containerd.io 
docker-buildx-plugin docker-compose-plugin --assume-yes
diff --git a/github-runner-ami/packer/files/source-list-additions.sh 
b/github-runner-ami/packer/files/source-list-additions.sh
index 6f8b0af..8b72354 100644
--- a/github-runner-ami/packer/files/source-list-additions.sh
+++ b/github-runner-ami/packer/files/source-list-additions.sh
@@ -17,7 +17,11 @@
 # specific language governing permissions and limitations
 # under the License.
 
-apt-key adv --recv-key 0x1646B01B86E50310 "1646B01B86E50310"
-echo "deb https://dl.yarnpkg.com/debian/ stable main" > 
/etc/apt/sources.list.d/yarn.list
-apt-key add /tmp/timber.key
-echo "deb https://repositories.timber.io/public/vector/deb/ubuntu focal main" 
> /etc/apt/sources.list.d/timber.list
+cp /tmp/yarn.key /etc/apt/keyrings/yarn.gpg
+sudo chmod a+r /etc/apt/keyrings/yarn.gpg
+
+cp /tmp/timber.key /etc/apt/keyrings/timber.gpg
+sudo chmod a+r /etc/apt/keyrings/timber.gpg
+
+echo "deb [signed-by=/etc/apt/keyrings/yarn.gpg] 
https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list
+echo "deb [signed-by=/etc/apt/keyrings/timber.gpg] 
https://repositories.timber.io/public/vector/deb/ubuntu jammy main" > 
/etc/apt/sources.list.d/timber.list
diff --git a/github-runner-ami/packer/files/yarn.key 
b/github-runner-ami/packer/files/yarn.key
new file mode 100644
index 000..509718b
--- /dev/null
+++ b/github-runner-ami/packer/files/yarn.key
@@ -0,0 +1,220 @@
+-BEGIN PGP PUBLIC KEY BLOCK-
+
+mQINBFf0j5oBEADS6cItqCbf4lOLICohq2aHqM5I1jsz3DC4ddIU5ONbKXP1t0wk
+FEUPRzd6m80cTo7Q02Bw7enh4J6HvM5XVBSSGKENP6XAsiOZnY9nkXlcQAPFRnCn
+CjEfoOPZ0cBKjn2IpIXXcC+7xh4p1yruBpOsCbT6BuzA+Nm9j4cpRjdRdWSSmdID
+TyMZClmYm/NIfCPduYvNZxZXhW3QYeieP7HIonhZSHVu/jauEUyHLVsieUIvAOJI
+cXYpwLlrw0yy4flHe1ORJzuA7EZ4eOWCuKf1PgowEnVSS7Qp7lksCuljtfXgWelB
+XGJlAMD90mMbsNpQPF8ywQ2wjECM8Q6BGUcQuGMDBtFihobb+ufJxpUOm4uDt0y4
+zaw+MVSi+a56+zvY0VmMGVyJstldPAcUlFYBDsfC9+zpzyrAqRY+qFWOT2tj29R5
+ZNYvUUjEmA/kXPNIwmEr4oj7PVjSTUSpwoKamFFE6Bbha1bzIHpdPIRYc6cEulp3
+dTOWfp+Cniiblp9gwz3HeXOWu7

(airflow-ci-infra) branch update-to-jammy created (now 7159e00)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch update-to-jammy
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git


  at 7159e00  Update to Ubuntu Jammy as base of our GitHub Runner AMI

This branch includes the following new commits:

 new 7159e00  Update to Ubuntu Jammy as base of our GitHub Runner AMI

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) branch main updated: Add missing condition in PROD image build (#38228)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new a0dd026fd4 Add missing condition in PROD image build (#38228)
a0dd026fd4 is described below

commit a0dd026fd482c3f2604260b1cbb2a172d426275d
Author: Jarek Potiuk 
AuthorDate: Sun Mar 17 14:14:35 2024 +0100

Add missing condition in PROD image build (#38228)

The "Regular" build requiring PROD image should skip image
verification in case it is not built in workflow.

Also "all-prod-image-tests" was waiting for a wrong condition
and the name of it was not the best. Fixed it in the same PR.
---
 .../{all-prod-image-tests.yml => additional-prod-image-tests.yml} | 0
 .github/workflows/ci.yml  | 8 
 .github/workflows/prod-image-build.yml| 1 +
 3 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/all-prod-image-tests.yml 
b/.github/workflows/additional-prod-image-tests.yml
similarity index 100%
rename from .github/workflows/all-prod-image-tests.yml
rename to .github/workflows/additional-prod-image-tests.yml
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c39565e92b..d9eac58da6 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -642,10 +642,10 @@ jobs:
   DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
 if: needs.build-info.outputs.in-workflow-build == 'false'
 
-  all-prod-image-tests:
-name: "All PROD image tests"
-needs: [build-info, build-prod-images, early-image-checks]
-uses: ./.github/workflows/all-prod-image-tests.yml
+  additional-prod-image-tests:
+name: "Additional PROD image tests"
+needs: [build-info, wait-for-prod-images, early-image-checks]
+uses: ./.github/workflows/additional-prod-image-tests.yml
 with:
   runs-on: ${{ needs.build-info.outputs.runs-on }}
   default-branch: ${{ needs.build-info.outputs.default-branch }}
diff --git a/.github/workflows/prod-image-build.yml 
b/.github/workflows/prod-image-build.yml
index 8dc2081b99..501b5b1b5b 100644
--- a/.github/workflows/prod-image-build.yml
+++ b/.github/workflows/prod-image-build.yml
@@ -214,3 +214,4 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
 run: >
   breeze prod-image verify --image-tag "${{ inputs.image-tag }}"
   --python "${{ matrix.python-version }}"
+if: inputs.do-build == 'true'

(airflow) branch fix-missing-condition-on-image-push updated (57d2a270d7 -> b5e51f0061)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch fix-missing-condition-on-image-push
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 57d2a270d7 Fix wrong & push condition on building CI image
 add b5e51f0061 Fix wrong & push condition on building CI image

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (57d2a270d7)
\
 N -- N -- N   refs/heads/fix-missing-condition-on-image-push 
(b5e51f0061)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci-image-build.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(airflow) 01/01: Fix wrong & push condition on building CI image

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch fix-missing-condition-on-image-push
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 57d2a270d747f7d0c9f57b9fd43236be3714c945
Author: Jarek Potiuk 
AuthorDate: Sun Mar 17 11:12:17 2024 +0100

Fix wrong & push condition on building CI image
---
 .github/workflows/ci-image-build.yml | 19 ---
 1 file changed, 4 insertions(+), 15 deletions(-)

diff --git a/.github/workflows/ci-image-build.yml 
b/.github/workflows/ci-image-build.yml
index 36d9653b5d..c1c986574a 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -137,21 +137,9 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
   - name: Login to ghcr.io
 run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ 
github.actor }} --password-stdin
 if: inputs.do-build == 'true'
-  - name: "Build & Push ${{ inputs.platform }}:${{ inputs.image-tag }} ${{ 
matrix.python-version }}"
-shell: bash
-run: >
-  breeze ci-image build --push --tag-as-latest --image-tag "${{ 
inputs.image-tag }}"
-  --python "${{ matrix.python-version }}"
-  --platform "linux/${{ inputs.platform }}"
-env:
-  DOCKER_CACHE: ${{ inputs.cache-directive }}
-  INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
-  UPGRADE_TO_NEWER_DEPENDENCIES: ${{ 
inputs.upgrade-to-newer-dependencies }}
-  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-  USE_UV: ${{ inputs.use-uv }}
-  BUILDER: ${{ inputs.platform == 'amd64' && 'default' || 
'airflow_cache' }}
-if: inputs.do-build == 'true'
-  - name: "Build ${{ inputs.platform }}:${{ inputs.image-tag }} ${{ 
matrix.python-version }}"
+  - name: >
+  Build ${{ inputs.push == 'true' && ' & push' || '' }}:${{ 
inputs.platform }}:${{ inputs.image-tag }}
+  ${{ matrix.python-version }}
 shell: bash
 run: >
   breeze ci-image build --tag-as-latest --image-tag "${{ 
inputs.image-tag }}"
@@ -164,6 +152,7 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
   GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
   USE_UV: ${{ inputs.use-uv }}
   BUILDER: ${{ inputs.platform == 'amd64' && 'default' || 
'airflow_cache' }}
+  PUSH: ${{ inputs.push-image }}
 if: inputs.do-build == 'true'
   - name: "Stop ARM instance"
 run: ./scripts/ci/images/ci_stop_arm_instance.sh

(airflow) branch fix-missing-condition-on-image-push created (now 57d2a270d7)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch fix-missing-condition-on-image-push
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at 57d2a270d7 Fix wrong & push condition on building CI image

This branch includes the following new commits:

 new 57d2a270d7 Fix wrong & push condition on building CI image

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) branch main updated: Extract early image checks as separate workflow (#38220)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 01635a021c Extract early image checks as separate workflow (#38220)
01635a021c is described below

commit 01635a021c1bafbf9478e6777a23f9c41dc671ac
Author: Jarek Potiuk 
AuthorDate: Sun Mar 17 07:21:19 2024 +0100

Extract early image checks as separate workflow (#38220)

Another (likely last) extraction of workflows from the main
ci.yml workflow.

It also moves the ARM CI build into the same early image workflow
as it could be run easly in the process.
---
 .github/workflows/all-prod-image-tests.yml |   2 +-
 .github/workflows/ci.yml   | 168 +++--
 .github/workflows/early-image-checks.yml   | 235 +
 .github/workflows/finalize-tests.yml   |  27 +---
 .github/workflows/push-image-cache.yml |   1 +
 5 files changed, 257 insertions(+), 176 deletions(-)

diff --git a/.github/workflows/all-prod-image-tests.yml 
b/.github/workflows/all-prod-image-tests.yml
index a2f0036442..b47acd25b4 100644
--- a/.github/workflows/all-prod-image-tests.yml
+++ b/.github/workflows/all-prod-image-tests.yml
@@ -46,7 +46,7 @@ on:  # yamllint disable-line rule:truthy
 required: true
 type: string
   docker-cache:
-description: "Whether to use Docker cache (true/false)"
+description: "Docker cache specification to build the image (registry, 
local, disabled)."
 required: true
 type: string
   canary-run:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c92087d43d..c39565e92b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -34,7 +34,6 @@ permissions:
   # changes. If it is for a fork, then the token is read-only anyway.
   packages: write
 env:
-  ANSWER: "yes"
   DB_RESET: "true"
   VERBOSE: "true"
   GITHUB_REPOSITORY: ${{ github.repository }}
@@ -172,70 +171,6 @@ jobs:
   PR_LABELS: ${{ steps.source-run-info.outputs.pr-labels }}
   GITHUB_CONTEXT: ${{ toJson(github) }}
 
-  # Push early BuildX cache to GitHub Registry in Apache repository, This 
cache does not wait for all the
-  # tests to complete - it is run very early in the build process for "main" 
merges in order to refresh
-  # cache using the current constraints. This will speed up cache refresh in 
cases when pyproject.toml
-  # changes or in case of Dockerfile changes. Failure in this step is not a 
problem (at most it will
-  # delay cache refresh. It does not attempt to upgrade to newer dependencies.
-  # We only push CI cache as PROD cache usually does not gain as much from 
fresh cache because
-  # it uses prepared airflow and provider packages that invalidate the cache 
anyway most of the time
-  push-early-buildx-cache-to-github-registry:
-name: Push Early Image Cache
-needs: [build-info, build-ci-images]
-uses: ./.github/workflows/push-image-cache.yml
-permissions:
-  contents: read
-  # This write is only given here for `push` events from "apache/airflow" 
repo. It is not given for PRs
-  # from forks. This is to prevent malicious PRs from creating images in 
the "apache/airflow" repo.
-  # For regular build for PRS this "build-prod-images" workflow will be 
skipped anyway by the
-  # "in-workflow-build" condition
-  packages: write
-secrets: inherit
-with:
-  cache-type: "Early"
-  include-prod-images: "false"
-  push-latest-images: "false"
-  image-tag: ${{ needs.build-info.outputs.image-tag }}
-  python-versions: ${{ needs.build-info.outputs.python-versions }}
-  branch: ${{ needs.build-info.outputs.default-branch }}
-  use-uv: "true"
-  breeze-python-version: ${{ 
needs.build-info.outputs.breeze-python-version }}
-  constraints-branch: ${{ 
needs.build-info.outputs.default-constraints-branch }}
-  docker-cache: ${{ needs.build-info.outputs.cache-directive }}
-if: needs.build-info.outputs.canary-run == 'true' && 
needs.build-info.outputs.default-branch == 'main'
-
-  # Check that after earlier cache push, breeze command will build quickly
-  check-that-image-builds-quickly:
-timeout-minutes: 5
-name: "Check that image builds quickly"
-runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
-needs:
-  - build-info
-  - push-early-buildx-cache-to-github-registry
-env:
-  RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
-  UPGRADE_TO_NEWER_DEPENDENCIES: false
-  PLATFORM: "linux/amd64"
-  PYTHON_MAJOR_MINOR_VERSION: 
"${{needs.build-info.outputs.default-python-version}}"
-  PYTHON_VERSION:

(airflow) branch extract-early-image-checks-workflow updated (c89261d242 -> 595902a458)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-early-image-checks-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard c89261d242 Extract early image checks as separate workflow
 add 595902a458 Extract early image checks as separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (c89261d242)
\
 N -- N -- N   refs/heads/extract-early-image-checks-workflow 
(595902a458)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/early-image-checks.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(airflow) branch extract-early-image-checks-workflow updated (fa1f08871e -> c89261d242)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-early-image-checks-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard fa1f08871e Extract early image checks as separate workflow
 add c89261d242 Extract early image checks as separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (fa1f08871e)
\
 N -- N -- N   refs/heads/extract-early-image-checks-workflow 
(c89261d242)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci.yml | 3 ---
 .github/workflows/early-image-checks.yml | 8 
 2 files changed, 4 insertions(+), 7 deletions(-)

(airflow) branch extract-early-image-checks-workflow updated (1731518fe4 -> fa1f08871e)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-early-image-checks-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit 1731518fe4 Extract early image checks as separate workflow
 add fa1f08871e Extract early image checks as separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (1731518fe4)
\
 N -- N -- N   refs/heads/extract-early-image-checks-workflow 
(fa1f08871e)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/push-image-cache.yml | 1 +
 1 file changed, 1 insertion(+)

(airflow) branch extract-early-image-checks-workflow updated (cf411afc50 -> 1731518fe4)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-early-image-checks-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard cf411afc50 Extract early image checks as separate workflow
 add 1731518fe4 Extract early image checks as separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (cf411afc50)
\
 N -- N -- N   refs/heads/extract-early-image-checks-workflow 
(1731518fe4)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(airflow) 01/01: Extract early image checks as separate workflow

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch extract-early-image-checks-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit cf411afc5016150fc71573354b547de562aa4a26
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 19:37:13 2024 +0100

Extract early image checks as separate workflow

Another (likely last) extraction of workflows from the main
ci.yml workflow.

It also moves the ARM CI build into the same early image workflow
as it could be run easly in the process.
---
 .github/workflows/all-prod-image-tests.yml |   2 +-
 .github/workflows/ci.yml   | 161 +++-
 .github/workflows/early-image-checks.yml   | 235 +
 .github/workflows/finalize-tests.yml   |  27 +---
 4 files changed, 254 insertions(+), 171 deletions(-)

diff --git a/.github/workflows/all-prod-image-tests.yml 
b/.github/workflows/all-prod-image-tests.yml
index a2f0036442..b47acd25b4 100644
--- a/.github/workflows/all-prod-image-tests.yml
+++ b/.github/workflows/all-prod-image-tests.yml
@@ -46,7 +46,7 @@ on:  # yamllint disable-line rule:truthy
 required: true
 type: string
   docker-cache:
-description: "Whether to use Docker cache (true/false)"
+description: "Docker cache specification to build the image (registry, 
local, disabled)."
 required: true
 type: string
   canary-run:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c92087d43d..0d0a4b2c4f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -172,70 +172,6 @@ jobs:
   PR_LABELS: ${{ steps.source-run-info.outputs.pr-labels }}
   GITHUB_CONTEXT: ${{ toJson(github) }}
 
-  # Push early BuildX cache to GitHub Registry in Apache repository, This 
cache does not wait for all the
-  # tests to complete - it is run very early in the build process for "main" 
merges in order to refresh
-  # cache using the current constraints. This will speed up cache refresh in 
cases when pyproject.toml
-  # changes or in case of Dockerfile changes. Failure in this step is not a 
problem (at most it will
-  # delay cache refresh. It does not attempt to upgrade to newer dependencies.
-  # We only push CI cache as PROD cache usually does not gain as much from 
fresh cache because
-  # it uses prepared airflow and provider packages that invalidate the cache 
anyway most of the time
-  push-early-buildx-cache-to-github-registry:
-name: Push Early Image Cache
-needs: [build-info, build-ci-images]
-uses: ./.github/workflows/push-image-cache.yml
-permissions:
-  contents: read
-  # This write is only given here for `push` events from "apache/airflow" 
repo. It is not given for PRs
-  # from forks. This is to prevent malicious PRs from creating images in 
the "apache/airflow" repo.
-  # For regular build for PRS this "build-prod-images" workflow will be 
skipped anyway by the
-  # "in-workflow-build" condition
-  packages: write
-secrets: inherit
-with:
-  cache-type: "Early"
-  include-prod-images: "false"
-  push-latest-images: "false"
-  image-tag: ${{ needs.build-info.outputs.image-tag }}
-  python-versions: ${{ needs.build-info.outputs.python-versions }}
-  branch: ${{ needs.build-info.outputs.default-branch }}
-  use-uv: "true"
-  breeze-python-version: ${{ 
needs.build-info.outputs.breeze-python-version }}
-  constraints-branch: ${{ 
needs.build-info.outputs.default-constraints-branch }}
-  docker-cache: ${{ needs.build-info.outputs.cache-directive }}
-if: needs.build-info.outputs.canary-run == 'true' && 
needs.build-info.outputs.default-branch == 'main'
-
-  # Check that after earlier cache push, breeze command will build quickly
-  check-that-image-builds-quickly:
-timeout-minutes: 5
-name: "Check that image builds quickly"
-runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
-needs:
-  - build-info
-  - push-early-buildx-cache-to-github-registry
-env:
-  RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
-  UPGRADE_TO_NEWER_DEPENDENCIES: false
-  PLATFORM: "linux/amd64"
-  PYTHON_MAJOR_MINOR_VERSION: 
"${{needs.build-info.outputs.default-python-version}}"
-  PYTHON_VERSION: ${{needs.build-info.outputs.default-python-version}}
-if: >
-  needs.build-info.outputs.canary-run == 'true'
-  && needs.build-info.outputs.default-branch == 'main'
-steps:
-  - name: "Cleanup repo"
-shell: bash
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
-  - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- 

(airflow) branch extract-early-image-checks-workflow created (now cf411afc50)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-early-image-checks-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at cf411afc50 Extract early image checks as separate workflow

This branch includes the following new commits:

 new cf411afc50 Extract early image checks as separate workflow

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) branch main updated (bf137f874b -> 8839e0a7dc)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from bf137f874b Update Hatchling to latest version (#38218)
 add 8839e0a7dc Extract finalization of tests to separate workflow (#38212)

No new revisions were added by this update.

Summary of changes:
 .github/actions/build-ci-images/action.yml |   3 +
 .github/actions/build-prod-images/action.yml   |   3 +
 .../actions/prepare_breeze_and_image/action.yml|   3 +
 .github/workflows/all-prod-image-tests.yml |  10 +-
 .github/workflows/ci-image-build.yml   |  25 ++-
 .github/workflows/ci.yml   | 167 ++---
 .github/workflows/finalize-tests.yml   | 198 +
 .github/workflows/k8s-tests.yml|   2 +
 .github/workflows/prod-image-build.yml |  11 +-
 .github/workflows/push-image-cache.yml |   5 +
 .github/workflows/release_dockerhub_image.yml  |   4 +-
 .github/workflows/run-unit-tests.yml   |   2 +-
 .../airflow_breeze/commands/ci_image_commands.py   |  24 +--
 .../commands/production_image_commands.py  |  22 ---
 dev/breeze/src/airflow_breeze/utils/registry.py|  61 ---
 .../ci_start_arm_instance_and_connect_to_docker.sh |  11 ++
 16 files changed, 290 insertions(+), 261 deletions(-)
 create mode 100644 .github/workflows/finalize-tests.yml
 delete mode 100644 dev/breeze/src/airflow_breeze/utils/registry.py

(airflow) branch main updated: Update Hatchling to latest version (#38218)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new bf137f874b Update Hatchling to latest version (#38218)
bf137f874b is described below

commit bf137f874bdb8ab1257211f13fce1cc00c85f399
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 21:11:08 2024 +0100

Update Hatchling to latest version (#38218)
---
 pyproject.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pyproject.toml b/pyproject.toml
index 2a26414bec..370e10d6d6 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -24,7 +24,7 @@
 requires = [
 "GitPython==3.1.42",
 "gitdb==4.0.11",
-"hatchling==1.22.1",
+"hatchling==1.22.2",
 "packaging==24.0",
 "pathspec==0.12.1",
 "pluggy==1.4.0",

(airflow) branch extract-finalization-of-tests-to-separate-workflow updated (a5c9fdb113 -> 457bb6fff3)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard a5c9fdb113 Extract finalization of tests to separate workflow
 add 457bb6fff3 Extract finalization of tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (a5c9fdb113)
\
 N -- N -- N   
refs/heads/extract-finalization-of-tests-to-separate-workflow (457bb6fff3)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:

(airflow) branch extract-finalization-of-tests-to-separate-workflow updated (bbf496566b -> a5c9fdb113)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit bbf496566b Extract finalization of tests to separate workflow
 add a5c9fdb113 Extract finalization of tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (bbf496566b)
\
 N -- N -- N   
refs/heads/extract-finalization-of-tests-to-separate-workflow (a5c9fdb113)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci-image-build.yml | 19 +++
 .github/workflows/finalize-tests.yml |  1 +
 2 files changed, 20 insertions(+)

(airflow) branch main updated: Fix broken link in README.md (#38216)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 7cac97467c Fix broken link in README.md (#38216)
7cac97467c is described below

commit 7cac97467c5450829d63282098351996dda3cf72
Author: Elad Kalif <45845474+elad...@users.noreply.github.com>
AuthorDate: Sat Mar 16 20:47:37 2024 +0200

Fix broken link in README.md (#38216)
---
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 502bf380d4..422168355a 100644
--- a/README.md
+++ b/README.md
@@ -506,7 +506,7 @@ fixed in time for the current release. Therefore, the 
milestone for issue is mor
 looked at, than promise it will be fixed in the version.
 
 More context and **FAQ** about the patchlevel release can be found in the
-[What goes into the next release](dev/WHAT_GOES_INTO_NEXT_RELEASE.md) document 
in the `dev` folder of the
+[What goes into the next release](dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md) 
document in the `dev` folder of the
 repository.
 
 ## Can I use the Apache Airflow logo in my presentation?

(airflow) branch extract-finalization-of-tests-to-separate-workflow updated (42b67528e0 -> bbf496566b)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 42b67528e0 Extract finalization of tests to separate workflow
 add bbf496566b Extract finalization of tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (42b67528e0)
\
 N -- N -- N   
refs/heads/extract-finalization-of-tests-to-separate-workflow (bbf496566b)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci.yml   | 10 ++
 .github/workflows/prod-image-build.yml |  4 
 2 files changed, 6 insertions(+), 8 deletions(-)

(airflow) branch extract-finalization-of-tests-to-separate-workflow updated (778c0f84f7 -> 42b67528e0)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 778c0f84f7 Extract finalization of tests to separate workflow
 add 42b67528e0 Extract finalization of tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (778c0f84f7)
\
 N -- N -- N   
refs/heads/extract-finalization-of-tests-to-separate-workflow (42b67528e0)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/actions/build-prod-images/action.yml |  3 +++
 .github/workflows/all-prod-image-tests.yml   | 10 --
 .github/workflows/ci.yml |  4 
 .github/workflows/k8s-tests.yml  |  2 ++
 .github/workflows/prod-image-build.yml   |  7 +--
 5 files changed, 22 insertions(+), 4 deletions(-)

(airflow) branch extract-finalization-of-tests-to-separate-workflow updated (276ce4b41f -> 778c0f84f7)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit 276ce4b41f Extract finalization of tests to separate workflow
 add 778c0f84f7 Extract finalization of tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (276ce4b41f)
\
 N -- N -- N   
refs/heads/extract-finalization-of-tests-to-separate-workflow (778c0f84f7)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/actions/build-ci-images/action.yml  | 2 +-
 .github/actions/prepare_breeze_and_image/action.yml | 2 +-
 .github/workflows/ci-image-build.yml| 5 ++---
 .github/workflows/ci.yml| 4 ++--
 .github/workflows/push-image-cache.yml  | 6 ++
 .github/workflows/release_dockerhub_image.yml   | 4 +---
 .github/workflows/run-unit-tests.yml| 2 +-
 7 files changed, 10 insertions(+), 15 deletions(-)

(airflow) branch extract-finalization-of-tests-to-separate-workflow updated (851cb0eae2 -> 276ce4b41f)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 851cb0eae2 Extract finalization of tests to separate workflow
 add a4977c14cf Extract production image tests to a separate workflow 
(#38210)
 add 4af69f9777 Extract integration tests to a spearate workflow (#38208)
 add 6c99fcc343 Fix coverage PEP 669 setting for tests (#38213)
 add 6ddf3ae0a8 Extract docs to a separate workflow (#38211)
 add 276ce4b41f Extract finalization of tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (851cb0eae2)
\
 N -- N -- N   
refs/heads/extract-finalization-of-tests-to-separate-workflow (276ce4b41f)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/actions/build-ci-images/action.yml |   3 +
 .github/actions/build-prod-images/action.yml   |   2 +-
 .../actions/prepare_breeze_and_image/action.yml|   3 +
 .github/workflows/all-prod-image-tests.yml | 158 
 .github/workflows/build-images.yml |   2 +-
 .github/workflows/ci-image-build.yml   |   5 +
 .github/workflows/ci.yml   | 424 +++--
 .github/workflows/docs.yml | 126 ++
 .github/workflows/finalize-tests.yml   |  25 ++
 .github/workflows/integration-tests.yml| 155 
 .github/workflows/prod-image-build.yml |   2 +-
 .github/workflows/push-image-cache.yml |   7 +
 .github/workflows/release_dockerhub_image.yml  |   6 +-
 .../airflow_breeze/commands/ci_image_commands.py   |  22 --
 .../commands/production_image_commands.py  |  22 --
 .../airflow_breeze/commands/testing_commands.py|   2 -
 dev/breeze/src/airflow_breeze/utils/registry.py|  61 ---
 .../ci_start_arm_instance_and_connect_to_docker.sh |  11 +
 scripts/in_container/run_ci_tests.sh   |  13 +
 19 files changed, 572 insertions(+), 477 deletions(-)
 create mode 100644 .github/workflows/all-prod-image-tests.yml
 create mode 100644 .github/workflows/docs.yml
 create mode 100644 .github/workflows/integration-tests.yml
 delete mode 100644 dev/breeze/src/airflow_breeze/utils/registry.py

(airflow) branch main updated: Extract docs to a separate workflow (#38211)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 6ddf3ae0a8 Extract docs to a separate workflow (#38211)
6ddf3ae0a8 is described below

commit 6ddf3ae0a874948c9d81aa5f0958b91ffd45233b
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 17:40:27 2024 +0100

Extract docs to a separate workflow (#38211)

Similarly to other extranctiosn - doc build can be separated to
a separate workflow, additionally, we can turn both docs and
spelling jobs into a single matrix job with some conditionals
easily.
---
 .github/workflows/ci.yml   | 122 ++-
 .github/workflows/docs.yml | 126 +
 2 files changed, 143 insertions(+), 105 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e65f3a94db..660ef62bae 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -453,111 +453,25 @@ jobs:
   needs-mypy: ${{ needs.build-info.outputs.needs-mypy }}
 if: needs.build-info.outputs.latest-versions-only != 'true'
 
-  build-docs:
-timeout-minutes: 60
-name: "Build docs"
-runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
-needs: [build-info, wait-for-ci-images]
-if: >
-  needs.build-info.outputs.docs-build == 'true' &&
-  needs.build-info.outputs.latest-versions-only != 'true'
-env:
-  RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
-  PYTHON_MAJOR_MINOR_VERSION: 
"${{needs.build-info.outputs.default-python-version}}"
-  INCLUDE_SUCCESS_OUTPUTS: 
"${{needs.build-info.outputs.include-success-outputs}}"
-steps:
-  - name: "Cleanup repo"
-shell: bash
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
-  - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
-uses: actions/checkout@v4
-with:
-  persist-credentials: false
-  - name: Cleanup docker
-uses: ./.github/actions/cleanup-docker
-  - name: >
-  Prepare breeze & CI image: 
${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
-uses: ./.github/actions/prepare_breeze_and_image
-  - uses: actions/cache@v4
-id: cache-doc-inventories
-with:
-  path: ./docs/_inventory_cache/
-  key: docs-inventory-${{ hashFiles('pyproject.toml;') }}
-  restore-keys: |
-docs-inventory-${{ hashFiles('pyproject.toml;') }}
-docs-inventory-
-  - name: "Build docs"
-run: >
-  breeze build-docs ${{ needs.build-info.outputs.docs-list-as-string 
}} --docs-only
-  - name: "Clone airflow-site"
-run: >
-  git clone https://github.com/apache/airflow-site.git 
${GITHUB_WORKSPACE}/airflow-site &&
-  echo "AIRFLOW_SITE_DIRECTORY=${GITHUB_WORKSPACE}/airflow-site" >> 
"$GITHUB_ENV"
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Publish docs"
-run: >
-  breeze release-management publish-docs
-  --override-versioned --run-in-parallel
-  ${{ needs.build-info.outputs.docs-list-as-string }}
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Generate back references for providers"
-run: breeze release-management add-back-references all-providers
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Generate back references for apache-airflow"
-run: breeze release-management add-back-references apache-airflow
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Generate back references for docker-stack"
-run: breeze release-management add-back-references docker-stack
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Generate back references for helm-chart"
-run: breeze release-management add-back-references helm-chart
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: Configure AWS credentials
-uses: 
aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a  
# v4.0.1
-if: needs.build-info.outputs.canary-run == 'true' && 
needs.build-info.outputs.default-branch == 'main'
-with:
-  aws-access-key-id: ${{ secrets.DOCS_AWS_ACCESS_KEY_ID }}
-  aws-secret-access-key: ${{ secrets.DOCS_AWS_SECRET_ACCESS_KEY }}
-  aws-region: eu-central-1
-  - name: "Upload documentation to AWS S3"
-if: needs.build-info.outputs.canary-run == 'true' && 
needs.build-info.outputs.def

(airflow) branch main updated (4af69f9777 -> 6c99fcc343)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 4af69f9777 Extract integration tests to a spearate workflow (#38208)
 add 6c99fcc343 Fix coverage PEP 669 setting for tests (#38213)

No new revisions were added by this update.

Summary of changes:
 dev/breeze/src/airflow_breeze/commands/testing_commands.py |  2 --
 scripts/in_container/run_ci_tests.sh   | 13 +
 2 files changed, 13 insertions(+), 2 deletions(-)

(airflow) branch main updated: Extract integration tests to a spearate workflow (#38208)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 4af69f9777 Extract integration tests to a spearate workflow (#38208)
4af69f9777 is described below

commit 4af69f9777264e98af6a602d3edf6d5dda031a25
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 17:09:12 2024 +0100

Extract integration tests to a spearate workflow (#38208)

Another step of ci workflow refactoring. This one extracts the
integration tests to a separate workflow, and turn the
integration tests there into a matrix strategy single job with
a bit more conditionals.
---
 .github/workflows/ci.yml| 145 +-
 .github/workflows/integration-tests.yml | 155 
 2 files changed, 176 insertions(+), 124 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 7dea803dd0..e65f3a94db 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -759,126 +759,26 @@ jobs:
   breeze-python-version: ${{ 
needs.build-info.outputs.breeze-python-version }}
 if: needs.build-info.outputs.run-tests == 'true'
 
-  tests-integration-postgres:
-timeout-minutes: 130
-name: Integration Tests Postgres
-runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
-needs: [build-info, wait-for-ci-images]
-env:
-  RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
-  PARALLEL_TEST_TYPES: 
"${{needs.build-info.outputs.parallel-test-types-list-as-string}}"
-  PR_LABELS: "${{needs.build-info.outputs.pull-request-labels}}"
-  FULL_TESTS_NEEDED: "${{needs.build-info.outputs.full-tests-needed}}"
-  DEBUG_RESOURCES: "${{needs.build-info.outputs.debug-resources}}"
-  BACKEND: "postgres"
-  ENABLE_COVERAGE: "${{needs.build-info.outputs.run-coverage}}"
-  PYTHON_MAJOR_MINOR_VERSION: 
"${{needs.build-info.outputs.default-python-version}}"
-  BACKEND_VERSION: "${{needs.build-info.outputs.default-postgres-version}}"
-  JOB_ID: "integration-postgres"
-  SKIP_PROVIDER_TESTS: "${{needs.build-info.outputs.skip-provider-tests}}"
-if: needs.build-info.outputs.run-tests == 'true'
-steps:
-  - name: "Cleanup repo"
-shell: bash
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
-  - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
-uses: actions/checkout@v4
-with:
-  persist-credentials: false
-  - name: Cleanup docker
-uses: ./.github/actions/cleanup-docker
-  - name: "Prepare breeze & CI image: 
${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{env.IMAGE_TAG}}"
-uses: ./.github/actions/prepare_breeze_and_image
-  - name: "Integration Tests Postgres: cassandra"
-run: |
-  breeze testing integration-tests --integration cassandra
-  breeze down
-if: needs.build-info.outputs.is-airflow-runner != 'true'
-  - name: "Integration Tests Postgres: mongo"
-run: |
-  breeze testing integration-tests --integration mongo
-  breeze down
-if: needs.build-info.outputs.is-airflow-runner != 'true'
-  - name: "Integration Tests Postgres: pinot"
-run: |
-  breeze testing integration-tests --integration pinot
-  breeze down
-if: needs.build-info.outputs.is-airflow-runner != 'true'
-  - name: "Integration Tests Postgres: celery"
-run: |
-  breeze testing integration-tests --integration celery
-  breeze down
-if: needs.build-info.outputs.is-airflow-runner != 'true'
-  - name: "Integration Tests Postgres: trino, kerberos"
-run: |
-  breeze testing integration-tests --integration trino --integration 
kerberos
-  breeze down
-if: needs.build-info.outputs.is-airflow-runner != 'true'
-  - name: "Integration Tests Postgres: Kafka"
-run: |
-  breeze testing integration-tests --integration kafka
-  breeze down
-if: needs.build-info.outputs.is-airflow-runner != 'true'
-  - name: "Integration Tests Postgres: Qdrant"
-run: breeze testing integration-tests --integration qdrant
-if: needs.build-info.outputs.is-airflow-runner == 'true'
-  - name: "Integration Tests Postgres: all-testable"
-run: breeze testing integration-tests --integration all-testable
-if: needs.build-info.outputs.is-airflow-runner == 'true'
-  - name: >
-  Post Tests success: Integration Postgres"
-uses: ./.github/actions/post_tests_success

(airflow) branch extract-docs-to-separate-workflow updated (56558dfa9a -> 0edeaafc13)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-docs-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit 56558dfa9a Extract docs to a separate workflow
 add a4977c14cf Extract production image tests to a separate workflow 
(#38210)
 add 0edeaafc13 Extract docs to a separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (56558dfa9a)
\
 N -- N -- N   refs/heads/extract-docs-to-separate-workflow 
(0edeaafc13)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/actions/build-prod-images/action.yml |   2 +-
 .github/workflows/all-prod-image-tests.yml   | 158 +++
 .github/workflows/build-images.yml   |   2 +-
 .github/workflows/ci.yml | 129 --
 .github/workflows/prod-image-build.yml   |   2 +-
 5 files changed, 183 insertions(+), 110 deletions(-)
 create mode 100644 .github/workflows/all-prod-image-tests.yml

(airflow) branch main updated: Extract production image tests to a separate workflow (#38210)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new a4977c14cf Extract production image tests to a separate workflow 
(#38210)
a4977c14cf is described below

commit a4977c14cf70b02dffb083ceae8fc527b61c2a08
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 16:57:49 2024 +0100

Extract production image tests to a separate workflow (#38210)

Another common part of the CI workflow that could be extracted to
a separate workflow. Those are all PROD image tests that can be
run in the same foldable section of the CI
---
 .github/actions/build-prod-images/action.yml |   2 +-
 .github/workflows/all-prod-image-tests.yml   | 158 +++
 .github/workflows/build-images.yml   |   2 +-
 .github/workflows/ci.yml | 125 +++--
 .github/workflows/prod-image-build.yml   |   2 +-
 5 files changed, 179 insertions(+), 110 deletions(-)

diff --git a/.github/actions/build-prod-images/action.yml 
b/.github/actions/build-prod-images/action.yml
index 0493c148c0..e50cc29e57 100644
--- a/.github/actions/build-prod-images/action.yml
+++ b/.github/actions/build-prod-images/action.yml
@@ -77,7 +77,7 @@ runs:
 name: source-constraints-${{ inputs.python-version }}
 path: ./docker-context-files
   if: ${{ inputs.build-provider-packages == 'true' }}
-- name: "Download constraints from the Generate & Verify build"
+- name: "Download constraints"
   uses: actions/download-artifact@v4
   with:
 name: constraints
diff --git a/.github/workflows/all-prod-image-tests.yml 
b/.github/workflows/all-prod-image-tests.yml
new file mode 100644
index 00..57ebd000d3
--- /dev/null
+++ b/.github/workflows/all-prod-image-tests.yml
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: All PROD image tests
+on:  # yamllint disable-line rule:truthy
+  workflow_call:
+inputs:
+  runs-on:
+description: "The array of labels (in json form) determining type of 
the runner to use for the build."
+required: false
+default: '["ubuntu-22.04"]'
+type: string
+  default-branch:
+description: "The default branch for the repository"
+required: true
+type: string
+  constraints-branch:
+description: "Branch used to construct constraints URL from."
+required: true
+type: string
+  image-tag:
+description: "Tag to set for the image"
+required: true
+type: string
+  upgrade-to-newer-dependencies:
+description: "Whether to upgrade to newer dependencies (true/false)"
+required: true
+type: string
+  chicken-egg-providers:
+description: "Whether to build chicken-egg provider packages in the 
same run (true/false)"
+required: true
+type: string
+  docker-cache:
+description: "Whether to use Docker cache (true/false)"
+required: true
+type: string
+  canary-run:
+description: "Whether to run the canary run (true/false)"
+required: true
+type: string
+  default-python-version:
+description: Which version of python should be used get CI image
+required: true
+type: string
+  breeze-python-version:
+description: >
+  Which version of python should be used to install Breeze (3.9 is 
minimum for reproducible builds)
+required: true
+type: string
+jobs:
+  prod-image-extra-checks-main:
+name: PROD image extra checks (main)
+uses: ./.github/workflows/prod-image-extra-checks.yml
+with:
+  python-versions: "[ '${{ inputs.default-python-version }}' ]"
+  branch: ${{ inputs.default-branch }}
+  image-tag: ${{ inputs.image-tag }}
+  build-provider-packages: ${{ inputs.default-branch == 'main' }}
+  upgrade-to-newer-depend

(airflow) branch extract-finalization-of-tests-to-separate-workflow updated (de8edb573b -> 851cb0eae2)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit de8edb573b Extract finalization of tests to separate workflow
 add 851cb0eae2 Extract finalization of tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (de8edb573b)
\
 N -- N -- N   
refs/heads/extract-finalization-of-tests-to-separate-workflow (851cb0eae2)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci-image-build.yml| 2 +-
 .github/workflows/ci.yml| 1 +
 dev/breeze/src/airflow_breeze/commands/ci_image_commands.py | 2 +-
 3 files changed, 3 insertions(+), 2 deletions(-)

(airflow) branch extract-docs-to-separate-workflow updated (cb9e596c9c -> 56558dfa9a)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-docs-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard cb9e596c9c Extract docs to a separate workflow
 add 56558dfa9a Extract docs to a separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (cb9e596c9c)
\
 N -- N -- N   refs/heads/extract-docs-to-separate-workflow 
(56558dfa9a)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:

(airflow) branch extract-finalization-of-tests-to-separate-workflow updated (43f59c4052 -> de8edb573b)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit 43f59c4052 Extract finalization of tests to separate workflow
 add de8edb573b Extract finalization of tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (43f59c4052)
\
 N -- N -- N   
refs/heads/extract-finalization-of-tests-to-separate-workflow (de8edb573b)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/finalize-tests.yml | 12 
 1 file changed, 12 deletions(-)

(airflow) branch extract-docs-to-separate-workflow updated (3adc8ff76e -> cb9e596c9c)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-docs-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 3adc8ff76e Extract docs to a separate workflow
 add cb9e596c9c Extract docs to a separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (3adc8ff76e)
\
 N -- N -- N   refs/heads/extract-docs-to-separate-workflow 
(cb9e596c9c)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/docs.yml | 19 +--
 1 file changed, 9 insertions(+), 10 deletions(-)

(airflow) 01/01: Extract finalization of tests to separate workflow

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 43f59c4052979b66654b8c5011eb9009449cd02c
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 14:28:18 2024 +0100

Extract finalization of tests to separate workflow

Finalization of the CI workflow is happening when all the regular
tests succeeded and consists of updating constrainst, refreshing
cache and building CI ARM images. All those jobs can be run in
a separate workflow - again nicely folded together.
---
 .github/workflows/ci.yml | 118 ++
 .github/workflows/finalize-tests.yml | 184 +++
 2 files changed, 193 insertions(+), 109 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 331782f1db..793d05ad50 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -1118,13 +1118,11 @@ jobs:
   ( needs.build-info.outputs.run-kubernetes-tests == 'true' ||
   needs.build-info.outputs.needs-helm-tests == 'true')
 
-  update-constraints:
+  finalize-tests:
 permissions:
   contents: write
   packages: write
-timeout-minutes: 80
-name: "Update constraints"
-runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+secrets: inherit
 needs:
   - build-info
   - build-docs
@@ -1141,115 +1139,17 @@ jobs:
   - tests-integration-postgres
   - tests-integration-mysql
   - generate-constraints
-env:
-  RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
-  DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
-  PYTHON_VERSIONS: ${{ 
needs.build-info.outputs.python-versions-list-as-string }}
-if: needs.build-info.outputs.upgrade-to-newer-dependencies != 'false'
-steps:
-  - name: "Cleanup repo"
-shell: bash
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
-  - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
-uses: actions/checkout@v4
-with:
-  # Needed to perform push action
-  persist-credentials: false
-  - name: Cleanup docker
-uses: ./.github/actions/cleanup-docker
-  - name: "Set constraints branch name"
-id: constraints-branch
-run: ./scripts/ci/constraints/ci_branch_constraints.sh >> 
${GITHUB_OUTPUT}
-  - name: Checkout ${{ steps.constraints-branch.outputs.branch }}
-uses: actions/checkout@v4
-with:
-  path: "constraints"
-  ref: ${{ steps.constraints-branch.outputs.branch }}
-  persist-credentials: true
-  fetch-depth: 0
-  - name: "Download constraints from the constraints generated by build CI 
image"
-uses: actions/download-artifact@v4
-with:
-  name: constraints
-  path: ./files
-  - name: "Diff in constraints for 
${{needs.build-info.outputs.python-versions}}"
-run: ./scripts/ci/constraints/ci_diff_constraints.sh
-  # only commit and push constraints in canary runs (main)
-  - name: "Commit changed constraint files for 
${{needs.build-info.outputs.python-versions}}"
-run: ./scripts/ci/constraints/ci_commit_constraints.sh
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Push changes"
-if: needs.build-info.outputs.canary-run == 'true'
-working-directory: "constraints"
-run:
-  git push
-  # Push BuildX cache to GitHub Registry in Apache repository, if all tests 
are successful and build
-  # is executed as result of direct push to "main" or one of the "vX-Y-test" 
branches
-  # It rebuilds all images using just-pushed constraints using buildx and 
pushes them to registry
-  # It will automatically check if a new python image was released and will 
pull the latest one if needed
-  push-buildx-cache-to-github-registry:
-name: Push Regular Image Cache
-needs: [build-info, update-constraints]
-uses: ./.github/workflows/push-image-cache.yml
-permissions:
-  contents: read
-  # This write is only given here for `push` events from "apache/airflow" 
repo. It is not given for PRs
-  # from forks. This is to prevent malicious PRs from creating images in 
the "apache/airflow" repo.
-  # For regular build for PRS this "build-prod-images" workflow will be 
skipped anyway by the
-  # "in-workflow-build" condition
-  packages: write
-secrets: inherit
+uses: ./.github/workflows/finalize-tests.yml
 with:
-  cache-type: "Regular"
-  include-prod-images: "true"
-  push-latest-images: "true"
- 

(airflow) branch extract-finalization-of-tests-to-separate-workflow created (now 43f59c4052)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
extract-finalization-of-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at 43f59c4052 Extract finalization of tests to separate workflow

This branch includes the following new commits:

 new 43f59c4052 Extract finalization of tests to separate workflow

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) branch extract-docs-to-separate-workflow updated (ae731d181b -> 3adc8ff76e)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-docs-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard ae731d181b Extract docs to a separate workflow
 add e9d3232895 Cleanup Airflow Configurations environment variables before 
run the tests (#38188)
 add 8660eef337 Migrating issue generation to breeze release-management 
(#38062)
 add 3adc8ff76e Extract docs to a separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (ae731d181b)
\
 N -- N -- N   refs/heads/extract-docs-to-separate-workflow 
(3adc8ff76e)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/check-providers.yml  |   4 +
 .github/workflows/docs.yml |   1 +
 .github/workflows/helm-tests.yml   |   5 +
 dev/README_RELEASE_AIRFLOW.md  |   3 +-
 dev/README_RELEASE_HELM_CHART.md   |   5 +-
 dev/breeze/doc/09_release_management_tasks.rst |  20 ++
 .../doc/images/output_release-management.svg   |  68 ++--
 .../doc/images/output_release-management.txt   |   2 +-
 ...ease-management_generate-issue-content-core.svg | 138 
 ...ease-management_generate-issue-content-core.txt |   1 +
 ...anagement_generate-issue-content-helm-chart.svg | 138 
 ...anagement_generate-issue-content-helm-chart.txt |   1 +
 .../output_setup_check-all-params-in-groups.svg|  50 +--
 .../output_setup_check-all-params-in-groups.txt|   2 +-
 .../output_setup_regenerate-command-images.svg |  58 ++--
 .../output_setup_regenerate-command-images.txt |   2 +-
 .../commands}/ISSUE_TEMPLATE.md.jinja2 |   0
 .../commands/release_management_commands.py| 358 -
 .../commands/release_management_commands_config.py |  26 ++
 dev/prepare_release_issue.py   | 323 ---
 tests/conftest.py  |  29 ++
 21 files changed, 825 insertions(+), 409 deletions(-)
 create mode 100644 
dev/breeze/doc/images/output_release-management_generate-issue-content-core.svg
 create mode 100644 
dev/breeze/doc/images/output_release-management_generate-issue-content-core.txt
 create mode 100644 
dev/breeze/doc/images/output_release-management_generate-issue-content-helm-chart.svg
 create mode 100644 
dev/breeze/doc/images/output_release-management_generate-issue-content-helm-chart.txt
 copy dev/{ => breeze/src/airflow_breeze/commands}/ISSUE_TEMPLATE.md.jinja2 
(100%)
 delete mode 100755 dev/prepare_release_issue.py

(airflow) branch main updated: Cleanup Airflow Configurations environment variables before run the tests (#38188)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new e9d3232895 Cleanup Airflow Configurations environment variables before 
run the tests (#38188)
e9d3232895 is described below

commit e9d3232895696e28ceb2373a7f9c3705c5aa7929
Author: Andrey Anshin 
AuthorDate: Sat Mar 16 16:36:01 2024 +0400

Cleanup Airflow Configurations environment variables before run the tests 
(#38188)
---
 tests/conftest.py | 29 +
 1 file changed, 29 insertions(+)

diff --git a/tests/conftest.py b/tests/conftest.py
index 133d78a32b..c1d846f8bc 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -19,6 +19,7 @@ from __future__ import annotations
 import json
 import os
 import platform
+import re
 import subprocess
 import sys
 import warnings
@@ -36,6 +37,34 @@ from itsdangerous import URLSafeSerializer
 
 assert "airflow" not in sys.modules, "No airflow module can be imported before 
these lines"
 
+# Clear all Environment Variables that might have side effect,
+# For example, defined in /files/airflow-breeze-config/variables.env
+_AIRFLOW_CONFIG_PATTERN = re.compile(r"^AIRFLOW__(.+)__(.+)$")
+_KEEP_CONFIGS_SETTINGS: dict[str, dict[str, set[str]]] = {
+# Keep always these configurations
+"always": {
+"database": {"sql_alchemy_conn"},
+"core": {"sql_alchemy_conn"},
+"celery": {"result_backend", "broker_url"},
+},
+# Keep per enabled integrations
+"celery": {"celery": {"*"}, "celery_broker_transport_options": {"*"}},
+"kerberos": {"kerberos": {"*"}},
+}
+_ENABLED_INTEGRATIONS = {e.split("_", 1)[-1].lower() for e in os.environ if 
e.startswith("INTEGRATION_")}
+_KEEP_CONFIGS: dict[str, set[str]] = {}
+for keep_settings_key in ("always", *_ENABLED_INTEGRATIONS):
+if keep_settings := _KEEP_CONFIGS_SETTINGS.get(keep_settings_key):
+for section, options in keep_settings.items():
+if section not in _KEEP_CONFIGS:
+_KEEP_CONFIGS[section] = options
+else:
+_KEEP_CONFIGS[section].update(options)
+for env_key in os.environ.copy():
+if m := _AIRFLOW_CONFIG_PATTERN.match(env_key):
+section, option = m.group(1).lower(), m.group(2).lower()
+if not (ko := _KEEP_CONFIGS.get(section)) or not ("*" in ko or option 
in ko):
+del os.environ[env_key]
 
 DEFAULT_WARNING_OUTPUT_PATH = Path("warnings.txt")
 

(airflow) branch extract-docs-to-separate-workflow updated (10ab00e997 -> ae731d181b)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-docs-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit 10ab00e997 Extract docs to a separate workflow
 add ae731d181b Extract docs to a separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (10ab00e997)
\
 N -- N -- N   refs/heads/extract-docs-to-separate-workflow 
(ae731d181b)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci.yml | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)

(airflow) 01/01: Extract docs to a separate workflow

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch extract-docs-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 10ab00e9975c859768365efc61c9d5d026cca2ac
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 13:21:27 2024 +0100

Extract docs to a separate workflow

Similarly to other extranctiosn - doc build can be separated to
a separate workflow, additionally, we can turn both docs and
spelling jobs into a single matrix job with some conditionals
easily.
---
 .github/workflows/ci.yml   | 112 
 .github/workflows/docs.yml | 126 +
 2 files changed, 137 insertions(+), 101 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 331782f1db..f8f1791a00 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -453,111 +453,21 @@ jobs:
   needs-mypy: ${{ needs.build-info.outputs.needs-mypy }}
 if: needs.build-info.outputs.latest-versions-only != 'true'
 
-  build-docs:
-timeout-minutes: 60
-name: "Build docs"
-runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
-needs: [build-info, wait-for-ci-images]
-if: >
-  needs.build-info.outputs.docs-build == 'true' &&
-  needs.build-info.outputs.latest-versions-only != 'true'
-env:
-  RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
-  PYTHON_MAJOR_MINOR_VERSION: 
"${{needs.build-info.outputs.default-python-version}}"
-  INCLUDE_SUCCESS_OUTPUTS: 
"${{needs.build-info.outputs.include-success-outputs}}"
-steps:
-  - name: "Cleanup repo"
-shell: bash
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
-  - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
-uses: actions/checkout@v4
-with:
-  persist-credentials: false
-  - name: Cleanup docker
-uses: ./.github/actions/cleanup-docker
-  - name: >
-  Prepare breeze & CI image: 
${{needs.build-info.outputs.default-python-version}}:${{env.IMAGE_TAG}}
-uses: ./.github/actions/prepare_breeze_and_image
-  - uses: actions/cache@v4
-id: cache-doc-inventories
-with:
-  path: ./docs/_inventory_cache/
-  key: docs-inventory-${{ hashFiles('pyproject.toml;') }}
-  restore-keys: |
-docs-inventory-${{ hashFiles('pyproject.toml;') }}
-docs-inventory-
-  - name: "Build docs"
-run: >
-  breeze build-docs ${{ needs.build-info.outputs.docs-list-as-string 
}} --docs-only
-  - name: "Clone airflow-site"
-run: >
-  git clone https://github.com/apache/airflow-site.git 
${GITHUB_WORKSPACE}/airflow-site &&
-  echo "AIRFLOW_SITE_DIRECTORY=${GITHUB_WORKSPACE}/airflow-site" >> 
"$GITHUB_ENV"
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Publish docs"
-run: >
-  breeze release-management publish-docs
-  --override-versioned --run-in-parallel
-  ${{ needs.build-info.outputs.docs-list-as-string }}
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Generate back references for providers"
-run: breeze release-management add-back-references all-providers
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Generate back references for apache-airflow"
-run: breeze release-management add-back-references apache-airflow
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Generate back references for docker-stack"
-run: breeze release-management add-back-references docker-stack
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: "Generate back references for helm-chart"
-run: breeze release-management add-back-references helm-chart
-if: needs.build-info.outputs.canary-run == 'true'
-  - name: Configure AWS credentials
-uses: 
aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a  
# v4.0.1
-if: needs.build-info.outputs.canary-run == 'true' && 
needs.build-info.outputs.default-branch == 'main'
-with:
-  aws-access-key-id: ${{ secrets.DOCS_AWS_ACCESS_KEY_ID }}
-  aws-secret-access-key: ${{ secrets.DOCS_AWS_SECRET_ACCESS_KEY }}
-  aws-region: eu-central-1
-  - name: "Upload documentation to AWS S3"
-if: needs.build-info.outputs.canary-run == 'true' && 
needs.build-info.outputs.default-branch == 'main'
-run: aws s3 sync --delete ./files/documentation 
s3://apache-airflow-docs
-
-  spellcheck-docs:
-timeout-minutes:

(airflow) branch extract-docs-to-separate-workflow created (now 10ab00e997)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-docs-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at 10ab00e997 Extract docs to a separate workflow

This branch includes the following new commits:

 new 10ab00e997 Extract docs to a separate workflow

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) branch main updated (0acd6b27af -> c79e7729ca)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 0acd6b27af templated fields logic checks for 
cloud_storage_transfer_service (#37519)
 add c79e7729ca Extract Kubernetes tests to a separate workflow (#38204)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci.yml   |  81 +++--
 .github/workflows/k8s-tests.yml| 127 +
 .../airflow_breeze/commands/kubernetes_commands.py |  14 +++
 dev/breeze/src/airflow_breeze/utils/registry.py|   4 +-
 4 files changed, 158 insertions(+), 68 deletions(-)
 create mode 100644 .github/workflows/k8s-tests.yml

(airflow) branch main updated: Restore Python 3.12 support for Databricks (#38207)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new aa75fbb2b8 Restore Python 3.12 support for Databricks (#38207)
aa75fbb2b8 is described below

commit aa75fbb2b849e1f43b04e2f73ccc834511ea01e5
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 12:45:18 2024 +0100

Restore Python 3.12 support for Databricks (#38207)

We excluded Python 3.12 from Databricks provider, because it was
failing our Python 3.12 tests intermittently (but often enough to
make a difference). It turned out that this was caused by running
the tests with coverage enabled and PEP 669 implementation in
Python 3.12 impacting intermittently performance of tests run
with coverage. However seems that experimenetal PEP 669 support
implemented in coverage 7.4.0 is nicely handling the performance
issues and after #38194 we shoudl be able to enable Python 3.12 for
Databricks without impacting our tests.

Related: https://github.com/databricks/databricks-sql-python/issues/369
---
 airflow/providers/databricks/provider.yaml |  6 --
 generated/provider_dependencies.json   |  4 +---
 pyproject.toml | 13 +++--
 3 files changed, 8 insertions(+), 15 deletions(-)

diff --git a/airflow/providers/databricks/provider.yaml 
b/airflow/providers/databricks/provider.yaml
index 6bc0bbfe87..ddc22d4a4c 100644
--- a/airflow/providers/databricks/provider.yaml
+++ b/airflow/providers/databricks/provider.yaml
@@ -60,12 +60,6 @@ versions:
   - 1.0.1
   - 1.0.0
 
-
-# Databricks is excluded for Python 3.12 because running databricks-sql-python 
imports on Python 3.12
-# Cause extremely long import times  
https://github.com/databricks/databricks-sql-python/issues/369
-# and until the problem is fixed, we exclude Python 3.12 for Databricks 
provider
-excluded-python-versions: ['3.12']
-
 dependencies:
   - apache-airflow>=2.6.0
   - apache-airflow-providers-common-sql>=1.10.0
diff --git a/generated/provider_dependencies.json 
b/generated/provider_dependencies.json
index 480ebc76fe..10d50d9bba 100644
--- a/generated/provider_dependencies.json
+++ b/generated/provider_dependencies.json
@@ -377,9 +377,7 @@
 "cross-providers-deps": [
   "common.sql"
 ],
-"excluded-python-versions": [
-  "3.12"
-],
+"excluded-python-versions": [],
 "state": "ready"
   },
   "datadog": {
diff --git a/pyproject.toml b/pyproject.toml
index 2fe574eaab..2a26414bec 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -346,7 +346,8 @@ devel-tests = [
 "aioresponses>=0.7.6",
 "backports.zoneinfo>=0.2.1;python_version<'3.9'",
 "beautifulsoup4>=4.7.1",
-"coverage>=7.2",
+# Coverage 7.4.0 added experimental support for Python 3.12 PEP669 which 
we use in Airflow
+"coverage>=7.4.0",
 "pytest-asyncio>=0.23.3",
 "pytest-cov>=4.1.0",
 "pytest-icdiff>=0.9",
@@ -665,12 +666,12 @@ common-sql = [ # source: 
airflow/providers/common/sql/provider.yaml
   "sqlparse>=0.4.2",
 ]
 databricks = [ # source: airflow/providers/databricks/provider.yaml
-  "aiohttp>=3.9.2, <4;python_version != \"3.12\"",
-  "apache-airflow[common_sql];python_version != \"3.12\"",
-  "databricks-sql-connector>=2.0.0, <3.0.0, !=2.9.0;python_version != 
\"3.12\"",
-  "requests>=2.27.0,<3;python_version != \"3.12\"",
+  "aiohttp>=3.9.2, <4",
+  "apache-airflow[common_sql]",
+  "databricks-sql-connector>=2.0.0, <3.0.0, !=2.9.0",
+  "requests>=2.27.0,<3",
   # Devel dependencies for the databricks provider
-  "deltalake>=0.12.0;python_version != \"3.12\"",
+  "deltalake>=0.12.0",
 ]
 datadog = [ # source: airflow/providers/datadog/provider.yaml
   "datadog>=0.14.0",

(airflow) branch main updated (3cff79bd25 -> 191036b737)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 3cff79bd25 Upgrade to latest build dependencies (#38205)
 add 191036b737 Extract basic tests to separate workflow (#38206)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/basic-tests.yml | 329 ++
 .github/workflows/ci.yml  | 290 ++---
 2 files changed, 343 insertions(+), 276 deletions(-)
 create mode 100644 .github/workflows/basic-tests.yml

(airflow) branch main updated (baa6f08a5c -> 3cff79bd25)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from baa6f08a5c Rename `SparkSubmitOperator`'s fields' names to comply with 
templated fields validation (#38051)
 add 3cff79bd25 Upgrade to latest build dependencies (#38205)

No new revisions were added by this update.

Summary of changes:
 Dockerfile | 2 +-
 Dockerfile.ci  | 4 ++--
 pyproject.toml | 5 ++---
 3 files changed, 5 insertions(+), 6 deletions(-)

(airflow) branch extract-basic-tests-to-separate-workflow updated (0901abdd4b -> fa42a8e21f)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-basic-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 0901abdd4b Extract basic tests to separate workflow
 add fa42a8e21f Extract basic tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (0901abdd4b)
\
 N -- N -- N   refs/heads/extract-basic-tests-to-separate-workflow 
(fa42a8e21f)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/basic-tests.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(airflow) branch extract-basic-tests-to-separate-workflow updated (ceab36d28e -> 0901abdd4b)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-basic-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard ceab36d28e Extract basic tests to separate workflow
 add 0901abdd4b Extract basic tests to separate workflow

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (ceab36d28e)
\
 N -- N -- N   refs/heads/extract-basic-tests-to-separate-workflow 
(0901abdd4b)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/basic-tests.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(airflow) branch extract-basic-tests-to-separate-workflow created (now ceab36d28e)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch extract-basic-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at ceab36d28e Extract basic tests to separate workflow

This branch includes the following new commits:

 new ceab36d28e Extract basic tests to separate workflow

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) 01/01: Extract basic tests to separate workflow

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch extract-basic-tests-to-separate-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit ceab36d28e450d4a963bb678b444be6b88148594
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 11:01:28 2024 +0100

Extract basic tests to separate workflow

Similarly as with other extractions - this one will extract all
the tests that need just basic "build-info" dependencies to a
separate workflow. This will allow to keep the ci.yml better
organised - eventually consisting of just calling other workflows.

It also assigns all the basic checks to be run on the public
ASF runners. There is no point in using our self-hosted runners
to run those tests, they are usually pretty fast even on a small
public runner (and they are not on a critical path for elapsed time
of the workflow run - no other tests depend on them).
---
 .github/workflows/basic-tests.yml | 329 ++
 .github/workflows/ci.yml  | 290 ++---
 2 files changed, 343 insertions(+), 276 deletions(-)

diff --git a/.github/workflows/basic-tests.yml 
b/.github/workflows/basic-tests.yml
new file mode 100644
index 00..03d282bb01
--- /dev/null
+++ b/.github/workflows/basic-tests.yml
@@ -0,0 +1,329 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Basic tests
+on:  # yamllint disable-line rule:truthy
+  workflow_call:
+inputs:
+  runs-www-tests:
+description: "Whether to run WWW tests (true/false)"
+required: true
+type: string
+  needs-api-codegen:
+description: "Whether to run API codegen (true/false)"
+required: true
+type: string
+  basic-checks-only:
+description: "Whether to run only basic checks (true/false)"
+required: true
+type: string
+  skip-pre-commits:
+description: "Whether to skip pre-commits (true/false)"
+required: true
+type: string
+  default-python-version:
+description: Which version of python should be used get CI image
+required: true
+type: string
+  breeze-python-version:
+description: >
+  Which version of python should be used to install Breeze (3.9 is 
minimum for reproducible builds)
+required: true
+type: string
+  canary-run:
+description: "Whether to run canary tests (true/false)"
+required: true
+type: string
+  latest-versions-only:
+description: "Whether to run only latest version checks (true/false)"
+required: true
+type: string
+jobs:
+  run-breeze-tests:
+timeout-minutes: 10
+name: Breeze unit tests
+runs-on: ["ubuntu-22.04"]
+steps:
+  - name: "Cleanup repo"
+shell: bash
+run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
+  - uses: actions/checkout@v4
+with:
+  # Need to fetch all history for selective checks tests
+  fetch-depth: 0
+  persist-credentials: false
+  - name: Cleanup docker
+uses: ./.github/actions/cleanup-docker
+  - uses: actions/setup-python@v5
+with:
+  python-version: "${{ inputs.default-python-version }}"
+  cache: 'pip'
+  cache-dependency-path: ./dev/breeze/setup*
+  - run: pip install --editable ./dev/breeze/
+  - run: python -m pytest -n auto --color=yes
+working-directory: ./dev/breeze/
+
+  tests-www:
+timeout-minutes: 10
+name: React WWW tests
+runs-on: ["ubuntu-22.04"]
+if: inputs.run-www-tests == 'true'
+steps:
+  - name: "Cleanup repo"
+shell: bash
+run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
+  - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+uses: actions/checkout@v4
+with:
+  persist

(airflow) branch main updated (63bc63c190 -> e30c93a9be)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 63bc63c190 Better diangostics for parallel test execution in CI 
(#38200)
 add e30c93a9be Make sure IMAGE_TAG is properly propagated to tests (#38201)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/helm-tests.yml | 2 +-
 .github/workflows/run-unit-tests.yml | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

(airflow) branch main updated: Better diangostics for parallel test execution in CI (#38200)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 63bc63c190 Better diangostics for parallel test execution in CI 
(#38200)
63bc63c190 is described below

commit 63bc63c190a4648ee71301759dd1e959f2095f70
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 09:25:48 2024 +0100

Better diangostics for parallel test execution in CI (#38200)

When running tests in parallel in our CI we switched off showing
success outputs by default in #38157. This is generally a good
thing, because it only shows outputs for the test types that failed
and makes the CI log output far snappier in case only one or few
test types failed.

However it also has a side effect that it's
not really easy to see what kind of commands were run in each
parallel test type, so we might get accidentally some mistakes
where we miss that our tests are not doing what we think they are
doing - for example not running tests for various Python versions,
only for one, or not running them for various database versions
(both cases happened in the past). By hiding the success outputs,
we make it harder to spot such issues.

This PR introduces two ways to improve it:

* we can now set a label on PR to `include success outputs` and
  any committer can set the label to show success outputs as well.

* When we run parallel unit tests we summarize what is going
  to be executed - showing all the ShellParams passed to the
  parallel test execution. This is the best place to show it
  because this ShellParams object contains all the actual values
  passed - after processing throught options, special environment
  variables and so-on - so those are the actual values used to
  run the parallel tests.

Also it will be printed in a prominent place - just before test
execution in the main console output, which means it will be
visible without the need to unfold any output.
---
 .github/workflows/ci.yml| 12 +++-
 .github/workflows/run-unit-tests.yml|  6 ++
 .github/workflows/special-tests.yml |  8 
 dev/breeze/doc/ci/04_selective_checks.md|  8 +++-
 .../src/airflow_breeze/commands/testing_commands.py | 17 ++---
 dev/breeze/src/airflow_breeze/utils/selective_checks.py |  7 +++
 6 files changed, 53 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 394ddf1553..620b0bfac4 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -48,7 +48,6 @@ env:
   GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
   IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
   USE_SUDO: "true"
-  INCLUDE_SUCCESS_OUTPUTS: "true"
   INCLUDE_NOT_READY_PROVIDERS: "true"
   AIRFLOW_ENABLE_AIP_44: "true"
   MOUNT_SOURCES: "skip"
@@ -98,6 +97,7 @@ jobs:
   full-tests-needed: ${{ steps.selective-checks.outputs.full-tests-needed 
}}
   parallel-test-types-list-as-string: >-
 ${{ steps.selective-checks.outputs.parallel-test-types-list-as-string 
}}
+  include-success-outputs: ${{ 
steps.selective-checks.outputs.include-success-outputs }}
   postgres-exclude: ${{ steps.selective-checks.outputs.postgres-exclude }}
   mysql-exclude: ${{ steps.selective-checks.outputs.mysql-exclude }}
   sqlite-exclude: ${{ steps.selective-checks.outputs.sqlite-exclude }}
@@ -421,6 +421,7 @@ jobs:
   BACKEND: sqlite
   # Force more parallelism for pull even on public images
   PARALLELISM: 6
+  INCLUDE_SUCCESS_OUTPUTS: 
"${{needs.build-info.outputs.include-success-outputs}}"
 steps:
   - name: "Cleanup repo"
 shell: bash
@@ -459,6 +460,7 @@ jobs:
   PYTHON_VERSIONS: 
${{needs.build-info.outputs.all-python-versions-list-as-string}}
   DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
   VERSION_SUFFIX_FOR_PYPI: "dev0"
+  INCLUDE_SUCCESS_OUTPUTS: "true"
 if: needs.build-info.outputs.ci-image-build == 'true'
 steps:
   - name: "Cleanup repo"
@@ -724,6 +726,7 @@ jobs:
 env:
   RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
   PYTHON_MAJOR_MINOR_VERSION: 
"${{needs.build-info.outputs.default-python-version}}"
+  INCLUDE_SUCCESS_OUTPUTS: 
"${{needs.build-info.outputs.include-success-outputs}}"
 steps:
   - name: "Cleanup repo"
 shell: bash
@@ -914,6 +917,7 @@ jobs:
   backend-versions: ${{ needs.build-info.outputs.postgres-versions }}
   excludes: ${{ needs.build-info.outputs.postgres-exclude }}
  

(airflow) branch main updated: Extract unit tests to separate workflows (#38157)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new edbf1e57d9 Extract unit tests to separate workflows (#38157)
edbf1e57d9 is described below

commit edbf1e57d9033c5f7877819edecf6363a8dbeaaf
Author: Jarek Potiuk 
AuthorDate: Sat Mar 16 08:06:51 2024 +0100

Extract unit tests to separate workflows (#38157)

This is continuation of decomposing of the huge ci.yaml of ours with
lot of copy-pasted yaml into smaller, separated workflows that are
called when needed. This one is by far the biggest one in terms of
how many repeated lines of workflow have been removed by that one.

It introduce two workflows:

1) Run unit tests workflow that run regular test in a matrix -
   we call them directly from CI yaml for postgres/mysql/sqlite
   and Non-DB tests - and they can run a matrix of tests - depending
   on type of the build we have - in simple PRs, it will be just
   matrix of default values, in more complex and canary build it
   will be a complete matrix of tests.

2) "Special test" - those are special tests that are always run with
   postgres, default version and Python default version but they
   have some "special" thing enabled - downgrade/upgrade of a specific
   important dependecy (boto/sqlalchemy), running quarantined tests
   or performing just collection of tests simulating ARM test
   collection. This workflow calls the regular unit test 1) workflow
   for each such special case - adding one more level of indirection
---
 .github/workflows/ci.yml | 659 ++-
 .github/workflows/run-unit-tests.yml | 185 ++
 .github/workflows/special-tests.yml  | 230 
 3 files changed, 522 insertions(+), 552 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 26e66bb070..394ddf1553 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -897,427 +897,122 @@ jobs:
   needs.build-info.outputs.latest-versions-only != 'true'
 
   tests-postgres:
-timeout-minutes: 130
-name: >
-  DB:Postgres${{matrix.backend-version}},Py${{matrix.python-version}}:
-  ${{needs.build-info.outputs.parallel-test-types-list-as-string}}
-runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
+name: "Postgres tests"
+uses: ./.github/workflows/run-unit-tests.yml
 needs: [build-info, wait-for-ci-images]
-strategy:
-  matrix:
-python-version: 
"${{fromJson(needs.build-info.outputs.python-versions)}}"
-backend-version: 
"${{fromJson(needs.build-info.outputs.postgres-versions)}}"
-exclude: "${{fromJson(needs.build-info.outputs.postgres-exclude)}}"
-  fail-fast: false
-env:
-  RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
-  PARALLEL_TEST_TYPES: 
"${{needs.build-info.outputs.parallel-test-types-list-as-string}}"
-  PR_LABELS: "${{needs.build-info.outputs.pull-request-labels}}"
-  FULL_TESTS_NEEDED: "${{needs.build-info.outputs.full-tests-needed}}"
-  DEBUG_RESOURCES: "${{needs.build-info.outputs.debug-resources}}"
-  BACKEND: "postgres"
-  PYTHON_MAJOR_MINOR_VERSION: "${{matrix.python-version}}"
-  BACKEND_VERSION: "${{matrix.backend-version}}"
-  JOB_ID: "postgres-${{matrix.backend-version}}-${{matrix.python-version}}"
-  ENABLE_COVERAGE: "${{needs.build-info.outputs.run-coverage}}"
+permissions:
+  contents: read
+  packages: read
+secrets: inherit
+with:
+  runs-on: ${{ needs.build-info.outputs.runs-on }}
+  backend: "postgres"
+  test-name: "Postgres"
+  test-scope: "DB"
+  image-tag: ${{ needs.build-info.outputs.image-tag }}
+  python-versions: ${{ needs.build-info.outputs.python-versions }}
+  backend-versions: ${{ needs.build-info.outputs.postgres-versions }}
+  excludes: ${{ needs.build-info.outputs.postgres-exclude }}
+  parallel-test-types-list-as-string: ${{ 
needs.build-info.outputs.parallel-test-types-list-as-string }}
+  run-migration-tests: "true"
+  run-coverage: ${{ needs.build-info.outputs.run-coverage }}
+  debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+  breeze-python-version: ${{ 
needs.build-info.outputs.breeze-python-version }}
 if: needs.build-info.outputs.run-tests == 'true'
-steps:
-  - name: "Cleanup repo"
-shell: bash
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
-  - name: "Checkout ${{ github.ref }} ( ${{ githu

(airflow) branch main updated: Improve speed and stability of Python 3.12 tests in canary build (#38194)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new dffc9e3fa8 Improve speed and stability of Python 3.12 tests in canary 
build (#38194)
dffc9e3fa8 is described below

commit dffc9e3fa83e3cd910a83e7c505fe23b26feddf2
Author: Jarek Potiuk 
AuthorDate: Fri Mar 15 23:32:34 2024 +0100

Improve speed and stability of Python 3.12 tests in canary build (#38194)

Python 3.12 introduced a new (much faster) way of tracking and
monitoring execution of python code by tools like coverage tracking
using sysmon (PEP 669). This however also apparently heavily impacted
performance of coverage tracking for Python 3.12 when PEP 669 is not
used. The coverage library since 7.4.0 has an experimental support
for PEP 669 that can be enabled with COVERAGE_CORE=sysmon env variable
and a number of users confirmed it fixes the problem.

We are using 7.4.4 coverage already so we should enable this mode
to speed up our coverage tracking. That should also allow us to
remove databricks from excluded providers.

See https://github.com/databricks/databricks-sql-python/issues/369
for databricks case and https://github.com/nedbat/coveragepy/issues/1665
for coverage bug.
---
 dev/breeze/src/airflow_breeze/commands/testing_commands.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands.py 
b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
index eceaa0a22e..cb8bd5723e 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
@@ -841,6 +841,8 @@ def helm_tests(
 python_version=shell_params.python,
 helm_test_package=helm_test_package,
 )
+if sys.version_info >= (3, 12):
+env["COVERAGE_CORE"] = "sysmon"
 cmd = ["docker", "compose", "run", "--service-ports", "--rm", "airflow", 
*pytest_args, *extra_pytest_args]
 result = run_command(cmd, check=False, env=env, 
output_outside_the_group=True)
 fix_ownership_using_docker()

(airflow) branch main updated: Add platform markers for the tests (#38173)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 4bd5e5bfff Add platform markers for the tests (#38173)
4bd5e5bfff is described below

commit 4bd5e5bfff7c7c31dbd778e768b37fa6984fcd7b
Author: Andrey Anshin 
AuthorDate: Sat Mar 16 02:18:30 2024 +0400

Add platform markers for the tests (#38173)
---
 contributing-docs/testing/unit_tests.rst | 12 
 tests/conftest.py| 23 +++
 tests/core/test_impersonation_tests.py   |  2 +-
 3 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/contributing-docs/testing/unit_tests.rst 
b/contributing-docs/testing/unit_tests.rst
index 53bf3a1210..59082ecf5a 100644
--- a/contributing-docs/testing/unit_tests.rst
+++ b/contributing-docs/testing/unit_tests.rst
@@ -1075,6 +1075,18 @@ This prepares airflow .whl package in the dist folder.
 
  breeze --use-airflow-version wheel --use-packages-from-dist 
--mount-sources skip
 
+Other Settings
+--
+
+Skip test on unsupported platform / environment
+...
+
+You can apply the marker ``pytest.mark.platform(name)`` to the specific test 
case, class or module
+for prevent to run on unsupported platform.
+
+- ``linux``: Run test only on linux platform
+- ``breeze``: Run test only inside of Breeze container, it might be useful in 
case of run
+  some potential dangerous things in tests or if it expects to use common 
Breeze things.
 
 Code Coverage
 -
diff --git a/tests/conftest.py b/tests/conftest.py
index 7cacce0621..133d78a32b 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -338,6 +338,7 @@ def pytest_configure(config):
 config.addinivalue_line("markers", "integration(name): mark test to run 
with named integration")
 config.addinivalue_line("markers", "backend(name): mark test to run with 
named backend")
 config.addinivalue_line("markers", "system(name): mark test to run with 
named system")
+config.addinivalue_line("markers", "platform(name): mark test to run with 
specific platform/environment")
 config.addinivalue_line("markers", "long_running: mark test that run for a 
long time (many minutes)")
 config.addinivalue_line(
 "markers", "quarantined: mark test that are in quarantine (i.e. flaky, 
need to be isolated and fixed)"
@@ -396,6 +397,26 @@ def skip_if_not_marked_with_backend(selected_backend, 
item):
 )
 
 
+def skip_if_platform_doesnt_match(marker):
+allowed_platforms = ("linux", "breeze")
+if not (args := marker.args):
+pytest.fail(f"No platform specified, expected one of: {', 
'.join(map(repr, allowed_platforms))}")
+elif not all(a in allowed_platforms for a in args):
+pytest.fail(
+f"Allowed platforms {', '.join(map(repr, allowed_platforms))}; "
+f"but got: {', '.join(map(repr, args))}"
+)
+if "linux" in args:
+if not sys.platform.startswith("linux"):
+pytest.skip("Test expected to run on Linux platform.")
+if "breeze" in args:
+if not os.path.isfile("/.dockerenv") or os.environ.get("BREEZE", 
"").lower() != "true":
+raise pytest.skip(
+"Test expected to run into Airflow Breeze container. "
+"Maybe because it is to dangerous to run it outside."
+)
+
+
 def skip_if_not_marked_with_system(selected_systems, item):
 for marker in item.iter_markers(name="system"):
 systems_name = marker.args[0]
@@ -539,6 +560,8 @@ def pytest_runtest_setup(item):
 skip_if_not_marked_with_system(selected_systems_list, item)
 else:
 skip_system_test(item)
+for marker in item.iter_markers(name="platform"):
+skip_if_platform_doesnt_match(marker)
 for marker in item.iter_markers(name="backend"):
 skip_if_wrong_backend(marker, item)
 selected_backend = item.config.option.backend
diff --git a/tests/core/test_impersonation_tests.py 
b/tests/core/test_impersonation_tests.py
index 3fa2d3b981..b17900820d 100644
--- a/tests/core/test_impersonation_tests.py
+++ b/tests/core/test_impersonation_tests.py
@@ -39,7 +39,7 @@ from tests.test_utils import db
 
 # The entire module into the quarantined mark, this might have unpredictable 
side effects to other tests
 # and should be moved into the isolated environment into the future.
-pytestmark = [pytest.mark.db_test, pytest.mark.quarantined]
+pytestmark = [pytest.mark.platform("breeze"), pytest.mark.db_test, 
pytest.mark.quarantined]
 
 DEV_NULL = "/dev/null"
 TEST_ROOT_FOLDER = os.path.abspath(os.path.join(os.path.dirname(__file__), 
".."))

(airflow) branch main updated: Set job labels for traceability in BigQuery jobs (#37736)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 3e2bfb8b3e Set job labels for traceability in BigQuery jobs (#37736)
3e2bfb8b3e is described below

commit 3e2bfb8b3ee80ddc18b00e461de53390dcc5a8b3
Author: Pat Leamon 
AuthorDate: Sat Mar 16 08:36:08 2024 +1100

Set job labels for traceability in BigQuery jobs (#37736)

* Set job labels for traceability

* Fixup

* Ensure the automatic labels will be valid

If not valid they won't be added.

* Fixup

* Merge length + regex check

* Simplify the label checking logic
---
 .../providers/google/cloud/operators/bigquery.py   |  17 +++
 .../google/cloud/operators/test_bigquery.py| 134 +
 2 files changed, 151 insertions(+)

diff --git a/airflow/providers/google/cloud/operators/bigquery.py 
b/airflow/providers/google/cloud/operators/bigquery.py
index 77862491b4..bb75b068b1 100644
--- a/airflow/providers/google/cloud/operators/bigquery.py
+++ b/airflow/providers/google/cloud/operators/bigquery.py
@@ -20,6 +20,7 @@ from __future__ import annotations
 
 import enum
 import json
+import re
 import warnings
 from functools import cached_property
 from typing import TYPE_CHECKING, Any, Iterable, Sequence, SupportsAbs
@@ -64,6 +65,8 @@ if TYPE_CHECKING:
 
 BIGQUERY_JOB_DETAILS_LINK_FMT = 
"https://console.cloud.google.com/bigquery?j={job_id};
 
+LABEL_REGEX = re.compile(r"^[a-z][\w-]{0,63}$")
+
 
 class BigQueryUIColors(enum.Enum):
 """Hex colors for BigQuery operators."""
@@ -2768,11 +2771,25 @@ class 
BigQueryInsertJobOperator(GoogleCloudBaseOperator, _BigQueryOpenLineageMix
 with open(self.configuration) as file:
 self.configuration = json.loads(file.read())
 
+def _add_job_labels(self) -> None:
+dag_label = self.dag_id.lower()
+task_label = self.task_id.lower()
+
+if LABEL_REGEX.match(dag_label) and LABEL_REGEX.match(task_label):
+automatic_labels = {"airflow-dag": dag_label, "airflow-task": 
task_label}
+if isinstance(self.configuration.get("labels"), dict):
+self.configuration["labels"].update(automatic_labels)
+elif "labels" not in self.configuration:
+self.configuration["labels"] = automatic_labels
+
 def _submit_job(
 self,
 hook: BigQueryHook,
 job_id: str,
 ) -> BigQueryJob:
+# Annotate the job with dag and task id labels
+self._add_job_labels()
+
 # Submit a new job without waiting for it to complete.
 return hook.insert_job(
 configuration=self.configuration,
diff --git a/tests/providers/google/cloud/operators/test_bigquery.py 
b/tests/providers/google/cloud/operators/test_bigquery.py
index 64e31913fb..ae0bcbfb89 100644
--- a/tests/providers/google/cloud/operators/test_bigquery.py
+++ b/tests/providers/google/cloud/operators/test_bigquery.py
@@ -1102,6 +1102,7 @@ class TestBigQueryInsertJobOperator:
 project_id=TEST_GCP_PROJECT_ID,
 )
 result = op.execute(context=MagicMock())
+assert configuration["labels"] == {"airflow-dag": "adhoc_airflow", 
"airflow-task": "insert_query_job"}
 
 mock_hook.return_value.insert_job.assert_called_once_with(
 configuration=configuration,
@@ -1143,6 +1144,7 @@ class TestBigQueryInsertJobOperator:
 project_id=TEST_GCP_PROJECT_ID,
 )
 result = op.execute(context=MagicMock())
+assert configuration["labels"] == {"airflow-dag": "adhoc_airflow", 
"airflow-task": "copy_query_job"}
 
 mock_hook.return_value.insert_job.assert_called_once_with(
 configuration=configuration,
@@ -1753,6 +1755,138 @@ class TestBigQueryInsertJobOperator:
 project_id=TEST_GCP_PROJECT_ID,
 )
 
+
@mock.patch("airflow.providers.google.cloud.operators.bigquery.BigQueryHook")
+def test_execute_adds_to_existing_labels(self, mock_hook):
+job_id = "123456"
+hash_ = "hash"
+real_job_id = f"{job_id}_{hash_}"
+
+configuration = {
+"query": {
+"query": "SELECT * FROM any",
+"useLegacySql": False,
+},
+"labels": {"foo": "bar"},
+}
+mock_hook.return_value.insert_job.return_value = 
MagicMock(job_id=real_job_id, error_result=False)
+mock_hook.return_value.generate_job_id.return_value = real_job_id
+
+o

(airflow) branch main updated: Extract provider checks as separate workflow (from CI workflow) (#38130)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 654038af5d Extract provider checks as separate workflow (from CI 
workflow) (#38130)
654038af5d is described below

commit 654038af5dbea30f2d630746390aebdcd11afd65
Author: Jarek Potiuk 
AuthorDate: Fri Mar 15 21:59:32 2024 +0100

Extract provider checks as separate workflow (from CI workflow) (#38130)
---
 .github/workflows/check-providers.yml | 250 ++
 .github/workflows/ci.yml  | 207 +++-
 2 files changed, 268 insertions(+), 189 deletions(-)

diff --git a/.github/workflows/check-providers.yml 
b/.github/workflows/check-providers.yml
new file mode 100644
index 00..23855aa510
--- /dev/null
+++ b/.github/workflows/check-providers.yml
@@ -0,0 +1,250 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Provider tests
+on:  # yamllint disable-line rule:truthy
+  workflow_call:
+inputs:
+  runs-on:
+description: "The array of labels (in json form) determining type of 
the runner to use for the build."
+required: false
+default: '["ubuntu-22.04"]'
+type: string
+  image-tag:
+description: "Tag to set for the image"
+required: true
+type: string
+  default-python-version:
+description: Which version of python should be used get CI image
+required: true
+type: string
+  upgrade-to-newer-dependencies:
+description: "Whether to upgrade to newer dependencies"
+required: true
+type: string
+  affected-providers-list-as-string:
+description: "List of affected providers as string"
+required: false
+type: string
+  providers-compatibility-checks:
+description: >
+  JSON-formatted array of providers compatibility checks in the form 
of array of dicts
+  (airflow-version, python-versions, remove-providers)
+required: true
+type: string
+  skip-provider-tests:
+description: "Whether to skip provider tests (true/false)"
+required: true
+type: string
+  python-versions:
+description: "JSON-formatted array of Python versions to build images 
from"
+required: true
+type: string
+  breeze-python-version:
+description: >
+  Which version of python should be used to install Breeze (3.9 is 
minimum for reproducible builds)
+required: true
+type: string
+jobs:
+  prepare-install-verify-provider-packages-wheel:
+timeout-minutes: 80
+name: "Provider packages wheel build and verify"
+runs-on: ${{fromJSON(inputs.runs-on)}}
+env:
+  RUNS_ON: "${{ inputs.runs-on }}"
+  PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+  IMAGE_TAG: "${{ inputs.image-tag }}"
+  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+steps:
+  - name: "Cleanup repo"
+shell: bash
+run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
+  - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+uses: actions/checkout@v4
+with:
+  persist-credentials: false
+  - name: Cleanup docker
+uses: ./.github/actions/cleanup-docker
+  - name: >
+  Prepare breeze & CI image: ${{ inputs.default-python-version }}:${{ 
inputs.image-tag }}
+uses: ./.github/actions/prepare_breeze_and_image
+with:
+  python-version: ${{ inputs.breeze-python-version }}
+  - name: "Cleanup dist files"
+run: rm -fv ./dist/*
+  - name: "Prepare provider documentation"
+run: >
+  breeze release-management prepare-provider-documentation 
--include-not-ready-providers
+  --non-interactive
+  - name: "Prepare provider pack

(airflow) 01/01: Improve speed and stability of Python 3.12 tests in canary build

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch improve-speed-and-stability-of-3.12-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 6a4488a34529a572052bbeb8ef7a49f4b724adc9
Author: Jarek Potiuk 
AuthorDate: Fri Mar 15 21:26:43 2024 +0100

Improve speed and stability of Python 3.12 tests in canary build

Python 3.12 introduced a new (much faster) way of tracking and
monitoring execution of python code by tools like coverage tracking
using sysmon (PEP 669). This however also apparently heavily impacted
performance of coverage tracking for Python 3.12 when PEP 669 is not
used. The coverage library since 7.4.0 has an experimental support
for PEP 669 that can be enabled with COVERAGE_CORE=sysmon env variable
and a number of users confirmed it fixes the problem.

We are using 7.4.4 coverage already so we should enable this mode
to speed up our coverage tracking. That should also allow us to
remove databricks from excluded providers.

See https://github.com/databricks/databricks-sql-python/issues/369
for databricks case and https://github.com/nedbat/coveragepy/issues/1665
for coverage bug.
---
 dev/breeze/src/airflow_breeze/commands/testing_commands.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands.py 
b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
index eceaa0a22e..cb8bd5723e 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
@@ -841,6 +841,8 @@ def helm_tests(
 python_version=shell_params.python,
 helm_test_package=helm_test_package,
 )
+if sys.version_info >= (3, 12):
+env["COVERAGE_CORE"] = "sysmon"
 cmd = ["docker", "compose", "run", "--service-ports", "--rm", "airflow", 
*pytest_args, *extra_pytest_args]
 result = run_command(cmd, check=False, env=env, 
output_outside_the_group=True)
 fix_ownership_using_docker()

(airflow) branch improve-speed-and-stability-of-3.12-tests created (now 6a4488a345)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch improve-speed-and-stability-of-3.12-tests
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at 6a4488a345 Improve speed and stability of Python 3.12 tests in canary 
build

This branch includes the following new commits:

 new 6a4488a345 Improve speed and stability of Python 3.12 tests in canary 
build

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) branch main updated (f78c71e8ff -> 3bac199a02)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from f78c71e8ff Bump follow-redirects from 1.15.4 to 1.15.6 in /airflow/www 
(#38156)
 add 3bac199a02 Ignore mypy complain about `directives._directives` in docs 
(#38190)

No new revisions were added by this update.

Summary of changes:
 docs/exts/substitution_extensions.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(airflow) branch main updated: Bump follow-redirects from 1.15.4 to 1.15.6 in /airflow/www (#38156)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new f78c71e8ff Bump follow-redirects from 1.15.4 to 1.15.6 in /airflow/www 
(#38156)
f78c71e8ff is described below

commit f78c71e8ff7e403fafdb784140de533528061b36
Author: Dependabot [bot] <49699333+dependabot[bot]@users.noreply.github.com>
AuthorDate: Fri Mar 15 18:04:09 2024 +0100

Bump follow-redirects from 1.15.4 to 1.15.6 in /airflow/www (#38156)

Bumps 
[follow-redirects](https://github.com/follow-redirects/follow-redirects) from 
1.15.4 to 1.15.6.
- [Release 
notes](https://github.com/follow-redirects/follow-redirects/releases)
- 
[Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.15.4...v1.15.6)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] 
Co-authored-by: dependabot[bot] 
<49699333+dependabot[bot]@users.noreply.github.com>
---
 airflow/www/yarn.lock | 58 ---
 1 file changed, 4 insertions(+), 54 deletions(-)

diff --git a/airflow/www/yarn.lock b/airflow/www/yarn.lock
index ba8c8fc336..25693116bb 100644
--- a/airflow/www/yarn.lock
+++ b/airflow/www/yarn.lock
@@ -5637,15 +5637,6 @@ dom-serializer@^1.0.1:
 domhandler "^4.2.0"
 entities "^2.0.0"
 
-dom-serializer@^2.0.0:
-  version "2.0.0"
-  resolved 
"https://registry.yarnpkg.com/dom-serializer/-/dom-serializer-2.0.0.tgz#e41b802e1eedf9f6cae183ce5e622d789d7d8e53;
-  integrity 
sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg==
-  dependencies:
-domelementtype "^2.3.0"
-domhandler "^5.0.2"
-entities "^4.2.0"
-
 domelementtype@1:
   version "1.3.1"
   resolved 
"https://registry.yarnpkg.com/domelementtype/-/domelementtype-1.3.1.tgz#d048c44b37b0d10a7f2a3d5fee3f4333d790481f;
@@ -5656,11 +5647,6 @@ domelementtype@^2.0.1, domelementtype@^2.2.0:
   resolved 
"https://registry.yarnpkg.com/domelementtype/-/domelementtype-2.2.0.tgz#9a0b6c2782ed6a1c7323d42267183df9bd8b1d57;
   integrity 
sha512-DtBMo82pv1dFtUmHyr48beiuq792Sxohr+8Hm9zoxklYPfa6n0Z3Byjj2IV7bmr2IyqClnqEQhfgHJJ5QF0R5A==
 
-domelementtype@^2.3.0:
-  version "2.3.0"
-  resolved 
"https://registry.yarnpkg.com/domelementtype/-/domelementtype-2.3.0.tgz#5c45e8e869952626331d7aab326d01daf65d589d;
-  integrity 
sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw==
-
 domexception@^2.0.1:
   version "2.0.1"
   resolved 
"https://registry.yarnpkg.com/domexception/-/domexception-2.0.1.tgz#fb44aefba793e1574b0af6aed2801d057529f304;
@@ -5689,13 +5675,6 @@ domhandler@^4.3.1:
   dependencies:
 domelementtype "^2.2.0"
 
-domhandler@^5.0.2, domhandler@^5.0.3:
-  version "5.0.3"
-  resolved 
"https://registry.yarnpkg.com/domhandler/-/domhandler-5.0.3.tgz#cc385f7f751f1d1fc650c21374804254538c7d31;
-  integrity 
sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w==
-  dependencies:
-domelementtype "^2.3.0"
-
 dompurify@^2.2.8:
   version "2.2.9"
   resolved 
"https://registry.yarnpkg.com/dompurify/-/dompurify-2.2.9.tgz#4b42e244238032d9286a0d2c87b51313581d9624;
@@ -5727,15 +5706,6 @@ domutils@^2.8.0:
 domelementtype "^2.2.0"
 domhandler "^4.2.0"
 
-domutils@^3.0.1:
-  version "3.1.0"
-  resolved 
"https://registry.yarnpkg.com/domutils/-/domutils-3.1.0.tgz#c47f551278d3dc4b0b1ab8cbb42d751a6f0d824e;
-  integrity 
sha512-H78uMmQtI2AhgDJjWeQmHwJJ2bLPD3GMmO7Zja/ZZh84wkm+4ut+IUnUdRa8uCGX88DiVx1j6FRe1XfxEgjEZA==
-  dependencies:
-dom-serializer "^2.0.0"
-domelementtype "^2.3.0"
-domhandler "^5.0.3"
-
 echarts@^5.4.2:
   version "5.4.2"
   resolved 
"https://registry.yarnpkg.com/echarts/-/echarts-5.4.2.tgz#9f38781c9c6ae323e896956178f6956952c77a48;
@@ -5792,11 +5762,6 @@ entities@^2.0.0:
   resolved 
"https://registry.yarnpkg.com/entities/-/entities-2.2.0.tgz#098dc90ebb83d8dffa089d55256b351d34c4da55;
   integrity 
sha512-p92if5Nz619I0w+akJrLZH0MX0Pb5DX39XOwQTtXSdQQOaYH03S1uIQp4mhOZtAXrxq4ViO67YTiLBo2638o9A==
 
-entities@^4.2.0, entities@^4.4.0:
-  version "4.5.0"
-  resolved 
"https://registry.yarnpkg.com/entities/-/entities-4.5.0.tgz#5d268ea5e7113ec74c4d033b79ea5a35a488fb48;
-  integrity 
sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==
-
 envinfo@^7.7.3:
   version "7.8.1"
   resolved 
"https://registry.yarnpkg.com/envinfo/-/envinfo-7.8.1.tgz#06377e3e5f4d379fea

(airflow) branch main updated: Update deferring.rst (#38122)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new bbb03a7c17 Update deferring.rst (#38122)
bbb03a7c17 is described below

commit bbb03a7c172aa4d688f6e126d399240f817b315f
Author: 9uzman 
AuthorDate: Fri Mar 15 12:52:20 2024 -0300

Update deferring.rst (#38122)
---
 docs/apache-airflow/authoring-and-scheduling/deferring.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/apache-airflow/authoring-and-scheduling/deferring.rst 
b/docs/apache-airflow/authoring-and-scheduling/deferring.rst
index 912e60dbca..218eb04a36 100644
--- a/docs/apache-airflow/authoring-and-scheduling/deferring.rst
+++ b/docs/apache-airflow/authoring-and-scheduling/deferring.rst
@@ -109,7 +109,7 @@ If you want to trigger deferral, at any place in your 
operator, you can call ``s
 
 When you opt to defer, your operator will stop executing at that point and be 
removed from its current worker. No state will persist, such as local variables 
or attributes set on ``self``. When your operator resumes, it resumes as a new 
instance of it. The only way you can pass state from the old instance of the 
operator to the new one is with ``method_name`` and ``kwargs``.
 
-When your operator resumes, Airflow adds an ``event`` object to the kwargs 
passed to the ``method_name`` method. This ``event`` object contains the 
payload from the trigger event that resumed your operator. Depending on the 
trigger, this can be useful to your operator, like it's a status code or URL to 
fetch results. Or, it might be unimportant information, like a datetime. Your 
``method_name`` method, however, *must* accept ``event`` as a keyword argument.
+When your operator resumes, Airflow adds a ``context`` object and an ``event`` 
object to the kwargs passed to the ``method_name`` method. This ``event`` 
object contains the payload from the trigger event that resumed your operator. 
Depending on the trigger, this can be useful to your operator, like it's a 
status code or URL to fetch results. Or, it might be unimportant information, 
like a datetime. Your ``method_name`` method, however, *must* accept 
``context`` and ``event`` as a keyword [...]
 
 If your operator returns from either its first ``execute()`` method when it's 
new, or a subsequent method specified by ``method_name``, it will be considered 
complete and finish executing.
 

(airflow) branch dependabot/npm_and_yarn/airflow/www/follow-redirects-1.15.6 updated (732fbb9c2a -> cfcc0b8d88)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch 
dependabot/npm_and_yarn/airflow/www/follow-redirects-1.15.6
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 732fbb9c2a Bump follow-redirects from 1.15.4 to 1.15.6 in /airflow/www
 add 90a5cd66ae Revert "Simplify dataset serialization code (#38089)" 
(#38158)
 add c77ce4c0ab Remove unnecessary cache for MyPy jobs (#38162)
 add a98e21896d Updated year in NOTICE (#38153)
 add cfcc0b8d88 Bump follow-redirects from 1.15.4 to 1.15.6 in /airflow/www

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (732fbb9c2a)
\
 N -- N -- N   
refs/heads/dependabot/npm_and_yarn/airflow/www/follow-redirects-1.15.6 
(cfcc0b8d88)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/mypy.yml  | 10 +-
 NOTICE  |  2 +-
 airflow/datasets/__init__.py| 11 ---
 airflow/models/dag.py   | 19 +++
 scripts/ci/pre_commit/pre_commit_mypy_folder.py |  2 +-
 5 files changed, 18 insertions(+), 26 deletions(-)

(airflow) branch main updated: Updated year in NOTICE (#38153)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new a98e21896d Updated year in NOTICE (#38153)
a98e21896d is described below

commit a98e21896d879648226971a7ed77427f9b327441
Author: Siva Kumar Edupuganti <161360372+esivakuma...@users.noreply.github.com>
AuthorDate: Thu Mar 14 15:12:21 2024 -0700

Updated year in NOTICE (#38153)
---
 NOTICE | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/NOTICE b/NOTICE
index 33371e44a7..f6040a224c 100644
--- a/NOTICE
+++ b/NOTICE
@@ -1,5 +1,5 @@
 Apache Airflow
-Copyright 2016-2023 The Apache Software Foundation
+Copyright 2016-2024 The Apache Software Foundation
 
 This product includes software developed at The Apache Software
 Foundation (http://www.apache.org/).

(airflow) branch main updated: Remove unnecessary cache for MyPy jobs (#38162)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new c77ce4c0ab Remove unnecessary cache for MyPy jobs (#38162)
c77ce4c0ab is described below

commit c77ce4c0abf3b259b739174523f3ae984100381e
Author: Jarek Potiuk 
AuthorDate: Thu Mar 14 22:23:03 2024 +0100

Remove unnecessary cache for MyPy jobs (#38162)

The MyPy jobs started to fail pretty randomly in docs and dev
builds and the main suspect for that is cache - either pre-commit
cache or docker cache stored locally in docker volume.

We can actually disable the cache, because a) pre-commit cache is
not really needed for this job as we only one venv to create and
b) the docker volume cache in case of CI job is at most harmless
as we exect to start all jobs from a clean state.
---
 .github/workflows/mypy.yml  | 10 +-
 scripts/ci/pre_commit/pre_commit_mypy_folder.py |  2 +-
 2 files changed, 2 insertions(+), 10 deletions(-)

diff --git a/.github/workflows/mypy.yml b/.github/workflows/mypy.yml
index debac81173..7fdaec187b 100644
--- a/.github/workflows/mypy.yml
+++ b/.github/workflows/mypy.yml
@@ -79,15 +79,6 @@ jobs:
 with:
   python-version: ${{ inputs.breeze-python-version }}
 if: inputs.needs-mypy == 'true'
-  - name: Cache pre-commit envs
-uses: actions/cache@v4
-with:
-  path: ~/.cache/pre-commit
-  # yamllint disable-line rule:line-length
-  key: "pre-commit-${{steps.breeze.outputs.host-python-version}}-${{ 
hashFiles('.pre-commit-config.yaml') }}"
-  restore-keys: |
-pre-commit-${{steps.breeze.outputs.host-python-version}}-
-if: inputs.needs-mypy == 'true'
   - name: "MyPy checks for ${{ matrix.mypy-folder }}"
 run: |
   pip install pre-commit
@@ -98,4 +89,5 @@ jobs:
   SKIP_GROUP_OUTPUT: "true"
   DEFAULT_BRANCH: ${{ inputs.default-branch }}
   RUFF_FORMAT: "github"
+  INCLUDE_MYPY_VOLUME: "false"
 if: inputs.needs-mypy == 'true'
diff --git a/scripts/ci/pre_commit/pre_commit_mypy_folder.py 
b/scripts/ci/pre_commit/pre_commit_mypy_folder.py
index 9a22a07c75..4fd4a1 100755
--- a/scripts/ci/pre_commit/pre_commit_mypy_folder.py
+++ b/scripts/ci/pre_commit/pre_commit_mypy_folder.py
@@ -53,7 +53,7 @@ res = run_command_via_breeze_shell(
 ],
 warn_image_upgrade_needed=True,
 extra_env={
-"INCLUDE_MYPY_VOLUME": "true",
+"INCLUDE_MYPY_VOLUME": os.environ.get("INCLUDE_MYPY_VOLUME", "true"),
 # Need to mount local sources when running it - to not have to rebuild 
the image
 # and to let CI work on it when running on PRs from forks - because 
mypy-dev uses files
 # that are not available at the time when image is built in CI

(airflow) branch main updated: Revert "Simplify dataset serialization code (#38089)" (#38158)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 90a5cd66ae Revert "Simplify dataset serialization code (#38089)" 
(#38158)
90a5cd66ae is described below

commit 90a5cd66ae19445dd7560caf3a5cff595dc43388
Author: Jarek Potiuk 
AuthorDate: Thu Mar 14 20:05:59 2024 +0100

Revert "Simplify dataset serialization code (#38089)" (#38158)

This reverts commit c14241b72c8057730adcd5cf7d969d8bf0b708f7.
---
 airflow/datasets/__init__.py | 11 ---
 airflow/models/dag.py| 19 +++
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/airflow/datasets/__init__.py b/airflow/datasets/__init__.py
index 1edcbb946d..4f8d587727 100644
--- a/airflow/datasets/__init__.py
+++ b/airflow/datasets/__init__.py
@@ -95,9 +95,6 @@ class BaseDatasetEventInput(Protocol):
 def __and__(self, other: BaseDatasetEventInput) -> DatasetAll:
 return DatasetAll(self, other)
 
-def as_expression(self) -> dict[str, Any]:
-raise NotImplementedError
-
 def evaluate(self, statuses: dict[str, bool]) -> bool:
 raise NotImplementedError
 
@@ -129,11 +126,6 @@ class Dataset(os.PathLike, BaseDatasetEventInput):
 def __hash__(self) -> int:
 return hash(self.uri)
 
-def as_expression(self) -> dict[str, Any]:
-if self.extra is None:
-return {"uri": self.uri}
-return {"uri": self.uri, "extra": self.extra}
-
 def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
 yield self.uri, self
 
@@ -149,9 +141,6 @@ class _DatasetBooleanCondition(BaseDatasetEventInput):
 def __init__(self, *objects: BaseDatasetEventInput) -> None:
 self.objects = objects
 
-def as_expression(self) -> dict[str, Any]:
-return {"objects": [o.as_expression() for o in self.objects]}
-
 def evaluate(self, statuses: dict[str, bool]) -> bool:
 return self.agg_func(x.evaluate(statuses=statuses) for x in 
self.objects)
 
diff --git a/airflow/models/dag.py b/airflow/models/dag.py
index 2ac247d739..ac38ae1eab 100644
--- a/airflow/models/dag.py
+++ b/airflow/models/dag.py
@@ -3034,6 +3034,16 @@ class DAG(LoggingMixin):
 )
 return cls.bulk_write_to_db(dags=dags, session=session)
 
+def simplify_dataset_expression(self, dataset_expression) -> dict | None:
+"""Simplifies a nested dataset expression into a 'any' or 'all' format 
with URIs."""
+if dataset_expression is None:
+return None
+if dataset_expression.get("__type") == "dataset":
+return dataset_expression["__var"]["uri"]
+
+new_key = "any" if dataset_expression["__type"] == "dataset_any" else 
"all"
+return {new_key: [self.simplify_dataset_expression(item) for item in 
dataset_expression["__var"]]}
+
 @classmethod
 @provide_session
 def bulk_write_to_db(
@@ -3053,6 +3063,8 @@ class DAG(LoggingMixin):
 if not dags:
 return
 
+from airflow.serialization.serialized_objects import BaseSerialization 
 # Avoid circular import.
+
 log.info("Sync %s DAGs", len(dags))
 dag_by_ids = {dag.dag_id: dag for dag in dags}
 
@@ -3117,10 +3129,9 @@ class DAG(LoggingMixin):
 )
 orm_dag.schedule_interval = dag.schedule_interval
 orm_dag.timetable_description = dag.timetable.description
-if (dataset_triggers := dag.dataset_triggers) is None:
-orm_dag.dataset_expression = None
-else:
-orm_dag.dataset_expression = dataset_triggers.as_expression()
+orm_dag.dataset_expression = dag.simplify_dataset_expression(
+BaseSerialization.serialize(dag.dataset_triggers)
+)
 
 orm_dag.processor_subdir = processor_subdir
 

(airflow) branch main updated: Allow to override concrete --postgres-version/--mysql-version (#38133)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 1403b1481d Allow to override concrete 
--postgres-version/--mysql-version (#38133)
1403b1481d is described below

commit 1403b1481dba463a11068b183c31d306bf7285ca
Author: Jarek Potiuk 
AuthorDate: Thu Mar 14 19:14:58 2024 +0100

Allow to override concrete --postgres-version/--mysql-version (#38133)

The `BACKEND_VERSION` env variable is now generic way to override
any version of any backend. When you set it before running breeze
and when the value maches possible options for given platform,
the value of concrete backend version will be overridden from the
variable.

This will be helpful to generalize a bit and make reusable test
workflow where `BACKEND_VERSION` will be treated in simiar way
in all tests.
---
 .github/workflows/ci.yml   |  29 ++---
 Dockerfile.ci  |  16 -
 dev/breeze/doc/03_developer_tasks.rst  |  36 -
 dev/breeze/doc/images/output_setup_config.txt  |   2 +-
 dev/breeze/doc/images/output_shell.txt |   2 +-
 dev/breeze/doc/images/output_start-airflow.txt |   2 +-
 dev/breeze/doc/images/output_testing_db-tests.txt  |   2 +-
 .../images/output_testing_integration-tests.txt|   2 +-
 dev/breeze/doc/images/output_testing_tests.txt |   2 +-
 dev/breeze/doc/images/version_information.png  | Bin 0 -> 24933 bytes
 .../src/airflow_breeze/commands/common_options.py  |   7 ++--
 .../src/airflow_breeze/utils/custom_param_types.py |  22 -
 .../src/airflow_breeze/utils/selective_checks.py   |   4 +--
 dev/breeze/src/airflow_breeze/utils/visuals.py |   2 +-
 scripts/docker/entrypoint_ci.sh|  16 -
 15 files changed, 94 insertions(+), 50 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e77ca2cb40..64ecdb93a1 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -1070,14 +1070,14 @@ jobs:
   tests-postgres:
 timeout-minutes: 130
 name: >
-  DB:Postgres${{matrix.postgres-version}},Py${{matrix.python-version}}:
+  DB:Postgres${{matrix.backend-version}},Py${{matrix.python-version}}:
   ${{needs.build-info.outputs.parallel-test-types-list-as-string}}
 runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
 needs: [build-info, wait-for-ci-images]
 strategy:
   matrix:
 python-version: 
"${{fromJson(needs.build-info.outputs.python-versions)}}"
-postgres-version: 
"${{fromJson(needs.build-info.outputs.postgres-versions)}}"
+backend-version: 
"${{fromJson(needs.build-info.outputs.postgres-versions)}}"
 exclude: "${{fromJson(needs.build-info.outputs.postgres-exclude)}}"
   fail-fast: false
 env:
@@ -1088,9 +1088,8 @@ jobs:
   DEBUG_RESOURCES: "${{needs.build-info.outputs.debug-resources}}"
   BACKEND: "postgres"
   PYTHON_MAJOR_MINOR_VERSION: "${{matrix.python-version}}"
-  POSTGRES_VERSION: "${{matrix.postgres-version}}"
-  BACKEND_VERSION: "${{matrix.postgres-version}}"
-  JOB_ID: 
"postgres-${{matrix.postgres-version}}-${{matrix.python-version}}"
+  BACKEND_VERSION: "${{matrix.backend-version}}"
+  JOB_ID: "postgres-${{matrix.backend-version}}-${{matrix.python-version}}"
   ENABLE_COVERAGE: "${{needs.build-info.outputs.run-coverage}}"
 if: needs.build-info.outputs.run-tests == 'true'
 steps:
@@ -1116,7 +1115,7 @@ jobs:
   --parallel-test-types 
"${{needs.build-info.outputs.parallel-test-types-list-as-string}}"
   - name: "Tests ARM Pytest collection: ${{matrix.python-version}}"
 run: breeze testing db-tests --collect-only --remove-arm-packages
-if: matrix.postgres-version == 
needs.build-info.outputs.default-postgres-version
+if: matrix.backend-version == 
needs.build-info.outputs.default-postgres-version
   - name: >
   Post Tests success: Postgres"
 uses: ./.github/actions/post_tests_success
@@ -1144,7 +1143,6 @@ jobs:
   ENABLE_COVERAGE: "${{needs.build-info.outputs.run-coverage}}"
   PYTHON_MAJOR_MINOR_VERSION: 
"${{needs.build-info.outputs.default-python-version}}"
   PYTHON_VERSION: "${needs.build-info.outputs.default-python-version}}"
-  POSTGRES_VERSION: 
"${{needs.build-info.outputs.default-postgres-version}}"
   BACKEND_VERSION: "${{needs.build-info.outputs.default-postgres-version}}"
   DOWNGRADE_SQLALCHEMY: "true"
   JOB_ID: >
@@ -1195,7 +1193,6 @@ jobs

(airflow) branch main updated (0df0e09be0 -> 9e97433dc3)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 0df0e09be0 Fix XComObjectStoreBackend config var in docs (#38142)
 add 9e97433dc3 Better and more complete cleanup for CI jobs (#38152)

No new revisions were added by this update.

Summary of changes:
 .../actions/cleanup-docker/action.yml  |  13 +-
 .github/workflows/build-images.yml |  17 ++-
 .github/workflows/ci-image-build.yml   |   6 +-
 .github/workflows/ci.yml   | 166 -
 .github/workflows/helm-tests.yml   |  10 +-
 .github/workflows/mypy.yml |   6 +-
 .github/workflows/prod-image-build.yml |   6 +-
 .github/workflows/push-image-cache.yml |   7 +-
 .github/workflows/release_dockerhub_image.yml  |  10 +-
 9 files changed, 187 insertions(+), 54 deletions(-)
 copy scripts/ci/docker-compose/backend-mysql-port.yml => 
.github/actions/cleanup-docker/action.yml (80%)

(airflow) branch better-cleanup-for-ci updated (a84dd8f525 -> 04883407e1)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch better-cleanup-for-ci
in repository https://gitbox.apache.org/repos/asf/airflow.git


omit a84dd8f525 Better and more complete cleanup for CI jobs
 add 04883407e1 Better and more complete cleanup for CI jobs

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (a84dd8f525)
\
 N -- N -- N   refs/heads/better-cleanup-for-ci (04883407e1)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../actions/{cleanup => cleanup-docker}/action.yml |   7 +-
 .github/workflows/build-images.yml |  21 +-
 .github/workflows/ci-image-build.yml   |   8 +-
 .github/workflows/ci.yml   | 249 +++--
 .github/workflows/helm-tests.yml   |  14 +-
 .github/workflows/mypy.yml |   8 +-
 .github/workflows/prod-image-build.yml |   8 +-
 .github/workflows/push-image-cache.yml |   7 +-
 .github/workflows/release_dockerhub_image.yml  |  14 +-
 9 files changed, 238 insertions(+), 98 deletions(-)
 rename .github/actions/{cleanup => cleanup-docker}/action.yml (80%)

(airflow) branch better-cleanup-for-ci created (now a84dd8f525)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch better-cleanup-for-ci
in repository https://gitbox.apache.org/repos/asf/airflow.git


  at a84dd8f525 Better and more complete cleanup for CI jobs

This branch includes the following new commits:

 new a84dd8f525 Better and more complete cleanup for CI jobs

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(airflow) 01/01: Better and more complete cleanup for CI jobs

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch better-cleanup-for-ci
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit a84dd8f525ba02ed5649b411739a85ec94e938af
Author: Jarek Potiuk 
AuthorDate: Thu Mar 14 15:35:08 2024 +0100

Better and more complete cleanup for CI jobs

Some of the recent refactors caused a problem that mypy jobs
started to return errors likely resulting from a bad cache volume
remaining after the jobs completed.

This PR extracts cleanup step to a separate, composite action and
uses this action everywhere, including pruning docker including
all containers and volumes to avoid the issue.
---
 .github/actions/cleanup/action.yml|  29 +
 .github/workflows/build-images.yml|  12 +-
 .github/workflows/ci-image-build.yml  |   4 +-
 .github/workflows/ci.yml  | 153 --
 .github/workflows/helm-tests.yml  |   8 +-
 .github/workflows/mypy.yml|   4 +-
 .github/workflows/prod-image-build.yml|   4 +-
 .github/workflows/push-image-cache.yml|   6 +-
 .github/workflows/release_dockerhub_image.yml |   8 +-
 9 files changed, 121 insertions(+), 107 deletions(-)

diff --git a/.github/actions/cleanup/action.yml 
b/.github/actions/cleanup/action.yml
new file mode 100644
index 00..87aa99b663
--- /dev/null
+++ b/.github/actions/cleanup/action.yml
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: 'Cleanup docker and repo'
+description: 'Cleans up docker and repository'
+runs:
+  using: "composite"
+  steps:
+- name: "Cleanup docker"
+  shell: bash
+  run: docker system prune --all --force --volumes
+- name: "Cleanup repo"
+  shell: bash
+  run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
diff --git a/.github/workflows/build-images.yml 
b/.github/workflows/build-images.yml
index c1619874a9..e290711b33 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -113,8 +113,8 @@ jobs:
   }
 }' --jq '.data.node.labels.nodes[]' | jq --slurp -c '[.[].name]' 
>> ${GITHUB_OUTPUT}
 if: github.event_name == 'pull_request_target'
-  - name: Cleanup repo
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
+  - name: Cleanup
+uses: ./.github/actions/cleanup
   - uses: actions/checkout@v4
 with:
   ref: ${{ env.TARGET_COMMIT_SHA }}
@@ -183,8 +183,8 @@ jobs:
   VERSION_SUFFIX_FOR_PYPI: "dev0"
   USE_UV: "true"
 steps:
-  - name: Cleanup repo
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
+  - name: Cleanup
+uses: ./.github/actions/cleanup
   - uses: actions/checkout@v4
 with:
   ref: ${{ needs.build-info.outputs.target-commit-sha }}
@@ -262,8 +262,8 @@ jobs:
   INCLUDE_NOT_READY_PROVIDERS: "true"
   USE_UV: "true"
 steps:
-  - name: Cleanup repo
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
+  - name: Cleanup
+uses: ./.github/actions/cleanup
   - uses: actions/checkout@v4
 with:
   ref: ${{ needs.build-info.outputs.target-commit-sha }}
diff --git a/.github/workflows/ci-image-build.yml 
b/.github/workflows/ci-image-build.yml
index c0ae69a4b8..8a2d81f13e 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -102,8 +102,8 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
   DEFAULT_CONSTRAINTS_BRANCH: ${{ inputs.constraints-branch }}
   VERSION_SUFFIX_FOR_PYPI: "dev0"
 steps:
-  - name: Cleanup repo
-run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
+  - name: Cleanup
+us

(airflow) branch main updated (baec968985 -> 73938f1425)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from baec968985 Upgrade uv to 0.1.20 and stop notifying about new uv 
versions (#38145)
 add 73938f1425 Add starting docker service for ARM instances (#38149)

No new revisions were added by this update.

Summary of changes:
 scripts/ci/images/initialize.sh | 4 
 1 file changed, 4 insertions(+)

(airflow) branch main updated (53ea51a9c4 -> baec968985)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 53ea51a9c4 Separate cache build to a separate workflow called from the 
main ci.yml (#38107)
 add baec968985 Upgrade uv to 0.1.20 and stop notifying about new uv 
versions (#38145)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci.yml   | 22 +-
 Dockerfile |  2 +-
 Dockerfile.ci  |  4 ++--
 pyproject.toml |  2 +-
 .../ci/pre_commit/pre_commit_update_installers.py  |  4 +++-
 5 files changed, 24 insertions(+), 10 deletions(-)

(airflow) branch main updated (c14241b72c -> 53ea51a9c4)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from c14241b72c Simplify dataset serialization code (#38089)
 add 53ea51a9c4 Separate cache build to a separate workflow called from the 
main ci.yml (#38107)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/build-images.yml |  81 -
 .github/workflows/ci.yml   | 198 +
 .github/workflows/push-image-cache.yml | 149 
 .../airflow_breeze/commands/ci_image_commands.py   |  90 ++
 .../commands/production_image_commands.py  | 102 +++
 .../airflow_breeze/utils/docker_command_utils.py   |  62 ---
 .../ci_start_arm_instance_and_connect_to_docker.sh |   3 +-
 scripts/ci/images/initialize.sh|  11 --
 8 files changed, 356 insertions(+), 340 deletions(-)
 create mode 100644 .github/workflows/push-image-cache.yml

(airflow-site) branch main updated: Add: Korea Airflow meetups group (#981)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-site.git


The following commit(s) were added to refs/heads/main by this push:
 new fd61ea5792 Add: Korea Airflow meetups group (#981)
fd61ea5792 is described below

commit fd61ea5792ed2694bab97b19518a7547ae65279c
Author: Yeonguk 
AuthorDate: Thu Mar 14 21:37:15 2024 +0900

Add: Korea Airflow meetups group (#981)

* Add: Korea Airflow meetups group

* Update meetups.json
---
 landing-pages/site/static/meetups.json | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/landing-pages/site/static/meetups.json 
b/landing-pages/site/static/meetups.json
index d1ddefb4b9..31f2c4e744 100644
--- a/landing-pages/site/static/meetups.json
+++ b/landing-pages/site/static/meetups.json
@@ -111,6 +111,13 @@
 "members": 196,
 "url": "https://www.meetup.com/toronto-apache-airflow-meetup/;
   },
+  {
+"city": "Seoul",
+"country": "Korea",
+"date": "",
+"members": 52,
+"url": "https://www.meetup.com/apache-airflow-users-korea/;
+  },
   {
 "city": "Taipei",
 "country": "Taiwan",

(airflow) branch separate-cache-workflow updated (3407bbaae0 -> 35c2b82802)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch separate-cache-workflow
in repository https://gitbox.apache.org/repos/asf/airflow.git


 discard 3407bbaae0 Separate cache build to a separate workflow called from the 
main ci.yml
 add 0727d0e605 Show if dag page filters are active (#38080)
 add 566fee79af Fix Scheduler in daemon mode doesn't create PID at the 
specified location (#38117)
 add 87df297851 Update uv to 0.1.19 (#38132)
 add 3c5bcaa0cc Fix Celery Provider 
`TestFlowerCommand::test_run_command_daemon` (#38143)
 add 35c2b82802 Separate cache build to a separate workflow called from the 
main ci.yml

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (3407bbaae0)
\
 N -- N -- N   refs/heads/separate-cache-workflow (35c2b82802)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 Dockerfile|   2 +-
 Dockerfile.ci |   4 +-
 airflow/cli/commands/daemon_utils.py  |   5 +-
 airflow/www/static/js/dag/nav/FilterBar.tsx   | 103 --
 airflow/www/static/js/dag/useFilters.test.tsx |   5 +-
 airflow/www/static/js/dag/useFilters.tsx  |   1 -
 pyproject.toml|   2 +-
 tests/cli/commands/test_celery_command.py |   1 +
 tests/cli/commands/test_kerberos_command.py   |   1 +
 tests/providers/celery/cli/test_celery_command.py |   1 +
 10 files changed, 86 insertions(+), 39 deletions(-)

(airflow) branch main updated: Fix Celery Provider `TestFlowerCommand::test_run_command_daemon` (#38143)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 3c5bcaa0cc Fix Celery Provider 
`TestFlowerCommand::test_run_command_daemon` (#38143)
3c5bcaa0cc is described below

commit 3c5bcaa0cc7f6b4571e3b9b2eba4b139edc3d4b0
Author: Andrey Anshin 
AuthorDate: Thu Mar 14 14:55:31 2024 +0400

Fix Celery Provider `TestFlowerCommand::test_run_command_daemon` (#38143)
---
 tests/providers/celery/cli/test_celery_command.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/providers/celery/cli/test_celery_command.py 
b/tests/providers/celery/cli/test_celery_command.py
index e960cc7b4e..67c28570ab 100644
--- a/tests/providers/celery/cli/test_celery_command.py
+++ b/tests/providers/celery/cli/test_celery_command.py
@@ -337,6 +337,7 @@ class TestFlowerCommand:
 assert mock_setup_locations.mock_calls == [
 mock.call(
 process="flower",
+pid="/tmp/flower.pid",
 stdout="/tmp/flower-stdout.log",
 stderr="/tmp/flower-stderr.log",
 log="/tmp/flower.log",