This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch ci-upgrade-main
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit f069b318fd0 Merge branch 'main' into ci-upgrade-main
omit b65580820a0 [main] CI: Upgrade important CI environment
add 658bd8a2885 Add tests for Keycloak token models (#70689)
add 221da637bd2 Breeze: Make postgres driver explicit and version-aware
(#70496)
add 8596a0fb643 Decouple SparkSubmitOperator resumable deployment backends
(#68679)
add 682cf06525c Revert AWS services toolset for common AI provider
(#70695)
add a15ef5081d9 Make Redshift system-test clusters non-public in
sql_to_s3/s3_to_sql (#70705)
add ffe077e8cd3 Add `DmsReloadTablesOperator` (#70569)
add 13d7df70ecd Fix DbtCloudRunJobOperator having false failures during
deferred polling (#70581)
add 07f5458df87 Add airflowctl dags clear command (#68706)
add abd837a382c Java SDK: Change coordinator comm to use multiplexing
(#69254)
add dbb5c7ffff9 Fail Neo4jOperator argument mistakes at Dag parse time
(#70508)
add b3014293344 Raise the datamodel-code-generator floor for the codegen
toolchain (#70670)
add d1bd535107d Restore Dag-parse-time validation for Databricks Repos
operator arguments (#70551)
add f781f8b8785 Keep S3DeleteObjectsOperator validation in __init__,
narrow to ValueError (#70359)
add 4cb967fd5c2 Add /ui/assets endpoint with last-event sort and use it in
the Assets list (#69940)
add 4f6186db09f Fix trigger kwargs decoding for asset watcher triggers
(#65992)
add 193b2e36b06 Document jwt_secret _secret and LocalFilesystemBackend
config support (#67940)
add ee8bb6d2649 Docs: clarify Databricks retry_delay uses exponential
backoff (#68622)
add 3fb080bf487 Add run_type tag to dagrun.duration.failed timeout metric
(#67765)
add d11422ee0c6 Fix getsection() dropping env-var overrides on
lazy-registered provider sections (#68895)
add 84ac434aaa1 Improve SQLite test DB health checks (#68021)
add f5f431378f1 Document effect of state-store cleanup for operators with
durable execution (#70721)
add 3de732ab9b2 Fix BashOperator script detection after templating (#70369)
add 724aaade2f6 Bump structlog>=26.1.0 and croniter>=6.2.2 to fix memory
leaks (#70694)
add ba105ea98b6 Add metrics for how often a dag gets serialised. (#68906)
add a69ef4c4928 Add airflowctl tasks failed-deps command (#69397)
add 521ad026ee4 cli: Fix TypeError in 'airflow db shell' when database
name is missing (#68913)
add ee5f3798e8d Emit GCSListObjectsOperator delimiter deprecation warning
after rendering (#70533)
add 13a3cd73173 i18n(ko): add missing translations(Jul 22) (#70211)
add 9fecfd5d995 Fix config update --option/--ignore-option never matching
options (#70240)
add 9481c8e2239 Fix KubernetesPodOperator dry_run requiring live
Kubernetes API client (#70234)
add f0d50bdb178 [AIP-94] Mark tasks CLI commands as migrated to airflowctl
(#69274)
add bc033da913d Handle ValueError for `trigger_dag_run` API request
(#67601)
add 20b9a85c557 Add custom metrics section in metrics docs (#69371)
add 61a0779687d Keep HTTPRoute schema definitions consistent (#70673)
add eca48163048 Add `--min-completed-minutes` to `cleanup-pods` to prevent
KPO race condition (#70595)
add 77838ed374f Validate common AI tool-call arguments to enable
pydantic-ai retries (#70096)
add 9f887794ef7 Fix incorrect code samples in the Deadline Alerts docs
(#70709)
add b74616b500e Helm: Scope KEDA_DB_CONN to worker and triggerer pods
(#70213)
add c86ba34f508 Documenting effect if task state store cleanup for KPO
(#70722)
add 92489784eeb Fix airflowctl failed-deps test fixture missing required
fields (#70776)
add cea9e6efdb5 Document effect of state-store cleanup in
ResumableJobMixin (#70718)
add e9dadbced43 Remove duplicate code (#70785)
add ecd4d951da1 Avoid duplicate trigger-rule upstream-count queries per
scheduling pass (#67672)
add 9d07d7c47ba Bump the edge-ui-package-updates group across 1 directory
with 31 updates (#70616)
add 4338fb82c38 i18n(Ko): add missing translations (Jul 30) (#70767)
add ce469bd89b2 i18n(ko): add missing translations(Jul 30) (#70765)
add 694e5b50352 Redact secrets from LLMRetryPolicy classification prompts
(#70229)
add fbac2f5ce19 Add a Retry Policies category to the provider registry
(#70499)
add 7996da357c1 [main] CI: Upgrade important CI environment
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f069b318fd0)
\
N -- N -- N refs/heads/ci-upgrade-main (7996da357c1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.pre-commit-config.yaml | 2 +-
.../logging-monitoring/metrics.rst | 71 +-
airflow-core/docs/howto/deadline-alerts.rst | 17 +-
airflow-core/docs/howto/set-config.rst | 1 +
.../local-filesystem-secrets-backend.rst | 47 +-
airflow-core/newsfragments/67672.improvement.rst | 1 +
airflow-core/newsfragments/67765.bugfix.rst | 1 +
airflow-core/newsfragments/70240.bugfix.rst | 1 +
airflow-core/pyproject.toml | 6 +-
.../src/airflow/api_fastapi/common/db/assets.py | 54 +
.../src/airflow/api_fastapi/common/parameters.py | 6 +
.../api_fastapi/core_api/datamodels/assets.py | 29 +
.../api_fastapi/core_api/openapi/_private_ui.yaml | 415 +++
.../api_fastapi/core_api/routes/ui/assets.py | 97 +-
.../airflow/api_fastapi/execution_api/AGENTS.md | 4 +-
.../api_fastapi/execution_api/routes/dag_runs.py | 8 +
.../src/airflow/cli/commands/config_command.py | 5 +-
.../src/airflow/cli/commands/db_command.py | 6 +
.../src/airflow/cli/commands/task_command.py | 3 +
.../src/airflow/jobs/scheduler_job_runner.py | 2 +-
airflow-core/src/airflow/models/dagrun.py | 16 +-
airflow-core/src/airflow/models/serialized_dag.py | 3 +
airflow-core/src/airflow/provider.yaml.schema.json | 24 +
.../src/airflow/secrets/local_filesystem.py | 1 +
airflow-core/src/airflow/serialization/decoders.py | 23 +-
airflow-core/src/airflow/ti_deps/dep_context.py | 31 +
.../src/airflow/ti_deps/deps/trigger_rule_dep.py | 42 +-
.../src/airflow/ui/openapi-gen/queries/common.ts | 21 +
.../ui/openapi-gen/queries/ensureQueryData.ts | 41 +
.../src/airflow/ui/openapi-gen/queries/prefetch.ts | 41 +
.../src/airflow/ui/openapi-gen/queries/queries.ts | 41 +
.../src/airflow/ui/openapi-gen/queries/suspense.ts | 41 +
.../ui/openapi-gen/requests/services.gen.ts | 53 +-
.../airflow/ui/openapi-gen/requests/types.gen.ts | 60 +
.../airflow/ui/public/i18n/locales/en/assets.json | 5 +-
.../airflow/ui/public/i18n/locales/ko/assets.json | 5 +-
.../airflow/ui/public/i18n/locales/ko/common.json | 9 +-
.../ui/public/i18n/locales/ko/components.json | 1 +
.../src/airflow/ui/public/i18n/locales/ko/dag.json | 9 +-
.../airflow/ui/public/i18n/locales/ko/dags.json | 10 +-
.../src/airflow/ui/src/constants/filterConfigs.tsx | 17 +-
.../src/airflow/ui/src/constants/searchParams.ts | 4 +
.../airflow/ui/src/pages/AssetsList/AssetsList.tsx | 84 +-
.../src/airflow/ui/src/utils/useFiltersHandler.ts | 2 +
airflow-core/src/airflow/utils/db_manager.py | 8 +
.../tests/unit/always/test_project_structure.py | 1 -
.../api_fastapi/core_api/routes/ui/test_assets.py | 286 +-
.../execution_api/versions/head/test_dag_runs.py | 26 +
.../unit/cli/commands/test_command_deprecations.py | 4 +
.../tests/unit/cli/commands/test_config_command.py | 33 +
.../tests/unit/cli/commands/test_db_command.py | 17 +
.../cli/commands/test_rotate_fernet_key_command.py | 2 +-
airflow-core/tests/unit/core/test_configuration.py | 30 +
.../tests/unit/dag_processing/test_processor.py | 8 +
airflow-core/tests/unit/jobs/test_scheduler_job.py | 38 +
airflow-core/tests/unit/jobs/test_triggerer_job.py | 102 +-
.../tests/unit/models/test_serialized_dag.py | 85 +-
.../unit/ti_deps/deps/test_trigger_rule_dep.py | 242 ++
airflow-core/tests/unit/utils/test_db_manager.py | 8 +
.../airflowctl_tests/test_airflowctl_commands.py | 4 +
airflow-ctl/docs/images/command_hashes.txt | 6 +-
airflow-ctl/docs/images/output_dags.svg | 138 +-
airflow-ctl/docs/images/output_taskinstances.svg | 74 +-
airflow-ctl/docs/images/output_tasks.svg | 74 +-
airflow-ctl/pyproject.toml | 4 +-
.../airflowctl/api/datamodels/auth_generated.py | 2 +-
.../src/airflowctl/api/datamodels/generated.py | 391 ++-
airflow-ctl/src/airflowctl/api/operations.py | 65 +-
airflow-ctl/src/airflowctl/ctl/cli_config.py | 92 +-
.../src/airflowctl/ctl/commands/dag_command.py | 166 +-
.../src/airflowctl/ctl/commands/task_command.py | 52 +
airflow-ctl/src/airflowctl/ctl/help_texts.yaml | 3 +
.../tests/airflow_ctl/api/test_operations.py | 212 +-
.../ctl/commands/test_connections_command.py | 4 +-
.../airflow_ctl/ctl/commands/test_dag_command.py | 314 +-
.../airflow_ctl/ctl/commands/test_task_command.py | 349 +++
.../ctl/commands/test_variable_command.py | 5 +-
.../tests/airflow_ctl/ctl/test_cli_config.py | 2 +
chart/templates/_helpers.yaml | 21 +-
.../templates/triggerer/triggerer-deployment.yaml | 3 +
chart/templates/workers/worker-deployment.yaml | 3 +
chart/tests/helm_tests/other/test_keda.py | 44 +
chart/values.schema.json | 287 +-
...elease-management_classify-provider-changes.svg | 16 +-
...elease-management_classify-provider-changes.txt | 2 +-
...management_generate-issue-content-providers.svg | 16 +-
...management_generate-issue-content-providers.txt | 2 +-
...ease-management_generate-providers-metadata.svg | 18 +-
...ease-management_generate-providers-metadata.txt | 2 +-
...e-management_prepare-provider-distributions.svg | 16 +-
...e-management_prepare-provider-distributions.txt | 2 +-
...e-management_prepare-provider-documentation.svg | 16 +-
...e-management_prepare-provider-documentation.txt | 2 +-
.../commands/release_management_commands.py | 2 +-
.../src/airflow_breeze/params/shell_params.py | 19 +
dev/breeze/tests/test_selective_checks.py | 16 +-
dev/breeze/tests/test_shell_params.py | 36 +
dev/breeze/uv.lock | 6 +-
dev/registry/registry_tools/types.py | 8 +
devel-common/src/tests_common/pytest_plugin.py | 181 +-
.../tests/unit/tests_common/test_pytest_plugin.py | 463 +++
generated/known_airflow_exceptions.txt | 2 +-
.../main/kotlin/org/apache/airflow/sdk/Server.kt | 29 +-
.../org/apache/airflow/sdk/execution/Comm.kt | 170 +-
.../org/apache/airflow/sdk/execution/Frame.kt | 18 +-
.../org/apache/airflow/sdk/execution/Logger.kt | 8 +-
.../kotlin/org/apache/airflow/sdk/ServerTest.kt | 142 +
.../org/apache/airflow/sdk/execution/CommTest.kt | 210 +-
providers/amazon/docs/operators/dms.rst | 32 +
.../docs/operators/redshift/redshift_data.rst | 7 +
.../airflow/providers/amazon/aws/operators/dms.py | 130 +
.../airflow/providers/amazon/aws/operators/s3.py | 8 +-
.../airflow/providers/amazon/aws/triggers/dms.py | 129 +-
.../airflow/providers/amazon/aws/waiters/dms.json | 122 +
.../amazon/tests/system/amazon/aws/example_dms.py | 193 +-
.../tests/system/amazon/aws/example_s3_to_sql.py | 1 +
.../tests/system/amazon/aws/example_sql_to_s3.py | 1 +
.../tests/unit/amazon/aws/hooks/test_base_aws.py | 6 +
.../amazon/tests/unit/amazon/aws/hooks/test_dms.py | 1 -
.../tests/unit/amazon/aws/operators/test_dms.py | 219 ++
.../tests/unit/amazon/aws/operators/test_s3.py | 2 +-
.../tests/unit/amazon/aws/triggers/test_dms.py | 153 +
.../tests/unit/amazon/aws/waiters/test_dms.py | 101 +
.../apache/spark/operators/spark_submit.py | 329 ++-
providers/cncf/kubernetes/docs/operators.rst | 9 +
.../providers/cncf/kubernetes/cli/definition.py | 12 +-
.../cncf/kubernetes/cli/kubernetes_command.py | 45 +-
.../providers/cncf/kubernetes/operators/pod.py | 19 +-
.../cncf/kubernetes/cli/test_kubernetes_command.py | 180 ++
.../unit/cncf/kubernetes/operators/test_pod.py | 28 +
providers/common/ai/docs/examples.rst | 4 +-
providers/common/ai/docs/index.rst | 5 +-
providers/common/ai/docs/retry_policies.rst | 22 +-
providers/common/ai/docs/toolsets.rst | 74 +-
providers/common/ai/provider.yaml | 8 +-
providers/common/ai/pyproject.toml | 9 -
.../common/ai/example_dags/example_aws_toolset.py | 54 -
.../airflow/providers/common/ai/policies/retry.py | 30 +-
.../providers/common/ai/toolsets/__init__.py | 10 +-
.../airflow/providers/common/ai/toolsets/aws.py | 509 ----
.../providers/common/ai/toolsets/datafusion.py | 7 +-
.../airflow/providers/common/ai/toolsets/hook.py | 30 +-
.../common/ai/toolsets/langchain_bridge.py | 42 +-
.../airflow/providers/common/ai/toolsets/sql.py | 7 +-
.../providers/common/ai/utils/tool_definition.py | 74 +-
.../ai/tests/unit/common/ai/policies/test_retry.py | 44 +
.../ai/tests/unit/common/ai/toolsets/test_aws.py | 359 ---
.../unit/common/ai/toolsets/test_datafusion.py | 19 +
.../ai/tests/unit/common/ai/toolsets/test_hook.py | 59 +-
.../common/ai/toolsets/test_langchain_bridge.py | 46 +
.../ai/tests/unit/common/ai/toolsets/test_sql.py | 18 +
.../unit/common/ai/utils/test_tool_definition.py | 134 +-
.../databricks/docs/operators/repos_create.rst | 2 +-
.../databricks/docs/operators/repos_delete.rst | 2 +-
.../databricks/docs/operators/repos_update.rst | 2 +-
providers/databricks/docs/operators/run_now.rst | 6 +
providers/databricks/docs/operators/submit_run.rst | 6 +
.../providers/databricks/hooks/databricks.py | 5 +-
.../providers/databricks/hooks/databricks_base.py | 5 +-
.../providers/databricks/operators/databricks.py | 33 +-
.../databricks/operators/databricks_repos.py | 46 +-
.../providers/databricks/triggers/databricks.py | 8 +-
.../databricks/operators/test_databricks_repos.py | 29 +-
.../src/airflow/providers/dbt/cloud/hooks/dbt.py | 45 +-
.../cloud/tests/unit/dbt/cloud/hooks/test_dbt.py | 101 +
.../providers/edge3/plugins/www/package.json | 60 +-
.../providers/edge3/plugins/www/pnpm-lock.yaml | 3044 ++++++++++----------
providers/google/docs/operators/cloud/bigquery.rst | 6 +
.../providers/google/cloud/operators/gcs.py | 13 +-
.../tests/unit/google/cloud/operators/test_gcs.py | 4 +-
.../keycloak/auth_manager/datamodels}/__init__.py | 0
.../keycloak/auth_manager/datamodels/test_token.py | 120 +
.../src/airflow/providers/neo4j/operators/neo4j.py | 8 +-
.../neo4j/tests/unit/neo4j/operators/test_neo4j.py | 16 +-
.../unit/openlineage/plugins/test_listener.py | 14 +-
providers/snowflake/docs/operators/snowflake.rst | 7 +
.../airflow/providers/standard/decorators/bash.py | 1 -
.../airflow/providers/standard/operators/bash.py | 5 +-
.../tests/unit/standard/operators/test_bash.py | 17 +
registry/src/_data/types.json | 5 +
registry/src/css/main.css | 1 +
registry/src/css/tokens.css | 1 +
scripts/ci/docker-compose/backend-postgres.yml | 4 +-
.../ci/prek/validate_operators_init_exemptions.txt | 3 -
.../in_container/run_provider_yaml_files_check.py | 48 +-
.../src/airflow_shared/configuration/parser.py | 2 +-
shared/dagnode/pyproject.toml | 2 +-
shared/listeners/pyproject.toml | 2 +-
shared/logging/pyproject.toml | 2 +-
.../src/airflow_shared/logging/structlog.py | 28 +-
shared/observability/pyproject.toml | 2 +-
.../observability/metrics/metrics_template.yaml | 7 +
shared/providers_discovery/pyproject.toml | 2 +-
task-sdk/docs/resumable-job-mixin.rst | 8 +
task-sdk/pyproject.toml | 4 +-
.../src/airflow/sdk/api/datamodels/_generated.py | 106 +-
.../airflow/sdk/execution_time/schema/schema.json | 494 ++--
task-sdk/tests/conftest.py | 2 +
task-sdk/tests/task_sdk/api/test_client.py | 14 +
task-sdk/tests/task_sdk/bases/test_hook.py | 2 +
.../tests/task_sdk/definitions/test_connection.py | 11 +-
.../execution_time/test_callback_supervisor.py | 22 +-
.../tests/task_sdk/execution_time/test_comms.py | 1 +
.../tests/task_sdk/execution_time/test_context.py | 50 +-
.../task_sdk/execution_time/test_context_cache.py | 17 +-
.../tests/task_sdk/execution_time/test_secrets.py | 3 +
.../task_sdk/execution_time/test_supervisor.py | 80 +-
.../task_sdk/execution_time/test_task_runner.py | 9 +
ts-sdk/src/generated/supervisor.ts | 194 +-
uv.lock | 101 +-
210 files changed, 9934 insertions(+), 4156 deletions(-)
create mode 100644 airflow-core/newsfragments/67672.improvement.rst
create mode 100644 airflow-core/newsfragments/67765.bugfix.rst
create mode 100644 airflow-core/newsfragments/70240.bugfix.rst
create mode 100644 airflow-core/src/airflow/api_fastapi/common/db/assets.py
create mode 100644 devel-common/tests/unit/tests_common/test_pytest_plugin.py
create mode 100644
java-sdk/sdk/src/test/kotlin/org/apache/airflow/sdk/ServerTest.kt
delete mode 100644
providers/common/ai/src/airflow/providers/common/ai/example_dags/example_aws_toolset.py
delete mode 100644
providers/common/ai/src/airflow/providers/common/ai/toolsets/aws.py
delete mode 100644
providers/common/ai/tests/unit/common/ai/toolsets/test_aws.py
copy {airflow-core/src/airflow/_shared =>
providers/keycloak/tests/unit/keycloak/auth_manager/datamodels}/__init__.py
(100%)
create mode 100644
providers/keycloak/tests/unit/keycloak/auth_manager/datamodels/test_token.py
