This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch ci-upgrade-main
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit e6d773d3e58 [main] CI: Upgrade important CI environment
add df2272c562a Add missing Dutch translations (#68742)
add 771c09c5f20 Fix common.ai durable execution skipping
Toolset-capability tools (#69881)
add 038ece6acf1 Don't deactivate DAG bundles owned by other dag-processors
(#69964)
add c5faeb1555b Authorize Dag reparse against the file's Dags, not the
query-string dag_id (#69471)
add 16754b56dae Restore graph task and group coloring via Chakra palette
tokens (#68760)
add 4f5e4b98437 Skip Go SDK CI jobs for doc-only changes under go-sdk/
(#70021)
add 424f6ec8be2 Enhance/standardize documentation for
`AssetOrTimeSchedule` (#69831)
add c7bc7ac73fc UI: Lazy-mount the pause confirmation dialog on Dag cards
and rows (#70016)
add 8037e9ae30d Skip ts-sdk supervisor schema check for doc-only changes
(#70022)
add b3cffc448ac Use atomic upsert for asset run queue on MySQL to avoid
deadlocks (#69977)
add fb69e2b4a32 Pin task bundle manifest to the dagrun's version (#69941)
add d6a6ceb9ec2 Allow configuring XCom sidecar container security context
(#69613)
add 45015fd4093 Add missing newsfragment to Helm Chart (#70043)
add b628e46a2a8 Update doc (#70047)
add daf347bf61c Fix `@task.llm_branch` import failure on Task SDK-only
workers (#70068)
add ffde96203e7 Reject unsupported require_approval in LLM branch and
schema compare operators (#70069)
add d413861ef7f Fix stale docs.sqlalchemy.org/en/14 pooling link in
config.yml (#69751)
add 41ed94591ea Java SDK: Throw MissingXComException instead of NPE for
absent XComs (#69257)
add 471df2dc220 Rename Kafka Listener to Kafka Event Producer and update
the documentation (#70014)
add 2b48289b230 Set stream=False for Snowflake Cortex Agent requests.
(#69731)
add cad7d0e2a9c Send Airflow CLI logs to stderr for -o commands so
structured output stays parseable (#68598)
add adb4b56a80d Document task subprocess lifecycle and pre-connect log
buffer (#70076)
add 4c671a90aaa Enhancing docs for `HttpEventTrigger` (#70073)
add 0b64e4eae4d Add Amazon ECR repository operators (#69886)
add 5820c02124d Prevent negative trigger queue delay metrics (#69971)
add 54980c13300 Bump the uv-dependency-updates group in /dev/breeze with 2
updates (#70033)
add 40871d0fae2 Bump the github-actions-updates group across 1 directory
with 9 updates (#70040)
add 1724107d560 Fix CI integration tests gated by the wrong platform
(#70015)
add 6bf629a0909 Fix mypy prek hooks skipping all files in dot-directory
(#70064)
add bddb64f8d6a Document 3.3.0 image databricks constraint fix in
Dockerfile changelog (#70083)
add d6246920c30 Fix DocumentLoaderOperator ignoring unknown parser on byte
input (#70071)
add 372025f2549 Send TS SDK logs to stderr on socket disconnect (#69302)
add e182169fbd0 Build Go SDK e2e example bundle natively in CI (#69909)
add 618faa7675d i18n(ko): translate Dag list run-state filter labels
(#69781)
add 69509289c96 Preserve output_type through human approval in LLM
operators (#70075)
add 5c967e7bcab Clarify the dags reserialize command scope (#69810)
add 2de022380ce Add Task SDK, Go and Java SDK execution architecture
diagrams (#69750)
add 582076cc273 Discount CI image-build time from duration-trend alerts
(#69789)
add 6e411d8baee Clarify AssetPartitionDagRun provisional-run docstring
(#69982)
add a7a9feddb56 fix/issue-70020: Adding cap to cloudpickle (#70086)
add 64f9cbddc0e Bring back edge worker metric compatibility with Airflow
3.2 (#67328)
add 21ed9fc97be Replace vars(response) with attribute access in Azure Data
Factory and Synapse operators for SDK v10 compatibility (#69689)
add a9a86ae5b91 Bump eslint (#70032)
add 5337b057d1c cncf-kubernetes: fix potential race condition in
trigger_reentry flow caused by `is_istio_enabled` (#69269)
add b71a6407919 Upgrade http to https in PR template (#70063)
add a76701fb474 Make schema_fields templated in GCSToBigQueryOperator
(#69108)
add e0cb240c64f docs: fix stale Airflow 2.0 references in dev/README.md
(#69961)
add 4c1ce4e3ff3 Refactor get_statsd_logger to use keyword arguments for
clarity and add a test for ipv6 support in StatsD logger (#69632)
add e6d898c2390 Let area:e2e-tests label force the airflow-e2e-tests suite
(#69993)
add 4dc4ebff93f Automate keeping documented "tested with" versions in sync
with the test matrix (#69124)
add 45df400413e Fix Breeze OpenLineage integration gate stuck on stale
Postgres versions (#69264)
add 3561a17b4b6 Bump @visx/shape from 3.12.0 to 4.0.0 in
/airflow-core/src/airflow/ui (#69722)
add 351f3895738 Rename Task and Asset store to Task and Asset state store
(#70097)
add 0cb7f218e04 i18n(ko): add missing translation for
triggerDag.partitionKeyHelp (#70005)
add 60b1848e4c6 Add integration tests for coordinator-mode TypeScript
tasks (#69328)
add 55e2d855307 [main] CI: Upgrade important CI environment
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (e6d773d3e58)
\
N -- N -- N refs/heads/ci-upgrade-main (55e2d855307)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 2 +-
.github/boring-cyborg.yml | 3 +
.github/workflows/additional-prod-image-tests.yml | 16 +
.github/workflows/airflow-e2e-tests.yml | 72 +-
.github/workflows/basic-tests.yml | 4 +-
.github/workflows/ci-amd.yml | 11 +-
.github/workflows/ci-arm.yml | 11 +-
.github/workflows/ci-duration-monitor.yml | 2 +-
.github/workflows/ci-image-checks.yml | 8 +-
.github/workflows/ci-notification.yml | 6 +-
.github/workflows/codeql-analysis.yml | 8 +-
.github/workflows/e2e-flaky-tests-report.yml | 2 +-
.github/workflows/generate-constraints.yml | 2 +-
.github/workflows/java-sdk-release-verify.yml | 4 +-
.github/workflows/k8s-tests.yml | 2 +-
.github/workflows/publish-docs-to-s3.yml | 2 +-
.github/workflows/recheck-old-bug-report.yml | 2 +-
.github/workflows/registry-backfill.yml | 6 +-
.github/workflows/registry-build.yml | 4 +-
.github/workflows/registry-tests.yml | 2 +-
.../scheduled-verify-release-calendar.yml | 2 +-
.github/workflows/stale.yml | 4 +-
.github/workflows/ui-e2e-tests.yml | 2 +-
.../update-constraints-on-push-stable.yml | 2 +-
.github/workflows/update-constraints-on-push.yml | 2 +-
.github/workflows/upgrade-check.yml | 4 +-
.pre-commit-config.yaml | 10 +-
Dockerfile | 2 +-
Dockerfile.ci | 2 +-
.../language-sdks/java.rst | 8 +
.../docs/authoring-and-scheduling/timetable.rst | 11 +-
airflow-core/docs/core-concepts/overview.rst | 85 ++
.../core-concepts/task-and-asset-state-store.rst | 16 +-
airflow-core/docs/howto/custom-operator.rst | 14 +-
airflow-core/docs/howto/usage-cli.rst | 16 +
.../diagram_java_sdk_execution_architecture.md5sum | 1 +
.../diagram_java_sdk_execution_architecture.png | Bin 0 -> 176205 bytes
.../img/diagram_java_sdk_execution_architecture.py | 287 +++++++
.../img/diagram_java_sdk_execution_sequence.md5sum | 1 +
.../img/diagram_java_sdk_execution_sequence.png | Bin 0 -> 189307 bytes
.../img/diagram_java_sdk_execution_sequence.py | 327 ++++++++
...diagram_native_language_sdk_architecture.md5sum | 1 +
.../diagram_native_language_sdk_architecture.png | Bin 0 -> 146251 bytes
.../diagram_native_language_sdk_architecture.py | 258 ++++++
.../diagram_task_sdk_execution_architecture.md5sum | 1 +
.../diagram_task_sdk_execution_architecture.png | Bin 0 -> 184731 bytes
.../img/diagram_task_sdk_execution_architecture.py | 291 +++++++
.../img/diagram_task_sdk_execution_sequence.md5sum | 1 +
.../img/diagram_task_sdk_execution_sequence.png | Bin 0 -> 144211 bytes
.../img/diagram_task_sdk_execution_sequence.py | 313 +++++++
airflow-core/docs/installation/prerequisites.rst | 4 +
airflow-core/newsfragments/68760.significant.rst | 11 +
airflow-core/pyproject.toml | 2 +-
airflow-core/src/airflow/__main__.py | 6 +
.../core_api/datamodels/ui/structure.py | 2 +
.../api_fastapi/core_api/openapi/_private_ui.yaml | 10 +
.../core_api/routes/public/dag_parsing.py | 49 +-
.../src/airflow/api_fastapi/core_api/security.py | 44 +
.../api_fastapi/core_api/services/ui/task_group.py | 9 +
airflow-core/src/airflow/assets/manager.py | 29 +-
airflow-core/src/airflow/cli/utils.py | 15 +
.../src/airflow/dag_processing/bundles/manager.py | 15 +-
airflow-core/src/airflow/dag_processing/manager.py | 4 +-
.../src/airflow/executors/workloads/task.py | 6 +-
.../src/airflow/jobs/scheduler_job_runner.py | 17 +-
.../src/airflow/jobs/triggerer_job_runner.py | 6 +-
airflow-core/src/airflow/models/asset.py | 12 +-
.../airflow/ui/openapi-gen/requests/schemas.gen.ts | 22 +
.../airflow/ui/openapi-gen/requests/types.gen.ts | 2 +
airflow-core/src/airflow/ui/package.json | 4 +-
airflow-core/src/airflow/ui/pnpm-lock.yaml | 250 +++---
.../ui/public/i18n/locales/ko/components.json | 1 +
.../airflow/ui/public/i18n/locales/ko/dags.json | 2 +
.../airflow/ui/public/i18n/locales/nl/assets.json | 17 +
.../airflow/ui/public/i18n/locales/nl/browse.json | 18 +
.../airflow/ui/public/i18n/locales/nl/common.json | 64 +-
.../ui/public/i18n/locales/nl/components.json | 24 +-
.../src/airflow/ui/public/i18n/locales/nl/dag.json | 105 ++-
.../airflow/ui/public/i18n/locales/nl/dags.json | 25 +-
.../ui/public/i18n/locales/nl/dashboard.json | 14 +
.../airflow/ui/public/i18n/locales/nl/hitl.json | 14 +
.../ui/src/components/ConfirmationModal.tsx | 8 +-
.../ui/src/components/Graph/TaskNode.test.tsx | 100 +++
.../airflow/ui/src/components/Graph/TaskNode.tsx | 37 +-
.../ui/src/components/Graph/elkGraphUtils.test.ts | 56 ++
.../ui/src/components/Graph/elkGraphUtils.ts | 6 +
.../airflow/ui/src/components/Graph/nodeColors.ts | 85 ++
.../ui/src/components/Graph/reactflowUtils.ts | 2 +
.../tests/unit/always/test_project_structure.py | 1 -
.../core_api/routes/public/test_dag_parsing.py | 11 +-
.../core_api/routes/public/test_plugins.py | 2 +-
.../core_api/routes/ui/test_structure.py | 79 ++
airflow-core/tests/unit/assets/test_manager.py | 41 +
airflow-core/tests/unit/cli/test_utils.py | 51 +-
.../bundles/test_dag_bundle_manager.py | 56 ++
.../tests/unit/dag_processing/test_manager.py | 14 +
.../tests/unit/executors/test_workloads.py | 55 +-
airflow-core/tests/unit/jobs/test_scheduler_job.py | 37 +-
airflow-core/tests/unit/jobs/test_triggerer_job.py | 10 +-
airflow-core/tests/unit/utils/test_task_group.py | 13 +-
.../docker/ts.yml | 37 +-
.../tests/airflow_e2e_tests/conftest.py | 206 ++++-
.../tests/airflow_e2e_tests/constants.py | 13 +-
.../airflow_e2e_tests/ts_sdk_tests}/__init__.py | 0
.../ts_sdk_tests/test_ts_sdk_dag.py | 134 +++
chart/newsfragments/65666.significant.rst | 3 -
chart/newsfragments/65852.significant.rst | 1 +
chart/newsfragments/66671.significant.rst | 3 +
chart/newsfragments/67681.significant.rst | 3 -
chart/newsfragments/69481.significant.rst | 2 +-
contributing-docs/30_new_language_sdk.rst | 29 +
dev/README.md | 12 +-
dev/README_RELEASE_AIRFLOW.md | 5 +-
dev/breeze/doc/ci/02_images.md | 2 +-
dev/breeze/doc/ci/04_selective_checks.md | 11 +
.../doc/images/output_ci_selective-check.svg | 46 +-
.../doc/images/output_ci_selective-check.txt | 2 +-
.../images/output_testing_airflow-e2e-tests.svg | 2 +-
.../images/output_testing_airflow-e2e-tests.txt | 2 +-
.../src/airflow_breeze/commands/ci_commands.py | 14 +
.../airflow_breeze/commands/ci_commands_config.py | 1 +
.../commands/release_management_commands.py | 4 +-
.../airflow_breeze/commands/testing_commands.py | 1 +
dev/breeze/src/airflow_breeze/global_constants.py | 13 +-
.../airflow_breeze/utils/docker_command_utils.py | 8 +-
.../src/airflow_breeze/utils/selective_checks.py | 119 +--
dev/breeze/tests/conftest.py | 8 -
dev/breeze/tests/test_docker_command_utils.py | 82 ++
dev/breeze/tests/test_selective_checks.py | 408 +++------
dev/breeze/uv.lock | 98 +--
devel-common/pyproject.toml | 2 +-
docker-stack-docs/changelog.rst | 7 +
docs/spelling_wordlist.txt | 4 +
.../org/apache/airflow/sdk/BuilderProcessor.kt | 25 +-
.../kotlin/org/apache/airflow/sdk/BuilderTest.kt | 83 +-
.../main/kotlin/org/apache/airflow/sdk/Client.kt | 16 +
.../kotlin/org/apache/airflow/sdk/ClientTest.kt | 12 +
.../test_kubernetes_pod_operator.py | 1 +
providers/amazon/docs/operators/ecr.rst | 90 ++
providers/amazon/provider.yaml | 5 +
.../airflow/providers/amazon/aws/operators/ecr.py | 231 +++++
.../airflow/providers/amazon/get_provider_info.py | 5 +
.../amazon/tests/system/amazon/aws/example_ecr.py | 99 +++
.../aws/executors/batch/test_batch_executor.py | 2 +
.../amazon/aws/executors/ecs/test_ecs_executor.py | 2 +
.../tests/unit/amazon/aws/operators/test_ecr.py | 208 +++++
providers/apache/kafka/docs/configurations-ref.rst | 42 +-
providers/apache/kafka/provider.yaml | 16 +-
providers/apache/kafka/pyproject.toml | 2 +-
.../providers/apache/kafka/get_provider_info.py | 12 +-
.../plugins/{listener.py => event_producer.py} | 33 +-
.../{test_listener.py => test_event_producer.py} | 14 +-
.../{test_listener.py => test_event_producer.py} | 176 ++--
.../kubernetes/docs/connections/kubernetes.rst | 11 +
providers/cncf/kubernetes/docs/operators.rst | 6 +
providers/cncf/kubernetes/provider.yaml | 6 +
.../providers/cncf/kubernetes/get_provider_info.py | 4 +
.../providers/cncf/kubernetes/hooks/kubernetes.py | 13 +
.../providers/cncf/kubernetes/operators/pod.py | 22 +-
.../cncf/kubernetes/utils/xcom_sidecar.py | 3 +
.../decorators/test_kubernetes_commons.py | 1 +
.../unit/cncf/kubernetes/hooks/test_kubernetes.py | 34 +
.../unit/cncf/kubernetes/operators/test_pod.py | 55 ++
.../cncf/kubernetes/utils/test_xcom_sidecar.py | 71 ++
.../providers/common/ai/decorators/llm_branch.py | 2 +-
.../airflow/providers/common/ai/durable/storage.py | 16 +-
.../airflow/providers/common/ai/mixins/approval.py | 8 +-
.../airflow/providers/common/ai/operators/agent.py | 85 +-
.../common/ai/operators/document_loader.py | 2 +
.../providers/common/ai/operators/llm_branch.py | 2 +
.../common/ai/operators/llm_schema_compare.py | 2 +
.../providers/common/ai/utils/output_type.py | 20 +-
.../tests/unit/common/ai/durable/test_storage.py | 29 +-
.../tests/unit/common/ai/mixins/test_approval.py | 13 +-
.../tests/unit/common/ai/operators/test_agent.py | 132 ++-
.../common/ai/operators/test_document_loader.py | 5 +
.../ai/tests/unit/common/ai/operators/test_llm.py | 19 +
.../unit/common/ai/operators/test_llm_branch.py | 9 +
.../common/ai/operators/test_llm_schema_compare.py | 5 +
.../tests/unit/common/ai/utils/test_output_type.py | 15 +-
providers/edge3/docs/edge_executor.rst | 27 +
.../airflow/providers/edge3/models/edge_worker.py | 25 +-
.../tests/unit/edge3/models/test_edge_worker.py | 64 ++
.../unit/edge3/worker_api/routes/test_worker.py | 17 +-
.../google/cloud/transfers/gcs_to_bigquery.py | 2 +
.../google/cloud/transfers/test_gcs_to_bigquery.py | 31 +
providers/http/docs/triggers.rst | 44 +-
.../microsoft/azure/operators/data_factory.py | 2 +-
.../providers/microsoft/azure/operators/synapse.py | 4 +-
.../microsoft/azure/operators/test_data_factory.py | 26 +
.../unit/microsoft/azure/operators/test_synapse.py | 46 +
.../snowflake/hooks/snowflake_cortex_agent.py | 5 +-
.../snowflake/hooks/test_snowflake_cortex_agent.py | 3 +
pyproject.toml | 6 +-
scripts/ci/analyze_ci_job_durations.py | 276 ++++--
scripts/ci/docker-compose/integration-kafka.yml | 8 +-
scripts/ci/prek/check_ci_workflows_in_sync.py | 7 +-
scripts/ci/prek/common_prek_utils.py | 41 +-
scripts/ci/prek/extract_permissions.py | 1 +
scripts/ci/prek/mypy_folder.py | 3 +-
...un_mypy_full_dist_local_venv_or_breeze_in_ci.py | 3 +-
scripts/ci/prek/update_tested_versions.py | 209 +++++
scripts/tests/ci/prek/test_common_prek_utils.py | 56 ++
.../tests/ci/prek/test_update_tested_versions.py | 123 +++
scripts/tests/ci/test_analyze_ci_job_durations.py | 233 ++++-
scripts/tools/setup_breeze | 2 +-
.../observability/metrics/statsd_logger.py | 7 +-
.../tests/observability/metrics/test_stats.py | 7 +
.../sdk/definitions/decorators/__init__.pyi | 6 +
task-sdk/src/airflow/sdk/definitions/taskgroup.py | 16 +-
ts-sdk/example/.gitignore | 1 +
ts-sdk/src/coordinator/log-channel.ts | 61 +-
ts-sdk/tests/coordinator/log-channel.test.ts | 121 ++-
uv.lock | 934 ++++++++++-----------
214 files changed, 6944 insertions(+), 1716 deletions(-)
create mode 100644
airflow-core/docs/img/diagram_java_sdk_execution_architecture.md5sum
create mode 100644
airflow-core/docs/img/diagram_java_sdk_execution_architecture.png
create mode 100644
airflow-core/docs/img/diagram_java_sdk_execution_architecture.py
create mode 100644
airflow-core/docs/img/diagram_java_sdk_execution_sequence.md5sum
create mode 100644
airflow-core/docs/img/diagram_java_sdk_execution_sequence.png
create mode 100644 airflow-core/docs/img/diagram_java_sdk_execution_sequence.py
create mode 100644
airflow-core/docs/img/diagram_native_language_sdk_architecture.md5sum
create mode 100644
airflow-core/docs/img/diagram_native_language_sdk_architecture.png
create mode 100644
airflow-core/docs/img/diagram_native_language_sdk_architecture.py
create mode 100644
airflow-core/docs/img/diagram_task_sdk_execution_architecture.md5sum
create mode 100644
airflow-core/docs/img/diagram_task_sdk_execution_architecture.png
create mode 100644
airflow-core/docs/img/diagram_task_sdk_execution_architecture.py
create mode 100644
airflow-core/docs/img/diagram_task_sdk_execution_sequence.md5sum
create mode 100644
airflow-core/docs/img/diagram_task_sdk_execution_sequence.png
create mode 100644 airflow-core/docs/img/diagram_task_sdk_execution_sequence.py
create mode 100644 airflow-core/newsfragments/68760.significant.rst
create mode 100644
airflow-core/src/airflow/ui/src/components/Graph/TaskNode.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/components/Graph/nodeColors.ts
copy scripts/ci/docker-compose/integration-mongo.yml =>
airflow-e2e-tests/docker/ts.yml (52%)
copy {airflow-core/src/airflow/_shared =>
airflow-e2e-tests/tests/airflow_e2e_tests/ts_sdk_tests}/__init__.py (100%)
create mode 100644
airflow-e2e-tests/tests/airflow_e2e_tests/ts_sdk_tests/test_ts_sdk_dag.py
delete mode 100644 chart/newsfragments/65666.significant.rst
create mode 100644 chart/newsfragments/65852.significant.rst
create mode 100644 chart/newsfragments/66671.significant.rst
delete mode 100644 chart/newsfragments/67681.significant.rst
create mode 100644 providers/amazon/docs/operators/ecr.rst
create mode 100644
providers/amazon/src/airflow/providers/amazon/aws/operators/ecr.py
create mode 100644 providers/amazon/tests/system/amazon/aws/example_ecr.py
create mode 100644 providers/amazon/tests/unit/amazon/aws/operators/test_ecr.py
rename
providers/apache/kafka/src/airflow/providers/apache/kafka/plugins/{listener.py
=> event_producer.py} (92%)
rename
providers/apache/kafka/tests/integration/apache/kafka/plugins/{test_listener.py
=> test_event_producer.py} (92%)
rename
providers/apache/kafka/tests/unit/apache/kafka/plugins/{test_listener.py =>
test_event_producer.py} (72%)
create mode 100644
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/utils/test_xcom_sidecar.py
create mode 100644 providers/edge3/tests/unit/edge3/models/test_edge_worker.py
create mode 100755 scripts/ci/prek/update_tested_versions.py
create mode 100644 scripts/tests/ci/prek/test_update_tested_versions.py
