This is an automated email from the ASF dual-hosted git repository.
ephraimbuddy pushed a change to branch dag-processor-work-delivery
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard ed8e8b74bf0 Tighten test coverage of Dag processor requested-work
delivery
discard 322fdd84fe6 Fix Dag processor maintenance logs and stale Dag warning
cleanup
discard 703c4482689 Carry publication receipts through Dag processor
maintenance
discard 34e2f9dee76 Preserve Dag cleanup and exact priority reparse targets
discard 8e4cd50bd33 Fix Dag processor requested-work cleanup and delivery
discard 22dc859ffe9 Adapt legacy Dag upgrade regression to bundle discovery
results
discard d013c6fc32f Align Dag processor claim migrations with main
discard a8876fcf8d6 Test legacy Dag deactivation after bundle-name backfill
discard 82d25c08cb2 Fix Dag processor work retries and metadata cleanup
discard 3293711654c Retain Dag processor work until delivery is acknowledged
discard 1232175f661 Cover publication API rejections and clarify its docs and
names
discard 29254b8b730 Reject Dags without a timetable or with incomplete
deadline alerts
discard 8d08c505ccd Preserve compact source publications and actionable Dag
diagnostics
discard 8c5e46d88f9 Clean asset references for stale Dags removed from bundles
discard eb57eb39899 Align API-mode Dag publication with DB-mode cleanup
discard 9edca876774 Allow Dag processors to publish metadata through the
Execution API
discard ef49963c778 Describe the git provider's refresh-error probe accurately
discard df5919166b7 Stop repeated bundle drains and keep cleanup overrides
working
discard 509cb5dcaa0 Keep Dag parsing and metrics healthy during bundle outages
discard 78401bac6a7 Revise Dag bundle refresh compatibility guidance
discard 36be438af12 Clarify bundle refresh compatibility and simplify its
regression test
discard f11a0cc5b37 Keep Git bundles importable on Airflow 3.4
discard 040e0c587b6 Keep Dag parsing responsive during bundle refresh failures
discard 73a139115df Make the bundle refresh regression test independent of
uptime
discard 8bb1297d4ba Avoid parsing stalls while a bundle refresh waits for
imports
discard 63c8c9bb9a8 Keep Dag parsing results aligned with refreshed bundle
sources
add 38b3c45415b Make the bundle name less prominent in the Dag import
error modals (#75958)
add 92bbf67c362 Refuse to publish constraints when the branch moved during
generation (#75989)
add 7a6acca54be Allow selecting several task instances in the Grid view
(#75221)
add 37b6b04e723 Add name filtering to the Deadline Alerts API (#74127)
add 083b82d7c4d Size CI test jobs on RunsOn runners to the parallelism
they can use (#75833)
add 6e6ae1ab7d2 UI: Show a loop's iterations and why each one decided
(#75984)
add 212fe09e1d0 UI: Stop Recent Configurations flashing on the trigger
form (#75991)
add 0f2563f232b Document the zh-TW translation workflow (#75702)
add 406474723a2 Add an end-to-end test for a Dag written in YAML (#75928)
add 8401a190fa8 Run the lang-SDK native Dag tests on Kubernetes (#75930)
add c6a077b0efe Serialise the workflows that publish shared state (#75996)
add 9e662c23fc0 Fix stale queued asset rows in scheduler cleanup and
readiness (#76002)
add 56b44decc8f Sync the session's copy with the DB for removed task
instances when scheduling (#75974)
add 1df83c5dff6 Type the ORM declarative base as a value of type Any
(#76025)
add 155560df668 Store and show the retry deadline; start retries with
empty run fields (#76027)
add 5af4c5bf12d Freeze the GC before the triggerer's forks so memory stays
flat (#76016)
add 5cc2413925a Honor Airflow logging levels in TypeScript SDK (#76005)
add acb2208f15f Let the ECS executor start tasks in parallel (#75907)
add 48435351c42 Pin the hardened base images to digests the CI has passed
on (#76023)
add a3a4f4f168e Stop bundling the ELK worker twice in the UI (#75864)
add 7b17f64bc2d Keep Dag parsing results aligned with refreshed bundle
sources
add 89fcad03b72 Avoid parsing stalls while a bundle refresh waits for
imports
add f8e30a00b63 Make the bundle refresh regression test independent of
uptime
add 5cb241c3b35 Keep Dag parsing responsive during bundle refresh failures
add 523bb5804e9 Keep Git bundles importable on Airflow 3.4
add d503ddd8ba0 Clarify bundle refresh compatibility and simplify its
regression test
add f37b1a9f5b6 Revise Dag bundle refresh compatibility guidance
add e7a502d87a4 Keep Dag parsing and metrics healthy during bundle outages
add ad0aa740087 Stop repeated bundle drains and keep cleanup overrides
working
add 56b6f8855ba Describe the git provider's refresh-error probe accurately
add a88a53aa8bf Allow Dag processors to publish metadata through the
Execution API
add ecd0a48bd1f Align API-mode Dag publication with DB-mode cleanup
add 4e908b30b03 Clean asset references for stale Dags removed from bundles
add fa432308864 Preserve compact source publications and actionable Dag
diagnostics
add e3aba00a3ce Reject Dags without a timetable or with incomplete
deadline alerts
add 0940033311a Cover publication API rejections and clarify its docs and
names
add b90c25695ca Reject only the invalid Dags in a Dag processor publication
add cc78654d756 Retain Dag processor work until delivery is acknowledged
add 696e7853522 Fix Dag processor work retries and metadata cleanup
add 4fbf8f373ec Test legacy Dag deactivation after bundle-name backfill
add 6321e5c0496 Align Dag processor claim migrations with main
add fe6187ca758 Adapt legacy Dag upgrade regression to bundle discovery
results
add df291f5a729 Fix Dag processor requested-work cleanup and delivery
add e44b432cc0c Preserve Dag cleanup and exact priority reparse targets
add 1f43f0132a0 Carry publication receipts through Dag processor
maintenance
add a78ef16cdbc Fix Dag processor maintenance logs and stale Dag warning
cleanup
add 7b53ef60a33 Tighten test coverage of Dag processor requested-work
delivery
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (ed8e8b74bf0)
\
N -- N -- N refs/heads/dag-processor-work-delivery (7b53ef60a33)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../skills/airflow-translations/locales/zh-TW.md | 44 ++-
.github/actions/breeze/action.yml | 6 +
.../actions/prepare_breeze_and_image/action.yml | 5 +
.github/actions/runs-on-cost-summary/action.yml | 4 +-
.github/workflows/additional-prod-image-tests.yml | 26 +-
.github/workflows/airflow-e2e-tests.yml | 12 +-
.github/workflows/basic-tests.yml | 4 +-
.github/workflows/ci-amd.yml | 42 +--
.github/workflows/ci-arm.yml | 42 +--
.github/workflows/finalize-tests.yml | 2 +-
.github/workflows/helm-tests.yml | 2 +-
.github/workflows/integration-system-tests.yml | 2 +-
.github/workflows/mirror-hardened-base-images.yml | 84 +++++-
.github/workflows/publish-docs-to-s3.yml | 74 ++++-
.github/workflows/refresh-image-registry-cache.yml | 7 +-
.github/workflows/registry-backfill.yml | 6 +
.github/workflows/registry-build.yml | 9 +
.github/workflows/release-constraints.yml | 12 +-
.github/workflows/release_dockerhub_image.yml | 11 +-
.../scheduled-legacy-prod-image-canary.yml | 9 +-
.github/workflows/scheduled-upgrade-check-main.yml | 2 +
.../scheduled-upgrade-check-v3-4-test.yml | 2 +
.github/workflows/special-tests.yml | 22 +-
.../update-constraints-on-push-stable.yml | 11 +-
.github/workflows/update-constraints-on-push.yml | 40 ++-
.github/workflows/upgrade-check.yml | 78 ++++-
Dockerfile | 83 +++++-
Dockerfile.ci | 79 ++++-
airflow-core/docs/ui.rst | 5 +
.../core_api/datamodels/task_instance_history.py | 4 +
.../core_api/datamodels/task_instances.py | 5 +
.../api_fastapi/core_api/datamodels/ui/grid.py | 1 +
.../api_fastapi/core_api/openapi/_private_ui.yaml | 31 +-
.../core_api/openapi/v2-rest-api-generated.yaml | 16 +
.../api_fastapi/core_api/routes/ui/deadlines.py | 10 +-
.../airflow/api_fastapi/core_api/routes/ui/grid.py | 2 +
.../api_fastapi/core_api/services/ui/grid.py | 5 +
.../execution_api/routes/dag_parsing.py | 17 +-
.../execution_api/routes/task_instances.py | 5 +-
airflow-core/src/airflow/assets/manager.py | 27 +-
.../src/airflow/cli/commands/triggerer_command.py | 9 +-
.../src/airflow/dag_processing/collection.py | 27 +-
.../src/airflow/jobs/scheduler_job_runner.py | 332 +++++++++++----------
.../src/airflow/jobs/triggerer_job_runner.py | 7 +
.../0142_3_4_0_unify_task_attempt_ownership.py | 3 +-
airflow-core/src/airflow/models/asset.py | 44 ++-
airflow-core/src/airflow/models/base.py | 13 +-
airflow-core/src/airflow/models/dag.py | 13 +-
airflow-core/src/airflow/models/dagrun.py | 13 +-
airflow-core/src/airflow/models/taskinstance.py | 88 +++++-
.../serialization/definitions/baseoperator.py | 2 +-
.../serialization/definitions/mappedoperator.py | 2 +-
.../ti_deps/deps/not_in_retry_period_dep.py | 6 +-
.../src/airflow/ui/openapi-gen/queries/common.ts | 5 +-
.../ui/openapi-gen/queries/ensureQueryData.ts | 8 +-
.../src/airflow/ui/openapi-gen/queries/prefetch.ts | 8 +-
.../src/airflow/ui/openapi-gen/queries/queries.ts | 8 +-
.../src/airflow/ui/openapi-gen/queries/suspense.ts | 8 +-
.../airflow/ui/openapi-gen/requests/schemas.gen.ts | 38 +++
.../ui/openapi-gen/requests/services.gen.ts | 4 +-
.../airflow/ui/openapi-gen/requests/types.gen.ts | 13 +
.../airflow/ui/public/i18n/locales/en/common.json | 3 +
.../ui/public/i18n/locales/en/dashboard.json | 1 +
.../ui/src/components/DagImportErrorDetails.tsx | 14 +-
.../ui/src/components/DagImportErrorModal.test.tsx | 27 +-
.../ui/src/components/Graph/useGraphLayout.ts | 3 +-
.../ui/src/components/TaskInstanceTooltip.test.tsx | 46 ++-
.../ui/src/components/TaskInstanceTooltip.tsx | 7 +
.../TriggerDag/RecentConfigSelect.test.tsx | 39 ++-
.../components/TriggerDag/RecentConfigSelect.tsx | 79 +++--
.../components/TriggerDag/TriggerDAGForm.test.tsx | 4 +-
.../ui/src/context/gridSelection/Context.ts | 56 ++++
.../gridSelection/GridSelectionProvider.tsx} | 18 +-
.../src/context/{groups => gridSelection}/index.ts | 3 +-
.../ui/src/context/keyboardShortcuts/shortcuts.ts | 10 +
.../src/airflow/ui/src/layouts/BaseLayout.tsx | 3 +-
.../ui/src/layouts/Details/Grid/Grid.test.tsx | 17 +-
.../airflow/ui/src/layouts/Details/Grid/Grid.tsx | 30 +-
.../Details/Grid/GridSelectionActionBar.tsx | 81 +++++
.../ui/src/layouts/Details/Grid/GridTI.test.tsx | 19 ++
.../airflow/ui/src/layouts/Details/Grid/GridTI.tsx | 9 +-
.../layouts/Details/Grid/TaskInstancesColumn.tsx | 15 +-
.../Grid/useGridSelectedTaskInstances.test.tsx | 97 ++++++
.../Details/Grid/useGridSelectedTaskInstances.ts | 77 +++++
.../layouts/Details/Grid/useGridSelection.test.tsx | 251 ++++++++++++++++
.../src/layouts/Details/Grid/useGridSelection.ts | 198 ++++++++++++
.../pages/Dashboard/Stats/DagImportErrorsModal.tsx | 14 +-
.../pages/GroupTaskInstance/GroupTaskInstance.tsx | 11 +-
.../ui/src/pages/GroupTaskInstance/Header.tsx | 26 +-
.../LoopIterations/LoopIterations.test.tsx | 136 +++++++++
.../LoopIterations/LoopIterations.tsx | 137 +++++++++
.../GroupTaskInstance/LoopIterations/loopUtils.ts | 2 +-
.../ui/src/pages/GroupTaskInstance/index.ts | 1 +
.../ui/src/pages/MappedTaskInstance/Header.tsx | 23 +-
.../src/airflow/ui/src/pages/Run/Header.tsx | 13 +-
.../ui/src/pages/TaskInstance/Details.test.tsx | 24 ++
.../airflow/ui/src/pages/TaskInstance/Details.tsx | 8 +
.../ui/src/pages/TaskInstance/Header.test.tsx | 21 ++
.../airflow/ui/src/pages/TaskInstance/Header.tsx | 28 +-
.../useHasMultipleDagBundles.ts} | 15 +-
airflow-core/src/airflow/ui/src/router.tsx | 7 +-
.../core_api/routes/public/test_hitl.py | 1 +
.../core_api/routes/public/test_task_instances.py | 92 ++++--
.../core_api/routes/ui/test_deadlines.py | 91 +++++-
.../api_fastapi/core_api/routes/ui/test_grid.py | 25 ++
.../versions/head/test_dag_parsing.py | 90 ++++--
.../versions/head/test_task_instances.py | 1 +
airflow-core/tests/unit/assets/test_manager.py | 4 +-
.../unit/cli/commands/test_triggerer_command.py | 27 ++
airflow-core/tests/unit/jobs/test_scheduler_job.py | 257 ++++++++++++----
airflow-core/tests/unit/jobs/test_triggerer_job.py | 24 ++
airflow-core/tests/unit/models/test_dag.py | 45 +++
airflow-core/tests/unit/models/test_dagrun.py | 25 ++
.../tests/unit/models/test_taskinstance.py | 64 +++-
airflow-core/tests/unit/models/test_trigger.py | 4 +-
.../src/airflowctl/api/datamodels/generated.py | 14 +
.../docker/{elasticsearch.yml => yaml-dag.yml} | 31 +-
.../tests/airflow_e2e_tests/conftest.py | 74 ++++-
.../tests/airflow_e2e_tests/constants.py | 6 +
.../airflow_e2e_tests/yaml_dag_tests}/__init__.py | 0
.../yaml_dag_tests/test_yaml_dag.py | 204 +++++++++++++
airflow-e2e-tests/yaml-dags/yaml_s3_roundtrip.yaml | 88 ++++++
...UALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md | 6 +
dev/README_RELEASE_AIRFLOW.md | 51 +++-
dev/breeze/doc/05_test_commands.rst | 2 +-
dev/breeze/doc/09_release_management_tasks.rst | 47 +++
dev/breeze/doc/ci/02_images.md | 2 +-
dev/breeze/doc/ci/04_selective_checks.md | 52 +++-
dev/breeze/doc/images/output_ci_upgrade.svg | 114 ++++---
dev/breeze/doc/images/output_ci_upgrade.txt | 2 +-
...utput_release-management_publish-docs-to-s3.svg | 64 ++--
...utput_release-management_publish-docs-to-s3.txt | 2 +-
.../output_setup_check-all-params-in-groups.svg | 24 +-
.../output_setup_check-all-params-in-groups.txt | 2 +-
.../output_setup_regenerate-command-images.svg | 4 +-
.../output_setup_regenerate-command-images.txt | 2 +-
.../images/output_testing_airflow-e2e-tests.svg | 2 +-
.../images/output_testing_airflow-e2e-tests.txt | 2 +-
dev/breeze/doc/images/output_workflow-run.svg | 34 ++-
dev/breeze/doc/images/output_workflow-run.txt | 2 +-
.../output_workflow-run_mirror-base-images.svg | 111 +++++++
.../output_workflow-run_mirror-base-images.txt | 1 +
.../images/output_workflow-run_upgrade-check.svg | 116 +++++++
.../images/output_workflow-run_upgrade-check.txt | 1 +
.../src/airflow_breeze/commands/ci_commands.py | 119 ++++++--
.../airflow_breeze/commands/ci_commands_config.py | 2 +
.../airflow_breeze/commands/kubernetes_commands.py | 106 +++++--
.../commands/release_management_commands.py | 124 ++++++--
.../commands/release_management_commands_config.py | 2 +
.../airflow_breeze/commands/testing_commands.py | 30 +-
.../airflow_breeze/commands/workflow_commands.py | 72 ++++-
.../commands/workflow_commands_config.py | 24 +-
.../files/hardened_python_image_digests.json | 10 +
.../files/legacy_python_versions.json | 6 +
dev/breeze/src/airflow_breeze/global_constants.py | 78 ++++-
.../src/airflow_breeze/params/build_prod_params.py | 11 +-
.../src/airflow_breeze/params/shell_params.py | 2 +
.../airflow_breeze/utils/ci_minutes_estimate.py | 6 +-
.../src/airflow_breeze/utils/gh_workflow_utils.py | 42 +++
.../src/airflow_breeze/utils/publish_docs_to_s3.py | 6 +-
dev/breeze/src/airflow_breeze/utils/run_tests.py | 101 +++++--
.../src/airflow_breeze/utils/selective_checks.py | 105 ++++++-
.../src/airflow_breeze/utils/verification_plan.py | 1 +
dev/breeze/tests/test_build_prod_params.py | 44 ++-
dev/breeze/tests/test_ci_upgrade.py | 100 +++++++
dev/breeze/tests/test_gh_workflow_utils.py | 54 ++++
.../tests/test_kubernetes_lang_sdk_commands.py | 44 ++-
dev/breeze/tests/test_mirror_base_images.py | 142 ++++++++-
dev/breeze/tests/test_publish_docs_to_s3.py | 25 ++
.../tests/test_pytest_args_for_test_types.py | 65 +++-
.../tests/test_release_management_commands.py | 70 +++++
dev/breeze/tests/test_run_test_args.py | 93 ++++++
dev/breeze/tests/test_selective_checks.py | 208 +++++++++++++
dev/breeze/tests/test_shell_params.py | 15 +-
dev/breeze/tests/test_workflow_commands.py | 57 ++++
docker-stack-docs/build-arg-ref.rst | 26 +-
docker-stack-docs/build.rst | 21 +-
.../customizing/add-build-essential-custom.sh | 3 +-
.../docker-examples/customizing/custom-sources.sh | 3 +-
.../customizing/pypi-dev-runtime-deps.sh | 3 +-
.../customizing/pypi-extras-and-deps.sh | 3 +-
.../customizing/pypi-selected-version.sh | 3 +-
.../restricted/restricted_environments.sh | 3 +-
.../test_examples_of_prod_image_building.py | 2 +
.../cmd/airflow-go-pack/pack_integration_test.go | 3 +
go-sdk/example/bundle/main.go | 5 +
go-sdk/example/bundle/yamldag/handlers.go | 43 +++
kubernetes-tests/lang_sdk/README.md | 18 +-
kubernetes-tests/lang_sdk/config/values.yaml | 9 +-
.../kubernetes_tests/test_lang_sdk_native_dag.py | 110 ++++++-
providers/amazon/docs/executors/ecs-executor.rst | 4 +
providers/amazon/provider.yaml | 9 +
.../amazon/aws/executors/ecs/ecs_executor.py | 230 +++++++++-----
.../providers/amazon/aws/executors/ecs/utils.py | 3 +
.../airflow/providers/amazon/get_provider_info.py | 7 +
.../amazon/aws/executors/ecs/test_ecs_executor.py | 265 +++++++++++++++-
scripts/ci/configure_runner_for_parallel_tests.sh | 46 +++
scripts/ci/prek/upgrade_important_versions.py | 141 ++++++++-
scripts/docker/entrypoint_ci.sh | 2 +
scripts/docker/install_os_dependencies.sh | 62 +++-
scripts/docker/install_postgres.sh | 6 +
.../ci/prek/test_upgrade_important_versions.py | 153 ++++++++++
.../observability/metrics/metrics_template.yaml | 6 +
ts-sdk/src/coordinator/log-channel.ts | 94 +++++-
ts-sdk/tests/coordinator/log-channel.test.ts | 173 +++++++++++
205 files changed, 7089 insertions(+), 1013 deletions(-)
create mode 100644
airflow-core/src/airflow/ui/src/context/gridSelection/Context.ts
copy airflow-core/src/airflow/ui/src/{system-components/Select/Item.tsx =>
context/gridSelection/GridSelectionProvider.tsx} (61%)
copy airflow-core/src/airflow/ui/src/context/{groups =>
gridSelection}/index.ts (93%)
create mode 100644
airflow-core/src/airflow/ui/src/layouts/Details/Grid/GridSelectionActionBar.tsx
create mode 100644
airflow-core/src/airflow/ui/src/layouts/Details/Grid/useGridSelectedTaskInstances.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/layouts/Details/Grid/useGridSelectedTaskInstances.ts
create mode 100644
airflow-core/src/airflow/ui/src/layouts/Details/Grid/useGridSelection.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/layouts/Details/Grid/useGridSelection.ts
create mode 100644
airflow-core/src/airflow/ui/src/pages/GroupTaskInstance/LoopIterations/LoopIterations.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/pages/GroupTaskInstance/LoopIterations/LoopIterations.tsx
copy airflow-core/src/airflow/ui/src/{context/groups/useGroups.ts =>
queries/useHasMultipleDagBundles.ts} (70%)
copy airflow-e2e-tests/docker/{elasticsearch.yml => yaml-dag.yml} (57%)
copy {airflow-core/src/airflow/_shared =>
airflow-e2e-tests/tests/airflow_e2e_tests/yaml_dag_tests}/__init__.py (100%)
create mode 100644
airflow-e2e-tests/tests/airflow_e2e_tests/yaml_dag_tests/test_yaml_dag.py
create mode 100644 airflow-e2e-tests/yaml-dags/yaml_s3_roundtrip.yaml
create mode 100644
dev/breeze/doc/images/output_workflow-run_mirror-base-images.svg
create mode 100644
dev/breeze/doc/images/output_workflow-run_mirror-base-images.txt
create mode 100644 dev/breeze/doc/images/output_workflow-run_upgrade-check.svg
create mode 100644 dev/breeze/doc/images/output_workflow-run_upgrade-check.txt
create mode 100644
dev/breeze/src/airflow_breeze/files/hardened_python_image_digests.json
create mode 100644
dev/breeze/src/airflow_breeze/files/legacy_python_versions.json
create mode 100644 go-sdk/example/bundle/yamldag/handlers.go
create mode 100755 scripts/ci/configure_runner_for_parallel_tests.sh
