This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch jason/lang-sdk-e2e/03e-lang-sdk-dag-bag
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 471491d2fe0 Default a Dag file processor's parse log to os.devnull
discard a806e5787ae Let a Dag file processor run without a parse log file
discard afbf901e917 Ignore the bagged Dag union in the external task sensor
test
discard 5888c5be40a List the tasks of a native Lang-SDK Dag
discard 6a1a42def5f Bag native Lang-SDK Dags in core, only when asked
discard c3669c2cf3a Let a Dag file processor run without an API client
discard 9b7542d5956 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
discard 79bef76bf7a Apply the team rules to the Dag that validation returns
discard d050e698ccc Test the import errors of a bundle with several
coordinators of a class
discard bf9c91a31c4 Find a claiming Dag importer instead of its coordinator in
Dag processing
discard d8e8e35f4fe Name the coordinator class a native Dag task must be
routed to
discard 7d894953d2e Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
discard 3400a3027d8 Pass a parse log handle to the Lang-SDK processor in tests
discard 57d4c6111e1 Find a native Dag's coordinator key through its importer
discard 62fff240b11 Wait for the killed leftover in the Lang-SDK processor test
discard e9677cbf7b5 Move the client-less Lang-SDK parse out of this change
discard 31c2b62bbbb Note in ADR-0010 that the manager routes claimed files for
now
discard 7e8364ff07b Clear import errors after the native-file Dag bag sync test
discard 4a4c5d426b8 Apply the multi-team task checks to Lang-SDK Dags
discard a60ad59da7f Spec the mocks of a hand-built Lang-SDK processor in tests
discard b3a1b4b19bc Name the setting in a Lang-SDK parse timeout error
discard 1cbc4dfb9a7 Kill what a Lang-SDK runtime leaves in its process group
discard 6102868b7da Match the schema error's JSON path in a Lang-SDK parse test
discard 6735f2a433d Fail a native Dag's task on a Python worker without retries
discard 1611330ed3e Find the claiming coordinator in the Task SDK and wrap it
in core
discard 29ae6e2fa6e Start each Dag file processor with explicit arguments
discard 96c32267afc Drop callbacks for Lang-SDK Dag files before queuing the
file
discard db47446a6ed Report a native Dag's task that reaches a Python worker
discard a658e511c49 Report a Lang-SDK parse past its import timeout as an
import error
discard ec74107656a Parse coordinator-claimed Dag files with their runtime in
the Dag processor
omit 6fd170439a4 Describe how a Dag importer finds its coordinator in
ADR-0010
omit b9bdf5419aa Let a Dag importer find the coordinator that parses its
files
add a44fd6d1f31 Allow explicit runs to drain a paused Dag (#73763)
add 40832901c59 Improve LocalExecutor bookkeeping: correctly add tasks to
the `running` list (#74142)
add 5487c080bea Add Azure Blob Storage support to the DataFusion object
storage layer (#73374)
add b191cc4767b Fail instead of silently falling behind when a Lang-SDK
vendored Dag schema drifts from airflow-core (#74091)
add e0202f40c78 Let a Dag importer find the coordinator that parses its
files (#74042)
add f1070b3884e Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add c8a3306646d Report a Lang-SDK parse past its import timeout as an
import error
add bbea6488ded Report a native Dag's task that reaches a Python worker
add 18de8bfa32d Drop callbacks for Lang-SDK Dag files before queuing the
file
add a6359232b06 Start each Dag file processor with explicit arguments
add 17f65afef02 Find the claiming coordinator in the Task SDK and wrap it
in core
add 2d6dd214ff5 Fail a native Dag's task on a Python worker without retries
add 1169dbb91fa Match the schema error's JSON path in a Lang-SDK parse test
add 303b0be7821 Kill what a Lang-SDK runtime leaves in its process group
add 19810c730b3 Name the setting in a Lang-SDK parse timeout error
add b5ed50699b4 Spec the mocks of a hand-built Lang-SDK processor in tests
add 58c66651378 Apply the multi-team task checks to Lang-SDK Dags
add 913e53620c5 Clear import errors after the native-file Dag bag sync test
add 22cd5aac3cd Note in ADR-0010 that the manager routes claimed files for
now
add e434a396ff7 Move the client-less Lang-SDK parse out of this change
add 0e680a3b2e2 Wait for the killed leftover in the Lang-SDK processor test
add eb258495592 Find a native Dag's coordinator key through its importer
add 4ad06144a0e Pass a parse log handle to the Lang-SDK processor in tests
add 435e639a5e9 Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
add e62a4815b8f Name the coordinator class a native Dag task must be
routed to
add 18744e8f554 Find a claiming Dag importer instead of its coordinator in
Dag processing
add 302b4e0c7b5 Test the import errors of a bundle with several
coordinators of a class
add 9007b91ed18 Apply the team rules to the Dag that validation returns
add d747c98f701 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
add 2855834d2bf Let a Dag file processor run without an API client
add 16099c61f3c Bag native Lang-SDK Dags in core, only when asked
add e77020e65bd List the tasks of a native Lang-SDK Dag
add ff9e8d23f10 Ignore the bagged Dag union in the external task sensor
test
add 29d2fa7090a Let a Dag file processor run without a parse log file
add 77a24d4ee6d Default a Dag file processor's parse log to os.devnull
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (471491d2fe0)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/03e-lang-sdk-dag-bag
(77a24d4ee6d)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.pre-commit-config.yaml | 42 +-
airflow-core/docs/core-concepts/backfill.rst | 7 +-
airflow-core/docs/core-concepts/dags.rst | 13 +
airflow-core/newsfragments/72407.significant.rst | 10 +-
.../api_fastapi/core_api/datamodels/backfills.py | 8 +
.../api_fastapi/core_api/datamodels/dag_run.py | 7 +
.../core_api/openapi/v2-rest-api-generated.yaml | 28 ++
.../api_fastapi/core_api/routes/public/assets.py | 10 +
.../core_api/routes/public/backfills.py | 17 +-
.../api_fastapi/core_api/routes/public/dag_run.py | 11 +
.../src/airflow/executors/local_executor.py | 75 +++-
airflow-core/src/airflow/models/backfill.py | 3 +
airflow-core/src/airflow/models/dag.py | 16 +
.../airflow/ui/openapi-gen/requests/schemas.gen.ts | 18 +
.../airflow/ui/openapi-gen/requests/types.gen.ts | 12 +
.../ui/public/i18n/locales/ar/components.json | 4 +-
.../ui/public/i18n/locales/ca/components.json | 4 +-
.../ui/public/i18n/locales/de/components.json | 4 +-
.../ui/public/i18n/locales/el/components.json | 4 +-
.../ui/public/i18n/locales/en/components.json | 15 +-
.../ui/public/i18n/locales/es/components.json | 4 +-
.../ui/public/i18n/locales/fr/components.json | 4 +-
.../ui/public/i18n/locales/he/components.json | 4 +-
.../ui/public/i18n/locales/hi/components.json | 4 +-
.../ui/public/i18n/locales/hu/components.json | 4 +-
.../ui/public/i18n/locales/it/components.json | 4 +-
.../ui/public/i18n/locales/ja/components.json | 4 +-
.../ui/public/i18n/locales/ko/components.json | 4 +-
.../ui/public/i18n/locales/nl/components.json | 4 +-
.../ui/public/i18n/locales/pl/components.json | 4 +-
.../ui/public/i18n/locales/pt/components.json | 4 +-
.../ui/public/i18n/locales/ru/components.json | 4 +-
.../ui/public/i18n/locales/th/components.json | 4 +-
.../ui/public/i18n/locales/tr/components.json | 4 +-
.../ui/public/i18n/locales/zh-CN/components.json | 4 +-
.../ui/public/i18n/locales/zh-TW/components.json | 4 +-
.../components/DagActions/RunBackfillForm.test.tsx | 96 ++++-
.../src/components/DagActions/RunBackfillForm.tsx | 35 +-
.../TriggerDag/PausedDagOptions.test.tsx | 250 +++++++++++
.../src/components/TriggerDag/PausedDagOptions.tsx | 123 ++++++
.../src/components/TriggerDag/TriggerDAGButton.tsx | 4 -
.../components/TriggerDag/TriggerDAGForm.test.tsx | 78 +++-
.../src/components/TriggerDag/TriggerDAGForm.tsx | 32 +-
.../components/TriggerDag/TriggerDAGModal.test.tsx | 238 +++++++++++
.../src/components/TriggerDag/TriggerDAGModal.tsx | 21 +-
.../airflow/ui/src/components/TriggerDag/types.ts | 3 +
.../ui/src/layouts/Details/DetailsLayout.tsx | 1 -
.../src/pages/Asset/CreateAssetEventModal.test.tsx | 161 +++++++-
.../ui/src/pages/Asset/CreateAssetEventModal.tsx | 20 +-
.../ui/src/pages/DagsList/DagCardActions.tsx | 1 -
.../src/airflow/ui/src/pages/DagsList/DagsList.tsx | 1 -
.../ui/src/queries/useCreateBackfill.test.tsx | 99 +++++
.../airflow/ui/src/queries/useCreateBackfill.ts | 39 +-
.../src/airflow/ui/src/queries/useTrigger.test.tsx | 98 +++++
.../src/airflow/ui/src/queries/useTrigger.ts | 31 +-
airflow-core/tests/unit/api_fastapi/conftest.py | 13 +
.../core_api/routes/public/test_assets.py | 48 ++-
.../core_api/routes/public/test_backfills.py | 72 +++-
.../core_api/routes/public/test_dag_run.py | 75 +++-
.../tests/unit/executors/test_local_executor.py | 455 ++++++++++++++++++++-
.../executors/test_local_executor_check_workers.py | 5 +-
airflow-core/tests/unit/jobs/test_scheduler_job.py | 9 +-
airflow-core/tests/unit/models/test_backfill.py | 133 +++++-
airflow-core/tests/unit/models/test_dag.py | 40 ++
.../src/airflowctl/api/datamodels/generated.py | 21 +
airflow-ctl/src/airflowctl/api/operations.py | 7 +-
.../tests/airflow_ctl/api/test_operations.py | 54 ++-
java-sdk/sdk/build.gradle.kts | 15 +
.../ai/docs/operators/llm_schema_compare.rst | 2 +-
providers/common/ai/docs/operators/llm_sql.rst | 2 +-
providers/common/ai/docs/toolsets/datafusion.rst | 2 +-
providers/common/sql/docs/operators.rst | 60 ++-
.../sql/src/airflow/providers/common/sql/config.py | 3 +
.../providers/common/sql/datafusion/engine.py | 185 +++++++++
.../sql/datafusion/object_storage_provider.py | 36 +-
.../common/sql/example_dags/example_analytics.py | 13 +
.../unit/common/sql/datafusion/test_engine.py | 447 ++++++++++++++++++++
.../sql/datafusion/test_object_storage_provider.py | 48 +++
.../sql/tests/unit/common/sql/test_config.py | 1 +
scripts/ci/prek/sync_go_sdk_schemas.py | 21 +-
.../src/airflow/sdk/coordinators/_subprocess.py | 2 +-
.../src/airflow/sdk/execution_time/coordinator.py | 4 +
82 files changed, 3243 insertions(+), 233 deletions(-)
create mode 100644
airflow-core/src/airflow/ui/src/components/TriggerDag/PausedDagOptions.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/components/TriggerDag/PausedDagOptions.tsx
create mode 100644
airflow-core/src/airflow/ui/src/components/TriggerDag/TriggerDAGModal.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/queries/useCreateBackfill.test.tsx
create mode 100644 airflow-core/src/airflow/ui/src/queries/useTrigger.test.tsx
