This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a change to branch jason/lang-sdk-e2e/03e-lang-sdk-dag-bag
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit 96f2076b96d Ignore the bagged Dag union in the external task sensor
test
omit fd0a742a2d7 List the tasks of a native Lang-SDK Dag
omit a8797750305 Bag native Lang-SDK Dags in core, only when asked
omit cab27a25c52 Let a Dag file processor run without an API client
omit c53c8c723e5 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
omit e072db64742 Wait for the killed leftover in the Lang-SDK processor test
omit 57a417b164d Move the client-less Lang-SDK parse out of this change
omit 2ee5ead4c9a Note in ADR-0010 that the manager routes claimed files for
now
omit 04706bdc0b5 Clear import errors after the native-file Dag bag sync test
omit 55311fe73c1 Apply the multi-team task checks to Lang-SDK Dags
omit 8b91affcf79 Spec the mocks of a hand-built Lang-SDK processor in tests
omit 56c6820d7c3 Name the setting in a Lang-SDK parse timeout error
omit f632602f806 Kill what a Lang-SDK runtime leaves in its process group
omit 280fb07887a Match the schema error's JSON path in a Lang-SDK parse test
omit 3c0b4ba6181 Build each bundle's Dag importer registry before freezing
the heap
omit 001b6d90f29 Fail a native Dag's task on a Python worker without retries
omit fd2a897053d Find the claiming coordinator in the Task SDK and wrap it
in core
omit 4f54283d0de Return a Dag importer from every SubprocessCoordinator call
omit ed8cbad9fa0 Start each Dag file processor with explicit arguments
omit 0cb6bf43a66 Drop callbacks for Lang-SDK Dag files before queuing the
file
omit 7c38262692d Report a native Dag's task that reaches a Python worker
omit 50981761a38 Report a Lang-SDK parse past its import timeout as an
import error
omit 9ef4d490013 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
omit 30187220d54 Say dag_bundle_name is only needed for two coordinators of
one class
omit 52b88ba0c85 Drop serves_bundle so get_parsed_bundles is the only answer
omit 75c77d508be Check Dag file claims per bundle on the parse path
omit 9b98c04bd91 Import the coordinator manager at the top of the importer
base
omit 60d7af4cad9 Let a coordinator parse the Dag files of the bundles it
serves
omit f9fd8e01be2 Sync the Java SDK copy of the Dag schema
omit 464c9273c0c Serialize the SDK Dag in tests that write a dag_maker Dag
omit e5d1a71ca9c Note in the parsing ADR that runtimes may omit
config-backed fields
omit 76389a9ffa2 Check SDK Dags with the Dag processor's validation in
conformance
omit d8cf5805ec7 Reject malformed task entries in a serialized Dag
omit c1226408285 Fill a serialized Dag's unset settings from the Airflow
config
omit adf872d0ac6 Check that a serialized Dag can be stored and loaded
omit 0649a43f468 Move the Dag file processor's shared plumbing into a base
class
omit 20603491841 Pin the empty task id case in the Dag cycle detection tests
omit 737097a0dfe Share Dag cycle detection between the Task SDK and core
add 383a9a87a98 Stop wiping the uv cache before every CI image build
(#74052)
add 58e987defec Fix Non-DB test collection failure in Google Stackdriver
hook test (#74066)
add 498724fc13a Go SDK: enumerate every user-facing enum in the airflow
package (#74064)
add 5e476d7cf89 TS SDK: answer the Dag-parsing request from bundle.serve
(#73442)
add dd271599939 Go SDK: vendor the schemas it generates from (#74047)
add 77927390c17 Move Dag cycle detection from the Task SDK to shared
(#74034)
add 7c112fe041f Move the Dag file processor's shared plumbing into a base
class
add 0834b7149ac Check that a serialized Dag can be stored and loaded
add d7e9d5f5e72 Fill a serialized Dag's unset settings from the Airflow
config
add 2482920cb7e Reject malformed task entries in a serialized Dag
add d3effc9c25d Check SDK Dags with the Dag processor's validation in
conformance
add f1f5d581143 Note in the parsing ADR that runtimes may omit
config-backed fields
add ad6eae0144f Serialize the SDK Dag in tests that write a dag_maker Dag
add 6a995f27589 Sync the Java SDK copy of the Dag schema
add 43d117ddbca Let a coordinator parse the Dag files of the bundles it
serves
add f27fb2d3de8 Import the coordinator manager at the top of the importer
base
add 27a987d2641 Check Dag file claims per bundle on the parse path
add 39fcab0bc98 Drop serves_bundle so get_parsed_bundles is the only answer
add d21aea99762 Say dag_bundle_name is only needed for two coordinators of
one class
add 693bdf2b4d4 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add 0f7db884a00 Report a Lang-SDK parse past its import timeout as an
import error
add 84d307204bd Report a native Dag's task that reaches a Python worker
add 88abc9efdd2 Drop callbacks for Lang-SDK Dag files before queuing the
file
add dab3acf4bee Start each Dag file processor with explicit arguments
add 49840104918 Return a Dag importer from every SubprocessCoordinator call
add bfd0b241a5c Find the claiming coordinator in the Task SDK and wrap it
in core
add 52c2c4b4929 Fail a native Dag's task on a Python worker without retries
add e480eae1fa7 Build each bundle's Dag importer registry before freezing
the heap
add 571336b499c Match the schema error's JSON path in a Lang-SDK parse test
add 1b9e9d7e407 Kill what a Lang-SDK runtime leaves in its process group
add f0302a6a041 Name the setting in a Lang-SDK parse timeout error
add 50a13677d5c Spec the mocks of a hand-built Lang-SDK processor in tests
add 68072d7cdf2 Apply the multi-team task checks to Lang-SDK Dags
add 2b1e171bfe6 Clear import errors after the native-file Dag bag sync test
add 6b773354fa0 Note in ADR-0010 that the manager routes claimed files for
now
add d0eafc41a00 Move the client-less Lang-SDK parse out of this change
add 31c1a6b5a72 Wait for the killed leftover in the Lang-SDK processor test
add df5ba5d5d62 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
add 0ea04fd6a01 Let a Dag file processor run without an API client
add e7657922dbb Bag native Lang-SDK Dags in core, only when asked
add ad8aaa2a6a9 List the tasks of a native Lang-SDK Dag
add 6fea2b43a31 Ignore the bagged Dag union in the external task sensor
test
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (96f2076b96d)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/03e-lang-sdk-dag-bag
(6fea2b43a31)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.pre-commit-config.yaml | 25 ++-
.../airflow_breeze/commands/ci_image_commands.py | 4 +-
dev/breeze/tests/test_ci_image_commands.py | 20 ++
go-sdk/README.md | 52 +++--
go-sdk/adr/0008-native-dag-interface.md | 5 +
go-sdk/airflow/dag.go | 5 +
go-sdk/airflow/enums.go | 156 +++++++++++++++
go-sdk/airflow/enums_test.go | 221 +++++++++++++++++++++
go-sdk/airflow/spec.go | 33 +--
go-sdk/airflow/spec_test.go | 71 -------
go-sdk/airflow/trigger_dag_run.go | 36 ++--
go-sdk/airflow/trigger_dag_run_test.go | 24 +--
go-sdk/internal/genspec/authoring.go | 6 +-
go-sdk/internal/genspec/main.go | 9 +-
go-sdk/internal/genspec/normalize_test.go | 5 +-
go-sdk/pkg/execution/genmodels/gen.go | 13 +-
go-sdk/pkg/execution/messages_test.go | 23 +--
{ts-sdk => go-sdk}/schema/dag-schema.json | 13 +-
.../schema/supervisor-schema.json | 0
.../unit/google/cloud/hooks/test_stackdriver.py | 2 +-
scripts/ci/prek/check_go_sdk_generated_drift.py | 25 ++-
scripts/ci/prek/sync_go_sdk_schemas.py | 142 +++++++++++++
.../ci/prek/test_check_go_sdk_generated_drift.py | 13 +-
scripts/tests/ci/prek/test_sync_go_sdk_schemas.py | 127 ++++++++++++
ts-sdk/src/coordinator/runtime.ts | 69 ++++++-
ts-sdk/tests/coordinator/integration.test.ts | 108 +++++++++-
26 files changed, 980 insertions(+), 227 deletions(-)
create mode 100644 go-sdk/airflow/enums.go
create mode 100644 go-sdk/airflow/enums_test.go
delete mode 100644 go-sdk/airflow/spec_test.go
copy {ts-sdk => go-sdk}/schema/dag-schema.json (97%)
copy task-sdk/src/airflow/sdk/execution_time/schema/schema.json =>
go-sdk/schema/supervisor-schema.json (100%)
create mode 100755 scripts/ci/prek/sync_go_sdk_schemas.py
create mode 100644 scripts/tests/ci/prek/test_sync_go_sdk_schemas.py
