This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch jason/lang-sdk-e2e/05-node-dag-importer
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 52bf274b956 Pin the Code view to the entry module in Node bundle tests
discard 83e1a0e950e Let NodeDagImporter source read errors reach the caller
discard 170bd0501a7 Import NodeDagImporter at the top of the Node coordinator
discard 4f597ad16e7 Warn against defining a Dag in both TypeScript and Python
discard ef4a6022cea Align the TypeScript build docs with the Code view
discard 6ff552bff78 Say when the Dag processor needs Node.js for TypeScript
bundles
discard 50fa4d83f06 Run a native TypeScript Dag's task from its own bundle
discard ef480595276 Share the bundle section mutation helper between Node tests
discard 7453fe7a3af Document parsing native TypeScript Dags
discard d1cc2a44291 Parse native TypeScript Dags with the Node coordinator
discard 81f952731a2 Keep the Java parse_dag test from changing the pytest
worker
discard 926eee46796 Stop promising that Java dependency JARs are not parsed
discard 47aaa41a857 Reject a Java Dag JAR whose SDK cannot answer a parse
discard ac66e0a3af2 Run the JAR a native Java Dag was parsed from
discard 5c769623a95 Report why a Java Dag JAR cannot be read
discard 8cd962566d3 Pass the task's Dag file to the coordinator's command
builder
discard 1cf7c2cc95e Run a task on its run's version when dag_bundle_name names
its bundle
discard fbd5b06be45 Document native Java Dag parsing
discard 52e1b94dfe7 Name the JAR manifest attributes the Java coordinator reads
discard a9ee37a8def Test the Java Dag source placeholder for a JAR without a
manifest
discard 6ed348f241a Treat an unreadable JAR like one without a manifest
discard 3160ab91bf8 Parse native Java Dags with their coordinator
discard 96234e6d17d Embed the Dag source in Java bundle JARs
discard 8f2cf075fd2 Build the JVM command that parses a Dag JAR
discard cd00618ba80 Read JAR manifests per the JAR specification
discard 6fea2b43a31 Ignore the bagged Dag union in the external task sensor
test
discard ad8aaa2a6a9 List the tasks of a native Lang-SDK Dag
discard e7657922dbb Bag native Lang-SDK Dags in core, only when asked
discard 0ea04fd6a01 Let a Dag file processor run without an API client
discard df5ba5d5d62 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
omit 31c1a6b5a72 Wait for the killed leftover in the Lang-SDK processor test
omit d0eafc41a00 Move the client-less Lang-SDK parse out of this change
omit 6b773354fa0 Note in ADR-0010 that the manager routes claimed files for
now
omit 2b1e171bfe6 Clear import errors after the native-file Dag bag sync test
omit 68072d7cdf2 Apply the multi-team task checks to Lang-SDK Dags
omit 50a13677d5c Spec the mocks of a hand-built Lang-SDK processor in tests
omit f0302a6a041 Name the setting in a Lang-SDK parse timeout error
omit 1b9e9d7e407 Kill what a Lang-SDK runtime leaves in its process group
omit 571336b499c Match the schema error's JSON path in a Lang-SDK parse test
omit e480eae1fa7 Build each bundle's Dag importer registry before freezing
the heap
omit 52c2c4b4929 Fail a native Dag's task on a Python worker without retries
omit bfd0b241a5c Find the claiming coordinator in the Task SDK and wrap it
in core
omit 49840104918 Return a Dag importer from every SubprocessCoordinator call
omit dab3acf4bee Start each Dag file processor with explicit arguments
omit 88abc9efdd2 Drop callbacks for Lang-SDK Dag files before queuing the
file
omit 84d307204bd Report a native Dag's task that reaches a Python worker
omit 0f7db884a00 Report a Lang-SDK parse past its import timeout as an
import error
omit 693bdf2b4d4 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
omit d21aea99762 Say dag_bundle_name is only needed for two coordinators of
one class
omit 39fcab0bc98 Drop serves_bundle so get_parsed_bundles is the only answer
omit 27a987d2641 Check Dag file claims per bundle on the parse path
omit f27fb2d3de8 Import the coordinator manager at the top of the importer
base
omit 43d117ddbca Let a coordinator parse the Dag files of the bundles it
serves
omit 6a995f27589 Sync the Java SDK copy of the Dag schema
omit ad6eae0144f Serialize the SDK Dag in tests that write a dag_maker Dag
omit f1f5d581143 Note in the parsing ADR that runtimes may omit
config-backed fields
omit d3effc9c25d Check SDK Dags with the Dag processor's validation in
conformance
omit 2482920cb7e Reject malformed task entries in a serialized Dag
omit d7e9d5f5e72 Fill a serialized Dag's unset settings from the Airflow
config
omit 0834b7149ac Check that a serialized Dag can be stored and loaded
omit 7c112fe041f Move the Dag file processor's shared plumbing into a base
class
add a3e07159a0b Add Dag ID filtering to Assets search (#70971)
add d3b9a0dd2b3 Add KafkaSharedStreamProducer and KafkaSharedStreamTrigger
(#68625)
add ae6139d58a8 Fix DecreasingPriorityStrategy example by initializing
try_number before weight evaluation (#62148)
add 402d7885194 Java SDK: Resolve a task's arguments from the Dag's own
wiring (#73596)
add ca0d7328288 Move the Dag file processor's shared plumbing into a base
class
add 235be2d6bda Check that a serialized Dag can be stored and loaded
add bcba1cbf97f Fill a serialized Dag's unset settings from the Airflow
config
add 2f770c94c9a Reject malformed task entries in a serialized Dag
add fc73b8cc582 Check SDK Dags with the Dag processor's validation in
conformance
add 8cfd06019f4 Note in the parsing ADR that runtimes may omit
config-backed fields
add a63b9f1c091 Serialize the SDK Dag in tests that write a dag_maker Dag
add 0afbeb1e11c Sync the Java SDK copy of the Dag schema
add fd5eb37bd70 Let a coordinator parse the Dag files of the bundles it
serves
add 2cb5c904df8 Import the coordinator manager at the top of the importer
base
add 266131caf2a Check Dag file claims per bundle on the parse path
add 13d82cc57ec Drop serves_bundle so get_parsed_bundles is the only answer
add 6afe857c7e0 Say dag_bundle_name is only needed for two coordinators of
one class
add e610aa76488 Hand out a coordinator's Dag importer through
get_dag_importer alone
add 42c00d465ae Explain dag_bundle_name by the coordinator that parses a
Dag file
add f72fb46b402 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add 426d3bb8956 Report a Lang-SDK parse past its import timeout as an
import error
add 3dc7c020ac7 Report a native Dag's task that reaches a Python worker
add 76b203ed127 Drop callbacks for Lang-SDK Dag files before queuing the
file
add fe4b272b1ae Start each Dag file processor with explicit arguments
add 77bd5b757d4 Find the claiming coordinator in the Task SDK and wrap it
in core
add f13c43fa99b Fail a native Dag's task on a Python worker without retries
add 2a253c147f6 Build each bundle's Dag importer registry before freezing
the heap
add 985271f2515 Match the schema error's JSON path in a Lang-SDK parse test
add 1697091e40a Kill what a Lang-SDK runtime leaves in its process group
add 4c5f7e2572f Name the setting in a Lang-SDK parse timeout error
add bfcc16ccfca Spec the mocks of a hand-built Lang-SDK processor in tests
add a68f51b07cd Apply the multi-team task checks to Lang-SDK Dags
add 85c86080632 Clear import errors after the native-file Dag bag sync test
add e3bd6f0b0b3 Note in ADR-0010 that the manager routes claimed files for
now
add 778193ac00d Move the client-less Lang-SDK parse out of this change
add 917db98d4a1 Wait for the killed leftover in the Lang-SDK processor test
add a70eb430b82 Find a native Dag's coordinator key through its importer
add e8fbf760526 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
add ff48b1f19aa Let a Dag file processor run without an API client
add dca01a5a0c6 Bag native Lang-SDK Dags in core, only when asked
add 897a3ef4c63 List the tasks of a native Lang-SDK Dag
add 6c3ae5d7c33 Ignore the bagged Dag union in the external task sensor
test
add 447eea4b249 Read JAR manifests per the JAR specification
add 00eaafa1752 Build the JVM command that parses a Dag JAR
add 4f8762224ec Embed the Dag source in Java bundle JARs
add a7093859961 Parse native Java Dags with their coordinator
add aab428b0bf7 Treat an unreadable JAR like one without a manifest
add 2bfbd658b47 Test the Java Dag source placeholder for a JAR without a
manifest
add cbeab5bb190 Name the JAR manifest attributes the Java coordinator reads
add f08c7823a97 Document native Java Dag parsing
add 0d10d21b2f1 Run a task on its run's version when dag_bundle_name names
its bundle
add 3ad83f79caf Pass the task's Dag file to the coordinator's command
builder
add cf32a1492cb Report why a Java Dag JAR cannot be read
add c44b01925f3 Run the JAR a native Java Dag was parsed from
add e02d9fb1f39 Reject a Java Dag JAR whose SDK cannot answer a parse
add ae5ebe24c46 Stop promising that Java dependency JARs are not parsed
add 9998e1ea102 Keep the Java parse_dag test from changing the pytest
worker
add 51198c6125c Hand out the Java Dag importer through get_dag_importer
add 4f3c85f0dc1 Parse native TypeScript Dags with the Node coordinator
add 6b8fd989bf6 Document parsing native TypeScript Dags
add 55d17f96ece Share the bundle section mutation helper between Node tests
add 42d6dd45114 Run a native TypeScript Dag's task from its own bundle
add d73eaf8b39e Say when the Dag processor needs Node.js for TypeScript
bundles
add 82d5a7ea320 Align the TypeScript build docs with the Code view
add 9869a1e4f3a Warn against defining a Dag in both TypeScript and Python
add 8a8c43e4344 Import NodeDagImporter at the top of the Node coordinator
add 47a97f0f08d Let NodeDagImporter source read errors reach the caller
add 4d6f5242f0f Pin the Code view to the entry module in Node bundle tests
add 44e11868ff9 Hand out the Node Dag importer through get_dag_importer
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (52bf274b956)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/05-node-dag-importer
(44e11868ff9)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../adr/lang-sdk/0010-native-dag-processing.md | 25 +-
.../authoring-and-scheduling/language-sdks/go.rst | 27 +-
.../language-sdks/java.rst | 10 +-
.../language-sdks/typescript.rst | 12 +-
.../airflow/dag_processing/lang_sdk_processor.py | 4 +-
airflow-core/src/airflow/models/taskinstance.py | 2 +-
.../ui/src/components/FilterBar/FilterBar.test.tsx | 21 +
.../ui/src/components/FilterBar/FilterBar.tsx | 31 +-
.../ui/src/pages/AssetsList/AssetsList.test.tsx | 85 +++-
.../airflow/ui/src/pages/AssetsList/AssetsList.tsx | 10 +-
.../tests/unit/dag_processing/fake_lang_sdk.py | 5 +-
airflow-core/tests/unit/models/test_dag.py | 2 +
go-sdk/README.md | 7 +-
java-sdk/README.md | 6 +-
.../org/apache/airflow/sdk/BuilderProcessor.kt | 2 +-
.../kotlin/org/apache/airflow/sdk/BuilderTest.kt | 4 +-
.../src/main/kotlin/org/apache/airflow/sdk/Arg.kt | 11 +-
.../main/kotlin/org/apache/airflow/sdk/Context.kt | 10 +-
.../main/kotlin/org/apache/airflow/sdk/DagDef.kt | 1 +
.../kotlin/org/apache/airflow/sdk/InputTask.kt | 2 +-
.../org/apache/airflow/sdk/execution/Task.kt | 7 +-
.../org/apache/airflow/sdk/internal/ArgValues.kt | 124 ++++++
.../org/apache/airflow/sdk/internal/TaskArgs.kt | 42 +-
.../org/apache/airflow/sdk/ArgTestSupport.kt | 15 +
.../kotlin/org/apache/airflow/sdk/ArgValuesTest.kt | 2 +-
.../kotlin/org/apache/airflow/sdk/InputTaskTest.kt | 90 ++++
.../org/apache/airflow/sdk/execution/TaskTest.kt | 24 +
.../apache/airflow/sdk/internal/ArgValuesTest.kt | 266 ++++++++++++
providers/apache/kafka/docs/triggers.rst | 33 ++
providers/apache/kafka/provider.yaml | 1 +
.../providers/apache/kafka/get_provider_info.py | 1 +
.../apache/kafka/triggers/shared_stream.py | 407 +++++++++++++++++
.../providers/apache/kafka/version_compat.py | 2 +
.../apache/kafka/triggers/test_shared_stream.py | 259 +++++++++++
.../apache/kafka/triggers/test_shared_stream.py | 481 +++++++++++++++++++++
.../src/airflow/sdk/coordinators/_subprocess.py | 21 +-
.../airflow/sdk/coordinators/java/coordinator.py | 5 +-
.../airflow/sdk/coordinators/node/coordinator.py | 5 +-
.../src/airflow/sdk/execution_time/coordinator.py | 93 ++--
.../src/airflow/sdk/execution_time/task_runner.py | 7 +-
task-sdk/src/airflow/sdk/importers/base.py | 18 +-
.../src/airflow/sdk/importers/python_importer.py | 4 +-
task-sdk/src/airflow/sdk/importers/zip_importer.py | 6 +-
.../coordinators/java/test_dag_importer.py | 1 -
.../task_sdk/coordinators/node/test_coordinator.py | 1 -
.../tests/task_sdk/coordinators/test_subprocess.py | 25 --
.../task_sdk/execution_time/test_coordinator.py | 119 +++--
.../task_sdk/execution_time/test_task_runner.py | 5 +-
task-sdk/tests/task_sdk/importers/test_registry.py | 18 +-
ts-sdk/README.md | 8 +-
ts-sdk/example/README.md | 6 +-
51 files changed, 2066 insertions(+), 307 deletions(-)
create mode 100644
java-sdk/sdk/src/test/kotlin/org/apache/airflow/sdk/internal/ArgValuesTest.kt
create mode 100644
providers/apache/kafka/src/airflow/providers/apache/kafka/triggers/shared_stream.py
create mode 100644
providers/apache/kafka/tests/integration/apache/kafka/triggers/test_shared_stream.py
create mode 100644
providers/apache/kafka/tests/unit/apache/kafka/triggers/test_shared_stream.py
