shivaam commented on issue #69288:
URL: https://github.com/apache/airflow/issues/69288#issuecomment-5081494623
Here is one possible TypeScript shape for discussion. The goal is to
preserve Python TaskFlow's dependency semantics while keeping the API natural
for TypeScript; none of these names are final.
The Dag could be declared first, with tasks defined on it:
```ts
const pipeline = new Dag({
dagId: "example_pipeline",
schedule: "@daily",
catchup: false,
});
const extract = pipeline.task({
taskId: "extract",
retries: 3,
run: extractHandler,
});
const transform = pipeline.task({
taskId: "transform",
run: transformHandler,
});
const load = pipeline.task({
taskId: "load",
run: loadHandler,
});
```
`pipeline.task(...)` could use the existing `registerTask()` mechanism
internally while also recording the information needed to serialize the Dag.
Passing task outputs would create data dependencies:
```ts
const extracted = extract();
const transformed = transform({ extracted });
const loaded = load({ transformed });
```
This constructs:
```text
extract -> transform -> load
```
For ordering without passing data, we could provide both `before()` and
`after()`:
```ts
extract.before(validate, transform);
publish.after(report, approval);
```
TypeScript cannot reproduce Python's `>>` and `<<` because JavaScript uses
those operators for numeric bit shifting. I also find `before()` and `after()`
easier to read than `setUpstream()` and `setDownstream()`. The Python-style
methods could potentially be added later as aliases. This is just a proposal
right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
