jason810496 commented on issue #69288:
URL: https://github.com/apache/airflow/issues/69288#issuecomment-5093501655
Hi @shivaam,
Nice work! I think the overall shape LGTM.
Regarding the task-level argument definition: I'd like to introduce a `spec`
level for Task, the reason being that we can generate the `Spec` type class
itself directly from `airflow-core/src/airflow/serialization/schema.json`.
Additionally, we could omit `spec` entirely for mixed-language Dags.
Another small thing (but I don't sure would the naming convention be weird
or not) is that would having the `dag_instance.Task` as upper case instead of
lower case to showcase the "factory" method make sense?
I'd also like to remove `registerTask` and introduce `registerDags(d1, d2)`
as the entrypoint.
The shape I have in mind:
**native Dag**
```ts
const pipeline = new Dag({
dagId: "example_pipeline"
});
const extract = pipeline.Task({
run: extractFn,
spec: {
taskId: "extract",
retries: 3,
},
});
registerDags(pipeline);
```
**mixed-lang Dag**
```ts
const pipeline = new Dag({
dagId: "example_pipeline",
schedule: "@daily",
catchup: false,
});
const extract = pipeline.Task({
run: extractFn,
spec: {
taskId: "extract",
retries: 3,
},
});
registerDags(pipeline);
```
For now, we only need to focus on the following TaskFlow-style dependency
definition (and treat it as first-class):
```ts
const extracted = extract();
const transformed = transform({ extracted });
const loaded = load({ transformed });
```
> The `chain` you mentioned definitely makes sense but we can support them
in the final follow-up.
Another thing to discuss is what the TaskFlow syntax for mixed-language Dags
should look like in the TS SDK.
For the Go SDK, Go itself can define an `arg:` tag with the real argument
name from the Python side:
```go
// Flat parameters: bound in declaration order after the injectables
// (sdk.TIRunContext, *slog.Logger, context.Context, client interfaces).
// Arity or declared-type mismatches fail the task before its body runs.
func ViaFlatArgs(ctx sdk.TIRunContext, log *slog.Logger,
name string, count int, ratio float64, enabled bool,
tags []string, config Config, numbers []int, note *string) (any, error)
// TaskInput struct: exported fields bind per call-argument name.
type ViaStructArgTagInput struct {
sdk.TaskInput
Region string `arg:"region_code"` // explicit argument name via tag
Threshold float64 `arg:"threshold"` // untagged fields bind their
verbatim Go field name
}
func ViaStructArgTag(ctx sdk.TIRunContext, log *slog.Logger, input
ViaStructArgTagInput) (any, error)
```
Do you have any thoughts on the equivalent for the TS world, where a user
could explicitly reference the source argument name?
> FYI that the source arguments will be introduced in
https://github.com/apache/airflow/pull/69757 -- that propagate as
`arg_bindings` in the `StartupDetails.TIRunContext`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
