JavierLopezT opened a new issue #16038:
URL: https://github.com/apache/airflow/issues/16038
The idea is to "append" operators to other operators without having to type
the code of them. For instance, after every S3ToSnowflakeOperator, I want a
SnowflakeOperator that makes an insert in a final table. So instead of doing
the current way, coding both operators, you could do something like this:
```
default_args = {'table': 'ex_table'}
s3_1 = S3ToSnowflakeOperator(
task_id='s3_1',
s3_bucket='blabla',
append_operator={'type': 'SnowflakeOperator',
arguments: {'sql': default_args['table']}}
```
And in the operators dependencies, it would be right after the operator it
is appended to. The task_id could be one by default, passed by the user or you
could create functions with rules to generate new task_ids based on the master
operator.
While it reduces a little bit the readability, I think it is very useful
for saving time writing some patterns that are always the same. I think it is
useful especially for checks, like for example GreatExpectations/SQL/dbt
operators right after PythonOperator/S3ToSQLOperators, etc.
I am willing to work on this but I don't know if it's technically possible.
And if so, I would need some guidance on how to approach it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
