potiuk commented on issue #16038: URL: https://github.com/apache/airflow/issues/16038#issuecomment-847705704
I think there are other mechanisms in Airflow that are better for that. The default way of handling it (and much better IMHO) is to write your custom operator and reuse and compose Hooks instead of having chain of operators. Vast majority (if not all) operators are written in the way that they almost transparently pass the parameters to underlying hooks for the very purpose. Writing your own operator is easy - just a class that derives from BaseOperator and implements 'execute' method. And it has the additional advantage that you can make it a really 'custom' operator where you could hard-code the args values that are specific for your case rather than expose them as __init__ parameters. Another advantage of this is that in this case the single 'execute' method runs always on one instance of airflow worker/same process and you can pass even vast amount of data between those hooks (for example streaming it) without worrying about passing the data through yet another external storage. This is how many transfer operators work. In short - in Airflow world composability is achieved by composing Hooks rather than operators. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
