wyattshapiro commented on PR #18615: URL: https://github.com/apache/airflow/pull/18615#issuecomment-1184733047
@alex-astronomer @RickRen7575 Thanks for advocating for this and opening up a PR. I have also run into this issue in my use case (invoke Meltano and dbt jobs via k8s pod operator) and would benefit from this change. I know that since the scheduler reloads DAGs at a relatively high frequency (every 30s) to fill the DagBag, it reimports the DAG files at that cadence and any code that is not inside an operator will be called on import. In order to prevent making metadb calls on DagBag realod, the code needs to be wrapped in an Airflow operator. Then, the results of that operator can be passed to another downstream operator. Enter XCOM. I tested viability of XCOM by generating Meltano env vars inside a Python operator then passing it to the Meltano/dbt k8s pod operator - this way a DAG won't ping the Airflow metadb for the required connection info when the DagBag is reloaded. However, it looks like XCOM will not work yet because of this template rendering issue! I agree that the deprecation of dict input for the k8s pod operator is a separate issue that can be solved separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
