KevinYang21 commented on issue #5085: [WIP][AIRFLOW-594] Add dag source plugin hooks URL: https://github.com/apache/airflow/pull/5085#issuecomment-485331808 @drewsonne Thank you for the detailed explaination. The more I think and dig into the issue the more interesting it becomes. Here's what I think: It is perfectly fine to implement some hooks for the use of both core Airflow and plugins. But in that way I think the code belongs more to the utils module instead of the plugin module. On the other hand, writing a hook to be used by both core and plugin is not what we are trying to pursue in this PR. From what I'm understanding, we're trying to make the DAG loading plugable. And I'm 100% supportive for this philosophy. In my imagination, to achieve this we would need roughly two steps, 1. is to extract a layer of abstration on DAG fetching in Airflow core and 2. is to make this layer plugable. I think both of them are within the scope of AIP-5 and seems like you are trying to achieve the latter in this PR. If what I was describing makes sense, then I think it is better to finish 1. first if we don't do 1. and 2. together. And about the implementation details of 2., to keep stuff consistent, I'd prefer to do it like [how we do plugin executor](https://github.com/apache/airflow/blob/master/airflow/executors/__init__.py#L84-L95), which we kinda need to have 1. in place to start. @ashb I image we'd still need local files as cache or so, but maybe this is not what you meant _need_. @milton0825 I'm sorry I didn't look close enough when u start the DISCUSS thread but one thing just came aross my mind is that we may want to make the DagFetcher module plugable, people like @drewsonne, me and more would love to have that( not saying that you didn't plan it, just didn't see it get pointed out loud in the AIP) @ashb @milton0825 @feng-tao Is above something similar to what you guys have in mind? I just start to put more thoughts into this and likely you guys got something better :D would love to hear more.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
