[GitHub] [airflow] drewsonne edited a comment on issue #5085: [WIP][AIRFLOW-594] Add dag source plugin hooks

[GitBox]

drewsonne edited a comment on issue #5085: [WIP][AIRFLOW-594] Add dag source 
plugin hooks
URL: https://github.com/apache/airflow/pull/5085#issuecomment-485105277
 
 
   Shooting from the hip, but to implement AIP-5, would be something like this 
on top of these hooks
   
   ```import json
   import os
   
   from airflow import LoggingMixin
   from airflow.configuration import conf
   from airflow.plugin import AirflowPluginDagSource
   from airflow.plugins_manager import AirflowPlugin
   
   log = LoggingMixin().log
   
   
   class DagBagFetcher(AirflowPluginDagSource):
       def add_dags_to_dagbag(self):
           pass
   
       def put_dags_on_disk(self):
           home_dir = conf.get('core', 'airflow_home')
           dag_repo_manifest = os.path.join(home_dir, 'dag_repositories.json')
           if os.path.isfile(dag_repo_manifest):
               log.info('Loading dag_repositories.json')
               with open(dag_repo_manifest, 'r') as fp:
                   manifest = json.load(fp)
                   for repo in manifest['dag_repositories']:
                       # ... download from s3
                       # ... write to disk
           else:
               log.info('Could not find dag_repositories.json')
   
   
   class Aip5Plugin(AirflowPlugin):
       dag_sources = [DagBagFetcher]
   ```
   
   and you could include it in the core, by adding to setup.py
   
   ```
       setup(
           # ...
           'entry_points': {
               'airflow.plugin': [
                   'dag_repo_loader = airflow.plugin.dag_repo_loader:Aip5Plugin'
               ]
           }
       )

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services