Hi Michael,
You could use a simple loop
Have a single DAG python script which parses a dictionary with the
different patterns and parameters.
------------------------------------
#imports
...
dag_info = {
'dag_1': {'schedule_interval': value1},
'dag_2': {'schedule_interval': value2},
}
...
for dag_id, params in dags_info.items():
dag = Dag(dag_id = dag_id, schedule_interval =
params['schedule_interval'])
# Add tasks to your DAG
...
----------------------------------
You could also wrap the DAG creation in a function call which is passed the
configuration parameters needed and creates the DAG objects and tasks.
def create_dag(config):
#Instantiate a DAG
# Add tasks
# return dag object.
dag = create_dag(config)
Best,
Arthur
On Wed, Nov 1, 2017 at 11:10 AM, Michael Crawford <
[email protected]> wrote:
> Hi All,
>
> Is there a best practice regarding registering similar DAGs which only
> differ in small variables.
>
> For instance say I have an certain ETL that I want to run on several
> different environments on different schedules.
>
> All of the DAGs would essentially be the exactly the same just with a few
> different parameters.
>
> It doesn’t seem like duplicating the DAG code over and over is the right
> way to do this.
>
> Thanks,
> Mike
