Yes, my script as shown tries to have os.system() call right after dag
definition part - sorry, I meant I had dag generation code (some commented out
parts) and then triggering the dag. Sorry for the confusion.
A better example would be
-----
import os
with open('../airflow/dags/uniqueId.py', 'w') as fp:
fcont = """
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
with DAG(
dag_id='uniqueId13',
default_args={
'depends_on_past': False,
'email': ['[email protected]'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
},
description='A simple tutorial DAG',
schedule_interval=timedelta(days=1),
start_date=datetime(2022, 4, 11),
catchup=False,
tags=['example'],
) as dag:
t1 = BashOperator(
task_id='print_date',
bash_command='date',
)
t2 = BashOperator(
task_id='sleep',
depends_on_past=False,
bash_command='sleep 5',
retries=3,
)
t1 >> t2
"""
fp.write(fcont)
os.system('airflow dags trigger uniqueId13')
-----
Thanks,
Alex
________________________________________
From: Alexander Mikhaylov <[email protected]>
Sent: Thursday, April 14, 2022 10:15 AM
To: [email protected]
Subject: Re: how to create and run DAG in Airflow 2?
No, Daniel, I'm trying to do a different thing.
I'm trying to have a program which would 1) create a DAG and then 2) launch
this created DAG. Preferably immediately after creation.
To have it, I started with trying to create DAG from the Python program,
programmatically - that is, I create DAG() object and then add it to globals()
list. There are examples (Astronomer website) where the author claims DAGs
created this way are showing in the list of DAGs of Airflow UI. I suspect it
worked with Airflow version 1.x, but it doesn't work for me with Airflow 2.2.4
, at least I don't see them.
Next, I've tried to generate a python file - a full dag file, with imports,
operators, >> chains etc. - and store it into the dags folder of Airflow. The
storing works - I have the file there, and that file later is picked up by
Airflow, and I see that generated DAG in UI, can launch it, that's ok. The
problem is that I want my program - which generates that mydag.py script - to
trigger the dag. This dag can be triggered from command line (airflow dags
trigger mydagid --conf '{}'), or using REST API (curl -X POST -d '{}' -H
'Content-Type...' -H 'Authorization...' http://localhost:8080/api/dags...) -
but my original program, the one which generates the dag file, can't launch it,
producing
airflow.exceptions.DagNotFound: Dag id mydagid not found in DagModel
(dag id is dag-specific). I've tried to call os.system('sleep 30') and
os.system('airflow dags trigger mydagid') - still getting the above error
message.
Again, os.system() calls are not in the dag file, but in the Python program
which generates dag file.
What can be changed here?..
Thanks,
Alex
________________________________________
From: Daniel Standish <[email protected]>
Sent: Thursday, April 14, 2022 10:03 AM
To: [email protected]
Subject: Re: how to create and run DAG in Airflow 2?
Ok so your dag should not do any triggering. dag file should just define dag.
so remove any `os.*` stuff.
don't need to mess with globals.
make sure your dag file is in dags folder
in terminal you can do `airflow info | grep dags_folder` to see what the
current configured dags folder is
you can test that your dag parses correctly with `python /path/dagfile.py` if
no errors there, and it's in your dag folder you should be good.
next check do `airflow dags list`. you should see it there.
next you can trigger from CLI with `airflow dags trigger` or something like
that. but scheduler must be running already for this to work. otherwise
trigger from UI. before starting webserver, in the terminal wher you will
start it, check that you can see the dag with airflow dags list.
don't trigger the dag from the dag file itself.
