GitHub user prayaagmahajan created a discussion: Need Help with deleting
DagFiles from DB
Hi, I am trying to delete my dag files from my internal file system. I know its
easy that I go and delete those but before that, I want to know the Dag Status
and delete only the Dag's with status 'Success\Failed'
Steps I am trying.
cleanup_old_dag_files() {
DAYS_AGO=2
CUTOFF_TS=$(date -u -d "$DAYS_AGO hours ago" +"%Y-%m-%dT%H:%M:%S")
echo "Cutoff timestamp: $CUTOFF_TS"
# 1. List DAGs (skip header)
**airflow dags list **| awk 'NR>2 {print $1, $4}' | while read -r DAG_ID
LAST_PARSED; do
# LAST_PARSED may be 'None'
[[ "$LAST_PARSED" == "None" || "$DAG_ID" == "|" || "$DAG_ID" !=
*"RUN_ONCE"* ]] && continue
[[ "$LAST_PARSED" > "$CUTOFF_TS" ]] && continue
#echo "Processing DAG: $DAG_ID"
airflow dags list-runs "$DAG_ID" | \
awk 'NR>2 {print $1, $2}' | while read -r DAG_ID STATE; do
echo "DAG_ID * $DAG_ID"
echo "STATE * $STATE"
#echo "Fetched DAGs with Success and Failed state: $1 - $2"
done
done
}
But this is inefficient as I have to iterate over many dag files get their Dag
Id and then check for their status and then Os.Remove from the path.
Is there any other way round which I can use to get all dags that has status
'success/failed' in a single cli script.
I am currently using airflow 3.1.3.
GitHub link: https://github.com/apache/airflow/discussions/60954
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
