o-nikolas commented on a change in pull request #19237:
URL: https://github.com/apache/airflow/pull/19237#discussion_r737742098
##########
File path:
airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_advanced.py
##########
@@ -33,13 +33,13 @@
on a YouTube channel you want to retrieve.
"""
+from datetime import datetime
from os import getenv
from airflow import DAG
from airflow.operators.dummy import DummyOperator
-from airflow.operators.python import BranchPythonOperator, get_current_context
+from airflow.operators.python import BranchPythonOperator
from airflow.providers.amazon.aws.transfers.google_api_to_s3 import
GoogleApiToS3Operator
-from airflow.utils.dates import days_ago
# [START howto_operator_google_api_to_s3_transfer_advanced_env_variables]
YOUTUBE_CONN_ID = getenv("YOUTUBE_CONN_ID", "google_cloud_default")
Review comment:
This is the default gcp conn id and I think can be removed as you have
been with `aws_default`.
##########
File path:
airflow/providers/amazon/aws/example_dags/example_dms_full_load_task.py
##########
@@ -50,20 +49,14 @@
with DAG(
dag_id='dms_full_load_task_run_dag',
- default_args={
- 'owner': 'airflow',
- 'depends_on_past': False,
- 'email': ['[email protected]'],
- 'email_on_failure': False,
- 'email_on_retry': False,
- },
dagrun_timeout=timedelta(hours=2),
- start_date=days_ago(2),
+ start_date=datetime(2021, 1, 1),
schedule_interval='0 3 * * *',
+ catchup=False,
Review comment:
Thanks for this change!
This brings up a worry I have with moving to static start dates. In the
previously example dags you haven't specified `catchup=False` (since there was
no schedule) but as soon as someone takes this example dag and starts to mutate
it to make it work for their usecase and adds a schedule to it, it's going to
cause many dag runs (which will just get worse the farther we get from 2021, 1,
1) if they don't know about `catchup=False`
I've personally been hit by this when I was starting out with Airflow.
With that in mind, a couple questions:
1. What was the motivation for moving to static start dates?
2. Should we add `catchup=False` to all examples to help new users avoid the
catchup explosion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
