rachael-ds commented on a change in pull request #14033:
URL: https://github.com/apache/airflow/pull/14033#discussion_r570919993
##########
File path: airflow/providers/google/cloud/example_dags/example_gcs.py
##########
@@ -151,6 +156,41 @@
copy_file >> delete_bucket_2
delete_files >> delete_bucket_1
+with models.DAG(
+ "example_gcs_sensors",
+ start_date=days_ago(1),
+ schedule_interval=None,
+ tags=['example'],
+) as dag2:
+ create_bucket = GCSCreateBucketOperator(
+ task_id="create_bucket", bucket_name=BUCKET_1, project_id=PROJECT_ID
+ )
+ upload_file = LocalFilesystemToGCSOperator(
+ task_id="upload_file",
+ src=PATH_TO_UPLOAD_FILE,
+ dst=BUCKET_FILE_LOCATION,
+ bucket=BUCKET_1,
+ )
+ # [START howto_sensor_object_exists_task]
+ gcs_object_exists = GCSObjectExistenceSensor(
+ bucket=BUCKET_1,
+ object=PATH_TO_UPLOAD_FILE,
+ mode='poke',
+ task_id="gcs_object_exists_task",
+ )
+ # [END howto_sensor_object_exists_task]
+ # [START howto_sensor_object_with_prefix_exists_task]
+ gcs_object_with_prefix_exists = GCSObjectsWtihPrefixExistenceSensor(
+ bucket=BUCKET_1,
+ prefix=PATH_TO_UPLOAD_FILE_PREFIX,
+ mode='poke',
+ task_id="gcs_object_with_prefix_exists_task",
+ )
+ # [END howto_sensor_object_with_prefix_exists_task]
+ delete_bucket = GCSDeleteBucketOperator(task_id="delete_bucket",
bucket_name=BUCKET_1)
+
+ create_bucket >> upload_file >> [gcs_object_exists,
gcs_object_with_prefix_exists] >> delete_bucket
Review comment:
The system test for this DAG will confirm that the
GCSObjectExistenceSensor and GCSObjectsWtihPrefixExistenceSensor can be called
in this context successfully.
It doesn't explicitly test for poke functionality (I see we have separate
unit tests for this) - I was hesitant to have one task executing the sensor
while try to have another upload a file in the same dag (I'm not sure how the
tests handle task concurrency/whether this is even best practice). Open to
thoughts/suggestions on this thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
