RemyOstyn commented on PR #32383:
URL: https://github.com/apache/airflow/pull/32383#issuecomment-1842065967
Hello @yupbank,
I found the same issue when using **Airflow Task Group**. If we forget to
specify the when pulling files paths from xcoms, the `objects` will be an empty
array between double quotes `"[]"` and the **GCSDeleteObjectsOperator** will
delete the whole bucket content.
**Example:** (let's assume these tasks are in a task group)
```
list_files_from_gcs = GCSListObjectsOperator(
task_id="list_files_from_gcs",
bucket="my-bucket",
match_glob=path + "/*.csv",
)
remove_files_from_gcs = GCSDeleteObjectsOperator(
task_id="remove_files_from_gcs",
bucket_name="my-bucket,
objects="{{ ti.xcom_pull('list_files_from_gcs') }}",
)
```
Instead of:
```
list_files_from_gcs = GCSListObjectsOperator(
task_id="list_files_from_gcs",
bucket="my-bucket",
match_glob=path + "/*.csv",
)
remove_files_from_gcs = GCSDeleteObjectsOperator(
task_id="remove_files_from_gcs",
bucket_name="my-bucket",
objects="{{ ti.xcom_pull('my_task_group.list_files_from_gcs')
}}", // As you can see I have to specify the task group here to avoid deleting
the whole bucket
)
```
Is there a way to add protection when using task group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
