mohamedawnallah commented on code in PR #33611:
URL: https://github.com/apache/beam/pull/33611#discussion_r1937576525
##########
sdks/python/apache_beam/io/gcp/gcsio.py:
##########
@@ -247,13 +247,35 @@ def open(
def delete(self, path):
"""Deletes the object at the given GCS path.
+ If the path is a directory (prefix), it deletes all blobs under that
prefix.
+
Args:
path: GCS file path pattern in the form gs://<bucket>/<name>.
"""
bucket_name, blob_name = parse_gcs_path(path)
bucket = self.client.bucket(bucket_name)
+
+ # Check if the blob is a directory (prefix) by listing objects
+ # under that prefix.
+ blobs = list(bucket.list_blobs(prefix=blob_name))
Review Comment:
> How about adding an optional parameter, such as recursive, with a default
value of false to ensure backward compatibility and avoid any performance
impact on existing pipelines using the delete function in gcsfilesystem.py?
@shunping What do you think about this comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
