royberkoweee commented on a change in pull request #9667:
URL: https://github.com/apache/airflow/pull/9667#discussion_r450825113
##########
File path: airflow/providers/google/cloud/transfers/gcs_to_gcs.py
##########
@@ -276,6 +280,21 @@ def _copy_source_with_wildcard(self, hook, prefix):
self.log.info('Delimiter ignored because wildcard is in prefix')
prefix_, delimiter = prefix.split(WILDCARD, 1)
objects = hook.list(self.source_bucket, prefix=prefix_,
delimiter=delimiter)
+ if not self.replace:
+ # If we are not replacing, list all files in the Destination GCS
bucket
+ # and only keep those files which are present in
+ # Source GCS bucket and not in Destination GCS bucket
+
+ existing_objects = hook.list(self.destination_bucket,
prefix=prefix_, delimiter=delimiter)
+
+ objects = list(set(objects) - set(existing_objects))
Review comment:
Testing on python3.7 I see set() - set() returns a set.
Therefore, I wanted to convert it to a list, in order to stay consistent
with `objects` when replace flag is True.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
