ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add
ability to specify multiple objects to copy in GCSToGCSOperator
URL: https://github.com/apache/airflow/pull/7728#discussion_r392667594
##########
File path: airflow/providers/google/cloud/operators/gcs_to_gcs.py
##########
@@ -40,32 +37,27 @@ class GCSToGCSOperator(BaseOperator):
:param source_bucket: The source Google Cloud Storage bucket where the
object is. (templated)
:type source_bucket: str
- :param source_object: The source name of the object to copy in the Google
cloud
+ :param source_objects: A list of prefix of the objects to copy in the
Google cloud
storage bucket. (templated)
- You can use only one wildcard for objects (filenames) within your
- bucket. The wildcard can appear inside the object name or at the
- end of the object name. Appending a wildcard to the bucket name is
- unsupported.
- :type source_object: str
+ :type source_objects: List[str]
:param destination_bucket: The destination Google Cloud Storage bucket
where the object should be. If the destination_bucket is None, it
defaults
to source_bucket. (templated)
:type destination_bucket: str
:param destination_object: The destination name of the object in the
destination Google Cloud Storage bucket. (templated)
- If a wildcard is supplied in the source_object argument, this is the
- prefix that will be prepended to the final destination objects' paths.
- Note that the source path's part before the wildcard will be removed;
- if it needs to be retained it should be appended to destination_object.
- For example, with prefix ``foo/*`` and destination_object ``blah/``,
the
- file ``foo/baz`` will be copied to ``blah/baz``; to retain the prefix
write
- the destination_object as e.g. ``blah/foo``, in which case the copied
file
- will be named ``blah/foo/baz``.
+ If destination object is not specified, then it defaults to each of
the source objects.
+ For example, if source_objects = ['foo/sales','bah/inventory'], then
destination will be
+ 'foo/sales' and 'bah/inventory' if destination_object is not specified.
:type destination_object: str
:param move_object: When move object is True, the object is moved instead
of copied to the new location. This is the equivalent of a mv command
as opposed to a cp command.
:type move_object: bool
+ :type delimiter: str
Review comment:
Ok. Thanks very much for the review.
I'm thinking of removing it entirely and add back
`google_cloud_storage_conn_id` which I removed. This is for backward
compatibility. The arguments are already 10 so leaving the both arguments will
make tests not to pass.
If `delimiter` is removed, then search will now be based on source object
wildcard.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
