chronitis commented on a change in pull request #4084: [AIRFLOW-3205] Support
multipart uploads to GCS
URL: https://github.com/apache/incubator-airflow/pull/4084#discussion_r228971761
##########
File path: airflow/contrib/hooks/gcs_hook.py
##########
@@ -177,7 +177,9 @@ def download(self, bucket, object, filename=None):
# pylint:disable=redefined-builtin
def upload(self, bucket, object, filename,
- mime_type='application/octet-stream', gzip=False):
+ mime_type='application/octet-stream', gzip=False,
+ multipart=False, chunksize=256 * 1024 * 1024,
Review comment:
Avoiding two options seems beneficial, but the onus is then on every call
site to specify not just that the upload should be multipart, but explicitly
select a size.
Would this work
```
upload(..., multipart=False) # default, single-part upload
upload(..., multipart=True) # multipart with default chunksize
upload(..., multipart=1024 * 1024) # multipart with chosen chunksize
```
with `false` being the default value in the signature?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
