[
https://issues.apache.org/jira/browse/AIRFLOW-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649995#comment-16649995
]
Gordon Ball commented on AIRFLOW-3205:
--------------------------------------
The behaviour of the MySQL->GCS operator is to split the output into multiple
files, whereas this is about uploading a single logical file in multiple HTTP
requests, avoiding a size limit.
The former behaviour is useful by itself (eg, for import to BigQuery the
multiple uploaded files can be imported in parallel, instead of a slow serial
import of a single file), but is orthogonal to this case.
> GCS: Support multipart upload
> -----------------------------
>
> Key: AIRFLOW-3205
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3205
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
> Reporter: Gordon Ball
> Priority: Minor
>
> GoogleCloudStorageHook currently only provides support for uploading files in
> a single HTTP request. This means that loads fail with SSL errors for files
> larger than 2GiB (presumably a int32 overflow, might depend on which SSL
> library is being used). Multipart uploads should be supported to allow large
> uploads, and possibly increase reliability for smaller uploads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
