Based on the AWS API, I'm guessing that the aws cli transparently breaks it into multiple requests using the multipart upload API.
"With a single PUT operation you can upload objects up to 5 GB in size. " - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html > Even if the processor is not already using the multipart upload, it shouldn't be hard to transition. Being able to cancel an upload and tracking the upload progress would add additional complexity if needed. On Tue, Sep 8, 2015 at 10:34 AM, John Titus (JIRA) <[email protected]> wrote: > John Titus created NIFI-937: > ------------------------------- > > Summary: PutS3 Processor cannot upload files larger than 5 GB > Key: NIFI-937 > URL: https://issues.apache.org/jira/browse/NIFI-937 > Project: Apache NiFi > Issue Type: Bug > Reporter: John Titus > Priority: Minor > > > Tried to use the PutS3 processor to upload a large file. Got the error > shown below. > > S3 doesn't allow you to upload > 5GB unless you use multi-part upload. > It'd be nice if the processor handled this transparently, as the aws cli > does. > > 2015-09-08 14:20:38,991 ERROR [Timer-Driven Process Thread-6] > o.a.nifi.processors.aws.s3.PutS3Object > PutS3Object[id=14280a26-a34a-47f6-aa7e-df404c960634] Failed to put > StandardFlowFileRecord[uuid=ba7e > > 8332-4140-44f4-a71d-e5529f0ab6ad,claim=1441721637937-1157587,offset=0,name=redacted,size > =5672981796] to Amazon S3 due to > com.amazonaws.services.s3.model.AmazonS3Exception: Your proposed up > load exceeds the maximum allowed size (Service: Amazon S3; Status Code: > 400; Error Code: EntityTooLa > rge; Request ID: 56B1AE1E6D8068C0), S3 Extended Request ID: > lQImEwX+c7TUC3+Qr97kFult/95qinlRTFM/CF6v > Z4b9DQwbwCn6auAc8epWlfHmHaU3AZCEvDc=: > com.amazonaws.services.s3.model.AmazonS3Exception: Your propos > ed upload exceeds the maximum allowed size (Service: Amazon S3; Status > Code: 400; Error Code: Entity > TooLarge; Request ID: 56B1AE1E6D8068C0), S3 Extended Request ID: > lQImEwX+c7TUC3+Qr97kFult/95qinlRTFM > /CF6vZ4b9DQwbwCn6auAc8epWlfHmHaU3AZCEvDc= > > > > > -- > This message was sent by Atlassian JIRA > (v6.3.4#6332) >
