Correction... try dropping to 4.6GB since 5.0GB = 4.65GiB, so you need it to be smaller than 4.65.
On Tue, Jan 12, 2021 at 8:11 PM Paul Kelly <pkelly.n...@gmail.com> wrote: > Hi Dennis, > > One thing to check... is your file 5.1 gibibytes or gigabytes? NiFi and > Amazon's multipart limits are in GiB, while Cloudian appears to be using GB > based on this Github thread [1]. 5.1 GB is 4.75 GiB, so I'm thinking NiFi > is seeing your file is 4.75 GiB and your threshold is "4.8GB" (which is > really GiB), so it's not trying to perform a multipart upload, but 4.75GiB > = 5.1GB, which is above Cloudian's single PUT operation limit, so the > Cloudian server is rejecting it. Could you try dropping your limit to "4.7 > GB" (which is really interpreted as 4.7 GiB in NiFi) to see if that works? > That should put your file above the multipart threshold for NiFi, which > will invoke the multipart upload functionality. > > [1] https://github.com/kahing/goofys/issues/139 > > Paul > > On Tue, Jan 12, 2021 at 6:13 PM Dennis N Brown <dbr...@lenovo.com> wrote: > >> Mark, Sorry… also NiFi version 1.12.1 >> >> >> >> Regards, >> >> >> >> *Dennis N Brown* >> Data Architect >> *Systems **CARE** Team* >> Lenovo USA >> >> >> US PostgreSQL Association >> >> >> [image: Email]dbr...@lenovo.com <dbr...@lenovo.com> >> >> >> >> [image: 2015-NewBrand-Lenovo-Services] >> >> >> >> >> >> *From:* Mark Payne <marka...@hotmail.com> >> *Sent:* January 12, 2021 12:20 >> *To:* users@nifi.apache.org >> *Subject:* Re: [External] Re: Having an issue with large files and >> PutS3Object >> >> >> >> Dennis, >> >> >> >> Do your logs have any stack traces in them? That would probably help to >> understand what’s happening pretty quickly. Also, which version of NiFi are >> you running? >> >> >> >> Thanks >> >> -Mark >> >> >> >> >> >> On Jan 12, 2021, at 12:12 PM, Dennis N Brown <dbr...@lenovo.com> wrote: >> >> >> >> Thanks Mark, The “Multipart Threshold” was defaulted to 5GB, and I have >> now adjusted it to 4.8GB to see if that made any difference, and it does >> not. It seems to me that NiFi should be detecting the file size and >> initiating the multipart upload, without even trying a normal S3 >> PutObject. But I’m not seeing any “multipart” messages in the error ( as I >> have seen in other posts about multipart uploads ). >> >> >> >> The Cloudian implementation appears to be using the AWS libraries, as all >> of the messages appear to have Amazon or AWS in them, and the documentation >> also states the 5GB limit for file size without using multipart upload. >> >> >> >> Regards, >> >> >> >> <image001.gif> >> >> *Dennis N Brown* >> Data Architect >> *Systems **CARE* *Team* >> Lenovo USA >> >> >> US PostgreSQL Association >> >> >> <image002.gif>dbr...@lenovo.com <dbr...@lenovo.com> >> >> >> >> <image003.gif> >> >> >> >> >> >> *From:* Mark Payne <marka...@hotmail.com> >> *Sent:* January 12, 2021 11:53 >> *To:* users@nifi.apache.org >> *Subject:* [External] Re: Having an issue with large files and >> PutS3Object >> >> >> >> Dennis, >> >> >> >> It appears that the PutS3Object processor looks at the size of the >> FlowFile and compares that to the value set in the “Multipart Threshold” >> property. If the size of the FlowFile is larger than that, it will use the >> multipart upload, with the configured size of the parts. I’m not familiar >> with the Cloudian implementation, but it may have different thresholds than >> S3? What do you have configured for the size of the Multipart Threshold? >> >> >> >> Thanks >> >> -Mark >> >> >> >> On Jan 12, 2021, at 11:32 AM, Dennis N Brown <dbr...@lenovo.com> wrote: >> >> >> >> Hello, I’m having an issue attempting to upload a large fie ( 5.1GB ) to >> S3 storage ( not AWS, but rather Cloudian implementation ). >> >> >> >> From everything I’ve read, it appears NiFi is supposed to revert to a >> multipart upload if the size of the file is greater than then “Multipart >> Threshold” defined in the PutS3Object processor. This is not happening for >> me, it just errors out with this message: >> >> ERROR o.a.nifi.processors.aws.s3.PutS3Object >> PutS3Object[id=cd683449-d9b3-1ce2-85ae-a0d900cfd488] Failed to put >> StandardFlowFileRecord[uuid=74a8d054-53cb-44d7-aca1-dabd94b50781,claim=StandardContentClaim >> [resourceClaim=StandardResourceClaim[id=1610459752598-174464, >> container=default, section=384], offset=59300, >> length=5109625339],offset=0,name=6477482,size=5109625339] to Amazon S3 due >> to com.amazonaws.services.s3.model.AmazonS3Exception: Your proposed upload >> exceeds the maximum allowed object size. (Service: Amazon S3; Status Code: >> 400; Error Code: EntityTooLarge; Request ID: >> 2f967706-9745-1564-a246-0a94ef6266cb; S3 Extended Request ID: >> a647d24f02954de69d161d24c3e48081), S3 Extended Request ID: >> a647d24f02954de69d161d24c3e48081: >> com.amazonaws.services.s3.model.AmazonS3Exception: Your proposed upload >> exceeds the maximum allowed object size. (Service: Amazon S3; Status Code: >> 400; Error Code: EntityTooLarge; Request ID: >> 2f967706-9745-1564-a246-0a94ef6266cb; S3 Extended Request ID: >> a647d24f02954de69d161d24c3e48081), S3 Extended Request ID: >> a647d24f02954de69d161d24c3e48081 >> >> com.amazonaws.services.s3.model.AmazonS3Exception: Your proposed upload >> exceeds the maximum allowed object size. (Service: Amazon S3; Status Code: >> 400; Error Code: EntityTooLarge; Request ID: >> 2f967706-9745-1564-a246-0a94ef6266cb; S3 Extended Request ID: >> a647d24f02954de69d161d24c3e48081) >> >> >> >> So my question is; Is NiFi supposed to detect the large file and then >> initiate the multipart upload, or is the server supposed to respond and >> cause NiFi to react to the size challenge? >> >> >> >> Regards, >> >> >> >> <image001.gif> >> >> *Dennis N Brown* >> Data Architect >> *Systems **CARE* *Team* >> Lenovo USA >> >> >> US PostgreSQL Association >> >> >> <image002.gif>dbr...@lenovo.com <dbr...@lenovo.com> >> >> >> >> <image003.gif> >> >> >> >
