Monitor queue duration?

Hi I would like to monitor the queue duration for the input to some Merge Content processors. I need to check if files are queued up for long time. If so it might be because a file is missing. The queue duration information is not part of the Prometheus metriks nor the site-to-site reports.

RE: [External] Re: Having an issue with large files and PutS3Object

Thanks so much Paul! That did the trick! Set the “Multipart Threshold” to 4.6 GB ( NiFi ) and the upload worked!! Regards, [cid:image001.gif@01D6E8FF.5966C460] Dennis N Brown Data Architect Systems CARE Team Lenovo USA US PostgreSQL Association

Re: [External] Re: Having an issue with large files and PutS3Object

Correction... try dropping to 4.6GB since 5.0GB = 4.65GiB, so you need it to be smaller than 4.65. On Tue, Jan 12, 2021 at 8:11 PM Paul Kelly wrote: > Hi Dennis, > > One thing to check... is your file 5.1 gibibytes or gigabytes? NiFi and > Amazon's multipart limits are in GiB, while Cloudian

Re: [External] Re: Having an issue with large files and PutS3Object

Hi Dennis, One thing to check... is your file 5.1 gibibytes or gigabytes? NiFi and Amazon's multipart limits are in GiB, while Cloudian appears to be using GB based on this Github thread [1]. 5.1 GB is 4.75 GiB, so I'm thinking NiFi is seeing your file is 4.75 GiB and your threshold is "4.8GB"

RE: [External] Re: Having an issue with large files and PutS3Object

Mark, Sorry… also NiFi version 1.12.1 Regards, [cid:image001.gif@01D6E8E4.9A3D8980] Dennis N Brown Data Architect Systems CARE Team Lenovo USA US PostgreSQL Association [Email]dbr...@lenovo.com [2015-NewBrand-Lenovo-Services] From: Mark Payne Sent: January 12,

RE: [External] Re: Having an issue with large files and PutS3Object

Hi Mark, Here is the full error and stack trace: ERROR o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=cd683449-d9b3-1ce2-85ae-a0d900cfd488] Failed to put StandardFlowFileRecord[uuid=74a8d054-53cb-44d7-aca1-dabd94b50781,claim=StandardContentClaim

Re: [External] Re: Having an issue with large files and PutS3Object

Dennis, Do your logs have any stack traces in them? That would probably help to understand what’s happening pretty quickly. Also, which version of NiFi are you running? Thanks -Mark On Jan 12, 2021, at 12:12 PM, Dennis N Brown mailto:dbr...@lenovo.com>> wrote: Thanks Mark, The “Multipart

RE: [External] Re: Having an issue with large files and PutS3Object

Thanks Mark, The “Multipart Threshold” was defaulted to 5GB, and I have now adjusted it to 4.8GB to see if that made any difference, and it does not. It seems to me that NiFi should be detecting the file size and initiating the multipart upload, without even trying a normal S3 PutObject. But

Re: Having an issue with large files and PutS3Object

Dennis, It appears that the PutS3Object processor looks at the size of the FlowFile and compares that to the value set in the “Multipart Threshold” property. If the size of the FlowFile is larger than that, it will use the multipart upload, with the configured size of the parts. I’m not

Having an issue with large files and PutS3Object

Hello, I'm having an issue attempting to upload a large fie ( 5.1GB ) to S3 storage ( not AWS, but rather Cloudian implementation ). >From everything I've read, it appears NiFi is supposed to revert to a >multipart upload if the size of the file is greater than then "Multipart >Threshold"

GPG2 keys

Hi all, Does Encryption/Decryption processors support keys from gpg2? secring.gpg doesn't really exist in gpg2 (gpg-agent has control over the private key). Tim

Re: [EXT] AW: ExecuteSQL and Teradata

Micron Confidential Can you try setting `FINALIZE_AUTO_CLOSE=ON` in your connection string to Teradata? It’s not a best practice, but based on what the docs say, I think it might work. >From the docs: FINALIZE_AUTO_CLOSE values are OFF (default) or ON: · When set to OFF (the default), the