John, If I understand correctly, your ListS3/PutS3Object processors are configured with the local cntlm settings?
Proxy Host: localhost Proxy Host Port: 3128 And have you tried hitting the us-east-1 endpoint by comparison (Google searching suggests variance in CA cert acceptance)? Thanks, James On Fri, Nov 4, 2016 at 1:41 AM, John Burns <[email protected]> wrote: > Hi James > > Yes, happy to share the configuration we use: > > We have an institute-wide proxy server that requires user credentials for > each request (domain, uname, passwd, port 80 and 443 only). We run NiFi on > Linux hosts using cntlm as the local proxy. Users provide their domain, > uname and passwd to cntlm, and point their applications to localhost:3128 > as the proxy, and cntlm sends on the proper credentials to the actual proxy > for each request (if that is clear). We point GetHTTP processors etc to > cntlm and it works fine, even for https web pages. > > We have one cert that is imported into browsers, and again, all browsers > point to localhost:3128 as the proxy. This seems to work fine, we just > export http_proxy=localhost:3128 and https_proxy=localhost:3128 at the bash > shell. > > AWS endpoints are https and unfortunately aws command line tools now only > work when we specify --no-verify-ssl option, otherwise we get the > following error: > > [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) > > So I was wondering what further configuration steps I need to take to get > S3/SQS working behind our proxy. > > Many thanks > > John > > > On Thu, Nov 3, 2016 at 6:42 PM, James Wing <[email protected]> wrote: > >> The short answer is no, PutS3Object does not currently support a direct >> equivalent of the AWS CLI's --no-verify-ssl option. There is an option to >> provide your own SSLContextService, if you need to establish trust with >> your proxy server (maybe, I'm not sure). >> >> https://nifi.apache.org/docs/nifi-docs/components/org.apache >> .nifi.ssl.StandardSSLContextService/index.html >> >> Can you share a bit more about your use case and proxy setup? I know >> there are other NiFi installations using proxy servers against S3, and I do >> not believe they have had this problem. >> >> Last, I believe I foolishly stated in an earlier email that the AWS CLI >> was a good comparison tool, but I might have to flip-flop now that we're >> bringing proxy settings and SSL verification into the picture. Are you >> sure the CLI is using your proxy similarly? >> >> Thanks, >> >> James >> >> On Thu, Nov 3, 2016 at 5:58 AM, John Burns <[email protected]> wrote: >> >>> Hi, >>> >>> I have a workflow that compresses an file then invokes PutS3Object to >>> store in an S3 bucket. This processor works fine in a non-proxy >>> environment, where PutS3Object is parameterised correctly with the proxy >>> settings, but in a proxy environment I get the following error shown in the >>> stack trace. >>> >>> Testing from the AWS cli tools, I need to use the --no-verify-ssl >>> parameter: >>> >>> aws s3 ls --no-verify-ssl s3://nifibucket/ >>> >>> Is there an equivalent "--no-verify-ss"for the PutS3Object processor? >>> >>> Thanks >>> >>> John >>> >>> >>> ERROR [Timer-Driven Process Thread-10] >>> o.a.nifi.processors.aws.s3.PutS3Object >>> PutS3Object[id=26ea1644-0158-1000-be29-271b59722ea4] Failed to put >>> StandardFlowFileRecord[uuid=72488dde-07c8-4236-8116-bd8b34d9 >>> 3716,claim=StandardContentClaim >>> [resourceClaim=StandardResourceClaim[id=1478122984174-68, >>> container=default, section=68], offset=233361, >>> length=34033],offset=0,name=bbctext.gz,size=34033] to Amazon S3 due to >>> com.amazonaws.AmazonClientException: Unable to execute HTTP request: >>> sun.security.validator.ValidatorException: PKIX path building failed: >>> sun.security.provider.certpath.SunCertPathBuilderException: unable to >>> find valid certification path to requested target: >>> com.amazonaws.AmazonClientException: Unable to execute HTTP request: >>> sun.security.validator.ValidatorException: PKIX path building failed: >>> sun.security.provider.certpath.SunCertPathBuilderException: unable to >>> find valid certification path to requested target >>> 2016-11-03 12:49:50,876 ERROR [Timer-Driven Process Thread-10] >>> o.a.nifi.processors.aws.s3.PutS3Object >>> com.amazonaws.AmazonClientException: Unable to execute HTTP request: >>> sun.security.validator.ValidatorException: PKIX path building failed: >>> sun.security.provider.certpath.SunCertPathBuilderException: unable to >>> find valid certification path to requested target >>> at >>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:706) >>> ~[aws-java-sdk-core-1.11.8.jar:na] >>> at >>> com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:447) >>> ~[aws-java-sdk-core-1.11.8.jar:na] >>> at >>> com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:409) >>> ~[aws-java-sdk-core-1.11.8.jar:na] >>> at >>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:358) >>> ~[aws-java-sdk-core-1.11.8.jar:na] >>> at >>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787) >>> ~[aws-java-sdk-s3-1.11.8.jar:na] >>> at >>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1399) >>> ~[aws-java-sdk-s3-1.11.8.jar:na] >>> at >>> org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:451) >>> ~[nifi-aws-processors-1.0.0.jar:1.0.0] >>> at org.apache.nifi.controller.repository.StandardProcessSession >>> .read(StandardProcessSession.java:1880) ~[na:na] >>> at org.apache.nifi.controller.repository.StandardProcessSession >>> .read(StandardProcessSession.java:1851) ~[na:na] >>> at >>> org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:401) >>> ~[nifi-aws-processors-1.0.0.jar:1.0.0] >>> at >>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>> [nifi-api-1.0.0.jar:1.0.0] >>> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(S >>> tandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0] >>> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >>> .call(ContinuallyRunProcessorTask.java:136) >>> [nifi-framework-core-1.0.0.jar:1.0.0] >>> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >>> .call(ContinuallyRunProcessorTask.java:47) >>> [nifi-framework-core-1.0.0.jar:1.0.0] >>> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA >>> gent$1.run(TimerDrivenSchedulingAgent.java:132) >>> [nifi-framework-core-1.0.0.jar:1.0.0] >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> [na:1.8.0_60] >>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >>> [na:1.8.0_60] >>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >>> tureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_60] >>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >>> tureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> [na:1.8.0_60] >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.ja >>> >> >> >
