I think you got it, Joe, it does make sense now. I had wondered about the Unlimited Strength policy, but for... MD5? I couldn't get over that. Now, I am a bit surprised more people haven't run into this.
Dan, I'm afraid I don't know anything about configuring the java.security providers. Telling you to sabotage BouncyCastle doesn't sound like good advice, but I'm not sure how to configure a happy coexistence between BouncyCastle and the AWS SDK. Thanks, James On Thu, Aug 18, 2016 at 7:54 AM, Joe Skora <[email protected]> wrote: > The JDK does not require MessageDigest > <https://github.com/openjdk-mirror/jdk/blob/caciocavallo/jdk7/master/src/share/classes/java/security/MessageDigest.java#L483> > implementations to implement Cloneable, but the AWS code > <https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/internal/MD5DigestCalculatingInputStream.java#L66> > seems to expect digests to be cloneable. Looking at Bouncy Castle, it's > BCMessageDigest > <https://github.com/bcgit/bc-java/blob/master/prov/src/main/java/org/bouncycastle/jcajce/provider/digest/BCMessageDigest.java#L7> > does not support Cloneable. > > I'm not familiar enough to offer specifics, but if you can change to a > different message digest that could eliminate the problem. > > On Thu, Aug 18, 2016 at 9:03 AM, dgm <[email protected]> wrote: > >> James, in addition to using the BC security provider, I was already using >> the latest JCE Unlimited Strength Jurisdiction Policy Files from Oracle. >> >> >> >> Thanks, >> >> Dan Morris >> >> 443-992-2848 >> >> >> >> *From: *Daniel Morris <[email protected]> >> *Date: *Thursday, August 18, 2016 at 7:23 AM >> *To: *<[email protected]> >> *Subject: *Re: PutS3Object Error >> >> >> >> Thanks James. So when you say the JRE security provider, are you >> referring to bouncycastle? If so, I am currently using >> bcprov-jdk16-1.46.jar. >> >> >> >> Thanks, >> >> Dan M >> >> >> >> *From: *James Wing <[email protected]> >> *Reply-To: *<[email protected]> >> *Date: *Wednesday, August 17, 2016 at 11:45 PM >> >> *To: *<[email protected]> >> *Subject: *Re: PutS3Object Error >> >> >> >> The IllegalStateException appears to be a response to a >> CloneNotSupportException thrown by java.security.MessageDigest. That >> suggests that your JRE's java.security provider does not support the clone >> method used by the AWS SDK. Is your java.security setup the JRE default? >> Would it be possible to try a different, maybe newer, JRE? >> >> And it's always a bonus when the code throwing your exception is >> foreshadowed with the ominous "should never occur": >> >> private MessageDigest cloneFrom(MessageDigest from) { >> try { >> return (MessageDigest)from.clone(); >> } catch (CloneNotSupportedException e) { // should never occur >> throw new IllegalStateException("unexpected", e); >> } >> } >> >> (see https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk >> -s3/src/main/java/com/amazonaws/services/s3/internal/MD5Dige >> stCalculatingInputStream.java#L54) >> >> Thanks, >> >> James >> >> >> >> On Wed, Aug 17, 2016 at 5:40 PM, dgm <[email protected]> wrote: >> >> Sure thing! >> >> >> >> Version: Nifi 0.7.0, RHEL 6.8, with Java JRE 1.7.0_101 >> >> >> >> Logs/Stacktrace: >> >> >> >> 2016-08-17 19:01:09,339 INFO [Timer-Driven Process Thread-3] >> o.a.n.p.attributes.UpdateAttribute UpdateAttribute[id=7b485633-a6e3-4aee- >> a671-d5b6f2599040] Updated attributes for StandardFlowFileRecord[uuid=bd >> a2e6c2-7a20-4354-a65f-3ed5b9ef4d06,claim=StandardContentClaim >> [resourceClaim=StandardResourceClaim[id=1471471716616-15, >> container=default, section=15], offset=44598, length=3467],offset=0, >> name=20160817190051056-low-text_json_good.json,size=3467]; transferring >> to 'success' >> >> 29 2016-08-17 19:01:09,346 ERROR [Timer-Driven Process Thread-3] >> o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=a0836995-c787-40b1- >> a8bb-c6f39bb9f37a] PutS3Object[id=a0836995-c787-40b1-a8bb-c6f39bb9f37a] >> failed to process due to java.lang.IllegalStateException: >> unexpected; rolling back session: java.lang.IllegalStateException: >> unexpected >> >> 30 2016-08-17 19:01:09,349 ERROR [Timer-Driven Process Thread-3] >> o.a.nifi.processors.aws.s3.PutS3Object >> >> 31 java.lang.IllegalStateException: unexpected >> >> 32 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInput >> Stream.cloneFrom(MD5DigestCalculatingInputStream.java:54) ~[na:na] >> >> 33 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInput >> Stream.mark(MD5DigestCalculatingInputStream.java:66) ~[na:na] >> >> 34 at >> com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96) >> ~[na:na] >> >> 35 at >> com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96) >> ~[na:na] >> >> 36 at >> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:472) >> ~[na:na] >> >> 37 at >> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317) >> ~[na:na] >> >> 38 at >> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595) >> ~[na:na] >> >> 39 at >> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1382) >> ~[na:na] >> >> 40 at >> org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:446) >> ~[na:na] >> >> 41 at org.apache.nifi.controller.repository.StandardProcessSession >> .read(StandardProcessSession.java:1851) ~[nifi-framework-core-0.7.0. >> jar:0.7.0] >> >> 42 at org.apache.nifi.controller.repository.StandardProcessSession >> .read(StandardProcessSession.java:1822) ~[nifi-framework-core-0.7.0. >> jar:0.7.0] >> >> 43 at >> org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:400) >> ~[na:na] >> >> 44 at >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> ~[nifi-api-0.7.0.jar:0.7.0] >> >> 45 at org.apache.nifi.controller.StandardProcessorNode.onTrigger(S >> tandardProcessorNode.java:1054) [nifi-framework-core-0.7.0.jar:0.7.0] >> >> 46 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >> .call(ContinuallyRunProcessorTask.java:136) >> [nifi-framework-core-0.7.0. jar:0.7.0] >> >> 47 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >> .call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0. >> jar:0.7.0] >> >> 48 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA >> gent$1.run(TimerDrivenSchedulingAgent.java:127) >> [nifi-framework-core-0. 7.0.jar:0.7.0] >> >> 49 at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> [na:1.7.0_101] >> >> 50 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >> [na:1.7.0_101] >> >> 51 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.access$301(ScheduledThreadPoolExecutor.java:178) >> [na:1.7. 0_101] >> >> 52 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_101] >> >> 53 at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> [na:1.7.0_101] >> >> 54 at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> [na:1.7.0_101] >> >> 55 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] >> >> 56 Caused by: java.lang.CloneNotSupportedException: null >> >> 57 at >> java.security.MessageDigest$Delegate.clone(MessageDigest.java:571) >> ~[na:1.7.0_101] >> >> 58 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInput >> Stream.cloneFrom(MD5DigestCalculatingInputStream.java:52) ~[na:na] >> >> 59 ... 23 common frames omitted >> >> 60 2016-08-17 19:01:09,350 ERROR [Timer-Driven Process Thread-3] >> o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=a0836995-c787-40b1- >> a8bb-c6f39bb9f37a] PutS3Object[id=a0836995-c787-40b1-a8bb-c6f39bb9f37a] >> failed to process session due to java.lang.IllegalStateException: >> unexpected: java.lang.IllegalStateException: unexpected >> >> 61 2016-08-17 19:01:09,354 ERROR [Timer-Driven Process Thread-3] >> o.a.nifi.processors.aws.s3.PutS3Object >> >> 62 java.lang.IllegalStateException: unexpected >> >> 63 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInput >> Stream.cloneFrom(MD5DigestCalculatingInputStream.java:54) ~[na:na] >> >> 64 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInput >> Stream.mark(MD5DigestCalculatingInputStream.java:66) ~[na:na] >> >> 65 at >> com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96) >> ~[na:na] >> >> 66 at >> com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96) >> ~[na:na] >> >> 67 at >> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:472) >> ~[na:na] >> >> 68 at >> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317) >> ~[na:na] >> >> 69 at >> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595) >> ~[na:na] >> >> 70 at >> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1382) >> ~[na:na] >> >> 71 at >> org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:446) >> ~[na:na] >> >> 72 at org.apache.nifi.controller.repository.StandardProcessSession >> .read(StandardProcessSession.java:1851) ~[nifi-framework-core-0.7.0. >> jar:0.7.0] >> >> 73 at org.apache.nifi.controller.repository.StandardProcessSession >> .read(StandardProcessSession.java:1822) ~[nifi-framework-core-0.7.0. >> jar:0.7.0] >> >> 74 at >> org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:400) >> ~[na:na] >> >> 75 at >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> ~[nifi-api-0.7.0.jar:0.7.0] >> >> 76 at org.apache.nifi.controller.StandardProcessorNode.onTrigger(S >> tandardProcessorNode.java:1054) ~[nifi-framework-core-0.7.0.jar:0.7.0] >> >> 77 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >> .call(ContinuallyRunProcessorTask.java:136) >> [nifi-framework-core-0.7.0. jar:0.7.0] >> >> 78 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >> .call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0. >> jar:0.7.0] >> >> 79 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA >> gent$1.run(TimerDrivenSchedulingAgent.java:127) >> [nifi-framework-core-0. 7.0.jar:0.7.0] >> >> 80 at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> [na:1.7.0_101] >> >> 81 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >> [na:1.7.0_101] >> >> 82 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.access$301(ScheduledThreadPoolExecutor.java:178) >> [na:1.7. 0_101] >> >> 83 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_101] >> >> 84 at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> [na:1.7.0_101] >> >> 85 at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> [na:1.7.0_101] >> >> 86 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] >> >> 87 Caused by: java.lang.CloneNotSupportedException: null >> >> 88 at >> java.security.MessageDigest$Delegate.clone(MessageDigest.java:571) >> ~[na:1.7.0_101] >> >> 89 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInput >> Stream.cloneFrom(MD5DigestCalculatingInputStream.java:52) ~[na:na] >> >> 90 ... 23 common frames omitted >> >> 91 2016-08-17 19:01:09,354 WARN [Timer-Driven Process Thread-3] >> o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=a0836995-c787-40b1- >> a8bb-c6f39bb9f37a] Processor Administratively Yielded for 1 sec due to >> processing failure >> >> >> >> >> >> >> >> Thanks, >> >> Dan M >> >> >> >> *From: *James Wing <[email protected]> >> *Reply-To: *<[email protected]> >> *Date: *Wednesday, August 17, 2016 at 8:32 PM >> *To: *<[email protected]> >> *Subject: *Re: PutS3Object Error >> >> >> >> Dan, >> >> Would you please share the version of NiFi you are using? Also, would >> you please look in logs/nifi-app.log for the stack trace of the exception >> and any nested exceptions? >> >> Thanks, >> >> James >> >> >> >> On Wed, Aug 17, 2016 at 5:05 PM, dgm <[email protected]> wrote: >> >> I’m just staring to use nifi and having an issue with the PutS3Object >> processor. I believe that it's configured consistently with my other S3 >> processors, using the same accounts/permissions... receiving the >> following error in nifi: PutS3Object[id=a3d6e547-9f71-4bcd-8873-108575604ab4] >> PutS3Object[id=a3d6e547-9f71-4bcd-8873-108575604ab4] failed to process >> session due to java.lang.IllegalStateException: unexpected: >> java.lang.IllegalStateException: unexpected >> >> >> >> When I configured the processor, I’m completing all the mandatory fields >> (i.e. object key, bucket, access key, secret key, storage class, and >> region). I’ve made sure these are consistent with my other S3 processors >> (List and Fetch – of which I can successful retrieve files). >> >> >> >> Any help or guidance would be greatly appreciated! >> >> >> >> Thanks, >> >> Dan M >> >> >> >> >> > >
