Thanks Michael! i didnt swap out openjdk for oracle version, i’ll try that next.
> On Mar 13, 2015, at 9:10 AM, Michael Starch <[email protected]> wrote: > > John, > > Did you remove the code at the top of the script that sets JAVA_EXT_JARS? > Are you running the oracle java still? > > If both are true, can you send me the latest error? > > -Michael > > > > On Thu, Mar 12, 2015 at 5:14 PM, John Reynolds <[email protected]> wrote: > >> Thanks Michael, this works for me (filemanager starts), however i’m still >> getting the same datatransferer error >> >>> On Mar 12, 2015, at 1:52 PM, Michael Starch <[email protected]> wrote: >>> >>> John, >>> >>> I should be more verbose. The java classpath traditionally did not pick >> up >>> multiple jars, so it was super labor intensive to setup. These days you >>> can use * inside the classpath to pick up multiple jars....but it must be >>> in " " because otherwise the shell will glob the * if it is outside of >>> quotes. >>> >>> If this doesn't work, try the other recommendations in: >>> >>> >>> >> http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath >>> >>> -Michael >>> >>> >>> On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch <[email protected]> >> wrote: >>> >>>> John, >>>> >>>> Change: >>>> -classpath "$FILEMGR_HOME"/lib \ >>>> To: >>>> -classpath "$FILEMGR_HOME/lib/*" \ >>>> >>>> -Michael >>>> >>>> >>>> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <[email protected]> >>>> wrote: >>>> >>>>> Thanks Michael, if i modify the filemgr-client to look like this (at >> the >>>>> end) >>>>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ >>>>> -classpath "$FILEMGR_HOME"/lib \ >>>>> >>>>> >> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties >>>>> \ >>>>> >> -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ >>>>> >>>>> >> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml >>>>> \ >>>>> >>>>> >> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml >>>>> \ >>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ >>>>> >>>>> (replacing ext jars with -classpath) then i get >>>>> Error: Could not find or load main class >>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient >>>>> i assume i’m doing something wrong with classpath but not sure what >>>>> >>>>>> On Mar 12, 2015, at 11:31 AM, Michael Starch <[email protected]> >>>>> wrote: >>>>>> >>>>>> John, >>>>>> >>>>>> Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS >>>>>> there. If so, it is clobbering the default path for "extension" jars, >>>>> and >>>>>> your java encryption jars are not being picked up. If it does set >>>>>> JAVA_EXT_JARS you have two options: >>>>>> >>>>>> 1. Move all your encryption jars into FILEMGR_HOME/lib/ >>>>>> 2. update filemgr-client script to us classpath to specify the jars in >>>>> the >>>>>> FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS >>>>>> >>>>>> >>>>>> -Michael >>>>>> >>>>>> >>>>>> On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <[email protected]> >>>>> wrote: >>>>>> >>>>>>> Hi Michael >>>>>>> yeah it’s openjdk 1.7 (“1.7.0_75") >>>>>>> i did download the the unlimited encryption jar from oracle and >>>>> replaced >>>>>>> the local_policy / us_export_policy jars in javahome/jre/lib/security >>>>>>> more i read, maybe limited by jce.jar >>>>>>> >>>>>>> i dont have anything special set for extension jars >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Mar 12, 2015, at 10:35 AM, Michael Starch <[email protected]> >>>>> wrote: >>>>>>>> >>>>>>>> John, >>>>>>>> >>>>>>>> What version of the JDK are you running, and what is your extension >>>>> jars >>>>>>>> environment variable set to. Do you have the java cryptology jar >>>>>>> included >>>>>>>> (Oracle JDK usually has this, I don't know if Open JDK does). >>>>>>>> >>>>>>>> "Algorithm HmacSHA1 not available" is usually thrown when Java >> cannot >>>>>>> find >>>>>>>> the java crypto jar used to calculate the given hash. >>>>>>>> >>>>>>>> -Michael >>>>>>>> >>>>>>>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <[email protected] >>> >>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Lewis, >>>>>>>>> using the latest docker buggtb/oodt image, which i assume is .8 >>>>>>>>> here’s the command i’m running to test the upload >>>>>>>>> >>>>>>>>> filemgr-client --url http://localhost:9000 --operation >>>>> --ingestProduct >>>>>>>>> --productName test --productStructure Flat --productTypeName >>>>> GenericFile >>>>>>>>> --metadataFile file:///root/test.txt.met --refs >> file:///root/test.txt >>>>>>>>> >>>>>>>>> i verified that i can upload to the path using the s3 tools on the >>>>> box / >>>>>>>>> with same credentials i put in the properties file >>>>>>>>> >>>>>>>>> here’s the full exception returned: >>>>>>>>> >>>>>>>>> >> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>>>> >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) >>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) >>>>>>>>> at >>>>>>>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) >>>>>>>>> at >>>>> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) >>>>>>>>> at >>>>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) >>>>>>>>> at >>>>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) >>>>>>>>> at >>>>> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) >>>>>>>>> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> Caused by: >>>>>>>>> >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) >>>>>>>>> ... 12 more >>>>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to >> calculate a >>>>>>>>> request signature: Unable to calculate a request signature: >> Algorithm >>>>>>>>> HmacSHA1 not available >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) >>>>>>>>> at >>>>>>>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) >>>>>>>>> at >>>>>>>>> >>>>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) >>>>>>>>> ... 13 more >>>>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to >> calculate a >>>>>>>>> request signature: Algorithm HmacSHA1 not available >>>>>>>>> at >>>>>>>>> >> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) >>>>>>>>> ... 20 more >>>>>>>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm >> HmacSHA1 >>>>>>> not >>>>>>>>> available >>>>>>>>> at javax.crypto.Mac.getInstance(Mac.java:176) >>>>>>>>> at >>>>>>>>> >> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86) >>>>>>>>> ... 21 more >>>>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: >>>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >>>>> Error >>>>>>>>> ingesting product >>>>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 >>>>>>> ] >>>>>>>>> : >>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) >>>>>>>>> at >>>>>>>>> >>>>> >> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) >>>>>>>>> at >>>>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) >>>>>>>>> at >>>>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) >>>>>>>>> at >>>>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) >>>>>>>>> at >>>>>>>>> >>>>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) >>>>>>>>> at >>>>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) >>>>>>>>> java.lang.Exception: Failed to ingest product >>>>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : >>>>>>>>> java.lang.Exception: >>>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >>>>> Error >>>>>>>>> ingesting product >>>>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 >>>>>>> ] >>>>>>>>> : >>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) >>>>>>>>> at >>>>>>>>> >>>>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) >>>>>>>>> at >>>>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) >>>>>>>>> at >>>>>>>>> >>>>>>> >>>>> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) >>>>>>>>> ERROR: Failed to ingest product 'test' : Failed to ingest product >>>>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : >>>>>>>>> java.lang.Exception: >>>>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >>>>> Error >>>>>>>>> ingesting product >>>>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 >>>>>>> ] >>>>>>>>> : >>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney < >>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>> Hi John, >>>>>>>>>> What beau on of OODT are you using? >>>>>>>>>> The file types should be encapsulated by the factory >> implementation >>>>> I >>>>>>>>> would >>>>>>>>>> guess. It should be. Case of mike this the default filemgr store >>>>> then >>>>>>> off >>>>>>>>>> you go. >>>>>>>>>> Can you provide a paste of how your arriving at your exception >>>>> please? >>>>>>>>>> Lewia >>>>>>>>>> >>>>>>>>>> On Wednesday, March 11, 2015, John Reynolds <[email protected]> >>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Should S3DataTransferer work with the generic file type from the >>>>> get >>>>>>> go >>>>>>>>> or >>>>>>>>>>> is there anything i need to change >>>>>>>>>>> i was using the radix docker but getting 'Caused by: >>>>>>>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not >>>>>>>>> available’ >>>>>>>>>>> i tried downloading the jce but no dice >>>>>>>>>>> i might try just building on my local machine and trying again >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney < >>>>>>>>>>> [email protected] <javascript:;>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hey John, >>>>>>>>>>>> You're right, you've nearly answered everything here ;) >>>>>>>>>>>> OK, so your properties can be located here >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124 >>>>>>>>>>>> You'll see the usual key, value pairs in there which you can ad >>>>>>> before >>>>>>>>>>>> build the project and packaging filemgr.properties along with >> your >>>>>>>>>>> compiled >>>>>>>>>>>> code. >>>>>>>>>>>> Regarding the following, >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds < >>>>> [email protected] >>>>>>>>>>> <javascript:;>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> in the near future, i will want to pass in a security token as >>>>> well >>>>>>> to >>>>>>>>>>> the >>>>>>>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer >>>>>>> class >>>>>>>>>>>>> (which expires after an hour), what’s the best practice to read >>>>> in >>>>>>>>> this >>>>>>>>>>>>> transient data from the command line (or wherever else i would >>>>>>>>> initiate >>>>>>>>>>> a >>>>>>>>>>>>> job from) since this wouldnt go in a properties file >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Do you mean that your security token expires hourly? As oppose >> to >>>>>>> your >>>>>>>>> s3 >>>>>>>>>>>> client? >>>>>>>>>>>> Lets try and defined exactly what you are after here. >>>>>>>>>>>> Thanks >>>>>>>>>>>> Lewis >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Lewis* >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>> >> >>
