Hi Shane, Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you are using?
You are also right that you can modify the class path in bin/compute-classpath.sh as a short term fix. The current order is following the output of your target system's `ls`, so the order is not guaranteed like you speak. Right before the last line (echo $CLASSPATH), you can add a line to make sure that the JAR you want to be loaded first is at the very beginning. The long term fix would be matching the hadoop-aws.jar, aws-java-sdk.jar, and PIO's S3 storage JAR version to have a binary compatible transitive dependency tree. This could be quite time consuming though. Regards, Donald On Mon, Mar 5, 2018 at 6:57 AM, Shane Johnson <[email protected]> wrote: > Hi team, > > I have been working with Mars to get a little more insight to an error I > am seeing. After narrowing down the issue, I am trying to find a way to > force the order of loading the jars. I am a novice in Scala and am looking > for a way to continue leveraging PredictionIO on Heroku but have run into a > roadblock as I believe ordering the jars would be a change to PredictionIO > itself. I would greatly appreciate your expertise and guidance. > > Error: > > Exception > > in thread "main" java.lang.NoSuchMethodError: > com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V > > 2018-03-02T02:49:42.245083+00:00 app[release.7253]: at > org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287) > > 2018-03-02T02:49:42.245086+00:00 app[release.7253]: at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) > > 2018-03-02T02:49:42.245087+00:00 app[release.7253]: at > org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) > > 2018-03-02T02:49:42.245088+00:00 app[release.7253]: at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) > > 2018-03-02T02:49:42.245090+00:00 app[release.7253]: at > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) > > > Context: > > - I am using the predictionIO buildpack on heroku and using version 12. > - I am using S3 to Persist the model objects as I am using the > PAlgorithm vs L2LAlgorithm. > - I see an intermittent error when the Algorithm.scala file tries to > connect to S3. Sometimes it works and sometime is does not. > - When it fails I get the message above - I think I have narrowed this > down to the ordering of loading the jars. > > *I have done a number of tests and can see that the problem always occurs > with a certain order of jars. This issue appears to be intermittent and > could be solved by forcing the order of the jars. I am assuming this could > be a straightforward fix, * > *I believe it is in the compute-classpath.sh. Perhaps it's something I can > even do on my end as an end user but I don't believe that to be the case.* > > *Please let me know if this is something that can be changed and committed > or if I can contribute in someway. Again, I am fairly new to Scala and big > data infrastructure so I am intimidated by the thought of making the change > myself but we are at a roadblock:)* > > *When aws-java-sdk.jar is loaded before > (hadoop-aws.jar and pio-data-s3-assembly-0.12.0-incubating.jar) it is > always successful. If it is loaded after it fails with the message above.* > > *Success Examples:* > > --jars > file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, > file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, > file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly- > 0.12.0-incubating.jar, > ... > > --jars > file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, > file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly- > 0.12.0-incubating.jar, > file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, > ... > > *Failure Examples:* > > --jars > file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, > file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly- > 0.12.0-incubating.jar, > file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, > ... > > --jars > file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly- > 0.12.0-incubating.jar, > file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar, > file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar, > ... > > > Thank you again for your help! > > *Shane Johnson | 801.360.3350 <(801)%20360-3350>* > LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook > <https://www.facebook.com/shane.johnson.71653> >
