Hello,
Thank you both for getting back to me so quickly,
Can you explain how I can get this to work with the latest version? I'm
trying to submit a spark job with the following ( With the hadoop 3
release).
Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages"
"com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:3.0.0-alpha1"
"sparkr-shell"',
But then I get:
16/12/16 21:47:56 ERROR RBackendHandler: createSparkContext on
org.apache.spark.api.r.RRDD failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
java.lang.IllegalAccessError: tried to access method
org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V
from class org.apache.hadoop.fs.s3a.S3AInstrumentation
How can I set my environment to pull down the hadoop 2.8 release?
Thank you again for your help.
On Fri, Dec 16, 2016 at 9:33 PM, Mingliang Liu <[email protected]> wrote:
> You may need the hadoop-aws*-version*.jar (and aws-java-sdk.jar) in your
> classpath.
>
> For new project, have a look at the latest S3A progress:
> https://github.com/apache/hadoop/blob/release-3.
> 0.0-alpha1-RC0/hadoop-tools/hadoop-aws/src/site/markdown/
> tools/hadoop-aws/index.md
>
> Good luck,
>
> L
>
> On Dec 16, 2016, at 1:24 PM, cs user <[email protected]> wrote:
>
> Hello,
>
> Can someone tell me which version of Hadoop I should be able to find this
> class in?
>
> Thanks!
>
>
>
