Re: Quick one... AWS SDK version?

2017-10-08 Thread Jonathan Kelly
a the s3a client, you will need >>> the SDK version to match the hadoop-aws JAR of the same version of Hadoop >>> your JARs have. Similarly, if you were using spark-kinesis, it needs to be >>> in sync there. >>> >>> >>> *From:* Steve Loughra

Re: Quick one... AWS SDK version?

2017-10-08 Thread Tushar Sudake
u were using spark-kinesis, it needs to be >> in sync there. >> >> >> *From:* Steve Loughran [mailto:ste...@hortonworks.com >> ] >> *Sent:* Tuesday, October 03, 2017 2:20 PM >> *To:* JG Perrin >> *Cc:* user@spark.apache.org >> *Subject:* Re: Quick on

Re: Quick one... AWS SDK version?

2017-10-07 Thread Jean Georges Perrin
gt; JARs have. Similarly, if you were using spark-kinesis, it needs to be in >>> sync there. >>>> >>>> From: Steve Loughran [mailto:ste...@hortonworks.com] >>>> Sent: Tuesday, October 03, 2017 2:20 PM >>>> To: JG Perrin >>>> Cc:

Re: Quick one... AWS SDK version?

2017-10-07 Thread Marco Mistroni
op >> your JARs have. Similarly, if you were using spark-kinesis, it needs to be >> in sync there. >> >> >> *From:* Steve Loughran [mailto:ste...@hortonworks.com >> ] >> *Sent:* Tuesday, October 03, 2017 2:20 PM >> *To:* JG Perrin >> *Cc:* user@

Re: Quick one... AWS SDK version?

2017-10-06 Thread Jonathan Kelly
om:* Steve Loughran [mailto:ste...@hortonworks.com > ] > *Sent:* Tuesday, October 03, 2017 2:20 PM > *To:* JG Perrin > *Cc:* user@spark.apache.org > *Subject:* Re: Quick one... AWS SDK version? > > > > On 3 Oct 2017, at 02:28, JG Perrin wrote: > > Hey Sparkians, > &g

Re: Quick one... AWS SDK version?

2017-10-04 Thread Steve Loughran
is.com>> Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: Quick one... AWS SDK version? On 3 Oct 2017, at 02:28, JG Perrin mailto:jper...@lumeris.com>> wrote: Hey Sparkians, What version of AWS Java SDK do you use with Spark 2.2? Do you stick with the

RE: Quick one... AWS SDK version?

2017-10-03 Thread JG Perrin
: Quick one... AWS SDK version? On 3 Oct 2017, at 02:28, JG Perrin mailto:jper...@lumeris.com>> wrote: Hey Sparkians, What version of AWS Java SDK do you use with Spark 2.2? Do you stick with the Hadoop 2.7.3 libs? You generally to have to stick with the version which hadoop was built wi

RE: Quick one... AWS SDK version?

2017-10-03 Thread JG Perrin
Thanks Yash… this is helpful! From: Yash Sharma [mailto:yash...@gmail.com] Sent: Tuesday, October 03, 2017 1:02 AM To: JG Perrin ; user@spark.apache.org Subject: Re: Quick one... AWS SDK version? Hi JG, Here are my cluster configs if it helps. Cheers. EMR: emr-5.8.0 Hadoop distribution

Re: Quick one... AWS SDK version?

2017-10-03 Thread Steve Loughran
On 3 Oct 2017, at 02:28, JG Perrin mailto:jper...@lumeris.com>> wrote: Hey Sparkians, What version of AWS Java SDK do you use with Spark 2.2? Do you stick with the Hadoop 2.7.3 libs? You generally to have to stick with the version which hadoop was built with I'm afraid...very brittle depende

Re: Quick one... AWS SDK version?

2017-10-02 Thread Yash Sharma
Hi JG, Here are my cluster configs if it helps. Cheers. EMR: emr-5.8.0 Hadoop distribution: Amazon 2.7.3 AWS sdk: /usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.160.jar Applications: Hive 2.3.0 Spark 2.2.0 Tez 0.8.4 On Tue, 3 Oct 2017 at 12:29 JG Perrin wrote: > Hey Sparkians, > > > > What ve