Re: How to read files from S3 from Spark local when there is a http proxy
s3a:// has a proxy option https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html s3n: apparently gets set up differently, though I've never tested it http://stackoverflow.com/questions/20241953/hadoop-distcp-to-s3-behind-http-proxy > On 8 Sep 2015, at 13:51, tariq <donta...@gmail.com> wrote: > > Hi svelusamy, > > Were you able to make it work? I am facing the exact same problem. Getting > connection timed when trying to access S3. > > Thank you. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-read-files-from-S3-from-Spark-local-when-there-is-a-http-proxy-tp21122p24604.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to read files from S3 from Spark local when there is a http proxy
Hi svelusamy, Were you able to make it work? I am facing the exact same problem. Getting connection timed when trying to access S3. Thank you. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-read-files-from-S3-from-Spark-local-when-there-is-a-http-proxy-tp21122p24604.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to read files from S3 from Spark local when there is a http proxy
In the shell, before running the job you can actually do a *export http_proxy="http://host:port"* and see if it works. Thanks Best Regards On Tue, Sep 8, 2015 at 6:21 PM, tariq <donta...@gmail.com> wrote: > Hi svelusamy, > > Were you able to make it work? I am facing the exact same problem. Getting > connection timed when trying to access S3. > > Thank you. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-read-files-from-S3-from-Spark-local-when-there-is-a-http-proxy-tp21122p24604.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
