Re: spark 2.02 error when writing to s3

2017-01-28 Thread Steve Loughran
On 27 Jan 2017, at 23:17, VND Tremblay, Paul > wrote: Not sure what you mean by "a consistency layer on top." Any explanation would be greatly appreciated! Paul netflix's s3mper: https://github.com/Netflix/s3mper EMR consistency:

RE: spark 2.02 error when writing to s3

2017-01-27 Thread VND Tremblay, Paul
Loughran; VND Tremblay, Paul Cc: Takeshi Yamamuro; user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: spark 2.02 error when writing to s3 Can you test by enabling emrfs consistent view and use s3:// uri. http://docs.aws.amazon.com/emr/latest/ManagementGuide/enable-consisten

Re: spark 2.02 error when writing to s3

2017-01-27 Thread Steve Loughran
l Cc: Takeshi Yamamuro; user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: spark 2.02 error when writing to s3 Can you test by enabling emrfs consistent view and use s3:// uri. http://docs.aws.amazon.com/emr/latest/ManagementGuide/enable-consistent-view.html Original

RE: spark 2.02 error when writing to s3

2017-01-26 Thread VND Tremblay, Paul
; user@spark.apache.org Subject: Re: spark 2.02 error when writing to s3 Can you test by enabling emrfs consistent view and use s3:// uri. http://docs.aws.amazon.com/emr/latest/ManagementGuide/enable-consistent-view.html Original message From: Steve Loughran Date:20/01/2017 21

Re: spark 2.02 error when writing to s3

2017-01-20 Thread Neil Jonkers
mblay, Paul" <tremblay.p...@bcg.com> Cc: Takeshi Yamamuro <linguin@gmail.com>,user@spark.apache.org Subject: Re: spark 2.02 error when writing to s3 AWS S3 is eventually consistent: even after something is deleted, a LIST/GET call may show it. You may be seeing that effect; even a

Re: spark 2.02 error when writing to s3

2017-01-20 Thread Steve Loughran
:user@spark.apache.org> Subject: Re: spark 2.02 error when writing to s3 Hi, Do you get the same exception also in v2.1.0? Anyway, I saw another guy reporting the same error, I think. https://www.mail-archive.com/user@spark.apache.org/msg60882.html // maropu On Fri, Jan 20, 2017 at 5:

RE: spark 2.02 error when writing to s3

2017-01-20 Thread VND Tremblay, Paul
Subject: Re: spark 2.02 error when writing to s3 Hi, Do you get the same exception also in v2.1.0? Anyway, I saw another guy reporting the same error, I think. https://www.mail-archive.com/user@spark.apache.org/msg60882.html // maropu On Fri, Jan 20, 2017 at 5:15 AM, VND Tremblay, Paul <trembla

Re: spark 2.02 error when writing to s3

2017-01-19 Thread Palash Gupta
Hi, You need to add mode overwrite option to avoid this error. //P.Gupta Sent from Yahoo Mail on Android On Fri, 20 Jan, 2017 at 2:15 am, VND Tremblay, Paul wrote: I have come across a problem when writing CSV files to S3 in Spark 2.02. The problem does not exist

Re: spark 2.02 error when writing to s3

2017-01-19 Thread Takeshi Yamamuro
Hi, Do you get the same exception also in v2.1.0? Anyway, I saw another guy reporting the same error, I think. https://www.mail-archive.com/user@spark.apache.org/msg60882.html // maropu On Fri, Jan 20, 2017 at 5:15 AM, VND Tremblay, Paul wrote: > I have come across a

spark 2.02 error when writing to s3

2017-01-19 Thread VND Tremblay, Paul
I have come across a problem when writing CSV files to S3 in Spark 2.02. The problem does not exist in Spark 1.6. 19:09:20 Caused by: java.io.IOException: File already exists:s3://stx-apollo-pr-datascience-internal/revenue_model/part-r-00025-c48a0d52-9600-4495-913c-64ae6bf888bd.csv My code