On 27 Jan 2017, at 23:17, VND Tremblay, Paul
> wrote:
Not sure what you mean by "a consistency layer on top." Any explanation would
be greatly appreciated!
Paul
netflix's s3mper: https://github.com/Netflix/s3mper
EMR consistency:
Loughran; VND Tremblay, Paul
Cc: Takeshi Yamamuro; user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: spark 2.02 error when writing to s3
Can you test by enabling emrfs consistent view and use s3:// uri.
http://docs.aws.amazon.com/emr/latest/ManagementGuide/enable-consisten
l
Cc: Takeshi Yamamuro; user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: spark 2.02 error when writing to s3
Can you test by enabling emrfs consistent view and use s3:// uri.
http://docs.aws.amazon.com/emr/latest/ManagementGuide/enable-consistent-view.html
Original
; user@spark.apache.org
Subject: Re: spark 2.02 error when writing to s3
Can you test by enabling emrfs consistent view and use s3:// uri.
http://docs.aws.amazon.com/emr/latest/ManagementGuide/enable-consistent-view.html
Original message
From: Steve Loughran
Date:20/01/2017 21
mblay, Paul" <tremblay.p...@bcg.com> Cc:
Takeshi Yamamuro <linguin@gmail.com>,user@spark.apache.org
Subject: Re: spark 2.02 error when writing to s3
AWS S3 is eventually consistent: even after something is deleted, a
LIST/GET call may show it. You may be seeing that effect; even a
:user@spark.apache.org>
Subject: Re: spark 2.02 error when writing to s3
Hi,
Do you get the same exception also in v2.1.0?
Anyway, I saw another guy reporting the same error, I think.
https://www.mail-archive.com/user@spark.apache.org/msg60882.html
// maropu
On Fri, Jan 20, 2017 at 5:
Subject: Re: spark 2.02 error when writing to s3
Hi,
Do you get the same exception also in v2.1.0?
Anyway, I saw another guy reporting the same error, I think.
https://www.mail-archive.com/user@spark.apache.org/msg60882.html
// maropu
On Fri, Jan 20, 2017 at 5:15 AM, VND Tremblay, Paul
<trembla
Hi,
You need to add mode overwrite option to avoid this error.
//P.Gupta
Sent from Yahoo Mail on Android
On Fri, 20 Jan, 2017 at 2:15 am, VND Tremblay, Paul
wrote:
I have come across a problem when writing CSV files to S3 in Spark 2.02. The
problem does not exist
Hi,
Do you get the same exception also in v2.1.0?
Anyway, I saw another guy reporting the same error, I think.
https://www.mail-archive.com/user@spark.apache.org/msg60882.html
// maropu
On Fri, Jan 20, 2017 at 5:15 AM, VND Tremblay, Paul
wrote:
> I have come across a
I have come across a problem when writing CSV files to S3 in Spark 2.02. The
problem does not exist in Spark 1.6.
19:09:20 Caused by: java.io.IOException: File already
exists:s3://stx-apollo-pr-datascience-internal/revenue_model/part-r-00025-c48a0d52-9600-4495-913c-64ae6bf888bd.csv
My code
10 matches
Mail list logo