Re: Missing output partition file in S3
On 19 Sep 2016, at 18:54, Chen, Kevin mailto:kevin.c...@neustar.biz>> wrote: Hi Steve, Our S3 is on US east. But this issue also occurred when we using a S3 bucket on US west. We are using S3n. We use Spark standalone deployment. We run the job in EC2. The datasets are about 25GB. We did not have speculative execution turned on. We did not use DirectCommiter. Thanks, Kevin the closest thing I know to that on the version of Spark you are using is : https://issues.apache.org/jira/browse/SPARK-4879 , but that's assuming speculative exection is on From: Steve Loughran mailto:ste...@hortonworks.com>> Date: Friday, September 16, 2016 at 3:46 AM To: Chen Kevin mailto:kevin.c...@neustar.biz>> Cc: "user@spark.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.org>> Subject: Re: Missing output partition file in S3 On 15 Sep 2016, at 19:37, Chen, Kevin mailto:kevin.c...@neustar.biz>> wrote: Hi, Has any one encountered an issue of missing output partition file in S3 ? My spark job writes output to a S3 location. Occasionally, I noticed one partition file is missing. As a result, one chunk of data was lost. If I rerun the same job, the problem usually goes away. This has been happening pretty random. I observed once or twice a week on a daily run job. I am using Spark 1.2.1. Very much appreciated on any input, suggestion of fix/workaround. This doesn't sound good Without making any promises about being able to fix this, I would like to understand the setup to see if there is something that could be done to address this 1. Which S3 installation? US East or elsewhere 2. Which s3 client: s3n or s3a. If on hadoop 2.7+, can you switch to S3a if you haven't already (exception, if you are using AWS EMR you have to stick with their s3:// client) 3. Are you running in-EC2 or remotely? 4. How big are the datasets being generated? 5. Do you have speculative execution turned on 6. which committer? is the external "DirectCommitter", or the classic Hadoop FileOutputCommitter? If so &you are using Hadoop 2.7.x, can you try the v2 algorithm (hadoop.mapreduce.fileoutputcommitter.algorithm.version 2) I should warn that the stance of myself and colleagues is "dont commit direct to S3", write to HDFS and do a distcp when you finally copy out the data. S3 itself doesn't have enough consistency for committing output to work in the presence of all race conditions and failure modes. At least here you've noticed the problem; the thing people fear is not noticing that a problem has arisen -Steve
Re: Missing output partition file in S3
Hi Steve, Our S3 is on US east. But this issue also occurred when we using a S3 bucket on US west. We are using S3n. We use Spark standalone deployment. We run the job in EC2. The datasets are about 25GB. We did not have speculative execution turned on. We did not use DirectCommiter. Thanks, Kevin From: Steve Loughran mailto:ste...@hortonworks.com>> Date: Friday, September 16, 2016 at 3:46 AM To: Chen Kevin mailto:kevin.c...@neustar.biz>> Cc: "user@spark.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.org>> Subject: Re: Missing output partition file in S3 On 15 Sep 2016, at 19:37, Chen, Kevin mailto:kevin.c...@neustar.biz>> wrote: Hi, Has any one encountered an issue of missing output partition file in S3 ? My spark job writes output to a S3 location. Occasionally, I noticed one partition file is missing. As a result, one chunk of data was lost. If I rerun the same job, the problem usually goes away. This has been happening pretty random. I observed once or twice a week on a daily run job. I am using Spark 1.2.1. Very much appreciated on any input, suggestion of fix/workaround. This doesn't sound good Without making any promises about being able to fix this, I would like to understand the setup to see if there is something that could be done to address this 1. Which S3 installation? US East or elsewhere 2. Which s3 client: s3n or s3a. If on hadoop 2.7+, can you switch to S3a if you haven't already (exception, if you are using AWS EMR you have to stick with their s3:// client) 3. Are you running in-EC2 or remotely? 4. How big are the datasets being generated? 5. Do you have speculative execution turned on 6. which committer? is the external "DirectCommitter", or the classic Hadoop FileOutputCommitter? If so &you are using Hadoop 2.7.x, can you try the v2 algorithm (hadoop.mapreduce.fileoutputcommitter.algorithm.version 2) I should warn that the stance of myself and colleagues is "dont commit direct to S3", write to HDFS and do a distcp when you finally copy out the data. S3 itself doesn't have enough consistency for committing output to work in the presence of all race conditions and failure modes. At least here you've noticed the problem; the thing people fear is not noticing that a problem has arisen -Steve
Re: Missing output partition file in S3
Sent from my iPhone > On Sep 15, 2016, at 1:37 PM, Chen, Kevin wrote: > > Hi, > > Has any one encountered an issue of missing output partition file in S3 ? My > spark job writes output to a S3 location. Occasionally, I noticed one > partition file is missing. As a result, one chunk of data was lost. If I > rerun the same job, the problem usually goes away. This has been happening > pretty random. I observed once or twice a week on a daily run job. I am using > Spark 1.2.1. > > Very much appreciated on any input, suggestion of fix/workaround. > > >
Re: Missing output partition file in S3
are you using speculation? On 15 September 2016 at 21:37, Chen, Kevin wrote: > Hi, > > Has any one encountered an issue of missing output partition file in S3 ? > My spark job writes output to a S3 location. Occasionally, I noticed one > partition file is missing. As a result, one chunk of data was lost. If I > rerun the same job, the problem usually goes away. This has been happening > pretty random. I observed once or twice a week on a daily run job. I am > using Spark 1.2.1. > > Very much appreciated on any input, suggestion of fix/workaround. > > > >
Re: Missing output partition file in S3
On 15 Sep 2016, at 19:37, Chen, Kevin mailto:kevin.c...@neustar.biz>> wrote: Hi, Has any one encountered an issue of missing output partition file in S3 ? My spark job writes output to a S3 location. Occasionally, I noticed one partition file is missing. As a result, one chunk of data was lost. If I rerun the same job, the problem usually goes away. This has been happening pretty random. I observed once or twice a week on a daily run job. I am using Spark 1.2.1. Very much appreciated on any input, suggestion of fix/workaround. This doesn't sound good Without making any promises about being able to fix this, I would like to understand the setup to see if there is something that could be done to address this 1. Which S3 installation? US East or elsewhere 2. Which s3 client: s3n or s3a. If on hadoop 2.7+, can you switch to S3a if you haven't already (exception, if you are using AWS EMR you have to stick with their s3:// client) 3. Are you running in-EC2 or remotely? 4. How big are the datasets being generated? 5. Do you have speculative execution turned on 6. which committer? is the external "DirectCommitter", or the classic Hadoop FileOutputCommitter? If so &you are using Hadoop 2.7.x, can you try the v2 algorithm (hadoop.mapreduce.fileoutputcommitter.algorithm.version 2) I should warn that the stance of myself and colleagues is "dont commit direct to S3", write to HDFS and do a distcp when you finally copy out the data. S3 itself doesn't have enough consistency for committing output to work in the presence of all race conditions and failure modes. At least here you've noticed the problem; the thing people fear is not noticing that a problem has arisen -Steve