are you using speculation?
On 15 September 2016 at 21:37, Chen, Kevin <kevin.c...@neustar.biz> wrote:
> Has any one encountered an issue of missing output partition file in S3 ?
> My spark job writes output to a S3 location. Occasionally, I noticed one
> partition file is missing. As a result, one chunk of data was lost. If I
> rerun the same job, the problem usually goes away. This has been happening
> pretty random. I observed once or twice a week on a daily run job. I am
> using Spark 1.2.1.
> Very much appreciated on any input, suggestion of fix/workaround.