I had similar problems with multi part uploads. In my case the real error was something else which was being masked by this issue https://issues.apache.org/jira/browse/SPARK-6560. In the end this bad digest exception was a side effect and not the original issue. For me it was some library version conflict on EMR.
Depending on the size of the output files, you might try to just disable the multipart upload using fs.s3n.multipart.uploads.enabled Cheers, Eugen 2016-02-07 15:05 GMT-08:00 Steve Loughran <[email protected]>: > > > On 7 Feb 2016, at 07:57, Dhimant <[email protected]> wrote: > > > > at > > > com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.uploadSinglePart(MultipartUploadOutputStream.java:245) > > ... 15 more > > Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The > > Content-MD5 you specified did not match what we received. (Service: > Amazon > > S3; Status Code: 400; Error Code: BadDigest; Request ID: > 5918216A5901FCC8), > > S3 Extended Request ID: > > > QSxtYln/yXqHYpdr4BWosin/TAFsGlK1FlKfE5PcuJkNrgoblGzTNt74kEhuNcrJCRZ3mXq0oUo= > > at > > > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) > > at > > > com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) > > at > > > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) > > at > > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) > > at > > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3796) > > at > > > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1482) > > at > > > com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:140) > > ... 22 more > > > > This is amazon's own s3 client. nothing in the apache hadoop source tree. > Normally I'd say "use s3a to make s3n problems go away", but I don't know > what that does on Amazon's own EMR libraries > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
