[
https://issues.apache.org/jira/browse/HBASE-13401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Reid updated HBASE-13401:
--------------------------------
Component/s: mapreduce
Environment: CentOS 6.5, Hadoop 2.4.0, Yarn + MapReduce2 2.4.0, Hortonworks
Data Platform 2.1.2
> Snapshot export to S3 fails with jets3t error - The Content-MD5 you specified
> did not match what we received.
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-13401
> URL: https://issues.apache.org/jira/browse/HBASE-13401
> Project: HBase
> Issue Type: Bug
> Components: hbase, mapreduce
> Affects Versions: 0.98.0
> Environment: CentOS 6.5, Hadoop 2.4.0, Yarn + MapReduce2 2.4.0,
> Hortonworks Data Platform 2.1.2
> Reporter: Joseph Reid
>
> We're running into issues exporting snapshots of large tables to Amazon S3.
> The snapshot completes successfully, but the snapshot export job throws
> jets3t errors when we attempt to export to S3.
> Error snippet, from job log:
> {code}
> 2015-04-03 16:59:16,425 INFO [main] mapreduce.Job: Task Id :
> attempt_1426532296228_55454_m_000008_1, Status : FAILED
> Error: org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode:
> 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0"
> encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5 you
> specified did not match what we
> received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
> at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleS3ServiceException(Jets3tNativeFileSystemStore.java:405)
> at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at org.apache.hadoop.fs.s3native.$Proxy19.storeFile(Unknown Source)
> at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:221)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:200)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:140)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:89)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.jets3t.service.S3ServiceException: S3 Error Message. --
> ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml
> version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The
> Content-MD5 you specified did not match what we
> received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
> at org.jets3t.service.S3Service.putObject(S3Service.java:2267)
> at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:113)
> ... 21 more
> 2015-04-03 17:03:50,613 INFO [main] mapreduce.Job: Task Id :
> attempt_1426532296228_55454_m_000010_1, Status : FAILED
> AttemptID:attempt_1426532296228_55454_m_000010_1 Timed out after 300 secs
> {\code}
> We've verified that exports to other clusters from these same snapshots work
> fine. Thus the issue appears to lie within the snapshot export utility,
> jets3t, and S3.
> "The Content-MD5 you specified did not match what we received" seems to
> indicate that the snapshot changed between when the upload started and the
> error. Can that be?
> Related to:
> [Discussion on jets3t user
> group|https://groups.google.com/forum/#!topic/jets3t-users/Bg2qh7OdE2U]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
