[
https://issues.apache.org/jira/browse/HADOOP-16900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052576#comment-17052576
]
Andrew Olson commented on HADOOP-16900:
---------------------------------------
Looked into the DistCp issue further. We didn't use the "-update" (aka sync
folders) option because we're copying the contents of a nested directory tree,
so the length check logic here [1] in the canSkip method wasn't reached. The
mere existence of the target file caused it to be skipped on the DistCp map
task attempt retry allowing the DistCp job to complete successfully.
{noformat}
2020-03-05 15:56:33,995 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper:
Copying hdfs://cluster/path/to/file.avro to s3a://bucket/path/to/file.avro
2020-03-05 15:56:34,163 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper:
Skipping copy of hdfs://cluster/path/to/file.avro to
s3a://bucket/path/to/file.avro
{noformat}
In the initial failed DistCp map task attempt, the DistCp
RetriableFileCopyCommand wrapper did attempt the copy 3 times with the same
"partNumber must be between 1 and 10000 inclusive, but is 10001" error each
time. I think that's all what we would expect to see, there's nothing to fix in
DistCp. If S3A had abandoned the multipart upload instead of incorrectly
sending a CompleteMultipartUpload request, the DistCp job would have failed
appropriately.
[1]
https://github.com/cloudera/hadoop-common/blob/cdh5.14.4-release/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L385
> Very large files can be truncated when written through S3AFileSystem
> --------------------------------------------------------------------
>
> Key: HADOOP-16900
> URL: https://issues.apache.org/jira/browse/HADOOP-16900
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.2.1
> Reporter: Andrew Olson
> Assignee: Steve Loughran
> Priority: Major
> Labels: s3
>
> If a written file size exceeds 10,000 * {{fs.s3a.multipart.size}}, a corrupt
> truncation of the S3 object will occur as the maximum number of parts in a
> multipart upload is 10,000 as specific by the S3 API and there is an apparent
> bug where this failure is not fatal, and the multipart upload is allowed to
> be marked as completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
