[
https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884298#comment-15884298
]
Steve Loughran commented on HADOOP-14028:
-----------------------------------------
OK, final patch is in to branch-2.8+.
[~mojodna] —thank you for helping with the beta testing, finding this, and
giving us the information we needed to track down and fix the problem. The next
beta releases will have the patches. This is why participation in OSS beta
releases is so important to users as well as us developers: the beta test phase
is the best chance to get the bugs you see fixed within a few days to weeks.
> S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or
> handle part upload failures
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-14028
> URL: https://issues.apache.org/jira/browse/HADOOP-14028
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.8.0
> Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2
> Reporter: Seth Fitzsimmons
> Assignee: Steve Loughran
> Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HADOOP-14028-006.patch, HADOOP-14028-007.patch,
> HADOOP-14028-branch-2-001.patch, HADOOP-14028-branch-2-008.patch,
> HADOOP-14028-branch-2-009.patch, HADOOP-14028-branch-2.8-002.patch,
> HADOOP-14028-branch-2.8-003.patch, HADOOP-14028-branch-2.8-004.patch,
> HADOOP-14028-branch-2.8-005.patch, HADOOP-14028-branch-2.8-007.patch,
> HADOOP-14028-branch-2.8-008.patch
>
>
> I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I
> was looking for after running into the same OOM problems) and don't see it
> cleaning up the disk-cached blocks.
> I'm generating a ~50GB file on an instance with ~6GB free when the process
> starts. My expectation is that local copies of the blocks would be deleted
> after those parts finish uploading, but I'm seeing more than 15 blocks in
> /tmp (and none of them have been deleted thus far).
> I see that DiskBlock deletes temporary files when closed, but is it closed
> after individual blocks have finished uploading or when the entire file has
> been fully written to the FS (full upload completed, including all parts)?
> As a temporary workaround to avoid running out of space, I'm listing files,
> sorting by atime, and deleting anything older than the first 20: `ls -ut |
> tail -n +21 | xargs rm`
> Steve Loughran says:
> > They should be deleted as soon as the upload completes; the close() call
> > that the AWS httpclient makes on the input stream triggers the deletion.
> > Though there aren't tests for it, as I recall.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
