[
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265416#comment-15265416
]
Steve Loughran commented on HADOOP-13028:
-----------------------------------------
Chris, I'm going to submit a patch with the logging enhanced.
It's not going to cover robustness of removeKeys as that turns out be
complicated enough to merit its own JIRA (HADOOP-12844), it's own patch, review
*and real test*.
in particular, as a way to create the problem is clearly "delete one of the
keys", we'll need the deletion code to not only skip over the failure of
individual files, —but to not treat the absence of a file as an error. That
means a test to create the failure condition on both single and multidelete,
see what comes back and then add the code to handle it (if there isn't enough
information in the exception error code, that means probing for each key's
existence).
Then the method needs to handle the situation "other failures", which could be
handled by rethrowing the exception (or one of the exceptions, for
single-file-delete failures, or otherwise reporting the failures to the caller
(e.g: return a list of nondeleted files). And that caller code needs to decide
what to do, which could vary between {{rename()}} and {{delete()}}
> add low level counter metrics for S3A; use in read performance tests
> --------------------------------------------------------------------
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, metrics
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch,
> HADOOP-13028-004.patch, HADOOP-13028-005.patch, HADOOP-13028-006.patch,
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt,
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive,
> closing connections may be expensive (a sign of a regression).
> S3A FS and individual input streams should have counters of the # of
> open/close/failure+reconnect operations, timers of how long things take. This
> can be used downstream to measure efficiency of the code (how often
> connections are being made), connection reliability, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
