[
https://issues.apache.org/jira/browse/HADOOP-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357725#comment-16357725
]
Aaron Fabbri commented on HADOOP-15216:
---------------------------------------
I'm working on a patch that uses {{retry()}} for {{onReadFailure()}} in the s3a
input stream, but only when s3guard is enabled.
What do you want to do for HEAD->200, GET -> 400 on the non-s3guard case?
Currently we retry once immediately. Was going to keep that behavior for now,
unless you think otherwise. We could add another retry policy config knob
"input stream retry always" or something and default to off.
{quote}
+on s3guard, GET could be 403 -> fail
{quote}
Trying to parse this. We have a couple cases in open(), when we call
getFileStatus():
- MetadataStore sees tombstone and throws FNFE.
- MetadataStore has no state for the path, returns null. We fall through to
s3GetFileStatus(), which should throw FNFE which bypasses the retry policy,
right?
> S3AInputStream to handle reconnect on read() failure better
> -----------------------------------------------------------
>
> Key: HADOOP-15216
> URL: https://issues.apache.org/jira/browse/HADOOP-15216
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Priority: Major
>
> {{S3AInputStream}} handles any IOE through a close() of stream and single
> re-invocation of the read, with
> * no backoff
> * no abort of the HTTPS connection, which is just returned to the pool, If
> httpclient hasn't noticed the failure, it may get returned to the caller on
> the next read
> Proposed
> * switch to invoker
> * retry policy explicitly for stream (EOF => throw, timeout => close, sleep,
> retry, etc)
> We could think about extending the fault injection to inject stream read
> failures intermittently too, though it would need something in S3AInputStream
> to (optionally) wrap the http input streams with the failing stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
