[
https://issues.apache.org/jira/browse/HADOOP-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17581763#comment-17581763
]
Steve Loughran commented on HADOOP-18410:
-----------------------------------------
I have a full log of a failing process, and I do not see the debug message
"initiating asynchronous drain" which I would expect when the async drain is
kicked off. This is worrying, as it implies it is not the async operation
itself. It's notable that one other optimisation here is moving from a simple
byte by byte read and discard
{code}
while(stream.read());
{code}
to one of reading in a buffer and discarding it.
Maybe it is that which is somehow wrong so that the read does not complete.
Note that the impala code is calling unbuffer() a lot so it is keeping the
S3AInputStream instances around, but should be closing the stream. If
references to the S3 object/connection are not being released and/or close() is
not called on the s3 stream, then there won't be any release taking place.
it's actually making me think maybe we would want a thread per fs to deal with
draining and freeing up references in long lived input streams. It would
recognise when a stream have been idle for more than a threshold and close its
inner stream, and be where the draining takes place. Given the work going on
with prefetching and vectored io, I'd like to see that stable before worrying
about this.
> S3AInputStream async drain not releasing http connections
> ---------------------------------------------------------
>
> Key: HADOOP-18410
> URL: https://issues.apache.org/jira/browse/HADOOP-18410
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.9
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> Impala tcp-ds setup to s3 is hitting problems with timeout fetching http
> connections from the s3a fs pool. Disabling s3a async drain makes this
> problem *go away*. assumption, either those async ops are blocking, or they
> are not releasing references properly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
