[
https://issues.apache.org/jira/browse/HTTPASYNC-99?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremy Barlow updated HTTPASYNC-99:
-----------------------------------
Attachment: ResponseCallback.java
AbnormalIOReactor.java
> Response handling problems when too many open files as request executed
> -----------------------------------------------------------------------
>
> Key: HTTPASYNC-99
> URL: https://issues.apache.org/jira/browse/HTTPASYNC-99
> Project: HttpComponents HttpAsyncClient
> Issue Type: Bug
> Affects Versions: 4.1.1
> Reporter: Jeremy Barlow
> Attachments: AbnormalIOReactor.java, ResponseCallback.java
>
>
> We have encountered a few issues with handling an HttpAsyncClient request
> which is made when the process has too many open file descriptors:
> 1) A .get() call made on the future returned from the httpclient's .execute()
> method hangs indefinitely.
> 2) No call is made to any of the methods on the
> {{FutureCallback<HttpResponse>}} object provided as an argument to the
> httpclient's .execute() method. I was expecting a call to .failed() to occur.
> 3) A call to the httpclient's .close() method does not error out but also
> does not result in all of the IOReactor "I/O Dispatcher" threads being shut
> down. In normal cases, these threads would get shut down when the .close()
> call is made.
> I've attached a couple of Java source files which demonstrate the problem -
> AbnormalIOReactor.java and ResponseCallback.java. AbnormalIOReactor.java is
> based on the example at
> https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchange.java.
> Just after the httpclient is started, this class allocates bare sockets
> repeatedly and holds onto them. This is done to force the application to get
> into the state where file descriptors have been exhausted. The
> httpclient.execute() call does not throw an error.
> The class repeatedly calls .get() on the future response with a prolonged
> timeout, breaking out of the loop when the client moves from the state of
> having .isRunning() be true to .isRunning() being false. Checking for a
> change in the .isRunning() state is a bit contrived here, ideally something
> that an API client would not have to do. I noticed from the implementation
> of the {{CloseableHttpAsyncClientBase}} constructor at
> https://svn.apache.org/repos/asf/httpcomponents/httpasyncclient/tags/4.1.1/httpasyncclient/src/main/java/org/apache/http/impl/nio/client/CloseableHttpAsyncClientBase.java
> that the reactor thread's run() method will catch an exception at request
> startup and set the client status to {{STOPPED}} and this seemed like the
> only way that a client could detect that a problem had occurred in this
> situation.
> {code:java}
> @Override
> public void run() {
> try {
> final IOEventDispatch ioEventDispatch = new
> InternalIODispatch(handler);
> connmgr.execute(ioEventDispatch);
> } catch (final Exception ex) {
> log.error("I/O reactor terminated abnormally", ex);
> } finally {
> status.set(Status.STOPPED);
> }
> }
> {code}
> The implementation of ResponseCallback.java writes a message to stdout when
> any of the methods is invoked. In this case, no message is written to stdout
> even though the request has effectively failed.
> When this program is run from the command line like {{java
> abnormalioreactor.AbnormalIOReactor}}, it will hang after the word "Done"
> appears, which occurs after the .close() method is called on the httpclient.
> When running jstack from the command line, I see that various "I/O
> Dispatcher" threads are still running for the client even though I had
> expected them to have been closed:
> {noformat}
> "I/O dispatcher 8" #19 prio=5 os_prio=31 tid=0x00007f978e800000 nid=0x6503
> runnable [0x0000000120cf6000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
> at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
> at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000000076ba79bf8> (a sun.nio.ch.Util$2)
> - locked <0x000000076ba79be8> (a
> java.util.Collections$UnmodifiableSet)
> - locked <0x000000076ba79ac8> (a sun.nio.ch.KQueueSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:257)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I had been thinking that in this situation it would be better for the
> httpclient to not be implicitly stopped - so it could be used for subsequent
> requests if/when the number of open file descriptors has dropped. Also, it
> would be good for either the httpclient.execute() call to throw an exception
> right away or at least for calls to .get() on the future returned from the
> .execute() call to throw an exception and for the .failed() method on
> {{FutureCallback<HttpResponse>}} object to be called.
> Note that I ran this application on my MacBook Pro, running OS-X Yosemite.
> We've seen this problem on other OSes as well, though, including CentOS 7.
> I reproduced this problem with a few different versions of HttpAsyncClient -
> 4.0.2, 4.1.1, custom build from trunk, custom build from the 4.1.x branch.
> This isn't a situation that we run into often with our applications - mostly
> in cases where users haven't tuned their application settings properly to
> account for the number of file descriptors that a Java process would need to
> use. It would be nice if the HttpAsyncClient handling could gracefully fail
> a request when open file descriptors have temporarily been exhausted under
> load spikes but leave the client in a usable state for future requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
