Hi Paul,
We implement the FutureCallback<HttpResponse> that gets passed into
HttpClient.execute().
In there we create the runnable and toss it on our executor service. And
inside the run() method of the runnable we read the content of the
HttpResponse.
class HttpCallback implements FutureCallback<HttpResponse> {
ContentReader _contentReader;
public HttpCallback(ContentReader contentReader) {
_contentReader = contentReader;
}
@Override
public void completed (HttpResponse result) {
_contentReader.setHttpResponse(result);
getContext().getReaderService().submit(_contentReader);
}
@Override
public void failed (Exception ex) {
ErrorType type = ErrorType.UNKNOWN_ERROR;
if (ex instanceof SocketTimeoutException) {
_log.warn(_contentReader.getEfsRequest().getFareSource().getFareSourceName()
+ ": No response ");
type = ErrorType.TIMEOUT_ABORT;
}
else {
_log.error("Error in asynchronous vendor request: " +
_contentReader.getEfsRequest().getFareSource().getFareSourceName(), ex);
}
_contentReader.getCallback().failure(type);
}
@Override
public void cancelled () {
_contentReader.getCallback().failure(ErrorType.CANCELLED);
}
}
On Fri, Mar 6, 2015 at 11:05 AM, Paul Bear <[email protected]> wrote:
> >
> > > What we do, is have the http callback handle the completed(), failed(),
> > > cancelled() but immediately wrap the HttpResponse within a Runnable and
> > > throw it over to an executor service to be handled. That way, the 2
> > threads
> > > that async http uses to handle incoming responses are doing the minimum
> > > amount of work.
> >
>
> Graeme, thanks for the details. Do you know in which class the Runnable is
> thrown over to the executor service ? I was looking into most classes of
> the HttpAsyncClient 4.1-beta1 package, but couldn't make out the class.
>
> Paul
>
>
>
> >
> > Our config looks like
> >
> > int timeout =
> > configuration.getAsyncConnectionTimeout();
> > int reactorThreadPoolSize = 16;
> > int maxConnectionsPerVendor = 10;
> > int maxConnections = 1024;
> > int numReaderThreads = 2;
> >
> > if (configuration.getAsyncReactorThreadPoolSize() !=
> null)
> > {
> > reactorThreadPoolSize =
> > configuration.getAsyncReactorThreadPoolSize();
> > }
> >
> > if (configuration.getAsyncMaxConnectionsPerVendor() !=
> > null) {
> > maxConnectionsPerVendor =
> > configuration.getAsyncMaxConnectionsPerVendor();
> > }
> >
> > if
> (configuration.getAsyncMaxConnectionsAcrossAllVendors()
> > != null) {
> > maxConnections =
> > configuration.getAsyncMaxConnectionsAcrossAllVendors();
> > }
> >
> > if (configuration.getAsyncReaderThreadPoolSize() !=
> null) {
> > numReaderThreads =
> > configuration.getAsyncReaderThreadPoolSize();
> > }
> >
> >
> > RequestConfig requestConfig;
> >
> > if (context.isUseProxy()) {
> > requestConfig = RequestConfig.custom()
> > .setSocketTimeout(timeout)
> > .setProxy(new
> HttpHost(context.getProxyHost(),
> > context.getProxyPort()))
> > .setConnectTimeout(timeout)
> > .build();
> >
> > }
> > else {
> > requestConfig = RequestConfig.custom()
> > .setSocketTimeout(timeout)
> > .setConnectTimeout(timeout)
> > .build();
> > }
> >
> > ConnectionConfig connectionConfig =
> > ConnectionConfig.custom()
> >
> .setMalformedInputAction(CodingErrorAction.IGNORE)
> >
> .setUnmappableInputAction(CodingErrorAction.IGNORE)
> > .setCharset(Consts.UTF_8)
> > .build();
> >
> > IOReactorConfig reactorConfig = IOReactorConfig.custom()
> > .setIoThreadCount(reactorThreadPoolSize)
> > .build();
> >
> > CloseableHttpAsyncClient httpClient =
> > HttpAsyncClients.custom()
> > .setDefaultRequestConfig(requestConfig)
> > .setDefaultIOReactorConfig(reactorConfig)
> > .setDefaultConnectionConfig(connectionConfig)
> > .setMaxConnPerRoute(maxConnectionsPerVendor)
> > .setMaxConnTotal(maxConnections)
> > .build();
> >
> > context.setHttpClient(httpClient);
> >
> >
> > context.setReaderService(Executors.newFixedThreadPool(numReaderThreads));
> >
> >
> >
> >
> > On Wed, Mar 4, 2015 at 10:55 AM, Paul Bear <[email protected]> wrote:
> >
> > > Graeme, thanks for the example. I wonder what the number of threads is
> > that
> > > process the responses ? Is it the number of Processors from Runtime
> > class,
> > > is it configurable ?
> > >
> > > 2015-03-04 11:04 GMT+01:00 Graeme Wallace <
> > [email protected]
> > > >:
> > >
> > > > Async Http package does what you are looking for.
> > > >
> > > > https://hc.apache.org/httpcomponents-asyncclient-dev/examples.html
> > > >
> > > > 3rd example down is where you want to start.
> > > >
> > > >
> > > > Graeme
> > > >
> > > > On Wed, Mar 4, 2015 at 9:54 AM, Paul Bear <[email protected]> wrote:
> > > >
> > > > > I have a large list of URLs (about 1 millon) and I want to run
> > > > >
> > > > > - one thread that runs through the list and asynchronously sends
> GET
> > > > > requests
> > > > > - several worker threads that process the responses
> > > > >
> > > > > Is it possible to separate sending GETs and processing responses in
> > > > > different threads using Apache Client ?
> > > > >
> > > > > Any ideas are welcome!
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Graeme Wallace
> > > > CTO
> > > > FareCompare.com
> > > > O: 972 588 1414
> > > > M: 214 681 9018
> > > >
> > >
> >
> >
> >
> > --
> > Graeme Wallace
> > CTO
> > FareCompare.com
> > O: 972 588 1414
> > M: 214 681 9018
> >
>
--
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018
