What we do, is have the http callback handle the completed(), failed(),
cancelled() but immediately wrap the HttpResponse within a Runnable and
throw it over to an executor service to be handled. That way, the 2 threads
that async http uses to handle incoming responses are doing the minimum
amount of work.
Our config looks like
int timeout =
configuration.getAsyncConnectionTimeout();
int reactorThreadPoolSize = 16;
int maxConnectionsPerVendor = 10;
int maxConnections = 1024;
int numReaderThreads = 2;
if (configuration.getAsyncReactorThreadPoolSize() != null) {
reactorThreadPoolSize =
configuration.getAsyncReactorThreadPoolSize();
}
if (configuration.getAsyncMaxConnectionsPerVendor() !=
null) {
maxConnectionsPerVendor =
configuration.getAsyncMaxConnectionsPerVendor();
}
if (configuration.getAsyncMaxConnectionsAcrossAllVendors()
!= null) {
maxConnections =
configuration.getAsyncMaxConnectionsAcrossAllVendors();
}
if (configuration.getAsyncReaderThreadPoolSize() != null) {
numReaderThreads =
configuration.getAsyncReaderThreadPoolSize();
}
RequestConfig requestConfig;
if (context.isUseProxy()) {
requestConfig = RequestConfig.custom()
.setSocketTimeout(timeout)
.setProxy(new HttpHost(context.getProxyHost(),
context.getProxyPort()))
.setConnectTimeout(timeout)
.build();
}
else {
requestConfig = RequestConfig.custom()
.setSocketTimeout(timeout)
.setConnectTimeout(timeout)
.build();
}
ConnectionConfig connectionConfig =
ConnectionConfig.custom()
.setMalformedInputAction(CodingErrorAction.IGNORE)
.setUnmappableInputAction(CodingErrorAction.IGNORE)
.setCharset(Consts.UTF_8)
.build();
IOReactorConfig reactorConfig = IOReactorConfig.custom()
.setIoThreadCount(reactorThreadPoolSize)
.build();
CloseableHttpAsyncClient httpClient =
HttpAsyncClients.custom()
.setDefaultRequestConfig(requestConfig)
.setDefaultIOReactorConfig(reactorConfig)
.setDefaultConnectionConfig(connectionConfig)
.setMaxConnPerRoute(maxConnectionsPerVendor)
.setMaxConnTotal(maxConnections)
.build();
context.setHttpClient(httpClient);
context.setReaderService(Executors.newFixedThreadPool(numReaderThreads));
On Wed, Mar 4, 2015 at 10:55 AM, Paul Bear <[email protected]> wrote:
> Graeme, thanks for the example. I wonder what the number of threads is that
> process the responses ? Is it the number of Processors from Runtime class,
> is it configurable ?
>
> 2015-03-04 11:04 GMT+01:00 Graeme Wallace <[email protected]
> >:
>
> > Async Http package does what you are looking for.
> >
> > https://hc.apache.org/httpcomponents-asyncclient-dev/examples.html
> >
> > 3rd example down is where you want to start.
> >
> >
> > Graeme
> >
> > On Wed, Mar 4, 2015 at 9:54 AM, Paul Bear <[email protected]> wrote:
> >
> > > I have a large list of URLs (about 1 millon) and I want to run
> > >
> > > - one thread that runs through the list and asynchronously sends GET
> > > requests
> > > - several worker threads that process the responses
> > >
> > > Is it possible to separate sending GETs and processing responses in
> > > different threads using Apache Client ?
> > >
> > > Any ideas are welcome!
> > >
> >
> >
> >
> > --
> > Graeme Wallace
> > CTO
> > FareCompare.com
> > O: 972 588 1414
> > M: 214 681 9018
> >
>
--
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018
