>
> > What we do, is have the http callback handle the completed(), failed(),
> > cancelled() but immediately wrap the HttpResponse within a Runnable and
> > throw it over to an executor service to be handled. That way, the 2
> threads
> > that async http uses to handle incoming responses are doing the minimum
> > amount of work.
>
Graeme, thanks for the details. Do you know in which class the Runnable is
thrown over to the executor service ? I was looking into most classes of
the HttpAsyncClient 4.1-beta1 package, but couldn't make out the class.
Paul
>
> Our config looks like
>
> int timeout =
> configuration.getAsyncConnectionTimeout();
> int reactorThreadPoolSize = 16;
> int maxConnectionsPerVendor = 10;
> int maxConnections = 1024;
> int numReaderThreads = 2;
>
> if (configuration.getAsyncReactorThreadPoolSize() != null)
> {
> reactorThreadPoolSize =
> configuration.getAsyncReactorThreadPoolSize();
> }
>
> if (configuration.getAsyncMaxConnectionsPerVendor() !=
> null) {
> maxConnectionsPerVendor =
> configuration.getAsyncMaxConnectionsPerVendor();
> }
>
> if (configuration.getAsyncMaxConnectionsAcrossAllVendors()
> != null) {
> maxConnections =
> configuration.getAsyncMaxConnectionsAcrossAllVendors();
> }
>
> if (configuration.getAsyncReaderThreadPoolSize() != null) {
> numReaderThreads =
> configuration.getAsyncReaderThreadPoolSize();
> }
>
>
> RequestConfig requestConfig;
>
> if (context.isUseProxy()) {
> requestConfig = RequestConfig.custom()
> .setSocketTimeout(timeout)
> .setProxy(new HttpHost(context.getProxyHost(),
> context.getProxyPort()))
> .setConnectTimeout(timeout)
> .build();
>
> }
> else {
> requestConfig = RequestConfig.custom()
> .setSocketTimeout(timeout)
> .setConnectTimeout(timeout)
> .build();
> }
>
> ConnectionConfig connectionConfig =
> ConnectionConfig.custom()
> .setMalformedInputAction(CodingErrorAction.IGNORE)
> .setUnmappableInputAction(CodingErrorAction.IGNORE)
> .setCharset(Consts.UTF_8)
> .build();
>
> IOReactorConfig reactorConfig = IOReactorConfig.custom()
> .setIoThreadCount(reactorThreadPoolSize)
> .build();
>
> CloseableHttpAsyncClient httpClient =
> HttpAsyncClients.custom()
> .setDefaultRequestConfig(requestConfig)
> .setDefaultIOReactorConfig(reactorConfig)
> .setDefaultConnectionConfig(connectionConfig)
> .setMaxConnPerRoute(maxConnectionsPerVendor)
> .setMaxConnTotal(maxConnections)
> .build();
>
> context.setHttpClient(httpClient);
>
>
> context.setReaderService(Executors.newFixedThreadPool(numReaderThreads));
>
>
>
>
> On Wed, Mar 4, 2015 at 10:55 AM, Paul Bear <[email protected]> wrote:
>
> > Graeme, thanks for the example. I wonder what the number of threads is
> that
> > process the responses ? Is it the number of Processors from Runtime
> class,
> > is it configurable ?
> >
> > 2015-03-04 11:04 GMT+01:00 Graeme Wallace <
> [email protected]
> > >:
> >
> > > Async Http package does what you are looking for.
> > >
> > > https://hc.apache.org/httpcomponents-asyncclient-dev/examples.html
> > >
> > > 3rd example down is where you want to start.
> > >
> > >
> > > Graeme
> > >
> > > On Wed, Mar 4, 2015 at 9:54 AM, Paul Bear <[email protected]> wrote:
> > >
> > > > I have a large list of URLs (about 1 millon) and I want to run
> > > >
> > > > - one thread that runs through the list and asynchronously sends GET
> > > > requests
> > > > - several worker threads that process the responses
> > > >
> > > > Is it possible to separate sending GETs and processing responses in
> > > > different threads using Apache Client ?
> > > >
> > > > Any ideas are welcome!
> > > >
> > >
> > >
> > >
> > > --
> > > Graeme Wallace
> > > CTO
> > > FareCompare.com
> > > O: 972 588 1414
> > > M: 214 681 9018
> > >
> >
>
>
>
> --
> Graeme Wallace
> CTO
> FareCompare.com
> O: 972 588 1414
> M: 214 681 9018
>
