Async httpclient with SOCKS5 proxy
Dear Users, Can you please help me with the examples to configure Async HttpClient with SOCKS5 proxy.. I read that the same is not supported for Async httpclient.. Thanks, Aparna
Re: Fwd: HttpClient5 : simple example on how to configure timeout after build()
On 5/17/2018 3:51 AM, Oleg Kalnichevski wrote: HttpClientBuilder in HC4 got overloaded with so many connection management parameters which could easily get rendered ineffective by explicitly passing an instance of HttpClientConnectionManager to the builder. The same could with HC5 would look like that. Thanks for the information on how to set the parameters I was looking for. I've upgraded to HC5 in one app, and all the tests are still passing. It also seems to work with real traffic. I'm explicitly creating the inner objects rather than building them on the fly like your code example does. Probably no real difference in what actually happens on execution, I just find this code style easier to read. final SocketConfig sc = SocketConfig.custom().setSoTimeout(Timeout.ofSeconds(10)).build(); final HttpClientConnectionManager cm = PoolingHttpClientConnectionManagerBuilder.create() .setMaxConnTotal(1000).setMaxConnPerRoute(200).setDefaultSocketConfig(sc).build(); final RequestConfig rc = RequestConfig.custom().setConnectionTimeout(Timeout.ofSeconds(5)) .build(); final CloseableHttpClient client = HttpClientBuilder.create().setConnectionManager(cm) .setDefaultRequestConfig(rc).build(); Are there any plans to make the creation of builder objects more uniform? Sometimes it's custom(), sometimes it's create().It would be nice if there was consistency. Thanks, Shawn - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Fwd: HttpClient5 : simple example on how to configure timeout after build()
Great! Thanks Oleg.
On 17/05/18 13:00, Oleg Kalnichevski wrote:
On Thu, 2018-05-17 at 02:21 +0300, / wrote:
Thanks I understand.
I would appreciate if some concrete code or pointer to examples is
given
regarding:
1) Create a shared HTTPClient
2) how a given thread is passed=given the shared client and changes,
say, connection timeout AND ALSO its userAgentString.
(the userAgentString is not something I want to change but I try to
push
this to its limit).
I tweaked the ClientMultiThreadedExecution example a little. Hope this
helps.
Oleg
---
public class ClientMultiThreadedExecution {
public static void main(String[] args) throws Exception {
// Create an HttpClient with the ThreadSafeClientConnManager.
// This connection manager must be used if more than one thread will
// be using the HttpClient.
PoolingHttpClientConnectionManager cm = new
PoolingHttpClientConnectionManager();
cm.setMaxTotal(100);
CloseableHttpClient httpclient = HttpClients.custom()
.setUserAgent(null)
.addInterceptorLast(new HttpRequestInterceptor() {
@Override
public void process(
HttpRequest request,
HttpContext context) throws HttpException,
IOException {
final String userAgent = (String)
context.getAttribute("user-agent");
if (userAgent != null) {
request.addHeader(HTTP.USER_AGENT, userAgent);
}
}
})
.setConnectionManager(cm)
.build();
try {
// create an array of URIs to perform GETs on
String[] urisToGet = {
"http://hc.apache.org/;,
"http://hc.apache.org/httpcomponents-core-ga/;,
"http://hc.apache.org/httpcomponents-client-ga/;,
};
// create a thread for each URI
GetThread[] threads = new GetThread[urisToGet.length];
for (int i = 0; i < threads.length; i++) {
HttpGet httpget = new HttpGet(urisToGet[i]);
HttpClientContext clientContext = HttpClientContext.create();
clientContext.setRequestConfig(RequestConfig.custom()
.setSocketTimeout(500 * i)
.build());
clientContext.setAttribute("user-agent", "My-User-Agent-" + (i
+ 1));
threads[i] = new GetThread(httpclient, clientContext, httpget,
i + 1);
}
// start the threads
for (int j = 0; j < threads.length; j++) {
threads[j].start();
}
// join the threads
for (int j = 0; j < threads.length; j++) {
threads[j].join();
}
} finally {
httpclient.close();
}
}
/**
* A thread that performs a GET.
*/
static class GetThread extends Thread {
private final CloseableHttpClient httpClient;
private final HttpClientContext context;
private final HttpGet httpget;
private final int id;
public GetThread(CloseableHttpClient httpClient, HttpClientContext
context, HttpGet httpget, int id) {
this.httpClient = httpClient;
this.context = context;
this.httpget = httpget;
this.id = id;
}
/**
* Executes the GetMethod and prints some status information.
*/
@Override
public void run() {
try {
System.out.println(id + " - about to get something from " +
httpget.getURI());
CloseableHttpResponse response = httpClient.execute(httpget,
context);
try {
System.out.println(id + " - get executed");
// get the response body as an array of bytes
HttpEntity entity = response.getEntity();
if (entity != null) {
byte[] bytes = EntityUtils.toByteArray(entity);
System.out.println(id + " - " + bytes.length + " bytes
read");
}
} finally {
response.close();
}
} catch (Exception e) {
System.out.println(id + " - error: " + e);
}
}
}
}
---
bw
a.
On 16/05/18 22:54, Oleg Kalnichevski wrote:
On Wed, 2018-05-16 at 15:56 +, Daly, Paul wrote:
I canot speak for HttpClient5, but If you are just looking to
change
some timeouts on the request, in 4.5.5, I do this (as recommended
from this emai list some time ago!).
I guess its not too differenent for HC5 (?)
- Create a shared HTTPClient instance. this is used by all
Re: Fwd: HttpClient5 : simple example on how to configure timeout after build()
On Thu, 2018-05-17 at 02:21 +0300, / wrote:
> Thanks I understand.
>
> I would appreciate if some concrete code or pointer to examples is
> given
> regarding:
>
> 1) Create a shared HTTPClient
> 2) how a given thread is passed=given the shared client and changes,
> say, connection timeout AND ALSO its userAgentString.
>
> (the userAgentString is not something I want to change but I try to
> push
> this to its limit).
>
I tweaked the ClientMultiThreadedExecution example a little. Hope this
helps.
Oleg
---
public class ClientMultiThreadedExecution {
public static void main(String[] args) throws Exception {
// Create an HttpClient with the ThreadSafeClientConnManager.
// This connection manager must be used if more than one thread will
// be using the HttpClient.
PoolingHttpClientConnectionManager cm = new
PoolingHttpClientConnectionManager();
cm.setMaxTotal(100);
CloseableHttpClient httpclient = HttpClients.custom()
.setUserAgent(null)
.addInterceptorLast(new HttpRequestInterceptor() {
@Override
public void process(
HttpRequest request,
HttpContext context) throws HttpException,
IOException {
final String userAgent = (String)
context.getAttribute("user-agent");
if (userAgent != null) {
request.addHeader(HTTP.USER_AGENT, userAgent);
}
}
})
.setConnectionManager(cm)
.build();
try {
// create an array of URIs to perform GETs on
String[] urisToGet = {
"http://hc.apache.org/;,
"http://hc.apache.org/httpcomponents-core-ga/;,
"http://hc.apache.org/httpcomponents-client-ga/;,
};
// create a thread for each URI
GetThread[] threads = new GetThread[urisToGet.length];
for (int i = 0; i < threads.length; i++) {
HttpGet httpget = new HttpGet(urisToGet[i]);
HttpClientContext clientContext = HttpClientContext.create();
clientContext.setRequestConfig(RequestConfig.custom()
.setSocketTimeout(500 * i)
.build());
clientContext.setAttribute("user-agent", "My-User-Agent-" + (i
+ 1));
threads[i] = new GetThread(httpclient, clientContext, httpget,
i + 1);
}
// start the threads
for (int j = 0; j < threads.length; j++) {
threads[j].start();
}
// join the threads
for (int j = 0; j < threads.length; j++) {
threads[j].join();
}
} finally {
httpclient.close();
}
}
/**
* A thread that performs a GET.
*/
static class GetThread extends Thread {
private final CloseableHttpClient httpClient;
private final HttpClientContext context;
private final HttpGet httpget;
private final int id;
public GetThread(CloseableHttpClient httpClient, HttpClientContext
context, HttpGet httpget, int id) {
this.httpClient = httpClient;
this.context = context;
this.httpget = httpget;
this.id = id;
}
/**
* Executes the GetMethod and prints some status information.
*/
@Override
public void run() {
try {
System.out.println(id + " - about to get something from " +
httpget.getURI());
CloseableHttpResponse response = httpClient.execute(httpget,
context);
try {
System.out.println(id + " - get executed");
// get the response body as an array of bytes
HttpEntity entity = response.getEntity();
if (entity != null) {
byte[] bytes = EntityUtils.toByteArray(entity);
System.out.println(id + " - " + bytes.length + " bytes
read");
}
} finally {
response.close();
}
} catch (Exception e) {
System.out.println(id + " - error: " + e);
}
}
}
}
---
> bw
> a.
>
> On 16/05/18 22:54, Oleg Kalnichevski wrote:
> > On Wed, 2018-05-16 at 15:56 +, Daly, Paul wrote:
> > > I canot speak for HttpClient5, but If you are just looking to
> > > change
> > > some timeouts on the request, in 4.5.5, I do this (as recommended
> > > from this emai list some time ago!).
> > >
> > > I guess its not too differenent for HC5 (?)
> > >
> > >
> > > - Create a shared HTTPClient instance. this is used by all
> > > requests
> > > in the JVM and is instantiated on first use.
>
Re: Fwd: HttpClient5 : simple example on how to configure timeout after build()
On Wed, 2018-05-16 at 14:11 -0600, Shawn Heisey wrote: > On 5/16/2018 8:42 AM, Shawn Heisey wrote: > > RequestConfig rc = > > RequestConfig.custom().setConnectTimeout(15000) > > .setSocketTimeout(12).build(); > > httpClient = HttpClients.custom().setDefaultRequestConfig(rc) > > .setMaxConnPerRoute(300).setMaxConnTotal(5000).disableAutomaticRetr > > ies() > > .build(); > > I have attempted to upgrade httpclient in one of my projects from > 4.5.x > to the latest 5.0 beta release, and this code no longer compiles. On > RequestConfig.Builder, the setSocketTimeout method no longer exists. > On > HttpClientBuilder, the setMaxConnPerRoute and setMaxConnTotal methods > no > longer exist. These methods were not deprecated in 4.5.x. > > Is it still possible to set a socket timeout default? Is it still > possible to increase the maximum number of connections that a client > can > make? If so, how is it done? > > Thanks, > Shawn > Hi Shawn HttpClientBuilder in HC4 got overloaded with so many connection management parameters which could easily get rendered ineffective by explicitly passing an instance of HttpClientConnectionManager to the builder. The same could with HC5 would look like that. --- CloseableHttpClient client = HttpClientBuilder.create() .setConnectionManager(PoolingHttpClientConnectionManagerBuilder.create() .setMaxConnTotal(200) .setMaxConnPerRoute(20) .setDefaultSocketConfig(SocketConfig.custom() .setSoTimeout(Timeout.ofMinutes(1L)) .build()) .build()) .setDefaultRequestConfig(RequestConfig.custom() .setConnectionTimeout(Timeout.ofSeconds(30L)) .build()) .build(); --- Does this help in any way? Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Default IOReactorConfig instance has no socket timeout
On Wed, 2018-05-16 at 22:40 +0300, Дмитрий Жихарев wrote: > Oleg, > > realistically I would expect the default timeout to be around 5 > seconds, as is the default value in the > PoolingHttpClientConnectionManager ( https://github.com/apache/httpco > mponents- > client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/ht > tp/impl/io/PoolingHttpClientConnectionManager.java). IIRC, it was 2 > seconds in the 4.4 release. To me, for an http call 3 minutes looks > like an eternity. But that's just my opinion. > > I've made a small example: https://github.com/jihor/async-http-client > -notimeout-demo > > In the demo project, HttpAsyncClient 4.1.3 is used. > > The example runs 2 http post's (one without timeout, the other with a > timeout) on each of the following clients: > > > > - (sync) client with default configuration > (HttpClients.createDefault()). This client will wait endlessly if no > timeout is defined in the request; > > - (sync) client with custom RequestConfig. This client will always > exit after its timeout expires even if the request has no timeout > defined; > > - asyncClient with default configuration > (HttpAsyncClients.createDefault()). This client will wait endlessly > if no timeout is defined in the request; > > - asyncClient with custom connection manager and custom ioReactor. > This client will always exit after its timeout expires even if the > request has no timeout defined. > > > > To me, it seems like a pitfall that default configurations offer > unlimited timeouts for the requests which don't request specific > timeouts using request configs. > Dmitry There are plenty of web services (specially in the corporate world) that take minutes to spit out some sort of a response back to the client. 5 seconds as a default would be _way_ too aggressive in my opinion. In fact, anything below a minute as a default could break a lot of applications. I am in favor of using a finite socket timeout by default in HC 5 but it should be reasonably conservative. Cheers Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
