Re: Redirect erases header?
Hi
Found the answer: when building the HttpAsyncClient set a redirect strategy and
add the appropriate header:
HttpAsyncClientBuilder builder = HttpAsyncClients.custom();
builder.setRedirectStrategy(new DefaultRedirectStrategy() {
@Override
public HttpUriRequest getRedirect(HttpRequest request, HttpResponse
response, HttpContext context) throws ProtocolException {
HttpUriRequest redirectRequest = super.getRedirect(request,
response, context);
// copy “Range headers, if exist
Header[] rangeHeaders = request.getHeaders(“Range);
if (rangeHeaders != null) {
for (Header header : rangeHeaders) {
redirectRequest.addHeader(header);
}
}
return redirectRequest;
}
});
Thanks,
--
Yoram Dayagi (Gmail)
Sent with Airmail
On February 16, 2014 at 5:06:34 PM, Yoram Dayagi (Gmail) (yor...@gmail.com)
wrote:
Hi
I’m using HttpAsyncClient to execute an async request.
I noticed that when requesting a GET request for a URL that redirects to
another, a header I put in the request (Range: bytes=0-1000) is ignored and
all content is returned in the response.
Notes:
1. When accessing directly the redirected URL the execution is as expected
(only first 1000 bytes are returned)
2. The response of the original URL doesn’t contain a header Accept-Ranges:
bytes”, where as the response of the redirected URL does contain this header.
How can I overcome this problem?
--
Yoram Dayagi (Gmail)
Sent with Airmail
Re: How to know which port is newly created for the connection ?
On Fri, 2014-02-14 at 21:15 +, Kiran Chitturi wrote:
Hi,
I am using http client 4.3.2 and I have a few questions.
1) I am making lot of concurrent requests to a server. When creating
requests, I want to debug and check which ports are newly created by the
client. Currently, I can see the ports in netstat as pasted below.
tcp4 0 0 172.16.11.138.7575 172.16.11.138.52014ESTABLISHED
I enabled the debug logging for MainClientExec class and it gives me the
below information. Here it does not exactly tell me what is the source port
that is being used for the connection. How can I use this ? Also, whenever
MainClientExec prints the below line, does it mean a new connection is
created with a new port or an existing port is being used ?
2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection
{}-http://172.16.11.138:8983
Kiran
Please raise a JIRA for this change request. I'll try to include it into
the upcoming 4.3.3 release.
2) My code looks like this for creating a http Client. I pass the client to
SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ which
uses this client to make requests. SolrJ makes requests to a Solr server.
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(100);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
final CloseableHttpClient httpClient = httpBuilder.build();
This httpClient created above also throws some exceptions. Can I assume retry
is successful and does this happen because I enabled socketReuseAddress ?
HttpClient should propagate I/O exceptions to the caller which it is
unable to re-try. If your code is not catching any exceptions it should
be safe to assume requests have been successfully re-tried.
Oleg
2014-02-14 12:53:21,415 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:53:21,443 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@106] - Retrying request
2014-02-14 12:53:47,299 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:53:47,327 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@106] - Retrying request
2014-02-14 12:54:24,740 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:54:24,763 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@106] - Retrying request
2014-02-14 12:54:39,121 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:54:39,157 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@106] - Retrying request
Please let me know your suggestions.
Many Thanks for your help,
--
Kiran Chitturi,
Software Engineer,
LucidWorks.
Cell Phone: 540-577-4852
Office Phone: 650-249-4452
Email: kiran.chitt...@lucidworks.com
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How can I abort an aynsc request using HttpAsyncClient
I want to start an async GET request for a big file. Then, at some point, while
the content is still being received, I would like to cancel the request from
another thread and close all relevant resources.
I tried to achieve this behaviour by implementing AsyncByteConsumer and doing
something like the following, but I have the feeling that the connection from
the connection pool was not released:
…
@Override
protected void onByteReceived(ByteBuffer buf, IOControl ioctrl) throws
IOException {
…
ioctrl.shutdown();
…
}
…
Also, I read in documentation that using “shutdown” on a channel is bad for SSL
connections, where “close” should be called instead.
Thanks,
Yoram
On February 17, 2014 at 11:16:03 AM, Oleg Kalnichevski (ol...@apache.org) wrote:
On Sun, 2014-02-16 at 16:30 +0200, Yoram Dayagi (Gmail) wrote:
Hi
I would like to use HttpAsyncClient in order to execute async requests.
Is it possible to abort an executing request from another thread?
What exactly do you mean by aborting request execution? Unblocking a
thread awaiting on the result future? Shutting down the underlying
connection? What is it exactly you are trying to achieve?
Oleg
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How can I abort an aynsc request using HttpAsyncClient
On Mon, 2014-02-17 at 11:23 +0200, Yoram Dayagi (Gmail) wrote:
I want to start an async GET request for a big file. Then, at some point,
while the content is still being received, I would like to cancel the request
from another thread and close all relevant resources.
I tried to achieve this behaviour by implementing AsyncByteConsumer and doing
something like the following, but I have the feeling that the connection from
the connection pool was not released:
…
@Override
protected void onByteReceived(ByteBuffer buf, IOControl ioctrl)
throws IOException {
…
ioctrl.shutdown();
…
}
…
This looks correct to me. Why do you think the connection was not
released back to the pool? You can run your client with connection
management context logging turned out to find out whether of not the
pool was leaking connections.
http://hc.apache.org/httpcomponents-client-4.3.x/logging.html
Also, I read in documentation that using “shutdown” on a channel is bad for
SSL connections, where “close” should be called instead.
Given you are shutting down the connection in the middle of an HTTP
exchange I doubt that should matter.
Oleg
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How can I abort an aynsc request using HttpAsyncClient
Hi
Below is the log of the following scenario:
1. Create an async client with MaxConnPerRoute=1
2. Start a request
3. Wait for 100ms and shutdown the channel (using ioctrl.shutdown)
4. Wait for 2000ms and start another request for same url
2014/02/17 12:34:15:587 IST [DEBUG] PoolingNHttpClientConnectionManager -
Connection request: [route:
{}-http://966b.http.dal05.cdn.softlayer.net:80][total kept alive: 0; route
allocated: 0 of 1; total allocated: 0 of 10]
2014/02/17 12:34:15:681 IST [DEBUG] PoolingNHttpClientConnectionManager -
Connection leased: [id: http-outgoing-0][route:
{}-http://966b.http.dal05.cdn.softlayer.net:80][total kept alive: 0; route
allocated: 1 of 1; total allocated: 1 of 10]
2014/02/17 12:34:15:689 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][r:]: Set attribute
http.nio.exchange-handler
2014/02/17 12:34:15:690 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][rw:]: Event set [w]
2014/02/17 12:34:15:691 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][rw:]: Set attribute
http.nio.http-exchange-state
2014/02/17 12:34:15:692 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][rw:]: Event set [w]
2014/02/17 12:34:15:694 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][rw:w]: 246 bytes
written
2014/02/17 12:34:15:694 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][r:w]: Event cleared
[w]
2014/02/17 12:34:15:770 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][r:r]: 1360 bytes read
2014/02/17 12:34:15:775 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][r:r]: Shutdown
2014/02/17 12:34:17:695 IST [DEBUG] PoolingNHttpClientConnectionManager -
Connection request: [route:
{}-http://966b.http.dal05.cdn.softlayer.net:80][total kept alive: 0; route
allocated: 1 of 1; total allocated: 1 of 10]
Note that the last log of “Connection request” reports that “route allocated: 1
of 1”.
Doesn’t it mean that the connection was not released when the channel was shut
down?
From the client perspective, the consumer’s onByteReceived is never called
after this point.
Thanks,
Yoram
On February 17, 2014 at 11:46:15 AM, Oleg Kalnichevski (ol...@apache.org) wrote:
On Mon, 2014-02-17 at 11:23 +0200, Yoram Dayagi (Gmail) wrote:
I want to start an async GET request for a big file. Then, at some point,
while the content is still being received, I would like to cancel the request
from another thread and close all relevant resources.
I tried to achieve this behaviour by implementing AsyncByteConsumer and doing
something like the following, but I have the feeling that the connection from
the connection pool was not released:
…
@Override
protected void onByteReceived(ByteBuffer buf, IOControl ioctrl) throws
IOException {
…
ioctrl.shutdown();
…
}
…
This looks correct to me. Why do you think the connection was not
released back to the pool? You can run your client with connection
management context logging turned out to find out whether of not the
pool was leaking connections.
http://hc.apache.org/httpcomponents-client-4.3.x/logging.html
Also, I read in documentation that using “shutdown” on a channel is bad for
SSL connections, where “close” should be called instead.
Given you are shutting down the connection in the middle of an HTTP
exchange I doubt that should matter.
Oleg
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How can I abort an aynsc request using HttpAsyncClient
On Mon, 2014-02-17 at 12:40 +0200, Yoram Dayagi (Gmail) wrote:
Hi
Below is the log of the following scenario:
1. Create an async client with MaxConnPerRoute=1
2. Start a request
3. Wait for 100ms and shutdown the channel (using ioctrl.shutdown)
4. Wait for 2000ms and start another request for same url
2014/02/17 12:34:15:587 IST [DEBUG] PoolingNHttpClientConnectionManager -
Connection request: [route:
{}-http://966b.http.dal05.cdn.softlayer.net:80][total kept alive: 0; route
allocated: 0 of 1; total allocated: 0 of 10]
2014/02/17 12:34:15:681 IST [DEBUG] PoolingNHttpClientConnectionManager -
Connection leased: [id: http-outgoing-0][route:
{}-http://966b.http.dal05.cdn.softlayer.net:80][total kept alive: 0; route
allocated: 1 of 1; total allocated: 1 of 10]
2014/02/17 12:34:15:689 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][r:]: Set attribute
http.nio.exchange-handler
2014/02/17 12:34:15:690 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][rw:]: Event set [w]
2014/02/17 12:34:15:691 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][rw:]: Set
attribute http.nio.http-exchange-state
2014/02/17 12:34:15:692 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][rw:]: Event set [w]
2014/02/17 12:34:15:694 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][rw:w]: 246 bytes
written
2014/02/17 12:34:15:694 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][r:w]: Event
cleared [w]
2014/02/17 12:34:15:770 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][r:r]: 1360 bytes
read
2014/02/17 12:34:15:775 IST [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 10.0.0.2:58623-93.184.221.133:80[ACTIVE][r:r]: Shutdown
2014/02/17 12:34:17:695 IST [DEBUG] PoolingNHttpClientConnectionManager -
Connection request: [route:
{}-http://966b.http.dal05.cdn.softlayer.net:80][total kept alive: 0; route
allocated: 1 of 1; total allocated: 1 of 10]
Note that the last log of “Connection request” reports that “route allocated:
1 of 1”.
Doesn’t it mean that the connection was not released when the channel was
shut down?
It probably does. Please raise a JIRA for this issue.
Oleg
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
[ANNOUNCEMENT] HttpComponents Core 4.3.2 GA released
The Apache HttpComponents project is pleased to announce 4.3.2 GA release of HttpComponents Core. This is a maintenance release that fixes a number of bugs and regressions found since 4.3.1, mostly in the NIO transport components. All users of HttpCore 4.3 are advised to upgrade. Download - http://hc.apache.org/downloads.cgi Release notes - http://www.apache.org/dist/httpcomponents/httpcore/RELEASE_NOTES.txt HttpComponents site - http://hc.apache.org/ About HttpComponents Core HttpCore is a set of low level HTTP transport components that can be used to build custom client and server side HTTP services with a minimal footprint. HttpCore supports two I/O models: a blocking I/O model based on the classic Java I/O and a non-blocking, event driven I/O model based on Java NIO. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How to know which port is newly created for the connection ?
On 2/17/14 1:12 AM, Oleg Kalnichevski ol...@apache.org wrote:
On Fri, 2014-02-14 at 21:15 +, Kiran Chitturi wrote:
Hi,
I am using http client 4.3.2 and I have a few questions.
1) I am making lot of concurrent requests to a server. When creating
requests, I want to debug and check which ports are newly created by the
client. Currently, I can see the ports in netstat as pasted below.
tcp4 0 0 172.16.11.138.7575 172.16.11.138.52014
ESTABLISHED
I enabled the debug logging for MainClientExec class and it gives me
the below information. Here it does not exactly tell me what is the
source port that is being used for the connection. How can I use this ?
Also, whenever MainClientExec prints the below line, does it mean a new
connection is created with a new port or an existing port is being used ?
2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection
{}-http://172.16.11.138:8983
Kiran
Please raise a JIRA for this change request. I'll try to include it into
the upcoming 4.3.3 release.
I opened a JIRA at HTTPClient-1462
(https://issues.apache.org/jira/browse/HTTPCLIENT-1462).
Thanks,
Kiran.
2) My code looks like this for creating a http Client. I pass the
client to
SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ
which uses this client to make requests. SolrJ makes requests to a Solr
server.
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(100);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
final CloseableHttpClient httpClient = httpBuilder.build();
This httpClient created above also throws some exceptions. Can I assume
retry is successful and does this happen because I enabled
socketReuseAddress ?
HttpClient should propagate I/O exceptions to the caller which it is
unable to re-try. If your code is not catching any exceptions it should
be safe to assume requests have been successfully re-tried.
Oleg
2014-02-14 12:53:21,415 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:53:21,443 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@106] - Retrying request
2014-02-14 12:53:47,299 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:53:47,327 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@106] - Retrying request
2014-02-14 12:54:24,740 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:54:24,763 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@106] - Retrying request
2014-02-14 12:54:39,121 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:54:39,157 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@106] - Retrying request
Please let me know your suggestions.
Many Thanks for your help,
--
Kiran Chitturi,
Software Engineer,
LucidWorks.
Cell Phone: 540-577-4852
Office Phone: 650-249-4452
Email: kiran.chitt...@lucidworks.com
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How to know which port is newly created for the connection ?
On 2/17/2014 12:34 AM, Kiran Chitturi wrote: Thank you so much for your reply. If I use the 'SystemDefaultHttpClient' for creating http client, there are many options for configuring the http client like socket reuse, disabling stale check, etc.. that are not possible to configure through SolrJ. May be after SOLR-5604, there will be more options through SolrJ for configuring or one could pass their own http client with the configured parameters. That's part of my idea with that issue. Get rid of all HttpClient deprecations in Solr code and make it possible to use an HttpClient built with HttpClientBuilder. A secondary but important goal would be to fix HttpClientUtil to use the builder object, and expand the list of options it can configure. If I use Http client from 4.3.x series, do you think there is code inside SolrJ that would make http client (4.3.2) incompatible with SolrJ? After I upgraded to 4.3.1 in my SolrJ application, I saw all the deprecated HttpClient stuff. I tried switching it to HttpClientBuilder and using the client with HttpSolrServer, but my application no longer worked, and threw some ugly exceptions. I am a little bit surprised that you can get it to work, but I haven't tried CloudSolrServer. I also haven't tried again now that I'm on 4.3.2, which might be why it's working for you. More boring details, many of which have already been discussed: I switched to SolrJ's HttpClientUtil so I no longer had deprecations in my own code, and forgot about it for a few weeks. Then someone opened SOLR-5590. I tested HttpClient 4.3.1 (latest at the time) in the Lucene ivy config. The tests still passed, so I committed the change. Now Solr and one small part of Lucene are using deprecated classes and methods, so we have SOLR-5604. Every time I start trying to fix it, the errors balloon, so I think it's going to require a significant overhaul. With Solr 4.x, API compatibility with earlier 4.x releases is preferred, so it might only be possible to implement SOLR-5604 in trunk. Thanks, Shawn - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
