Re: UTF8 problem?
Oleg, Thanks for the info. I ended up suspecting as much. I confirmed it by locally changing java.net.SocketOutputStream to confirm the bytes written to the socket were exactly as reported by your logging (which they were). It seems that I was being misled by the output from my HTTP sniffer (org.apache.axis.utils.tcpmon) and a failure on the part of the receiving program. Cheers, Mike From: Oleg Kalnichevski [EMAIL PROTECTED] To: HttpClient User Discussion httpclient-users@hc.apache.org Sent: Saturday, 25 October, 2008 13:23:16 Subject: Re: UTF8 problem? On Thu, 2008-10-23 at 09:36 +, Mike Kyle wrote: I set an EntityEnclosingMethod request entity to be a ByteArrayRequestEntity. This entity has the Java characters \u4E2D\u6587 as the corresponding UTF8 bytes (UTF8 = 0xE4,0xB8,0xAD,0xE6,0x96,0x87). This is confirmed by logging httpclient.wire.content. There I see the UTF8 values. Mike, What is logged in the wire log is exactly what gets written to the underlying socket. I do not think HttpClient is culprit. Oleg However what appears to really get transmitted is the corresponding Java characters rather than the UTF8 values! As this is supposedly a UTF8 encoded XML document the receiver is not best pleased. This is confirmed by performing HTTP sniffing using org.apache.axis.utils.tcpmon.My suspicion is that somehow a character handler is intervening? Debugging HttpConnection implied that the output stream is a BufferedOutputStream wrapping a java.net.SocketOutputStream. I had assumed that the socket streams would be byte oriented. The content type is set to 'text/xml; chartset=utf-8'. I am normally using HttpClient 3.0 + but the latest 3.1 appeared to react exactly the same. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
AW: NTLM-Proxy authentication and SSL-target
Hi Oleg,
Thanks for your reply.
Correct, I use the NTLM-support like it is described in the documentation
(with jcifs etc.). And that worked fine for no-SSL-targets.
Thanks for your help,
Ulrich
Ok, here is the information you requested:
executing request: GET / HTTP/1.1
via proxy: http://s-hqw2k3bd:3128
to target: https://www.verisign.com:443
[DEBUG] ClientParamsStack - 'http.protocol.max-redirects': null
[DEBUG] ClientParamsStack - 'http.route.forced-route': null
[DEBUG] ClientParamsStack - 'http.route.local-address': null
[DEBUG] ClientParamsStack - 'http.route.default-proxy':
http://s-hqw2k3bd:3128
[DEBUG] ClientParamsStack - 'http.conn-manager.timeout': null
[DEBUG] SingleClientConnManager - Get connection for route
HttpRoute[{tls}-http://s-hqw2k3bd:3128-https://www.verisign.com:443]
[DEBUG] ClientParamsStack - 'http.connection.stalecheck': null
[DEBUG] DefaultRequestDirector - Stale connection check
[DEBUG] DefaultRequestDirector - Stale connection detected
[DEBUG] DefaultClientConnection - Connection closed
[DEBUG] ClientParamsStack - 'http.connection.timeout': null
[DEBUG] ClientParamsStack - 'http.tcp.nodelay': null
[DEBUG] ClientParamsStack - 'http.socket.timeout': null
[DEBUG] ClientParamsStack - 'http.socket.linger': null
[DEBUG] ClientParamsStack - 'http.socket.buffer-size': null
[DEBUG] ClientParamsStack - 'http.protocol.element-charset': null
[DEBUG] ClientParamsStack - 'http.connection.max-line-length': null
[DEBUG] ClientParamsStack - 'http.protocol.element-charset': null
[DEBUG] ClientParamsStack - 'http.connection.max-header-count': null
[DEBUG] ClientParamsStack - 'http.connection.max-line-length': null
[DEBUG] ClientParamsStack - 'http.connection.max-status-line-garbage': null
[DEBUG] ClientParamsStack - 'http.protocol.version': HTTP/1.1
[DEBUG] ClientParamsStack - 'http.useragent': Apache-HttpClient/4.0-beta1
(java 1.4)
[DEBUG] wire - CONNECT www.verisign.com:443 HTTP/1.1[EOL]
[DEBUG] wire - User-Agent: Apache-HttpClient/4.0-beta1 (java 1.4)[EOL]
[DEBUG] wire - [EOL]
[DEBUG] headers - CONNECT www.verisign.com:443 HTTP/1.1
[DEBUG] headers - User-Agent: Apache-HttpClient/4.0-beta1 (java 1.4)
[DEBUG] wire - HTTP/1.0 407 Proxy Authentication Required[EOL]
[DEBUG] wire - Server: squid/2.6.STABLE6-NT[EOL]
[DEBUG] wire - Date: Wed, 29 Oct 2008 07:36:24 GMT[EOL]
[DEBUG] wire - Content-Type: text/html[EOL]
[DEBUG] wire - Content-Length: 1347[EOL]
[DEBUG] wire - Expires: Wed, 29 Oct 2008 07:36:24 GMT[EOL]
[DEBUG] wire - X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0[EOL]
[DEBUG] wire - Proxy-Authenticate: NTLM[EOL]
[DEBUG] wire - X-Cache: MISS from s-hqw2k3bd.pmbelz.de[EOL]
[DEBUG] wire - X-Cache-Lookup: NONE from s-hqw2k3bd.pmbelz.de:3128[EOL]
[DEBUG] wire - Via: 1.0 s-hqw2k3bd.pmbelz.de:3128
(squid/2.6.STABLE6-NT)[EOL]
[DEBUG] wire - Proxy-Connection: close[EOL]
[DEBUG] headers - HTTP/1.0 407 Proxy Authentication Required
[DEBUG] headers - Server: squid/2.6.STABLE6-NT
[DEBUG] headers - Date: Wed, 29 Oct 2008 07:36:24 GMT
[DEBUG] headers - Content-Type: text/html
[DEBUG] headers - Content-Length: 1347
[DEBUG] headers - Expires: Wed, 29 Oct 2008 07:36:24 GMT
[DEBUG] headers - X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
[DEBUG] headers - Proxy-Authenticate: NTLM
[DEBUG] headers - X-Cache: MISS from s-hqw2k3bd.pmbelz.de
[DEBUG] headers - X-Cache-Lookup: NONE from s-hqw2k3bd.pmbelz.de:3128
[DEBUG] headers - Via: 1.0 s-hqw2k3bd.pmbelz.de:3128
(squid/2.6.STABLE6-NT)
[DEBUG] headers - Proxy-Connection: close
[INFO] ResponseProcessCookies - CookieSpec not available in HTTP context
[DEBUG] ClientParamsStack - 'http.protocol.handle-authentication': null
[DEBUG] DefaultRequestDirector - Proxy requested authentication
[DEBUG] DefaultProxyAuthenticationHandler - Authentication schemes in the
order of preference: [ntlm, digest, basic]
[DEBUG] DefaultProxyAuthenticationHandler - ntlm authentication scheme
selected
[DEBUG] DefaultRequestDirector - Authorization challenge processed
[DEBUG] DefaultRequestDirector - Authentication scope: NTLM any
realm@s-hqw2k3bd:3128
[DEBUG] DefaultRequestDirector - Found credentials
[DEBUG] DefaultClientConnection - Connection closed
[DEBUG] ClientParamsStack - 'http.connection.timeout': null
[DEBUG] ClientParamsStack - 'http.tcp.nodelay': null
[DEBUG] ClientParamsStack - 'http.socket.timeout': null
[DEBUG] ClientParamsStack - 'http.socket.linger': null
[DEBUG] ClientParamsStack - 'http.socket.buffer-size': null
[DEBUG] ClientParamsStack - 'http.protocol.element-charset': null
[DEBUG] ClientParamsStack - 'http.connection.max-line-length': null
[DEBUG] ClientParamsStack - 'http.protocol.element-charset': null
[DEBUG] ClientParamsStack - 'http.connection.max-header-count': null
[DEBUG] ClientParamsStack - 'http.connection.max-line-length': null
[DEBUG] ClientParamsStack - 'http.connection.max-status-line-garbage': null
[DEBUG] ClientParamsStack - 'http.protocol.version': HTTP/1.1
[DEBUG] ClientParamsStack - 'http.useragent': Apache-HttpClient/4.0-beta1
httpclient and xpath
i know it may not be the right place to ask this but maybe one of you faces this issue before. i want to validate the response that i get from my web application and i thought about xpath. did anybody tried this before? i want to parse the response into document and then do xpath.getvalue(myxpath) and validate the data. first is there a response handler that can do this? or does someone can recommend a package than can do it? -- Regards Lior Grinfeld
RE: httpclient and xpath
maybe NekoHTML (http://nekohtml.sourceforge.net/) could be a solution? christophe -- Christophe Furmaniak Technical Operations - Reporting Testing - Atos Worldline [EMAIL PROTECTED] Tél: +33 3 20 60 81 80 - Fax : +33 3 20 60 76 76 Atos Worldline www.atosworldline.com is an Atos Origin company www.atosorigin.com -Message d'origine- De : lior grinfeld [mailto:[EMAIL PROTECTED] Envoyé : mercredi 29 octobre 2008 16:06 À : HttpClient User Discussion Objet : httpclient and xpath i know it may not be the right place to ask this but maybe one of you faces this issue before. i want to validate the response that i get from my web application and i thought about xpath. did anybody tried this before? i want to parse the response into document and then do xpath.getvalue(myxpath) and validate the data. first is there a response handler that can do this? or does someone can recommend a package than can do it? -- Regards Lior Grinfeld Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: MultithreadedHttpConnectionManager and high loads
On Tue, 2008-10-28 at 23:51 +, sebb wrote: On 28/10/2008, Oleg Kalnichevski [EMAIL PROTECTED] wrote: On Tue, 2008-10-28 at 09:19 -0700, Tatu Saloranta wrote: --- On Mon, 10/27/08, sebb [EMAIL PROTECTED] wrote: From: sebb [EMAIL PROTECTED] Subject: Re: MultithreadedHttpConnectionManager and high loads To: HttpClient User Discussion httpclient-users@hc.apache.org Date: Monday, October 27, 2008, 12:03 PM On 27/10/2008, De Groot, Cees [EMAIL PROTECTED] wrote: Hi, We're using HC in order to access an internal high-volume service (thousands reqs/sec), and we noticed that DefaultHttpParams is synchronized all over the place. This kills concurrency (I have a thread dump showing ~1200 threads waiting there ;-)), and I don't think it is necessary - it should be possible to read settings without having to acquire locks first. That's not necessarily true. Synchronize does more than provide mutual exclusion - i.e. locking - it also ensures that fields written in one thread are correctly seen in another. This is certainly correct and good point (details of how the memory view syncing is done can be even more complicated than simple flush, conceptually it's a memory barrier). For anyone unfamiliar with the concept (mutex and memory consistency) should read Java Memory Model article: http://www.cs.umd.edu/~pugh/java/memoryModel/ Anyway, one thing I was wondering was whether syncs (or, the alternative, using volatile) could still be avoided for default values. This because it would seem like such values would be immutable? This is indeed the case. HttpParams are used in write once / ready many mode and therefore its methods do not necessarily need to be synchronized to be threading safe. As far as I know this is not the case unless the variable is final, volatile, or set up during class initialisation. Otherwise, the JVM is free to cache the written or previously read value. Whether it does so is another matter; there's nothing to stop the the JVM from flushing the variable earlier. So unsafe code may still work. However, if the writer thread and reader thread(s) both synchronise on the same object then any variables that were set before the synch calls will be seen by the other thread - the variables don't have to set as part of the synch block. This is part of the happens-before rule, if I've understood it correctly. While this is certainly true, it is very unlikely this can affect HttpParams given the fact they are initialized when HttpClient instance is created and then read from _much_ later in time (many, many CPU cycles after) when requests are executed. Oleg HttpClient 4.0 uses unsynchronized implementation of HttpParams per default Oleg -+ Tatu +- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: AW: NTLM-Proxy authentication and SSL-target
On Wed, 2008-10-29 at 09:17 +0100, Cech. Ulrich wrote: Hi Oleg, Thanks for your reply. Correct, I use the NTLM-support like it is described in the documentation (with jcifs etc.). And that worked fine for no-SSL-targets. Thanks for your help, Ulrich Ulrich, The version of Squid you are using appears broken. The proxy keeps one sending 'Proxy-Connection: close' which is wrong given the fact that NTLM requires a persistent connection in order to function. See wire log below Oleg [DEBUG] headers - CONNECT www.verisign.com:443 http://www.verisign.com:443 HTTP/1.1 [DEBUG] headers - User-Agent: Apache-HttpClient/4.0-beta1 (java 1.4) [DEBUG] headers - HTTP/1.0 407 Proxy Authentication Required [DEBUG] headers - Server: squid/2.6.STABLE6-NT [DEBUG] headers - Date: Wed, 29 Oct 2008 07:36:24 GMT [DEBUG] headers - Content-Type: text/html [DEBUG] headers - Content-Length: 1347 [DEBUG] headers - Expires: Wed, 29 Oct 2008 07:36:24 GMT [DEBUG] headers - X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0 [DEBUG] headers - Proxy-Authenticate: NTLM [DEBUG] headers - X-Cache: MISS from s-hqw2k3bd.pmbelz.de [DEBUG] headers - X-Cache-Lookup: NONE from s-hqw2k3bd.pmbelz.de:3128 [DEBUG] headers - Via: 1.0 s-hqw2k3bd.pmbelz.de:3128 (squid/2.6.STABLE6-NT) [DEBUG] headers - Proxy-Connection: close [DEBUG] headers - CONNECT www.verisign.com:443 http://www.verisign.com:443 HTTP/1.1 [DEBUG] headers - User-Agent: Apache-HttpClient/4.0-beta1 (java 1.4) [DEBUG] headers - Proxy-Authorization: NTLM TlRMTVNTUAABATIAAAYABgAgBwAHACYAAABQTUJFTFpQQy0xNjM0 [DEBUG] headers - HTTP/1.0 407 Proxy Authentication Required [DEBUG] headers - Server: squid/2.6.STABLE6-NT [DEBUG] headers - Date: Wed, 29 Oct 2008 07:36:24 GMT [DEBUG] headers - Content-Type: text/html [DEBUG] headers - Content-Length: 1347 [DEBUG] headers - Expires: Wed, 29 Oct 2008 07:36:24 GMT [DEBUG] headers - X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0 [DEBUG] headers - Proxy-Authenticate: NTLM TlRMTVNTUAACADgBAgACJxWYyBecHP84BQLODg8= [DEBUG] headers - X-Cache: MISS from s-hqw2k3bd.pmbelz.de [DEBUG] headers - X-Cache-Lookup: NONE from s-hqw2k3bd.pmbelz.de:3128 [DEBUG] headers - Via: 1.0 s-hqw2k3bd.pmbelz.de:3128 (squid/2.6.STABLE6-NT) [DEBUG] headers - Proxy-Connection: close [DEBUG] headers - CONNECT www.verisign.com:443 http://www.verisign.com:443 HTTP/1.1 [DEBUG] headers - User-Agent: Apache-HttpClient/4.0-beta1 (java 1.4) [DEBUG] headers - Proxy-Authorization: NTLM TlRMTVNTUAADGAAYAEAYABgAWAwADABwCAAIAHwOAA4AhQIAAOOOenVD3rqAqXU7pJjbK5QyfrCD58Nj97AldWYk//1QtLyNjRSYH89g7e4f95Vx0lAATQBCAEUATABaAGMAZQBjAGgAUABDAC0AMQA2ADMANAA= [DEBUG] headers - HTTP/1.0 407 Proxy Authentication Required [DEBUG] headers - Server: squid/2.6.STABLE6-NT [DEBUG] headers - Date: Wed, 29 Oct 2008 07:36:24 GMT [DEBUG] headers - Content-Type: text/html [DEBUG] headers - Content-Length: 1347 [DEBUG] headers - Expires: Wed, 29 Oct 2008 07:36:24 GMT [DEBUG] headers - X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0 [DEBUG] headers - Proxy-Authenticate: NTLM [DEBUG] headers - X-Cache: MISS from s-hqw2k3bd.pmbelz.de [DEBUG] headers - X-Cache-Lookup: NONE from s-hqw2k3bd.pmbelz.de:3128 [DEBUG] headers - Via: 1.0 s-hqw2k3bd.pmbelz.de:3128 (squid/2.6.STABLE6-NT) [DEBUG] headers - Proxy-Connection: close - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: httpclient and xpath
Lior, I've had excellent results using dom4j. Just load the xml into an object and you can query the nodes using xpath expressions. Raymond On Wed, Oct 29, 2008 at 08:05, lior grinfeld [EMAIL PROTECTED] wrote: i know it may not be the right place to ask this but maybe one of you faces this issue before. i want to validate the response that i get from my web application and i thought about xpath. did anybody tried this before? i want to parse the response into document and then do xpath.getvalue(myxpath) and validate the data. first is there a response handler that can do this? or does someone can recommend a package than can do it? -- Regards Lior Grinfeld -- - Raymond Kroeker - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: httpclient and xpath
I am unclear on whether Lior wants to parse XML or HTML. Dom4j would work well for XML. I've used TagSoup for HTML (http://home.ccil.org/~cowan/XML/tagsoup/). --joe Raymond Kroeker wrote: Lior, I've had excellent results using dom4j. Just load the xml into an object and you can query the On Wed, Oct 29, 2008 at 08:58, Furmaniak Christophe [EMAIL PROTECTED] wrote: maybe NekoHTML (http://nekohtml.sourceforge.net/) could be a solution? christophe -- Christophe Furmaniak Technical Operations - Reporting Testing - Atos Worldline [EMAIL PROTECTED] Tél: +33 3 20 60 81 80 - Fax : +33 3 20 60 76 76 Atos Worldline www.atosworldline.com is an Atos Origin company www.atosorigin.com -Message d'origine- De : lior grinfeld [mailto:[EMAIL PROTECTED] Envoyé : mercredi 29 octobre 2008 16:06 À : HttpClient User Discussion Objet : httpclient and xpath i know it may not be the right place to ask this but maybe one of you faces this issue before. i want to validate the response that i get from my web application and i thought about xpath. did anybody tried this before? i want to parse the response into document and then do xpath.getvalue(myxpath) and validate the data. first is there a response handler that can do this? or does someone can recommend a package than can do it? -- Regards Lior Grinfeld Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
