On 23/02/17 12:43, André Warnier (tomcat) wrote:
On 22.02.2017 19:22, Aaron Gray wrote:
So this is interesting.

So from HTTP server #1 (172.1.1.1 example) I hit:
https://172.1.1.1:23270/static and I see this in the HTTP log:
172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 200 32
I see this in the Tomcat log:
172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 200 32
((( Yes I know they look identical, but the logformat is the same, so it
comes out looking the same, but they are  two diff log files totally )))

I then hit the https://loadbalancer.domain.com/static from my Win10 laptop
  (10.1.1.2 example)
and I see this in the HTTP server #1 log, but NOTHING in the tomcat access
log.
10.1.1.2 - - [22/Feb/2017:10:20:02 -0800] "-" 408 -

So 408.  Timeout.  Hmm... Why?  We KNOW that it can connect from http ->
tomcat:18080 perfectly.   So is it a timeout BACK to the F5? That i dont
know yet.

It looks as if, in that case, httpd is trying to connect to tomcat, but
- either it /is/ connecting, and sending the request, but tomcat never answers
- or it is connecting to something else, which isn't tomcat
(but which accepts the request and doesn't answer either)

I am not *sure*, but I believe, that if httpd was trying to connect to something that isn't tomcat, and is not listening on port 18080, it would get a different error, and a different logfile message. So it would seem that at least the TCP connection works.


Is the F5 loadbalancer configured to make reverse proxy to a SSL http?
The directive SSLProxyEngine onthe F5 loadbalencer has to be enabled.
http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslproxyengine

If the loadbalancer it's delegating the SSL work to the server1
server1 https listen = 172.1.1.1:23270

then the F5 could be connecting as http regular TCP connection to a SSL TCP one and therefor the SSL handshake it's not happening.

Antonio






Now there is still a puzzle there : when you connect as
browser --> httpd --> tomcat
then everything works : httpd connects to tomcat, sends a request, gets a response, and returns it to the browser.

But when you do the same as :
browser --> F5 --> httpd --> tomcat
it "does not work", and the problem seems to be at httpd --> tomat
Why would it make a difference in httpd --> tomcat, if it is either the browser or the F5 which sends the request to httpd ?
the httpd --> tomcat connection should be the same, no matter what.

Hmm, for now as puzzled as you are.
Maybe we have a quantum phenomenon here, where the mere fact of you looking at the logfile, changes what happened beforehand. Just kidding.

Informational note : the Apache access log line, is written when Apache *has finished* processing the request (including the possible round-trip through tomcat) and has sent the response to the client. (That's because, for example, it can also write the time spent, and the size of the response; so it has to know those before it can write the log line).
I presume that it is the same for tomcat.
Just mentioning this, so that you would maybe have a second look at the tomcat access log, after a few minutes, to make sure that there really is no corresponding line there. It could be that tomcat actually /is/ receiving the request, and processing it, but that it takes a long time, and that httpd tires of waiting and already times out in the meantime. In such a case, you would probably see an error some time later in the tomcat error log, when it is trying to eventually return the response, and finds a closed connection instead. Now again, why it would do that in one case and not the other, is still a mystery.

Another note : in the "timeout" log line above, httpd is not showing the request URL which triggered this timeout.
a) it may be showing some additional information in the Apache error log
b) these exists an httpd add-on module (mod_log_forensic ?) which could provide more details in the log, about what happens precisely



On Tue, Feb 21, 2017 at 3:39 PM, André Warnier (tomcat) <a...@ice-sa.com>
wrote:

On 21.02.2017 23:28, Aaron Gray wrote:

Antonio:  The Tomcat server has no knowledge of the F5, or that it is
being
fronted by an Apache HTTP Server.  I do SSL termination in Apache HTTP
Server, and clear-text from HTTP to Tomcat.
My redirect port for the normal HTTP listen in Tomcat is commented out.
      <Connector port="18080" protocol="HTTP/1.1"
                 connectionTimeout="20000" />
      <!-- A "Connector" using the shared thread pool-->
      <!--
      <Connector executor="tomcatThreadPool"
                 port="8080" protocol="HTTP/1.1"
                 connectionTimeout="20000"
                 redirectPort="8443" />
      -->

Andre:
The URL I am using is https://loadbalancer.domain.com
It is listening on port 80 and 443, if you hit 80, internally it redirects
you to 443.  No SSL cert on the F5 load balancer.  It simply sends the
traffic to one of the two HTTP servers (round-robin, also tried
persistence, no difference). The HTTP server is listening only for HTTPS
on 23270/tcp.
When I access these DMZ webservers via the F5 load balancer (to which I dont have access to, but the network folks configure for me), it hangs.
Eventually returns: server1 https listen = 172.1.1.1:23270
https://loadbalancer.domain.com:23270/SelfService
cant load.

No idea why the URL is being re-written with the ":23270".
I added static content to the server.xml on 10.1.1.1 (Tomcat) to test:
<Context docBase="/path/to/tomcat/static" path="/static" />
Then put a simple index.html in there.  Accessing via the Apache Web
Servers works fine, but if you hit it with the Load Balancer it once
again
adds the https://loadbalancer.domain.com:23270/static

Hitting https://loadbalancer.domain.com
I see my "Hello world!" which is all that is in index.html. This is the
DocumentRoot of HTTP, and *not* proxied over at this time.


So in this case, there is no delay, and you get the Apache httpd-hosted
"index.html" containing "Hello World. Right ?

   Only

/SelfService and /static are proxied
/static just being my test of static content, but still served up by
Tomcat..

It's exactly 30 seconds before the page cannot be loaded when trying
anything proxied to Tomcat, but also accessed via the F5 load balancer.
Not sure where the 30 seconds comes from; perhaps a load balancer time
out,
as I dont see a "30" in my httpd configurations or my tomcat server.xml


You can certainly look at the Apache httpd logs, and the tomcat logs, to
see if you get a request or not.
In Apache httpd, you can set the loglevel individually for mod_proxy (if
you are running v 2.4), and it should show something if it gets this
request and forwards it to tomcat.
In tomcat, you can either enable an access log (which will show if it
receives this request), or you could temporarily remove/rename the /static webapp. This way, it should trigger an error "not found" which you would
also see in the error log.

There should be nothing between them to hinder it.  We have many load
balancers and this one specifically you dont need to open any firewall
requests for the specific networks the HTTP servers are on. I did have to
get the firewall opened up to allow me to hit
https://loadbalancer.domain.com because the VIP for "
loadbalancer.domain.com"
is in the DMZ, and my Desktop & VPN networks cannot hit it on 80/443
without opening holes. But beyond that, any connection from the F5 to the
HTTP Server should be 100% open bi-directional, since same subnet.


But something isn't working, otherwise you would not be asking.
So,

a) hitting the tomcat webapps through httpd seems to be working fine
   (browser -> httpd:23270 -> tomcat:18080 -> webapp or static)

b) hitting a non-proxied-to-tomcat resource of httpd seems to work fine
too, even through the F5
   (browser -> F5:443 -> httpd:23270 -> html page)

c) it is only when you do :
(browser -> F5:443 -> httpd:23270 -> tomcat:18080 -> webapp or static)
   that you see this issue

It would really help if you looked in the logs of both httpd and tomcat,
and checked for differences betweens cases a, b and c above.

I believe that the F5 message with the port 23270 is a minor issue, of
information disclosure by the F5, that it should not disclose.

But the reason why it returns this error is obviously that in that case,
it does not get a response from his request to httpd.
The reason for this response not coming back to the F5 (in case c only), can be due to either httpd or tomcat. But F5 doesn't know about tomcat. So
for the F5, it is httpd which is not responding. Thus,
- either httpd is never getting the request from the F5 (unlikely, because
in b above it gets it and responds)
- or httpd is getting the request from the F5, but not forwarding it to
tomcat, but also not returning an immediate error response to F5 (which
seems also unlikely, because of a and b)
- or httpd is getting the request, forwarding it to tomcat, but not
getting a response from tomcat. So
- either tomcat is never getting the request from httpd (but in a, it
gets it)
- or tomcat is getting the request from httpd, but not responding (but
in a, it does)
   - or tomcat is getting the request and responding, but the response
never gets back to httpd (but in a, it does)

So if a and b and c are all accurate, there is something apparently
illogical happening.
This would lead to the conclusion that a and b and c cannot all be
accurate.

The logs.. ?



On Tue, Feb 21, 2017 at 2:05 PM, Antonio S. Cofino <cofi...@gmail.com>
wrote:

Aaron, on tomcat instances change the redirectPort attributte on the http
conectó to the loabbalancer's port 443

My guess is that your webapp has restriction rule requesting SSL con
fidntial channel. Therefore the non-confidential to the 18080 port from
the
balancer are redirected to the 23270 port, but it should be 443.

Antonio



El 21/2/2017 19:46, "Aaron Gray" <aaronmg...@gmail.com> escribió:

I have an application server from a vendor that comes bundled with an
additional Apache Tomcat server. The webapp SelfService.war is vendor
supplied too.

Here's my problem (IP's replaced to protect the innocent):

networks:
DMZ=172.x.x.x
INTERNAL=10.x.x.x

server1 https listen = 172.1.1.1:23270
server2 https listen = 172.1.1.2:23270
F5 load balancer hostname = loadbalancer.domain.com:443
backend tomcat server = 10.1.1.1:18080

mod_proxy configuration:
ProxyPass /SelfService http://10.1.1.1:18080/SelfService
ProxyPassReverse /SelfService http://10.1.1.1:18080/SelfService

When I access these DMZ webservers which mod_proxy back to Apache Tomcat
as:
https://172.1.1.1:23270/SelfService
and
https://172.1.1.2:23270/SelfService <https://172.1.1.1:23270/SelfService

They load properly. Perfectly, every time!

When I access these DMZ webservers via the F5 load balancer (to which I dont have access to, but the network folks configure for me), it hangs.
Eventually returns:
https://loadbalancer.domain.com:23270/SelfService
cant load.

No idea why the URL is being re-written with the ":23270".
I added static content to the server.xml on 10.1.1.1 (Tomcat) to test:
<Context docBase="/path/to/tomcat/static" path="/static" />
Then put a simple index.html in there.  Accessing via the Apache Web
Servers works fine, but if you hit it with the Load Balancer it once
again
adds the https://loadbalancer.domain.com:23270/static

Do you have any thoughts? Thanks so much, I have been working with this
for weeks now with no success




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org





---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Reply via email to