Working with my F5 guy, we had an idea, since 80/tcp and 443/tcp were already open to the VIP on the F5, we simply turned of 80 -> 443 redirect on the F5, and then configured the F5 to use the non-HTTP port in Apache HTTP Server (the two backend servers). So its HTTP the entire way through, and the same issue persisted.
browser -> F5(http) -> httpd(http port) -> tomcat(http port, nonssl) So i think i can rule out any possibility that the fact I was terminating SSL in Apache HTTP Server then proxying clear-text back as a reason. It worked fine like I said if you just go direct to the HTTP Server and slip the F5. And F5 works landing the /healthcheck.html (HTTP KeepAlive in the root, which isnt proxied), so all good there. Just cant /SelfService (configured w/ Proxy) thanks for your support here though. On Thu, Feb 23, 2017 at 9:12 AM, Aaron Gray <aaronmg...@gmail.com> wrote: > SSLProxyEngine On > Was already turned on this entire time inside the ssl.conf (I include it) > VirtualHost section. > > I am debating turning on HTTPS in Tomcat on the backend 10.x.x.x app > server, and then HTTPS the whole way through and see if that makes any > difference. I may need to request a new firewall to be opened, which is > not able to open until 3/2. Gonna see. > > > On Thu, Feb 23, 2017 at 8:22 AM, Antonio S. Cofiño <cofi...@gmail.com> > wrote: > >> >> On 23/02/17 12:43, André Warnier (tomcat) wrote: >> >>> On 22.02.2017 19:22, Aaron Gray wrote: >>> >>>> So this is interesting. >>>> >>>> So from HTTP server #1 (172.1.1.1 example) I hit: >>>> https://172.1.1.1:23270/static and I see this in the HTTP log: >>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 200 >>>> 32 >>>> I see this in the Tomcat log: >>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 200 >>>> 32 >>>> ((( Yes I know they look identical, but the logformat is the same, so it >>>> comes out looking the same, but they are two diff log files totally ))) >>>> >>>> I then hit the https://loadbalancer.domain.com/static from my Win10 >>>> laptop >>>> (10.1.1.2 example) >>>> and I see this in the HTTP server #1 log, but NOTHING in the tomcat >>>> access >>>> log. >>>> 10.1.1.2 - - [22/Feb/2017:10:20:02 -0800] "-" 408 - >>>> >>>> So 408. Timeout. Hmm... Why? We KNOW that it can connect from http -> >>>> tomcat:18080 perfectly. So is it a timeout BACK to the F5? That i dont >>>> know yet. >>>> >>> >>> It looks as if, in that case, httpd is trying to connect to tomcat, but >>> - either it /is/ connecting, and sending the request, but tomcat never >>> answers >>> - or it is connecting to something else, which isn't tomcat >>> (but which accepts the request and doesn't answer either) >>> >>> I am not *sure*, but I believe, that if httpd was trying to connect to >>> something that isn't tomcat, and is not listening on port 18080, it would >>> get a different error, and a different logfile message. So it would seem >>> that at least the TCP connection works. >>> >> >> >> Is the F5 loadbalancer configured to make reverse proxy to a SSL http? >> The directive SSLProxyEngine onthe F5 loadbalencer has to be enabled. >> http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslproxyengine >> >> If the loadbalancer it's delegating the SSL work to the server1 >> server1 https listen = 172.1.1.1:23270 >> >> then the F5 could be connecting as http regular TCP connection to a SSL >> TCP one and therefor the SSL handshake it's not happening. >> >> Antonio >> >> >> >> >> >> >>> Now there is still a puzzle there : when you connect as >>> browser --> httpd --> tomcat >>> then everything works : httpd connects to tomcat, sends a request, gets >>> a response, and returns it to the browser. >>> >>> But when you do the same as : >>> browser --> F5 --> httpd --> tomcat >>> it "does not work", and the problem seems to be at httpd --> tomat >>> Why would it make a difference in httpd --> tomcat, if it is either the >>> browser or the F5 which sends the request to httpd ? >>> the httpd --> tomcat connection should be the same, no matter what. >>> >>> Hmm, for now as puzzled as you are. >>> Maybe we have a quantum phenomenon here, where the mere fact of you >>> looking at the logfile, changes what happened beforehand. Just kidding. >>> >>> Informational note : the Apache access log line, is written when Apache >>> *has finished* processing the request (including the possible round-trip >>> through tomcat) and has sent the response to the client. (That's because, >>> for example, it can also write the time spent, and the size of the >>> response; so it has to know those before it can write the log line). >>> I presume that it is the same for tomcat. >>> Just mentioning this, so that you would maybe have a second look at the >>> tomcat access log, after a few minutes, to make sure that there really is >>> no corresponding line there. >>> It could be that tomcat actually /is/ receiving the request, and >>> processing it, but that it takes a long time, and that httpd tires of >>> waiting and already times out in the meantime. In such a case, you would >>> probably see an error some time later in the tomcat error log, when it is >>> trying to eventually return the response, and finds a closed connection >>> instead. >>> Now again, why it would do that in one case and not the other, is still >>> a mystery. >>> >>> Another note : in the "timeout" log line above, httpd is not showing the >>> request URL which triggered this timeout. >>> a) it may be showing some additional information in the Apache error log >>> b) these exists an httpd add-on module (mod_log_forensic ?) which could >>> provide more details in the log, about what happens precisely >>> >>> >>> >>>> On Tue, Feb 21, 2017 at 3:39 PM, André Warnier (tomcat) <a...@ice-sa.com> >>>> wrote: >>>> >>>> On 21.02.2017 23:28, Aaron Gray wrote: >>>>> >>>>> Antonio: The Tomcat server has no knowledge of the F5, or that it is >>>>>> being >>>>>> fronted by an Apache HTTP Server. I do SSL termination in Apache HTTP >>>>>> Server, and clear-text from HTTP to Tomcat. >>>>>> My redirect port for the normal HTTP listen in Tomcat is commented >>>>>> out. >>>>>> <Connector port="18080" protocol="HTTP/1.1" >>>>>> connectionTimeout="20000" /> >>>>>> <!-- A "Connector" using the shared thread pool--> >>>>>> <!-- >>>>>> <Connector executor="tomcatThreadPool" >>>>>> port="8080" protocol="HTTP/1.1" >>>>>> connectionTimeout="20000" >>>>>> redirectPort="8443" /> >>>>>> --> >>>>>> >>>>>> Andre: >>>>>> The URL I am using is https://loadbalancer.domain.com >>>>>> It is listening on port 80 and 443, if you hit 80, internally it >>>>>> redirects >>>>>> you to 443. No SSL cert on the F5 load balancer. It simply sends the >>>>>> traffic to one of the two HTTP servers (round-robin, also tried >>>>>> persistence, no difference). The HTTP server is listening only for >>>>>> HTTPS >>>>>> on 23270/tcp. >>>>>> When I access these DMZ webservers via the F5 load balancer (to which >>>>>> I >>>>>> dont have access to, but the network folks configure for me), it >>>>>> hangs. >>>>>> Eventually returns: server1 https listen = 172.1.1.1:23270 >>>>>> >>>>>> https://loadbalancer.domain.com:23270/SelfService >>>>>> cant load. >>>>>> >>>>>> No idea why the URL is being re-written with the ":23270". >>>>>> I added static content to the server.xml on 10.1.1.1 (Tomcat) to test: >>>>>> <Context docBase="/path/to/tomcat/static" path="/static" /> >>>>>> Then put a simple index.html in there. Accessing via the Apache Web >>>>>> Servers works fine, but if you hit it with the Load Balancer it once >>>>>> again >>>>>> adds the https://loadbalancer.domain.com:23270/static >>>>>> >>>>>> Hitting https://loadbalancer.domain.com >>>>>> I see my "Hello world!" which is all that is in index.html. This is >>>>>> the >>>>>> DocumentRoot of HTTP, and *not* proxied over at this time. >>>>>> >>>>>> >>>>> So in this case, there is no delay, and you get the Apache httpd-hosted >>>>> "index.html" containing "Hello World. Right ? >>>>> >>>>> Only >>>>> >>>>> /SelfService and /static are proxied >>>>>> /static just being my test of static content, but still served up by >>>>>> Tomcat.. >>>>>> >>>>>> It's exactly 30 seconds before the page cannot be loaded when trying >>>>>> anything proxied to Tomcat, but also accessed via the F5 load >>>>>> balancer. >>>>>> Not sure where the 30 seconds comes from; perhaps a load balancer time >>>>>> out, >>>>>> as I dont see a "30" in my httpd configurations or my tomcat >>>>>> server.xml >>>>>> >>>>>> >>>>>> You can certainly look at the Apache httpd logs, and the tomcat logs, >>>>> to >>>>> see if you get a request or not. >>>>> In Apache httpd, you can set the loglevel individually for mod_proxy >>>>> (if >>>>> you are running v 2.4), and it should show something if it gets this >>>>> request and forwards it to tomcat. >>>>> In tomcat, you can either enable an access log (which will show if it >>>>> receives this request), or you could temporarily remove/rename the >>>>> /static >>>>> webapp. This way, it should trigger an error "not found" which you >>>>> would >>>>> also see in the error log. >>>>> >>>>> There should be nothing between them to hinder it. We have many load >>>>> >>>>>> balancers and this one specifically you dont need to open any firewall >>>>>> requests for the specific networks the HTTP servers are on. I did >>>>>> have to >>>>>> get the firewall opened up to allow me to hit >>>>>> https://loadbalancer.domain.com because the VIP for " >>>>>> loadbalancer.domain.com" >>>>>> is in the DMZ, and my Desktop & VPN networks cannot hit it on 80/443 >>>>>> without opening holes. But beyond that, any connection from the F5 >>>>>> to the >>>>>> HTTP Server should be 100% open bi-directional, since same subnet. >>>>>> >>>>>> >>>>>> But something isn't working, otherwise you would not be asking. >>>>> So, >>>>> >>>>> a) hitting the tomcat webapps through httpd seems to be working fine >>>>> (browser -> httpd:23270 -> tomcat:18080 -> webapp or static) >>>>> >>>>> b) hitting a non-proxied-to-tomcat resource of httpd seems to work fine >>>>> too, even through the F5 >>>>> (browser -> F5:443 -> httpd:23270 -> html page) >>>>> >>>>> c) it is only when you do : >>>>> (browser -> F5:443 -> httpd:23270 -> tomcat:18080 -> webapp or >>>>> static) >>>>> that you see this issue >>>>> >>>>> It would really help if you looked in the logs of both httpd and >>>>> tomcat, >>>>> and checked for differences betweens cases a, b and c above. >>>>> >>>>> I believe that the F5 message with the port 23270 is a minor issue, of >>>>> information disclosure by the F5, that it should not disclose. >>>>> >>>>> But the reason why it returns this error is obviously that in that >>>>> case, >>>>> it does not get a response from his request to httpd. >>>>> The reason for this response not coming back to the F5 (in case c >>>>> only), >>>>> can be due to either httpd or tomcat. But F5 doesn't know about >>>>> tomcat. So >>>>> for the F5, it is httpd which is not responding. Thus, >>>>> - either httpd is never getting the request from the F5 (unlikely, >>>>> because >>>>> in b above it gets it and responds) >>>>> - or httpd is getting the request from the F5, but not forwarding it to >>>>> tomcat, but also not returning an immediate error response to F5 (which >>>>> seems also unlikely, because of a and b) >>>>> - or httpd is getting the request, forwarding it to tomcat, but not >>>>> getting a response from tomcat. So >>>>> - either tomcat is never getting the request from httpd (but in a, >>>>> it >>>>> gets it) >>>>> - or tomcat is getting the request from httpd, but not responding >>>>> (but >>>>> in a, it does) >>>>> - or tomcat is getting the request and responding, but the response >>>>> never gets back to httpd (but in a, it does) >>>>> >>>>> So if a and b and c are all accurate, there is something apparently >>>>> illogical happening. >>>>> This would lead to the conclusion that a and b and c cannot all be >>>>> accurate. >>>>> >>>>> The logs.. ? >>>>> >>>>> >>>>> >>>>> On Tue, Feb 21, 2017 at 2:05 PM, Antonio S. Cofino <cofi...@gmail.com> >>>>> >>>>>> wrote: >>>>>> >>>>>> Aaron, on tomcat instances change the redirectPort attributte on the >>>>>> http >>>>>> >>>>>>> conectó to the loabbalancer's port 443 >>>>>>> >>>>>>> My guess is that your webapp has restriction rule requesting SSL con >>>>>>> fidntial channel. Therefore the non-confidential to the 18080 port >>>>>>> from >>>>>>> the >>>>>>> balancer are redirected to the 23270 port, but it should be 443. >>>>>>> >>>>>>> Antonio >>>>>>> >>>>>>> >>>>>>> >>>>>>> El 21/2/2017 19:46, "Aaron Gray" <aaronmg...@gmail.com> escribió: >>>>>>> >>>>>>> I have an application server from a vendor that comes bundled with an >>>>>>> additional Apache Tomcat server. The webapp SelfService.war is >>>>>>> vendor >>>>>>> supplied too. >>>>>>> >>>>>>> Here's my problem (IP's replaced to protect the innocent): >>>>>>> >>>>>>> networks: >>>>>>> DMZ=172.x.x.x >>>>>>> INTERNAL=10.x.x.x >>>>>>> >>>>>>> server1 https listen = 172.1.1.1:23270 >>>>>>> server2 https listen = 172.1.1.2:23270 >>>>>>> F5 load balancer hostname = loadbalancer.domain.com:443 >>>>>>> backend tomcat server = 10.1.1.1:18080 >>>>>>> >>>>>>> mod_proxy configuration: >>>>>>> ProxyPass /SelfService http://10.1.1.1:18080/SelfService >>>>>>> ProxyPassReverse /SelfService http://10.1.1.1:18080/SelfService >>>>>>> >>>>>>> When I access these DMZ webservers which mod_proxy back to Apache >>>>>>> Tomcat >>>>>>> as: >>>>>>> https://172.1.1.1:23270/SelfService >>>>>>> and >>>>>>> https://172.1.1.2:23270/SelfService <https://172.1.1.1:23270/SelfS >>>>>>> ervice >>>>>>> >>>>>>>> >>>>>>>> They load properly. Perfectly, every time! >>>>>>> >>>>>>> When I access these DMZ webservers via the F5 load balancer (to >>>>>>> which I >>>>>>> dont have access to, but the network folks configure for me), it >>>>>>> hangs. >>>>>>> Eventually returns: >>>>>>> https://loadbalancer.domain.com:23270/SelfService >>>>>>> cant load. >>>>>>> >>>>>>> No idea why the URL is being re-written with the ":23270". >>>>>>> I added static content to the server.xml on 10.1.1.1 (Tomcat) to >>>>>>> test: >>>>>>> <Context docBase="/path/to/tomcat/static" path="/static" /> >>>>>>> Then put a simple index.html in there. Accessing via the Apache Web >>>>>>> Servers works fine, but if you hit it with the Load Balancer it once >>>>>>> again >>>>>>> adds the https://loadbalancer.domain.com:23270/static >>>>>>> >>>>>>> Do you have any thoughts? Thanks so much, I have been working with >>>>>>> this >>>>>>> for weeks now with no success >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>>> >>>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >
