Hi Aleks,
Yes, I should have done last in my last email post. Sorry about that.
haproxy version:
[root@crsplabnet2 haproxy]# haproxy -vv
HA-Proxy version 1.8.14-52e4d43 2018/09/20
Copyright 2000-2018 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -fno-strict-overflow -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
-----------------------------
Shibboleth version ( running in both of my backend nodes, as HAproxy can't
interact with shib , to do authentication offload in HAProxy )
[root@web1 ~]# rpm -qa |grep hibb
shibboleth-3.0.2-1.1.x86_64
-------------------------------
Apache version ( running in both backend nodes ONLY ) :
[root@web1 ~]# rpm -qa |grep httpd
httpd-devel-2.4.6-80.el7.centos.x86_64
httpd-tools-2.4.6-80.el7.centos.x86_64
httpd-2.4.6-80.el7.centos.x86_64
-----------------------------------
Linux version in all systems:
Centos 7.5 , running stock kernel 3.10.0-862.el7.x86_64
here is a workflow and some background info:
background info:
A. the application that we are trying to use: jupyterhub --
https://github.com/jupyterhub
B. We have shibboleth IDP that all our internal sites do authentication
from.
C. This application Jupyterhub spawns its own ( internally built ) proxy
instance and binds that to port 8000, as you saw in the <Location> config
section in apache config above.
D. Jupyterhub can directly authenticate with shibboleth ( some additional
work is needed ), but since we are putting this behind HAProxy, running
Shibboleth SP in the backend node ( where the jupyterhub instance runs ) is
needed. therefore, I decided to wrap jupytrehub within apache as apache
has built-in support for shibboleth ( mod_shib , that gets installed with
shibboleth RPM ).
Here is what I was thinking expected workflow would be:
Client [ public IP ] connects ---> HAproxy [ public IP ] connects --->
backend node [ private IP:80 ]
Note: Shibboleth is running in the backend node with its public IP: 443, as
it loves to work with SSL and I needed an in-common SSL cert installed to
get the SP working correctly.
So, for example, in the browser, I go to:
-- proxy.example.com ( HAProxy system )
-- from the backend, shibboleth responds to me with a login screen.
-- I authenticate with username/password
-- Shibboleth hands my username as an attribute called "UserID". to
Apache, and I forward that "UserID" to jupyterhub. Apache pulls the
jupyterhub instance, presents to the user.
Here is what I expect:
Type:
URL in the browser: 'proxy.exmaple.com/jhub'
After authentication, the browse URL shown to connect directly to the
backend node, such as:
https://backendnode1.example.com/jhub/itoufiqu/tree?
whereas, I would expect it to keep the proxy URL to be:
https://proxy.exmaple.com/jhub/itoufiqu/tree?
Interesting thing is, this redirect to the backend machine, connects the
client directly to the backend machine over the public IP of the backend
machine. I think this is due to SSO being done by shibboleth. Since
Shibboleth fails authentication from within the internal LAN ( as the IDP
could not talk to the SP in the private network), I had to get a public
IP. In this process, HAProxy loses all accounting data to this session,
except the first connection attempt.
This setup could be good or bad, I am not sure. My initial thoughts were,
all traffic passes through HAProxy. In retrospect, HAProxy is not being
taxed at all, there is a direct connection between the client and the
backend node over a public IP ( of the backend node). But, what I don't
understand is, if a connection attempt is made through HAProxy, why would
it allow the connection to be handed off to the backend node directly (
client -> backend node )?
Thoughts?
On Sat, Oct 27, 2018 at 4:31 AM Aleksandar Lazic <al-hapr...@none.at> wrote:
> Hi Imam.
>
> It would be helpfull to know your used versions:
>
> haproxy -vv
> apache httpd version
> shibboleth version
>
> A small workflow picture like:
>
> Client -> haproxy -> apache httpd -> shibboleth ?
>
> Am 27.10.2018 um 07:44 schrieb Imam Toufique:
> > Hi Igor,
> >
> > Thanks very much for offering to help! I will do this in sections,
> hopefully, I
> > can keep this from being too cluttered.
> >
> > haproxy.cfg:
> >
> --------------------------------------------------------------------------------------
> > global
> > #log /dev/log local0 debug
> > #log /dev/log local1 debug
> > log 127.0.0.1 local2
> > chroot /var/lib/haproxy
> > stats timeout 30s
> > user haproxy
> > group haproxy
> > tune.ssl.default-dh-param 2048
> > daemon
> >
> > defaults
> > log global
> > mode http
> > option tcplog
> > option dontlognull
> > timeout connect 5000
> > timeout client 50000
> > timeout server 50000
> > timeout tunnel 9h
> > option tcp-check
> >
> > frontend http_front
> > bind :80
> > bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem
> > stats uri /haproxy?stats
> > default_backend web1_cluster
> > option httplog
> > log global
> > #option dontlognull
> > log /dev/log local0 debug
>
> The 2 log entries are redundant, imho.
>
> > mode http
>
> Is set at default block. Please take a look at
>
> https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/
>
> > option forwardfor # forward IP
> > http-request set-header X-Forwarded-Port %[dst_port]
> > http-request add-header X-Forwarded-Proto https if { ssl_fc }
>
> I personally would use here also set-header instead of add.
>
> > redirect scheme https if !{ ssl_fc }
> >
> > acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav
>
> This should be only the host name.
>
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.6-hdr
> and
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.6-req.hdr
>
> I would add a addition acl.
>
> acl host_web2 hdr(host) -i crsplab2.oit.uci.edu
> acl path_web2 path_beg -i /webdav
>
> > use_backend webdav_cluster if host_web2
>
> an add it to the use_backend line
>
> use_backend webdav_cluster if host_web2 path_web2
>
> > acl host_web3 path_beg /jhub
> > use_backend web3_cluster if host_web3
> >
> >
> > backend webdav_cluster
> > balance roundrobin
>
> I would add here the following line.
>
> cookie SRV1 insert indirect nocache
> > server web1 10.1.100.156:8080 check inter 2000 cookie w1
> > server web2 10.1.100.160:8080 check inter 2000 cookie w2
> >
> > backend web3_cluster
>
> I would add here the following line.
>
> cookie SRV2 insert indirect nocache
> > server publicIP:443 check ssl verify none inter 2000 cookie w1
> >
> -----------------------------------------------------------------------------------------------------
> > Note: I have a single backend node, as it was easy to test with just one
> node,
> > instead of making changes to 2 nodes at a time.
> >
> > Here is my apache config:
> >
> > in httpd.conf, only change I have made is ( the rest is a stock centos
> 7.5
> > httpd.conf ):
> > -------------------------------------
> > ServerName 10.1.100.160:80 ( Internal IP of the backend node)
> > Redirect permanent /jhub https://crsplabweb1.domain.com/jhub
> > -------------------------------------
> >
> > in my ssl.conf, where I access the jupyterhub instance running in
> 127.0.0.1:8000
> > <http://127.0.0.1:8000> . Also, note that the backend is running
> shibboleth
> > SP. One of the issues I encountered is, If I did not have SSL , i was
> getting a
> > browser warning for not having SSL.
> Can you set up shibboleth in that manner that he answers with
> proxy.domain.com?
> As we don't know which version is in place I send you just as a hint here
> some
> links.
>
> https://wiki.shibboleth.net/confluence/display/SHIB2/SPReverseProxy
>
> http://shibboleth.1660669.n2.nabble.com/shibboleth-sp-behind-an-HAproxy-td5960149.html
>
> https://duckduckgo.com/?q=shibboleth+behind+haproxy
>
>
> > Here is my ssl.conf:
> >
> >
> --------------------------------------------------------------------------
> > Listen 443 https
> > SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
> > SSLSessionCache shmcb:/run/httpd/sslcache(512000)
> > SSLSessionCacheTimeout 300
> > SSLRandomSeed startup file:/dev/urandom 256
> > SSLRandomSeed connect builtin
> > SSLCryptoDevice builtin
> >
> > <VirtualHost _default_:443>
> >
> > UseCanonicalName on
> > ServerName crsplabweb1.domain.com:443
>
> Maybe you can change this to proxy.domain.com and add a ServerAlias
>
> ServerName proxy.domain.com
> ServerAlias crsplabweb1 crsplabweb1.domain.com
>
> https://httpd.apache.org/docs/2.4/mod/core.html#serveralias
>
> > ErrorLog logs/ssl_error_log
> > TransferLog logs/ssl_access_log
> > LogLevel warn
> >
> > SSLEngine on
> >
> > SSLProtocol all -SSLv2 -SSLv3
> > SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA
> > SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer
> > SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key
> > SSLCertificateChainFile
> /etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c
> >
> > <Files ~ "\.(cgi|shtml|phtml|php3?)$">
> > SSLOptions +StdEnvVars
> > </Files>
> > <Directory "/var/www/cgi-bin">
> > SSLOptions +StdEnvVars
> > </Directory>
> >
> > <Location /jhub>
> > ProxyPass http://127.0.0.1:8000/jhub
> > ProxyPassReverse http://127.0.0.1:8000/jhub
> > RequestHeader unset Accept-Encoding
> > ProxyPreserveHost on
> > AuthType shibboleth
> > ShibRequestSetting requireSession 1
> > Require shibboleth
> > ShibUseHeaders On
> > ShibBasicHijack On
> > RewriteEngine On
> > RequestHeader set X-Remote-User %{REMOTE_USER}s
> > </Location>
> >
> > <LocationMatch
> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)">
> > ProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3
> > ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3
> > </LocationMatch>
> >
> > BrowserMatch "MSIE [2-5]" \
> > nokeepalive ssl-unclean-shutdown \
> > downgrade-1.0 force-response-1.0
> >
> > CustomLog logs/ssl_request_log \
> > "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
> > </VirtualHost>
> >
> ----------------------------------------------------------------------------------
> >
> > Thanks
>
> Regards
> Aleks
>
> > On Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov <
> ig...@encompasscorporation.com
> > <mailto:ig...@encompasscorporation.com>> wrote:
> >
> > Hi Imam,
> >
> > On Sat, Oct 27, 2018 at 9:37 AM Imam Toufique <techie...@gmail.com
> > <mailto:techie...@gmail.com>> wrote:
> >
> > Hi,
> >
> > I came up with the following config, things seem to be working
> now, for
> > the most part.
> >
> > frontend http_front
> > bind :80
> > bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem
> > stats uri /haproxy?stats
> > default_backend web1_cluster
> > option httplog
> > log global
> > #option dontlognull
> > log /dev/log local0 debug
> > mode http
> > option forwardfor # forward IP
> > http-request set-header X-Forwarded-Port %[dst_port]
> > http-request add-header X-Forwarded-Proto https if { ssl_fc }
> > redirect scheme https if !{ ssl_fc }
> > acl host_web3 path_beg /jhub
> > use_backend web3_cluster if host_web3
> >
> > web3_cluster
> >
> > backend web3_cluster
> > mode http
> > balance source
> > server crsplabweb1.domain.com publicIP:443 check ssl verify
> none inter 2000 cookie w1
> >
> > The above config gets me to the backend node -- where I have a
> > jupyterhub instance running + . Shibboleth SP running for
> > authentication. As I could not get shibboleth SP to work by
> staying in
> > my private network, I had to set up a public IP for the backend
> node,
> > get SSL certs - so shibboleth authentication could be done. I
> am sure
> > there is a better approach to this, but I don't know what it
> is. I will
> > be trying out SNAT to see if that will allow me to keep using my
> private
> > IP for the backend nodes. If any of you know how to do SNAT,
> please
> > chime in, it would be worth the time/effort to try it out.
> >
> > Now, the interesting thing I have noticed with the above setup
> -- when I
> > connect to HAProxy, let's say with https://proxy.domain.com , I
> > authenticate with shibboleth, and then the URL in the browser
> points to
> > the backend node.
> >
> > For example:
> >
> > my proxy address: https://proxy.domain.com/jhub
> >
> > after I connect to the backend, the URL turns into -
> > https://crsplabweb1.domain.com/jhub/tree?
> >
> > ...and everything works thereafter.
> >
> > I tried the rewrite method that Igor has suggested before, that
> did not
> > make any difference. But what I noticed is, after I connect, no
> traffic
> > go through the proxy anymore, my client ( i.e. laptop) connects
> directly
> > to the backend server. Not sure if this good or bad though (?) ,
> but, I
> > am not sure how to configure this so that I will go through a
> proxy but
> > still be connected in the backend via a private IP and I can (
> still )
> > authenticate via shibboleth.
> >
> > So, when I change the 'web3_cluster' backend to :
> >
> > server crsplabweb1 privateIP:80 inter 2000 cookie w1
> >
> > and, I set backend apache to accept connection on port 80, then
> I break
> > shibboleth authentication.
> >
> > Any inputs here?
> >
> > thanks, guys!
> >
> >
> > I think it is time for you to provide the full HAP and Apache
> configs so we
> > can see what is going on (please obfuscate any sensitive data). Also
> the use
> > of the "cookie w1" is not clear since you are not setting it in HAP
> and is
> > kinda redundant for single backend setup.
> >
> >
> >
> > On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov
> > <ig...@encompasscorporation.com <mailto:
> ig...@encompasscorporation.com>>
> > wrote:
> >
> >
> >
> > On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov
> > <ig...@encompasscorporation.com
> > <mailto:ig...@encompasscorporation.com>> wrote:
> >
> >
> >
> > On Thu, 25 Oct 2018 6:13 pm Imam Toufique <
> techie...@gmail.com
> > <mailto:techie...@gmail.com>> wrote:
> >
> > so I almost got this to work, based on the situation
> I am
> > in. To elaborate just a bit, my setup involves a
> shibboleth
> > SP that I need to authenticate my application.
> Since I
> > can't set up the HA proxy node with shibboleth SP -
> I had to
> > wrap my application in the backend with apache so I
> can pass
> > REMOTE_USER to the application. the application I
> have is -
> > jupyterhub and it start with its own proxy. Long
> story
> > short, here is my current setup:
> >
> > frontend
> > bind :80
> > bind :443 ssl crt /etc/haproxy/crsplab2_1.pem
> > stats uri /haproxy?stats
> > default_backend web1_cluster
> > option httplog
> > log global
> > #option dontlognull
> > log /dev/log local0 debug
> > mode http
> > option forwardfor # forward IP
> > http-request set-header X-Forwarded-Port
> %[dst_port]
> > http-request add-header X-Forwarded-Proto https
> if { ssl_fc }
> > redirect scheme https if !{ ssl_fc }
> >
> > acl host_web3 path_beg /jhub
> > use_backend web3_cluster if host_web3
> >
> > backend
> > server web1.oit.uci.edu <http://web1.oit.uci.edu>
> > 128.110.80.5:80 <http://128.110.80.5:80> check
> >
> > this works for the most part. But I am confused
> with a
> > problem. when I get to my application, my backend IP
> address
> > shows up in the browser URL.
> >
> > for example, I see this in my browser:
> >
> > http://128.110.80.5/jhub/user/itoufiqu/tree?
> >
> > whereas, I was expecting that it would show the
> original
> > URL, such as:
> >
> > http://crsplab2.domain.com/jhub/user/itoufiqu/tree
> > <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>?
> (
> > where crsplab2.domain.com <
> http://crsplab2.domain.com> is
> > the URL to get HAproxy )
> >
> >
> > You need to tell your backend app that it runs behind
> reverse
> > proxy with ssl termination and that it's domain/url
> > is https://crsplab2.domain.com
> > <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>.
> How you do
> > that depends on the backend app you are using but most
> of them
> > like apache2, tomcat etc. have specific configs that you
> can
> > find in their documentation. For example if your backend
> is
> > apache2 I bet you don't have the DomainName set in the
> config in
> > which case it defaults to the host ip address.
> >
> >
> > You can also try:
> >
> > rspirep ^Location:\ http://(.*):80(.*) Location:\
> > https://crsplab2.domain.com
> > <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2
> if {
> > ssl_fc }
> >
> > to fix the URL but note that this will not save you from
> hard coded
> > url's in the returned html pages the way apache does.
> >
> >
> >
> > While I am no expert in HA proxy world, I think this
> might
> > due to the fact that my backend does not have SSL and
> > HAproxy frontend does have SSL. At this point, I
> would
> > avoid that IP address showing up in the browser.
> what is
> > the best way to accomplish this?
> >
> > thanks for your continues help!
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic
> > <al-hapr...@none.at <mailto:al-hapr...@none.at>>
> wrote:
> >
> > Hi.
> >
> > Am 23.10.2018 um 09:04 schrieb Imam Toufique:
> > > I am looking for some help on how to write the
> > following apache proxypass rules
> > > in HAproxy. Not to mention I am at a bit of
> loss with
> > my first try :-) . Here
> > > are my current proxypass rules:
> > >
> > > ProxyPass http://10.1.100.156:8000/jhub
> > > ProxyPassReverse http://10.1.100.156:8000/jhub
> >
> > Well ProxyPass and ProxyPassReverse do a lot of
> thinks
> > not just rewrites, as
> > mentioned in the doc
> >
> >
> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass
> >
> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse
> >
> >
> > > <LocationMatch
> >
> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)">
> > > ProxyPassMatch ws://
> 10.1.100.156:8000/jhub/$1/$2$3
> > <http://10.1.100.156:8000/jhub/$1/$2$3>
> > > ProxyPassReverse
> > ws://10.1.100.156:8000/jhub/$1/$2$3
> > <http://10.1.100.156:8000/jhub/$1/$2$3>
> > > </LocationMatch>
> > >
> > > As I am not well versed in the massive HAproxy
> > configuration guide, if any of
> > > you can give me a hand with this, I would
> very much
> > appreciate it.
> >
> > I'm also not "that" expert but I would try the
> > following, untested.
> >
> > ###
> > defaults
> > mode http
> > log global
> >
> > #... maybe some other settings
> > timeout tunnel 10h
> >
> > frontend https_001
> >
> > #... maybe some other settings
> >
> > acl websocket path_beg /jhub
> >
> > #... maybe some other acls
> >
> > use_backend websocket_001 if websocket
> >
> > backend websocket_001
> >
> > reqrep "^([^\ :]*)
> >
> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"
> > "/jhub/\1/\2\3"
> >
> > # You will need to replace the first column
> with the
> > response from the
> > # backend response
> > # rspirep "^Location:
> >
> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"
> > "Location:
> > /jhub/\1/\2\3"
> > # OR
> > # http-response replace-header Location
> >
> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"
> > "/jhub/\1/\2\3"
> >
> > # add some checks
> >
> > server ws_01 10.1.100.156:8000
> > <http://10.1.100.156:8000> check
> > ###
> >
> > Here are some links which may help you also.
> >
> >
> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/
> >
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep
> >
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep
> >
> > I would run haproxy in Debug mode and see how the
> > request pass haproxy and adopt
> > the config.
> >
> > It would be nice when you show us the working
> conf ;-)
> >
> > It would be nice to have a
> >
> > http-request replace-uri <match-regex>
> <replace-fmt>
> >
> > to replace the reqrep.
> >
>
--
Regards,
*Imam Toufique*
*213-700-5485*
