Re: lua haproxy-auth-request - round 2
Hi Tim, On 2018-10-09 10:40 AM, Tim Düsterhus wrote: Bob, Am 05.10.2018 um 22:10 schrieb Computerisms Corporation: Okay, this is something I hadn't looked at. Not 100% sure I am interpreting correctly, but assuming I am, then the browser is reporting that the connection is successfully upgrading to websockets both with and without the Lua script enabled. As far as I can tell, the only thing that changes is the cookies and keys. There is a significant difference in the "waiting" response, though; ~350ms with the lua script, but less than 5ms without it. Use your browser's dev tools (F12 for Firefox, Chrome). Search for the websocket request. It's just called '/websocket' for Etherpad. It should return a 101 Switching Protocols. In Google Chrome you can even look at the WebSocket Frames. okay, that is what I was looking at, and yes, it does show the websocket upgrade request, see below. it looks like the websockets upgrade is /socket.io followed by some arguments denoted with a /?. Can't paste, but it starts with a call to this file with code 200; getting en-gb.json with code 304; another call to /socket.io with code 200; then the code 101 with an upgrade; 10 seconds later it shows a POST to the /socket.io with code 200. b) Send credentials for basic authentication for Websockets. hm, so I don't seem to be able to connect to etherpad directly using a ws:// schematic in chrome or firefox, and I think that is what you mean. not sure if that is just me, though, will work some more on that and see if I can figure out if I am doing something wrong there... No, use the dev tools and look at the request headers, whether there is an Authorization header. Of course oauth_proxy needs to be enabled. I think by oauth_proxy you mean the auth method I am using with the lua script, in my case authnz_external from apache. if you literally mean oauth_proxy, maybe this is the link I am missing. I have not configured oauth_proxy any where, as I am/(was?) not planning to use it. But by the request headers, I understand what you mean now. I am using firefox primarily, but appears I get the same basic thing in chrome. Firefox and Chrome both report 101 switching protocols, the websocket upgrade header, and the authorization header: Response headers (175 B) Connection Upgrade Sec-WebSocket-AcceptgU996yXfDiutgquFxiRxGuffglc= Sec-WebSocket-Extensionspermessage-deflate Upgrade websocket Request headers (936 B) Accept text/html,application/xhtml+xm…plication/xml;q=0.9,*/*;q=0.8 Accept-Encoding gzip, deflate, br Accept-Language en-CA,en-US;q=0.7,en;q=0.3 Authorization Basic Ym9iQGNvbXB1dGVyaXNtcy5j…HMHJnZTB1cyhXZWUpRWdnbGl0cw== Cache-Control no-cache Connection keep-alive, Upgrade Cookie _ga=GA1.2.1611432529.153149834…fFXUaWksCAAAJ; language=en-gb Hostpad.computerisms.ca Origin https://pad.computerisms.ca Pragma no-cache Sec-WebSocket-Extensionspermessage-deflate Sec-WebSocket-Key DhxDcq4PkH+/TF2kaSW8JQ== Sec-WebSocket-Version 13 Upgrade websocket User-Agent Mozilla/5.0 (X11; Ubuntu; Linu…) Gecko/20100101 Firefox/62.0 I attached a screenshot of Chrome's dev tools. Thanks, that confirms I am using the correct thing. If you send me credentials in private I can take a look myself, if you want. will follow up off list... Best regards Tim Düsterhus
Re: lua haproxy-auth-request - round 2
Hi Tim,
Thanks for the response, and apologies for the delay. Popularity is
advertised as a good thing, but I have my doubts. Regardless, I am back
on track with this project for the moment.
While I don't have any idea from the top of my head your configuration
surely would be helpful.
right, should have included that the first time, will put it at the
bottom of this mail.
You might also want to check whether the webbrowser is able to:
a) Set-Up the Websocket with auth-request in between (you should see a
101 Switching Protocols in it's network console).
Okay, this is something I hadn't looked at. Not 100% sure I am
interpreting correctly, but assuming I am, then the browser is reporting
that the connection is successfully upgrading to websockets both with
and without the Lua script enabled. As far as I can tell, the only
thing that changes is the cookies and keys. There is a significant
difference in the "waiting" response, though; ~350ms with the lua
script, but less than 5ms without it.
b) Send credentials for basic authentication for Websockets.
hm, so I don't seem to be able to connect to etherpad directly using a
ws:// schematic in chrome or firefox, and I think that is what you mean.
not sure if that is just me, though, will work some more on that and
see if I can figure out if I am doing something wrong there...
My haproxy.cfg:
Note in the front end I have two lines commented; with these lines
commented, everything works, can reload hundreds of times with no error.
With the lines uncommented, the auth works, the main landing page
works, but accessing the actual pad does not work. the tables entries
seem to work fine either way.
I have uncommented these lines to induce failure, and pasted a copy of
the haproxy logs of the event at
http://www.computerisms.ca/haproxy.txt
At the bottom of the log file, it appears to me that I get a 200 after
the websocket upgrade, which I interpret to mean it was successful, but
at that point the page spins and some 150-300 seconds later I get the
error page displayed on the screen with no more log entries in haproxy.
global
debug
log /dev/loglocal1 debug
chroot /var/lib/haproxy
userhaproxy
group haproxy
daemon
ca-base /etc/ssl/certs
crt-base/etc/ssl/private
ssl-default-bind-optionsno-sslv3
lua-load/Computerisms/config/etc/haproxy.auth.lua
defaults
log global
modehttp
option httplog
option dontlognull
frontendhttpfront
bind${ADDRESS}:80 v4v6
bind ${ADDRESS}:443 v4v6 ssl crt /Computerisms/config/certificates/
redirectscheme https code 301 if !{ ssl_fc }
modehttp
option httplog
log global
# http-requestlua.auth-requestauth_request /index.html
## ACLs
acl tables.computerisms.ca ssl_fc_sni -i tables.computerisms.ca
acl pad.computerisms.ca ssl_fc_sni -i pad.computerisms.ca
## AUTHREQ
use_backend auth_request if ! {
var(txn.auth_response_successful) -m bool } tables.computerisms.ca
# use_backend auth_request if ! {
var(txn.auth_response_successful) -m bool } pad.computerisms.ca
## AUTHBACKEND
use_backend tables.computerisms.ca if tables.computerisms.ca
use_backend pad.computerisms.ca if pad.computerisms.ca
default_backend mooglehttps
backend auth_request
modehttp
server auth-request127.0.0.1:8044 check
# option httpclose
# option forwardfor
backend mooglehttps
balance leastconn
modehttp
option httpclose
option forwardfor
option log-health-checks
option httpchk
server sand1lian 192.168.25.52:48443 check send-proxy-v2 ssl
verify none
server sand2lian 192.168.25.53:48443 check send-proxy-v2 ssl
verify none
## BEGIN pad.computerisms.ca
backend pad.computerisms.ca
balance leastconn
modehttp
cookie sessionID insert nocache indirect
#option httpclose
option forwardfor
server sand1lian 192.168.25.52:19008 cookie sand1pad
server sand2lian 192.168.25.53:19008 cookie sand2pad
## END pad.computerisms.ca
## BEGIN tables.computerisms.ca
backend tables.computerisms.ca
balance leastconn
modehttp
cookie sessionID insert nocache indirect
option httpclose
option forwardfor
server sand1lian 192.168.25.52:29000 check cookie sand1tables
server sand2lian 192.168.25.53:29000 check
lua haproxy-auth-request - round 2
Hi Tim and other Gurus, I am using the auth-request lua script from here (thanks Tim, you rock for publishing this): https://github.com/TimWolla/haproxy-auth-request Haproxy is sitting in front of two nodejs apps; etherpad and ethercalc. it is also sitting in front of apache which is serving standard websites such as static html, Joomla, Wordpress, Drupal, etc. I am using the lua auth script to employ apache's authnz external authentication program, which is authenticating against an IMAP server, and I have got my haproxy configured such that it only fires the lua script when accessing the nodejs apps. Things are mostly working, but there is one problem I am having with etherpad that can be fixed by disabling the lua script, so even though I am not 100% sure the problem is not on the etherpad side, I am starting from here. To date I am not able to replicate this issue on ethercalc, the lua auth script seems to work fine every where else. specifically; I can load the etherpad landing page reliably. to load a pad one changes the path of the url, for example one could have two pads with the following urls: http://pad.domain.tld/p/nameofpad and http://pad.domain.tld/p/otherpadname. These pads eventually time out and produce an error regarding a null variable from etherpad if the lua auth script is enabled in the haproxy config. etherpad has other modules that can be accessed at other URL paths, such as /metrics and /stats, these work fine, the error seems to be limited to loading the pads only. The error I am seeing in etherpad is described quite accurately here: https://github.com/ether/etherpad-lite/issues/3047 The short version of that is that a variable or variables are empty and should not be. According to the guy who got things working there, he "redid" his proxy setup and it solved the problem. unfortunately, etherpad doesn't have any documentation for using haproxy as of yet, so trying to puzzle out what is different. this is my one/only indication that this problem might need to be solved on the etherpad side. I figured there must be some difference with a cookie or header or something, so I stripped all SSL configs and did a tcpdump to capture the full text of the traffic. I expected and found some differences, but every thing looks pretty much the same to me. I have spent quite a few hours searching those dumps, if the answer is there it is too slippery for my eye to land on. Log files are equally unhelpful; the etherpad log shows the null variable, but I am able to find no clues in logs for haproxy, etherpad, or the apache instance that is doing the authnz auth. In my investigation, I found that sometimes apache would return a 304 or 408 from the authentication instance. I altered the lua script to return an auth_response_successful on these codes, but it didn't fix the problem. through my tcpdump, I discovered that etherpad is using websockets. I found this page: https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/ which indicates that haproxy will automatically change to tunnel mode and support websockets. I can find no reason why the lua script might interfere with this, and ethercalc also uses websockets, but so far it is the only thing I can find that *might be causing data to not be passed for the variable(s) to be populated. if you are still with me, thank you so much for reading this far. I would truly appreciate any thoughts you might have on how to diagnose what is causing this issue... -- Bob Miller Cell: 867-334-7117 Office: 867-633-3760 www.computerisms.ca
Re: haproxy-auth-request
Hi Tim,
Wanted to say thank you for your help, I got every thing working.
in case it helps others new to this figure out how to accomplish the
task, here is the config I ended up with:
frontendhttpfront
bind ${ADDRESS}:80 v4v6
bind ${ADDRESS}:443 v4v6 ssl crt /Computerisms/config/certificates/
redirect scheme https code 301 if !{ ssl_fc }
mode http
option httplog
log global
http-request lua.auth-requestauth_request/index.html
## ACLs
acl test.computerisms.ca ssl_fc_sni -i test.computerisms.ca
## AUTHREQ
use_backend auth_request if ! { var(txn.auth_response_successful) -m
bool } test.computerisms.ca
## AUTHBACKEND
use_backendtest.computerisms.caif test.computerisms.ca
default_backendmooglehttps
On 2018-09-04 02:42 PM, Tim Düsterhus wrote:
Bob,
Am 04.09.2018 um 23:27 schrieb Computerisms Corporation:
First, apologies for the breach in etiquette, will use reply-all on this
list.
FWIW: I removed Thierry again, because at this point this is no longer
about Lua itself :-)
After following the thread in github and your hint that a apt-gettable
package for luasocket exists, I purged everything from luarocks and
installed from debian repos and the script is no longer producing errors
and the backend is successfully logging connections.
Perfect!
A follow up question, if I may: my backend leads to a simple apache
Sure.
DocumentRoot with auth that works as expected when accessed directly. I
was expecting when accessing through haproxy that when the auth-request
script did its subrequest, I would get the apache credentials pop up in
the browser. However, no pop up happens, and the backend immediately
fails. Did I misunderstand how this would work?
The only thing my script does is checking the response code of some
subrequest. What you do with it is entirely up to you. The example in
the README on GitHub denies the request.
Instead of denying the request you could select a different backend
(i.e. Apache) which then would be able to show the authentication screen.
In my blog post I use this to force the request to go to OAuth Proxy if
the verification fails:
use_backend oauth_proxy if ! { var(txn.auth_response_successful) -m
bool }
Instead of `oauth_proxy` you would use `auth_request` based on the
configuration you gave previously.
I thought that maybe the user/pass needs to be included in the url
(http://user:p...@domain.tld), but the behaviour remains the same
HTTP Basic authentication should work out of the box, because all the
request headers are forwarded to the backend.
Best regards
Tim Düsterhus
Re: haproxy-auth-request
Hi Tim, First, apologies for the breach in etiquette, will use reply-all on this list. After following the thread in github and your hint that a apt-gettable package for luasocket exists, I purged everything from luarocks and installed from debian repos and the script is no longer producing errors and the backend is successfully logging connections. A follow up question, if I may: my backend leads to a simple apache DocumentRoot with auth that works as expected when accessed directly. I was expecting when accessing through haproxy that when the auth-request script did its subrequest, I would get the apache credentials pop up in the browser. However, no pop up happens, and the backend immediately fails. Did I misunderstand how this would work? I thought that maybe the user/pass needs to be included in the url (http://user:p...@domain.tld), but the behaviour remains the same On 2018-09-04 07:35 AM, Tim Düsterhus wrote: Hi all, Am 04.09.2018 um 13:50 schrieb Tim Düsterhus: Someone reported the same error in the issue tracker on GitHub: https://github.com/TimWolla/haproxy-auth-request/issues/4 The issue in the bug tracker was caused by an old version of lua-socket. Unfortunately the author of lua-socket does not seem to do regular releases (at least there are no git tags) which makes it hard to specify what a supported version is. Best regards Tim Düsterhus
Re: haproxy-auth-request
Hi Tim, Joseph,
Thank you both very much for answering; so very much appreciated.
Am 02.09.2018 um 00:12 schrieb Joseph Sible:
Try removing the highlighted block of code:
https://github.com/TimWolla/haproxy-auth-request/blob/e6a686e6f192200a6c7001c303b87d2d6e9d4788/auth-request.lua#L45-L51
It's a monkey-patch to haproxy's socket that you might not need.
Yes, it's obsolete as of haproxy 1.8.4. But I don't believe it to be the
cause of the issues. It looks like the monkey patched function itself is
given invalid parameters.
so, this is good to know, but for the sake of completeness, I will
mention that I have commented that particular block of code, as well as
just the :old_settimeout line, as well as the block of code above it,
and I commented the timeout in the http.lua file, as suggested in the
blog post, plus I tried commenting and changing several other lines in
the auth-request.lua file.
the commenting and changing mostly(always?) ended up in the following error:
Lua function 'auth-request': runtime error: attempt to yield across a
C-call boundary from [C] field 'request',
/Computerisms/config/etc/haproxy.auth.lua:95 C function line 56.
I believe the issue might be that your version of LuaSocket calls
`settimeout` differently that I anticipated in haproxy-auth-request.
What version of LuaSocket are you using? Can you give your configuration?
Absolutely! I am really new to lua and haproxy both, so very possibly I
didn't do something as I was supposed to. I installed luasocket via
luarocks, I had to do some digging around to get it to install the
lua5.3 version, on debian it apparently has preference over lua5.1.
As per luarocks:
root@sand2lian:/etc/apache2# luarocks list
Installed rocks:
luasocket
3.0rc1-2 (installed) - /usr/local/lib/luarocks/rocks
As per dpkg:
root@sand2lian:/etc/apache2# dpkg -l lua5.3
ii lua5.3 5.3.3-1
my haproxy.cfg file is largely by the book as per your instructions on
github:
backend auth_request
modehttp
option forwardfor
server auth-request127.0.0.1:8044
frontendhttpfront
bind${ADDRESS}:80
modehttp
option httplog
log global
http-requestlua.auth-request auth_request /index.html
http-request deny if ! { var(txn.auth_response_successful) -m bool }
# redirectscheme https code 301 if !{ ssl_fc }
default_backend mooglehttp
hm, Tim, what you say about different versions actually might be the
right track; in your blog post the http.lua file you link to shows the line:
h.try(c:settimeout(_M.TIMEOUT))
as line 119, but in my file, that line is number 116, so doesn't seem to
be the same file... will see if I can follow this trail to a solution...
Best regards
Tim Düsterhus
haproxy-auth-request
Hi Gurus; I was in the IRC channel the other day looking for a way to get authentication through apache's authnz-external working from haproxy; specifically I have a few nodejs applications and I want to put an authentication system using imap in front. I already have authnz-external working so it would be convenient to continue using it. dcorbett suggested I investigate lua, which led me to finding https://github.com/TimWolla/haproxy-auth-request. seems like just the thing I need. After I checked that I met all the criteria and installed it and got every thing setup and worked out my mistakes, I was left with log entries like: Lua function 'auth-request': runtime error: haproxy.auth.lua:48: bad argument #1 to 'old_settimeout' (number expected, got nil) from [C] field 'request', haproxy.auth.lua:95 C function line 56. Fortunately, the author was most excellent and documented the history of this script here: https://bl.duesterhus.eu/20180119/, specifically the section regarding haproxy's sockets. Through that post and also comments in the github script containing links to https://www.mail-archive.com/haproxy@formilux.org/msg28604.html and https://www.mail-archive.com/haproxy@formilux.org/msg28574.html, I am lead to believe this should be an fixed in my apt-installed version of haproxy 1.8.13-1. I have been investigating, commenting code, and hacking by trial and error to find a solution, but so far I am not able to get past this point. Clearly my skills and understanding are not yet where they need to be, being relatively new to both haproxy and lua. It occurs that this could be a lua problem (as opposed to haproxy), but according to what I have read and understood, it seems that this is related to haproxy's implementation of lua more than lua itself. Hence I am asking here first. Wondering if anyone can offer some insight, or point me at some required reading that might shed some light on this but isn't aimed at a developer level of understanding? I have read through https://www.arpalert.org/haproxy-lua.html#h211 a time or two, but if there is a shining light bulb in there it hasn't blinded me yet. -- Bob Miller Cell: 867-334-7117 Office: 867-633-3760 www.computerisms.ca
