Hi there,
I'm currently debugging are very strange problem: If a header has a
length of exactly 2048 bytes (2046 characters +CRLF), Pound silently
removes the LF, effectively breaking the header (as it's RFC-defined
that headers have to end with CRLF, not CR). Amazingly the problem does
not occur for headers with 2045 or 2047 characters (+CRLF), but only for
those with exactly 2046 bytes.
My first thought was that it's a MAXBUF issue because in older versions
MAXBUF had a value of 2048, which was then dropped to 1024, but later
raised to 4096 in current versions of Pound (I've found some posts about
problems with e.g. cookies with large values, which were directly
related to MAXBUF, but my current problem case seems different).
My test scenario is built on CentOS 6.6 (fully up-to-date).
I have tested both Pound 2.6 and 2.7; the problem arises in both of
them. While we're using some patches in production, I am using the
original, unpatched source from apsis.ch for the tests to make sure that
I'm not talking about a problem introduced by some patch.
In the following tests I'm referring to Pound 2.6, but Pound 2.7
provides the same results.
I have compiled Pound without any specials (no libunwind, no
gperftools), against the distro's OpenSSL as well as a manually compiled
OpenSSL (doesn't make a difference, and I'm not using HTTPS for the
tests anyway). My configure line for the tests is:
server$ ./configure --prefix=/home/poundtest/pound26 \
--with-owner=poundtest --with-group=poundtest --disable-super
I have stripped down /home/poundtest/pound26/etc/pound.cfg to a minimal
version which only contains this:
----- begin cut -----
Daemon 0
LogFacility -
LogLevel 2
Service
BackEnd
Address 127.0.0.1
Port 81
End
End
ListenHTTP
Address ::
Port 63161
xHTTP 2
AddHeader "X-Forwarded-Port: 80"
End
----- end cut -----
To test, I'm using curl, sending a header of exactly 2046 bytes length
(I'm using a series of AAA..., ending with ...AAABCD so that the end of
the header is easily recognizable in the following steps):
client$ curl --header "X-tra-long1: AAAAA[...]AAAAABCD" \
--header "X-tra-long2: done" http://musca.uberspace.de:63161/
On the server I'm running Pound under strace, restricted to read() and
write(), with a snaplen of 4096 so I'm not cutting anything from output:
server$ strace -f -eread,write -s 4096 ~/pound26/sbin/pound
Now, here comes the interesting thing. I've only shortened the "AAA..."
series for better readability.
Request comes in:
[pid 11594] read(4, "GET / HTTP/1.1\r\nUser-Agent: curl/7.29.0\r\nHost:
musca.uberspace.de:63161\r\nAccept: */*\r\nX-tra-long1: AAAAA[...]AAAAA", 4096)
= 1440
[pid 11594] read(4, "AAAAA[...]AAAAABCD\r\nX-tra-long2: done\r\n\r\n", 4096) =
715
Request passes through to backend:
[pid 11594] write(5, "GET / HTTP/1.1\r\nUser-Agent: curl/7.29.0\r\nHost:
musca.uberspace.de:63161\r\nAccept: */*\r\nX-tra-long1:
AAAAA[...]AAAAABCD\rX-tra-long2: done\r\nX-Forwarded-Port:
80\r\nX-Forwarded-For: ::ffff:82.207.131.175\r\n\r\n", 2216) = 2216
Holy shit. You can clearly see in front of the "X-tra-long2:" header
that there's "\r\n" coming in, but only "\r" is sent to the backend.
It's the only header where this happens; all others are correctly sent
terminated with "\r\n".
The problem arises at _exactly_ 2046 bytes + CRLF. To validate, I have
added an "E" to my very long header. This is the result:
Request comes in:
[pid 31433] read(4, "GET / HTTP/1.1\r\nUser-Agent: curl/7.29.0\r\nHost:
musca.uberspace.de:63161\r\nAccept: */*\r\nX-tra-long1: AAAAA[...]AAAAA", 4096)
= 1400
[pid 31433] read(4, "AAAAA[...]AAAAABCDE\r\nX-tra-long2: done\r\n\r\n", 4096) =
756
Request passes through to backend:
[pid 31433] write(5, "GET / HTTP/1.1\r\nUser-Agent: curl/7.29.0\r\nHost:
musca.uberspace.de:63161\r\nAccept: */*\r\nX-tra-long1:
AAAAA[...]AAAAABCDE\r\nX-tra-long2: done\r\nX-Forwarded-Port:
80\r\nX-Forwarded-For: ::ffff:62.216.165.244\r\n\r\n", 2218) = 2218
As you can see, the "\r\n" separator is correctly transported to the
backend in this case. Same with any other tested length != 2046.
I have to admit that I'm totally stuck here: I have no idea why this is
happening or what part of the code could possibly relate to this
problem. Any help on this is highly appreciated.
(The real-world scenario for this is a user running "Firefox Sync" as
the Pound backend. His Firefox browser is sending "Authorization:
BrowserID <assertion>" headers that have a length 2046 bytes + CRLF;
Firefox Sync on the server side effectively chokes on the broken header,
rendering the application useless.)
Best regards,
Jonas
--
To unsubscribe send an email with subject unsubscribe to [email protected].
Please contact [email protected] for questions.
