Woohoo!! :) thanks, Willy, for the work. Seems like a really great
list of stuff there.
Especially love the "HTTP invalid request and response captures per
frontend/backend" feature - I would definitely love to be able to see
what we're getting over here where we use haproxy....
One question if u dont mind - session rate limiting on frontends -
what's the use case for this?
-jf
--
In the meantime, here is your PSA:
"It's so hard to write a graphics driver that open-sourcing it would not help."
-- Andrew Fear, Software Product Manager, NVIDIA Corporation
http://kerneltrap.org/node/7228
On Sat, Mar 7, 2009 at 7:11 AM, Willy Tarreau <w...@1wt.eu> wrote:
>
> Hi all !
>
> About 3 months ago I told you that 1.3.16 was getting closer. Now I
> really think it's getting even closer. Since then, we have fixed all
> remaining visible bugs, which does not mean that all bugs went away of
> course. Also several new useful features have been implemented due to
> concrete opportunities :
>
> - doc is finished. All keywords, log options etc... have been migrated
> to the new doc. The old one is still there "just in case", but should
> not be needed anymore.
>
> - autonomous forwarding layer between sockets without waking the
> task up : when data must be forwarded from one socket to another
> one, we don't wake the task up anymore if not needed. This saves
> many CPU cycles on large objects and has improved the maximum data
> rate by 10-15% depending on the workload. A further improvement
> will consist in allocating buffers on the fly from a pool just
> during the transfer, and releasing empty buffers when not in use
> in order to reduce memory requirements.
>
> - TCP splicing on very recent linux 2.6 (2.6.27.19, 2.6.28.6, or
> 2.6.29). TCP splicing enables zero-copy forward between two network
> interfaces. With most network interfaces it does not seem to change
> anything, however with some other NICs (at least Myricom's 10GE), we
> observe huge savings. I could even reach 10 Gbps of layer 7 data
> forwarding with only 25% CPU usage! Please note that this must not
> be used with any kernel earlier than versions above since all
> previous tcp-splice implementations are buggy and will randomly and
> silently corrupt your data.
>
> - unix stats socket is working again.
>
> - complete session table dumps on the unix socket. It reports
> pointers, states, protocol, timeouts, etc... It's primarily meant
> for development, but will help understand why sometimes a process
> refuses to die when some sessions remain present.
>
> - HTTP invalid request and response captures per frontend/backend :
> those who are fed up with tracking 502 coming from buggy servers
> will love this one. Each invalid request or response (non HTTP
> compliant) is copied into a reserved space in the frontend (request)
> or backend (response) with info about the client's IP, the server,
> the exact date, etc... so that the admin can later consult those
> errors by simply sending a "show errors" request on the unix stats
> socket. The exact position of the invalid character is indicated,
> and that eases the troubleshooting a lot ! It's also useful to keep
> complete captures of attacks, when the attacker sends invalid
> requests :-)
>
> - the layering work has been continued for a long time and a massive
> cleanup has been performed (another one is still needed though)
>
> - the internal I/O and scheduler subsystems are progressively getting
> more mature, making it easier to integrate new features.
>
> - add support for options "clear-cookie", "set-cookie" and "drop-query"
> to the "redirect" keyword.
>
> - ability to bind to a specific interface for listeners as well as for
> source of outgoing connections. This will help on complex setups where
> several interfaces are attached to the same LAN.
>
> - ability to bind some instances to some processes in multi-process
> mode. Some people want to run frontend X on process 1 and frontend
> Y on process 2. This is now possible. A future improvement will
> consist in defining what CPU/core each process can run on (OS
> dependant).
>
> - session rate measurement per frontend, backend and server. This is
> now reported in the stats in a "rate" column, in number of sessions
> per second. Right now this is only on the last second, but at least
> the algorithm gives an accurate measurement with very low CPU usage.
> This value may also be checked in ACLs in order to write conditions
> based on performance.
>
> - session rate limiting on frontends : using "rate-limit sessions XXX"
> it is now possible to limit the rate at which a frontend will accept
> connections. This is very accurate too. Right now it's only limited
> to sessions per second, but it will evolve to different periods. I've
> successfully tried values as low as 1 and as high as 55000 sessions/s,
> all of them gave me the expected performance within +/- 0.1% due to
> the measuring resolution. This means you can simply set a limit per
> application. Massive collocations will surely love this :-)
>
>
> Some other features are pending : the recent http-check content inspection,
> refined request/response ACLs, etc... those will more likely be merged after
> 1.3.16.
>
> After that, I plan to work on smarter memory allocation (for buffers), and
> a general QoS framework supporting shared and maybe hierarchical classes
> of traffic, as well as per-IP and per-group classes. The idea is simply to
> be able to limit the per-client data rate, request rate, and do that globally
> or per frontend, etc...
>
> I'm not even discussing keep-alive, SSL or SMTP because I'm fairly certain
> that those are the immediate big questions that will be asked after the
> release.
>
> The current version has been running in production with tcp-splicing on a
> very big site for more than one month, and on my small site for about as
> long. I know that some of the people on the list also use it regularly.
> For that reason, I think I'll finish a little bit of polishing and release
> it as 1.3.16 very soon. The last GIT version will appear as a new snapshot
> in a few hours in the usual "snapshot" directory as 20090307.
>
> As usual, you will have to use it with care, as there certainly are a lot
> of remaining bugs. But there are nice features too :-) I think that bugs
> will then be easier to fix once identified. Time will tell if they are
> also easier to spot :-)
>
> Regards,
> Willy
>
>
>
