Thanks for the quick response! Just a quick correction and question.
When I mentioned requests above 1000k, i actually meant 1000 bytes. Basically it seemed like it didn't work whenever the request was big enough to force a 100-Continue. I'm not sure if this would change anything, since we still want to be capable of handling large requests, but I wanted to make sure that I provided as much information as possible. You mentioned enabling the request POST body analyzer. What is the best way to use this? On Tue, Mar 1, 2011 at 12:19 PM, Willy Tarreau <[email protected]> wrote: > Hi Nathaniel, > > On Tue, Mar 01, 2011 at 11:15:48AM -0800, Nathaniel Irvin wrote: > > We have been looking at modifying HAProxy to be able to create ACL's > that > > read in the request body and forward based upon whether or not a certain > > string is contained within it. > > It's planned for 1.5 but not done yet. I think there's not too much work > anymore to do that. I would have preferred to do it *after* we convert > the ACL to use the pattern frameworks in order to avoid duplicate the > work, but given how late we are, having it right now should be fine. > > > It seems like there is everything needed except the "acl_fetch_line" > > function. In this function, we have been able to read in most requests, > but > > requests bigger than 1000k seem to break because the entire message is > not > > read in before the function is called. > > POST body analysis is limited to the request buffer length anyway (16 kB by > default). You should never expect to buffer nor analyse large POSTs such as > the ones you're talking about above. That does not scale at all. At 10000 > concurrent connections, you're eating 10gigs of RAM ! Some proxy products > even store POST requests on disk to save memory. > > > The flag "ACL_TEST_F_MAY_CHANGE" appears to be made to cover this > purpose, > > but setting this flag doesn't seem to do anything because the flag > > "ACL_PARTIAL" is never set for http acl's, and there doesn't seem to be > any > > support for calling the fetch function again once the rest of the message > is > > received. > > I see. That's one of the reasons it would have been better to conver the > ACL framework first :-/ > > Right now, what I'd suggest would be to simply enable the request POST body > analyser when at least one such ACL is needed. This analyser will wait for > either the whole content-length to be read or for the buffer to be full. > Then you can run your ACL on the contents and see if the content is there > or not. > > It will also avoid the risk of rechecking the whole buffer upon each read > (potentially every 1460 bytes when dealing with remote clients sending one > segment at a time). > > Regards, > Willy > > -- Nathaniel Irvin Junior Developer True North Service, Inc.
