On Mon, Jul 25, 2016 at 2:02 AM, Pavlos Parissis <pavlos.paris...@gmail.com> wrote:
> On 22/07/2016 11:53 πμ, Willy Tarreau wrote: > > Hi Cyrus, > > > > On Thu, Jul 21, 2016 at 11:22:06PM -0700, Cyrus Katrak wrote: > >> Greetings from Slack Technologies, > >> > >> We are evaluating HAProxy as a load balancing solution and so far are > quite pleased with > >> it. One feature we require is the ability to log when a connection > started (option > >> logasap), and again when it is terminated. We have many long lived > websocket connections, > >> and this level of introspection is invaluable for debugging. > >> > >> We are currently accomplishing this with the following patch: diff -u -r > >> haproxy-1.6.5-orig/src/proto_http.c haproxy-1.6.5/src/proto_http.c --- > >> haproxy-1.6.5-orig/src/proto_http.c 2016-05-10 06:42:00.000000000 -0700 > +++ > >> haproxy-1.6.5/src/proto_http.c 2016-07-21 14:19:10.131437264 -0700 > @@ -6926,6 +6926,7 > >> @@ s->logs.bytes_out = txn->rsp.eoh; s->do_log(s); s->logs.bytes_out = > 0; + > >> s->logs.logwait |= LW_BYTES; } return 1; } > >> > >> Questions: 1. Is this correct? Are there unforeseen consequences of > this change? > > > > No, no bad consequences, it will work for you in this situation. > > > >> 2. Would this functionality be welcomed upstream? If so, I can look > into creating a > >> separate option for this. > > > > We found that there are various situations where people would like to > have different places > > to produce logs. The most common demands I've collected over the years > were : - after > > request is full received - after server starts to respond - after server > finishes. > > > > I had an experimental patch adding up to 6 locations (more or less > similar to yours above). > > But I found that a side effect of this was that logformat doesn't play > nicely with it since > > some elements are not necessarily present. So either we stop emitting > warnings for bad log > > formats (I'd prefer not), or better, we plan on having different log > formats for each step, > > which requires different configuration settings. > > > > For those setups which forward logs to ElasticSearch/druid/similar > products it may mess up dashboards as information for single request is > spread across multiple > log lines. > > Not good but can still be fixed with the multiline module in Logstash like we do with Java exceptions. > For the past 3 weeks I have been working on 'dashboarding' HAProxy logs > using Graphite and ElasticSearch (soon with druid). > I combine time series and event based information, thus I have a lot of > data available to me. > I am now in the process to go beyond simply measuring num of req/secs and > apply 'behavioral > analytics'. The limiting factor for doing this is *not* HAProxy logs but > how Graphite and > ElasticSearch work/store data. Druid should help on this, I hope... > Have done the same with Logstash, Statsd, Graphite and Grafana. Collecting request/response time, status codes and bytes transfer from Haproxy. Grafana UI makes a big difference compared to the Graphite charts. > > What is 'behavioral analytics' in the context of HAProxy logs? > Let's say you enable support for ECC and RSA SSL certs, you know by fact > the less CPU cycles > will be utilized. But, what else is effected by that, either positively or > negatively. > Right now, you can't easily answer this question. > > Why do you care to know about it ? > For multiple reasons: > > #1 You want to measure ROI of the change > #2 You want to reduce the 'surprise' factor. That is, a change occurred in > area that > you didn't think of. > > To sum up, HAProxy logs are very reach in information and any effort to > make them to > provide more information will be very much appreciated. > > > In addition, some people would like to be able to emit a log from an > http-request rule, > > which also comes with the idea of a separate log format description. > > > > +1 as it very handy during debug sessions. > > > And the last point is that we definitely need to be able to log the step > we're in to > > distinguish the various log lines. But it could simply be a log format > tag so that users > > place it where they want. > > I am doing that to separate logs between TCP and HTTP frontends: > > mode tcp > option tcplog > log-tag haproxy-tcp > > and I have template on rsyslogd to pass fields to ElasticSearch based on > the mode. > Adding another tag is an easy win. > > > Cheers, > Pavlos > >
