unique-id-header with capture request header

2014-05-13 Thread Bryan Talbot
We have more than 1 proxy tier. The edge proxy generates a unique ID and
the other tiers (and apps in between) log the value and pass it around as a
per-request id.

Middle tier haproxy instances capture and log the unique id using capture
request header which works fine; however, for the edge proxy this doesn't
work since the ID doesn't seem to be available as a request header yet and
a custom log-format must be used instead.

This means that the logs generated by edge and non-edge proxies have a
different format and be parsed specially. Also, a custom log-format must be
maintained just for this purpose.

What's the best way to capture the unique-id generated in a proxy or from a
request and log it in a consistent way?

-Bryan


Re: unique-id-header with capture request header

2014-05-13 Thread Patrick Hemmer
*From: *Bryan Talbot bryan.tal...@playnext.com
*Sent: * 2014-05-13 11:52:32 E
*To: *HAProxy haproxy@formilux.org
*Subject: *unique-id-header with capture request header

 We have more than 1 proxy tier. The edge proxy generates a unique ID
 and the other tiers (and apps in between) log the value and pass it
 around as a per-request id.

 Middle tier haproxy instances capture and log the unique id using
 capture request header which works fine; however, for the edge proxy
 this doesn't work since the ID doesn't seem to be available as a
 request header yet and a custom log-format must be used instead.

 This means that the logs generated by edge and non-edge proxies have a
 different format and be parsed specially. Also, a custom log-format
 must be maintained just for this purpose.

 What's the best way to capture the unique-id generated in a proxy or
 from a request and log it in a consistent way?

 -Bryan


We do this exact same thing. Unique ID generated on the outside, and
passed around on the inside.
However we use a custom log format on both the internal and external
haproxy so that the format is exactly the same.

External:
unique-id-format %{+X}o%pid-%rt
capture request header X-Request-Id len 12
log-format %ci:%cp\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\
%ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %U/%B\ %ST\ %tsc\
%ID,%[capture.req.hdr(0)]\ +\ %r

Internal:
capture request header X-Request-Id len 12
log-format %ci:%cp\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\
%ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %U/%B\ %ST\ %tsc\ %[capture.req.hdr(0)]\ +\ %r

We capture on external as well as occasionally an internal service will
re-route back through the external, and so it gets a new ID, and we can
correlate the two. But the fields are space delimited, so that even
showing 2 IDs, it's still a single field separated by comma (assuming
some client doesn't pass an invalid value in. but that's also why this
is at the end of the log line)

The other advantage of doing it that the default http log format is
missing %U, which we find useful to have.

-Patrick