Hi Marcin,
On Mon, Jul 12, 2021 at 04:59:32PM +0200, Marcin Deranek wrote:
> Hi,
>
> Over a past few weeks I have been working on implementing JA3 compatible
> TLS Fingerprinting[1] in the HAProxy. You can find the outcome in
> attachments. Feel free to review/comment them.
Thanks for this. First, I had never heard about this JA3 thing. I'm
seeing at your link that there's even a preliminary list of well-known
signatures, I hope it will not start to be used too much as an access
restriction, or we risk to observe some ossifications of the clients
in field, and a real difficulty to introduce new ones. At least the
signature could be used to rate-limit unknown ones. Time will tell...
Just out of curiosity (feel free not to respond if you'd prefer not to),
how are you using this result ? Is it to try to figure outliers by
matching signatures against what the user-agent claims to be, or just
for monitoring/logging or maybe for rate limiting bots ? Did you detect
different SSL libraries for a same user-agent ?
> Here are some choices I made which you should be aware of:
> - I decided to go with a "modular" approach where you can build JA3
> compatible fingerprint with available fetchers/converters rather than a
> single JA3 fetcher. This makes approach more "reusable" in some other
> scenarios.
I agree with this approach and generally that's what we're doing, because
whatever you implement in this context may serve other cases later and
speed up development of new features.
> - Each Client Hello related fetcher has option to include/exclude GREASE
> (RFC8701) values from the output. This is mainly for backward compatibility
> and ability to get "pure" data. I suspect in most cases people do not want
> GREASE values to be present in the output (which is not the case right now
> for cipherlist fetchers).
Indeed, it makes sense to offer the option to exclude purposely added noise
from the computation. I don't know what JA3 specifies regarding this, but I
guess it excludes it.
> - exclude_grease function allocates trash on demand depending on GREASE
> (RFC8701) values position in the list. We can get away without creating
> trash buffer if GREASE values are present at the very beginning and/or the
> very end of the list. I decided to allocate trash buffer only when it's
> really needed, so that's why it's creation is "hidden" inside exlude_grease
> function.
>From what I'm seeing there, you could probably simplify the function and
consider that you always allocate and copy if you need to exclude grease.
A client hello is not huge anyway, and the time saved in the memcpy() of
a few hundred bytes is not much compared to the overall processing of an
SSL hello. By the way, it would be nice if you could use a different name
(e.g. "temp") for your local trash pointer, as it shadows the thread-local
"trash" and can be confusing.
> - Now ssl_capture (next to ciphersuite) contains data about extensions, ec
> ciphers etc. One of the reasons I decided to merge all those values in a
> single ssl_capture buffer is easier control of buffer size limit. I think
> it's beneficial to have a single buffer limit for all those values rather
> than separate values for each. Having said that probably
> tune.ssl.capture-cipherlist-size needs to change it's name to eg.
> tune.ssl.capture-buffer-limit to better reflect it's function.
All this would indeed make a lot of sense. However, just renaming a config
setting is not an option. What can be done is to create the new one and
continue to process the old one while emitting a deprecation warning asking
to use the other one instead. Maybe the size will differ and the doc will
need to explain how to transform the values.
I'm having an issue with your new definition of ssl_capture_location
and its use in ssl_capture:
/* Location and size of the data in the buffer */
struct ssl_capture_location {
unsigned char len; // offset 0, size 1 byte, followed by a 7-byte
hole
size_t offset; // offset 8, size 8 bytes
};
=> this structure takes 16 bytes of memory
/* This memory pool is used for capturing clienthello parameters. */
struct ssl_capture {
unsigned long long int xxh64;
unsigned int protocol_version;
struct ssl_capture_location ciphersuite;
struct ssl_capture_location extensions;
struct ssl_capture_location ec;
struct ssl_capture_location ec_formats;
char data[VAR_ARRAY];
};
=> thus above just for the lengths we're using 64 bytes of memory, this
starts to be quite a lot per capture. Given that no TLS record can be
larger than 16kB (or is that 64?), you could use two unsigned shorts
and divide this overhead by 4.
> - Instead of creating a new converter I decided to extend existing hex
> conveter to provide a similar functionality to bin2int. I thought this
> makes more sense as extended hex converter is fully backward compatible. It
> has to be noted that extended hex converter is not strictly necessary to
> produce JA3 TLS Fingerprint, but but might useful in some other scenarios.
Actually I've already missed this ability to decode larger ints so it's
welcome. But there's an important point that your change doesn't take into
account (for both bin2int and hex), which is the input byte ordering. At
the moment only big endian is supported. In addition, the "bin2int" makes
me think it emits an integer while it emits an ASCII decimal representation
of it.
What I could suggest instead would be to add the following converters:
be2dec() // big endian to decimal
be2hex() // big endian to hexadecimal
le2dec() // little endian to decimal
le2hex() // little endian to hexadecimal
We could later complete these with other less useful variants like octal
or raw ints (e.g. to extract dates). Just like we could imagine supporting
some flavors of varints on input later if needed for some protocols.
> Example usage:
> http-request set-header X-SSL-JA3
> %[ssl_fc_protocol_hello_id],%[ssl_fc_cipherlist_bin(1),bin2int(-,2)],%[ssl_fc_extlist_bin(1),bin2int(-,2)],%[ssl_fc_eclist_bin(1),bin2int(-,2)],%[ssl_fc_ecformats_bin,bin2int(-,1)]
> http-request set-header X-SSL-JA3-Hash
> %[req.fhdr(x-ssl-ja3),digest(md5),hex]
I think in the doc you should add an example showing how to match a
signature against those listed in file "lists/osx-nix-ja3.csv" in the
project. It will help you verify if the solution completely works and
is practically usable. Maybe it can involve intermediary variables
for example.
> Question: I noticed that during Client Hello parsing we calculate xxh64
> right away and store it. Isn't better to calculate it when it's actually
> used?
Maybe, but quite honestly I really don't know that area. xxh64 is extremely
cheap so it's probably no big deal, and I don't know if we keep all the
elements needed to compute it or just the hash itself, in which case it
can make sense to do it as early as possible. Just guessing :-/
Thanks!
Willy
