On 08.09.20 22:54, Tim Düsterhus wrote:
Reinhard,
Björn,
Am 08.09.20 um 21:39 schrieb Björn Jacke:
the only official supported way to identify a google bot is to run a
reverse DNS lookup on the accessing IP address and run a forward DNS
lookup on the result to verify that it points to accessing IP address
and the resulting domain name is in either googlebot.com or google.com
domain.
...
thanks for asking this again, I brought this up earlier this year and I
got no answer:
https://www.mail-archive.com/[email protected]/msg37301.html
I would expect that this is something that most sites would actually
want to check and I'm surprised that there is no solution for this or at
least none that is obvious to find.
The usually recommended solution for this kind of checks is either Lua
or the SPOA, running the actual logic out of process.
For Lua my haproxy-auth-request script is a batteries included solution
to query an arbitrary HTTP service:
https://github.com/TimWolla/haproxy-auth-request. It comes with the
drawback that Lua runs single-threaded within HAProxy, so you might not
want to use this if the checks need to run in the hot path, handling
thousands of requests per second.
It should be possible to cache the results of the script using a stick
table or a map.
Back in nginx times I used nginx' auth_request to query a local service
that checked whether the client IP address was a Tor exit node. It
worked well.
For SPOA there's this random IP reputation service within the HAProxy
repository:
https://github.com/haproxy/haproxy/tree/master/contrib/spoa_example. I
never used the SPOA feature, so I can't comment on whether that example
generally works and how hard it would be to extend it. It certainly
comes with the restriction that you are limited to C or Python (or a
manual implementation of the SPOA protocol) vs anything that speaks HTTP.
In addition to Tim's answer you can also try to use spoa_server which
supports `-n <workers>`.
https://github.com/haproxy/haproxy/tree/master/contrib/spoa_server
Best regards
Tim Düsterhus
Regards
Aleks
