Hi Mark, thanks for your feedback. Please see below:
On 23/01/2020 13:40, Manuel Dominguez Sarmiento wrote:> Hi, we started
noticing that HttpServletRequest.getRemoteAddr() was
sometimes returning NULL (which is invalid according to the servlet
spec), about 20-50 times per day (we have high-load servers which
routinely handle over 100 requests per second). This only happens on
secure HTTP/2 requests.
We noticed this while running the latest 9.0.30, but looking at our logs
for the past few months, this was also happening on 9.0.21 as well as
9.0.22 which are the previous versions we had deployed. This issue was
not present in 9.0.17, which was the version we were using before 9.0.21.
Before anyone cries out "there is a bug in your code", Tomcat itself
reports a NULL remoteAddr via AccessLogValve in these cases. It is also
interesting that the remotePort is reported as "-1".
We looked through the changelog, but we can't figure out what change
might have triggered this (presumably some change between 9.0.17 and
9.0.21). It looks like some kind of async race condition during the
HTTP/2 upgrade, but this is only an educated guess.
I'll take a look.
You mention async. Are these requests using the Servlet async API?
No, we're not using the async API in any capacity. Just the regular
servlet API: doGet() / doPost() / etc.
Perhaps my usage of "async" was confusing (as that can refer to several
different things in Tomcat). I just meant that it feels like a race
condition between different threads.
Our server.xml is below (confidential data has been modified). We are
using the NIO connector, Tomcat Native + APR libraries (but NOT the APR
connector). The useAsyncIO flag is disabled because of server lockups
we've experienced with this flag enabled on our production servers. Any
ideas?
Nothing comes to mind immediately.
Mark
Please let me know if there's any diagnostic info that might help with
this and we'll produce it.
Thanks again,
- Manuel Dominguez Sarmiento
