Follow up on the resolution.
Data was being left in the socket, although it wasn’t immediately obvious why.
The affected end point received gzipped POST bodies, and would process this
somewhat like this:
uncompressed_body = StringIO.new(Zlib::GzipReader.new(request.body).read)
process(uncompressed_body)
At the end of each request, all of the uncompressed data had been consumed and
`uncompressed_body.eof?` would always be true. However, GzipReader#close wasn’t
explicitly called, causing anywhere between 1 and 7 trailing bytes of the 8
byte gzip footer to remain unread in request.body (rack.input) in a small
portion of requests. Rewriting this to explicitly close the GzipReader forces
it to read and validate the footer, and leaves us with a completely consumed
request.body.
Thanks for your help identifying the root cause.