On 6/13/2018 4:04 AM, Markus Jelsma wrote:
You mentioned shard handler tweaks, thanks. I see we have an incorrect setting there for maximumPoolSize, way too high, but that doesn't account for the number of threads created. After reducing the number, for dubious reasons, twice the number of threads are created and the node dies.
The specific config in the shard handler I was thinking of was socket timeouts.
The EofException in Jetty almost always indicates that the client disconnected before the server responded. A low socket timeout will cause a client to disconnect if the server takes longer than the timeout to finish its work. When the server finally does finish and try to respond, it throws the EofException because the connection it was expecting to use is no longer there.
I believe that the default socket timeout in Solr's shard handler is 60 seconds. Which is a relative eternity in most situations. It would take a particularly nasty GC pause problem to exceed that timeout.
Regarding memory leaks, of course, the first that came to mind is that i made an error which only causes trouble on 7.3, but it is unreproducible so far, even if i fully replicate production in a test environment. Since it only leaks on commits, first suspect were URPs, and the URPs are the only things i can disable in production without affecting customers. Needless to say, it weren't the URPs.
Custom update processors could cause leaks, but it is not something that I would expect from a typical URP implementation. If you've disabled them and it's still happening, then that's probably not it. It's plugins for queries that have the most potential for not closing resources correctly even when written by experienced programmers. I'm not sure what the potential for leaks is on index-time plugins, but I suspect that it's less likely than problems with query plugins.
Thanks, Shawn
