Hi,
we want to use gzip-compression between our application and the solr server.
We use a standalone solr server version 8.4.1 and the prepackaged jetty as
application server.
We have enabled the jetty gzip module by adding these two files:
{path_to_solr}/server/modules/gzip.mod (see below the question)
{path_to_solr}/server/etc/jetty-gzip.xml (see below the question)
Within the application we use a HttpSolrServer that is configured with
allowCompression=true.
After we had released our application we saw that the number of connections
within the TCP-state CLOSE_WAIT rising up until the application was not
able to open new connections.
After a long debugging session we think the problem is that the header
"Content-Length" that is returned by the jetty is sometimes wrong when
gzip-compression is enabled.
The solrj client uses a ContentLengthInputStream, that uses the header
"Content-Lenght" to detect if all data was received. But the InputStream
can not be fully consumed because the value of the header "Content-Lenght"
is higher than the actual content-length.
Usually the method PoolingHttpClientConnectionManager.releaseConnection is
called after the InputStream was fully consumed. This give the connection
free to be reused or to be closed by the application.
Due to the incorrect header "Content-Length" the
PoolingHttpClientConnectionManager.releaseConnection method is never called
and the connection stays active. After the connection-timeout of the jetty
is reached, it closes the connection from the server-side and the TCP-state
switches into CLOSE_WAIT. The client never closes the connection and so the
number of connections in use rises up.
Currently we try to configure the jetty gzip module to return no
"Content-Length" if gzip-compression was used. We hope that in this case
another InputStream implementation is used that uses the NULL-terminator to
see when the InputStream was fully consumed.
Do you have any experiences with this problem or any suggestions for us?
Thanks,
Johannes
gzip.mod
-----
DO NOT EDIT - See:
https://www.eclipse.org/jetty/documentation/current/startup-modules.html
[description]
Enable GzipHandler for dynamic gzip compression
for the entire server.
[tags]
handler
[depend]
server
[xml]
etc/jetty-gzip.xml
[ini-template]
## Minimum content length after which gzip is enabled
jetty.gzip.minGzipSize=2048
## Check whether a file with *.gz extension exists
jetty.gzip.checkGzExists=false
## Gzip compression level (-1 for default)
jetty.gzip.compressionLevel=-1
## User agents for which gzip is disabled
jetty.gzip.excludedUserAgent=.*MSIE.6\.0.*
-----
jetty-gzip.xml
-----
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "
http://www.eclipse.org/jetty/configure_9_3.dtd";>
<!-- =============================================================== -->
<!-- Mixin the GZIP Handler -->
<!-- This applies the GZIP Handler to the entire server -->
<!-- If a GZIP handler is required for an individual context, then -->
<!-- use a context XML (see test.xml example in distribution) -->
<!-- =============================================================== -->
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<Call name="insertHandler">
<Arg>
<New id="GzipHandler"
class="org.eclipse.jetty.server.handler.gzip.GzipHandler">
<Set name="minGzipSize">
<Property name="jetty.gzip.minGzipSize"
deprecated="gzip.minGzipSize" default="2048" />
</Set>
<Set name="checkGzExists">
<Property name="jetty.gzip.checkGzExists"
deprecated="gzip.checkGzExists" default="false" />
</Set>
<Set name="compressionLevel">
<Property name="jetty.gzip.compressionLevel"
deprecated="gzip.compressionLevel" default="-1" />
</Set>
<Set name="inflateBufferSize">
<Property name="jetty.gzip.inflateBufferSize"
default="0" />
</Set>
<Set name="deflaterPoolCapacity">
<Property name="jetty.gzip.deflaterPoolCapacity"
default="-1" />
</Set>
<Set name="syncFlush">
<Property name="jetty.gzip.syncFlush" default="false" />
</Set>
<Set name="excludedAgentPatterns">
<Array type="String">
<Item>
<Property name="jetty.gzip.excludedUserAgent"
deprecated="gzip.excludedUserAgent" default=".*MSIE.6\.0.*" />
</Item>
</Array>
</Set>
<Set name="includedMethodList">
<Property name="jetty.gzip.includedMethodList"
default="GET,POST" />
</Set>
<Set name="excludedMethodList">
<Property name="jetty.gzip.excludedMethodList"
default="" />
</Set>
</New>
</Arg>
</Call>
</Configure>
-----
