PerMessageDeflate - Native Memory Pressure

[Sridhar Rao]

Hi All,

We are using tomcat 8.5.47 with Java8. Recently we were seeing a lot of
nodes getting restarted because of insufficient memory. Those nodes seem to
be under native memory pressure. After doing some analysis, It appears that
the native memory pressure is caused by the number of WebSocket
connections. It appears that by default tomcat adds "permessage-deflate"
extension. And each WebSocket connection seem to have an instance of
PerMessageDeflate object, which intern has an instance of a Deflater object.

Since each Deflater object is creating native memory buffers we seem to be
running into this issue. As a quick test, we wrote a simple test which
creates 5000 Deflater objects. It increases process memory to ~1G. i.e. If
we have 5000 WebSocket connections, that is 1G.


import java.io.IOException;
import java.util.zip.Deflater;
public class DeflatorTest {
   public static void main(String[] args) throws IOException {
      String content = "blahblahblah";
      byte[] input = content.getBytes("UTF-8");
      System.out.println("Test size = " + 5000);
      for (int i = 0; i < 5000; i++) {
         byte[] output = new byte[500];
         Deflater deflater = new Deflater(-1, true);
      }
      System.out.println("Done");
      //Let it run for a while so I can monitor the java process.
      try {
         Thread.sleep(60 * 60 * 1000);
      } catch (InterruptedException ex) {
      }
   }
}

So we are exploring disabling "permessage-deflate" extension. Although it
could increase network overhead, it should help with native memory and CPU
cycles. Most of our messages are small anyway. Just wanted to check, if
there are any potential gotcha's with disabling message deflater.

Thanks for the feedback
PS

Relevant parts of the tomcat code

Constants.java

static {
if (DISABLE_BUILTIN_EXTENSIONS) {
INSTALLED_EXTENSIONS = Collections.unmodifiableList(new ArrayList<Extension>
());
} else {
List<Extension> installed = new ArrayList<>(1);
installed.add(new WsExtension("permessage-deflate"));
INSTALLED_EXTENSIONS = Collections.unmodifiableList(installed);
}
}
 PerMessageDeflate.java
private final Deflater deflater = new Deflater(Deflater.DEFAULT_COMPRESSION,
true);