Hi Joakim
Thanks for the quick reply. Please have another look at my demo https://github.com/softappeal/yass/blob/master/src/test/java/ch/softappeal/yass/transport/ws/test/JettyWebSocketLeak.java . I enhanced it slightly: - only explicitly closing client session (with Session.close()) - added sleeps - client session closing is now asynchronous If you look at the program output, you see that: - the client connects to the server, client and server side sessions will be created: >>> opening server session 678919052 >>> opening client session 342587933 - after 5s, the client session is closed with Session.close() and then the Endpoint.onClose callbacks for both session are called: >>> closing client session 342587933 >>> client session 342587933 closed with CloseReason[1000] >>> server session 678919052 closed with CloseReason[1000] - after 10s, the leaked server session is printed: >>> printing leaked server sessions ... >>> session 678919052 I'm quite sure that I use the Java WebSocket API correctly. The program output seems to show this. I also programmed the same demo for Undertow/Wildfly https://github.com/softappeal/yass/blob/master/src/test/java/ch/softappeal/yass/transport/ws/test/UndertowWebSocketNoLeak.java . It also behaves the same. Please notice that I configure the endpoints using the interface-driven approach and not the annotation-driven one. You used the annotation-driven approach in your example. I believe that its quite clear that there is really a memory leak in Jetty using the interface-driven approach. The correctly closed JsrSessions will never be removed from the WebSocketServerFactory._beans (removeBean() is never called); even after days. In fact, we have productive servers crashing after a couple of days with OutOfMemoryErrors. The dumps show tens of thousands of correctly closed JsrSessions in WebSocketServerFactory._beans. No JsrSession is ever removed. In our productive setup the clients are browsers connecting with WebSockets. I hope that you agree that this is really a bug. If so, could you please open an issue? Regards, Angelo ------------------------------------------------------------------------------------------------------------------------ package ch.softappeal.yass.transport.ws.test; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; import org.eclipse.jetty.servlet.ServletContextHandler; import org.eclipse.jetty.websocket.jsr356.ClientContainer; import org.eclipse.jetty.websocket.jsr356.JsrSession; import org.eclipse.jetty.websocket.jsr356.server.ServerContainer; import org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer; import org.eclipse.jetty.websocket.server.WebSocketServerFactory; import javax.websocket.ClientEndpointConfig; import javax.websocket.CloseReason; import javax.websocket.Endpoint; import javax.websocket.EndpointConfig; import javax.websocket.Extension; import javax.websocket.HandshakeResponse; import javax.websocket.Session; import javax.websocket.server.HandshakeRequest; import javax.websocket.server.ServerEndpointConfig; import java.net.URI; import java.util.ArrayList; import java.util.List; import java.util.concurrent.TimeUnit; public class JettyWebSocketLeak { public static class WsConfigurator extends ServerEndpointConfig.Configurator { private final String side; public WsConfigurator(String side) { this.side = side; } @SuppressWarnings("unchecked") @Override public <T> T getEndpointInstance(Class<T> endpointClass) { return (T)new Endpoint() { @Override public void onOpen(Session session, EndpointConfig config) { System.out.println("opening " + side + " session " + session.hashCode()); if ("client".equals(side)) { new Thread(() -> { try { TimeUnit.SECONDS.sleep(5); System.out.println(); System.out.println("closing " + side + " session " + session.hashCode()); session.close(); } catch (Exception e) { throw new RuntimeException(e); } }).start(); } } @Override public void onClose(Session session, CloseReason closeReason) { System.out.println(side + " session " + session.hashCode() + " closed with " + closeReason); } @Override public void onError(Session session, Throwable throwable) { } }; } @Override public String getNegotiatedSubprotocol(List<String> supported, List<String> requested) { return ""; } @Override public List<Extension> getNegotiatedExtensions(List<Extension> installed, List<Extension> requested) { return new ArrayList<>(); } @Override public boolean checkOrigin(String originHeaderValue) { return true; } @Override public void modifyHandshake(ServerEndpointConfig sec, HandshakeRequest request, HandshakeResponse response) { } } public static int PORT = 9090; public static String PATH = "/test"; public static URI THE_URI = URI.create("ws://localhost:" + PORT + PATH); public static void main(String... args) throws Exception { Server server = new Server(); ServerConnector serverConnector = new ServerConnector(server); serverConnector.setPort(PORT); server.addConnector(serverConnector); ServletContextHandler servletContextHandler = new ServletContextHandler(ServletContextHandler.NO_SESSIONS); servletContextHandler.setContextPath("/"); server.setHandler(servletContextHandler); ServerContainer serverContainer = WebSocketServerContainerInitializer.configureContext(servletContextHandler); serverContainer.addEndpoint( ServerEndpointConfig.Builder.create(Endpoint.class, PATH).configurator(new WsConfigurator("server")).build() ); server.start(); ClientContainer clientContainer = new ClientContainer(); clientContainer.start(); clientContainer.connectToServer( new WsConfigurator("client").getEndpointInstance(Endpoint.class), ClientEndpointConfig.Builder.create().build(), THE_URI ); TimeUnit.SECONDS.sleep(10); System.out.println(); System.out.println("printing leaked server sessions ..."); serverContainer.getBeans(WebSocketServerFactory.class).forEach( factory -> factory.getBeans(JsrSession.class).forEach( session -> System.out.println(" session " + session.hashCode()) ) ); /* program output: opening server session 678919052 opening client session 342587933 closing client session 342587933 client session 342587933 closed with CloseReason[1000] server session 678919052 closed with CloseReason[1000] printing leaked server sessions ... session 678919052 */ } } ------------------------------------------------------------------------------------------------------------------------ Von: [email protected] [mailto:[email protected]] Im Auftrag von Joakim Erdfelt Gesendet: Dienstag, 12. Januar 2016 17:14 An: JETTY user mailing list Betreff: [EXTERN] Re: [jetty-users] Memory leak with websockets Here's yours modified to wait for close before testing the list of beans. https://gist.github.com/joakime/1abe75ca14bd05a747b0 The output of that I get on my Linux machine ... java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment java.awt.printerjob = sun.print.PSPrinterJob java.class.path = /home/joakim/code/jetty/Playground/target/classes:/home/joakim/.m2/repository/org/jboss/weld/servlet/weld-servlet/2.2.9.Final/weld-servlet-2.2.9.Final.jar:/home/joakim/.m2/repository/javax/websocket/javax.websocket-api/1.0/javax.websocket-api-1.0.jar:/home/joakim/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-deploy/9.3.7.RC0/jetty-deploy-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-xml/9.3.7.RC0/jetty-xml-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-http/9.3.7.RC0/jetty-http-9.3.7.RC0-tests.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-util/9.3.7.RC0/jetty-util-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-webapp/9.3.7.RC0/jetty-webapp-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-servlet/9.3.7.RC0/jetty-servlet-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-servlets/9.3.7.RC0/jetty-servlets-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-continuation/9.3.7.RC0/jetty-continuation-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-http/9.3.7.RC0/jetty-http-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-io/9.3.7.RC0/jetty-io-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-annotations/9.3.7.RC0/jetty-annotations-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-plus/9.3.7.RC0/jetty-plus-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-jndi/9.3.7.RC0/jetty-jndi-9.3.7.RC0.jar:/home/joakim/.m2/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/home/joakim/.m2/repository/org/ow2/asm/asm/5.0.1/asm-5.0.1.jar:/home/joakim/.m2/repository/org/ow2/asm/asm-commons/5.0.1/asm-commons-5.0.1.jar:/home/joakim/.m2/repository/org/ow2/asm/asm-tree/5.0.1/asm-tree-5.0.1.jar:/home/joakim/.m2/repository/org/eclipse/jetty/websocket/javax-websocket-server-impl/9.3.7.RC0/javax-websocket-server-impl-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/websocket/javax-websocket-client-impl/9.3.7.RC0/javax-websocket-client-impl-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/websocket/websocket-client/9.3.7.RC0/websocket-client-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/websocket/websocket-server/9.3.7.RC0/websocket-server-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/websocket/websocket-common/9.3.7.RC0/websocket-common-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/websocket/websocket-api/9.3.7.RC0/websocket-api-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/websocket/websocket-servlet/9.3.7.RC0/websocket-servlet-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-rewrite/9.3.7.RC0/jetty-rewrite-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-server/9.3.7.RC0/jetty-server-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-client/9.3.7.RC0/jetty-client-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-proxy/9.3.7.RC0/jetty-proxy-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-servlet/9.3.7.RC0/jetty-servlet-9.3.7.RC0-tests.jar:/home/joakim/.m2/repository/org/eclipse/jetty/jetty-security/9.3.7.RC0/jetty-security-9.3.7.RC0.jar:/home/joakim/.m2/repository/org/eclipse/jetty/toolchain/jetty-test-helper/2.9/jetty-test-helper-2.9.jar:/home/joakim/.m2/repository/junit/junit/4.11/junit-4.11.jar:/home/joakim/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/home/joakim/.m2/repository/org/hamcrest/hamcrest-library/1.3/hamcrest-library-1.3.jar java.class.version = 52.0 java.endorsed.dirs = /home/joakim/java/jvm/jdk-8u60/jre/lib/endorsed java.ext.dirs = /home/joakim/java/jvm/jdk-8u60/jre/lib/ext:/usr/java/packages/lib/ext java.home = /home/joakim/java/jvm/jdk-8u60/jre java.io.tmpdir = /tmp java.library.path = /usr/lib/jni:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.8.0_60-b27 java.specification.name = Java Platform API Specification java.specification.vendor = Oracle Corporation java.specification.version = 1.8 java.vendor = Oracle Corporation java.vendor.url = http://java.oracle.com/ java.vendor.url.bug = http://bugreport.sun.com/bugreport/ java.version = 1.8.0_60 java.vm.info = mixed mode java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.name = Java Virtual Machine Specification java.vm.specification.vendor = Oracle Corporation java.vm.specification.version = 1.8 java.vm.vendor = Oracle Corporation java.vm.version = 25.60-b23 sun.arch.data.model = 64 sun.boot.class.path = /home/joakim/java/jvm/jdk-8u60/jre/lib/resources.jar:/home/joakim/java/jvm/jdk-8u60/jre/lib/rt.jar:/home/joakim/java/jvm/jdk-8u60/jre/lib/sunrsasign.jar:/home/joakim/java/jvm/jdk-8u60/jre/lib/jsse.jar:/home/joakim/java/jvm/jdk-8u60/jre/lib/jce.jar:/home/joakim/java/jvm/jdk-8u60/jre/lib/charsets.jar:/home/joakim/java/jvm/jdk-8u60/jre/lib/jfr.jar:/home/joakim/java/jvm/jdk-8u60/jre/classes sun.boot.library.path = /home/joakim/java/jvm/jdk-8u60/jre/lib/amd64 sun.cpu.endian = little sun.cpu.isalist = sun.desktop = gnome sun.io.unicode.encoding = UnicodeLittle sun.java.command = jetty.websocket.JettyWebSocketLeak sun.java.launcher = SUN_STANDARD sun.jnu.encoding = UTF-8 sun.management.compiler = HotSpot 64-Bit Tiered Compilers sun.os.patch.level = unknown user.country = US user.dir = /home/joakim/code/jetty/Playground user.home = /home/joakim user.language = en user.name = joakim user.timezone = 2016-01-12 09:07:21.879:INFO::main: Logging initialized @182ms 2016-01-12 09:07:21.979:INFO:oejs.Server:main: jetty-9.3.7.RC0 2016-01-12 09:07:22.228:INFO:oejsh.ContextHandler:main: Started o.e.j.s.ServletContextHandler@62bd765{/,null,AVAILABLE} 2016-01-12 09:07:22.236:INFO:oejs.ServerConnector:main: Started ServerConnector@351d0846{HTTP/1.1,[http/1.1]}{0.0.0.0:9090} 2016-01-12 09:07:22.237:INFO:oejs.Server:main: Started @541ms printing the leaked server side sessions ... 2016-01-12 09:07:22.353:INFO:oejs.ServerConnector:main: Stopped ServerConnector@351d0846{HTTP/1.1,[http/1.1]}{0.0.0.0:9090} 2016-01-12 09:07:22.354:INFO:oejsh.ContextHandler:main: Stopped o.e.j.s.ServletContextHandler@62bd765{/,null,UNAVAILABLE} If I get rid of the 2 lines ... cli1.waitForClose(); cli2.waitForClose(); Then I occasionally (its not 100% of the time) see some (still open) WebSocketSession's lingering as beans on the WebSocketServerFactory. Joakim Erdfelt / [email protected] On Tue, Jan 12, 2016 at 8:54 AM, Joakim Erdfelt <[email protected]> wrote: You are testing for beans against client connections that are still actively connected. If you change this to actually wait for the clients to register an onClose() before testing the server side beans list you'll see its empty. Merely issuing clientContainer.stop() starts the close handshake, abruptly, and abnormally, not giving the server a chance to complete the close handshake. example: https://github.com/eclipse/jetty.project/blob/master/jetty-websocket/websocket-client/src/test/java/org/eclipse/jetty/websocket/client/SessionTest.java#L98-L107 Joakim Erdfelt / [email protected] On Tue, Jan 12, 2016 at 8:33 AM, Salvadè Angelo <[email protected]> wrote: Hi I have the same problem. Please have a look at https://github.com/softappeal/yass/blob/master/src/test/java/ch/softappeal/yass/transport/ws/test/JettyWebSocketLeak.java . It shows the leak. Regards, Angelo --------------------------------------------------------------------------------------------------------------------------------------------- package ch.softappeal.yass.transport.ws.test; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; import org.eclipse.jetty.servlet.ServletContextHandler; import org.eclipse.jetty.websocket.jsr356.ClientContainer; import org.eclipse.jetty.websocket.jsr356.JsrSession; import org.eclipse.jetty.websocket.jsr356.server.ServerContainer; import org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer; import org.eclipse.jetty.websocket.server.WebSocketServerFactory; import javax.websocket.ClientEndpointConfig; import javax.websocket.CloseReason; import javax.websocket.Endpoint; import javax.websocket.EndpointConfig; import javax.websocket.Extension; import javax.websocket.HandshakeResponse; import javax.websocket.Session; import javax.websocket.server.HandshakeRequest; import javax.websocket.server.ServerEndpointConfig; import java.io.IOException; import java.net.URI; import java.util.ArrayList; import java.util.List; public class JettyWebSocketLeak { public static class WsConfigurator extends ServerEndpointConfig.Configurator { @SuppressWarnings("unchecked") @Override public <T> T getEndpointInstance(Class<T> endpointClass) { return (T)new Endpoint() { @Override public void onOpen(Session session, EndpointConfig config) { try { System.out.println("closing " + session); session.close(); } catch (IOException e) { throw new RuntimeException(e); } } @Override public void onClose(Session session, CloseReason closeReason) { } @Override public void onError(Session session, Throwable throwable) { } }; } public Endpoint getEndpointInstance() { return getEndpointInstance(null); } @Override public String getNegotiatedSubprotocol(List<String> supported, List<String> requested) { for (String r : requested) { if (supported.contains(r)) { return r; } } return ""; } @Override public List<Extension> getNegotiatedExtensions(List<Extension> installed, List<Extension> requested) { List<Extension> extensions = new ArrayList<>(requested.size()); for (Extension r : requested) { for (Extension i : installed) { if (i.getName().equals(r.getName())) { extensions.add(r); break; } } } return extensions; } @Override public boolean checkOrigin(String originHeaderValue) { return true; } @Override public void modifyHandshake(ServerEndpointConfig sec, HandshakeRequest request, HandshakeResponse response) { } } private static int PORT = 9090; private static String PATH = "/test"; private static URI THE_URI = URI.create("ws://localhost:" + PORT + PATH); private static void connectClient() throws Exception { ClientContainer clientContainer = new ClientContainer(); clientContainer.start(); clientContainer.connectToServer(new WsConfigurator().getEndpointInstance(), ClientEndpointConfig.Builder.create().build(), THE_URI); clientContainer.stop(); } public static void main(String... args) throws Exception { Server server = new Server(); ServerConnector serverConnector = new ServerConnector(server); serverConnector.setPort(PORT); server.addConnector(serverConnector); ServletContextHandler servletContextHandler = new ServletContextHandler(ServletContextHandler.NO_SESSIONS); servletContextHandler.setContextPath("/"); server.setHandler(servletContextHandler); ServerContainer serverContainer = WebSocketServerContainerInitializer.configureContext(servletContextHandler); serverContainer.addEndpoint(ServerEndpointConfig.Builder.create(Endpoint.class, PATH).configurator(new WsConfigurator()).build()); server.start(); connectClient(); connectClient(); System.out.println("printing the leaked server side sessions ..."); serverContainer.getBeans(WebSocketServerFactory.class).forEach(factory -> factory.getBeans(JsrSession.class).forEach(System.out::println)); } } ________________________________ Disclaimer: Aufgrund der bisherigen E-Mail-Korrespondenz bzw. der getroffenen Absprache, betrachtet sich die Zürcher Kantonalbank als berechtigt, mit Ihnen per E-Mail zu kommunizieren. Die Zürcher Kantonalbank geht davon aus, dass Sie die Risiken von E-Mails kennen und in Kauf nehmen (insbesondere fehlende Vertraulichkeit, Manipulation oder Missbrauch durch Dritte, Fehlleitung, verzögerte Übermittlung oder Bearbeitung, Viren, etc.). Die Zürcher Kantonalbank lehnt jede Haftung für Schäden im Zusammenhang mit der Verwendung von E-Mails ab, sofern die Bank die geschäftsübliche Sorgfalt nicht verletzt hat. E-Mails werden nur während den üblichen Geschäftszeiten der Bank bearbeitet. Sie können nicht von der sofortigen Kenntnisnahme Ihrer E-Mails ausgehen. E-Mail eignet sich daher nicht für die Übermittlung von Handelsaufträgen und wertverschiebenden oder zeitkritischen Aufträgen (z.B. Zahlungsaufträge). Sollten Sie dieses E-Mail irrtümlicherweise erhalten haben, bitten wir Sie, das E-Mail an die Zürcher Kantonalbank zurückzusenden und das E-Mail anschliessend mitsamt allen Anhängen von ihrem System zu löschen. Der Gebrauch der im E-Mail enthaltenen Information ist verboten. Based on previous e-mail correspondence or an arrangement we have reached with you, Zürcher Kantonalbank considers itself to be entitled to communicate with you via e-mail. Zürcher Kantonalbank assumes that you know the risks associated with e-mails and that you accept them (in particular, the lack of confidentiality, manipulation or misuse by third parties, misdirection, delayed transmission or processing, viruses, etc.). Zürcher Kantonalbank accepts no liability whatsoever for damage caused in connection with the use of e-mail, provided that the Bank has not failed to exercise customary due care. E-mails are processed only during the Bank’s normal business hours. You cannot assume that your e-mails will be read immediately. E-mails are therefore not suitable for sending trading orders and orders that change the value of an account or are time-critical (e.g. payment orders). If you have received this e-mail in error, please respond to Zürcher Kantonalbank and then delete this e-mail and your response together with all attachments from your system. The use of the information contained in the e-mail is prohibited. _______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users ________________________________ Disclaimer: Aufgrund der bisherigen E-Mail-Korrespondenz bzw. der getroffenen Absprache, betrachtet sich die Zürcher Kantonalbank als berechtigt, mit Ihnen per E-Mail zu kommunizieren. Die Zürcher Kantonalbank geht davon aus, dass Sie die Risiken von E-Mails kennen und in Kauf nehmen (insbesondere fehlende Vertraulichkeit, Manipulation oder Missbrauch durch Dritte, Fehlleitung, verzögerte Übermittlung oder Bearbeitung, Viren, etc.). Die Zürcher Kantonalbank lehnt jede Haftung für Schäden im Zusammenhang mit der Verwendung von E-Mails ab, sofern die Bank die geschäftsübliche Sorgfalt nicht verletzt hat. E-Mails werden nur während den üblichen Geschäftszeiten der Bank bearbeitet. Sie können nicht von der sofortigen Kenntnisnahme Ihrer E-Mails ausgehen. E-Mail eignet sich daher nicht für die Übermittlung von Handelsaufträgen und wertverschiebenden oder zeitkritischen Aufträgen (z.B. Zahlungsaufträge). Sollten Sie dieses E-Mail irrtümlicherweise erhalten haben, bitten wir Sie, das E-Mail an die Zürcher Kantonalbank zurückzusenden und das E-Mail anschliessend mitsamt allen Anhängen von ihrem System zu löschen. Der Gebrauch der im E-Mail enthaltenen Information ist verboten. Based on previous e-mail correspondence or an arrangement we have reached with you, Zürcher Kantonalbank considers itself to be entitled to communicate with you via e-mail. Zürcher Kantonalbank assumes that you know the risks associated with e-mails and that you accept them (in particular, the lack of confidentiality, manipulation or misuse by third parties, misdirection, delayed transmission or processing, viruses, etc.). Zürcher Kantonalbank accepts no liability whatsoever for damage caused in connection with the use of e-mail, provided that the Bank has not failed to exercise customary due care. E-mails are processed only during the Bank’s normal business hours. You cannot assume that your e-mails will be read immediately. E-mails are therefore not suitable for sending trading orders and orders that change the value of an account or are time-critical (e.g. payment orders). If you have received this e-mail in error, please respond to Zürcher Kantonalbank and then delete this e-mail and your response together with all attachments from your system. The use of the information contained in the e-mail is prohibited. _______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
