Re: remote jmx monitoring through ssh tunnel
Chris‘, > Am 10.12.2019 um 18:59 schrieb Chris Cheshire : > > On Tue, Dec 10, 2019 at 11:58 AM Chris Cheshire wrote: >> >>> On Tue, Dec 10, 2019 at 9:42 AM Christopher Schultz >>> wrote: >>> >>> -BEGIN PGP SIGNED MESSAGE- >>> Hash: SHA256 >>> >>> Chris, >>> >>> On 12/9/19 17:10, Chris Cheshire wrote: In CATALINA_BASE/bin/setenv.sh I have the following : CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" >>> >>> Okay. >>> In CATALINA_BASE/conf/server.xml I have a listener configured : >>> className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" useLocalPorts="true" /> Upon startup I see in logs : INFO [main] org.apache.catalina.mbeans.JmxRemoteLifecycleListener.createServer The JMX Remote Listener has configured the registry on port [10001] and the server on port [10002] for the [Platform] server I didn‘t read it anywhere. Did you add the catalina-jmx.jar to the classpath? Peter $ netstat -an | grep 10001 tcp4 0 0 127.0.0.1.10001 *.*LISTEN tcp6 0 0 ::1.10001 *.*LISTEN On my local machine I have a tunnel set up as follows : ssh -N -L10001:localhost:10001 -L10002:localhost:10002 user@remotehost (where user is the user tomcat is running under) When I try to add a remote JMX connection in VisualVM on my client machine to localhost:10001 I get an error dialog after a brief delay with the message "Cannot connect to localhost:10001 using service:jmx:rmi:///jndi/rmi://localhost:10001/jmxrmi". If I change it to port 10002 I get the same error. On the server at this time : $ netstat -an | grep 10001 tcp4 0 0 127.0.0.1.10001 *.*LISTEN tcp6 0 0 ::1.10001 *.*LISTEN tcp4 0 0 127.0.0.1.62637 127.0.0.1.10001TIME_WAIT If I try to use jconsole connecting to port 10001 I get the error "Connection failed: non-JRMP server at remote endpoint". Connecting to port 10002 I get the error "Connection failed: no such object in table" >>> >>> You should be using the port defined by rmiRegistryPortPlatform, so >>> 10001 is the correct port to use. >>> I've been through the tomcat configuration documentation a couple times but I can't see what else I need to configure. >>> >>> What you have looks good to me without reproducing it myself. Can you do >>> : >>> >>> $ netstat -an | grep 1000[0-9] >>> >>> ? >>> >>> Just to be sure about both ports? >>> >> >> $ netstat -an | grep 1000[0-9] >> tcp6 0 0 :::10001:::*LISTEN >> tcp6 0 0 :::10002:::*LISTEN >> >> >> H. Tomcat is only listening on ipv6 ports, but my tunnel is using >> ipv4. After digging around [1], I added this to CATALINA_OPTS in >> setenv.sh >> >> -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true >> >> $ netstat -an | grep 1000[0-9] >> tcp0 0 0.0.0.0:10001 0.0.0.0:* LISTEN >> tcp0 0 0.0.0.0:10002 0.0.0.0:* LISTEN >> >> When I try to connect with jconsole I get the same error (non-JRMP >> server at remote endpoint), with the server showing >> >> tcp0 0 0.0.0.0:10001 0.0.0.0:* LISTEN >> tcp0 0 0.0.0.0:10002 0.0.0.0:* LISTEN >> tcp0 0 127.0.0.1:10001 127.0.0.1:43803 TIME_WAIT >> tcp0 0 127.0.0.1:10001 127.0.0.1:43815 TIME_WAIT >> >> >> I have also updated sshd_config with >> >> PermitTunnel yes >> >> and restarted that. Still no change. >> >> Chris >> >> >> [1] >> https://serverfault.com/questions/390840/how-does-one-get-tomcat-to-bind-to-ipv4-address > > > As a followup to take the tunnel out of the equation I downloaded > jmxterm [1] on the server and tried to connect > > > $ java -jar jmxterm-1.0.0-uber.jar > Welcome to JMX terminal. Type "help" for available commands. > $>open localhost:10001 > #RuntimeIOException: Runtime IO exception: Failed to retrieve > RMIServer stub: javax.naming.CommunicationException [Root exception is > java.rmi.ConnectIOException: non-JRMP server at remote endpoint] > $> > > > Back to the tomcat documentation, I added this to CATALINA_OPTS > (based on listener config and assumed defaults) > > -Dcom.sun.management.jmxremote.registry.ssl=false > > and now I get a different error : > $>open localhost:10001 > #RuntimeIOException: Runtime IO exception: Failed to retrieve > RMIServer stub: javax.naming.CommunicationException [Root exception is >
Re: Expected behavior of calling javax.servlet.ServletRequest#getInputStream after javax.servlet.http.HttpServletRequest#getPart
On 10.12.2019 15:31, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Behrang, On 12/8/19 05:18, Behrang Saeedzadeh wrote: If I call javax.servlet.ServletRequest#getInputStream after having called javax.servlet.http.HttpServletRequest#getPart, even without performing any operations on the given part, I am getting an empty stream (0-bytes). Is this in compliance with the spec? What does the servlet spec say about that situation? A bit in defense of the OP, I have to say that the Servlet Spec (4.0 final) does not seem extremely clear on that subject. In section "3.1.1 When Parameters Are Available", it /does/ say this : "If the conditions are met, post form data will no longer be available for reading directly from the request object’s input stream." But among those conditions that have to be met (just above that quote) is "4. The servlet has made an initial call of any of the "getParameter family" of methods on the request object." (So there, it talks about "the getParameter family", not the "getPart" ones). It only talks about the "getPart" methods in the next section "3.2 File upload", and in that section, all it says about the input stream is : "If the servlet container does not provide the multi-part/form-data processing, the data will be available through the HttpServletReuqest.getInputStream." So the way I read this, is that there is nothing that explicitly says that the InputStream is no longer available if you have called "getPart". (It is also in fact not very clear about what happens to the parameters, when the content-type of the Request is "multipart/form-data", which is only mentioned in section 3.2.) On the other hand, http://tomcat.apache.org/tomcat-9.0-doc/servletapi/index.html --> Interface HttpServletRequest says : Part getPart(java.lang.String name) throws java.io.IOException, ServletException Gets the named Part or null if the Part does not exist. *Triggers upload of all Parts*. (emphasis mine) which /may/ be taken to mean (and probably does) that it consumes the whole InputStream. But again, this may be lacking some overall clarity. In any case, the OP's question is not really unwarranted, unless there is some other explanation somewhere which I have missed. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: remote jmx monitoring through ssh tunnel
On Tue, Dec 10, 2019 at 11:58 AM Chris Cheshire wrote: > > On Tue, Dec 10, 2019 at 9:42 AM Christopher Schultz > wrote: > > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > Chris, > > > > On 12/9/19 17:10, Chris Cheshire wrote: > > > In CATALINA_BASE/bin/setenv.sh I have the following : > > > > > > CATALINA_OPTS="-Dcom.sun.management.jmxremote > > > -Dcom.sun.management.jmxremote.ssl=false > > > -Dcom.sun.management.jmxremote.authenticate=false" > > > > Okay. > > > > > In CATALINA_BASE/conf/server.xml I have a listener configured : > > > > > > > > className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" > > > rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" > > > useLocalPorts="true" /> > > > > > > > > > Upon startup I see in logs : INFO [main] > > > org.apache.catalina.mbeans.JmxRemoteLifecycleListener.createServer > > > The JMX Remote Listener has configured the registry on port > > > [10001] and the server on port [10002] for the [Platform] server > > > > > > > > > $ netstat -an | grep 10001 tcp4 0 0 127.0.0.1.10001 > > > *.*LISTEN tcp6 0 0 ::1.10001 > > > *.*LISTEN > > > > > > On my local machine I have a tunnel set up as follows : ssh -N > > > -L10001:localhost:10001 -L10002:localhost:10002 user@remotehost > > > > > > (where user is the user tomcat is running under) > > > > > > When I try to add a remote JMX connection in VisualVM on my client > > > machine to localhost:10001 I get an error dialog after a brief > > > delay with the message "Cannot connect to localhost:10001 using > > > service:jmx:rmi:///jndi/rmi://localhost:10001/jmxrmi". If I change > > > it to port 10002 I get the same error. On the server at this time > > > : $ netstat -an | grep 10001 tcp4 0 0 127.0.0.1.10001 > > > *.*LISTEN tcp6 0 0 ::1.10001 > > > *.*LISTEN tcp4 0 0 127.0.0.1.62637 > > > 127.0.0.1.10001TIME_WAIT > > > > > > > > > If I try to use jconsole connecting to port 10001 I get the error > > > "Connection failed: non-JRMP server at remote endpoint". Connecting > > > to port 10002 I get the error "Connection failed: no such object > > > in table" > > > > You should be using the port defined by rmiRegistryPortPlatform, so > > 10001 is the correct port to use. > > > > > I've been through the tomcat configuration documentation a couple > > > times but I can't see what else I need to configure. > > > > What you have looks good to me without reproducing it myself. Can you do > > : > > > > $ netstat -an | grep 1000[0-9] > > > > ? > > > > Just to be sure about both ports? > > > > $ netstat -an | grep 1000[0-9] > tcp6 0 0 :::10001:::*LISTEN > tcp6 0 0 :::10002:::*LISTEN > > > H. Tomcat is only listening on ipv6 ports, but my tunnel is using > ipv4. After digging around [1], I added this to CATALINA_OPTS in > setenv.sh > > -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true > > $ netstat -an | grep 1000[0-9] > tcp0 0 0.0.0.0:10001 0.0.0.0:* LISTEN > tcp0 0 0.0.0.0:10002 0.0.0.0:* LISTEN > > When I try to connect with jconsole I get the same error (non-JRMP > server at remote endpoint), with the server showing > > tcp0 0 0.0.0.0:10001 0.0.0.0:* LISTEN > tcp0 0 0.0.0.0:10002 0.0.0.0:* LISTEN > tcp0 0 127.0.0.1:10001 127.0.0.1:43803 TIME_WAIT > tcp0 0 127.0.0.1:10001 127.0.0.1:43815 TIME_WAIT > > > I have also updated sshd_config with > > PermitTunnel yes > > and restarted that. Still no change. > > Chris > > > [1] > https://serverfault.com/questions/390840/how-does-one-get-tomcat-to-bind-to-ipv4-address As a followup to take the tunnel out of the equation I downloaded jmxterm [1] on the server and tried to connect $ java -jar jmxterm-1.0.0-uber.jar Welcome to JMX terminal. Type "help" for available commands. $>open localhost:10001 #RuntimeIOException: Runtime IO exception: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint] $> Back to the tomcat documentation, I added this to CATALINA_OPTS (based on listener config and assumed defaults) -Dcom.sun.management.jmxremote.registry.ssl=false and now I get a different error : $>open localhost:10001 #RuntimeIOException: Runtime IO exception: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: error unmarshalling return; nested exception is: java.lang.ClassNotFoundException: org/apache/catalina/mbeans/JmxRemoteLifecycleListener$RmiClientLocalhostSocketFactory (no security manager: RMI class loader disabled)] So
Re: remote jmx monitoring through ssh tunnel
On Tue, Dec 10, 2019 at 9:42 AM Christopher Schultz wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Chris, > > On 12/9/19 17:10, Chris Cheshire wrote: > > In CATALINA_BASE/bin/setenv.sh I have the following : > > > > CATALINA_OPTS="-Dcom.sun.management.jmxremote > > -Dcom.sun.management.jmxremote.ssl=false > > -Dcom.sun.management.jmxremote.authenticate=false" > > Okay. > > > In CATALINA_BASE/conf/server.xml I have a listener configured : > > > > > className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" > > rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" > > useLocalPorts="true" /> > > > > > > Upon startup I see in logs : INFO [main] > > org.apache.catalina.mbeans.JmxRemoteLifecycleListener.createServer > > The JMX Remote Listener has configured the registry on port > > [10001] and the server on port [10002] for the [Platform] server > > > > > > $ netstat -an | grep 10001 tcp4 0 0 127.0.0.1.10001 > > *.*LISTEN tcp6 0 0 ::1.10001 > > *.*LISTEN > > > > On my local machine I have a tunnel set up as follows : ssh -N > > -L10001:localhost:10001 -L10002:localhost:10002 user@remotehost > > > > (where user is the user tomcat is running under) > > > > When I try to add a remote JMX connection in VisualVM on my client > > machine to localhost:10001 I get an error dialog after a brief > > delay with the message "Cannot connect to localhost:10001 using > > service:jmx:rmi:///jndi/rmi://localhost:10001/jmxrmi". If I change > > it to port 10002 I get the same error. On the server at this time > > : $ netstat -an | grep 10001 tcp4 0 0 127.0.0.1.10001 > > *.*LISTEN tcp6 0 0 ::1.10001 > > *.*LISTEN tcp4 0 0 127.0.0.1.62637 > > 127.0.0.1.10001TIME_WAIT > > > > > > If I try to use jconsole connecting to port 10001 I get the error > > "Connection failed: non-JRMP server at remote endpoint". Connecting > > to port 10002 I get the error "Connection failed: no such object > > in table" > > You should be using the port defined by rmiRegistryPortPlatform, so > 10001 is the correct port to use. > > > I've been through the tomcat configuration documentation a couple > > times but I can't see what else I need to configure. > > What you have looks good to me without reproducing it myself. Can you do > : > > $ netstat -an | grep 1000[0-9] > > ? > > Just to be sure about both ports? > $ netstat -an | grep 1000[0-9] tcp6 0 0 :::10001:::*LISTEN tcp6 0 0 :::10002:::*LISTEN H. Tomcat is only listening on ipv6 ports, but my tunnel is using ipv4. After digging around [1], I added this to CATALINA_OPTS in setenv.sh -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true $ netstat -an | grep 1000[0-9] tcp0 0 0.0.0.0:10001 0.0.0.0:* LISTEN tcp0 0 0.0.0.0:10002 0.0.0.0:* LISTEN When I try to connect with jconsole I get the same error (non-JRMP server at remote endpoint), with the server showing tcp0 0 0.0.0.0:10001 0.0.0.0:* LISTEN tcp0 0 0.0.0.0:10002 0.0.0.0:* LISTEN tcp0 0 127.0.0.1:10001 127.0.0.1:43803 TIME_WAIT tcp0 0 127.0.0.1:10001 127.0.0.1:43815 TIME_WAIT I have also updated sshd_config with PermitTunnel yes and restarted that. Still no change. Chris [1] https://serverfault.com/questions/390840/how-does-one-get-tomcat-to-bind-to-ipv4-address - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On Tue, Dec 10, 2019 at 4:36 PM Mark Thomas wrote: > On 10/12/2019 14:27, Christopher Schultz wrote: > > > > > Would using org.apache.catalina.connector.RECYCLE_FACADES=true have > > made this problem go away? Or would the behavior have been the same, > > just less dangerous? > > I think it would have triggered some NPEs in the background thread. > > > I'm wondering if Tomcat could or should have another safety feature to > > help catch this sort of thing in development. In all my development > > environments, I have the JDBC connection pool size set to a fixed > > maximum of 1 connection. This means that any potential deadlocks in > > the application due to sloppy connection-management will cause pretty > > early because we'll get pool-fetch timeouts, missing-return-connection > > errors, etc. > > > > Request object reuse has a measurable positive effect on performance > > in production, > > It would be worth confirming that is still the case for the Request and > Response objects. I suspect it is but it would be good to get some > recent, hard data. > It would generate more GC per request and it is measurably slower on a best case scenario with a static file (the worst would be starting to use charsets and char input/output, or "large" Servlet buffers - the default is 8KB but people often increase it). Quick test: add request = null; line 305 of CoyoteAdapter. -10% with ab -k on tomcat.gif > > > but in development probably doesn't matter quite so > > much. In the same way that WebappClassLoader becomes inert when the > > application has been stopped, perhaps we could "shut-down" request / > > response / session objects that have been loaned to a > > request-processor thread. > > > > Something like this: > > I'm not sure why we need this over and above the RequestFacade object. > Rémy
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 10/12/2019 14:27, Christopher Schultz wrote: > Would using org.apache.catalina.connector.RECYCLE_FACADES=true have > made this problem go away? Or would the behavior have been the same, > just less dangerous? I think it would have triggered some NPEs in the background thread. > I'm wondering if Tomcat could or should have another safety feature to > help catch this sort of thing in development. In all my development > environments, I have the JDBC connection pool size set to a fixed > maximum of 1 connection. This means that any potential deadlocks in > the application due to sloppy connection-management will cause pretty > early because we'll get pool-fetch timeouts, missing-return-connection > errors, etc. > > Request object reuse has a measurable positive effect on performance > in production, It would be worth confirming that is still the case for the Request and Response objects. I suspect it is but it would be good to get some recent, hard data. > but in development probably doesn't matter quite so > much. In the same way that WebappClassLoader becomes inert when the > application has been stopped, perhaps we could "shut-down" request / > response / session objects that have been loaned to a > request-processor thread. > > Something like this: I'm not sure why we need this over and above the RequestFacade object. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: http2 async timeout setting
On 10/12/2019 11:23, Arief Hasani wrote: > Hi Mark, > Very much appreciated. Fixed in: - master for 9.0.31 onwards - 8.5.x for 8.5.51 onwards 7.0.x was not affected. Mark > Thanks > On Monday, December 9, 2019, 05:59:58 PM GMT+7, Mark Thomas > wrote: > > On 07/12/2019 03:46, Arief Hasani wrote: >> Hi Chris, >> Thanks for the reminder. following is the code that runs the timeout >> listener on time while running on http1.1 but not on http2. tested on 9.0.29 > > > > I can reproduce this. I'm working on a fix now. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: remote jmx monitoring through ssh tunnel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chris, On 12/9/19 17:10, Chris Cheshire wrote: > In CATALINA_BASE/bin/setenv.sh I have the following : > > CATALINA_OPTS="-Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false" Okay. > In CATALINA_BASE/conf/server.xml I have a listener configured : > > className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" > rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" > useLocalPorts="true" /> > > > Upon startup I see in logs : INFO [main] > org.apache.catalina.mbeans.JmxRemoteLifecycleListener.createServer > The JMX Remote Listener has configured the registry on port > [10001] and the server on port [10002] for the [Platform] server > > > $ netstat -an | grep 10001 tcp4 0 0 127.0.0.1.10001 > *.*LISTEN tcp6 0 0 ::1.10001 > *.*LISTEN > > On my local machine I have a tunnel set up as follows : ssh -N > -L10001:localhost:10001 -L10002:localhost:10002 user@remotehost > > (where user is the user tomcat is running under) > > When I try to add a remote JMX connection in VisualVM on my client > machine to localhost:10001 I get an error dialog after a brief > delay with the message "Cannot connect to localhost:10001 using > service:jmx:rmi:///jndi/rmi://localhost:10001/jmxrmi". If I change > it to port 10002 I get the same error. On the server at this time > : $ netstat -an | grep 10001 tcp4 0 0 127.0.0.1.10001 > *.*LISTEN tcp6 0 0 ::1.10001 > *.*LISTEN tcp4 0 0 127.0.0.1.62637 > 127.0.0.1.10001TIME_WAIT > > > If I try to use jconsole connecting to port 10001 I get the error > "Connection failed: non-JRMP server at remote endpoint". Connecting > to port 10002 I get the error "Connection failed: no such object > in table" You should be using the port defined by rmiRegistryPortPlatform, so 10001 is the correct port to use. > I've been through the tomcat configuration documentation a couple > times but I can't see what else I need to configure. What you have looks good to me without reproducing it myself. Can you do : $ netstat -an | grep 1000[0-9] ? Just to be sure about both ports? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3vrsQACgkQHPApP6U8 pFjccg//RYofm1FSliR3L5WEqeyxfVrxD/w7HwASLCo61yLeJHz9bRcavv78wdlJ tQAyWtZMUkQlnct5398qLXrrwhUrmmkqKwn5qnKW5z2zwx7h53dTYgP6Zx/pDyRa 7zsCLtPulkb0kf+/5Ok7c43Qt8KLu1dEbQdjNuUKthHpESWW3yhNtH/QjEjIxU+l L0A8cNGq5XhvBgohCoDLdrqZLSBoqbWIOaunhfz2VBzzpqjDzJLdB9nBy8KRpxXk viSa4cMYwdc53sim0UZWm9HP58G1ZD/ZAi+2ES9BiqZqmP5Gbf+2a5ulyss1SYVP rfKg/dPAxX0BNVIwAYwxKdtm9kphgKwqHrh4NRydljqHNyaa4Hx8ncLFSFAVpCIN u57HIxAjX3LOtPAM0GON8ZnU4NVzZNtNb5y3gxtDHRBAurg4jIfufh08baLCBHmE BVR+WORNhxy6h8t1N+lyr9GDNraPyN7nEa9fLf679UilYUBr2UnCjL7f0UbH77sZ 32b/roteai1veoiw7+xiffV0Mb1+s86VfRxN5fuRhub2tko6uWY5PvDp5oVLZgKj vfF1EIQZPsoqSkc2d86M/0PsOMLWajeGAZ3TWRyrrQqZJEVBsopNWfECCawhIECT 7mQwQpk0AQAvBy3m+WcOkLD6Iva+xzMG3dN/azUILownSkGjZso= =QWRH -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Expected behavior of calling javax.servlet.ServletRequest#getInputStream after javax.servlet.http.HttpServletRequest#getPart
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Behrang, On 12/8/19 05:18, Behrang Saeedzadeh wrote: > If I call javax.servlet.ServletRequest#getInputStream after having > called javax.servlet.http.HttpServletRequest#getPart, even without > performing any operations on the given part, I am getting an empty > stream (0-bytes). > > Is this in compliance with the spec? What does the servlet spec say about that situation? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3vrCgACgkQHPApP6U8 pFgXaRAAyIjSDXvbXJvqgKnjRoZch/OnPP8gKviPywjH+aOtI1pzDz/U1XoCWaba GJJfw12LLrX/NcbtYG5hADQLnayn26Xi203F6TPG7PckHzwIp8UbhWa5qI/QNIeu JRM1P7LZgsBVMX+we0ziXAwt1UtWIGZp6gU0V6z3zg0V2DSsEcu/zuRzqJqKweVH fCmkJk2i6RInhw1eeEa4B6eRuAEl69WKPuddpJwNFdzj3hbAy0yMdnK/K9v/60BQ igxdlITy7xL6V9tT9/DDXS07DByLop2bbectiMJ7J+6j7PksZygneGf4gPjRiDPw eH2r6SloPN+sZGy7ttYqLI3/eMKKd8ad5s3+5ArnDgmBp/95y+pgaBUStUQQrycc mcywbl2cxPF+7hv4vMquYIi24uQjQcBpSVKuiFllO/ThP2NDgII8kZ7pNBfOUJQd pRDv4jwmpgh5ozm5dYfysLZ+fyxakkY0I1PnUSsJ7j0FEU9TT3+5UvbNjyPmPr0/ 7dfHiiM+wLOQ9b1vc1wISwzZ1tgS2sLUU9js4RlpDXp5Ru5vIgREAF1fO3Za2MBT hoRGbL4eAdzkAGPfMMYwCLr36FXv+Ra6KBScWd0V2u8cehz0nIp+QpyNrFXhdaK6 HQJLkaNmC/rY7Aw4nQ/AEIbT4a5ECUa4kHtimWYDse/I14zvYG4= =ISFn -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Mark,
On 12/10/19 03:26, Mark Thomas wrote:
> On 09/12/2019 19:54, Jerry Malcolm wrote:
>
>
>
>> Thanks to all of the responders on this problem. From the
>> beginning of this problem, I was convinced that I was doing
>> something bad wrong to cause this. I think now I have finally
>> identified the 'bad thing'. Periodically the model code that runs
>> in the request determines that I need to spin off another thread
>> to do some background work. That thread needs a few parameters
>> from the request object. So I pass the request object into the
>> thread object. I realize now that doing that is a horrible thing
>> to do. I was not aware until yesterday that request objects are
>> pooled and recycled. So I figured holding onto the request
>> object a little longer before it is discarded for that thread to
>> use was not going to be a problem. Obviously, that was
>> incorrect.
>
> Yep, that'll do it.
>
>> From what I can deduce, the main request thread finishes and
>> returns. The request object recycle code runs and the object is
>> returned to the pool. However, then the async thread later gets
>> a parm and must apparently still be able to use the request
>> object even though it's back in the pool. The
>> 'parametersProcessed' gets set to true. Then the next request
>> comes in, and that request object is assigned with
>> parametersProcessed=true.
>
> I think this analysis is correct.
>
>> I'm assuming the fix is to get everything I need out of the
>> request object and pass those parm values to the thread before
>> returning instead of passing the request object itself to the
>> thread.
>
> Correct.
>
>> Does all of this sound plausible as the source for this. The
>> good news is that I'm still teachable... :-)
>
> Very plausible.
>
> Kudos for figuring this out so quickly - and for the very clear
> write up.
Would using org.apache.catalina.connector.RECYCLE_FACADES=true have
made this problem go away? Or would the behavior have been the same,
just less dangerous?
I'm wondering if Tomcat could or should have another safety feature to
help catch this sort of thing in development. In all my development
environments, I have the JDBC connection pool size set to a fixed
maximum of 1 connection. This means that any potential deadlocks in
the application due to sloppy connection-management will cause pretty
early because we'll get pool-fetch timeouts, missing-return-connection
errors, etc.
Request object reuse has a measurable positive effect on performance
in production, but in development probably doesn't matter quite so
much. In the same way that WebappClassLoader becomes inert when the
application has been stopped, perhaps we could "shut-down" request /
response / session objects that have been loaned to a
request-processor thread.
Something like this:
Request realRequest = ... // fetch pooled request object
HttpServletRequest req;
RecyclableWrapper wrapper;
if(developmentSafetyMode) {
wrapper = new RecyclableWrapper(realRequest);
req =
(HttpServletRequest)Proxy.newProxyInstance(MigrateConfiguration.class.ge
tClassLoader(),
new Class[] { HttpServletRequest.class }, wrapper);
} else {
req = realRequest;
}
// dispatch request
if(null != wrapper) {
wrapper.recycle();
}
And then the invocation handler class:
static class RecyclableWrapper
implements InvocationHandler
{
private T target;
private boolean recycled;
public RecyclableWrapper(T target) {
this.target = target;
}
public void recycle() {
recycled = true;
}
@Override
public Object invoke(Object proxy, Method method, Object[] args)
throws Throwable {
if(recycled)
throw new InvocationTargetException((Throwable)null,
"Object has been recycled.");
return method.invoke(target, args);
}
}
This would catch these kinds of errors very early, and we could use a
very specific error message that is easy to find using e.g. Google. OR
just put a URL to a wiki page explaining the problem directly in the
error message.
I suppose this could even be done as a Valve (or Filter) so it's
completely optional and trivially configurable. If a Valve, we'd have
to wrap the Request instead of HttpServletRequest.
I've only shown how to wrap the request above, but of course the
response, session, and maybe even the streams should be wrapped, too,
to be completely safe.
Is anyone interested in this kind of thing being packaged with Tomcat?
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3vqzkACgkQHPApP6U8
pFg2qg/+PhgRgGxELAoqv/ZHPBjpv5IutL1X+uLa52Nl3NyuIPSWTFV7SB+ORE0l
5x8DVjW7MR7OUuu4Ptg2RPVmrufWDSANfujtJ9RS/ugzAii33O+3MIY2zHAZgfMk
AKmcIQ3eGK4Ep1SByPTvD7nK3nmIlSzn7Z6d4bBeSawOFYcb1i1zs026gxfgTnnq
Re: http2 async timeout setting
Hi Mark, Very much appreciated. Thanks On Monday, December 9, 2019, 05:59:58 PM GMT+7, Mark Thomas wrote: On 07/12/2019 03:46, Arief Hasani wrote: > Hi Chris, > Thanks for the reminder. following is the code that runs the timeout listener > on time while running on http1.1 but not on http2. tested on 9.0.29 I can reproduce this. I'm working on a fix now. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Crashed when the concurrent Users reached 150
Thanks a Lot Mark On Mon, Dec 9, 2019 at 2:22 PM Mark Thomas wrote: > On 09/12/2019 06:41, Jayaram Ponnusamy wrote: > > Thanks for your Valuable Comments: > > We are using Apache Tomcat/8.0.50 and Mod_JK mod_jk/1.2.28 > > > > The Above configuration made by some consultant. We are in the > situation to > > correct all the things to make it for high availability. > > Also we are in the process of upgrading the apache to 2.4.25 with mod_jk/ > > 1.2.42. And Event MPM instead of Prefork. > > > > Please let us know do we need to use worker MPM also or EVENT MPM is > fine? > > If Event MPM with below Configuration then how many concurrent users can > > access the site and what are the relevant changes needs to be done on > > Tomcat Side? > > On Tomcat Servers We have 16GB RAM with 8CPU [Two Nodes] > > > > # event MPM > > > > StartServers 1 > > ServerLimit 7 > > MinSpareThreads 250 > > MaxSpareThreads 2500 > > ThreadsPerChild 500 > > ThreadLimit 500 > > MaxRequestWorkers 3500 > > MaxConnectionsPerChild 0 > > > > > > > > # Tomcat Configuration > > > URIEncoding="UTF-8" emptySessionPath="true" maxThreads="600" > > minSpareThreads="10" connectionTimeout="-1" /> > > > URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" > > connectionTimeout="2" acceptCount="2000" /> > > > > > > # Tomcat Configuration > > -Xms2048M -Xmx6144M -XX:PermSize=1024m -XX:MaxPermSize=2048m > > > > > > > > > > On Thu, Dec 5, 2019 at 7:27 PM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Jayaram, > > > > On 12/5/19 03:19, Jayaram Ponnusamy wrote: > We are using apache 2.2.21 on RHEL6.9 and in Backend we are using > Tomcat AppServer with JavaBased CMS. Our Server Architecture is > *(F5 [For Load Balance] -> Apache WebServer [For Redirection & URL > Mapping] -> Tomcat AppServer [Running a Java Based CMS]).* > > > > That version of Apache httpd is pretty old and unsupported. RHEL 6.9 > > is also out of support (if I'm reading their Wikipedia page > > correctly). You may want to upgrade everything while you are looking > > at all this. > > > We are using Apache to connect Tomcat by MOD_JK and mapping the > URL (Attached VirtualHost.conf), and No applications/code are > running on WebServers. > > 1. Two TomcatServers (8 CPU & 16GB RAM on Each Servers) 2. Two > WebServers (4 CPU & 8GB RAM on Each Servers) > > *PROBLEM:* When the Child Process count is reached / crossed 200 > like below then site is not accessible > > > > Do you mean that new connections from httpd -> Tomcat cannot be made? > > > when the child processes reached 300 or More than site crashed > > > > Please define "crashed". Got an exception? JVM crash (with hs_pid* > > file)? Kernel panic? > > > and then we bring the site back by restarting Tomcat & Apache > HTTP. > > > > Do you need to restart httpd? Or only Tomcat? Or will restarting httpd > > work? > > > Kindly please check my configuration and help to allow at-least > 500 concurrent users. > > *CONFIGURATION:* * httpd.conf* StartServers > 80 ServerLimit 3500 MaxClients 3500 MaxRequestsPerChild 0 > StartServers 6 MaxClients 230 > MinSpareThreads 75 MaxSpareThreads 150 ThreadsPerChild 25 > MaxRequestsPerChild 0 > > > > Which MPM are you actually using? You have listed both prefork and > > worker. AFAICT you have to pick one at runtime. > > > > For prefork, the maximum number of connections you should expect to be > > made to each Tomcat backend is 3500 (the value of MaxClients). > > > > For worker, the maximum number of connections you should expect to be > > made to each Tomcat backend is 230. > > > *Tomcat:* redirectPort="8443" URIEncoding="UTF-8" emptySessionPath="true" > maxThreads="600" minSpareThreads="10" connectionTimeout="-1" /> > URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" > connectionTimeout="2" acceptCount="2000" /> > > > > Note that AJP connections are expected to be persistent; they don't > > close when the request has completed. They don't even really do "keep > > alive" in the same sense as HTTP keepalive. It's more like "always > > keepalive". Once an httpd process opens a connection to a Tomcat > > instance, that connection will remain open for quite a while as long > > as requests continue to be made. > > > > If you are using prefork MPM, the the version of Tomcat you are using > > is critical to understanding what is happening, here. If you are using > > a BIO-based connector, then you will need to increase the number of > > threads on each Tomcat server from 600 to 3500. > > > > If you are using the worker MPM, then your Tomcat should be able to > > handle the 230 maximum connections configured there. > > TL;DR - Try switching to the NIO AJP connector on Tomcat. > > Take a look at this session I uploaded from TomcatCon London. You > probably want to start around
Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms
On 09/12/2019 19:54, Jerry Malcolm wrote: > Thanks to all of the responders on this problem. From the beginning of > this problem, I was convinced that I was doing something bad wrong to > cause this. I think now I have finally identified the 'bad thing'. > Periodically the model code that runs in the request determines that I > need to spin off another thread to do some background work. That thread > needs a few parameters from the request object. So I pass the request > object into the thread object. I realize now that doing that is a > horrible thing to do. I was not aware until yesterday that request > objects are pooled and recycled. So I figured holding onto the request > object a little longer before it is discarded for that thread to use was > not going to be a problem. Obviously, that was incorrect. Yep, that'll do it. > From what I can deduce, the main request thread finishes and returns. > The request object recycle code runs and the object is returned to the > pool. However, then the async thread later gets a parm and must > apparently still be able to use the request object even though it's back > in the pool. The 'parametersProcessed' gets set to true. Then the next > request comes in, and that request object is assigned with > parametersProcessed=true. I think this analysis is correct. > I'm assuming the fix is to get everything I need out of the request > object and pass those parm values to the thread before returning instead > of passing the request object itself to the thread. Correct. > Does all of this sound plausible as the source for this. The good news > is that I'm still teachable... :-) Very plausible. Kudos for figuring this out so quickly - and for the very clear write up. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
