Re: Tomcat 8 epoll spinning issue (100% CPU)
On 2019/10/04 22:47:17, Christopher Schultz wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Emmanuel, > > On 10/4/19 16:38, Emmanuel Lecharny wrote: > > Hi remy, > > > > On 2019/10/04 15:37:36, Rémy Maucherat wrote: > >> On Fri, Oct 4, 2019 at 3:40 PM Emmanuel Lecharny > >> wrote: > >> > >>> Hi ! > >>> > >>> I filled a ticket yesterday about a pb we face with many NIO > >>> framework, which I think could hit Tomcat too (see > >>> https://bz.apache.org/bugzilla/show_bug.cgi?id=63802). > >>> Actually, I think I'm facing this problem on a project I'm > >>> working on atm. > >>> > >>> Remy suggested we discuss it on this mailing list. > >>> > >>> Bottom line, what happens is that under some circumstances not > >>> well defined, the call to select() might end to an infinite > >>> loop eating all the CPU (select() returns 0, so select is > >>> immediately called again, and we loop). > >>> > >>> In various NIO framworks - and being a MINA committer, I have > >>> implemented the discussed workaround -, we are controlling this > >>> situation by breaking this infinite loop this way : - if the > >>> select() call returns 0 - then if we have called select() more > >>> than N times in less than M ms (N=10, M=100 in MINA) - then we > >>> create a new Selector, register all the selectionKey that were > >>> registered on the broken selector, and ditch the old selector. > >>> > >>> This workaround does not cost a lot when the selector works as > >>> designed, as a select() call should never return 0. > >>> > >> > >> There's actually a very similar hack for APR that has been placed > >> by myself a long time ago [ > >> https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/u > til/net/AprEndpoint.java#L1410 > >> > >> > ], I don't even know if it's actually useful and it's certainly not > >> testable. Overall what it does is pretty terrible :( > >> > >> Personally I would like to know more about this "long lived bug > >> either in the JDK or even in Linux epoll implementation" like > >> actual platform details and JVM versions used since I've never > >> heard about it in the first place. > > > > for the record, I had a discussion yesterday with one of my close > > friend and co-worker back in the 90's. He remember clearly, while > > working on the SUN TCP stack, that such a problem occorded back > > then. Yes, 25 years ago... Ok, that was just for the fun, it's > > likely be perfectly unrelated ;-) > > > > At MINA, we were hit by this bug in 2009 (see > > https://issues.apache.org/jira/browse/DIRMINA-678), and it was > > linked to a bug reported on Jetty > > (http://jetty.4.x6.nabble.com/jira-Created-JETTY-937-SelectChannelConn > ector-100-CPU-usage-on-Linux-td36385.html), > > itself related to some JDK bugs, supposedly fixed since then. > > > > I had a long conversation with Jean-François Arcand somewhere > > around this date, and he suggested we adopt the same workaround he > > applied to Grizzly. We also had a convo with Alan Bateman during a > > Java One in SF, but nothing specific resulted from this convo, > > except that AFAICR, he aknowledge there is an issue. > > > > So this problem started with JDK 6, but I can't guarantee it wasn't > > already present in JDK 5 or 4, on linux, and not on any other OS > > like windows or Mac OSX. It's not exactly fresh in my mind, because > > it was already 10 years ago. > > > >> Also I'd like to know since NIO2 doesn't expose its poller and > >> almost certainly doesn't have such a platform specific mysterious > >> thing inside it [we can check I guess]. > > > > No idea, but I think NIO.2 has just added some coating over what > > was NIO.1 (guts feeling here...). > > > > In the context of NIO, do you have evidence the > >> hack has been tested to work (besides avoiding the CPU loop) and > >> allowed the server to continue its regular operation without any > >> impact ? > > > > Absolutely. We do log in MINA when a new selector is created, and > > we have had some issue related to a case where this piece of code > > was called, fixed since : > > https://issues.apache.org/jira/browse/DIRMINA-762?page=com.atlassian.j > ira.plugin.system.issuetabpanels%3Aall-tabpanel > > > > So we definitively know that people get hit by the initial issue > > (select returns 0), a new selector is being created, and everything > > is fine from the user perspective (I do believe that creating the > > new selector and registering all the SelectionKey on it is not > > worse than having to restart the server manually...) > > > > In any case, Grizzly has probably the best possible approach to > > this problem: make the workaround optional. > > > > For Tomcat, I'm tempted to use the Http11AprProtocol class instead > > of the NIO one, as one can swap the protocol in the configuration, > > but the impact is that you need OpenSSL already installed on your > > machine. That would be an acceptable workaround in my case, but a > > painful one. A similar approach would be
Re: Tomcat 8 epoll spinning issue (100% CPU)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Emmanuel, On 10/4/19 16:38, Emmanuel Lecharny wrote: > Hi remy, > > On 2019/10/04 15:37:36, Rémy Maucherat wrote: >> On Fri, Oct 4, 2019 at 3:40 PM Emmanuel Lecharny >> wrote: >> >>> Hi ! >>> >>> I filled a ticket yesterday about a pb we face with many NIO >>> framework, which I think could hit Tomcat too (see >>> https://bz.apache.org/bugzilla/show_bug.cgi?id=63802). >>> Actually, I think I'm facing this problem on a project I'm >>> working on atm. >>> >>> Remy suggested we discuss it on this mailing list. >>> >>> Bottom line, what happens is that under some circumstances not >>> well defined, the call to select() might end to an infinite >>> loop eating all the CPU (select() returns 0, so select is >>> immediately called again, and we loop). >>> >>> In various NIO framworks - and being a MINA committer, I have >>> implemented the discussed workaround -, we are controlling this >>> situation by breaking this infinite loop this way : - if the >>> select() call returns 0 - then if we have called select() more >>> than N times in less than M ms (N=10, M=100 in MINA) - then we >>> create a new Selector, register all the selectionKey that were >>> registered on the broken selector, and ditch the old selector. >>> >>> This workaround does not cost a lot when the selector works as >>> designed, as a select() call should never return 0. >>> >> >> There's actually a very similar hack for APR that has been placed >> by myself a long time ago [ >> https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/u til/net/AprEndpoint.java#L1410 >> >> ], I don't even know if it's actually useful and it's certainly not >> testable. Overall what it does is pretty terrible :( >> >> Personally I would like to know more about this "long lived bug >> either in the JDK or even in Linux epoll implementation" like >> actual platform details and JVM versions used since I've never >> heard about it in the first place. > > for the record, I had a discussion yesterday with one of my close > friend and co-worker back in the 90's. He remember clearly, while > working on the SUN TCP stack, that such a problem occorded back > then. Yes, 25 years ago... Ok, that was just for the fun, it's > likely be perfectly unrelated ;-) > > At MINA, we were hit by this bug in 2009 (see > https://issues.apache.org/jira/browse/DIRMINA-678), and it was > linked to a bug reported on Jetty > (http://jetty.4.x6.nabble.com/jira-Created-JETTY-937-SelectChannelConn ector-100-CPU-usage-on-Linux-td36385.html), > itself related to some JDK bugs, supposedly fixed since then. > > I had a long conversation with Jean-François Arcand somewhere > around this date, and he suggested we adopt the same workaround he > applied to Grizzly. We also had a convo with Alan Bateman during a > Java One in SF, but nothing specific resulted from this convo, > except that AFAICR, he aknowledge there is an issue. > > So this problem started with JDK 6, but I can't guarantee it wasn't > already present in JDK 5 or 4, on linux, and not on any other OS > like windows or Mac OSX. It's not exactly fresh in my mind, because > it was already 10 years ago. > >> Also I'd like to know since NIO2 doesn't expose its poller and >> almost certainly doesn't have such a platform specific mysterious >> thing inside it [we can check I guess]. > > No idea, but I think NIO.2 has just added some coating over what > was NIO.1 (guts feeling here...). > > In the context of NIO, do you have evidence the >> hack has been tested to work (besides avoiding the CPU loop) and >> allowed the server to continue its regular operation without any >> impact ? > > Absolutely. We do log in MINA when a new selector is created, and > we have had some issue related to a case where this piece of code > was called, fixed since : > https://issues.apache.org/jira/browse/DIRMINA-762?page=com.atlassian.j ira.plugin.system.issuetabpanels%3Aall-tabpanel > > So we definitively know that people get hit by the initial issue > (select returns 0), a new selector is being created, and everything > is fine from the user perspective (I do believe that creating the > new selector and registering all the SelectionKey on it is not > worse than having to restart the server manually...) > > In any case, Grizzly has probably the best possible approach to > this problem: make the workaround optional. > > For Tomcat, I'm tempted to use the Http11AprProtocol class instead > of the NIO one, as one can swap the protocol in the configuration, > but the impact is that you need OpenSSL already installed on your > machine. That would be an acceptable workaround in my case, but a > painful one. A similar approach would be pleasant to have : a > Http11NIONoSpinProtocol class that we can use if needed. I'm inclined to just build this into the standard protocol class with some good documentation explaining why the hack is in there. You will never know you need it until you suddenly need it, and
Re: Tomcat 8 epoll spinning issue (100% CPU)
Hi remy, On 2019/10/04 15:37:36, Rémy Maucherat wrote: > On Fri, Oct 4, 2019 at 3:40 PM Emmanuel Lecharny > wrote: > > > Hi ! > > > > I filled a ticket yesterday about a pb we face with many NIO framework, > > which I think could hit Tomcat too (see > > https://bz.apache.org/bugzilla/show_bug.cgi?id=63802). Actually, I think > > I'm facing this problem on a project I'm working on atm. > > > > Remy suggested we discuss it on this mailing list. > > > > Bottom line, what happens is that under some circumstances not well > > defined, the call to select() might end to an infinite loop eating all the > > CPU (select() returns 0, so select is immediately called again, and we > > loop). > > > > In various NIO framworks - and being a MINA committer, I have implemented > > the discussed workaround -, we are controlling this situation by breaking > > this infinite loop this way : > > - if the select() call returns 0 > > - then if we have called select() more than N times in less than M ms > > (N=10, M=100 in MINA) > > - then we create a new Selector, register all the selectionKey that were > > registered on the broken selector, and ditch the old selector. > > > > This workaround does not cost a lot when the selector works as designed, > > as a select() call should never return 0. > > > > There's actually a very similar hack for APR that has been placed by myself > a long time ago [ > https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/AprEndpoint.java#L1410 > ], I don't even know if it's actually useful and it's certainly not > testable. Overall what it does is pretty terrible :( > > Personally I would like to know more about this "long lived bug either in > the JDK or even in Linux epoll implementation" like actual platform details > and JVM versions used since I've never heard about it in the first place. for the record, I had a discussion yesterday with one of my close friend and co-worker back in the 90's. He remember clearly, while working on the SUN TCP stack, that such a problem occorded back then. Yes, 25 years ago... Ok, that was just for the fun, it's likely be perfectly unrelated ;-) At MINA, we were hit by this bug in 2009 (see https://issues.apache.org/jira/browse/DIRMINA-678), and it was linked to a bug reported on Jetty (http://jetty.4.x6.nabble.com/jira-Created-JETTY-937-SelectChannelConnector-100-CPU-usage-on-Linux-td36385.html), itself related to some JDK bugs, supposedly fixed since then. I had a long conversation with Jean-François Arcand somewhere around this date, and he suggested we adopt the same workaround he applied to Grizzly. We also had a convo with Alan Bateman during a Java One in SF, but nothing specific resulted from this convo, except that AFAICR, he aknowledge there is an issue. So this problem started with JDK 6, but I can't guarantee it wasn't already present in JDK 5 or 4, on linux, and not on any other OS like windows or Mac OSX. It's not exactly fresh in my mind, because it was already 10 years ago. > Also I'd like to know since NIO2 doesn't expose its poller and almost > certainly doesn't have such a platform specific mysterious thing inside it > [we can check I guess]. No idea, but I think NIO.2 has just added some coating over what was NIO.1 (guts feeling here...). In the context of NIO, do you have evidence the > hack has been tested to work (besides avoiding the CPU loop) and allowed > the server to continue its regular operation without any impact ? Absolutely. We do log in MINA when a new selector is created, and we have had some issue related to a case where this piece of code was called, fixed since : https://issues.apache.org/jira/browse/DIRMINA-762?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel So we definitively know that people get hit by the initial issue (select returns 0), a new selector is being created, and everything is fine from the user perspective (I do believe that creating the new selector and registering all the SelectionKey on it is not worse than having to restart the server manually...) In any case, Grizzly has probably the best possible approach to this problem: make the workaround optional. For Tomcat, I'm tempted to use the Http11AprProtocol class instead of the NIO one, as one can swap the protocol in the configuration, but the impact is that you need OpenSSL already installed on your machine. That would be an acceptable workaround in my case, but a painful one. A similar approach would be pleasant to have : a Http11NIONoSpinProtocol class that we can use if needed. WDYT ? Emmanuel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache SSI breaks with tomcat-connectors-1.2.43 or newer
The SVN Build works for us! Thanks Mark. When do you think the official release will be ready? --Ez On Wed, Oct 2, 2019 at 10:02 AM Mark Thomas wrote: > On 02/10/2019 15:39, Mark Thomas wrote: > > On 02/10/2019 14:51, Mark Thomas wrote: > > > > > > There is a work-around. Use virtual="..." in the SSI includes. > > > > Meanwhile, I am working on a fix for mod_jk. > > Done. If you want to test it out you'll have to build from svn. > Meanwhile, I'll start thinking about a mod_jk release. We haven't had > one for about a year and this bug fix seems like a good reason to have one. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat 8 epoll spinning issue (100% CPU)
On Fri, Oct 4, 2019 at 3:40 PM Emmanuel Lecharny wrote: > Hi ! > > I filled a ticket yesterday about a pb we face with many NIO framework, > which I think could hit Tomcat too (see > https://bz.apache.org/bugzilla/show_bug.cgi?id=63802). Actually, I think > I'm facing this problem on a project I'm working on atm. > > Remy suggested we discuss it on this mailing list. > > Bottom line, what happens is that under some circumstances not well > defined, the call to select() might end to an infinite loop eating all the > CPU (select() returns 0, so select is immediately called again, and we > loop). > > In various NIO framworks - and being a MINA committer, I have implemented > the discussed workaround -, we are controlling this situation by breaking > this infinite loop this way : > - if the select() call returns 0 > - then if we have called select() more than N times in less than M ms > (N=10, M=100 in MINA) > - then we create a new Selector, register all the selectionKey that were > registered on the broken selector, and ditch the old selector. > > This workaround does not cost a lot when the selector works as designed, > as a select() call should never return 0. > There's actually a very similar hack for APR that has been placed by myself a long time ago [ https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/AprEndpoint.java#L1410 ], I don't even know if it's actually useful and it's certainly not testable. Overall what it does is pretty terrible :( Personally I would like to know more about this "long lived bug either in the JDK or even in Linux epoll implementation" like actual platform details and JVM versions used since I've never heard about it in the first place. Also I'd like to know since NIO2 doesn't expose its poller and almost certainly doesn't have such a platform specific mysterious thing inside it [we can check I guess]. In the context of NIO, do you have evidence the hack has been tested to work (besides avoiding the CPU loop) and allowed the server to continue its regular operation without any impact ? Rémy > > I suggest Tomcat add such a workaround in the various versions (8 and 9 at > least). > > Emmanuel Lécharny > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Tomcat 8 epoll spinning issue (100% CPU)
Hi ! I filled a ticket yesterday about a pb we face with many NIO framework, which I think could hit Tomcat too (see https://bz.apache.org/bugzilla/show_bug.cgi?id=63802). Actually, I think I'm facing this problem on a project I'm working on atm. Remy suggested we discuss it on this mailing list. Bottom line, what happens is that under some circumstances not well defined, the call to select() might end to an infinite loop eating all the CPU (select() returns 0, so select is immediately called again, and we loop). In various NIO framworks - and being a MINA committer, I have implemented the discussed workaround -, we are controlling this situation by breaking this infinite loop this way : - if the select() call returns 0 - then if we have called select() more than N times in less than M ms (N=10, M=100 in MINA) - then we create a new Selector, register all the selectionKey that were registered on the broken selector, and ditch the old selector. This workaround does not cost a lot when the selector works as designed, as a select() call should never return 0. I suggest Tomcat add such a workaround in the various versions (8 and 9 at least). Emmanuel Lécharny - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Content Security policy for Tomcat 8.5
Hello, Thanks for replying... My current tomcat version is 8.5.x hosted on windows 2012 R2 server and no other web server as fronted web server. CSP values shared with me is : "default-src 'self' 'unsafe-eval' 'unsafe-inline' *.mycompany.com; script-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' *.mycompany.com data:; connect-src ‘self’ *. mycompany.com" I am new to Tomcat setup and able to add headerfilter for other header but didnt find much help for CSP ones On Fri, Oct 4, 2019 at 3:08 AM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Nitin, > > On 10/3/19 09:54, Nitin Kadam wrote: > > Hell All, > > > > Internal security team recommended to set *Content security policy* > > header for Web server as same is not complaint with security > > standard. can you please help me setting CSP filters for my Tomcat > > application hosted on windows server. > > Do you know the value you want to use for your CSP header? > > Enabling the header can be done in a number of ways, including using > http://tomcat.apache.org/tomcat-9.0-doc/rewrite.html > > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl2WajgACgkQHPApP6U8 > pFg9VxAAuhUwoIwgFmT23ynF/DNJxVaHVcIpu3v6ekHOE59T8mL4wd6s8356nw7G > tR19Q8S8aiRNiPWIfa9N5Ifis2p9KCJVCxck9PPxzqCVYM2wLaBWIzyoJz3GRQ4S > hDLdEhGJYEDUY6Oc1LLaa/ZhFz6+cb03NXRtmMT+ynVyO1w3BgL9+DbRhbqdbEd2 > SeFlAQTudakOcHe1nfy5r0pyaoGAVcPp5G6vLLtanWTPSpe2lWlRlW3Y6UAPFYBz > g2iNoIfsvIUR4sGcHcJXQZZ4hPFCvmOdziCXx1duG3P2ki4HZ11Zn3FyqfexCAwb > 7Di1f7m0kIZ52b/a6gDagZ5zg3FPKkDw4esW7ml0Bm73va4yD0hmg7Pv/nBIalcI > hNOl0fxpPnuq/XzfCzZM8ep7MweHD9U0xDnQQ6nVdLz8HjbM0fvUxe375brASGcT > KuCC3xqLy2xokVwNN+AAi5ccsOB+b5hPzF69XT4DlvZszTuwsYpIFCudfvVY/Zzk > SSogvNDGF5ERll7xVS6//NguwPfMFzeS7v01AtP+ojf6Bl4c6jEoH8mEgckTaVyR > R5kX9yeDOwnA2Q8DHOw32R748UcfoErophkGLbqpuS3uHIkQQQA0UuWgFWZHDUfl > H2DBkFtDmlCLQR4m8F6WCbANsllZvf9LQBfsysCDb66CvMep9wQ= > =oC/r > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Regards Nitin Kadam (9967688959)
