RE: Invalid HTTP Header - attack?
My apologies, the version of Apache that came with the application is 9.0.13. Tony Justiniano Engineer I, EUS Engineering Wyndham Destinations 6277 Sea Harbor Drive Orlando, FL 32821 Office: +1-407-626-5416 Mobile: +1-407-463-4297 tony.justini...@wyn.com -Original Message- From: John Dale Sent: Thursday, August 1, 2019 4:37 PM To: Tomcat Users List Subject: Re: Invalid HTTP Header - attack? This e-mail is from an external source. Use caution when opening attachments or clicking on links. 9.0.16.0 - this is the version installed with apt-get tomcat9 on ubuntu 18.04 Thank you for your feedback. John On 8/1/19, Konstantin Kolinko wrote: > чт, 1 авг. 2019 г. в 22:11, John Dale : >> >> Great feedback. Thanks. >> >> I am the network department. :) >> >> This is a public facing service and shortly after I see this in the >> log, I get an OOM exception and server shutdown. Twice now this >> morning. >> > > The exception text is a bit misleading. It says "header", but it > actually caused by sanity checks that are done when parsing the first > line of the request (it precedes all the headers) aka the "request > line". Thus you can see "parseRequestLine()" in the stack trace. > > As you may know, starting with HTTP/1.1 a client can send several HTTP > request over the same connection (aka "keep alive", also "request > pipelining"). If the length of the preceding request was not processed > correctly either because the client sent an incorrect value of > Content-Length header or if there is a bug, Tomcat will start parsing > a new request at a wrong place and you will see such an error. > > Other cause of similar errors is when a client tries to connect using > https: protocol to a http: connector. A small difference is that in > that case the sanity check will be triggered earlier: when parsing the > HTTP method name (the first component of the request line). In your > case the error message says about the HTTP protocol version (the third > component of the request line). > > > 1. Personally, I always run with > org.apache.catalina.connector.RECYCLE_FACADES=true > as documented in [1]. > > This property helps if there is a bug in a web application. > > 2. Make sure that you use an up-to-date version of Tomcat. You didn't > tell us what version of Tomcat 9.0.x you are using. > > 3. If there is bug that causes Tomcat to incorrectly process a length > of a request (a known way to trigger such a bug), I think that it will > be treated as a security vulnerability that leads to an information > leak. > > See CVE-2018-8037 )fixed in 9.0.10), CVE-2017-5651 and CVE-2017-5647 > (both fixed in 9.0.0.M19) for an idea. > > https://tomcat.apache.org/security-9.html > > Maybe you can configure creation of a heap dump during the OOM, so > that it could be diagnosed what is causing a memory leak? > > Note that there is a procedure to report security issues [2]. A public > Bugzilla should not be used for such reports. > > 4. The error message that you saw is printed only once in every 24 > hours. The latter occurrences during the same day are suppressed > (logged at DEBUG level) to prevent flooding one's log files. This > behaviour is controlled by system properties [3], > > org.apache.juli.logging.UserDataHelper.CONFIG > org.apache.juli.logging.UserDataHelper.SUPPRESSION_TIME > > [1] > https://tomcat.apache.org/tomcat-9.0-doc/security-howto.html#System_Pr > operties > > [2] https://tomcat.apache.org/security.html > > [3] > https://tomcat.apache.org/tomcat-9.0-doc/config/systemprops.html#Loggi > ng > > Best regards, > Konstantin Kolinko > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org This email message (including all attachments) is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information, or may otherwise be protected by work product or other legal rules. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Unless otherwise indicated in the body of this email, nothing in this communication is intended to operate as an electronic signature and this transmission cannot be used to form, document, or authenticate a contract. Wyndham Destinations, Inc., and/or its affiliates may monitor all incoming and outgoing
Re: Invalid HTTP Header - attack?
9.0.16.0 - this is the version installed with apt-get tomcat9 on ubuntu 18.04 Thank you for your feedback. John On 8/1/19, Konstantin Kolinko wrote: > чт, 1 авг. 2019 г. в 22:11, John Dale : >> >> Great feedback. Thanks. >> >> I am the network department. :) >> >> This is a public facing service and shortly after I see this in the >> log, I get an OOM exception and server shutdown. Twice now this >> morning. >> > > The exception text is a bit misleading. It says "header", but it > actually caused by sanity checks that are done when parsing the first > line of the request (it precedes all the headers) aka the "request > line". Thus you can see "parseRequestLine()" in the stack trace. > > As you may know, starting with HTTP/1.1 a client can send several HTTP > request over the same connection (aka "keep alive", also "request > pipelining"). If the length of the preceding request was not processed > correctly either because the client sent an incorrect value of > Content-Length header or if there is a bug, Tomcat will start parsing > a new request at a wrong place and you will see such an error. > > Other cause of similar errors is when a client tries to connect using > https: protocol to a http: connector. A small difference is that in > that case the sanity check will be triggered earlier: when parsing the > HTTP method name (the first component of the request line). In your > case the error message says about the HTTP protocol version (the third > component of the request line). > > > 1. Personally, I always run with > org.apache.catalina.connector.RECYCLE_FACADES=true > as documented in [1]. > > This property helps if there is a bug in a web application. > > 2. Make sure that you use an up-to-date version of Tomcat. You didn't > tell us what version of Tomcat 9.0.x you are using. > > 3. If there is bug that causes Tomcat to incorrectly process a length > of a request (a known way to trigger such a bug), I think that it will > be treated as a security vulnerability that leads to an information > leak. > > See CVE-2018-8037 )fixed in 9.0.10), CVE-2017-5651 and CVE-2017-5647 > (both fixed in 9.0.0.M19) for an idea. > > https://tomcat.apache.org/security-9.html > > Maybe you can configure creation of a heap dump during the OOM, so > that it could be diagnosed what is causing a memory leak? > > Note that there is a procedure to report security issues [2]. A public > Bugzilla should not be used for such reports. > > 4. The error message that you saw is printed only once in every 24 > hours. The latter occurrences during the same day are suppressed > (logged at DEBUG level) to prevent flooding one's log files. This > behaviour is controlled by system properties [3], > > org.apache.juli.logging.UserDataHelper.CONFIG > org.apache.juli.logging.UserDataHelper.SUPPRESSION_TIME > > [1] > https://tomcat.apache.org/tomcat-9.0-doc/security-howto.html#System_Properties > > [2] https://tomcat.apache.org/security.html > > [3] > https://tomcat.apache.org/tomcat-9.0-doc/config/systemprops.html#Logging > > Best regards, > Konstantin Kolinko > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Invalid HTTP Header - attack?
чт, 1 авг. 2019 г. в 22:11, John Dale : > > Great feedback. Thanks. > > I am the network department. :) > > This is a public facing service and shortly after I see this in the > log, I get an OOM exception and server shutdown. Twice now this > morning. > The exception text is a bit misleading. It says "header", but it actually caused by sanity checks that are done when parsing the first line of the request (it precedes all the headers) aka the "request line". Thus you can see "parseRequestLine()" in the stack trace. As you may know, starting with HTTP/1.1 a client can send several HTTP request over the same connection (aka "keep alive", also "request pipelining"). If the length of the preceding request was not processed correctly either because the client sent an incorrect value of Content-Length header or if there is a bug, Tomcat will start parsing a new request at a wrong place and you will see such an error. Other cause of similar errors is when a client tries to connect using https: protocol to a http: connector. A small difference is that in that case the sanity check will be triggered earlier: when parsing the HTTP method name (the first component of the request line). In your case the error message says about the HTTP protocol version (the third component of the request line). 1. Personally, I always run with org.apache.catalina.connector.RECYCLE_FACADES=true as documented in [1]. This property helps if there is a bug in a web application. 2. Make sure that you use an up-to-date version of Tomcat. You didn't tell us what version of Tomcat 9.0.x you are using. 3. If there is bug that causes Tomcat to incorrectly process a length of a request (a known way to trigger such a bug), I think that it will be treated as a security vulnerability that leads to an information leak. See CVE-2018-8037 )fixed in 9.0.10), CVE-2017-5651 and CVE-2017-5647 (both fixed in 9.0.0.M19) for an idea. https://tomcat.apache.org/security-9.html Maybe you can configure creation of a heap dump during the OOM, so that it could be diagnosed what is causing a memory leak? Note that there is a procedure to report security issues [2]. A public Bugzilla should not be used for such reports. 4. The error message that you saw is printed only once in every 24 hours. The latter occurrences during the same day are suppressed (logged at DEBUG level) to prevent flooding one's log files. This behaviour is controlled by system properties [3], org.apache.juli.logging.UserDataHelper.CONFIG org.apache.juli.logging.UserDataHelper.SUPPRESSION_TIME [1] https://tomcat.apache.org/tomcat-9.0-doc/security-howto.html#System_Properties [2] https://tomcat.apache.org/security.html [3] https://tomcat.apache.org/tomcat-9.0-doc/config/systemprops.html#Logging Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Invalid HTTP Header - attack?
Great feedback. Thanks. I am the network department. :) This is a public facing service and shortly after I see this in the log, I get an OOM exception and server shutdown. Twice now this morning. Hmm .. :\ John On 8/1/19, Michael Osipov wrote: > Am 2019-08-01 um 20:36 schrieb Mark Thomas: >> On 01/08/2019 19:08, John Dale wrote: >>> I'm getting this in my logs - is this an attack do you think? >> >> Unlikely to be an attack. Most likely a broken client. > > There is another scenario: > > Regular security scans on all corporate subnets from sec dept. I have > these almost every day in access.log and via SSH. > > Ask your network department who's IP this is and you should get better > information. > > See also: https://bz.apache.org/bugzilla/show_bug.cgi?id=55372 > > Michael > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Invalid HTTP Header - attack?
Am 2019-08-01 um 20:36 schrieb Mark Thomas: On 01/08/2019 19:08, John Dale wrote: I'm getting this in my logs - is this an attack do you think? Unlikely to be an attack. Most likely a broken client. There is another scenario: Regular security scans on all corporate subnets from sec dept. I have these almost every day in access.log and via SSH. Ask your network department who's IP this is and you should get better information. See also: https://bz.apache.org/bugzilla/show_bug.cgi?id=55372 Michael - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Invalid HTTP Header - attack?
On 01/08/2019 19:08, John Dale wrote: > I'm getting this in my logs - is this an attack do you think? Unlikely to be an attack. Most likely a broken client. > How > might I determine this? debug logging for org.apache.coyote.http11.Http11InputBuffer is going to log the request line and HTTP headers for every request. tcpdump / wireshark is another option. > Could this be pushing bytes to the handler and causing a memory issue? No. Mark > > Error parsing HTTP request header > Aug 1 17:37:58 dom1 tomcat9[9793]: Note: further occurrences of HTTP > request parsing errors will be logged at DEBUG level. > Aug 1 17:37:58 dom1 tomcat9[9793]: > java.lang.IllegalArgumentException: Invalid character found in the > HTTP protocol > Aug 1 17:37:58 dom1 tomcat9[9793]: at > org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:531) > Aug 1 17:37:58 dom1 tomcat9[9793]: at > org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:294) > Aug 1 17:37:58 dom1 tomcat9[9793]: at > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) > Aug 1 17:37:58 dom1 tomcat9[9793]: at > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) > Aug 1 17:37:58 dom1 tomcat9[9793]: at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) > Aug 1 17:37:58 dom1 tomcat9[9793]: at > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) > Aug 1 17:37:58 dom1 tomcat9[9793]: at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > Aug 1 17:37:58 dom1 tomcat9[9793]: at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > Aug 1 17:37:58 dom1 tomcat9[9793]: at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > Aug 1 17:37:58 dom1 tomcat9[9793]: at > java.lang.Thread.run(Thread.java:748) > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Invalid HTTP Header - attack?
I'm getting this in my logs - is this an attack do you think? How might I determine this? Could this be pushing bytes to the handler and causing a memory issue? Error parsing HTTP request header Aug 1 17:37:58 dom1 tomcat9[9793]: Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. Aug 1 17:37:58 dom1 tomcat9[9793]: java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol Aug 1 17:37:58 dom1 tomcat9[9793]: at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:531) Aug 1 17:37:58 dom1 tomcat9[9793]: at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:294) Aug 1 17:37:58 dom1 tomcat9[9793]: at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) Aug 1 17:37:58 dom1 tomcat9[9793]: at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) Aug 1 17:37:58 dom1 tomcat9[9793]: at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) Aug 1 17:37:58 dom1 tomcat9[9793]: at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) Aug 1 17:37:58 dom1 tomcat9[9793]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) Aug 1 17:37:58 dom1 tomcat9[9793]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) Aug 1 17:37:58 dom1 tomcat9[9793]: at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) Aug 1 17:37:58 dom1 tomcat9[9793]: at java.lang.Thread.run(Thread.java:748) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
