Re: Our tomcat just crashed due to classnotfound, not sure how to investigate
pentesting going on? On Feb 16, 2015 10:44 AM, "Mathias af Jochnick" wrote: > Christopher, thanks for responding. See my comments below. > > -- > Mathias af Jochnick, +46703414084 > > On 16 February 2015 at 16:36:32, Christopher Schultz ( > ch...@christopherschultz.net) wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Mathias, > > On 2/16/15 5:43 AM, Mathias af Jochnick wrote: > > Our server has been in production for years without issues, but > > today it crashed out of the blue. > > > > Last in the logs: > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > > aasxknsakadskdskdskdsakmxxads Caused by: > > java.lang.ClassNotFoundException: aasknsakadskdskdskdsakmads at > > java.net.URLClassLoader$1.run(URLClassLoader.java:217) at > > java.security.AccessController.doPrivileged(Native Method) at > > java.net.URLClassLoader.findClass(URLClassLoader.java:205) at > > java.lang.ClassLoader.loadClass(ClassLoader.java:321) at > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) at > > java.lang.ClassLoader.loadClass(ClassLoader.java:266) > > > > Could not find the main class: aasxknsakadskdskdskdsakxxmads. > > Program will exit. > > Where do you see this? Which log file? > >>Catalina.out > > > > When Java can't load a class, a ClassNotFoundException or > NoClassDefFoundError is thrown, but it does not terminate the JVM. > > The message above says "Could not find the main class", which > indicates that a JVM was trying to launch with a new main class, not > that an existing JVM (i.e. Tomcat running your web apps) was trying to > load a class. > >>Right, well as i said, it’s in catalina.out > > > > When I run my JVM with a bad main class, I don't get a stack trace: > > $ java -showversion foo > java version "1.8.0_31" > Java(TM) SE Runtime Environment (build 1.8.0_31-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode) > > Error: Could not find or load main class foo > > Other versions of Java: > > $ java -showversion foo > java version "1.7.0_76" > Java(TM) SE Runtime Environment (build 1.7.0_76-b13) > Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode) > > Error: Could not find or load main class foo > > $ java -showversion foo > java version "1.6.0_65" > Java(TM) SE Runtime Environment (build 1.6.0_65-b14-466.1-11M4716) > Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-466.1, mixed mode) > > Exception in thread "main" java.lang.NoClassDefFoundError: foo > Caused by: java.lang.ClassNotFoundException: foo > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > So, I get the exception and stack trace, but no message about how the > "Program will exit." > > So... how is this being launched? > >>>Well, that’s what i’m wondering too. Our Tomcat process died, that was > what was last in the logfile, the server had been up for a week. Nobody had > logged in and done anything, unless we’re missing something. The server is > at a hosting partner and i’m pretty confident they’re not complete tools if > you see what i mean. > > > > > While its obvious what caused the crash, i have no clue as to why. > > We have no new production code, nothing has changed as far as we > > know. I'm not sure how to investigate this, to me it seems like > > some sort of dynamic class-loading hack attempt? > > > > Can i look for some configured service to turn off? > > > > I'm at a loss so any pointers on how to investigate / prevent this > > would be extremely appreciated. > > > > Tomcat/6.0.28 > > No currently-supported version of Tomcat prints the message "Program > will exit.". I'm at a loss to explain why you would *ever* see this > message, let alone what is causing it in your particular case. > > I would treat the server as suspicious and take appropriate steps. > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG v1 > Comment: GPGTools - http://gpgtools.org > > iQIcBAEBCAAGBQJU4g4aAAoJEBzwKT+lPKRYdAUP/0ud2VoU5Ts5ztmHCHGBkeF4 > bGAvY7/eKNWxOG7Zt7KlDvIl06dZa7yQarnD6EtpdAWtg1lP3YO5GM6HXlS8EN9g > 14MrTxsC0rGJXji2+z7EHXwi2v1bosLqadufgaS6RXgbhiZ/YhoiHK2MWTEDgONX > 5GLe8yjQQ2ADZeb8JaOO3fT0Tt1/ZQd/hRLsyCw2QFCUF/VP9puczMb7cQj6v2L/ > xizSxJenSZsVljVSEnwC/XBv/JpIja/F9FD2qvB0da9hSa5Usuvg6GSPihegldMS > nIXgbdYQcb8m3aFysF/LVTd2cRHp3RQfz/O10mNjSI0eDFkPOHXeEXOy0Sqdew8u > VtrUm1haWrbo/sAUDBpgIvL5RP4L/YBH84CFcG2syXeff/kfCE1wkKH/HaOx9B31 > S7wLVp/fk1RW1N2KHv3SdxWh1Ual7uvidyP3+Gmztk3278OwPX8Ji49tXB9aLc8R > fzXi06laGkQ8T/mQggzeSvr70BW6rBmcK9gziFmNxUbOrCvOCP987h97YXd2NRZc > qTXOWVdO+GkO09thdVCRCd3IPbvt+P7en+A0b+p7eJUQZjIYlFHA1pL9hoDxYLD1 > C3Zfedk4tpig4ynDM+VHZ9n44zEJO8RSZlzknNiLiy0GE1pFMr0TJePTMsc4tCu0 > zb/vU3pKvKrQNFIdydaE > =jXyS > -END PGP SIGNATURE
Re: undefined method
Thanks for the additional suggestions! Will try those on Monday. On Sat, Feb 21, 2015 at 3:26 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Sean, > > On 2/20/15 5:00 PM, Sean Dawson wrote: > > On Fri, Feb 20, 2015 at 4:41 PM, Konstantin Kolinko > > wrote: > > > >> 2015-02-21 0:10 GMT+03:00 Sean Dawson > >> : > >>> We have a GWT app deployed to tomcat (7_59) and fairly often > >>> when we > >> send a > >>> bunch of request quickly we're seeing undefined methods in the > >>> logs - and the calls fail, causing issues with our app. We > >>> make calls via RestyGwt (latest version) but GwtRequests all > >>> show this - both though after a > >> number > >>> of REST calls in a short period of time. So for example... > >>> > >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path1 > >>> HTTP/1.1" 200 > >> 304 > >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path2 > >>> HTTP/1.1" 200 > >> 310 > >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path3 > >>> HTTP/1.1" 200 > >> 307 > >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "undefinedDELETE > >>> /path4 HTTP/1.1" 501 304 [ip-addr] - - [20/Feb/2015:15:24:34 > >>> -0500] "DELETE /path5 HTTP/1.1" 200 > >> 304 > >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path6 > >>> HTTP/1.1" 200 > >> 310 > >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path7 > >>> HTTP/1.1" 200 > >> 307 > >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "undefinedDELETE > >>> /path8 HTTP/1.1" 501 304 > >>> > >>> Similarly... > >>> > >>> ... "undefinedPOST /gwtRequest HTTP/1.1" 501 1136 > >>> > >>> Very little info online, but did come across this old bug... > >>> > >>> https://bz.apache.org/bugzilla/show_bug.cgi?id=49779 > >>> > >>> In fiddler, the headers are identical between the requests that > >>> work and those that fail. Resending the failed request > >>> completes normally. > >>> > >>> So far we've only be able to reproduce this when using Internet > >>> Explorer (10 & 11) and we've spent a lot of time trying to > >>> figure out what's going on - but have been unable. Any > >>> pointers/explanations? > >>> > >>> Thanks! > >> > >> "undefined" is a JavaScript word. In Java I would expect "null" > >> instead of that word. > >> > >>> In fiddler, the headers are identical between the requests that > >>> work and those that fail. > >> > >> The string in access log is not a header. It is HTTP request > >> line. The first line of an HTTP request. > >> > >> > > Ok, but this is in the standard tomcat access logs, using standard > > logging, and is in the method name, not URL. Maybe I'm not > > understanding what you're saying here. > > > > > >> BTW, a similar issue at stackoverflow (but the "undefined" string > >> was added to URL part of request line): > >> > >> > >> > http://stackoverflow.com/questions/11017609/undefined-randomly-appended-in-1-of-requested-urls-on-my-website-since-12-jun > >> > >> > Title: “undefined” randomly appended in 1% of requested urls on my > >> website since 12 june 2012 > >> > >> > > We did come across it but again our's is in the method, not in the > > URL. > > > > > >> > >> One of theories there is that some browser addon was > >> malfunctioning. > >> > >> > > Ok, this has happened on about 5 people's machines with a couple > > different versions of IE - I don't think we have any addons at all > > in some cases. > > > > > >> If nothing else helps, it should be easy to implement a Valve > >> for Tomcat that will fix the wrong request.getMethod() value > >> before passing it to a web application. > >> > >> > > I don't know much about that but we could give it a try - so > > someone else is changing the method somewhere before it gets to > > tomcat? and the Valve will change it back? > > Fiddler isn't the authority when it comes to what is going across the > wire. It's possible that something is happening after Fiddler takes > its samples. > > Are you able to hook-up something like Wireshark or other > packet-capturing software to see what actually goes over the wire? > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG v1 > Comment: GPGTools - http://gpgtools.org > > iQIcBAEBCAAGBQJU6On7AAoJEBzwKT+lPKRY8vEQAJiupeJotnZVWXFeL5wbwAfw > 5nDiXz1wu6IcY0FNingOx/cgorIxJP1Ti6qNY06gXfEG/sJ+Fk1MNn4bbtxoPv5P > nRpMjSC1jloxmMK+Y6RKTt5815yCAiggd/mJONzK6NN+vfG6hg4C0l9GnCnuuMte > 9mDUkkqogkn2EGYJQua3JiCoQT+qAJbOA+zxRJJcLHB+GzSQLHT48KYAmJQVRWH2 > CRtFXQxPtuE0QVaMCWJQcSKqFuJ2y9ZiP77E2DJfo644/4VP2sDk2rIk3MtJCT3F > gfLWbMMFcV27QTXQvH3uYXhdEfrVhTUGxurio95gVD6y0g7F4pMYeJAcvTnYVV8Y > C9OhHLJrn4NXJ34D7XIzefTaVc8kcp/oVKe7irLK9IapIIqdX+H0P3uHuFCPFEPg > aKSNVJ80jD72/yjUAiULgagjOJ7k4b9WhnsrZJFCRydT5yCcK7w3UrNdIDgQSltp > TjfJTfCitCzb6/pXMnT+DE7PyPQyeIviU+7rCs89xBNHAoFyYJ+agJOu6CE/hMhg > LT672uLvkt4XD2eLE5yUYOHY7KkIKV6WfYPtyyamnoy9vpCPLTxlH6ZyRg+xt17D > zP201nRf4Hyay6x7vi+cB4SZ1f5nUS8eV5hPrDZmLiIksdSqzkZFD/a5/JMsa07C > esM43RAOa8/LxmJCiyqz > =gMzK > -END PGP S
Re: undefined method
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Sean, On 2/20/15 5:00 PM, Sean Dawson wrote: > On Fri, Feb 20, 2015 at 4:41 PM, Konstantin Kolinko > wrote: > >> 2015-02-21 0:10 GMT+03:00 Sean Dawson >> : >>> We have a GWT app deployed to tomcat (7_59) and fairly often >>> when we >> send a >>> bunch of request quickly we're seeing undefined methods in the >>> logs - and the calls fail, causing issues with our app. We >>> make calls via RestyGwt (latest version) but GwtRequests all >>> show this - both though after a >> number >>> of REST calls in a short period of time. So for example... >>> >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path1 >>> HTTP/1.1" 200 >> 304 >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path2 >>> HTTP/1.1" 200 >> 310 >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path3 >>> HTTP/1.1" 200 >> 307 >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "undefinedDELETE >>> /path4 HTTP/1.1" 501 304 [ip-addr] - - [20/Feb/2015:15:24:34 >>> -0500] "DELETE /path5 HTTP/1.1" 200 >> 304 >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path6 >>> HTTP/1.1" 200 >> 310 >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "DELETE /path7 >>> HTTP/1.1" 200 >> 307 >>> [ip-addr] - - [20/Feb/2015:15:24:34 -0500] "undefinedDELETE >>> /path8 HTTP/1.1" 501 304 >>> >>> Similarly... >>> >>> ... "undefinedPOST /gwtRequest HTTP/1.1" 501 1136 >>> >>> Very little info online, but did come across this old bug... >>> >>> https://bz.apache.org/bugzilla/show_bug.cgi?id=49779 >>> >>> In fiddler, the headers are identical between the requests that >>> work and those that fail. Resending the failed request >>> completes normally. >>> >>> So far we've only be able to reproduce this when using Internet >>> Explorer (10 & 11) and we've spent a lot of time trying to >>> figure out what's going on - but have been unable. Any >>> pointers/explanations? >>> >>> Thanks! >> >> "undefined" is a JavaScript word. In Java I would expect "null" >> instead of that word. >> >>> In fiddler, the headers are identical between the requests that >>> work and those that fail. >> >> The string in access log is not a header. It is HTTP request >> line. The first line of an HTTP request. >> >> > Ok, but this is in the standard tomcat access logs, using standard > logging, and is in the method name, not URL. Maybe I'm not > understanding what you're saying here. > > >> BTW, a similar issue at stackoverflow (but the "undefined" string >> was added to URL part of request line): >> >> >> http://stackoverflow.com/questions/11017609/undefined-randomly-appended-in-1-of-requested-urls-on-my-website-since-12-jun >> >> Title: “undefined” randomly appended in 1% of requested urls on my >> website since 12 june 2012 >> >> > We did come across it but again our's is in the method, not in the > URL. > > >> >> One of theories there is that some browser addon was >> malfunctioning. >> >> > Ok, this has happened on about 5 people's machines with a couple > different versions of IE - I don't think we have any addons at all > in some cases. > > >> If nothing else helps, it should be easy to implement a Valve >> for Tomcat that will fix the wrong request.getMethod() value >> before passing it to a web application. >> >> > I don't know much about that but we could give it a try - so > someone else is changing the method somewhere before it gets to > tomcat? and the Valve will change it back? Fiddler isn't the authority when it comes to what is going across the wire. It's possible that something is happening after Fiddler takes its samples. Are you able to hook-up something like Wireshark or other packet-capturing software to see what actually goes over the wire? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJU6On7AAoJEBzwKT+lPKRY8vEQAJiupeJotnZVWXFeL5wbwAfw 5nDiXz1wu6IcY0FNingOx/cgorIxJP1Ti6qNY06gXfEG/sJ+Fk1MNn4bbtxoPv5P nRpMjSC1jloxmMK+Y6RKTt5815yCAiggd/mJONzK6NN+vfG6hg4C0l9GnCnuuMte 9mDUkkqogkn2EGYJQua3JiCoQT+qAJbOA+zxRJJcLHB+GzSQLHT48KYAmJQVRWH2 CRtFXQxPtuE0QVaMCWJQcSKqFuJ2y9ZiP77E2DJfo644/4VP2sDk2rIk3MtJCT3F gfLWbMMFcV27QTXQvH3uYXhdEfrVhTUGxurio95gVD6y0g7F4pMYeJAcvTnYVV8Y C9OhHLJrn4NXJ34D7XIzefTaVc8kcp/oVKe7irLK9IapIIqdX+H0P3uHuFCPFEPg aKSNVJ80jD72/yjUAiULgagjOJ7k4b9WhnsrZJFCRydT5yCcK7w3UrNdIDgQSltp TjfJTfCitCzb6/pXMnT+DE7PyPQyeIviU+7rCs89xBNHAoFyYJ+agJOu6CE/hMhg LT672uLvkt4XD2eLE5yUYOHY7KkIKV6WfYPtyyamnoy9vpCPLTxlH6ZyRg+xt17D zP201nRf4Hyay6x7vi+cB4SZ1f5nUS8eV5hPrDZmLiIksdSqzkZFD/a5/JMsa07C esM43RAOa8/LxmJCiyqz =gMzK -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: undefined method
Hello Sean, If all the requests are originating from one browser then try running http watch or any other browser plugin of your preference to check the status codes for your request... Also another handy tool is nettool to sniff the requests.. Good luck! Regards, -Yogesh On Saturday, February 21, 2015, Sean Dawson wrote: > Ok thanks for the replies. It's weird that everything works fine except in > the case that there are a bunch of requests in a short period and that's > the only time we see this issue - if we debug, slow it down, etc - no > problem. Feels to me like a threading issue - but that can't be the case > on the (browser/js) client. > > We may also try with jboss to see if it reproduces there - probably won't > get to that before Monday though. > > I have no idea who would be adding the undefined - maybe RestyGwt? > > > On Fri, Feb 20, 2015 at 5:44 PM, Konstantin Kolinko < > knst.koli...@gmail.com > > wrote: > > > 2015-02-21 1:00 GMT+03:00 Sean Dawson >: > > > On Fri, Feb 20, 2015 at 4:41 PM, Konstantin Kolinko < > > knst.koli...@gmail.com > > > > wrote: > > > > > >> 2015-02-21 0:10 GMT+03:00 Sean Dawson >: > > >> > > > >> > ... > > >> > "undefinedPOST /gwtRequest HTTP/1.1" 501 1136 > > >> > > > >> > <...> > > >> > > >> > In fiddler, the headers are identical between the requests that work > > and > > >> > those that fail. > > >> > > >> The string in access log is not a header. It is HTTP request line. > > >> The first line of an HTTP request. > > >> > > >> > > > Ok, but this is in the standard tomcat access logs, using standard > > logging, > > > and is in the method name, not URL. Maybe I'm not understanding what > > > you're saying here. > > > > I mean that your phrase "the headers are identical" is irrelevant. > > The broken value is not in a header, but in the request line of an > > HTTP request. > > > > HTTP request = request line + CRLF + headers + CRLF CRLF + body > > > > > > > > > >> BTW, a similar issue at stackoverflow (but the "undefined" string was > > >> added to URL part of request line): > > >> > > >> > > >> > > > http://stackoverflow.com/questions/11017609/undefined-randomly-appended-in-1-of-requested-urls-on-my-website-since-12-jun > > >> Title: “undefined” randomly appended in 1% of requested urls on my > > >> website since 12 june 2012 > > >> > > >> > > > We did come across it but again our's is in the method, not in the URL. > > > > You are also using strings, concatenation, and javascript. > > > > >> > > >> One of theories there is that some browser addon was malfunctioning. > > >> > > >> > > > Ok, this has happened on about 5 people's machines with a couple > > different > > > versions of IE - I don't think we have any addons at all in some cases. > > > > Some addons are popular. Some people do not pay attention when > > installing 3rd party toolbars bundled with legit software installers. > > > > > > > >> If nothing else helps, it should be easy to implement a Valve for > > >> Tomcat that will fix the wrong request.getMethod() value before > > >> passing it to a web application. > > >> > > >> > > > I don't know much about that but we could give it a try - so > someone > > > else is changing the method somewhere before it gets to tomcat? and the > > > Valve will change it back? > > > > Yes. > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > >
