Re: [Resin-interest] com.caucho.vfs.JniStream.readNative flakiness? [Was: Another weird OpenSSL/Resin Error.]
On Dec 8, 2006, at 18:43 , Scott Ferguson wrote: > On Dec 8, 2006, at 6:00 PM, Rob Lockstone wrote: > >>> We can add the OpenSSL version we tested with. That's a good idea. >>> We can't distribute the dll itself. >> >> Damn. :-( Well, can you distribute the source AND your particular >> build flags, etc? That should allow us to build the dll with the same >> source/process and hopefully obtain the exact same dll's. > > I've been using the shining light build: http://www.slproweb.com/ > products/Win32OpenSSL.html. Yes, that's exactly the one I'm using. Except the 'b' version. I tried using 'd' with 3.0.21 and it blew up, wouldn't even load the dll's at startup. Have you used 'd' with .21? If so, which dll's and where do you place them? Currently, I place libeay32.dll, libssl32.dll, and ssleay32.dll (the latter two appear to be identical, at least in size) into the resin home directory. Rob > > If you're rebuilding openssl, you need to make sure you've set the > threaded flag (although looking at their docs, it looks like it's set > automatically in windows.) It used to be that openssl's default > build would compile in non-multithreaded on unix. > > The build info for resinssl should be part of the pro distribution in > modules/c/resinssl if that's helpful tracking this down. > > -- Scott >> >> Rob >> >>> I'll see if I can figure out how to reproduce it. It may be a bit >>> tricky to track down, though. Java's thread dumps are so much >>> nicer. >>> >>> -- Scott >>> Rob On Dec 8, 2006, at 09:10 , Scott Ferguson wrote: > > On Dec 8, 2006, at 8:56 AM, Rob Lockstone wrote: > >> Environment: Windows 2003 Server, Resin Pro 3.0.21, OpenSSL >> 0.9.8b, >> Java 1.4.2_12 >> >> More information on what caused the JVM/Resin to spontaneously >> reboot. I get these fairly often (multiple times per month). >> They're not always the same exact thing. Often the stack trace >> references one of the SSL DLL's, but other times, like this >> one, it >> references the JVM DLL. >> >> In any event, since the "Java frames" section references some >> Caucho JNI calls, I'm sending this in. Scott, let me know if you >> want an official bug filed. > > That would be great. > >> >> I'm attaching two files to this email since the emailer likes to >> reformat stuff into unreadable chunks. One is the weird >> SSL_UNDEFINED_FUNCTION error I have never seen before and >> reported >> a few minutes. The other is the top portion of the JVM crash log >> where the jvm.dll was the source of the crash and Caucho JNI >> routines are being called. >> >> Btw, whenever the crash log mentions an SLL DLL as the source of >> the crash, the "Java frames" section is always something along >> these lines: >> >> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) >> j com.caucho.vfs.JniStream.readNative(J[BII)I+0 >> j com.caucho.vfs.JniStream.read([BII)I+43 >> j com.caucho.vfs.ReadStream.readBuffer()Z+53 >> j com.caucho.vfs.ReadStream.waitForRead()Z+12 >> j com.caucho.server.port.TcpConnection.run()V+181 >> j com.caucho.util.ThreadPool.runTasks()V+187 >> j com.caucho.util.ThreadPool.run()V+85 >> j java.lang.Thread.run()V+11 >> v ~StubRoutines::call_stub >> Which is similar to the attached JVM DLL crash. What's up with >> com.caucho.vfs.JniStream.readNative ? > > readNative itself should be solid. We've run it for days under > very > heavy load (CPU load around 40 on multi-cpu systems) with no > issues. > > But there has been less time stressing openssl, so there might be > something odd there. > > -- Scott > >> >> Rob >> >> >> >> >> >> ___ >> resin-interest mailing list >> resin-interest@caucho.com >> http://maillist.caucho.com/mailman/listinfo/resin-interest > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest >>> >>> >>> ___ >>> resin-interest mailing list >>> resin-interest@caucho.com >>> http://maillist.caucho.com/mailman/listinfo/resin-interest >> >> >> ___ >> resin-interest mailing list >> resin-interest@caucho.com >> http://maillist.caucho.com/mailman/listinfo/resin-interest > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest _
Re: [Resin-interest] com.caucho.vfs.JniStream.readNative flakiness? [Was: Another weird OpenSSL/Resin Error.]
On Dec 8, 2006, at 6:00 PM, Rob Lockstone wrote: >> >> We can add the OpenSSL version we tested with. That's a good idea. >> We can't distribute the dll itself. > > Damn. :-( Well, can you distribute the source AND your particular > build flags, etc? That should allow us to build the dll with the same > source/process and hopefully obtain the exact same dll's. I've been using the shining light build: http://www.slproweb.com/ products/Win32OpenSSL.html. If you're rebuilding openssl, you need to make sure you've set the threaded flag (although looking at their docs, it looks like it's set automatically in windows.) It used to be that openssl's default build would compile in non-multithreaded on unix. The build info for resinssl should be part of the pro distribution in modules/c/resinssl if that's helpful tracking this down. -- Scott > > Rob > >> I'll see if I can figure out how to reproduce it. It may be a bit >> tricky to track down, though. Java's thread dumps are so much nicer. >> >> -- Scott >> >>> >>> Rob >>> >>> >>> >>> >>> On Dec 8, 2006, at 09:10 , Scott Ferguson wrote: >>> On Dec 8, 2006, at 8:56 AM, Rob Lockstone wrote: > Environment: Windows 2003 Server, Resin Pro 3.0.21, OpenSSL > 0.9.8b, > Java 1.4.2_12 > > More information on what caused the JVM/Resin to spontaneously > reboot. I get these fairly often (multiple times per month). > They're not always the same exact thing. Often the stack trace > references one of the SSL DLL's, but other times, like this > one, it > references the JVM DLL. > > In any event, since the "Java frames" section references some > Caucho JNI calls, I'm sending this in. Scott, let me know if you > want an official bug filed. That would be great. > > I'm attaching two files to this email since the emailer likes to > reformat stuff into unreadable chunks. One is the weird > SSL_UNDEFINED_FUNCTION error I have never seen before and reported > a few minutes. The other is the top portion of the JVM crash log > where the jvm.dll was the source of the crash and Caucho JNI > routines are being called. > > Btw, whenever the crash log mentions an SLL DLL as the source of > the crash, the "Java frames" section is always something along > these lines: > > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > j com.caucho.vfs.JniStream.readNative(J[BII)I+0 > j com.caucho.vfs.JniStream.read([BII)I+43 > j com.caucho.vfs.ReadStream.readBuffer()Z+53 > j com.caucho.vfs.ReadStream.waitForRead()Z+12 > j com.caucho.server.port.TcpConnection.run()V+181 > j com.caucho.util.ThreadPool.runTasks()V+187 > j com.caucho.util.ThreadPool.run()V+85 > j java.lang.Thread.run()V+11 > v ~StubRoutines::call_stub > Which is similar to the attached JVM DLL crash. What's up with > com.caucho.vfs.JniStream.readNative ? readNative itself should be solid. We've run it for days under very heavy load (CPU load around 40 on multi-cpu systems) with no issues. But there has been less time stressing openssl, so there might be something odd there. -- Scott > > Rob > > > > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest >>> >>> ___ >>> resin-interest mailing list >>> resin-interest@caucho.com >>> http://maillist.caucho.com/mailman/listinfo/resin-interest >> >> >> ___ >> resin-interest mailing list >> resin-interest@caucho.com >> http://maillist.caucho.com/mailman/listinfo/resin-interest > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] com.caucho.vfs.JniStream.readNative flakiness? [Was: Another weird OpenSSL/Resin Error.]
On Dec 8, 2006, at 12:45 PM, Scott Ferguson wrote: > On Dec 8, 2006, at 10:34 AM, Rob Lockstone wrote: > >> Fwiw, I just restarted a server and got three of these SSL-related >> crashes in a row. It's just really, really bad when you try to >> restart a server and add it to a pool of active servers and it >> starts taking requests. Just awful. > > So it sounds like there might be a startup synchronization issue. > Your previous case shows something similar. Unfortunately, it doesn't always happen at start up. The server can run for hours or days or even weeks, and then, blame, it happens. However, it does seem to happen more frequently just after resin is restarted, so that's certainly a place to start. >> Again, I'd like to request that, at least for Windows, Caucho >> includes its own "we trust this build of OpenSSL DLL's for use with >> Resin". The constant crashing is painful to watch. And since I'm >> the "resin guy", even though I'm not a sysadmin, I get blamed when >> everything goes to pot. It's not fun. :-( > > We can add the OpenSSL version we tested with. That's a good idea. > We can't distribute the dll itself. Damn. :-( Well, can you distribute the source AND your particular build flags, etc? That should allow us to build the dll with the same source/process and hopefully obtain the exact same dll's. Rob > I'll see if I can figure out how to reproduce it. It may be a bit > tricky to track down, though. Java's thread dumps are so much nicer. > > -- Scott > >> >> Rob >> >> >> >> >> On Dec 8, 2006, at 09:10 , Scott Ferguson wrote: >> >>> >>> On Dec 8, 2006, at 8:56 AM, Rob Lockstone wrote: >>> Environment: Windows 2003 Server, Resin Pro 3.0.21, OpenSSL 0.9.8b, Java 1.4.2_12 More information on what caused the JVM/Resin to spontaneously reboot. I get these fairly often (multiple times per month). They're not always the same exact thing. Often the stack trace references one of the SSL DLL's, but other times, like this one, it references the JVM DLL. In any event, since the "Java frames" section references some Caucho JNI calls, I'm sending this in. Scott, let me know if you want an official bug filed. >>> >>> That would be great. >>> I'm attaching two files to this email since the emailer likes to reformat stuff into unreadable chunks. One is the weird SSL_UNDEFINED_FUNCTION error I have never seen before and reported a few minutes. The other is the top portion of the JVM crash log where the jvm.dll was the source of the crash and Caucho JNI routines are being called. Btw, whenever the crash log mentions an SLL DLL as the source of the crash, the "Java frames" section is always something along these lines: Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j com.caucho.vfs.JniStream.readNative(J[BII)I+0 j com.caucho.vfs.JniStream.read([BII)I+43 j com.caucho.vfs.ReadStream.readBuffer()Z+53 j com.caucho.vfs.ReadStream.waitForRead()Z+12 j com.caucho.server.port.TcpConnection.run()V+181 j com.caucho.util.ThreadPool.runTasks()V+187 j com.caucho.util.ThreadPool.run()V+85 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub Which is similar to the attached JVM DLL crash. What's up with com.caucho.vfs.JniStream.readNative ? >>> >>> readNative itself should be solid. We've run it for days under very >>> heavy load (CPU load around 40 on multi-cpu systems) with no issues. >>> >>> But there has been less time stressing openssl, so there might be >>> something odd there. >>> >>> -- Scott >>> Rob ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest >>> >>> >>> ___ >>> resin-interest mailing list >>> resin-interest@caucho.com >>> http://maillist.caucho.com/mailman/listinfo/resin-interest >> >> ___ >> resin-interest mailing list >> resin-interest@caucho.com >> http://maillist.caucho.com/mailman/listinfo/resin-interest > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] com.caucho.vfs.JniStream.readNative flakiness? [Was: Another weird OpenSSL/Resin Error.]
On Dec 8, 2006, at 10:34 AM, Rob Lockstone wrote: > Fwiw, I just restarted a server and got three of these SSL-related > crashes in a row. It's just really, really bad when you try to > restart a server and add it to a pool of active servers and it > starts taking requests. Just awful. So it sounds like there might be a startup synchronization issue. Your previous case shows something similar. > Again, I'd like to request that, at least for Windows, Caucho > includes its own "we trust this build of OpenSSL DLL's for use with > Resin". The constant crashing is painful to watch. And since I'm > the "resin guy", even though I'm not a sysadmin, I get blamed when > everything goes to pot. It's not fun. :-( We can add the OpenSSL version we tested with. That's a good idea. We can't distribute the dll itself. I'll see if I can figure out how to reproduce it. It may be a bit tricky to track down, though. Java's thread dumps are so much nicer. -- Scott > > Rob > > > > > On Dec 8, 2006, at 09:10 , Scott Ferguson wrote: > >> >> On Dec 8, 2006, at 8:56 AM, Rob Lockstone wrote: >> >>> Environment: Windows 2003 Server, Resin Pro 3.0.21, OpenSSL 0.9.8b, >>> Java 1.4.2_12 >>> >>> More information on what caused the JVM/Resin to spontaneously >>> reboot. I get these fairly often (multiple times per month). >>> They're not always the same exact thing. Often the stack trace >>> references one of the SSL DLL's, but other times, like this one, it >>> references the JVM DLL. >>> >>> In any event, since the "Java frames" section references some >>> Caucho JNI calls, I'm sending this in. Scott, let me know if you >>> want an official bug filed. >> >> That would be great. >> >>> >>> I'm attaching two files to this email since the emailer likes to >>> reformat stuff into unreadable chunks. One is the weird >>> SSL_UNDEFINED_FUNCTION error I have never seen before and reported >>> a few minutes. The other is the top portion of the JVM crash log >>> where the jvm.dll was the source of the crash and Caucho JNI >>> routines are being called. >>> >>> Btw, whenever the crash log mentions an SLL DLL as the source of >>> the crash, the "Java frames" section is always something along >>> these lines: >>> >>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) >>> j com.caucho.vfs.JniStream.readNative(J[BII)I+0 >>> j com.caucho.vfs.JniStream.read([BII)I+43 >>> j com.caucho.vfs.ReadStream.readBuffer()Z+53 >>> j com.caucho.vfs.ReadStream.waitForRead()Z+12 >>> j com.caucho.server.port.TcpConnection.run()V+181 >>> j com.caucho.util.ThreadPool.runTasks()V+187 >>> j com.caucho.util.ThreadPool.run()V+85 >>> j java.lang.Thread.run()V+11 >>> v ~StubRoutines::call_stub >>> Which is similar to the attached JVM DLL crash. What's up with >>> com.caucho.vfs.JniStream.readNative ? >> >> readNative itself should be solid. We've run it for days under very >> heavy load (CPU load around 40 on multi-cpu systems) with no issues. >> >> But there has been less time stressing openssl, so there might be >> something odd there. >> >> -- Scott >> >>> >>> Rob >>> >>> >>> >>> >>> >>> ___ >>> resin-interest mailing list >>> resin-interest@caucho.com >>> http://maillist.caucho.com/mailman/listinfo/resin-interest >> >> >> ___ >> resin-interest mailing list >> resin-interest@caucho.com >> http://maillist.caucho.com/mailman/listinfo/resin-interest > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] com.caucho.vfs.JniStream.readNative flakiness? [Was: Another weird OpenSSL/Resin Error.]
Fwiw, I just restarted a server and got three of these SSL-related crashes in a row. It's just really, really bad when you try to restart a server and add it to a pool of active servers and it starts taking requests. Just awful. Again, I'd like to request that, at least for Windows, Caucho includes its own "we trust this build of OpenSSL DLL's for use with Resin". The constant crashing is painful to watch. And since I'm the "resin guy", even though I'm not a sysadmin, I get blamed when everything goes to pot. It's not fun. :-( Rob SSLdllCrash.log Description: Binary data On Dec 8, 2006, at 09:10 , Scott Ferguson wrote: On Dec 8, 2006, at 8:56 AM, Rob Lockstone wrote: Environment: Windows 2003 Server, Resin Pro 3.0.21, OpenSSL 0.9.8b, Java 1.4.2_12 More information on what caused the JVM/Resin to spontaneously reboot. I get these fairly often (multiple times per month). They're not always the same exact thing. Often the stack trace references one of the SSL DLL's, but other times, like this one, it references the JVM DLL. In any event, since the "Java frames" section references some Caucho JNI calls, I'm sending this in. Scott, let me know if you want an official bug filed. That would be great. I'm attaching two files to this email since the emailer likes to reformat stuff into unreadable chunks. One is the weird SSL_UNDEFINED_FUNCTION error I have never seen before and reported a few minutes. The other is the top portion of the JVM crash log where the jvm.dll was the source of the crash and Caucho JNI routines are being called. Btw, whenever the crash log mentions an SLL DLL as the source of the crash, the "Java frames" section is always something along these lines: Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j com.caucho.vfs.JniStream.readNative(J[BII)I+0 j com.caucho.vfs.JniStream.read([BII)I+43 j com.caucho.vfs.ReadStream.readBuffer()Z+53 j com.caucho.vfs.ReadStream.waitForRead()Z+12 j com.caucho.server.port.TcpConnection.run()V+181 j com.caucho.util.ThreadPool.runTasks()V+187 j com.caucho.util.ThreadPool.run()V+85 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub Which is similar to the attached JVM DLL crash. What's up with com.caucho.vfs.JniStream.readNative ? readNative itself should be solid. We've run it for days under very heavy load (CPU load around 40 on multi-cpu systems) with no issues. But there has been less time stressing openssl, so there might be something odd there. -- Scott Rob ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] com.caucho.vfs.JniStream.readNative flakiness? [Was: Another weird OpenSSL/Resin Error.]
On Dec 8, 2006, at 09:10 , Scott Ferguson wrote: > > On Dec 8, 2006, at 8:56 AM, Rob Lockstone wrote: > >> Environment: Windows 2003 Server, Resin Pro 3.0.21, OpenSSL 0.9.8b, >> Java 1.4.2_12 >> >> More information on what caused the JVM/Resin to spontaneously >> reboot. I get these fairly often (multiple times per month). >> They're not always the same exact thing. Often the stack trace >> references one of the SSL DLL's, but other times, like this one, it >> references the JVM DLL. >> >> In any event, since the "Java frames" section references some >> Caucho JNI calls, I'm sending this in. Scott, let me know if you >> want an official bug filed. > > That would be great. > Ok, I reported it. Looks like Mantis took it twice. The first time, it complained that the file I was trying to upload wasn't valid. So I created a new report. But once I submitted that one, it took me to the list of issues and I saw that it had created two entries for the same bug. The numbers are 1499 and 1500. Feel free to remove one of them as a dup. Rob ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] com.caucho.vfs.JniStream.readNative flakiness? [Was: Another weird OpenSSL/Resin Error.]
On Dec 8, 2006, at 8:56 AM, Rob Lockstone wrote: > Environment: Windows 2003 Server, Resin Pro 3.0.21, OpenSSL 0.9.8b, > Java 1.4.2_12 > > More information on what caused the JVM/Resin to spontaneously > reboot. I get these fairly often (multiple times per month). > They're not always the same exact thing. Often the stack trace > references one of the SSL DLL's, but other times, like this one, it > references the JVM DLL. > > In any event, since the "Java frames" section references some > Caucho JNI calls, I'm sending this in. Scott, let me know if you > want an official bug filed. That would be great. > > I'm attaching two files to this email since the emailer likes to > reformat stuff into unreadable chunks. One is the weird > SSL_UNDEFINED_FUNCTION error I have never seen before and reported > a few minutes. The other is the top portion of the JVM crash log > where the jvm.dll was the source of the crash and Caucho JNI > routines are being called. > > Btw, whenever the crash log mentions an SLL DLL as the source of > the crash, the "Java frames" section is always something along > these lines: > > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > j com.caucho.vfs.JniStream.readNative(J[BII)I+0 > j com.caucho.vfs.JniStream.read([BII)I+43 > j com.caucho.vfs.ReadStream.readBuffer()Z+53 > j com.caucho.vfs.ReadStream.waitForRead()Z+12 > j com.caucho.server.port.TcpConnection.run()V+181 > j com.caucho.util.ThreadPool.runTasks()V+187 > j com.caucho.util.ThreadPool.run()V+85 > j java.lang.Thread.run()V+11 > v ~StubRoutines::call_stub > Which is similar to the attached JVM DLL crash. What's up with > com.caucho.vfs.JniStream.readNative ? readNative itself should be solid. We've run it for days under very heavy load (CPU load around 40 on multi-cpu systems) with no issues. But there has been less time stressing openssl, so there might be something odd there. -- Scott > > Rob > > > > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
