Hi everyone, I've been struggling with a problem for 4 days and I'm still unable to solve it. First I will describe the situation.
I've built a website with Asp.NET: .NET 4 MVC 3; No razor S#arp Architecture 1.9.6.0 MySQL I compile my project on Windows with Visual Studio 2010. I then publish it to a Ubuntu 10.04 x86 virtual machine with Mono 2.10.2 compiled from source. After making some changes to some 3rd party libs I got everything working. Then I started to load test with JMeter and I found out that Mono hangs after a few hundred requests when I'm using more than 1 thread in JMeter. The Mono process will hang with ~100% CPU load and after a minute or 2 it will return a 500 with Request timed out. The webserver won't process any more requests while it hangs. I tested this with Apache and xsp4, same results. I tried to reproduce this with a clean MVC 3 project but it didn't hang. It really seems to be something in S#arp Architecture. To ensure it wasn't because of that specific virtual machine I tried it with Mono 2.10.5 on another virtual machine. Same result. So I tried to find out why and where in the process it was hanging. I followed the steps from this article: http://www.mono-project.com/Debugging#Debugging_Managed_Lock_Deadlocks because I thought it was because of a deadlock. I attached gdb to the hanging process and tried some of the commands listed on the page. Executing "call mono_locks_dump (0)" usually returns this: Total locks (in 6 array(s)): 1008, used: 391, on freelist: 2, to recycle: 614 or this: Total locks (in 6 array(s)): 1008, used: 583, on freelist: 93, to recycle: 332 Executing "thread apply all bt" will return a very long list of about 50 threads, some waiting for a lock but most of them are similar to: Thread 1 (Thread 0xb7725b40 (LWP 2832)): #0 0xb78cf430 in __kernel_vsyscall () #1 0xb788cf5b in read () from /lib/tls/i686/cmov/libpthread.so.0 #2 0x081f9183 in read (handle=0x0, buffer=0x1ef010, numbytes=1024, bytesread=0xbfbf656c, overlapped=0x0) at /usr/include/bits/unistd.h:45 #3 console_read (handle=0x0, buffer=0x1ef010, numbytes=1024, bytesread=0xbfbf656c, overlapped=0x0) at io.c:1109 #4 0x081fa86d in ReadFile (handle=0xfffffe00, buffer=0x1ef010, numbytes=1024, bytesread=0xbfbf656c, overlapped=0x0) at io.c:2314 #5 0x0823ab9e in ves_icall_System_IO_MonoIO_Read (handle=0x0, dest=0x1ef000, dest_offset=0, count=1024, error= 0xbfbf6600) at file-io.c:838 #6 0xb665832a in ?? () #7 0xb6658270 in ?? () #8 0xb6657cb4 in ?? () #9 0xb6657ab2 in ?? () #10 0xb66577f3 in ?? () #11 0xb66599a4 in ?? () #12 0xb6560197 in ?? () #13 0xb655e56a in ?? () #14 0xb655e48a in ?? () #15 0xb655e436 in ?? () #16 0xb65533bc in ?? () #17 0xb6551e0c in ?? () #18 0xb6551c80 in ?? () #19 0xb65dbf84 in ?? () #20 0xb65dbdfc in ?? () #21 0xb714d7ae in ?? () #22 0xb714be60 in ?? () #23 0xb714bf94 in ?? () #24 0x08062c78 in mono_jit_runtime_invoke (method="Mono.WebServer.XSP.Server:Main ()", obj=0x0, params=0xbfbf6f1c, exc= 0x0) at mini.c:5751 #25 0x08171a20 in mono_runtime_invoke (method="Mono.WebServer.XSP.Server:Main ()", obj=0x0, params=0xbfbf6f1c, exc=0x0) at object.c:2754 #26 0x08175773 in mono_runtime_exec_main (method="Mono.WebServer.XSP.Server:Main ()", args=0x3be00, exc=0x0) at object.c:3929 #27 0x080bb0a5 in main_thread_handler (argc=3, argv=0xbfbf7104) at driver.c:1003 #28 mono_main (argc=3, argv=0xbfbf7104) at driver.c:1855 #29 0x0805944a in mono_main_with_options (argc=3, argv=0xbfbf7104) at main.c:66 #30 main (argc=3, argv=0xbfbf7104) at main.c:97 I'm actually very interested what's going on at the steps 6-23. I guess these are the calls in my application. Seems gdb is unable to use the pdb files I supply from Visual Studio. Running pdb2mdb on my DLL's gives a BadImageFormatException. I fiddled around a bit with the settings from http://www.mono-project.com/ThreadPool_DeadLocks but no luck. Back to Windows... I installed the latest Mono 2.10.6 on my Windows machine and ran my website with xsp4. With 1 thread in JMeter it doesn't hang, with 2 or more it does after some requests. 2 threads will make it hang at 50%, 3 thread 75% and so on, I have a quad core machine. Because of all the threads hanging at the same time I'm very eager to say it's a deadlock or something. I tried to find out what was going wrong by attaching Visual Studio to the hanging process but it only gives disassembled code. Process explorer did give me a lot of insight. If I make Mono hang with 8 threads in JMeter and inspect the process I get these results: 18 threads with 8 of them having >1% cpu usage. These 8 are stuck here: mono-2.0.dll!mono_thread_interruption_requested+0x4e mono-2.0.dll!mono_get_special_static_data+0x1a mono-2.0.dll!mono_error_raise_exception+0x12559 mono-2.0.dll!mono_get_special_static_data+0x1a mono-2.0.dll!mono_get_special_static_data+0x1a mono-2.0.dll!mono_get_special_static_data+0x1a mono-2.0.dll!mono_get_special_static_data+0x1a mono-2.0.dll!mono_get_special_static_data+0x1a 9 threads are at wow64cpu.dll!TurboDispatchJumpAddressEnd+0x6c0 1 thread is at ntdll.dll!NtRequestWaitReplyPort+0xa But I don't know what to do with this. The Development Server distributed with Visual Studio actually never hangs. I haven't tested Windows Server yet. So my problem is a hanging webserver and I don't know why it's happening. Could you help me with troubleshooting and solving this? I hope I've given enough information. I could provide a full report of "thread apply all bt" if you'd like. Thank you for your time! _______________________________________________ Mono-list maillist - [email protected] http://lists.ximian.com/mailman/listinfo/mono-list
