Re: [Mono-dev] Shutdown hangs

2016-07-12 Thread Chris Swiedler 
Switching to mono 4.0.5 seems a little better, but the problem (or a similar 
problem) is still there, and I get segfaults and other problems with 4.0.

Does anyone have any suggestions for brute-force ways to fix this problem? 
E.g., can I modify ves_icall_System_Environment_Exit or similar to just call 
exit()? I haven't looked at the code but from the call stacks it seems like 
this happens in an attempt to make an orderly shutdown of threads. While that 
would be nice, it's much more important to me for the application to actually 
exit. If I have to, I'll make the process kill itself with SIGKILL, but I'd 
rather not since then I lose the actual exit code.

chris

-Original Message-
From: mono-devel-list-boun...@lists.ximian.com 
[mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Chris Swiedler
Sent: Monday, June 13, 2016 9:55 AM
To: mono-devel-list <mono-devel-list@lists.ximian.com>
Subject: [Mono-dev] Shutdown hangs

I'm getting semi-reliable shutdown hangs in a mono command-line app using mono 
4.2.3 on Centos 7 64-bit. The app does different things at different times, and 
the shutdowns only seem to happen when it connects to our SQL server database 
(using the Mono database bindings, not ODBC). We've also seen an occasional 
segfault, also seemingly only when doing database work, though those may have 
gone away since upgrading to 4.2.3.

Does anyone have any thoughts? Pstack output below.

Thanks,
chris


Thread 5 (Thread 0x7f4c777ff700 (LWP 11658)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x005f786c in thread_func (thread_data=0x0) at 
sgen-thread-pool.c:118
#2  0x7f4c77ffedf5 in start_thread (arg=0x7f4c777ff700) at 
pthread_create.c:308
#3  0x7f4c77d2c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 4 (Thread 0x7f4c75560700 (LWP 11659)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061aa37 in mono_sem_wait (sem=sem@entry=0x953c40 
, alertable=alertable@entry=1) at mono-semaphore.c:107
#2  0x005a1736 in finalizer_thread (unused=) at gc.c:1096
#3  0x005841e9 in start_wrapper_internal (data=) at 
threads.c:725
#4  start_wrapper (data=) at threads.c:772
#5  0x00621026 in inner_start_thread (arg=0x7fffda0d42e0) at 
mono-threads-posix.c:97
#6  0x7f4c77ffedf5 in start_thread (arg=0x7f4c75560700) at 
pthread_create.c:308
#7  0x7f4c77d2c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f4c74fd3700 (LWP 11660)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c68000940, 
alertable=alertable@entry=0) at mono-semaphore.c:107
#2  0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110
#3  mono_thread_info_end_self_suspend () at mono-threads.c:692
#4  0x00583d2c in self_suspend_internal (thread=0x7f4c78a74330) at 
threads.c:4546
#5  mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74330) at 
threads.c:4050
#6  0x00583f50 in mono_thread_interruption_checkpoint_request 
(bypass_abort_protection=bypass_abort_protection@entry=1) at threads.c:4184
#7  0x00584dae in mono_thread_force_interruption_checkpoint () at 
threads.c:4221

Thread 2 (Thread 0x7f4c746ff700 (LWP 11661)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c6c000940, 
alertable=alertable@entry=0) at mono-semaphore.c:107
#2  0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110
#3  mono_thread_info_end_self_suspend () at mono-threads.c:692
#4  0x00583d2c in self_suspend_internal (thread=0x7f4c78a74430) at 
threads.c:4546
#5  mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74430) at 
threads.c:4050
#6  0x005851f9 in ves_icall_System_Threading_Thread_Sleep_internal 
(ms=1000) at threads.c:1206
#17 0x7f4c756fa2e7 in System_Threading_Thread_Sleep_int 
(millisecondsTimeout=1000) at 
/home/cswiedler/mono-main/Extern/mono/source/external/referencesource/mscorlib/system/threading/thread.cs:715

Thread 1 (Thread 0x7f4c78b34780 (LWP 11657)):
#0  0x7f4c7800599d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00632a5a in monoeg_g_usleep (microseconds=1935890) at 
gdate-unix.c:53
#2  0x0061f2bb in suspend_sync_nolock (id=id@entry=139966347736832, 
interrupt_kernel=interrupt_kernel@entry=1) at mono-threads.c:870
#3  0x0061f732 in mono_thread_info_safe_suspend_and_run 
(id=139966347736832, interrupt_kernel=interrupt_kernel@entry=1, 
callback=callback@entry=0x5813c0 , 
user_data=user_data@entry=0x7fffda0d3a10) at mono-threads.c:891
#4  0x00581d29 in suspend_thread_internal (thread=0x7f4c78a74430, 
interrupt=1) at threads.c:4531
#5  0x00

Re: [Mono-dev] Shutdown hangs

2016-07-08 Thread Chris Swiedler 
I've switched to 4.0.5, and the hangs are less frequent (though they still 
happen). Now, though, we're getting this shutdown crash. Is it possibly fixed 
by a later patch?

Full stack attached. Short version:

sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85movq  %rax, %rcx
Id   Target Id Frame
14   Thread 0x7f7d3726b700 (LWP 19847) "Finalizer" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
13   Thread 0x7f7d279cc700 (LWP 19882) "Threadpool moni" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
12   Thread 0x7f7d2757d700 (LWP 19890) "IO Threadpool w" 0x7f7d3de26cd9 in 
__libc_waitpid (pid=pid@entry=24339, stat_loc=stat_loc@entry=0x7f7d2753c19c, 
options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40
11   Thread 0x7f7d268ff700 (LWP 19903) "Threadpool work" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
10   Thread 0x7f7d25dfe700 (LWP 19916) "Threadpool work" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
9Thread 0x7f7d26d7a700 (LWP 19920) "Threadpool work" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
8Thread 0x7f7cf79e3700 (LWP 19940) "hydrogen_main" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
7Thread 0x7f7cf77e2700 (LWP 19941) "Timer-Scheduler" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
6Thread 0x7f7cf6dff700 (LWP 19951) "Threadpool work" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
5Thread 0x7f7cf5bfd700 (LWP 19967) "Threadpool work" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
4Thread 0x7f7cf59fc700 (LWP 19971) "Threadpool work" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
3Thread 0x7f7cf57fb700 (LWP 19978) "Threadpool work" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
2Thread 0x7f7cf55fa700 (LWP 19980) "Threadpool work" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
* 1Thread 0x7f7d3e955780 (LWP 19843) "LobbyServer" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
...
Thread 1 (Thread 0x7f7d3e955780 (LWP 19843)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061def7 in mono_sem_wait (sem=sem@entry=0x7f7d04000970, 
alertable=alertable@entry=0) at mono-semaphore.c:101
#2  0x006230ca in mono_threads_core_suspend 
(info=info@entry=0x7f7d040008e0, interrupt_kernel=interrupt_kernel@entry=1) at 
mono-threads-posix.c:418
#3  0x00622513 in mono_thread_info_suspend_sync (error_condition=, 
interrupt_kernel=1, tid=140175507707648) at mono-threads.c:474
#4  mono_thread_info_safe_suspend_sync (id=140175507707648, 
interrupt_kernel=interrupt_kernel@entry=1) at mono-threads.c:652
#5  0x00589705 in suspend_thread_internal (thread=0x7f7d37312940, 
interrupt=1) at threads.c:4707
#6  0x00589bdd in mono_thread_suspend_all_other_threads () at 
threads.c:3142
#7  0x0053c0f6 in ves_icall_System_Environment_Exit (result=0) at 
icall.c:6471

-Original Message-
From: David Evans [mailto:dev...@pacificbiosciences.com] 
Sent: Monday, June 13, 2016 12:40 PM
To: Chris Swiedler <cswied...@trionworlds.com>; mono-devel-list 
<mono-devel-list@lists.ximian.com>
Subject: RE: Shutdown hangs

I saw something similar when I was testing 4.2.2 awhile back, a rare hang on 
Exit with the main thread waiting like yours. I haven't tried 4.2.3 to see if 
our hang reproduces yet. We're still on 4.0.4 and we don't see it there.

In the hang we saw, I also had one thread in pthread_cond_wait like yours, 
though we had two threads also in poll() with 
libusb_handle_events_timeout_completed () further up the stack.

(gdb) info threads
  Id   Target Id Frame
  16   Thread 0x7f5eb0bff700 (LWP 2763) "mono" pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  15   Thread 0x7f5eae7e1700 (LWP 2765) "Finalizer" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  14   Thread 0x7f5ead26b700 (LWP 2784) "Timer-Scheduler" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
 13   Thread 0x7f5eacf2f700 (LWP 2806) "Top" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
  12   Thread 0x7f5eacd2a700 (LWP 2807) "Top" 0x7f5eb1b48b9d in nanosleep 
() at ../sysdeps/unix/syscall-template.S:81
  11   Thread 0x7f5e8e07b700 (LWP 2840) "Top" sem_wait () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
* 10   Thread 0x7f5e72fa5700 (LWP 2980) "Top" 0x7f5eb186112d in poll () at 
../sysdeps/unix/syscall-template.S:81
  9Thread 0x7f5e727a4700 (LWP 2981) "Top

Re: [Mono-list] Mono C# Console Application dies unexpectedly every ~20 minuts

2016-06-14 Thread Chris Swiedler 
What is the exit code of the process? If it's in the 128-150 range then it's 
receiving a signal.

chris

-Original Message-
From: mono-list-boun...@lists.ximian.com 
[mailto:mono-list-boun...@lists.ximian.com] On Behalf Of akurilo
Sent: Tuesday, June 14, 2016 10:23 AM
To: mono-list@lists.ximian.com
Subject: [Mono-list] Mono C# Console Application dies unexpectedly every ~20 
minuts

Hi, i've developed an application for raspberry pi (wheezy) using C# .Net it 
has a lot of code and about 5 active threads all the time, i've set up event 
handlers for ExitProcess, UnhandeledException, whatever events, but every 20 
minutes my app dies with no warning or any message.

I've spent hours trying to find any information about how the app died, but 
haven't found anything in system logs. So i'm wondering if someone knows about 
if mono can kill the process by some reason.

P.S. My process uses 2.5% of memory and ususally no CPU. It has an infinity 
while loop where it does all the mess, but it doesn't seem to be reason why it 
should be killed.

Thanks



--
View this message in context: 
http://mono.1490590.n4.nabble.com/Mono-C-Console-Application-dies-unexpectedly-every-20-minuts-tp4667992.html
Sent from the Mono - General mailing list archive at Nabble.com.
___
Mono-list maillist  -  Mono-list@lists.ximian.com 
http://lists.ximian.com/mailman/listinfo/mono-list
___
Mono-list maillist  -  Mono-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-list

[Mono-dev] Shutdown hangs

2016-06-13 Thread Chris Swiedler 
I'm getting semi-reliable shutdown hangs in a mono command-line app using mono 
4.2.3 on Centos 7 64-bit. The app does different things at different times, and 
the shutdowns only seem to happen when it connects to our SQL server database 
(using the Mono database bindings, not ODBC). We've also seen an occasional 
segfault, also seemingly only when doing database work, though those may have 
gone away since upgrading to 4.2.3.

Does anyone have any thoughts? Pstack output below.

Thanks,
chris


Thread 5 (Thread 0x7f4c777ff700 (LWP 11658)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x005f786c in thread_func (thread_data=0x0) at 
sgen-thread-pool.c:118
#2  0x7f4c77ffedf5 in start_thread (arg=0x7f4c777ff700) at 
pthread_create.c:308
#3  0x7f4c77d2c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 4 (Thread 0x7f4c75560700 (LWP 11659)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061aa37 in mono_sem_wait (sem=sem@entry=0x953c40 
, alertable=alertable@entry=1) at mono-semaphore.c:107
#2  0x005a1736 in finalizer_thread (unused=) at gc.c:1096
#3  0x005841e9 in start_wrapper_internal (data=) at 
threads.c:725
#4  start_wrapper (data=) at threads.c:772
#5  0x00621026 in inner_start_thread (arg=0x7fffda0d42e0) at 
mono-threads-posix.c:97
#6  0x7f4c77ffedf5 in start_thread (arg=0x7f4c75560700) at 
pthread_create.c:308
#7  0x7f4c77d2c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f4c74fd3700 (LWP 11660)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c68000940, 
alertable=alertable@entry=0) at mono-semaphore.c:107
#2  0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110
#3  mono_thread_info_end_self_suspend () at mono-threads.c:692
#4  0x00583d2c in self_suspend_internal (thread=0x7f4c78a74330) at 
threads.c:4546
#5  mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74330) at 
threads.c:4050
#6  0x00583f50 in mono_thread_interruption_checkpoint_request 
(bypass_abort_protection=bypass_abort_protection@entry=1) at threads.c:4184
#7  0x00584dae in mono_thread_force_interruption_checkpoint () at 
threads.c:4221

Thread 2 (Thread 0x7f4c746ff700 (LWP 11661)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c6c000940, 
alertable=alertable@entry=0) at mono-semaphore.c:107
#2  0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110
#3  mono_thread_info_end_self_suspend () at mono-threads.c:692
#4  0x00583d2c in self_suspend_internal (thread=0x7f4c78a74430) at 
threads.c:4546
#5  mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74430) at 
threads.c:4050
#6  0x005851f9 in ves_icall_System_Threading_Thread_Sleep_internal 
(ms=1000) at threads.c:1206
#17 0x7f4c756fa2e7 in System_Threading_Thread_Sleep_int 
(millisecondsTimeout=1000) at 
/home/cswiedler/mono-main/Extern/mono/source/external/referencesource/mscorlib/system/threading/thread.cs:715

Thread 1 (Thread 0x7f4c78b34780 (LWP 11657)):
#0  0x7f4c7800599d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00632a5a in monoeg_g_usleep (microseconds=1935890) at 
gdate-unix.c:53
#2  0x0061f2bb in suspend_sync_nolock (id=id@entry=139966347736832, 
interrupt_kernel=interrupt_kernel@entry=1) at mono-threads.c:870
#3  0x0061f732 in mono_thread_info_safe_suspend_and_run 
(id=139966347736832, interrupt_kernel=interrupt_kernel@entry=1, 
callback=callback@entry=0x5813c0 , 
user_data=user_data@entry=0x7fffda0d3a10) at mono-threads.c:891
#4  0x00581d29 in suspend_thread_internal (thread=0x7f4c78a74430, 
interrupt=1) at threads.c:4531
#5  0x005834ff in mono_thread_suspend_all_other_threads () at 
threads.c:3154
#6  0x00538176 in ves_icall_System_Environment_Exit (result=0) at 
icall.c:6006
#7  0x40731832 in ?? ()
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

Re: [Mono-dev] High threadpool CPU usage

2016-06-03 Thread Chris Swiedler 
Well, looking at some pstack+top output, I’m not sure it’s GC anymore. I ran 
pstack and top as close to simultaneously as I could, and the high-cpu threads 
were just in pthread_cond_timedwait. Is it possible we get into a scenario 
where mono_cond_timedwait_ms is sleeping with a timeout_ms that's extremely 
short? Is there an easy way to force the compiler (#pragma or something?) to 
not optimize out the timeout_ms value? I'd rather not build all of mono with 
optimizations off, but I could disable optimizations for a few files.

chris

  PID USER  PR  NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
17369 hydrogen  20   0 6491744 157928  17216 S 53.3  0.1   0:05.08 Threadpool 
work
17385 hydrogen  20   0 6491744 157928  17216 S 46.7  0.1   0:09.38 Threadpool 
work

Thread 8 (Thread 0x7f52c65fc700 (LWP 17369)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00618817 in mono_cond_timedwait_ms 
(cond=cond@entry=0x7f52c65fbdc0, mutex=0x2887808, timeout_ms=) 
at mono-mutex.c:181
#2  0x00586f28 in worker_park () at threadpool-ms.c:509

Thread 4 (Thread 0x7f528a7fe700 (LWP 17385)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00618817 in mono_cond_timedwait_ms 
(cond=cond@entry=0x7f528a7fddc0, mutex=0x2887808, timeout_ms=) 
at mono-mutex.c:181
#2  0x00586f28 in worker_park () at threadpool-ms.c:509

From: Rodrigo Kumpera [mailto:kump...@gmail.com] 
Sent: Friday, May 27, 2016 5:07 PM
To: Chris Swiedler <cswied...@trionworlds.com>
Cc: Ludovic Henry <ludo...@xamarin.com>; mono-devel-list 
<mono-devel-list@lists.ximian.com>
Subject: Re: [Mono-dev] High threadpool CPU usage

Chris,

Can you at least enable GC logging? That will tell you how frequently and for 
how long the collector is running.

On Fri, May 27, 2016 at 11:04 AM, Chris Swiedler <cswied...@trionworlds.com> 
wrote:
Thanks. I’ll try to use the profiler, but this problem doesn’t happen at 
startup, and running the application under profiling for eight hours is 
probably infeasible.
 
I see that at one point mono support signal-based enabling/disabling of the 
profiler, like how SIGUSR2 toggles the trace mechanism on and off. Is that sort 
of functionality ever going to be re-added? Currently there’s the heapshot 
option via the TCP listener, perhaps there could be enable/disable commands via 
that mechanism?
 
chris
 
From: Ludovic Henry [mailto:ludo...@xamarin.com] 
Sent: Friday, May 27, 2016 1:27 AM

To: Chris Swiedler <cswied...@trionworlds.com>; mono-devel-list 
<mono-devel-list@lists.ximian.com>
Subject: Re: [Mono-dev] High threadpool CPU usage
 
Hi Chris,
 
The signal handler you are seeing is the GC signalling every thread to suspend 
them. So yes you are right, that's the sgen collector stopping the world.
 
To track down the large number of allocation, I would encourage you to try the 
log profiler (documentation: 
http://www.mono-project.com/docs/debug+profile/profile/profiler/ ). That should 
give you some more insights in where the allocations are coming from.
 
Thank you,
Ludovic
 
On Thu, May 26, 2016 at 8:20 PM Chris Swiedler <cswied...@trionworlds.com> 
wrote:
Thanks, Ludovic. I’m using mono-4.2.3. The ‘massive amounts of GC’ idea makes 
sense based on what I’m seeing. When I run pstack, I get a number of threadpool 
threads in stacks with:
 
#0  0x7fdff1c6a952 in do_sigsuspend (set=0x954220 ) at 
../sysdeps/unix/sysv/linux/sigsuspend.c:32
#1  __GI___sigsuspend (set=set@entry=0x954220 ) at 
../sysdeps/unix/sysv/linux/sigsuspend.c:46
#2  0x005c7534 in suspend_thread (info=0x7fdf480008c0, 
context=context@entry=0x7fde997ea1c0) at sgen-os-posix.c:126
#3  0x005c771f in suspend_handler (_dummy=, 
_info=, context=0x7fde997ea1c0) at sgen-os-posix.c:153
#4  
 
I thought this was related to GDB / pstack attaching, but it’s actually the 
suspend handling for the sgen collector, right? 
 
Is a thread suspending itself CPU-intensive? I see lots of threads with high 
CPU at any point, which seems to indicate that this wouldn’t just be the CPU 
usage of the collector thread itself.
 
Do you have any suggestions for how to track down the large numbers of 
allocations? This isn’t very easy to reproduce, but now that I know what to 
look for, I might be able to get it to happen in a test environment.
 
chris
 
From: Ludovic Henry [mailto:ludo...@xamarin.com] 
Sent: Thursday, May 26, 2016 8:43 AM
To: Chris Swiedler <cswied...@trionworlds.com>; mono-devel-list 
<mono-devel-list@lists.ximian.com>
Subject: Re: [Mono-dev] High threadpool CPU usage
 
Hi Chris,
 
The first stacktrace you are observing is for threadpool thread parking. We use 
this technique for threads that are currently not doing anything, to keep them 
around for a little while (5-60 seconds) so if we have burst of usage, we do 
n

Re: [Mono-dev] High threadpool CPU usage

2016-05-27 Thread Chris Swiedler 
Thanks. I’ll try to use the profiler, but this problem doesn’t happen at 
startup, and running the application under profiling for eight hours is 
probably infeasible.

I see that at one point mono support signal-based enabling/disabling of the 
profiler, like how SIGUSR2 toggles the trace mechanism on and off. Is that sort 
of functionality ever going to be re-added? Currently there’s the heapshot 
option via the TCP listener, perhaps there could be enable/disable commands via 
that mechanism?

chris

From: Ludovic Henry [mailto:ludo...@xamarin.com]
Sent: Friday, May 27, 2016 1:27 AM
To: Chris Swiedler <cswied...@trionworlds.com>; mono-devel-list 
<mono-devel-list@lists.ximian.com>
Subject: Re: [Mono-dev] High threadpool CPU usage

Hi Chris,

The signal handler you are seeing is the GC signalling every thread to suspend 
them. So yes you are right, that's the sgen collector stopping the world.

To track down the large number of allocation, I would encourage you to try the 
log profiler (documentation: 
http://www.mono-project.com/docs/debug+profile/profile/profiler/ ). That should 
give you some more insights in where the allocations are coming from.

Thank you,
Ludovic

On Thu, May 26, 2016 at 8:20 PM Chris Swiedler 
<cswied...@trionworlds.com<mailto:cswied...@trionworlds.com>> wrote:
Thanks, Ludovic. I’m using mono-4.2.3. The ‘massive amounts of GC’ idea makes 
sense based on what I’m seeing. When I run pstack, I get a number of threadpool 
threads in stacks with:

#0  0x7fdff1c6a952 in do_sigsuspend (set=0x954220 ) at 
../sysdeps/unix/sysv/linux/sigsuspend.c:32
#1  __GI___sigsuspend (set=set@entry=0x954220 ) at 
../sysdeps/unix/sysv/linux/sigsuspend.c:46
#2  0x005c7534 in suspend_thread (info=0x7fdf480008c0, 
context=context@entry=0x7fde997ea1c0) at sgen-os-posix.c:126
#3  0x005c771f in suspend_handler (_dummy=, 
_info=, context=0x7fde997ea1c0) at sgen-os-posix.c:153
#4  

I thought this was related to GDB / pstack attaching, but it’s actually the 
suspend handling for the sgen collector, right?

Is a thread suspending itself CPU-intensive? I see lots of threads with high 
CPU at any point, which seems to indicate that this wouldn’t just be the CPU 
usage of the collector thread itself.

Do you have any suggestions for how to track down the large numbers of 
allocations? This isn’t very easy to reproduce, but now that I know what to 
look for, I might be able to get it to happen in a test environment.

chris

From: Ludovic Henry [mailto:ludo...@xamarin.com<mailto:ludo...@xamarin.com>]
Sent: Thursday, May 26, 2016 8:43 AM
To: Chris Swiedler 
<cswied...@trionworlds.com<mailto:cswied...@trionworlds.com>>; mono-devel-list 
<mono-devel-list@lists.ximian.com<mailto:mono-devel-list@lists.ximian.com>>
Subject: Re: [Mono-dev] High threadpool CPU usage

Hi Chris,

The first stacktrace you are observing is for threadpool thread parking. We use 
this technique for threads that are currently not doing anything, to keep them 
around for a little while (5-60 seconds) so if we have burst of usage, we do 
not end up destroying and creating threads all the time.

The second stacktrace you are observing is, as you noted, when performing a 
garbage collection, when the GC thread is suspending all the running threads. 
So if you are witnessing this trace very often, it means a thread is allocating 
memory very quickly triggering GC collection very often.

So I would need more information to tell you exactly why it would use 100% CPU. 
Also which version of Mono are you running?

Thank you very much,
Ludovic
On Wed, May 25, 2016 at 8:30 PM Chris Swiedler 
<cswied...@trionworlds.com<mailto:cswied...@trionworlds.com>> wrote:
We have a server app which is periodically going into a mode where all 
threadpool threads start running at very high CPU. I've run pstack when it's in 
this mode, and every time I do it, nearly all the threadpool threads have this 
stack:

#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00618817 in mono_cond_timedwait_ms 
(cond=cond@entry=0x7fe7ee1fddc0, mutex=0x241eb78, timeout_ms=) 
at mono-mutex.c:181
#2  0x00586f28 in worker_park () at threadpool-ms.c:509
#3  worker_thread (data=) at threadpool-ms.c:607
#4  0x005841e9 in start_wrapper_internal (data=) at 
threads.c:725
#5  start_wrapper (data=) at threads.c:772
#6  0x00621026 in inner_start_thread (arg=0x7fe831df4650) at 
mono-threads-posix.c:97
#7  0x7fe88a55edf5 in start_thread (arg=0x7fe7ee1fe700) at 
pthread_create.c:308
#8  0x7fe88a28c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Usually one thread will have a stack like this:

#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061aa37 in mono_sem_wait (sem=0x9542c0 , 
alertable=alertable@entry=0) at mono-semaphore.c:107
#2  0x005c77

Re: [Mono-dev] High threadpool CPU usage

2016-05-26 Thread Chris Swiedler 
Thanks, Ludovic. I’m using mono-4.2.3. The ‘massive amounts of GC’ idea makes 
sense based on what I’m seeing. When I run pstack, I get a number of threadpool 
threads in stacks with:

#0  0x7fdff1c6a952 in do_sigsuspend (set=0x954220 ) at 
../sysdeps/unix/sysv/linux/sigsuspend.c:32
#1  __GI___sigsuspend (set=set@entry=0x954220 ) at 
../sysdeps/unix/sysv/linux/sigsuspend.c:46
#2  0x005c7534 in suspend_thread (info=0x7fdf480008c0, 
context=context@entry=0x7fde997ea1c0) at sgen-os-posix.c:126
#3  0x005c771f in suspend_handler (_dummy=, 
_info=, context=0x7fde997ea1c0) at sgen-os-posix.c:153
#4  

I thought this was related to GDB / pstack attaching, but it’s actually the 
suspend handling for the sgen collector, right?

Is a thread suspending itself CPU-intensive? I see lots of threads with high 
CPU at any point, which seems to indicate that this wouldn’t just be the CPU 
usage of the collector thread itself.

Do you have any suggestions for how to track down the large numbers of 
allocations? This isn’t very easy to reproduce, but now that I know what to 
look for, I might be able to get it to happen in a test environment.

chris

From: Ludovic Henry [mailto:ludo...@xamarin.com]
Sent: Thursday, May 26, 2016 8:43 AM
To: Chris Swiedler <cswied...@trionworlds.com>; mono-devel-list 
<mono-devel-list@lists.ximian.com>
Subject: Re: [Mono-dev] High threadpool CPU usage

Hi Chris,

The first stacktrace you are observing is for threadpool thread parking. We use 
this technique for threads that are currently not doing anything, to keep them 
around for a little while (5-60 seconds) so if we have burst of usage, we do 
not end up destroying and creating threads all the time.

The second stacktrace you are observing is, as you noted, when performing a 
garbage collection, when the GC thread is suspending all the running threads. 
So if you are witnessing this trace very often, it means a thread is allocating 
memory very quickly triggering GC collection very often.

So I would need more information to tell you exactly why it would use 100% CPU. 
Also which version of Mono are you running?

Thank you very much,
Ludovic
On Wed, May 25, 2016 at 8:30 PM Chris Swiedler 
<cswied...@trionworlds.com<mailto:cswied...@trionworlds.com>> wrote:
We have a server app which is periodically going into a mode where all 
threadpool threads start running at very high CPU. I've run pstack when it's in 
this mode, and every time I do it, nearly all the threadpool threads have this 
stack:

#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00618817 in mono_cond_timedwait_ms 
(cond=cond@entry=0x7fe7ee1fddc0, mutex=0x241eb78, timeout_ms=) 
at mono-mutex.c:181
#2  0x00586f28 in worker_park () at threadpool-ms.c:509
#3  worker_thread (data=) at threadpool-ms.c:607
#4  0x005841e9 in start_wrapper_internal (data=) at 
threads.c:725
#5  start_wrapper (data=) at threads.c:772
#6  0x00621026 in inner_start_thread (arg=0x7fe831df4650) at 
mono-threads-posix.c:97
#7  0x7fe88a55edf5 in start_thread (arg=0x7fe7ee1fe700) at 
pthread_create.c:308
#8  0x7fe88a28c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Usually one thread will have a stack like this:

#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061aa37 in mono_sem_wait (sem=0x9542c0 , 
alertable=alertable@entry=0) at mono-semaphore.c:107
#2  0x005c77bd in sgen_wait_for_suspend_ack (count=count@entry=18) at 
sgen-os-posix.c:188
#3  0x005c78f9 in sgen_thread_handshake (suspend=suspend@entry=1) at 
sgen-os-posix.c:224
#4  0x005c7e92 in sgen_client_stop_world 
(generation=generation@entry=0) at sgen-stw.c:234
#5  0x005e6aca in sgen_stop_world (generation=0) at sgen-gc.c:3389
#6  0x005e6c29 in sgen_perform_collection (requested_size=4096, 
generation_to_collect=0, reason=0x6d9595 "Nursery full", wait_to_finish=0) at 
sgen-gc.c:2322#7  0x005da62a in sgen_alloc_obj_nolock 
(vtable=vtable@entry=0x7fe85c0343c0, size=size@entry=128) at sgen-alloc.c:291
#8  0x005da913 in sgen_alloc_obj (vtable=vtable@entry=0x7fe85c0343c0, 
size=128) at sgen-alloc.c:457
#9  0x005c86e9 in mono_gc_alloc_obj 
(vtable=vtable@entry=0x7fe85c0343c0, size=) at sgen-mono.c:936
#10 0x005a8b54 in mono_object_new_alloc_specific 
(vtable=vtable@entry=0x7fe85c0343c0) at object.c:4385
#11 0x005a8bf0 in mono_object_new_specific (vtable=0x7fe85c0343c0) at 
object.c:4379
#12 0x005a8c8c in mono_object_new (domain=domain@entry=0x1ded1c0, 
klass=) at object.c:4318
#13 0x005ac1c9 in mono_async_result_new (domain=domain@entry=0x1ded1c0, 
handle=handle@entry=0x0, state=0x0, data=data@entry=0x0, 
object_data=object_data@entry=0x7fe8838af020) at object.c:5768
#14 0x005887ff in mono_threadpool_ms_begin_invoke (doma

[Mono-dev] High threadpool CPU usage

2016-05-25 Thread Chris Swiedler 
We have a server app which is periodically going into a mode where all 
threadpool threads start running at very high CPU. I've run pstack when it's in 
this mode, and every time I do it, nearly all the threadpool threads have this 
stack:

#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00618817 in mono_cond_timedwait_ms 
(cond=cond@entry=0x7fe7ee1fddc0, mutex=0x241eb78, timeout_ms=) 
at mono-mutex.c:181
#2  0x00586f28 in worker_park () at threadpool-ms.c:509
#3  worker_thread (data=) at threadpool-ms.c:607
#4  0x005841e9 in start_wrapper_internal (data=) at 
threads.c:725
#5  start_wrapper (data=) at threads.c:772
#6  0x00621026 in inner_start_thread (arg=0x7fe831df4650) at 
mono-threads-posix.c:97
#7  0x7fe88a55edf5 in start_thread (arg=0x7fe7ee1fe700) at 
pthread_create.c:308
#8  0x7fe88a28c1ad in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Usually one thread will have a stack like this:

#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x0061aa37 in mono_sem_wait (sem=0x9542c0 , 
alertable=alertable@entry=0) at mono-semaphore.c:107
#2  0x005c77bd in sgen_wait_for_suspend_ack (count=count@entry=18) at 
sgen-os-posix.c:188
#3  0x005c78f9 in sgen_thread_handshake (suspend=suspend@entry=1) at 
sgen-os-posix.c:224
#4  0x005c7e92 in sgen_client_stop_world 
(generation=generation@entry=0) at sgen-stw.c:234
#5  0x005e6aca in sgen_stop_world (generation=0) at sgen-gc.c:3389
#6  0x005e6c29 in sgen_perform_collection (requested_size=4096, 
generation_to_collect=0, reason=0x6d9595 "Nursery full", wait_to_finish=0) at 
sgen-gc.c:2322#7  0x005da62a in sgen_alloc_obj_nolock 
(vtable=vtable@entry=0x7fe85c0343c0, size=size@entry=128) at sgen-alloc.c:291
#8  0x005da913 in sgen_alloc_obj (vtable=vtable@entry=0x7fe85c0343c0, 
size=128) at sgen-alloc.c:457
#9  0x005c86e9 in mono_gc_alloc_obj 
(vtable=vtable@entry=0x7fe85c0343c0, size=) at sgen-mono.c:936
#10 0x005a8b54 in mono_object_new_alloc_specific 
(vtable=vtable@entry=0x7fe85c0343c0) at object.c:4385
#11 0x005a8bf0 in mono_object_new_specific (vtable=0x7fe85c0343c0) at 
object.c:4379
#12 0x005a8c8c in mono_object_new (domain=domain@entry=0x1ded1c0, 
klass=) at object.c:4318
#13 0x005ac1c9 in mono_async_result_new (domain=domain@entry=0x1ded1c0, 
handle=handle@entry=0x0, state=0x0, data=data@entry=0x0, 
object_data=object_data@entry=0x7fe8838af020) at object.c:5768
#14 0x005887ff in mono_threadpool_ms_begin_invoke (domain=0x1ded1c0, 
target=target@entry=0x7fe8838aee38, method=method@entry=0x2963d28, 
params=params@entry=0x7fe7ed9f8f10) at threadpool-ms.c:1300
#15 0x0054b547 in mono_delegate_begin_invoke (delegate=0x7fe8838aee38, 
params=0x7fe7ed9f8f10) at marshal.c:2111
#16 0x416d29d8 in ?? ()
#17 0x in ?? ()

Just from reading the first stack, it doesn't look like mono_cond_timedwait_ms 
would spin, at least as long as the timeout_ms wasn't 0. For the second stack, 
I don't know whether that's a normal garbage collection pass or (since we see 
it frequently) a sign that garbage collection is happening too often.

Can anyone give me some pointers for where to dig more deeply?

Thanks,
chris
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

Re: [Mono-dev] Reading threadpool performance counters

2016-05-24 Thread Chris Swiedler 
Thanks, Ludovic. I’ve created 
https://bugzilla.xamarin.com/show_bug.cgi?id=41294 with sample code. I’ve also 
verified that it works as expected with mono 3.12, so it sounds like your 
explanation is correct.

chris


From: Ludovic Henry [mailto:ludo...@xamarin.com]
Sent: Tuesday, May 24, 2016 5:46 AM
To: Chris Swiedler <cswied...@trionworlds.com>; mono-devel-list 
<mono-devel-list@lists.ximian.com>
Subject: Re: [Mono-dev] Reading threadpool performance counters

Hi Chris,

I don't think we are keeping this counters up to date in the threadpool. Some 
of them are kept up to date, but as we imported the threadpool implementation 
from referencesources since 4.2, we must have missed this performance counter. 
So you are certainly not doing something wrong.

If you want, you should fill a bug with the information you have so we have a 
look at it as soon as possible.

Thank you very much,
Ludovic

On Mon, May 23, 2016 at 11:59 PM Chris Swiedler 
<cswied...@trionworlds.com<mailto:cswied...@trionworlds.com>> wrote:
Per https://bugzilla.xamarin.com/show_bug.cgi?id=1772 I've been trying to get 
Mono threadpool metrics with lines like this:

m_workItemsAdded = new PerformanceCounter("Mono Threadpool", "Work Items 
Added", Process.GetCurrentProcess().Id.ToString());

However, the counter I get back always has its value set to zero. Are these 
threadpool counters still supported? Am I doing something wrong? This is using 
mono 4.2.3.

Thanks,
chris
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com<mailto:Mono-devel-list@lists.ximian.com>
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[Mono-dev] Reading threadpool performance counters

2016-05-23 Thread Chris Swiedler 
Per https://bugzilla.xamarin.com/show_bug.cgi?id=1772 I've been trying to get 
Mono threadpool metrics with lines like this:

m_workItemsAdded = new PerformanceCounter("Mono Threadpool", "Work Items 
Added", Process.GetCurrentProcess().Id.ToString());

However, the counter I get back always has its value set to zero. Are these 
threadpool counters still supported? Am I doing something wrong? This is using 
mono 4.2.3.

Thanks,
chris
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

Re: [Mono-dev] Cross Platform on Linux/Windows with Mono.Posix reference on Linux

2016-03-30 Thread Chris Swiedler 
To Alan's point, we did see crashes when we (unintentionally) deployed 
Mono.Posix.dll along with our app. Our solution was, like he said, to Not Do 
That, and use our Mono.Posix as a compile-time library only.

I do think it's much easier to do it this way than to have different compile 
targets, which (if I'm understanding correctly) is what the other suggestions 
all require.

chris

-Original Message-
From: mono-devel-list-boun...@lists.ximian.com 
[mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Jonathan Pryor
Sent: Wednesday, March 30, 2016 7:04 AM
To: Alan McGovern 
Cc: mono-devel-list 
Subject: Re: [Mono-dev] Cross Platform on Linux/Windows with Mono.Posix 
reference on Linux

On Mar 30, 2016, at 8:33 AM, Alan  wrote:
> If you package Mono.Posix.dll your app *will crash* on different systems.

Not necessarily.

> This binary is platform specific and is not safe to copy between OS’s.

Mono.Posix.dll *itself* contains no platform-specific code and is perfectly 
safe to copy between operating systems.

Mono.Posix.dll contains P/Invokes into “MonoPosixHelper” 
(libMonoPosixHelper.dylib on OS X, MonoPosixHelper.dll on Windows, etc.), and 
MonoPosixHelper contains operating system-specific code. It *cannot* be copied 
between operating systems; it’s a native library.

> It's fine to have a copy of Mono.Posix.dll used purely for compilation 
> purposes. But at runtime you have to use the system provided one, which 
> typically means the one provided by the system's mono installation.

It’s entirely fine to include Mono.Posix.dll with your app, SO LONG AS you 
*also* copy and distribute MonoPosixHelper with your app.

Additionally, Mono.Posix.dll also P/Invokes other native libraries such as 
INTL.DLL (Mono.Unix.Catalog) and MSVCRT.DLL (Mono.Unix.Native.Stdlib), which 
should be usable on Windows (so long as you also distribute INTL.DLL, etc.).

 - Jon

___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

Re: [Mono-dev] Cross Platform on Linux/Windows with Mono.Posix reference on Linux

2016-03-29 Thread Chris Swiedler 
Why not just include references to Mono.Posix.dll in the Windows build? You 
don't have to install the full framework. We have a similar issue with server 
apps that are developed on Windows and run on Linux. We just put a win32 build 
of Mono into our source tree and added Mono.Posix as a reference. Everywhere we 
invoke any Posix code, we just wrap the statements in something like "if  
(Environment.OSVersion.Platform == PlatformID.Unix)". 

chris

-Original Message-
From: mono-devel-list-boun...@lists.ximian.com 
[mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Edward Ned 
Harvey (mono)
Sent: Tuesday, March 29, 2016 5:49 AM
To: Greg Young ; Jason Curl 
Cc: mono-devel-list@lists.ximian.com
Subject: Re: [Mono-dev] Cross Platform on Linux/Windows with Mono.Posix 
reference on Linux

> From: mono-devel-list-boun...@lists.ximian.com 
> [mailto:mono-devel-list- boun...@lists.ximian.com] On Behalf Of Greg 
> Young
> 
>  />

(for clarification of my previous response, because I was asked for 
clarification off-list)

If you have some code that calls Mono.Posix.Something(), and you have a 
conditional reference to Mono.Posix in the project, then you're able to build 
on Mono, but on windows, without the reference, your call to 
Mono.Posix.Something() will not compile, which prevents you from building, even 
though at runtime you might never make that call. In order to compile on 
windows, you'll need to comment-out your Mono.Posix.Something() line, via #if 
or [Conditional], which means you need a compiler symbol to indicate whether 
you're building on mono or windows. This could be either separate project 
files, or multiple build configurations. (Or dynamically generated project 
files).

Maybe you can have a conditional reference to Mono.Posix on mono, and have a 
separate conditional reference to ../../ExtraClassLibraries/Mono.Posix.dll on 
windows, with copy-to-destination, so you can get the same code to build on 
both mono and windows, resulting in an effectively identical cross-platform 
binary... But I'm almost certain I recall somebody on this list telling me, 
that the JIT compiler can or will or might in the future, eagerly link 
assemblies at runtime (or just before runtime), so the Mono.Posix.dll file must 
be present on windows at runtime in order to avoid runtime exceptions, even 
though the runtime code execution will never execute that path.

You avoid all these problems by abstracting the functionality you want into an 
interface or abstract class, and using a factory to perform a runtime dynamic 
load of a mono-specific or windows-specific assembly with a derivative class. 
You can build the mono-specific assembly on mono, or maybe you can use the 
double-conditional described above and get the mono-specific assembly to also 
be buildable on windows. No need to copy Mono.Posix.dll to the destination, 
because Mono.Posix will only be called at runtime when run on mono.
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

Re: [Mono-dev] Debugging Mono applications under GDB

2015-11-15 Thread Chris Swiedler 
Do you mean for development, or for debugging? The application is running on a 
headless server and I can’t run monodevelop or Xamarin on it. I’m building the 
exes with the mono toolchain under Windows, by installing it as a .net profile.

chris

> On Nov 14, 2015, at 6:32 PM, Edward Ned Harvey (mono) 
> <edward.harvey.m...@clevertrove.com> wrote:
> 
>> From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-
>> boun...@lists.ximian.com] On Behalf Of Chris Swiedler
>> 
>> We have a server application that's being developed under Visual Studio and
>> run under Mono 3.12. We're generating .mdb files from the .pdbs that VS
>> creates. With those deployed next to the executable, mono will give full
>> callstacks with source and line information when logging exceptions, so I
>> believe the .mdbs are correct.
> 
> Have you considered monodevelop or Xamarin Studio?

___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

[Mono-dev] Debugging Mono applications under GDB

2015-11-09 Thread Chris Swiedler 
We have a server application that's being developed under Visual Studio and run 
under Mono 3.12. We're generating .mdb files from the .pdbs that VS creates. 
With those deployed next to the executable, mono will give full callstacks with 
source and line information when logging exceptions, so I believe the .mdbs are 
correct.



I've read http://www.mono-project.com/docs/debug+profile/debug/ so many times I 
can quote it by heart, but I still can't get gdb to give me any useful symbols. 
I'm using gdb 7.6.1, so according to the docs, it should just use the JIT 
interface to get symbols.



Is there any way to troubleshoot the loading of symbols by gdb? There are 
examples for how to generate xdb shared object files (I've tried them, but as 
doesn't like some of the assembly that MONO_XDEBUG generates) but nothing on 
the new JIT interface other than "it should work".



I do get errors like "../../gdb/dwarf2-frame.c:683: internal-error: Unknown CFI 
encountered." I'm not sure if those are relevant to my problem. A sample gdb 
session is below.



Thanks,

chris



GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7 Copyright (C) 2013 Free 
Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later 

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

.

Attaching to process 21185

Reading symbols from /usr/bin/mono-sgen...(no debugging symbols found)...done.

Reading symbols from /lib64/libm.so.6...Reading symbols from 
/usr/lib/debug/usr/lib64/libm-2.17.so.debug...done.

done.

Loaded symbols for /lib64/libm.so.6

Reading symbols from /lib64/librt.so.1...Reading symbols from 
/usr/lib/debug/usr/lib64/librt-2.17.so.debug...done.

done.

Loaded symbols for /lib64/librt.so.1

Reading symbols from /lib64/libdl.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libdl-2.17.so.debug...done.

done.

Loaded symbols for /lib64/libdl.so.2

Reading symbols from /lib64/libpthread.so.0...Reading symbols from 
/usr/lib/debug/usr/lib64/libpthread-2.17.so.debug...done.

done.

[New LWP 21269]

...

[New LWP 21187]

[Thread debugging using libthread_db enabled] Using host libthread_db library 
"/lib64/libthread_db.so.1".

Loaded symbols for /lib64/libpthread.so.0 Reading symbols from 
/lib64/libgcc_s.so.1...Reading symbols from 
/usr/lib/debug/usr/lib64/libgcc_s-4.8.3-20140911.so.1.debug...done.

done.

Loaded symbols for /lib64/libgcc_s.so.1

Reading symbols from /lib64/libc.so.6...Reading symbols from 
/usr/lib/debug/usr/lib64/libc-2.17.so.debug...done.

done.

Loaded symbols for /lib64/libc.so.6

Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.

done.

Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from 
/lib64/libnss_files.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libnss_files-2.17.so.debug...done.

done.

Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from 
/usr/lib/mono/4.5/mscorlib.dll.so...done.

Loaded symbols for /usr/lib/mono/4.5/mscorlib.dll.so Reading symbols from 
/usr/lib64/libMonoPosixHelper.so...done.

Loaded symbols for /usr/lib64/libMonoPosixHelper.so Reading symbols from 
/lib64/libz.so.1...Reading symbols from 
/usr/lib/debug/usr/lib64/libz.so.1.2.7.debug...done.

done.

Loaded symbols for /lib64/libz.so.1

Reading symbols from /lib64/libnss_dns.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libnss_dns-2.17.so.debug...done.

done.

Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from 
/lib64/libresolv.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libresolv-2.17.so.debug...done.

done.

Loaded symbols for /lib64/libresolv.so.2 clock_nanosleep (clock_id=1, flags=1, 
req=0x7fff271a8e80, rem=0x) at 
../sysdeps/unix/sysv/linux/clock_nanosleep.c:49

49LIBC_CANCEL_RESET (oldstate); Mono support loaded.

(gdb) mono_backtrace 10

#0  clock_nanosleep (clock_id=1, flags=1, req=0x7fff271a8e80, 
rem=0x) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:49

49LIBC_CANCEL_RESET (oldstate);

#1  0x00618c78 in SleepEx ()

#2  0x0058cc7a in ves_icall_System_Threading_Thread_Sleep_internal ()

#3 0x413215c2 in  (wrapper managed-to-native) 
System.Threading.Thread:Sleep_internal (int) + 0x52 (0x41321570 0x413215ee) 
[0x20ee4f0 - LobbyServer.exe]

#4  0x02146a10 in ?? ()

#5  0x7fff271a9410 in ?? ()

#6  0x7f252cc01338 in ?? ()

#7  0x7fff271a9410 in ?? ()

#8  0x7f252cc00b18 in ?? ()

#9  0x02139ac0 in ?? ()

(gdb) info threads

  Id   Target Id Frame

  32   Thread 0x7f252b0ed700 (LWP 21187) "Finalizer" sem_wait () at

[Mono-list] Debugging Mono applications under GDB

2015-10-19 Thread Chris Swiedler 
We have a server application that's being developed under Visual Studio and run 
under Mono 3.12. We're generating .mdb files from the .pdbs that VS creates. 
With those deployed next to the executable, mono will give full callstacks with 
source and line information when logging exceptions, so I believe the .mdbs are 
correct.

I've read http://www.mono-project.com/docs/debug+profile/debug/ so many times I 
can quote it by heart, but I still can't get gdb to give me any useful symbols. 
I'm using gdb 7.6.1, so according to the docs, it should just use the JIT 
interface to get symbols.

Is there any way to troubleshoot the loading of symbols by gdb? There are 
examples for how to generate xdb shared object files (I've tried them, but as 
doesn't like some of the assembly that MONO_XDEBUG generates) but nothing on 
the new JIT interface other than "it should work". 

I do get errors like "../../gdb/dwarf2-frame.c:683: internal-error: Unknown CFI 
encountered." I'm not sure if those are relevant to my problem. A sample gdb 
session is below.

Thanks,
chris

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7 Copyright (C) 2013 Free 
Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
.
Attaching to process 21185
Reading symbols from /usr/bin/mono-sgen...(no debugging symbols found)...done.
Reading symbols from /lib64/libm.so.6...Reading symbols from 
/usr/lib/debug/usr/lib64/libm-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...Reading symbols from 
/usr/lib/debug/usr/lib64/librt-2.17.so.debug...done.
done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libdl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...Reading symbols from 
/usr/lib/debug/usr/lib64/libpthread-2.17.so.debug...done.
done.
[New LWP 21269]
[New LWP 21266]
[New LWP 21254]
[New LWP 21253]
[New LWP 21252]
[New LWP 21250]
[New LWP 21248]
[New LWP 21246]
[New LWP 21244]
[New LWP 21242]
[New LWP 21240]
[New LWP 21237]
[New LWP 21235]
[New LWP 21223]
[New LWP 21221]
[New LWP 21219]
[New LWP 21217]
[New LWP 21216]
[New LWP 21214]
[New LWP 21213]
[New LWP 21211]
[New LWP 21209]
[New LWP 21208]
[New LWP 21207]
[New LWP 21205]
[New LWP 21204]
[New LWP 21203]
[New LWP 21202]
[New LWP 21200]
[New LWP 21188]
[New LWP 21187]
[Thread debugging using libthread_db enabled] Using host libthread_db library 
"/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0 Reading symbols from 
/lib64/libgcc_s.so.1...Reading symbols from 
/usr/lib/debug/usr/lib64/libgcc_s-4.8.3-20140911.so.1.debug...done.
done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...Reading symbols from 
/usr/lib/debug/usr/lib64/libc-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from 
/lib64/libnss_files.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libnss_files-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from 
/usr/lib/mono/4.5/mscorlib.dll.so...done.
Loaded symbols for /usr/lib/mono/4.5/mscorlib.dll.so Reading symbols from 
/usr/lib64/libMonoPosixHelper.so...done.
Loaded symbols for /usr/lib64/libMonoPosixHelper.so Reading symbols from 
/lib64/libz.so.1...Reading symbols from 
/usr/lib/debug/usr/lib64/libz.so.1.2.7.debug...done.
done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libnss_dns.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libnss_dns-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from 
/lib64/libresolv.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libresolv-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libresolv.so.2 clock_nanosleep (clock_id=1, flags=1, 
req=0x7fff271a8e80, rem=0x) at 
../sysdeps/unix/sysv/linux/clock_nanosleep.c:49
49    LIBC_CANCEL_RESET (oldstate);
Mono support loaded.
(gdb) mono_backtrace 10
#0  clock_nanosleep (clock_id=1, flags=1, req=0x7fff271a8e80, 
rem=0x) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:49
49    LIBC_CANCEL_RESET (oldstate);
#1  0x00618c78 in SleepEx ()
#2  0x0058cc7a in ves_icall_System_Threading_Thread_Sleep_internal ()
#3 0x413215c2 in  (wrapper managed-to-native) 
System.Threading.Thread:Sleep_internal (int) + 0x52 (0x41321570 0x413215ee)