Re: [Flightgear-devel] [OT] Best pratice for debugging multi threaded programs?

[Tim Moore]

On Mon, Mar 21, 2011 at 3:04 PM, Holger Wirtz dcored...@googlemail.com wrote:
 Hi all,

 sorry - a little bit off-topic (in fact not so much as you might think,
 it's for a third-party software for FG):

 Has anyone some hints/websites/programs for debugging C -based multi
 threaded programs (exactly: glib based C code)? Currently I am
 developing with vi and a little bit gdb (command line) and very much
 debug output. But this setup seems to be very frustrating while
 searching for dead-locks and race-conditions :-(

 On the other hand I won't invest too much time for studying rocket
 engeneering only to use framework XYZ. Has anyone some ideas how to
 debug with less effort?

Just a couple of ideas to add to others' suggestions. First, if you
are starting  from scratch, do some reading  on design of
multi-threaded programs in order to reduce sharing -- and the
potential for races -- right from the start. A good strategy is to set
up queues of work for auxiliary threads and also have queues where
those threads can place their results.

The helgrind tool, which is part of valgrind, is very useful for
rooting out race conditions at an API level. You might not be able to
run FlightGear with it in a practical way, but you can certainly run
graphical applications with it.

In gdb there are several breakpoint and scheduling commands which give
you the flexibility to only break in certain threads and to only let
one thread continue. This can be useful when several threads have
entered the same function (which might be perfectly reasonable) and
gdb appears to be bouncing all around.

Finally, command line gdb is intolerable; I haven't debugged that way
for years. I run gdb inside of emacs, which gives you instant access
to the source code. I won't push emacs on a vi user :), but there are
many alternatives these days on Linux which will provide a friendlier
debugging  experience.

Tim
 Thanks, Holger

 --
 Colocation vs. Managed Hosting
 A question and answer guide to determining the best fit
 for your organization - today and in the future.
 http://p.sf.net/sfu/internap-sfd2d
 ___
 Flightgear-devel mailing list
 Flightgear-devel@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/flightgear-devel


--
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software 
be a part of the solution? Download the Intel(R) Manageability Checker 
today! http://p.sf.net/sfu/intel-dev2devmar
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel

Re: [Flightgear-devel] [OT] Best pratice for debugging multi threaded programs?

[Erik Hofman]

On Mon, 2011-03-21 at 15:04 +0100, Holger Wirtz wrote:
 Hi all,
 
 sorry - a little bit off-topic (in fact not so much as you might think, 
 it's for a third-party software for FG):
 
 Has anyone some hints/websites/programs for debugging C -based multi 
 threaded programs (exactly: glib based C code)? Currently I am 
 developing with vi and a little bit gdb (command line) and very much 
 debug output. But this setup seems to be very frustrating while 
 searching for dead-locks and race-conditions :-(

I have my own locking functions that call the pthread ones and added a
small piece of code to check for the mutex value:


pthread_mutex_t mutex;
int mtx = mutex.__data.__count;

r = pthread_mutex_lock(mutex);
if (mtx != 1) {
   printf(Mutex was already locked\n);
}

You could add all kinds of fancy stuff using __FILE__, __FUNCTION__ and
__LINE__ to detect from where the code has been called.
Note: This only works for __GNUC__

Erik


--
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software 
be a part of the solution? Download the Intel(R) Manageability Checker 
today! http://p.sf.net/sfu/intel-dev2devmar
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel

Re: [Flightgear-devel] [OT] Best pratice for debugging multi threaded programs?

[Holger Wirtz]

Hi Curt,

thanks for your comments to multi threading. I noticed that I am not 
doing everything wrong with my current approach for debugging. It seems 
that I have to go one (or more) steps back to fix the problems.

I think that multi-threaded applications need _very_ much more coding 
discipline than I thought. I tapped in nearly every trap that might 
exist and after fixing one problem, two new problems apeared :-(

But how to implement a conference server without multi-threading? I 
don't know, so I will go on with fixing strange problems hopefully I get 
the server running before I am too old for this s**t (TM).

Thanks, Holger

On 03/21/11 15:46, Curtis Olson wrote:
 Hi Holger,

 This is a very interesting question.  I don't have any good answers, so I am
 looking forward to seeing if anyone else on the list has any good
 methodologies or even small hints and tips they can suggest.

 I could stand up on my soap box here and give my long speech on threading
 architectures ... but I'll spare everyone. :-)

 The summary though is that threaded architectures add complexity and can
 create timing bugs or order dependent bugs.  The end result can be a system
 that works most of the time, and then randomly crashes for no apparent
 reason.  Often it can take hours (if not days or weeks) of running the code
 to tickle the bug ... and most of the time you have to trigger the bug to
 see what happened and get any kind of idea where it happened so you can fix
 it.  Also, even with a perfectly written threaded system, it's easy to come
 back in 6 months and forget something about the design and make a change
 that introduces a subtle problem.  This is even more likely if someone else
 comes later and touches the code without having ever understood all the
 nuances (sequence, and timing, etc.) of how all the threads work together.

 Personally I try to avoid threads whenever possible, and only use them when
 there is no sane alternative solution.  But I understand that threads are a
 fact of life so they can't always be avoided.

 When I've had to debug threaded code I've used a mix of gdb (and compiling
 everything with full debugging symbols) to get a back trace if there is a
 crash.  I also like to use printf's to see the sequence or progress of the
 code as it runs.  If I get desperate I might haul out valgrind and see if
 that offers any clues.  I've also spent hours just staring at the code and
 mentally working through the sequencing of the threads and the
 locking/mutual-exclusion primatives trying to analyze it all in my head and
 look for logic bugs that way.  All of this is pretty tedious, un-fun
 stuff.

 Maybe someone else has other tips and tricks.  I don't think I've run across
 any higher level methodologies that you can work through to always sniff out
 any threading bug.

 I know other people may feel differently about the use of threads and their
 relative benefits and dangers ... and that's fine.  I like to think of
 threads like nun-chucks.  I'll probably get moved over to the soprano
 section of the choir before I manage to do anything useful too spectacular
 with them. :-)

 Curt.


 On Mon, Mar 21, 2011 at 9:04 AM, Holger Wirtz wrote:

 Hi all,

 sorry - a little bit off-topic (in fact not so much as you might think,
 it's for a third-party software for FG):

 Has anyone some hints/websites/programs for debugging C -based multi
 threaded programs (exactly: glib based C code)? Currently I am
 developing with vi and a little bit gdb (command line) and very much
 debug output. But this setup seems to be very frustrating while
 searching for dead-locks and race-conditions :-(

 On the other hand I won't invest too much time for studying rocket
 engeneering only to use framework XYZ. Has anyone some ideas how to
 debug with less effort?

 Thanks, Holger


 --
 Colocation vs. Managed Hosting
 A question and answer guide to determining the best fit
 for your organization - today and in the future.
 http://p.sf.net/sfu/internap-sfd2d
 ___
 Flightgear-devel mailing list
 Flightgear-devel@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/flightgear-devel






 --
 Colocation vs. Managed Hosting
 A question and answer guide to determining the best fit
 for your organization - today and in the future.
 http://p.sf.net/sfu/internap-sfd2d



 ___
 Flightgear-devel mailing list
 Flightgear-devel@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/flightgear-devel

--
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) 

Re: [Flightgear-devel] [OT] Best pratice for debugging multi threaded programs?

[Holger Wirtz]

Hi Erik,

thanks for responding!

On 03/22/11 08:47, Erik Hofman wrote:
 On Mon, 2011-03-21 at 15:04 +0100, Holger Wirtz wrote:
 Hi all,

 sorry - a little bit off-topic (in fact not so much as you might think,
 it's for a third-party software for FG):

 Has anyone some hints/websites/programs for debugging C -based multi
 threaded programs (exactly: glib based C code)? Currently I am
 developing with vi and a little bit gdb (command line) and very much
 debug output. But this setup seems to be very frustrating while
 searching for dead-locks and race-conditions :-(

 I have my own locking functions that call the pthread ones and added a
 small piece of code to check for the mutex value:


 pthread_mutex_t mutex;
 int mtx = mutex.__data.__count;

 r = pthread_mutex_lock(mutex);
 if (mtx != 1) {
 printf(Mutex was already locked\n);
 }

I tried the same thing with the glib-pthreads wrappers, but 
pthread_mutex_lock blocks until the mutex is unlocked. Perhaps this 
should work with pthread_mutex_trylock()?

 You could add all kinds of fancy stuff using __FILE__, __FUNCTION__ and
 __LINE__ to detect from where the code has been called.
 Note: This only works for __GNUC__

Ok, this seems to be a good way to get more information where the dead 
locks are located. I will try this.

Thanks a lot!

Holger


 Erik


 --
 Enable your software for Intel(R) Active Management Technology to meet the
 growing manageability and security demands of your customers. Businesses
 are taking advantage of Intel(R) vPro (TM) technology - will your software
 be a part of the solution? Download the Intel(R) Manageability Checker
 today! http://p.sf.net/sfu/intel-dev2devmar
 ___
 Flightgear-devel mailing list
 Flightgear-devel@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/flightgear-devel

--
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software 
be a part of the solution? Download the Intel(R) Manageability Checker 
today! http://p.sf.net/sfu/intel-dev2devmar
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel

Re: [Flightgear-devel] [OT] Best pratice for debugging multi threaded programs?

[Erik Hofman]

On Tue, 2011-03-22 at 11:20 +0100, Holger Wirtz wrote:

 I tried the same thing with the glib-pthreads wrappers, but 
 pthread_mutex_lock blocks until the mutex is unlocked. Perhaps this 
 should work with pthread_mutex_trylock()?

It depends on the mode but you could also test prior to locking if it is
already 1 but then you're a but late in the game.

  You could add all kinds of fancy stuff using __FILE__, __FUNCTION__ and
  __LINE__ to detect from where the code has been called.
  Note: This only works for __GNUC__
 
 Ok, this seems to be a good way to get more information where the dead 
 locks are located. I will try this.
 
 Thanks a lot!

You're welcome.

Erik


--
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software 
be a part of the solution? Download the Intel(R) Manageability Checker 
today! http://p.sf.net/sfu/intel-dev2devmar
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel

[Flightgear-devel] [OT] Best pratice for debugging multi threaded programs?

[Holger Wirtz]

Hi all,

sorry - a little bit off-topic (in fact not so much as you might think, 
it's for a third-party software for FG):

Has anyone some hints/websites/programs for debugging C -based multi 
threaded programs (exactly: glib based C code)? Currently I am 
developing with vi and a little bit gdb (command line) and very much 
debug output. But this setup seems to be very frustrating while 
searching for dead-locks and race-conditions :-(

On the other hand I won't invest too much time for studying rocket 
engeneering only to use framework XYZ. Has anyone some ideas how to 
debug with less effort?

Thanks, Holger

--
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel

Re: [Flightgear-devel] [OT] Best pratice for debugging multi threaded programs?

[Curtis Olson]

Hi Holger,

This is a very interesting question.  I don't have any good answers, so I am
looking forward to seeing if anyone else on the list has any good
methodologies or even small hints and tips they can suggest.

I could stand up on my soap box here and give my long speech on threading
architectures ... but I'll spare everyone. :-)

The summary though is that threaded architectures add complexity and can
create timing bugs or order dependent bugs.  The end result can be a system
that works most of the time, and then randomly crashes for no apparent
reason.  Often it can take hours (if not days or weeks) of running the code
to tickle the bug ... and most of the time you have to trigger the bug to
see what happened and get any kind of idea where it happened so you can fix
it.  Also, even with a perfectly written threaded system, it's easy to come
back in 6 months and forget something about the design and make a change
that introduces a subtle problem.  This is even more likely if someone else
comes later and touches the code without having ever understood all the
nuances (sequence, and timing, etc.) of how all the threads work together.

Personally I try to avoid threads whenever possible, and only use them when
there is no sane alternative solution.  But I understand that threads are a
fact of life so they can't always be avoided.

When I've had to debug threaded code I've used a mix of gdb (and compiling
everything with full debugging symbols) to get a back trace if there is a
crash.  I also like to use printf's to see the sequence or progress of the
code as it runs.  If I get desperate I might haul out valgrind and see if
that offers any clues.  I've also spent hours just staring at the code and
mentally working through the sequencing of the threads and the
locking/mutual-exclusion primatives trying to analyze it all in my head and
look for logic bugs that way.  All of this is pretty tedious, un-fun
stuff.

Maybe someone else has other tips and tricks.  I don't think I've run across
any higher level methodologies that you can work through to always sniff out
any threading bug.

I know other people may feel differently about the use of threads and their
relative benefits and dangers ... and that's fine.  I like to think of
threads like nun-chucks.  I'll probably get moved over to the soprano
section of the choir before I manage to do anything useful too spectacular
with them. :-)

Curt.


On Mon, Mar 21, 2011 at 9:04 AM, Holger Wirtz wrote:

 Hi all,

 sorry - a little bit off-topic (in fact not so much as you might think,
 it's for a third-party software for FG):

 Has anyone some hints/websites/programs for debugging C -based multi
 threaded programs (exactly: glib based C code)? Currently I am
 developing with vi and a little bit gdb (command line) and very much
 debug output. But this setup seems to be very frustrating while
 searching for dead-locks and race-conditions :-(

 On the other hand I won't invest too much time for studying rocket
 engeneering only to use framework XYZ. Has anyone some ideas how to
 debug with less effort?

 Thanks, Holger


 --
 Colocation vs. Managed Hosting
 A question and answer guide to determining the best fit
 for your organization - today and in the future.
 http://p.sf.net/sfu/internap-sfd2d
 ___
 Flightgear-devel mailing list
 Flightgear-devel@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/flightgear-devel




-- 
Curtis Olson:
http://www.atiak.com - http://aem.umn.edu/~uav/
http://www.flightgear.org -
http://www.flightgear.org/blogs/category/curt/http://www.flightgear.org/blogs/category/personal/curt/
--
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel