Re: [osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
Hi Adam, On 9/24/07, Adam Coates [EMAIL PROTECTED] wrote: It looks like there's still a problem with the current setup. The Thread object itself can get deleted before the pthread actually gets to the lock (which, I guess is what you just added). If I understand correctly you are saying that the Thread object is being deleted before the static StartThread function gets called. In this case its rather an awkward problem to solve. Perhaps one could have lock made in the Thread::startThread method that is unlocked by the static StartThread function once it's complete. While it looks like this doesn't cause a crash, the problem is evidenced by valgrind (reports uninitalized memory reads/writes to the lock -- which got freed by the Thread destructor). You seem like you're pretty busy, so if you want, I can take a crack at this -- I think the Thread code is simple enough, but I'll need to think a bit to figure out how to get around this. One (simple) solution is to just join() the thread from the ~Thread() destructor. I expect the valgrind warnings are unrelated to this start problem. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
Hi Adamn, On 9/25/07, Robert Osfield [EMAIL PROTECTED] wrote: On 9/24/07, Adam Coates [EMAIL PROTECTED] wrote: It looks like there's still a problem with the current setup. The Thread object itself can get deleted before the pthread actually gets to the lock (which, I guess is what you just added). I have modified osgunittests so it now has a function that tests the creation, start and deletion of a thread, as per code example, you can run it via: osgunittests thread With the yesterdays code it worked fine on my system about 2/3rd of the times that it was run, more if I didn't add the delay (this is now a Thread::microSleep). The tests I did yesterday were done without the delay which is probably why I didn't see any crashes after my first round of fixes yesterday. Armed with a relatively reliable means for recreating the crash I've now set about refactoring the OpenThreads::Thread code so that instead of using a mutex in the start method and destructor it uses a OpenThreads::Block in the Thread::start() method and static StartThread() function such that the thread that calls the thread-startThread() will be held back until the new thread has been successfully started. With this change I'm now able to run the osgunittest thread test 1000 times without error (I use a repeat script to do this). I have also rolled these changes into Windows and IRIX spoc code paths, revert the previous Mutex based code. Could you check out the latest version in SVN and let me know if this problem is now properly fixed. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
Hi Adam,
I have recreated the crash, and have applied a fix to
OpenThreads::Thread to resolve it. The fix involved added a Mutex to
Thread which is used to prevent the startThread and destructor running
in parallel. This certainly resolves the crash under Linux for my
tests. I have rolled exactly the same changes into the Win32 and
Sproc code too as these will have exhibited exactly the same problem.
Could you do a svn update and let me know how you get on.
Cheers,
Robert.
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
I've attached the version I'm using that demonstrates the unusability
of the isRunning() flag.
The code pasted into the last e-mail is all that's necessary to
illustrate the crash, though it doesn't try to avoid it using the
'isRunning' flag. You shouldn't need to modify the OpenThreads
library.
I get the following error after running the attached code:
pure virtual method called
terminate called without an active exception
Aborted
presumably because thread-run() is being called on an object that has
been trashed.
AC
On 9/20/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
Could you send the exact code that you can recreate the issue with.
Robert.
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
Ah - just double-checked. It'll crash just fine without adding the
usleep() to OpenThreads; I had my builds mixed up.
AC
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
Hi, Robert,
The following is sufficient (basically what you said):
#include OpenThreads/Thread
class MyThread : public OpenThreads::Thread {
public:
void run(void) { }
};
int main(int argc, char* argv[]) {
{
MyThread thread;
thread.startThread();
}
while(1) ;
return 0;
}
You need the while(1) ; to prevent the process from dying and cleaning
up the thread before it wrecks. This alone wouldn't reproduce the bug
because the interleaving doesn't happen (it's a very short window in
which it can happen). To force the bug, I added ::usleep(100) at
the beginning of ThreadPrivateActions::StartThread().
AC
On 9/20/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
So, I just realized that the OSG folks are not synonymous with the
OpenThreads folks. ^_^ Sorry for dropping this on the wrong list
then.
Well OpenThreads list is a flat line, almost all (99.9%) OpenThreads
activity and development is done by members of the OpenSceneGraph
community. So this is probably the best place to raise issues like
this. Responsibility for maintaining OpenThreads has also fallen on
my shoulders as it effectively became an orphaned project when the
original project lead got sucked into non coding management.
In that light, it may not even be considered a bug in OT,
since, theoretically, it might be unfair to call the Thread
destructor
while the thread is running (even though I'm pretty sure they
intended
isRunning() to work as we'd expect).
Well this type of usage is kinda abusing the Thread class, but I'd
still say this way a bug.
It should be possible to add a workaround in the following way:
1. Before starting the thread, set a flag to indicate that the
thread
is starting up.
2. Have the thread's run() routine unset this flag.
Before deleting the Thread, make sure that both the starting up
flag
and the isRunning flag are false. One of these will have to be true
if the thread has not reached the run() method yet, or has not
completely exited from run() -- thus avoiding the problem.
I'll have ponder in this issue, I'm more inclined to use a mutex in
some way that just a straight flag.
My question: does somebody want me to implement this workaround and
submit it to you? Alternatively, I can just point out the break to
the OpenThreads people and we can wait for a patch...
It's best to just roll your sleeves up and get stuck in. First up we
need a test that can reliably reproduce the error. Would the
following code block do the trick?
{
OpenThreads::Thread thread;
thread.startThread();
}
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
Re: [osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
Yup - builds fine, and it resolved the crash in my system as well.
Thanks for taking care of it!!
AC
On 9/24/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
I have recreated the crash, and have applied a fix to
OpenThreads::Thread to resolve it. The fix involved added a Mutex to
Thread which is used to prevent the startThread and destructor running
in parallel. This certainly resolves the crash under Linux for my
tests. I have rolled exactly the same changes into the Win32 and
Sproc code too as these will have exhibited exactly the same problem.
Could you do a svn update and let me know how you get on.
Cheers,
Robert.
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
I've attached the version I'm using that demonstrates the unusability
of the isRunning() flag.
The code pasted into the last e-mail is all that's necessary to
illustrate the crash, though it doesn't try to avoid it using the
'isRunning' flag. You shouldn't need to modify the OpenThreads
library.
I get the following error after running the attached code:
pure virtual method called
terminate called without an active exception
Aborted
presumably because thread-run() is being called on an object that has
been trashed.
AC
On 9/20/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
Could you send the exact code that you can recreate the issue with.
Robert.
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
Ah - just double-checked. It'll crash just fine without adding the
usleep() to OpenThreads; I had my builds mixed up.
AC
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
Hi, Robert,
The following is sufficient (basically what you said):
#include OpenThreads/Thread
class MyThread : public OpenThreads::Thread {
public:
void run(void) { }
};
int main(int argc, char* argv[]) {
{
MyThread thread;
thread.startThread();
}
while(1) ;
return 0;
}
You need the while(1) ; to prevent the process from dying and cleaning
up the thread before it wrecks. This alone wouldn't reproduce the bug
because the interleaving doesn't happen (it's a very short window in
which it can happen). To force the bug, I added ::usleep(100) at
the beginning of ThreadPrivateActions::StartThread().
AC
On 9/20/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
So, I just realized that the OSG folks are not synonymous with the
OpenThreads folks. ^_^ Sorry for dropping this on the wrong list
then.
Well OpenThreads list is a flat line, almost all (99.9%)
OpenThreads
activity and development is done by members of the OpenSceneGraph
community. So this is probably the best place to raise issues like
this. Responsibility for maintaining OpenThreads has also fallen on
my shoulders as it effectively became an orphaned project when the
original project lead got sucked into non coding management.
In that light, it may not even be considered a bug in OT,
since, theoretically, it might be unfair to call the Thread
destructor
while the thread is running (even though I'm pretty sure they
intended
isRunning() to work as we'd expect).
Well this type of usage is kinda abusing the Thread class, but I'd
still say this way a bug.
It should be possible to add a workaround in the following way:
1. Before starting the thread, set a flag to indicate that the
thread
is starting up.
2. Have the thread's run() routine unset this flag.
Before deleting the Thread, make sure that both the starting up
flag
and the isRunning flag are false. One of these will have to be
true
if the thread has not reached the run() method yet, or has not
completely exited from run() -- thus avoiding the problem.
I'll have ponder in this issue, I'm more inclined to use a mutex in
some way that just a straight flag.
My question: does somebody want me to implement this workaround
and
submit it to you? Alternatively, I can just point out the break
to
the OpenThreads people and we can wait for a patch...
It's best to just roll your sleeves up and get stuck in. First up
we
need a test that can reliably reproduce the error. Would the
following code block do the trick?
{
OpenThreads::Thread thread;
thread.startThread();
}
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
Re: [osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
Hey, Robert, It looks like there's still a problem with the current setup. The Thread object itself can get deleted before the pthread actually gets to the lock (which, I guess is what you just added). While it looks like this doesn't cause a crash, the problem is evidenced by valgrind (reports uninitalized memory reads/writes to the lock -- which got freed by the Thread destructor). You seem like you're pretty busy, so if you want, I can take a crack at this -- I think the Thread code is simple enough, but I'll need to think a bit to figure out how to get around this. One (simple) solution is to just join() the thread from the ~Thread() destructor. Cheers, Adam ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
Hi, Robert,
The following is sufficient (basically what you said):
#include OpenThreads/Thread
class MyThread : public OpenThreads::Thread {
public:
void run(void) { }
};
int main(int argc, char* argv[]) {
{
MyThread thread;
thread.startThread();
}
while(1) ;
return 0;
}
You need the while(1) ; to prevent the process from dying and cleaning
up the thread before it wrecks. This alone wouldn't reproduce the bug
because the interleaving doesn't happen (it's a very short window in
which it can happen). To force the bug, I added ::usleep(100) at
the beginning of ThreadPrivateActions::StartThread().
AC
On 9/20/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
So, I just realized that the OSG folks are not synonymous with the
OpenThreads folks. ^_^ Sorry for dropping this on the wrong list
then.
Well OpenThreads list is a flat line, almost all (99.9%) OpenThreads
activity and development is done by members of the OpenSceneGraph
community. So this is probably the best place to raise issues like
this. Responsibility for maintaining OpenThreads has also fallen on
my shoulders as it effectively became an orphaned project when the
original project lead got sucked into non coding management.
In that light, it may not even be considered a bug in OT,
since, theoretically, it might be unfair to call the Thread destructor
while the thread is running (even though I'm pretty sure they intended
isRunning() to work as we'd expect).
Well this type of usage is kinda abusing the Thread class, but I'd
still say this way a bug.
It should be possible to add a workaround in the following way:
1. Before starting the thread, set a flag to indicate that the thread
is starting up.
2. Have the thread's run() routine unset this flag.
Before deleting the Thread, make sure that both the starting up flag
and the isRunning flag are false. One of these will have to be true
if the thread has not reached the run() method yet, or has not
completely exited from run() -- thus avoiding the problem.
I'll have ponder in this issue, I'm more inclined to use a mutex in
some way that just a straight flag.
My question: does somebody want me to implement this workaround and
submit it to you? Alternatively, I can just point out the break to
the OpenThreads people and we can wait for a patch...
It's best to just roll your sleeves up and get stuck in. First up we
need a test that can reliably reproduce the error. Would the
following code block do the trick?
{
OpenThreads::Thread thread;
thread.startThread();
}
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
Ah - just double-checked. It'll crash just fine without adding the
usleep() to OpenThreads; I had my builds mixed up.
AC
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
Hi, Robert,
The following is sufficient (basically what you said):
#include OpenThreads/Thread
class MyThread : public OpenThreads::Thread {
public:
void run(void) { }
};
int main(int argc, char* argv[]) {
{
MyThread thread;
thread.startThread();
}
while(1) ;
return 0;
}
You need the while(1) ; to prevent the process from dying and cleaning
up the thread before it wrecks. This alone wouldn't reproduce the bug
because the interleaving doesn't happen (it's a very short window in
which it can happen). To force the bug, I added ::usleep(100) at
the beginning of ThreadPrivateActions::StartThread().
AC
On 9/20/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
So, I just realized that the OSG folks are not synonymous with the
OpenThreads folks. ^_^ Sorry for dropping this on the wrong list
then.
Well OpenThreads list is a flat line, almost all (99.9%) OpenThreads
activity and development is done by members of the OpenSceneGraph
community. So this is probably the best place to raise issues like
this. Responsibility for maintaining OpenThreads has also fallen on
my shoulders as it effectively became an orphaned project when the
original project lead got sucked into non coding management.
In that light, it may not even be considered a bug in OT,
since, theoretically, it might be unfair to call the Thread destructor
while the thread is running (even though I'm pretty sure they intended
isRunning() to work as we'd expect).
Well this type of usage is kinda abusing the Thread class, but I'd
still say this way a bug.
It should be possible to add a workaround in the following way:
1. Before starting the thread, set a flag to indicate that the thread
is starting up.
2. Have the thread's run() routine unset this flag.
Before deleting the Thread, make sure that both the starting up flag
and the isRunning flag are false. One of these will have to be true
if the thread has not reached the run() method yet, or has not
completely exited from run() -- thus avoiding the problem.
I'll have ponder in this issue, I'm more inclined to use a mutex in
some way that just a straight flag.
My question: does somebody want me to implement this workaround and
submit it to you? Alternatively, I can just point out the break to
the OpenThreads people and we can wait for a patch...
It's best to just roll your sleeves up and get stuck in. First up we
need a test that can reliably reproduce the error. Would the
following code block do the trick?
{
OpenThreads::Thread thread;
thread.startThread();
}
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
Hi Adam,
Could you send the exact code that you can recreate the issue with.
Robert.
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
Ah - just double-checked. It'll crash just fine without adding the
usleep() to OpenThreads; I had my builds mixed up.
AC
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
Hi, Robert,
The following is sufficient (basically what you said):
#include OpenThreads/Thread
class MyThread : public OpenThreads::Thread {
public:
void run(void) { }
};
int main(int argc, char* argv[]) {
{
MyThread thread;
thread.startThread();
}
while(1) ;
return 0;
}
You need the while(1) ; to prevent the process from dying and cleaning
up the thread before it wrecks. This alone wouldn't reproduce the bug
because the interleaving doesn't happen (it's a very short window in
which it can happen). To force the bug, I added ::usleep(100) at
the beginning of ThreadPrivateActions::StartThread().
AC
On 9/20/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
So, I just realized that the OSG folks are not synonymous with the
OpenThreads folks. ^_^ Sorry for dropping this on the wrong list
then.
Well OpenThreads list is a flat line, almost all (99.9%) OpenThreads
activity and development is done by members of the OpenSceneGraph
community. So this is probably the best place to raise issues like
this. Responsibility for maintaining OpenThreads has also fallen on
my shoulders as it effectively became an orphaned project when the
original project lead got sucked into non coding management.
In that light, it may not even be considered a bug in OT,
since, theoretically, it might be unfair to call the Thread destructor
while the thread is running (even though I'm pretty sure they intended
isRunning() to work as we'd expect).
Well this type of usage is kinda abusing the Thread class, but I'd
still say this way a bug.
It should be possible to add a workaround in the following way:
1. Before starting the thread, set a flag to indicate that the thread
is starting up.
2. Have the thread's run() routine unset this flag.
Before deleting the Thread, make sure that both the starting up flag
and the isRunning flag are false. One of these will have to be true
if the thread has not reached the run() method yet, or has not
completely exited from run() -- thus avoiding the problem.
I'll have ponder in this issue, I'm more inclined to use a mutex in
some way that just a straight flag.
My question: does somebody want me to implement this workaround and
submit it to you? Alternatively, I can just point out the break to
the OpenThreads people and we can wait for a patch...
It's best to just roll your sleeves up and get stuck in. First up we
need a test that can reliably reproduce the error. Would the
following code block do the trick?
{
OpenThreads::Thread thread;
thread.startThread();
}
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
I've attached the version I'm using that demonstrates the unusability
of the isRunning() flag.
The code pasted into the last e-mail is all that's necessary to
illustrate the crash, though it doesn't try to avoid it using the
'isRunning' flag. You shouldn't need to modify the OpenThreads
library.
I get the following error after running the attached code:
pure virtual method called
terminate called without an active exception
Aborted
presumably because thread-run() is being called on an object that has
been trashed.
AC
On 9/20/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
Could you send the exact code that you can recreate the issue with.
Robert.
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
Ah - just double-checked. It'll crash just fine without adding the
usleep() to OpenThreads; I had my builds mixed up.
AC
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
Hi, Robert,
The following is sufficient (basically what you said):
#include OpenThreads/Thread
class MyThread : public OpenThreads::Thread {
public:
void run(void) { }
};
int main(int argc, char* argv[]) {
{
MyThread thread;
thread.startThread();
}
while(1) ;
return 0;
}
You need the while(1) ; to prevent the process from dying and cleaning
up the thread before it wrecks. This alone wouldn't reproduce the bug
because the interleaving doesn't happen (it's a very short window in
which it can happen). To force the bug, I added ::usleep(100) at
the beginning of ThreadPrivateActions::StartThread().
AC
On 9/20/07, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Adam,
On 9/20/07, Adam Coates [EMAIL PROTECTED] wrote:
So, I just realized that the OSG folks are not synonymous with the
OpenThreads folks. ^_^ Sorry for dropping this on the wrong list
then.
Well OpenThreads list is a flat line, almost all (99.9%) OpenThreads
activity and development is done by members of the OpenSceneGraph
community. So this is probably the best place to raise issues like
this. Responsibility for maintaining OpenThreads has also fallen on
my shoulders as it effectively became an orphaned project when the
original project lead got sucked into non coding management.
In that light, it may not even be considered a bug in OT,
since, theoretically, it might be unfair to call the Thread destructor
while the thread is running (even though I'm pretty sure they intended
isRunning() to work as we'd expect).
Well this type of usage is kinda abusing the Thread class, but I'd
still say this way a bug.
It should be possible to add a workaround in the following way:
1. Before starting the thread, set a flag to indicate that the thread
is starting up.
2. Have the thread's run() routine unset this flag.
Before deleting the Thread, make sure that both the starting up flag
and the isRunning flag are false. One of these will have to be true
if the thread has not reached the run() method yet, or has not
completely exited from run() -- thus avoiding the problem.
I'll have ponder in this issue, I'm more inclined to use a mutex in
some way that just a straight flag.
My question: does somebody want me to implement this workaround and
submit it to you? Alternatively, I can just point out the break to
the OpenThreads people and we can wait for a patch...
It's best to just roll your sleeves up and get stuck in. First up we
need a test that can reliably reproduce the error. Would the
following code block do the trick?
{
OpenThreads::Thread thread;
thread.startThread();
}
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
ottest.cpp
Description: Binary data
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
[osg-users] OpenThreads threading bug on creation/deletion (leads to crashes)
I ran into a crash caused (apprently) by OpenThreads (pthread implementation). If you create a new thread, and then immediately destroy it, it is possible to delete the thread before it has been flagged as running. Thus, whoever in osgViewer is waiting on the thread to close ends up deleting the thread, never realizing that it has already been started (resulting in a crash). The OpenThread source (see Thread::~Thread() in Pthread.c++) suggests that even attempting to delete a thread while it's running is an error. But it, too, uses the isRunning flag as its signal that the thread is either running or not, so it doesn't even notice the error when the thread gets destroyed. isRunning is not actually a correct indicator of whether the thread is running, as presently implemented. I temporarily solved this by adding: pd-isRunning = true; just before pthread_create() in Thread::start(). This solves my problem, but it should probably be looked at / solved by somebody more familiar with the code. Modifying asynchronous code you didn't write is a bad bad feeling... =( This is in the OpenSceneGraph-2.1.11 SVN source. Thanks all, AC ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
