Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Al 15/10/10 22:31, En/na Vinzent Höfler ha escrit: - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) I don't agree, the creator of the thread should be able to see what's happening to it (i.e. keeping a reference to the thread without risking a SIGSEV because the thread has terminated). Bye -- Luca ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Sat, 16 Oct 2010, Luca Olivetti wrote: Al 15/10/10 22:31, En/na Vinzent Höfler ha escrit: - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) I don't agree, the creator of the thread should be able to see what's happening to it (i.e. keeping a reference to the thread without risking a SIGSEV because the thread has terminated). That's what the OnTerminate() event is for, that should definitely remain. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
- FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) (IIRC FreeOnTerminate was even set to False by the default constructor, so you had the choice of either using the default or introducing a race condition by setting it after the inherited Create - which starts the execute.) This problem has been solved with http://bugs.freepascal.org/view.php?id=16884 Jonas___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal i don't agree with always FreeOnTerminate. I have many prgramms that want to know the last status of the thread before they free it. But i think also the bug report you are linking to, shouldn't be a fpc bug. It is more a programmer bug. As programmer i have to know at which time i call the create methode of my TThread. So the correct programming would be: constructor TAThread.SomeCreate(aSuspended : boolean); begin [... do my things here ...] inherited create(aSuspended); end; or constructor TAThread.SomeCreate(aSuspended : boolean); begin inherited create(True); [... do my things here ...] if not aSuspended then Resume; end; I don't think fpc should slow down the thread creating, just to resolve problems of not good thread programming of the fpc users. When someone starts to programm threads, he has to know what he is doing. cu all Beni ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 16 Oct 2010, at 13:57, Benedikt Schindler wrote: - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) (IIRC FreeOnTerminate was even set to False by the default constructor, so you had the choice of either using the default or introducing a race condition by setting it after the inherited Create - which starts the execute.) This problem has been solved with http://bugs.freepascal.org/view.php?id=16884 But i think also the bug report you are linking to, shouldn't be a fpc bug. It is more a programmer bug. As programmer i have to know at which time i call the create methode of my TThread. He cannot. If freeonterminate=true, then even if you call inherited create(false) as the very last statement of your constructor, the thread may already have finished running and freed itself before AfterConstruction is called. The AfterConstruction call will then crash. Jonas___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Sat, Oct 16, 2010 at 7:04 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: He cannot. If freeonterminate=true, then even if you call inherited create(false) as the very last statement of your constructor, the thread may already have finished running and freed itself before AfterConstruction is called. The AfterConstruction call will then crash. Right. I think that it would have been totally acceptable due to the nature of threads especially on differing os(es) to not have AfterConstruction because of this very issue. The fact that threads can zip through their execute event and self terminate before the AfterConstruction is even called is definitely an issue that can only be solved with synchronization of some sort. Threads are a special case where these objects themselves represent a kernel process with its own distinct timeline. Logically, performance cannot be a priority when requiring synchronization of differing timelines. But for small projects where Delphi offers TThread with these features, it is good for FPC to accommodate Delphi source. There is always going to be a demographic who will never need performance and performance is not much of an issue for projects with 1000 threads to the observer. IMO, it would be better to leverage the present performance condition to muster support for a high performance thread component set. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Sat, 16 Oct 2010, Andrew Brunner wrote: On Sat, Oct 16, 2010 at 7:04 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: He cannot. If freeonterminate=true, then even if you call inherited create(false) as the very last statement of your constructor, the thread may already have finished running and freed itself before AfterConstruction is called. The AfterConstruction call will then crash. Right. I think that it would have been totally acceptable due to the nature of threads especially on differing os(es) to not have AfterConstruction because of this very issue. The fact that threads can zip through their execute event and self terminate before the AfterConstruction is even called is definitely an issue that can only be solved with synchronization of some sort. Threads are a special case where these objects themselves represent a kernel process with its own distinct timeline. Logically, performance cannot be a priority when requiring synchronization of differing timelines. But for small projects where Delphi offers TThread with these features, it is good for FPC to accommodate Delphi source. There is always going to be a demographic who will never need performance and performance is not much of an issue for projects with 1000 threads to the observer. IMO, it would be better to leverage the present performance condition to muster support for a high performance thread component set. How about creating a) A native TThreadManager record in system unit for unices. b) Create a alternative thread class (TFPThread, TRTLThread, TFastThread or whatever) that uses the TThreadManager record to create 'Fast' threads, which would not be encumbered by the Delphi heritage of TThread. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 16.10.2010 15:36, Michael Van Canneyt wrote: How about creating a) A native TThreadManager record in system unit for unices. I don't think this can be done for all Unix systems in a generic way, because e.g. on Linux you should use the clone syscall (which is Linux only) while on e.g. Mac OS X you should rely on the c library cause the syscall interface might be changed by Apple (@OS X devs: please correct me if this is wrong). Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
In our previous episode, Michael Van Canneyt said: IMO, it would be better to leverage the present performance condition to muster support for a high performance thread component set. How about creating a) A native TThreadManager record in system unit for unices. b) Create a alternative thread class (TFPThread, TRTLThread, TFastThread or whatever) that uses the TThreadManager record to create 'Fast' threads, which would not be encumbered by the Delphi heritage of TThread. If Delphi legacy is of no object, and it is new development, simply design in a threadpool. If construct/destruction is slow, then don't. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
In our previous episode, Sven Barth said: How about creating a) A native TThreadManager record in system unit for unices. I don't think this can be done for all Unix systems in a generic way, because e.g. on Linux you should use the clone syscall (which is Linux only) FreeBSD supports rfork, which has similar functionality (the subset (?) implemented by Clone in the linuxator is implemented by a rfork call Yrying to build on the clone call has been done in 1.0.x times, and I don't have fond memories about it. while on e.g. Mac OS X you should rely on the c library cause the syscall interface might be changed by Apple (@OS X devs: please correct me if this is wrong). First we would have to get rid of this solution in search of a problem attitude. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 16.10.2010 15:45, Marco van de Voort wrote: In our previous episode, Sven Barth said: How about creating a) A native TThreadManager record in system unit for unices. I don't think this can be done for all Unix systems in a generic way, because e.g. on Linux you should use the clone syscall (which is Linux only) FreeBSD supports rfork, which has similar functionality (the subset (?) implemented by Clone in the linuxator is implemented by a rfork call Yrying to build on the clone call has been done in 1.0.x times, and I don't have fond memories about it. I did a quick check: NPTL and thus the pthreads implementation on current Linux systems use clone as well. I don't know what the trouble were, but if we want a native thread manager on Linux we should try clone again (as well as using rfork on FreeBSD). while on e.g. Mac OS X you should rely on the c library cause the syscall interface might be changed by Apple (@OS X devs: please correct me if this is wrong). First we would have to get rid of this solution in search of a problem attitude. What do you mean by the this solution in search of a problem attitude? I can't follow you currently... :( Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
In our previous episode, Sven Barth said: FreeBSD supports rfork, which has similar functionality (the subset (?) implemented by Clone in the linuxator is implemented by a rfork call Yrying to build on the clone call has been done in 1.0.x times, and I don't have fond memories about it. I did a quick check: NPTL and thus the pthreads implementation on current Linux systems use clone as well. No doubt. I don't know what the trouble were, but if we want a native thread manager on Linux we should try clone again (as well as using rfork on FreeBSD). Well, I still would have to see a good case for a native manager made. But one of the problems with clone was IIRC that it wasn't easily wrapped like other syscalls. It will need to be done in asm per architecture IIRC. while on e.g. Mac OS X you should rely on the c library cause the syscall interface might be changed by Apple (@OS X devs: please correct me if this is wrong). First we would have to get rid of this solution in search of a problem attitude. What do you mean by the this solution in search of a problem attitude? I can't follow you currently... :( People seem to first want to bash pthreads, and discuss own threadmanagers before actually defining what the problem of the current solution is, and why it is absolutely needed to create something new. Like, euh, why simply having a thread pool to workaround slow creation is not a better solution (and as done for decades by vast numbers of programmers) All I have seen is some complaint about slow starting threads by sb who can't even find the proper pthread source, and now we are suddenly making room for a native threadmanager? A bit premature IMHO. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Sat, 16 Oct 2010 10:45:49 +0200, Luca Olivetti l...@ventoso.org wrote: Al 15/10/10 22:31, En/na Vinzent Höfler ha escrit: - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) I don't agree, the creator of the thread should be able to see what's happening to it (i.e. keeping a reference to the thread without risking a SIGSEV because the thread has terminated). Well, the usual implementation of an externally called Destroy is - first a call to the Terminate method - then a WaitFor() Unfortunately WaitFor maps quite directly to pthread_join() while a self-destruction is done by pthread_exit(). You can NOT call both without triggering undefined behaviour which - beginning with some 2.6 kernel - was a segmentation fault already, and you can not call pthread_join() from the same thread, this would deadlock. There are several solutions to the problem, each one with its own drawbacks. I would like to avoid additional synchronisation requirements as well as introducing new race conditions. With these implementation issues, implementing a race condition free WaitFor is already hard enough, but allowing Destroy to be called externally adds more issues than it can possibly solve. Vinzent. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 16.10.2010 16:47, Marco van de Voort wrote: In our previous episode, Sven Barth said: FreeBSD supports rfork, which has similar functionality (the subset (?) implemented by Clone in the linuxator is implemented by a rfork call Yrying to build on the clone call has been done in 1.0.x times, and I don't have fond memories about it. I did a quick check: NPTL and thus the pthreads implementation on current Linux systems use clone as well. No doubt. I don't know what the trouble were, but if we want a native thread manager on Linux we should try clone again (as well as using rfork on FreeBSD). Well, I still would have to see a good case for a native manager made. But one of the problems with clone was IIRC that it wasn't easily wrapped like other syscalls. It will need to be done in asm per architecture IIRC. Right... I forgot about that... while on e.g. Mac OS X you should rely on the c library cause the syscall interface might be changed by Apple (@OS X devs: please correct me if this is wrong). First we would have to get rid of this solution in search of a problem attitude. What do you mean by the this solution in search of a problem attitude? I can't follow you currently... :( People seem to first want to bash pthreads, and discuss own threadmanagers before actually defining what the problem of the current solution is, and why it is absolutely needed to create something new. One of the problem seems to be that we need to link an external library. While FPC allows us to do so much things with Pascal code only, we need to add a dependency when we want to use threading. Note: On Windows this feels different, as here kernel32.dll is THE interface to the OS, while libc/pthread aren't on Unix systems (there the OS interface consists of the syscalls to the kernel). Like, euh, why simply having a thread pool to workaround slow creation is not a better solution (and as done for decades by vast numbers of programmers) Maybe we should provide a generic thread pool system in the FCL? Then some FPC users might realize that they don't really need a faster thread creation. All I have seen is some complaint about slow starting threads by sb who can't even find the proper pthread source, and now we are suddenly making room for a native threadmanager? Isn't a native threadmanager for Linux on the wishlist of FPC users for quite some time now? :) Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Al 16/10/10 16:57, En/na Vinzent Höfler ha escrit: Well, the usual implementation of an externally called Destroy is - first a call to the Terminate method - then a WaitFor() Nope, I avoid the WaitFor. I usually do a while not FFinished do CheckSynchronize(100); (where FFinished is set when Execute ends). Bye -- Luca ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Sat, 16 Oct 2010 17:42:03 +0200, Luca Olivetti l...@ventoso.org wrote: Al 16/10/10 16:57, En/na Vinzent Höfler ha escrit: Well, the usual implementation of an externally called Destroy is - first a call to the Terminate method - then a WaitFor() Nope, I avoid the WaitFor. Nice for you, but I'd rather avoid polling solutions. I usually do a while not FFinished do CheckSynchronize(100); Check again. You can't even access FFinished from a foreign thread. Vinzent. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Al 16/10/10 17:42, En/na Luca Olivetti ha escrit:
Al 16/10/10 16:57, En/na Vinzent Höfler ha escrit:
Well, the usual implementation of an externally called Destroy is
- first a call to the Terminate method
- then a WaitFor()
Nope, I avoid the WaitFor. I usually do a
while not FFinished do
CheckSynchronize(100);
(where FFinished is set when Execute ends).
Btw, this is what a C++ class (cThread in vdr, simple but nice, even it
it's C++ ;) does
cThread::~cThread()
{
Cancel(); // just in case the derived class didn't call it
free(description);
}
void cThread::Cancel(int WaitSeconds)
{
running = false;
if (active WaitSeconds -1) {
if (WaitSeconds 0) {
for (time_t t0 = time(NULL) + WaitSeconds; time(NULL) t0; ) {
if (!Active())
return;
cCondWait::SleepMs(10);
}
esyslog(ERROR: %s thread %d won't end (waited %d seconds) -
canceling it..., description ? description : , childThreadId,
WaitSeconds);
}
pthread_cancel(childTid);
childTid = 0;
active = false;
}
}
bool cThread::Active(void)
{
if (active) {
//
// Single UNIX Spec v2 says:
//
// The pthread_kill() function is used to request
// that a signal be delivered to the specified thread.
//
// As in kill(), if sig is zero, error checking is
// performed but no signal is actually sent.
//
int err;
if ((err = pthread_kill(childTid, 0)) != 0) {
if (err != ESRCH)
LOG_ERROR;
childTid = 0;
active = running = false;
}
else
return true;
}
return false;
}
You can check the whole class here:
http://projects.vdr-developer.org/projects/vdr/repository/revisions/master/entry/thread.h
http://projects.vdr-developer.org/projects/vdr/repository/revisions/master/entry/thread.c
Bye
--
Luca
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Sat, 16 Oct 2010 18:04:59 +0200, Luca Olivetti l...@ventoso.org wrote: I use that only for threads I start at the beginning of the app and terminate at the end, so polling isn't a big issue. Those kind of threads are not a big issue at all. As they are running during the whole application's lifetime, race conditions with concurrently running code are about as improbable as the universe collapsing right now. It's not strictly necessary since the OS will clean up everything afterwards, but I prefer to do things properly. And not call pthread_cancel() like the C++ class you given as example and stop threads in the middle of their work? ;) They don't even bother and create all threads in detached state, so you can not pthread_join() and get an exitcode from them. while not FFinished do CheckSynchronize(100); Check again. You can't even access FFinished from a foreign thread. I can (FFinished is a private member defined by me, and Destroy, being a method of the same class, can access private members). Yes, my mistake. I misinterpreted the lines. Vinzent. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
In our previous episode, Sven Barth said: First we would have to get rid of this solution in search of a problem attitude. What do you mean by the this solution in search of a problem attitude? I can't follow you currently... :( People seem to first want to bash pthreads, and discuss own threadmanagers before actually defining what the problem of the current solution is, and why it is absolutely needed to create something new. One of the problem seems to be that we need to link an external library. While FPC allows us to do so much things with Pascal code only, we need to add a dependency when we want to use threading. Yes. So all your programs don't use any libraries except libc because pthreads? If so, what kind of programs do you make? Note: On Windows this feels different, as here kernel32.dll is THE interface to the OS, while libc/pthread aren't on Unix systems (there the OS interface consists of the syscalls to the kernel). I don't see it that way. I'm perfectly happy to take libc/pthread as system interface. If they actually went to the trouble to make it a decent API, somewhat stable in time, and decoupled from e.g. compiler version changes. THAT is the *nix problem, not the name of the library, be it user32 or libc or libroot. Still, while I agree with the problem, I don't think a native threadlib will change anything in this picture. The amount of pain won't lessen. One only adds yet another factor (libc threads vs pascal threads) to the chaos. The only possible thing is wait till Linux and the Unix world at large comes to its senses. Like, euh, why simply having a thread pool to workaround slow creation is not a better solution (and as done for decades by vast numbers of programmers) Maybe we should provide a generic thread pool system in the FCL? Then some FPC users might realize that they don't really need a faster thread creation. I think all sane contributions are welcomed. One can never fully exclude that somebody might actually come up with both a good Pascal only threading library and a good motivation to actually use it. ( I can imagine some limited uses to really quick initialize CGIs or so) But talking about making room in the FPC implementation before these two conditions are satisfied, just because some people think there is a problem, and a solution for that problem. It will just lead to orphaned attempts and a drain on FPC resources. Just look at the discussion about the current TThread implementation, which afaik stems from the 2003-2004 timeframe (when what is now the 2.x series was finalized in the 1.9 beta series). And that is just library usage. How long do you think it will take to get a native implementation up to snuff for real use, beyond being some minimalists pet project ? Even if I forget for the moment that it is DOA out of principle (since it won't play nice with external libs probably), it would take years. And all the while the forces concentrate on that, the current implementation is on ice, waiting for the glorious new code that will fix everything and will never come. Better invest in dressing up the current implementation, synchronization features (syncobjs, TMonitor, locked queues etc), stuff that is really going to be used. Less glory, more result. All I have seen is some complaint about slow starting threads by sb who can't even find the proper pthread source, and now we are suddenly making room for a native threadmanager? Isn't a native threadmanager for Linux on the wishlist of FPC users for quite some time now? :) Not mine. And yes, there has been a lot of echoing of sentiment, but I never saw a decent problem definition, and what a native solution would actually fix. The native threadlib has been a recurring topic since the 1.0.x implementation was trashed, which surfaces at any possible problem in the current thread implementation (which is blown out of proportion to make a case for start over). ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 16.10.2010 18:30, Marco van de Voort wrote: In our previous episode, Sven Barth said: First we would have to get rid of this solution in search of a problem attitude. What do you mean by the this solution in search of a problem attitude? I can't follow you currently... :( People seem to first want to bash pthreads, and discuss own threadmanagers before actually defining what the problem of the current solution is, and why it is absolutely needed to create something new. One of the problem seems to be that we need to link an external library. While FPC allows us to do so much things with Pascal code only, we need to add a dependency when we want to use threading. Yes. So all your programs don't use any libraries except libc because pthreads? If so, what kind of programs do you make? I didn't say that I personally have this problem; it's just the one problem that I'm really aware of. :) But I even have an example where I don't even use libc, but use threads: a downloader (using lnet). Ok... I've recently switched from lnet and threading to calling wget with TProcess, because it was to unreliable, but I HAD such an application where I'd liked to not have any dependency. ^^ Another application in need of native threading might come in some time, because I have an arm-linux system with uclibc where I not yet know how good and reliable its NPTL implementation is. Note: On Windows this feels different, as here kernel32.dll is THE interface to the OS, while libc/pthread aren't on Unix systems (there the OS interface consists of the syscalls to the kernel). I don't see it that way. I'm perfectly happy to take libc/pthread as system interface. If they actually went to the trouble to make it a decent API, somewhat stable in time, and decoupled from e.g. compiler version changes. THAT is the *nix problem, not the name of the library, be it user32 or libc or libroot. That's somehow that what I meant, but with different words. libc/pthread aren't yet stable enough in my opinion to be seen as OS interface. I'd have no problem with them as well if they'd remain stable... Still, while I agree with the problem, I don't think a native threadlib will change anything in this picture. The amount of pain won't lessen. One only adds yet another factor (libc threads vs pascal threads) to the chaos. Yes, that's the biggest problem... :( The only possible thing is wait till Linux and the Unix world at large comes to its senses. Which might take some time... Like, euh, why simply having a thread pool to workaround slow creation is not a better solution (and as done for decades by vast numbers of programmers) Maybe we should provide a generic thread pool system in the FCL? Then some FPC users might realize that they don't really need a faster thread creation. I think all sane contributions are welcomed. One can never fully exclude that somebody might actually come up with both a good Pascal only threading library and a good motivation to actually use it. ( I can imagine some limited uses to really quick initialize CGIs or so) But talking about making room in the FPC implementation before these two conditions are satisfied, just because some people think there is a problem, and a solution for that problem. Who's talking about making room? I thought the recent discussion was about having an alternative thread class that doesn't have the TThread heritage. And also having the possibility to switch between native threads or pthreads (on platforms of which the kernel provides such threading facilities). I don't see why you'd need to make room in FPC's implementation for them. They'd simply be alternatives. :) It will just lead to orphaned attempts and a drain on FPC resources. Just look at the discussion about the current TThread implementation, which afaik stems from the 2003-2004 timeframe (when what is now the 2.x series was finalized in the 1.9 beta series). And that is just library usage. How long do you think it will take to get a native implementation up to snuff for real use, beyond being some minimalists pet project ? Even if I forget for the moment that it is DOA out of principle (since it won't play nice with external libs probably), it would take years. Regarding orphaned attempts: I know what you're talking about. I always think that I've started something without finishing it when I think about my two FPC pet projects cppclass and nativent. But I still have the hope that I'll continue working on them :) And all the while the forces concentrate on that, the current implementation is on ice, waiting for the glorious new code that will fix everything and will never come. Better invest in dressing up the current implementation, synchronization features (syncobjs, TMonitor, locked queues etc), stuff that is really going to be used. I really, really must do more serious work with threads... Regards, Sven
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Sat, 16 Oct 2010, Marco van de Voort wrote: All I have seen is some complaint about slow starting threads by sb who can't even find the proper pthread source, and now we are suddenly making room for a native threadmanager? Isn't a native threadmanager for Linux on the wishlist of FPC users for quite some time now? :) Not mine. And yes, there has been a lot of echoing of sentiment, but I never saw a decent problem definition, and what a native solution would actually fix. It would allow me to write programs that are multithreaded and don't use external libs. And yes, they do exist. Not GUI, but server apps. Now, the only dependency is often libc for the threads. There is a clear target: FPC programs without external libs, capable of multi-threading and thread synchronization, which basically means: - Ability to create threads. - Ability to use mutexes across threads. - Ability to use semaphores. It's not because you don't see the value, that other people don't. So at least, please don't stop them from trying to go somewhere, even if you don't want to go there. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 20:09, schrieb Andrew Brunner: What are the exact differences from this test to your last one? Would be nice to know that... :) Every barrier causes a significant increase in time. In high performance parallel computing time is something we minimize. Big reductions in linear execution yield massive performance gains in the parallel environments. So when I removed the FPC barrier that gave me a significant increase in performance. I also used the parent's RTLEvent instead of creating one for each thread which added another boost. Then i removed another barrier for adding elements to the watch list. All together, that solved my problem. Which FPC barrier did you remove? What parent are you talking about? Which other barrier did you remove? Or to put it into a single question: what exactly (code) did you change in the RTL? Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Thu, 14 Oct 2010, Andrew Brunner wrote: What are the exact differences from this test to your last one? Would be nice to know that... :) Every barrier causes a significant increase in time. In high performance parallel computing time is something we minimize. Big reductions in linear execution yield massive performance gains in the parallel environments. So when I removed the FPC barrier that gave me a significant increase in performance. I also used the parent's RTLEvent instead of creating one for each thread which added another boost. Then i removed another barrier for adding elements to the watch list. All together, that solved my problem. Can you share your changes ? Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 15 Oct 2010, at 15:54, Andrew Brunner wrote:
rtl changes alone added 20-30% speed increase over the test case.
And in the process they break all tthread suspending under Unix.
CurrentTM.SuspendThread() will always fail there (it calls through to
CSuspentThread):
function CSuspendThread (threadHandle : TThreadID) : dword;
begin
{ pthread_kill(SIGSTOP) cannot be used, because posix-compliant
implementations then freeze the entire process instead of only
the target thread. Suspending a particular thread is not
supported by posix nor by most *nix implementations, presumably
because of concerns mentioned in E.4 at
http://pauillac.inria.fr/~xleroy/linuxthreads/faq.html#E and in
http://java.sun.com/j2se/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html
}
// result := pthread_kill(threadHandle,SIGSTOP);
result:=dword(-1);
end;
You replaced a bunch of semaphore create/lock/unlock/destroy
operations with calls to a function that does not do anything.
My reasoning for removing the barrier was because that it's purpose
was not warranted. Users who create threads know to set values of
variables before calling the inherited create method to their own
creation. If I'm on a massively scaled machine I know how to do get
threads to execute properly and wait w/o taking up CPU time. I don't
want the development platform slowing things down for the few that
don't understand how to create a thread.
Code snippet from actual source to pThreads...
void __pthread_restart_new(pthread_descr th)
{
/* The barrier is proabably not needed, in which case it still
documents
our assumptions. The intent is to commit previous writes to shared
memory so the woken thread will have a consistent view.
Complementary
read barriers are present to the suspend functions. */
WRITE_MEMORY_BARRIER();
kill(th-p_pid, __pthread_sig_restart);
}
By their own documentation they admit their own barrier is not
needed. LOL.
That barrier has nothing to do with set values of variables before
calling the inherited create method to their own creation (although
it's not entirely clear to me you mean by that). If you set values
that the child thread will read after that thread has already been
created, this memory barrier will not help either. That barrier is
again about memory write ordering, which could cause the newly started
thread to not see all writes performed by the parent thread *before*
creating the child thread.
And the reason that's it's probably not required, is because there's
so much code in between that it's unlikely that any of those writes
would still be outstanding at that point. It's not for the few that
don't understand how to create a thread.
Jonas
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Thank you,
I'll study it to see if/how we can do something to increase speed of creating
threads
in FPC.
But if I understand you correctly, the WRITE_MEMORY_BARRIER() call is
beyond our reach, so there's little we can do about that.
Michael.
On Fri, 15 Oct 2010, Andrew Brunner wrote:
rtl changes alone added 20-30% speed increase over the test case. The
other changes were made to the actual test case.
My reasoning for removing the barrier was because that it's purpose
was not warranted. Users who create threads know to set values of
variables before calling the inherited create method to their own
creation. If I'm on a massively scaled machine I know how to do get
threads to execute properly and wait w/o taking up CPU time. I don't
want the development platform slowing things down for the few that
don't understand how to create a thread.
Code snippet from actual source to pThreads...
void __pthread_restart_new(pthread_descr th)
{
/* The barrier is proabably not needed, in which case it still documents
our assumptions. The intent is to commit previous writes to shared
memory so the woken thread will have a consistent view. Complementary
read barriers are present to the suspend functions. */
WRITE_MEMORY_BARRIER();
kill(th-p_pid, __pthread_sig_restart);
}
By their own documentation they admit their own barrier is not needed. LOL.
On Fri, Oct 15, 2010 at 3:09 AM, Michael Van Canneyt
mich...@freepascal.org wrote:
Can you share your changes ?
No problem. See attachments.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: You replaced a bunch of semaphore create/lock/unlock/destroy operations with calls to a function that does not do anything. I did not. I REMOVED create/destroy/lock/unlock. I ADDED calls to the ThreadManager's own functions for thread Resume and Suspend. Did you try executing the suspend command? What happened? That barrier has nothing to do with set values of variables before calling the inherited create method to their own creation (although it's not entirely clear to me you mean by that). If you set values that the child thread will read after that thread has already been created, this memory barrier will not help either. That barrier is again about memory write ordering, which could cause the newly started thread to not see all writes performed by the parent thread *before* creating the child thread. Other than waiting for the values to be committed, I just don't see it's value. And the reason that's it's probably not required, is because there's so much code in between that it's unlikely that any of those writes would still be outstanding at that point. It's not for the few that don't understand how to create a thread. Exactly. The other code being what? ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, Oct 15, 2010 at 9:24 AM, Michael Van Canneyt mich...@freepascal.org wrote: Thank you, I'll study it to see if/how we can do something to increase speed of creating threads in FPC. But if I understand you correctly, the WRITE_MEMORY_BARRIER() call is beyond our reach, so there's little we can do about that. Michael. Thanks for all your help too! I'm just wanting a really stable program executable. My goal is to make sure FPC is stable and fast as possible under Linux. So any tire kicking I do is for the benefit of FPC - nothing else. It was also suggested that PThread's Suspend and Resume may not do anything. What I would do (if that is the case), is try to find where PThread has access to the Semaphore or CriticalSection it has on that thread... If you could tie into that system, you would be able to not have that extra inefficiency of the FPC Semaphore (and any the creator of TThread may have). All these barriers add serious inline time. IMO. You must assume that all variables of the TThread descendant have already been assigned and are valid. There is no need to block the thread from execution. That is unless they are asking to have it suspended on creation. And if that is the case... We need to tie into PThread's Semaphore or Event or CriticalSection and exploit that. It's going to lead to a really significant gain for TThread.Create. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 15.10.2010 16:31, schrieb Andrew Brunner: On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebejonas.ma...@elis.ugent.be wrote: You replaced a bunch of semaphore create/lock/unlock/destroy operations with calls to a function that does not do anything. I did not. I REMOVED create/destroy/lock/unlock. I ADDED calls to the ThreadManager's own functions for thread Resume and Suspend. Did you try executing the suspend command? What happened? What do you believe happens with a thread when you execute a function that only returns dword(-1) and does nothing else? :) Take a look at Jonas code excerpt from rtl/unix/cthreads.pp again and you'll see that the call to pthread_kill in CSuspendThread is disabled and has a big comment before it. Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 15 Oct 2010, at 16:31, Andrew Brunner wrote:
On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebe jonas.ma...@elis.ugent.be
wrote:
You replaced a bunch of semaphore create/lock/unlock/destroy
operations with
calls to a function that does not do anything.
I did not. I REMOVED create/destroy/lock/unlock. I ADDED calls to
the ThreadManager's own functions for thread Resume and Suspend.
Yes, that's what I meant, because the ThreadManeger's SuspendThread
function does not do anything under Unix.
Did
you try executing the suspend command? What happened?
First of all, if you use tthread.create(true) (i.e., create a
suspended thread), then the execute method will never be called:
***
{$mode delphi}
uses
cthreads, classes, sysutils;
type
tmythread = class(tthread)
procedure execute;override;
end;
procedure tmythread.execute;
begin
writeln('executing');
end;
var
c: tmythread;
begin
writeln('creating thread suspended');
c:=tmythread.create(true);
sleep(500);
writeln('resuming thread');
c.resume;
c.waitfor;
c.free;
end.
***
Currently writes:
creating thread suspended
resuming thread
executing
With the patch, the last line is missing.
Secondly, ignoring this bug and suspending the thread inside itself:
***
{$mode delphi}
uses
cthreads, classes, sysutils;
type
tmythread = class(tthread)
procedure execute;override;
end;
procedure tmythread.execute;
begin
suspend;
writeln('executing');
end;
var
c: tmythread;
begin
writeln('creating thread suspended');
c:=tmythread.create(false);
sleep(500);
writeln('resuming thread');
c.resume;
c.waitfor;
c.free;
end.
***
Currently writes:
creating thread suspended
resuming thread
executing
With the patch:
creating thread suspended
executing
resuming thread
That barrier has nothing to do with set values of variables before
calling
the inherited create method to their own creation (although it's not
entirely clear to me you mean by that). If you set values that the
child
thread will read after that thread has already been created, this
memory
barrier will not help either. That barrier is again about memory
write
ordering, which could cause the newly started thread to not see all
writes
performed by the parent thread *before* creating the child thread.
Other than waiting for the values to be committed, I just don't see
it's value.
The whole value is in waiting for the values to be committed. Without
the write barrier, this could e.g. happen:
a) in parent thread
* globalvar:=1;
* create_thread();
b) in child thread:
writeln(gobalvar); // prints 0, or whatever the value of globalvar was
before the parent thread assigned 1 to it
That can even happen on x86 in its default mode, because this is a
Stores reordered after Loads situation (but as mentioned below, it's
unlikely to happen).
And the reason that's it's probably not required, is because
there's so much
code in between that it's unlikely that any of those writes would
still be
outstanding at that point. It's not for the few that don't
understand how
to create a thread.
Exactly. The other code being what?
If you mean the code in between by the other code: any code
executed between setting the global variable in the parent thread and
the actual starting of the child thread (which includes all of the
code in pthread_create, and in case of FPC in tthread.create and/or
BeginThread, although in case of tthread.create there is also some
extra communication between the parent thread and the child thread via
the heap).
Jonas
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, Oct 15, 2010 at 9:55 AM, Sven Barth pascaldra...@googlemail.com wrote: Am 15.10.2010 16:31, schrieb Andrew Brunner: On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebejonas.ma...@elis.ugent.be wrote: You replaced a bunch of semaphore create/lock/unlock/destroy operations with calls to a function that does not do anything. I did not. I REMOVED create/destroy/lock/unlock. I ADDED calls to the ThreadManager's own functions for thread Resume and Suspend. Did you try executing the suspend command? What happened? What do you believe happens with a thread when you execute a function that only returns dword(-1) and does nothing else? :) Take a look at Jonas code excerpt from rtl/unix/cthreads.pp again and you'll see that the call to pthread_kill in CSuspendThread is disabled and has a big comment before it. Regards, Sven Now that is funny. I guess I made the assumption that the call the Pthreads Suspend thread would actually do what is advertised, at lease on good faith... Who else has the source to PThreads? Anyone care to read it over? I have it. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 15.10.2010 17:11, schrieb Andrew Brunner: On Fri, Oct 15, 2010 at 9:55 AM, Sven Barthpascaldra...@googlemail.com wrote: Am 15.10.2010 16:31, schrieb Andrew Brunner: On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebejonas.ma...@elis.ugent.be wrote: You replaced a bunch of semaphore create/lock/unlock/destroy operations with calls to a function that does not do anything. I did not. I REMOVED create/destroy/lock/unlock. I ADDED calls to the ThreadManager's own functions for thread Resume and Suspend. Did you try executing the suspend command? What happened? What do you believe happens with a thread when you execute a function that only returns dword(-1) and does nothing else? :) Take a look at Jonas code excerpt from rtl/unix/cthreads.pp again and you'll see that the call to pthread_kill in CSuspendThread is disabled and has a big comment before it. Regards, Sven Now that is funny. I guess I made the assumption that the call the Pthreads Suspend thread would actually do what is advertised, at lease on good faith... Who else has the source to PThreads? Anyone care to read it over? I have it. The problem isn't the pthreads library. The problem is the POSIX threading specification of which pthreads is an implementation (look at the comment in CSuspendThread again). Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, Oct 15, 2010 at 10:06 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: First of all, if you use tthread.create(true) (i.e., create a suspended thread), then the execute method will never be called: BTW: I did notice that the semaphore was being used to suspend the thread. And I also know that that was it's only function... Suspend and Resume. At this point I would suggest we can do one of two things: 1.) take pthreads and re-build with access to a semaphore that it probably has (again I assume) or call method to trigger same functionality to suspend thread via pThreads. or 2.) take pthreads and port line by line into FPC so we make calls to either kernel or libc. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, Oct 15, 2010 at 10:24 AM, Sven Barth pascaldra...@googlemail.com wrote: The problem isn't the pthreads library. The problem is the POSIX threading specification of which pthreads is an implementation (look at the comment in CSuspendThread again). POSIX is just like you say a specification. Suspend and Resume is a feature. There are many ways to implement a feature such as suspend and resume. A semaphore is one way. Unfortunately, because FPC uses PThreads and PThreads uses libc FPC apps get slow during a high scale thread creation process where a barrier is required. I was able to work around not having an RTLEvent per thread to bybass in this case, but other cases will come where I need to have events or signals going to each thread. And if a semaphore is created for each thread object over each layer to implement you can see these inefficiencies compound. If we go native, at lease we can share the 1 semaphore that is needed to introduce suspend and resume features. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, 15 Oct 2010 16:45:04 +0200, Andrew Brunner
andrew.t.brun...@gmail.com wrote:
It was also suggested that PThread's Suspend and Resume may not do
anything. What I would do (if that is the case), is try to find where
PThread has access to the Semaphore or CriticalSection it has on that
thread...
Huh? Why should the pthreads library have a semaphore or similar on each
thread?
IMO. You must assume that all variables of the TThread descendant have
already been assigned and are valid.
Yes, but you can't. Even if you assign all values before calling the
inherited Create, there is a hidden function called AfterConstruction
which gets executed out of the threads context, so for instance the
thread id may not be set yet (actually, the thread id is returned
earliest when Execute gets called, so there's no way to retrieve it
*before*). If you access it inside the execute method, you more or
less crash (or at least leak memory).
It took me a couple of days to find the reason for our application
crashing and I believe that was fixed quite a while ago. (Somewhere
around 2.0.4, IIRC). Jonas may remember.
Excerpt from the workaround I had written then:
type
//-- @abstract(Thread class circumventing a race condition in thread
//-- creation present in the fpc 2.0.x tree.)
//-- The workaround works by starting the thread routine in suspended
//-- mode and waking it up after the constructor is known to have
//-- finished initializing all needed instance internals.
tThread = class (Classes.tThread)
public
{/= Create =\}
{}
{\==/}
//-- @abstract(The usual thread constructor.)
//-- Overwritten to ensure it starts in suspended state, woken up
//-- by @link(AfterConstruction) a little moment later.
constructor Create;
{$IFDEF UNIX}
{/= Destroy \}
{}
{\==/}
//-- @abstract(Destroys the thread and does a best effort to
//-- avoid memory leaks by deciding if it should call
//-- pthread_detach or pthread_join (WaitFor).)
destructor Destroy; override;
{$ENDIF UNIX}
{/= AfterConstruction ==\}
{}
{\==/}
//-- @abstract(Resumes the suspended thread.)
//-- This ensures that the constructor completed before @code(
//-- Execute) is actually running.
procedure AfterConstruction; override;
end {tThread};
A similar workaround got into FPC a while later and it *was* for a
good reason: Safety.
There is no need to block the thread from execution.
Actually there currently is. Otherwise you ask for trouble.
That is unless they are asking to have it
suspended on creation. And if that is the case... We need to tie into
PThread's Semaphore or Event or CriticalSection and exploit that.
What do you want to tie into there?
Vinzent.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, 15 Oct 2010 17:24:03 +0200, Sven Barth pascaldra...@googlemail.com wrote: The problem isn't the pthreads library. The problem is the POSIX threading specification of which pthreads is an implementation (look at the comment in CSuspendThread again). Actually, it's not even a problem. Externally suspending a thread is inherently unsafe and should not be done at all. If you'd do *that* would be the problem (yes, I know that there are possible use cases, but these are negligible). Vinzent. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, 15 Oct 2010 17:39:59 +0200, Andrew Brunner andrew.t.brun...@gmail.com wrote: BTW: I did notice that the semaphore was being used to suspend the thread. And I also know that that was it's only function... Suspend and Resume. The problem is not pthreads or anything, the problem is how the semantics of the Classes.TThread is defined. You can still use BeginThread directly if you want to avoid that. Vinzent. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, Oct 15, 2010 at 10:57 AM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: If you access it inside the execute method, you more or less crash (or at least leak memory). You obviously had a problem with access ThreadID before it was assigned. Accessing it should not arrive at a RAV. I'm not sure this is on topic... It took me a couple of days to find the reason for our application crashing and I believe that was fixed quite a while ago. (Somewhere around 2.0.4, IIRC). Jonas may remember. Excerpt from the workaround I had written then: //-- Overwritten to ensure it starts in suspended state, woken up //-- by @link(AfterConstruction) a little moment later. constructor Create; Exactly. It was poor implementation. You should have had a global barrier onExecute. That would unlock the thread after everything you needed was readable. See my project attached a few emails back. It demonstrates how things should be done with threads. Now - fpc carries one private barrier per thread instance. Developers of FPC should not always blame the development platform. They are sometimes going to need to redirect an issue back onto the developer. But I do recognize this is a delicate issue. But slowing us all down for issues such as these... Perhaps a wiki article on how to get threads to wait during execute would have an easier fix. And I'm sure a 100 people too would have problems. But I make a case, that slowing down the entire platform for those who need help getting threads to synchronize makes for a poor movement collectively. Perhaps a plethora of examples could help. A similar workaround got into FPC a while later and it *was* for a good reason: Safety. Anyone can make safety an issue. IMO, cThreads has a lot of RFI. (Room for improvement). There is no need to block the thread from execution. Actually there currently is. Otherwise you ask for trouble. I disagree. Completely. But that's just because I'm not tied to any one way of thinking. That is unless they are asking to have it suspended on creation. And if that is the case... We need to tie into PThread's Semaphore or Event or CriticalSection and exploit that. What do you want to tie into there? I reviewed the pthreads code. BTW: pthreads is what FPC uses to implement threading. The barrier may be accessible therby eliminating the need for an additional instance being created inside TThread. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, Oct 15, 2010 at 10:57 AM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: Huh? Why should the pthreads library have a semaphore or similar on each thread? Good question. And according to pThreads source comments they themselves did unwillingly. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, Oct 15, 2010 at 11:03 AM, jellyfish.softw...@gmx.net wrote: On Fri, 15 Oct 2010 17:24:03 +0200, Sven Barth pascaldra...@googlemail.com wrote: The problem isn't the pthreads library. The problem is the POSIX threading specification of which pthreads is an implementation (look at the comment in CSuspendThread again). Actually, it's not even a problem. Externally suspending a thread is inherently unsafe and should not be done at all. If you'd do *that* would be the problem (yes, I know that there are possible use cases, but these are negligible). Externally suspending a thread *IS* absolutely unsafe. But the notion of suspending an ENGINE at runtime externally is totally expected. The thread itself MUST put itself in a suspended state. +1 ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, 15 Oct 2010 18:19:14 +0200, Andrew Brunner andrew.t.brun...@gmail.com wrote: On Fri, Oct 15, 2010 at 10:57 AM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: If you access it inside the execute method, you more or less crash (or at least leak memory). You obviously had a problem with access ThreadID before it was assigned. Accessing it should not arrive at a RAV. No, but trying to free the thread with an invalid handle via the pthread_destroy() function does. I'm not sure this is on topic... It was the reason for the changes introduced into FPC. I don't remember the exact bug report number. Excerpt from the workaround I had written then: //-- Overwritten to ensure it starts in suspended state, woken up //-- by @link(AfterConstruction) a little moment later. constructor Create; Exactly. It was poor implementation. You should have had a global barrier onExecute. That would unlock the thread after everything you needed was readable. If it's global, it would unlock *any* thread at that time. See my project attached a few emails back. It demonstrates how things should be done with threads. Now - fpc carries one private barrier per thread instance. Sure, it's not the optimum design. But using one global barrier each thread could block each other on creation, potentially introducing excessive context switches. Which is not optimal, neither. Maybe it's less expensive, yet this blocking is actually required. Developers of FPC should not always blame the development platform. I don't think anyone did. IMO, the problem is neither FPC nor pthreads, the problem is the required semantics of TThread. They are sometimes going to need to redirect an issue back onto the developer. But I do recognize this is a delicate issue. But slowing us all down for issues such as these... The faster alternative is bound to fail with FreeOnTerminate := True. Perhaps - instead of looking into the pthreads library - you should better take a look at the TThreads implementation and try to understand *why* all the code there actually *is* there. Perhaps a wiki article on how to get threads to wait during execute would have an easier fix. Yeah, right. Creating the semaphore to wait in the execute method is bound to be faster than creating it before thread creation. Sure. But I make a case, that slowing down the entire platform for those who need help getting threads to synchronize makes for a poor movement collectively. Perhaps a plethora of examples could help. The reason wasn't people need help getting threads to synchronize, the reason was - TThread does not work as advertised under all circumstances. And some of those circumstances were not at all controllable by the user. I presume, most people rather have a *working* thread implementation than a fast, but - for no apparent reason - occasionally crashing one due to race conditions. A similar workaround got into FPC a while later and it *was* for a good reason: Safety. Anyone can make safety an issue. Yes. But if you prefer crashing the application in favor of speed just use C. :- IMO, cThreads has a lot of RFI. (Room for improvement). No. TThread has. Last time I looked into CThreads it was a more or less thin binding to the pthreads API. There is no need to block the thread from execution. Actually there currently is. Otherwise you ask for trouble. I disagree. Completely. But that's just because I'm not tied to any one way of thinking. Try calling pthread_destroy() with an uninitialized handle. Maybe that changes your way of thinking. I reviewed the pthreads code. Oh, great. That probably was never done before. Sorry for my sarcasm. BTW: pthreads is what FPC uses to implement threading. On Unix like targets. The barrier may be accessible therby eliminating the need for an additional instance being created inside TThread. *What* barrier? Vinzent. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, Oct 15, 2010 at 12:01 PM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: Exactly. It was poor implementation. You should have had a global barrier onExecute. That would unlock the thread after everything you needed was readable. If it's global, it would unlock *any* thread at that time. Yes, belonging to a system with of collection of said threads where managing system oversees any worker threads. Sure, it's not the optimum design. But using one global barrier each thread could block each other on creation, potentially introducing excessive context switches. Which is not optimal, neither. Ok. Say a project manager is looking to give FPC a try for a highly scale system that requires managing 1 million concurrent operations across say 30,000 threads. Say it's a 10 core system with 32Gigs ram. The first thing they are going to do is compare a general case with TThreads. As is - it's slow as crap. They aren't going to want to re-implement the threading system. They aren't going to take FPC seriously as a viable compiler for parallel computing. Pascal offers people the ability to quickly read what they are seeing. IMO, FPC would be a great solution. But with so many competing platforms and with M$ literally pumping millions into their parallel computing platform... Hey... It's probably worth just going with the compiler du jour ;-) Maybe it's less expensive, yet this blocking is actually required. It is not required if threading is implemented that ALREADY has a barrier in this case see pthreads source. It does. IMO, the problem is neither FPC nor pthreads, the problem is the required semantics of TThread. The problem is not with TThread. The problem is with Suspend and Resume features. The faster alternative is bound to fail with FreeOnTerminate := True. How did you draw this conclusion? Perhaps - instead of looking into the pthreads library - you should better take a look at the TThreads implementation and try to understand *why* all the code there actually *is* there. I'm going to look at severe linear inefficiencies. That is my nature. :-) Yeah, right. Creating the semaphore to wait in the execute method is bound to be faster than creating it before thread creation. Sure. LOL. Again, how in the world did you make the assumption, especially given all the code I put out there regarding a manager overseeing all the threads, that I would create a semaphore inside a worker thread's execute body? Did you read the uThreads.pas unit for this discussion? Yes. But if you prefer crashing the application in favor of speed just use C. :- I would most likely prefer objective-c. IMO, cThreads has a lot of RFI. (Room for improvement). No. TThread has. Last time I looked into CThreads it was a more or less thin binding to the pthreads API. TThread has RFI. But are you then suggesting you want to improve w/o using libc !?! IMO, using whatever we can would be great. There is no need to block the thread from execution. Actually there currently is. Otherwise you ask for trouble. We're going to disagree here. It's just that simple. If you found yourself in the ranks with developers saying What would anyone need 30,000 threads camp, please ignore this entire discussion. I beg this with authority :-) Try calling pthread_destroy() with an uninitialized handle. Maybe that changes your way of thinking. No, I pretty much question EVERYTHING. Oh, great. That probably was never done before. Sorry for my sarcasm. My question is... Sorry to whom? *What* barrier? Seriously? There are four possibly five (depending on prospective) barriers mentioned in the course of this entire discussion. List them and I will be able to oblige. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, 15 Oct 2010 20:15:51 +0200, Andrew Brunner andrew.t.brun...@gmail.com wrote: On Fri, Oct 15, 2010 at 12:01 PM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: Exactly. It was poor implementation. You should have had a global barrier onExecute. That would unlock the thread after everything you needed was readable. If it's global, it would unlock *any* thread at that time. Yes, belonging to a system with of collection of said threads where managing system oversees any worker threads. Adding more administration won't solve any issues. Sure, it's not the optimum design. But using one global barrier each thread could block each other on creation, potentially introducing excessive context switches. Which is not optimal, neither. Ok. Say a project manager is looking to give FPC a try for a highly scale system that requires managing 1 million concurrent operations across say 30,000 threads. Say it's a 10 core system with 32Gigs ram. The first thing they are going to do is compare a general case with TThreads. As is - it's slow as crap. Yes. They aren't going to want to re-implement the threading system. They aren't going to take FPC seriously as a viable compiler for parallel computing. They don't need to. Just throw away the abstraction and call BeginThread. According to you this will be magnitudes faster, because it's more or less a direct call to the underlying pthreads subsystem. Maybe it's less expensive, yet this blocking is actually required. It is not required if threading is implemented that ALREADY has a barrier in this case see pthreads source. It does. You CAN NOT use this barrier whatever they may do in the pthreads implementation you're referring to. It may even be solved different on another platform. Simple rule: If there's no API for it, you can't do it. IMO, the problem is neither FPC nor pthreads, the problem is the required semantics of TThread. The problem is not with TThread. The problem is with Suspend and Resume features. Which are features of TThread. The faster alternative is bound to fail with FreeOnTerminate := True. How did you draw this conclusion? Simple. I wrote the bug report. Perhaps - instead of looking into the pthreads library - you should better take a look at the TThreads implementation and try to understand *why* all the code there actually *is* there. I'm going to look at severe linear inefficiencies. That is my nature. :-) But by looking at *some* specific implemention of pthreads you're starting at the wrong end, I'd say. Yeah, right. Creating the semaphore to wait in the execute method is bound to be faster than creating it before thread creation. Sure. LOL. Again, how in the world did you make the assumption, especially given all the code I put out there regarding a manager overseeing all the threads, that I would create a semaphore inside a worker thread's execute body? You wanted to suspend the thread? Did you read the uThreads.pas unit for this discussion? Where can I find it? I only read your changes to the RTL and you seem to assume that neither TThread.Create nor TThread.Destroy can run concurrently with ThreadFunc (i.e. TThread.Execute). You do not seem to understand that the variables accessed here *may* change during execution of any of those functions. IOW: Your optimization is wrong in the sense that it will not work reliably. Yes. But if you prefer crashing the application in favor of speed just use C. :- I would most likely prefer objective-c. Does that lead to crashes that often? ;) IMO, cThreads has a lot of RFI. (Room for improvement). No. TThread has. Last time I looked into CThreads it was a more or less thin binding to the pthreads API. TThread has RFI. But are you then suggesting you want to improve w/o using libc !?! I suggest to remove the bottlenecks causing the slowdown at the point where they are created. Alas, I was long enough involved with that stuff that I came to the conclusion that the original Delphi-API is seriously flawed and can not be implemented both efficiently *and* correctly. So, I'd rather see a FPThread class than hacking some pthreads libraries. IMO, using whatever we can would be great. There is no need to block the thread from execution. Actually there currently is. Otherwise you ask for trouble. We're going to disagree here. It's just that simple. You agreement is not necessary to see that synchronisation is required in some way. But understanding all possible impacts of call sequences /is/. The problem is that the code does not do what it is supposed to do if the TThread.Destroy is called while TThread.Create is still running. And that *can* and *will* happen. It happened to me, so it will to others. One reason for that is that the Destroy method is designed to be called from either the executing thread (FreeOnTerminate = True) or some other external thread explicitely destroying it (FreeOnTerminate = False). To
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, 15 Oct 2010 18:26:59 +0200, Andrew Brunner andrew.t.brun...@gmail.com wrote: Externally suspending a thread *IS* absolutely unsafe. At least we're on the same page here. :) But the notion of suspending an ENGINE at runtime externally is totally expected. The thread itself MUST put itself in a suspended state. Yes. This could be done. Yet, there's no way of knowing if the thread is actually suspended without some sort of answer from the to be suspended thread, means you'd need synchronisation again. A possible solution would be to lower the to be suspended thread's priority below that of the Idle-Task. Yet, there are very few systems that provide such kind of asynchronous thread control and AFAIK none of them is an FPC target. (Sorry, Andrew. Freaking different mail program. I'm sooo used to press R for Reply To Mailinglist...) Vinzent. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 15.10.2010 21:26, Vinzent Höfler wrote: IMO, cThreads has a lot of RFI. (Room for improvement). No. TThread has. Last time I looked into CThreads it was a more or less thin binding to the pthreads API. TThread has RFI. But are you then suggesting you want to improve w/o using libc !?! I suggest to remove the bottlenecks causing the slowdown at the point where they are created. Alas, I was long enough involved with that stuff that I came to the conclusion that the original Delphi-API is seriously flawed and can not be implemented both efficiently *and* correctly. So, I'd rather see a FPThread class than hacking some pthreads libraries. I'm curious. How should this FPThread class look like? You saw flaws in Delphi's API so what would you do better? Note: I'm not implying that Delphi's API is good... as a Delphi/FPC user I don't know of another Object Oriented Pascal alternative and thus would like to know what you'd improve. Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, 15 Oct 2010 22:13:36 +0200, Sven Barth pascaldra...@googlemail.com wrote: I'm curious. How should this FPThread class look like? You saw flaws in Delphi's API so what would you do better? Hmm. Actually, I did not think too hard about it yet. So for a quick brainstorming approach: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) (IIRC FreeOnTerminate was even set to False by the default constructor, so you had the choice of either using the default or introducing a race condition by setting it after the inherited Create - which starts the execute.) That should solve most problems. It does not solve the problem that object validity errors occur when the execute method is compiled with range checks on and the VMT (I believe it is) is checked, because this is not done before the constructor has been finished, but at that point the Execute method may already be running. So I suppose, a Start method must be introduced which must be called after the creation of the thread (which would be real nasty), and that still does not solve the problem with the missing ThreadId which means, the Execute method must be wrapped to call CurrentThread before the actual overriden method is called... Note: I'm not implying that Delphi's API is good... as a Delphi/FPC user I don't know of another Object Oriented Pascal alternative and thus would like to know what you'd improve. The Delphi-API simply is too Windows-centric, that's all. I wouldn't actually blame it on Borland either, but simulating Windows semantics on systems other than Windows is not likely to lead to very efficient designs. Vinzent. -- Using Opera's revolutionary email client: http://www.opera.com/mail/ ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 15 Oct 2010, at 22:31, Vinzent Höfler wrote: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) FWIW, they're deprecated since Delphi 2009, so they will disappear over time. To resume a tthread created with fcreatesuspended=true, they introduced a new Start method (which does basically the same as Resume, but it's not deprecated). - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) (IIRC FreeOnTerminate was even set to False by the default constructor, so you had the choice of either using the default or introducing a race condition by setting it after the inherited Create - which starts the execute.) This problem has been solved with http://bugs.freepascal.org/view.php?id=16884 Jonas___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Fri, 15 Oct 2010 22:50:29 +0200, Jonas Maebe jonas.ma...@elis.ugent.be wrote: On 15 Oct 2010, at 22:31, Vinzent Höfler wrote: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) FWIW, they're deprecated since Delphi 2009, so they will disappear over time. To resume a tthread created with fcreatesuspended=true, they introduced a new Start method (which does basically the same as Resume, but it's not deprecated). So they already stole my idea! ;) Seriously that's good to hear, because it solves some of the possible compatibility issues. If users complain they can be told: Delphi is doing it, too, then. ;) - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) (IIRC FreeOnTerminate was even set to False by the default constructor, so you had the choice of either using the default or introducing a race condition by setting it after the inherited Create - which starts the execute.) This problem has been solved with http://bugs.freepascal.org/view.php?id=16884 Yes. But isn't that precisely the (forced) thread suspension on creation, Andrew is complaining about? Vinzent. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
In our previous episode, Vinzent H?fler said: Hmm. Actually, I did not think too hard about it yet. So for a quick brainstorming approach: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) Your wish is my command :-) As of now they are deprecated in 2.5.1 The methods suspend and resume are deprecated, and a new method start has been added to be able to start a suspended-on-creation thread. Trunk contains .start since april, so FPC 2.4.2 will contain this method start (but doesn't add the deprecated modifiers to resume and suspend, something that I did just now, and update http://wiki.freepascal.org/User_Changes_Trunk ) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Sat, 16 Oct 2010 00:17:33 +0200, Marco van de Voort mar...@stack.nl wrote: In our previous episode, Vinzent H?fler said: Hmm. Actually, I did not think too hard about it yet. So for a quick brainstorming approach: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) Your wish is my command :-) That was quick. I still have two wishes left, don't I? ;) Vinzent. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
In our previous episode, Vinzent H?fler said: So for a quick brainstorming approach: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) Your wish is my command :-) That was quick. I still have two wishes left, don't I? ;) If my administration is correct, you spent those in 2007 ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 04:15, schrieb Andrew Brunner: On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt mich...@freepascal.org wrote: Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? I seriously doubt it. What you could do to test, is write your program using direct Pthread calls. This way we can ascertain whether it is the FPC or Pthread code which is the bottleneck. Right. I'm going to do more reading on the POSIX threading system. I did get to trace into the threading unit under linux and the first thing I noticed was that a mutex was used to suspend and create the thread instead of using the ThreadManager.Suspend and Resume features. My local copy has removed the semaphore and I instantly noticed a speed increase in thread creation due to the lack of the extra semaphore per thread. Based on what I see as far as performance goes, the current version of threading under Unix takes 2 semaphores per thread. One by use in the threads.inc file and at least one by the pthreads.so ! It could be that RTL initialization of the thread slows down as well (just a possibility). You might want to disable the calls to InitThread and DoneThread in ThreadMain inside in rtl/unix/cthreads.pp. But be careful: now you must not use Pascal's I/O, Heap and Exceptions inside your thread function, cause they aren't initialized now (use direct syscalls like fpwrite and allocate the memory manually). If need be, I can dig up my texts for the Kylix book, it describes how to do this in Object Pascal. I suspect the code would still be pretty much the same. I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. Yes, write an object pascal version of it, which accesses the kernel directly and bypasses the C library. That's exactly what I'm thinking. There are only like 36 methods to implement. Depending on how hard will be to hook into the kernel... The only thing is we're going not going to have diversity as found in the pthreads.so. I'd bet they have tons of code for Darwin, Debian, Redhat, etc. I guess it's unknown at this point but well worth the time to explore. The problem with an own version of pthreads is that those threads will be limited to Pascal code only, cause other (C based) libraries will still use pthreads. An interesting lecture on this topic is in the wiki of Wine, cause they had to implement their own threading implementation as well (the old pthreads library wasn't capable enough for the needs of the WinAPI). eYou can find the article here: http://www.winehq.org/docs/winedev-guide/threading (the interesting paragraph is POSIX threading vs. kernel threading). Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Wed, 13 Oct 2010, Andrew Brunner wrote: On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt mich...@freepascal.org wrote: Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? I seriously doubt it. What you could do to test, is write your program using direct Pthread calls. This way we can ascertain whether it is the FPC or Pthread code which is the bottleneck. Right. I'm going to do more reading on the POSIX threading system. I did get to trace into the threading unit under linux and the first thing I noticed was that a mutex was used to suspend and create the thread instead of using the ThreadManager.Suspend and Resume features. My local copy has removed the semaphore and I instantly noticed a speed increase in thread creation due to the lack of the extra semaphore per thread. Based on what I see as far as performance goes, the current version of threading under Unix takes 2 semaphores per thread. One by use in the threads.inc file and at least one by the pthreads.so ! If need be, I can dig up my texts for the Kylix book, it describes how to do this in Object Pascal. I suspect the code would still be pretty much the same. I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. Yes, write an object pascal version of it, which accesses the kernel directly and bypasses the C library. That's exactly what I'm thinking. There are only like 36 methods to implement. Depending on how hard will be to hook into the kernel... The kernel thing is easy, just a vfork() call, if I'm correct. It's all the rest that is difficult: synchronization, mutex, semaphores :) The only thing is we're going not going to have diversity as found in the pthreads.so. I'd bet they have tons of code for Darwin, Debian, Redhat, etc. I guess it's unknown at this point but well worth the time to explore. I don't think the various Linux distros work different in this respect. They all use pthreads now, and they all use the same kernel interface. It has been on many people's wish list for ages. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Remains the question why anyone would want to create 1500 threads ? Any number of threads above the number of CPUs is no longer efficient anyway (give or take some corner cases with lots of I/O). Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 09:28, schrieb Michael Van Canneyt: Yes, write an object pascal version of it, which accesses the kernel directly and bypasses the C library. That's exactly what I'm thinking. There are only like 36 methods to implement. Depending on how hard will be to hook into the kernel... The kernel thing is easy, just a vfork() call, if I'm correct. It's all the rest that is difficult: synchronization, mutex, semaphores :) I don't know about other Unix variants, but for Linux one should use clone to create a thread. See also this example by - I believe - Linus himself: http://www.ibiblio.org/pub/Linux/docs/faqs/Threads-FAQ/html/clone.c Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Thu, 14 Oct 2010, Sven Barth wrote: The problem with an own version of pthreads is that those threads will be limited to Pascal code only, cause other (C based) libraries will still use pthreads. This is not a problem for pascal-only libs. An interesting lecture on this topic is in the wiki of Wine, cause they had to implement their own threading implementation as well (the old pthreads library wasn't capable enough for the needs of the WinAPI). eYou can find the article here: http://www.winehq.org/docs/winedev-guide/threading (the interesting paragraph is POSIX threading vs. kernel threading). It makes you wonder how Andrew's programs would behave under wine :-) Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 09:35, schrieb Michael Van Canneyt: On Thu, 14 Oct 2010, Sven Barth wrote: The problem with an own version of pthreads is that those threads will be limited to Pascal code only, cause other (C based) libraries will still use pthreads. This is not a problem for pascal-only libs. As long as you don't need to use non-pascal libraries, you are correct. But according to the Wine article it is already a problem if you use libc, because it only enables its thread safety routines if pthreads is detected... :( An interesting lecture on this topic is in the wiki of Wine, cause they had to implement their own threading implementation as well (the old pthreads library wasn't capable enough for the needs of the WinAPI). eYou can find the article here: http://www.winehq.org/docs/winedev-guide/threading (the interesting paragraph is POSIX threading vs. kernel threading). It makes you wonder how Andrew's programs would behave under wine :-) Indeed ^^ Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. Henry ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. One after the other ? That is not a meaningful test in this case ? You should know how many threads existed in parallel. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 14/10/10 10:20, Michael Van Canneyt wrote: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. One after the other ? That is not a meaningful test in this case ? You should know how many threads existed in parallel. No, parallel: http://lwn.net/Articles/10740/ Henry ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 10:20, Michael Van Canneyt wrote: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. One after the other ? That is not a meaningful test in this case ? You should know how many threads existed in parallel. No, parallel: http://lwn.net/Articles/10740/ Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. Possible culprits would then be the semaphores and TLS allocation. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 11:29, schrieb Michael Van Canneyt: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 10:20, Michael Van Canneyt wrote: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. One after the other ? That is not a meaningful test in this case ? You should know how many threads existed in parallel. No, parallel: http://lwn.net/Articles/10740/ Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. When running the attached test program without a debugger, the FPC performance won't be that bad either. Volker ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Thu, Oct 14, 2010 at 4:29 AM, Michael Van Canneyt mich...@freepascal.org wrote: Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. Possible culprits would then be the semaphores and TLS allocation. I obtained the source to pThreads system late last night. It does use a barrier using which is not necessary. And FPC (trunk) has another barrier on top of that. I may re-compile pthreads library and update on my machine but I think I already have it running up to speed! Debugger slows things down too. Present version of ThreadMemTest running outside debugger: 10500 threads create in 1.5 seconds :-) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 15:32, schrieb Andrew Brunner: On Thu, Oct 14, 2010 at 4:29 AM, Michael Van Canneyt mich...@freepascal.org wrote: Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. Possible culprits would then be the semaphores and TLS allocation. I obtained the source to pThreads system late last night. It does use a barrier using which is not necessary. And FPC (trunk) has another barrier on top of that. I may re-compile pthreads library and update on my machine but I think I already have it running up to speed! Debugger slows things down too. Present version of ThreadMemTest running outside debugger: 10500 threads create in 1.5 seconds :-) What are the exact differences from this test to your last one? Would be nice to know that... :) Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
What are the exact differences from this test to your last one? Would be nice to know that... :) Every barrier causes a significant increase in time. In high performance parallel computing time is something we minimize. Big reductions in linear execution yield massive performance gains in the parallel environments. So when I removed the FPC barrier that gave me a significant increase in performance. I also used the parent's RTLEvent instead of creating one for each thread which added another boost. Then i removed another barrier for adding elements to the watch list. All together, that solved my problem. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 14 Oct 2010, at 15:32, Andrew Brunner wrote: I obtained the source to pThreads system late last night. It does use a barrier using which is not necessary. I would be extremely surprised if the pthreads library contained any unnecessary barriers. Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
[fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Tue, Oct 12, 2010 at 5:51 PM, Andrew Brunner andrew.t.brun...@gmail.com wrote: Another problem demonstrated with this application is the limiting factor of thread creation. I'd like to make a complaint using this code as well. Change the number of threads to 3000. The system gets to about 1,000 and starts to bog down. I have a significantly fast computer here and I can tell that thread creation is not supposed to be this slow. Under delphi 2006 (windows) I was able to create up to 3000 threads. I recall stories of other programs in Java running well past that 3000. Why does fpc handle threads in a way that causes creation to slow down the more you have (as they approach 1,000) things slow to a stall. Can someone explain the difference? It seems to me like there is a list of threads being tracked by FPC or the memory manager is using locks and the other threads are borrowing the locks causing a sigificant reduction in performance over competing development platforms (such as C# or Java or Delphi). I'd like to get this resolved b/c my main system will potentially have tens of thousands of threads on serious hardware. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Wed, 13 Oct 2010, Andrew Brunner wrote: On Tue, Oct 12, 2010 at 5:51 PM, Andrew Brunner andrew.t.brun...@gmail.com wrote: Another problem demonstrated with this application is the limiting factor of thread creation. I'd like to make a complaint using this code as well. Change the number of threads to 3000. The system gets to about 1,000 and starts to bog down. I have a significantly fast computer here and I can tell that thread creation is not supposed to be this slow. Under delphi 2006 (windows) I was able to create up to 3000 threads. I recall stories of other programs in Java running well past that 3000. Why does fpc handle threads in a way that causes creation to slow down the more you have (as they approach 1,000) things slow to a stall. Probably because it uses a heap manager per thread. You may try to use 'cmem', which will replace the heap manager with the C memory manager (one for the whole app, not per thread). That will allow you to test this hypothesis. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Wed, Oct 13, 2010 at 8:28 AM, Michael Van Canneyt mich...@freepascal.org wrote: Probably because it uses a heap manager per thread. You may try to use 'cmem', which will replace the heap manager with the C memory manager (one for the whole app, not per thread). That will allow you to test this hypothesis. Ok. Trying to speed up the creating of threads I have enabled cmem by putting it as the first unit in the project. cmem,cthreads, are the first two units to be exact. looking at the status of the process using the system monitor it shows me that process switches from ptrace_stop to futex_wait_queue_me (cycling). None of my CPUs look to be all consumed so this is clearly an inefficiency with the way threads are created and added internally to the fpc rtl. I tried upping the nice level to -15 and it does speed up a little, it still takes minutes to create anything over 1,500 threads. Futexs can be avoided with interlocked calls to assign pointer values. I really hate to see the application thread slowed down to a crawl just because of memory or class creation. Please (anyone is welcome) help me speed fpc thread creation up. I might be able to suggest improvements to the threading system. Anyone intimately know FPC and it's linux guts w/r/t/ TThread that is willing to take a look at this with me? ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Wed, 13 Oct 2010, Andrew Brunner wrote: On Wed, Oct 13, 2010 at 8:28 AM, Michael Van Canneyt mich...@freepascal.org wrote: Probably because it uses a heap manager per thread. You may try to use 'cmem', which will replace the heap manager with the C memory manager (one for the whole app, not per thread). That will allow you to test this hypothesis. Ok. Trying to speed up the creating of threads I have enabled cmem by putting it as the first unit in the project. cmem,cthreads, are the first two units to be exact. looking at the status of the process using the system monitor it shows me that process switches from ptrace_stop to futex_wait_queue_me (cycling). None of my CPUs look to be all consumed so this is clearly an inefficiency with the way threads are created and added internally to the fpc rtl. FPC doesn't have anything to say about CPU allocation. The threads are created by the C pthread library and Linux kernel. They do the heavy work. I tried upping the nice level to -15 and it does speed up a little, it still takes minutes to create anything over 1,500 threads. Futexs can be avoided with interlocked calls to assign pointer values. I really hate to see the application thread slowed down to a crawl just because of memory or class creation. Please (anyone is welcome) help me speed fpc thread creation up. I might be able to suggest improvements to the threading system. Anyone intimately know FPC and it's linux guts w/r/t/ TThread that is willing to take a look at this with me? Why do you think I answered you ? :-) Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Wed, Oct 13, 2010 at 2:12 PM, Michael Van Canneyt mich...@freepascal.org wrote: FPC doesn't have anything to say about CPU allocation. The threads are created by the C pthread library and Linux kernel. They do the heavy work. Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? What is needed to get directly to kernel api to /queue events/create/destroy/suspend/resume threads? Are there any alternatives to the cthreads unit? Thank for the help Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Wed, 13 Oct 2010, Andrew Brunner wrote: On Wed, Oct 13, 2010 at 2:12 PM, Michael Van Canneyt mich...@freepascal.org wrote: FPC doesn't have anything to say about CPU allocation. The threads are created by the C pthread library and Linux kernel. They do the heavy work. Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? I seriously doubt it. What you could do to test, is write your program using direct Pthread calls. This way we can ascertain whether it is the FPC or Pthread code which is the bottleneck. If need be, I can dig up my texts for the Kylix book, it describes how to do this in Object Pascal. I suspect the code would still be pretty much the same. What is needed to get directly to kernel api to /queue events/create/destroy/suspend/resume threads? Are there any alternatives to the cthreads unit? Yes, write an object pascal version of it, which accesses the kernel directly and bypasses the C library. It has been on many people's wish list for ages. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt mich...@freepascal.org wrote: Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? I seriously doubt it. What you could do to test, is write your program using direct Pthread calls. This way we can ascertain whether it is the FPC or Pthread code which is the bottleneck. Right. I'm going to do more reading on the POSIX threading system. I did get to trace into the threading unit under linux and the first thing I noticed was that a mutex was used to suspend and create the thread instead of using the ThreadManager.Suspend and Resume features. My local copy has removed the semaphore and I instantly noticed a speed increase in thread creation due to the lack of the extra semaphore per thread. Based on what I see as far as performance goes, the current version of threading under Unix takes 2 semaphores per thread. One by use in the threads.inc file and at least one by the pthreads.so ! If need be, I can dig up my texts for the Kylix book, it describes how to do this in Object Pascal. I suspect the code would still be pretty much the same. I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. Yes, write an object pascal version of it, which accesses the kernel directly and bypasses the C library. That's exactly what I'm thinking. There are only like 36 methods to implement. Depending on how hard will be to hook into the kernel... The only thing is we're going not going to have diversity as found in the pthreads.so. I'd bet they have tons of code for Darwin, Debian, Redhat, etc. I guess it's unknown at this point but well worth the time to explore. It has been on many people's wish list for ages. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
