Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Luca Olivetti
Al 15/10/10 22:31, En/na Vinzent Höfler ha escrit: - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) I don't agree, the creator of the thread should be able to see

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Michael Van Canneyt
On Sat, 16 Oct 2010, Luca Olivetti wrote: Al 15/10/10 22:31, En/na Vinzent Höfler ha escrit: - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) I don't agree, the

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Benedikt Schindler
- FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) (IIRC FreeOnTerminate was even set to False by the default constructor, so you had the choice of either

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Jonas Maebe
On 16 Oct 2010, at 13:57, Benedikt Schindler wrote: - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute method) (IIRC FreeOnTerminate was even set to False by the default

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Andrew Brunner
On Sat, Oct 16, 2010 at 7:04 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: He cannot. If freeonterminate=true, then even if you call inherited create(false) as the very last statement of your constructor, the thread may already have finished running and freed itself before AfterConstruction

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Michael Van Canneyt
On Sat, 16 Oct 2010, Andrew Brunner wrote: On Sat, Oct 16, 2010 at 7:04 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: He cannot. If freeonterminate=true, then even if you call inherited create(false) as the very last statement of your constructor, the thread may already have finished

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Sven Barth
On 16.10.2010 15:36, Michael Van Canneyt wrote: How about creating a) A native TThreadManager record in system unit for unices. I don't think this can be done for all Unix systems in a generic way, because e.g. on Linux you should use the clone syscall (which is Linux only) while on e.g.

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Marco van de Voort
In our previous episode, Michael Van Canneyt said: IMO, it would be better to leverage the present performance condition to muster support for a high performance thread component set. How about creating a) A native TThreadManager record in system unit for unices. b) Create a

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Marco van de Voort
In our previous episode, Sven Barth said: How about creating a) A native TThreadManager record in system unit for unices. I don't think this can be done for all Unix systems in a generic way, because e.g. on Linux you should use the clone syscall (which is Linux only) FreeBSD

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Sven Barth
On 16.10.2010 15:45, Marco van de Voort wrote: In our previous episode, Sven Barth said: How about creating a) A native TThreadManager record in system unit for unices. I don't think this can be done for all Unix systems in a generic way, because e.g. on Linux you should use the clone

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Marco van de Voort
In our previous episode, Sven Barth said: FreeBSD supports rfork, which has similar functionality (the subset (?) implemented by Clone in the linuxator is implemented by a rfork call Yrying to build on the clone call has been done in 1.0.x times, and I don't have fond memories about it.

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Vinzent Höfler
On Sat, 16 Oct 2010 10:45:49 +0200, Luca Olivetti l...@ventoso.org wrote: Al 15/10/10 22:31, En/na Vinzent Höfler ha escrit: - FreeOnTerminate should be gone, (meaning no way to actively call TThread.Destroy from another thread, a thread gets destroyed automatically when it leaves its execute

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Sven Barth
On 16.10.2010 16:47, Marco van de Voort wrote: In our previous episode, Sven Barth said: FreeBSD supports rfork, which has similar functionality (the subset (?) implemented by Clone in the linuxator is implemented by a rfork call Yrying to build on the clone call has been done in 1.0.x times,

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Luca Olivetti
Al 16/10/10 16:57, En/na Vinzent Höfler ha escrit: Well, the usual implementation of an externally called Destroy is - first a call to the Terminate method - then a WaitFor() Nope, I avoid the WaitFor. I usually do a while not FFinished do CheckSynchronize(100); (where FFinished is

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Vinzent Höfler
On Sat, 16 Oct 2010 17:42:03 +0200, Luca Olivetti l...@ventoso.org wrote: Al 16/10/10 16:57, En/na Vinzent Höfler ha escrit: Well, the usual implementation of an externally called Destroy is - first a call to the Terminate method - then a WaitFor() Nope, I avoid the WaitFor. Nice for

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Luca Olivetti
Al 16/10/10 17:42, En/na Luca Olivetti ha escrit: Al 16/10/10 16:57, En/na Vinzent Höfler ha escrit: Well, the usual implementation of an externally called Destroy is - first a call to the Terminate method - then a WaitFor() Nope, I avoid the WaitFor. I usually do a while not FFinished do

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Vinzent Höfler
On Sat, 16 Oct 2010 18:04:59 +0200, Luca Olivetti l...@ventoso.org wrote: I use that only for threads I start at the beginning of the app and terminate at the end, so polling isn't a big issue. Those kind of threads are not a big issue at all. As they are running during the whole

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Marco van de Voort
In our previous episode, Sven Barth said: First we would have to get rid of this solution in search of a problem attitude. What do you mean by the this solution in search of a problem attitude? I can't follow you currently... :( People seem to first want to bash pthreads, and

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Sven Barth
On 16.10.2010 18:30, Marco van de Voort wrote: In our previous episode, Sven Barth said: First we would have to get rid of this solution in search of a problem attitude. What do you mean by the this solution in search of a problem attitude? I can't follow you currently... :( People seem to

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-16 Thread Michael Van Canneyt
On Sat, 16 Oct 2010, Marco van de Voort wrote: All I have seen is some complaint about slow starting threads by sb who can't even find the proper pthread source, and now we are suddenly making room for a native threadmanager? Isn't a native threadmanager for Linux on the wishlist of FPC

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Sven Barth
Am 14.10.2010 20:09, schrieb Andrew Brunner: What are the exact differences from this test to your last one? Would be nice to know that... :) Every barrier causes a significant increase in time. In high performance parallel computing time is something we minimize. Big reductions in linear

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Michael Van Canneyt
On Thu, 14 Oct 2010, Andrew Brunner wrote: What are the exact differences from this test to your last one? Would be nice to know that... :) Every barrier causes a significant increase in time. In high performance parallel computing time is something we minimize. Big reductions in linear

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Jonas Maebe
On 15 Oct 2010, at 15:54, Andrew Brunner wrote: rtl changes alone added 20-30% speed increase over the test case. And in the process they break all tthread suspending under Unix. CurrentTM.SuspendThread() will always fail there (it calls through to CSuspentThread): function

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Michael Van Canneyt
Thank you, I'll study it to see if/how we can do something to increase speed of creating threads in FPC. But if I understand you correctly, the WRITE_MEMORY_BARRIER() call is beyond our reach, so there's little we can do about that. Michael. On Fri, 15 Oct 2010, Andrew Brunner wrote: rtl

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Andrew Brunner
On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: You replaced a bunch of semaphore create/lock/unlock/destroy operations with calls to a function that does not do anything. I did not. I REMOVED create/destroy/lock/unlock. I ADDED calls to the ThreadManager's own

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Andrew Brunner
On Fri, Oct 15, 2010 at 9:24 AM, Michael Van Canneyt mich...@freepascal.org wrote: Thank you, I'll study it to see if/how we can do something to increase speed of creating threads in FPC. But if I understand you correctly, the WRITE_MEMORY_BARRIER() call is beyond our reach, so there's

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Sven Barth
Am 15.10.2010 16:31, schrieb Andrew Brunner: On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebejonas.ma...@elis.ugent.be wrote: You replaced a bunch of semaphore create/lock/unlock/destroy operations with calls to a function that does not do anything. I did not. I REMOVED

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Jonas Maebe
On 15 Oct 2010, at 16:31, Andrew Brunner wrote: On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: You replaced a bunch of semaphore create/lock/unlock/destroy operations with calls to a function that does not do anything. I did not. I REMOVED

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Andrew Brunner
On Fri, Oct 15, 2010 at 9:55 AM, Sven Barth pascaldra...@googlemail.com wrote: Am 15.10.2010 16:31, schrieb Andrew Brunner: On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebejonas.ma...@elis.ugent.be  wrote: You replaced a bunch of semaphore create/lock/unlock/destroy operations with calls to a

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Sven Barth
Am 15.10.2010 17:11, schrieb Andrew Brunner: On Fri, Oct 15, 2010 at 9:55 AM, Sven Barthpascaldra...@googlemail.com wrote: Am 15.10.2010 16:31, schrieb Andrew Brunner: On Fri, Oct 15, 2010 at 9:24 AM, Jonas Maebejonas.ma...@elis.ugent.be wrote: You replaced a bunch of semaphore

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Andrew Brunner
On Fri, Oct 15, 2010 at 10:06 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: First of all, if you use tthread.create(true) (i.e., create a suspended thread), then the execute method will never be called: BTW: I did notice that the semaphore was being used to suspend the thread. And I also

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Andrew Brunner
On Fri, Oct 15, 2010 at 10:24 AM, Sven Barth pascaldra...@googlemail.com wrote: The problem isn't the pthreads library. The problem is the POSIX threading specification of which pthreads is an implementation (look at the comment in CSuspendThread again). POSIX is just like you say a

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Vinzent Höfler
On Fri, 15 Oct 2010 16:45:04 +0200, Andrew Brunner andrew.t.brun...@gmail.com wrote: It was also suggested that PThread's Suspend and Resume may not do anything. What I would do (if that is the case), is try to find where PThread has access to the Semaphore or CriticalSection it has on that

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread JeLlyFish . software
On Fri, 15 Oct 2010 17:24:03 +0200, Sven Barth pascaldra...@googlemail.com wrote: The problem isn't the pthreads library. The problem is the POSIX threading specification of which pthreads is an implementation (look at the comment in CSuspendThread again). Actually, it's not even a

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Vinzent Höfler
On Fri, 15 Oct 2010 17:39:59 +0200, Andrew Brunner andrew.t.brun...@gmail.com wrote: BTW: I did notice that the semaphore was being used to suspend the thread. And I also know that that was it's only function... Suspend and Resume. The problem is not pthreads or anything, the problem is how

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Andrew Brunner
On Fri, Oct 15, 2010 at 10:57 AM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: If you access it inside the execute method, you more or less crash (or at least leak memory). You obviously had a problem with access ThreadID before it was assigned. Accessing it should not arrive at a RAV.

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Andrew Brunner
On Fri, Oct 15, 2010 at 10:57 AM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: Huh? Why should the pthreads library have a semaphore or similar on each thread? Good question. And according to pThreads source comments they themselves did unwillingly.

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Andrew Brunner
On Fri, Oct 15, 2010 at 11:03 AM, jellyfish.softw...@gmx.net wrote: On Fri, 15 Oct 2010 17:24:03 +0200, Sven Barth pascaldra...@googlemail.com wrote: The problem isn't the pthreads library. The problem is the POSIX threading specification of which pthreads is an implementation (look at the

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Vinzent Höfler
On Fri, 15 Oct 2010 18:19:14 +0200, Andrew Brunner andrew.t.brun...@gmail.com wrote: On Fri, Oct 15, 2010 at 10:57 AM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: If you access it inside the execute method, you more or less crash (or at least leak memory). You obviously had a problem

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Andrew Brunner
On Fri, Oct 15, 2010 at 12:01 PM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: Exactly.  It was poor implementation.  You should have had a global barrier onExecute.  That would unlock the thread after everything you needed was readable. If it's global, it would unlock *any* thread at

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Vinzent Höfler
On Fri, 15 Oct 2010 20:15:51 +0200, Andrew Brunner andrew.t.brun...@gmail.com wrote: On Fri, Oct 15, 2010 at 12:01 PM, Vinzent Höfler jellyfish.softw...@gmx.net wrote: Exactly. It was poor implementation. You should have had a global barrier onExecute. That would unlock the thread after

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Vinzent Höfler
On Fri, 15 Oct 2010 18:26:59 +0200, Andrew Brunner andrew.t.brun...@gmail.com wrote: Externally suspending a thread *IS* absolutely unsafe. At least we're on the same page here. :) But the notion of suspending an ENGINE at runtime externally is totally expected. The thread itself MUST put

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Sven Barth
On 15.10.2010 21:26, Vinzent Höfler wrote: IMO, cThreads has a lot of RFI. (Room for improvement). No. TThread has. Last time I looked into CThreads it was a more or less thin binding to the pthreads API. TThread has RFI. But are you then suggesting you want to improve w/o using libc !?! I

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Vinzent Höfler
On Fri, 15 Oct 2010 22:13:36 +0200, Sven Barth pascaldra...@googlemail.com wrote: I'm curious. How should this FPThread class look like? You saw flaws in Delphi's API so what would you do better? Hmm. Actually, I did not think too hard about it yet. So for a quick brainstorming approach:

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Jonas Maebe
On 15 Oct 2010, at 22:31, Vinzent Höfler wrote: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) FWIW, they're deprecated since Delphi 2009, so they will disappear over time. To resume a tthread created with fcreatesuspended=true, they

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Vinzent Höfler
On Fri, 15 Oct 2010 22:50:29 +0200, Jonas Maebe jonas.ma...@elis.ugent.be wrote: On 15 Oct 2010, at 22:31, Vinzent Höfler wrote: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) FWIW, they're deprecated since Delphi 2009, so they will

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Marco van de Voort
In our previous episode, Vinzent H?fler said: Hmm. Actually, I did not think too hard about it yet. So for a quick brainstorming approach: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) Your wish is my command :-) As of now they

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Vinzent Höfler
On Sat, 16 Oct 2010 00:17:33 +0200, Marco van de Voort mar...@stack.nl wrote: In our previous episode, Vinzent H?fler said: Hmm. Actually, I did not think too hard about it yet. So for a quick brainstorming approach: - Suspend/Resume should be gone (you can't guarantee it to work, so

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-15 Thread Marco van de Voort
In our previous episode, Vinzent H?fler said: So for a quick brainstorming approach: - Suspend/Resume should be gone (you can't guarantee it to work, so there's no point in doing it at all) Your wish is my command :-) That was quick. I still have two wishes left, don't I? ;) If

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Sven Barth
Am 14.10.2010 04:15, schrieb Andrew Brunner: On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt mich...@freepascal.org wrote: Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? I seriously doubt it. What you could do

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Michael Van Canneyt
On Wed, 13 Oct 2010, Andrew Brunner wrote: On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt mich...@freepascal.org wrote: Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? I seriously doubt it. What you could do

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Sven Barth
Am 14.10.2010 09:28, schrieb Michael Van Canneyt: Yes, write an object pascal version of it, which accesses the kernel directly and bypasses the C library. That's exactly what I'm thinking. There are only like 36 methods to implement. Depending on how hard will be to hook into the kernel...

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Michael Van Canneyt
On Thu, 14 Oct 2010, Sven Barth wrote: The problem with an own version of pthreads is that those threads will be limited to Pascal code only, cause other (C based) libraries will still use pthreads. This is not a problem for pascal-only libs. An interesting lecture on this topic is in

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Sven Barth
Am 14.10.2010 09:35, schrieb Michael Van Canneyt: On Thu, 14 Oct 2010, Sven Barth wrote: The problem with an own version of pthreads is that those threads will be limited to Pascal code only, cause other (C based) libraries will still use pthreads. This is not a problem for pascal-only

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Henry Vermaak
On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Michael Van Canneyt
On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Henry Vermaak
On 14/10/10 10:20, Michael Van Canneyt wrote: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Michael Van Canneyt
On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 10:20, Michael Van Canneyt wrote: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Volker Zipfel
Am 14.10.2010 11:29, schrieb Michael Van Canneyt: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 10:20, Michael Van Canneyt wrote: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something.

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Andrew Brunner
On Thu, Oct 14, 2010 at 4:29 AM, Michael Van Canneyt mich...@freepascal.org wrote: Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. Possible culprits would then be the semaphores and TLS allocation. I obtained the

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Sven Barth
Am 14.10.2010 15:32, schrieb Andrew Brunner: On Thu, Oct 14, 2010 at 4:29 AM, Michael Van Canneyt mich...@freepascal.org wrote: Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. Possible culprits would then be the

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Andrew Brunner
What are the exact differences from this test to your last one? Would be nice to know that... :) Every barrier causes a significant increase in time. In high performance parallel computing time is something we minimize. Big reductions in linear execution yield massive performance gains in the

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-14 Thread Jonas Maebe
On 14 Oct 2010, at 15:32, Andrew Brunner wrote: I obtained the source to pThreads system late last night. It does use a barrier using which is not necessary. I would be extremely surprised if the pthreads library contained any unnecessary barriers. Jonas

[fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-13 Thread Andrew Brunner
On Tue, Oct 12, 2010 at 5:51 PM, Andrew Brunner andrew.t.brun...@gmail.com wrote: Another problem demonstrated with this application is the limiting factor of thread creation. I'd like to make a complaint using this code as well. Change the number of threads to 3000. The system gets to about

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-13 Thread Michael Van Canneyt
On Wed, 13 Oct 2010, Andrew Brunner wrote: On Tue, Oct 12, 2010 at 5:51 PM, Andrew Brunner andrew.t.brun...@gmail.com wrote: Another problem demonstrated with this application is the limiting factor of thread creation. I'd like to make a complaint using this code as well. Change the number

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-13 Thread Andrew Brunner
On Wed, Oct 13, 2010 at 8:28 AM, Michael Van Canneyt mich...@freepascal.org wrote: Probably because it uses a heap manager per thread. You may try to use 'cmem', which will replace the heap manager with the C memory manager (one for the whole app, not per thread). That will allow you to test

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-13 Thread Michael Van Canneyt
On Wed, 13 Oct 2010, Andrew Brunner wrote: On Wed, Oct 13, 2010 at 8:28 AM, Michael Van Canneyt mich...@freepascal.org wrote: Probably because it uses a heap manager per thread. You may try to use 'cmem', which will replace the heap manager with the C memory manager (one for the whole app,

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-13 Thread Andrew Brunner
On Wed, Oct 13, 2010 at 2:12 PM, Michael Van Canneyt mich...@freepascal.org wrote: FPC doesn't have anything to say about CPU allocation. The threads are created by the C pthread library and Linux kernel. They do the heavy work. Is it possible I have the pthread library in some sort of debug

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-13 Thread Michael Van Canneyt
On Wed, 13 Oct 2010, Andrew Brunner wrote: On Wed, Oct 13, 2010 at 2:12 PM, Michael Van Canneyt mich...@freepascal.org wrote: FPC doesn't have anything to say about CPU allocation. The threads are created by the C pthread library and Linux kernel. They do the heavy work. Is it possible I

Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

2010-10-13 Thread Andrew Brunner
On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt mich...@freepascal.org wrote: Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? I seriously doubt it. What you could do to test, is write your program using direct