Re: [fpc-pascal] Re: Posting to the list via GMAIL
Op 2010-10-14 00:47, Jonas Maebe het geskryf: I've just received confirmation from the GMANE admin that they've made the gateway read-write now. Awesome, thanks Jonas. Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net:8080/fpgui/ ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 04:15, schrieb Andrew Brunner: On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt mich...@freepascal.org wrote: Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? I seriously doubt it. What you could do to test, is write your program using direct Pthread calls. This way we can ascertain whether it is the FPC or Pthread code which is the bottleneck. Right. I'm going to do more reading on the POSIX threading system. I did get to trace into the threading unit under linux and the first thing I noticed was that a mutex was used to suspend and create the thread instead of using the ThreadManager.Suspend and Resume features. My local copy has removed the semaphore and I instantly noticed a speed increase in thread creation due to the lack of the extra semaphore per thread. Based on what I see as far as performance goes, the current version of threading under Unix takes 2 semaphores per thread. One by use in the threads.inc file and at least one by the pthreads.so ! It could be that RTL initialization of the thread slows down as well (just a possibility). You might want to disable the calls to InitThread and DoneThread in ThreadMain inside in rtl/unix/cthreads.pp. But be careful: now you must not use Pascal's I/O, Heap and Exceptions inside your thread function, cause they aren't initialized now (use direct syscalls like fpwrite and allocate the memory manually). If need be, I can dig up my texts for the Kylix book, it describes how to do this in Object Pascal. I suspect the code would still be pretty much the same. I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. Yes, write an object pascal version of it, which accesses the kernel directly and bypasses the C library. That's exactly what I'm thinking. There are only like 36 methods to implement. Depending on how hard will be to hook into the kernel... The only thing is we're going not going to have diversity as found in the pthreads.so. I'd bet they have tons of code for Darwin, Debian, Redhat, etc. I guess it's unknown at this point but well worth the time to explore. The problem with an own version of pthreads is that those threads will be limited to Pascal code only, cause other (C based) libraries will still use pthreads. An interesting lecture on this topic is in the wiki of Wine, cause they had to implement their own threading implementation as well (the old pthreads library wasn't capable enough for the needs of the WinAPI). eYou can find the article here: http://www.winehq.org/docs/winedev-guide/threading (the interesting paragraph is POSIX threading vs. kernel threading). Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Wed, 13 Oct 2010, Andrew Brunner wrote: On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt mich...@freepascal.org wrote: Is it possible I have the pthread library in some sort of debug mode that is slowing down the process of thread creation? I seriously doubt it. What you could do to test, is write your program using direct Pthread calls. This way we can ascertain whether it is the FPC or Pthread code which is the bottleneck. Right. I'm going to do more reading on the POSIX threading system. I did get to trace into the threading unit under linux and the first thing I noticed was that a mutex was used to suspend and create the thread instead of using the ThreadManager.Suspend and Resume features. My local copy has removed the semaphore and I instantly noticed a speed increase in thread creation due to the lack of the extra semaphore per thread. Based on what I see as far as performance goes, the current version of threading under Unix takes 2 semaphores per thread. One by use in the threads.inc file and at least one by the pthreads.so ! If need be, I can dig up my texts for the Kylix book, it describes how to do this in Object Pascal. I suspect the code would still be pretty much the same. I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. Yes, write an object pascal version of it, which accesses the kernel directly and bypasses the C library. That's exactly what I'm thinking. There are only like 36 methods to implement. Depending on how hard will be to hook into the kernel... The kernel thing is easy, just a vfork() call, if I'm correct. It's all the rest that is difficult: synchronization, mutex, semaphores :) The only thing is we're going not going to have diversity as found in the pthreads.so. I'd bet they have tons of code for Darwin, Debian, Redhat, etc. I guess it's unknown at this point but well worth the time to explore. I don't think the various Linux distros work different in this respect. They all use pthreads now, and they all use the same kernel interface. It has been on many people's wish list for ages. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Remains the question why anyone would want to create 1500 threads ? Any number of threads above the number of CPUs is no longer efficient anyway (give or take some corner cases with lots of I/O). Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 09:28, schrieb Michael Van Canneyt: Yes, write an object pascal version of it, which accesses the kernel directly and bypasses the C library. That's exactly what I'm thinking. There are only like 36 methods to implement. Depending on how hard will be to hook into the kernel... The kernel thing is easy, just a vfork() call, if I'm correct. It's all the rest that is difficult: synchronization, mutex, semaphores :) I don't know about other Unix variants, but for Linux one should use clone to create a thread. See also this example by - I believe - Linus himself: http://www.ibiblio.org/pub/Linux/docs/faqs/Threads-FAQ/html/clone.c Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Thu, 14 Oct 2010, Sven Barth wrote: The problem with an own version of pthreads is that those threads will be limited to Pascal code only, cause other (C based) libraries will still use pthreads. This is not a problem for pascal-only libs. An interesting lecture on this topic is in the wiki of Wine, cause they had to implement their own threading implementation as well (the old pthreads library wasn't capable enough for the needs of the WinAPI). eYou can find the article here: http://www.winehq.org/docs/winedev-guide/threading (the interesting paragraph is POSIX threading vs. kernel threading). It makes you wonder how Andrew's programs would behave under wine :-) Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 09:35, schrieb Michael Van Canneyt: On Thu, 14 Oct 2010, Sven Barth wrote: The problem with an own version of pthreads is that those threads will be limited to Pascal code only, cause other (C based) libraries will still use pthreads. This is not a problem for pascal-only libs. As long as you don't need to use non-pascal libraries, you are correct. But according to the Wine article it is already a problem if you use libc, because it only enables its thread safety routines if pthreads is detected... :( An interesting lecture on this topic is in the wiki of Wine, cause they had to implement their own threading implementation as well (the old pthreads library wasn't capable enough for the needs of the WinAPI). eYou can find the article here: http://www.winehq.org/docs/winedev-guide/threading (the interesting paragraph is POSIX threading vs. kernel threading). It makes you wonder how Andrew's programs would behave under wine :-) Indeed ^^ Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. Henry ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
[fpc-pascal] Can variables be declared within a block?
Can variables be declared within a begin end block? eg. if y = xxx then begin var xxx; var yyy; end There are there as shortnames for long expressions -- Frank Church === http://devblog.brahmancreations.com ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. One after the other ? That is not a meaningful test in this case ? You should know how many threads existed in parallel. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Can variables be declared within a block?
On Thu, 14 Oct 2010, Frank Church wrote: Can variables be declared within a begin end block? No. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 14/10/10 10:20, Michael Van Canneyt wrote: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. One after the other ? That is not a meaningful test in this case ? You should know how many threads existed in parallel. No, parallel: http://lwn.net/Articles/10740/ Henry ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 10:20, Michael Van Canneyt wrote: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. One after the other ? That is not a meaningful test in this case ? You should know how many threads existed in parallel. No, parallel: http://lwn.net/Articles/10740/ Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. Possible culprits would then be the semaphores and TLS allocation. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
[fpc-pascal] Running FPC without extdebug
Can somebody confirm that the -an option (List node info in assembler file) is unavailable unless FPC is compiled with EXTDEBUG? -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues] ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Running FPC without extdebug
On 14 Oct 2010, at 11:44, Mark Morgan Lloyd wrote: Can somebody confirm that the -an option (List node info in assembler file) is unavailable unless FPC is compiled with EXTDEBUG? Correct. Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Building FPC with debug information
On 14 Oct 2010, at 02:58, Andrew Brunner wrote: I was reading on building FPC with debug enabled so I can trace into a unit make OPT=-O- DEBUG=1 all Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 11:29, schrieb Michael Van Canneyt: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 10:20, Michael Van Canneyt wrote: On Thu, 14 Oct 2010, Henry Vermaak wrote: On 14/10/10 03:15, Andrew Brunner wrote: I would say let's try to obtain source to pthreads or something. I'd bet we can just do a straight shoot into something from there. If it's open source i'd bet we can bother them perhaps for a newer version more high-scale friendly. NPTL source is in glibc. I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2 threads per second... 25min! Pathetic... Ingo Molnar have started and stopped 100,000 threads in less than 2 seconds on a dual P4 machine in some of the early NPTL tests. One after the other ? That is not a meaningful test in this case ? You should know how many threads existed in parallel. No, parallel: http://lwn.net/Articles/10740/ Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. When running the attached test program without a debugger, the FPC performance won't be that bad either. Volker ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Running FPC without extdebug
Jonas Maebe wrote: On 14 Oct 2010, at 11:44, Mark Morgan Lloyd wrote: Can somebody confirm that the -an option (List node info in assembler file) is unavailable unless FPC is compiled with EXTDEBUG? Correct. Thanks Jonas. I was using -an to look at some of the SPARC code generation a few months ago, EXTDEBUG results in a lot of noise in the compiler output particularly if the same compiler is then used by Lazarus. I think there might be an argument for either moving -an outside EXTDEBUG or alternatively for removing the description of -an from the help output unless EXTDEBUG is defined. -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues] ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On Thu, Oct 14, 2010 at 4:29 AM, Michael Van Canneyt mich...@freepascal.org wrote: Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. Possible culprits would then be the semaphores and TLS allocation. I obtained the source to pThreads system late last night. It does use a barrier using which is not necessary. And FPC (trunk) has another barrier on top of that. I may re-compile pthreads library and update on my machine but I think I already have it running up to speed! Debugger slows things down too. Present version of ThreadMemTest running outside debugger: 10500 threads create in 1.5 seconds :-) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Building FPC with debug information
Thanks, Jonas. On Thu, Oct 14, 2010 at 6:43 AM, Jonas Maebe jonas.ma...@elis.ugent.be wrote: On 14 Oct 2010, at 02:58, Andrew Brunner wrote: I was reading on building FPC with debug enabled so I can trace into a unit make OPT=-O- DEBUG=1 all Jonas ___ fpc-pascal maillist - fpc-pas...@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
Am 14.10.2010 15:32, schrieb Andrew Brunner: On Thu, Oct 14, 2010 at 4:29 AM, Michael Van Canneyt mich...@freepascal.org wrote: Impressive. So the FPC implementation on top of this obviously does something on top of this which causes it to slow down. Possible culprits would then be the semaphores and TLS allocation. I obtained the source to pThreads system late last night. It does use a barrier using which is not necessary. And FPC (trunk) has another barrier on top of that. I may re-compile pthreads library and update on my machine but I think I already have it running up to speed! Debugger slows things down too. Present version of ThreadMemTest running outside debugger: 10500 threads create in 1.5 seconds :-) What are the exact differences from this test to your last one? Would be nice to know that... :) Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
[fpc-pascal] TR: Strange Behaviour of TBits
Hello I found a strange behaviour of TBits. Example : procedure TForm1.Button1Click(Sender: TObject); var x:TBits; begin x:=TBits.Create(); x[0] := true; showmessage(inttostr( x.size)); x[1] := false; showmessage(inttostr( x.size)); x[2] := false; showmessage(inttostr( x.size)); x[3] := true; showmessage(inttostr( x.size)); x[4] := true; showmessage(inttostr( x.size)); end; The above code returns 1, 2, 3, 3, 4 while it should return 1, 2, 3, 4, 5. I Built a TMyBits class using strictly the TBits.inc file from fpc and... the bug does not appear... ! I'm using lazarus 0.9.28.2 beta and fpc 2.2.4 with linux 64. Did I miss something ? regards Julien __ Information provenant d'ESET NOD32 Antivirus, version de la base des signatures de virus 5531 (20101014) __ Le message a été vérifié par ESET NOD32 Antivirus. http://www.eset.com ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
What are the exact differences from this test to your last one? Would be nice to know that... :) Every barrier causes a significant increase in time. In high performance parallel computing time is something we minimize. Big reductions in linear execution yield massive performance gains in the parallel environments. So when I removed the FPC barrier that gave me a significant increase in performance. I also used the parent's RTLEvent instead of creating one for each thread which added another boost. Then i removed another barrier for adding elements to the watch list. All together, that solved my problem. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64
On 14 Oct 2010, at 15:32, Andrew Brunner wrote: I obtained the source to pThreads system late last night. It does use a barrier using which is not necessary. I would be extremely surprised if the pthreads library contained any unnecessary barriers. Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
