Re: [Lazarus] Multi-threading support in IDE
On Fri, Aug 14, 2009 at 02:58:34PM +0200, Michael Van Canneyt wrote: On Thu, Aug 13, 2009 at 09:26:40PM +0200, Michael Van Canneyt wrote: works well and does not require to much code changes, it could be wrapped in IFDEF's for more advanced users that want to try it for themselves. Possible or not (in the Lazarus IDE), I'll take a look at the FP IDE to see how such an integration works. All in the name of learning multi-threading. ;-) The FP IDE does not use multi-threading. While the compiler is compiling, the IDE is stuck. Isn't there a callback to report progres (lines etc) ? There is, but the user cannot do anything except look at the progress indicator. Probably, but that is something that can be resolved, not a fundamental problem. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Jürgen Hestermann wrote: That's true. The general problem is not that the IDE is not responsive when compiling/linking but that compiling/linking takes so long. FPC is a fast compiler and more speed is obviously also nice. The compiling is not the issue to me, it's the unresponsive IDE that gets me. I would like it to be similir to other users experiences. For example, you can print a large document and continue browsing your text or changing some setting in the menus etc... It doesn't block you from continuing. The Lazarus IDE does. I compile very often - probably once every minute or two. I also practise Test-Drive Development, which makes sure you compile often. I would like to start a full build (which could take a while for large projects) and continue writing fpdoc documentation or start a new unit test (obviously without saving before the compile is complete) etc... It's called multi-tasking. Ever heard of it? ;-) Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Fri, 14 Aug 2009 07:16:36 +0200 Jürgen Hestermann juergen.hesterm...@gmx.de wrote: BTW, when the compiler has to read and write disk files, a separate thread will not speed up much. Then the compiler thread will be idle much time, waiting for disk I/O, and the main thread will be idle, waiting for the compiler. IMO. That's true. The general problem is not that the IDE is not responsive when compiling/linking but that compiling/linking takes so long. If it would be as fast as in Delphi noone would complain. It takes so long, because compiler and linker have to read all ppus on every run. A persistent compiler and linker could keep some things. And having compiler and linker in separate (asynchronous) threads would be the same as starting them as a new process in asynchronous mode. It would open a can of worms because of synchronising problems. They only share files. The synchronising is easy. The are good reasons to not put the compiler into the same process: -a process can be aborted/killed easily -various compiler versions -compiler crashes do not effect the IDE I would not mind to wait until compiling has finished if it would only be faster. Mattias -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Mattias Gaertner schrieb: On Thu, 13 Aug 2009 22:24:37 +0200 Florian Klaempfl flor...@freepascal.org wrote: Mattias Gaertner schrieb: On Thu, 13 Aug 2009 14:01:38 +0200 Florian Klaempfl flor...@freepascal.org wrote: Michael Van Canneyt schrieb: So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Efficient, yes; As I pointed out on fpc-core a few days ago, I'd like to create an fpc which stays in memory and takes commands via pipes/sockets/ipc whatever to reduce process startup times. Hooray. \O/ Can I help somehow? Well, first we have to decide which method to use. I even think a slightly modified fpc which can be controlled through stdin is the easiest. Ok. Don't forget proper encoding. ;) But an async. process doesn't work for lazarus, right? So controlling through stdin/stdout is not possible? -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Mattias Gaertner wrote: -compiler crashes do not effect the IDE I like Florian's idea too. As for your statement above, how often does this really happen? In the last 4-5 years that I have been using FPC, the compiler has not once crashed on me. Yes it might report that there is a compiler error in my project, but I do not consider that a crash. Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Fri, 14 Aug 2009, Graeme Geldenhuys wrote: Mattias Gaertner wrote: -compiler crashes do not effect the IDE I like Florian's idea too. As for your statement above, how often does this really happen? In the last 4-5 years that I have been using FPC, the compiler has not once crashed on me. Yes it might report that there is a compiler error in my project, but I do not consider that a crash. It's one thing if the compiler compiles once and exits. It's a whole other matter if the compiler compiles multiple times, and must properly reset itself prior to recompiling. Michael. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Graeme Geldenhuys wrote: Jürgen Hestermann wrote: That's true. The general problem is not that the IDE is not responsive when compiling/linking but that compiling/linking takes so long. FPC is a fast compiler and more speed is obviously also nice. The compiling is not the issue to me, it's the unresponsive IDE that gets me. I would like it to be similir to other users experiences. For example, you can print a large document and continue browsing your text or changing some setting in the menus etc... It doesn't block you from continuing. The Lazarus IDE does. I compile very often - probably once every minute or two. I also practise Test-Drive Development, which makes sure you compile often. I would like to start a full build (which could take a while for large projects) and continue writing fpdoc documentation or start a new unit test (obviously without saving before the compile is complete) etc... Try ide\outputfilter.pas around line 295 if (Applicationnil) and (abs(LastProcessMessages-Now)((1/86400)/3)) make it if (Applicationnil) and (abs(LastProcessMessages-Now)((1/86400)/10)) and maybe add an application.idle(flase); too Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Mattias Gärtner schreef: Zitat von Graeme Geldenhuys grae...@opensoft.homeip.net: Mattias Gaertner wrote: -compiler crashes do not effect the IDE I like Florian's idea too. As for your statement above, how often does this really happen? In the last 4-5 years that I have been using FPC, the compiler has not once crashed on me. Yes it might report that there is a compiler error in my project, but I do not consider that a crash. The compiler is started on every compile which is the cleanest start you can get. A threaded compiler must have a restart, which is less clean. There will be more crashes and bugs. What happens if the restart fails? For the debugger, we have reset debugger. For the compiler, we would add a reset compiler option. Vincent -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Vincent Snijders wrote: It may happen to me more than for you, because I may use the trunk compiler more often, so that these things get noticed before it gets in the release and more people are hurt. That would be it. I stopped using the trunk compiler for our day-to-day development. Plus what Michael and Mattias mentioned makes perfect sense too. Thanks for the info guys. Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Graeme Geldenhuys wrote: Martin wrote: make it if (Applicationnil) and (abs(LastProcessMessages-Now)((1/86400)/10)) This does not make much difference on my system. :-( Hm I tested on windows only. Try adding the 2 lines I inserted with comments (I have not tested this!!) Just seeing if (TheAsyncProcess=nil) and (fProcess.Outputnil) then begin // using a blocking TProcess //Count := 0; //if fProcess.Output.NumBytesAvailable 0 then Count:=fProcess.Output.Read(Buf[1],length(Buf)); if Count=0 then begin // no output on blocking means, process has ended break; end; end; One more remark, if it works, it will allow normal editing,etc. But you can not do: - search and replace - find in files ... Because those themself are long running, and while being inside the compile's processMessages, the compile will not read the output, and that may block compiling. This is because all this stuff is not event-driven, so it cannnot be mixed. It would work in a single thread and all this, but it must be fully event-driven. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Michael Van Canneyt schreef: On Fri, 14 Aug 2009, Vincent Snijders wrote: Mattias Gärtner schreef: Zitat von Graeme Geldenhuys grae...@opensoft.homeip.net: Mattias Gaertner wrote: -compiler crashes do not effect the IDE I like Florian's idea too. As for your statement above, how often does this really happen? In the last 4-5 years that I have been using FPC, the compiler has not once crashed on me. Yes it might report that there is a compiler error in my project, but I do not consider that a crash. The compiler is started on every compile which is the cleanest start you can get. A threaded compiler must have a restart, which is less clean. There will be more crashes and bugs. What happens if the restart fails? For the debugger, we have reset debugger. That's easy: it stops and restarts gdb. an external process. For the compiler, we would add a reset compiler option. If the compiler is internal, what would 'reset compiler' do ? If the compiler people can't guarantee a proper reset, then neither can the IDE... With the proposal of Florian, the 'reset compiler' is possible, just as with GDB... Sorry, I was reading it as part of an usage question about Florians proposal (external in memory compiler), not about an internal compiler. Vincent -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
sorry tested now. doesn't work, outside windows world... So even a blocking pipe should have some kind of CanRead that returns immediately? Martin wrote: Graeme Geldenhuys wrote: Martin wrote: make it if (Applicationnil) and (abs(LastProcessMessages-Now)((1/86400)/10)) This does not make much difference on my system. :-( Hm I tested on windows only. Try adding the 2 lines I inserted with comments (I have not tested this!!) Just seeing if (TheAsyncProcess=nil) and (fProcess.Outputnil) then begin // using a blocking TProcess //Count := 0; //if fProcess.Output.NumBytesAvailable 0 then Count:=fProcess.Output.Read(Buf[1],length(Buf)); if Count=0 then begin // no output on blocking means, process has ended break; end; end; One more remark, if it works, it will allow normal editing,etc. But you can not do: - search and replace - find in files ... Because those themself are long running, and while being inside the compile's processMessages, the compile will not read the output, and that may block compiling. This is because all this stuff is not event-driven, so it cannnot be mixed. It would work in a single thread and all this, but it must be fully event-driven. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin schreef: Graeme Geldenhuys wrote: Martin wrote: make it if (Applicationnil) and (abs(LastProcessMessages-Now)((1/86400)/10)) This does not make much difference on my system. :-( Hm I tested on windows only. You can not really test it on windows, because on windows TAsyncProcess is used. Vincent -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Mattias Gärtner wrote: Zitat von Vincent Snijders vsnijd...@vodafonevast.nl: I like Florian's idea too. As for your statement above, how often does this really happen? In the last 4-5 years that I have been using FPC, the compiler has not once crashed on me. Yes it might report that there is a compiler error in my project, but I do not consider that a crash. The compiler is started on every compile which is the cleanest start you can get. A threaded compiler must have a restart, which is less clean. There will be more crashes and bugs. What happens if the restart fails? For the debugger, we have reset debugger. For the compiler, we would add a reset compiler option. I meant, when the compiler is in another thread and the restart fails, what should happen? You can only restart the IDE. Just out of curiosity, what are we trying to solve? How much benefit would a build in or preloaded fpc inside Lazarus be? Most Operating System cache disk access. So once you used, it is in memory, never mind if that memory belongs to lazarus, or the OS. Of course another thread would solve issues with unresponsiveness. But that could be done by having another thread, that runs the external process. Or if the AsyncProcess worked, it could be done, by using this + moving to an event-driven model. Or am I missing some crucial point? Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin schreef: Just out of curiosity, what are we trying to solve? snip Or am I missing some crucial point? 3 idle cores on a quad core CPU. Why pay for a quad core, if Lazarus + FPC use only one? I am still waiting during a compile and those 3 other cores are idling. Let them to part of the work to get the job done faster. ;-) Vincent P.S. I couldn't resist not helping in this discussion. It is nice to have an active mailing list not about unicode. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Vincent Snijders wrote: Or am I missing some crucial point? 3 idle cores on a quad core CPU. Why pay for a quad core, if Lazarus + FPC use only one? I am still waiting during a compile and those 3 other cores are idling. Let them to part of the work to get the job done faster. ;-) +10 :-) My new computer is on order and will arrive next week. Quad Core with 4GB memory. A huge upgrade from my current P4 2.4Ghz with 1GB memory. I would like all my cpu cores to work as hard as they can! So I can code faster. ;-) P.S. I couldn't resist not helping in this discussion. It is nice to have an active mailing list not about unicode. Yes, it is a pleasure not having to do the SVN vs Git thing either. :) Did I mention Git uses multiple threads for log, diff and delta packing if a multi-core cpu is detected. [...Graeme runs and hides behind his desk...] Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Vincent Snijders wrote: Martin schreef: Just out of curiosity, what are we trying to solve? snip Or am I missing some crucial point? 3 idle cores on a quad core CPU. Why pay for a quad core, if Lazarus + FPC use only one? I am still waiting during a compile and those 3 other cores are idling. Let them to part of the work to get the job done faster. ;-) I guess lazarus uses one, fpc uses one and the os uses one, so that leaves one for me to answer mails or forummessages or browse or chat. No idle cores here ;-) Marc -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Mattias Gärtner wrote: Do you have time to debug TAsyncProcess? No harm in trying... Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Zitat von Martin laza...@mfriebe.de: sorry tested now. doesn't work, outside windows world... So even a blocking pipe should have some kind of CanRead that returns immediately? Then it would be non blocking. Mattias -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Zitat von Vincent Snijders vsnijd...@vodafonevast.nl: Mattias Gärtner schreef: Zitat von Graeme Geldenhuys grae...@opensoft.homeip.net: Mattias Gaertner wrote: -compiler crashes do not effect the IDE I like Florian's idea too. As for your statement above, how often does this really happen? In the last 4-5 years that I have been using FPC, the compiler has not once crashed on me. Yes it might report that there is a compiler error in my project, but I do not consider that a crash. The compiler is started on every compile which is the cleanest start you can get. A threaded compiler must have a restart, which is less clean. There will be more crashes and bugs. What happens if the restart fails? For the debugger, we have reset debugger. For the compiler, we would add a reset compiler option. I meant, when the compiler is in another thread and the restart fails, what should happen? You can only restart the IDE. Mattias -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Friday 14 August 2009 10:48:05 Vincent Snijders wrote: Graeme Geldenhuys schreef: Mattias Gaertner wrote: -compiler crashes do not effect the IDE I like Florian's idea too. As for your statement above, how often does this really happen? It happens from time to time. For example: http://bugs.freepascal.org/view.php?id=13323 Same for me, FPC fixes_2_2 crashes about 20 times a day with EList or EAcessViolation if I forget to use -B to compile some modified kernel units which are used in many other units. Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Friday 14 August 2009 08:55:07 Graeme Geldenhuys wrote: Jürgen Hestermann wrote: That's true. The general problem is not that the IDE is not responsive when compiling/linking but that compiling/linking takes so long. FPC is a fast compiler and more speed is obviously also nice. Last time I compared FPC and Delphi 7 Delphi was ten times faster. Yes, more speed would be nice. Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Fri, 14 Aug 2009, Martin Schreiber wrote: On Friday 14 August 2009 08:55:07 Graeme Geldenhuys wrote: Jürgen Hestermann wrote: That's true. The general problem is not that the IDE is not responsive when compiling/linking but that compiling/linking takes so long. FPC is a fast compiler and more speed is obviously also nice. Last time I compared FPC and Delphi 7 Delphi was ten times faster. Yes, more speed would be nice. Yes, but Delphi 7 has 10 times less platforms to support, and 10 times more people for it's development, people working full time on nothing else than Delphi. So where are the compiler geniuses, and the super patches to improve FPC ? Michael.-- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Florian Klaempfl wrote: Probably simple. The compiler is started by the compile function in fpc/compiler/compiler.pas and takes simply a command line arguments. Florian, you were 100% correct, it is VERY easy! I looked at the FP IDE, and within 5 minutes I had my own fpGUI based ide. :-) Actually just a GUI application with one edit field that takes the compiler parameters and then a big COMPILE button that starts the compile process in a separate thread. Currently all output goes to stdout, but the compiling works perfectly. Next step is to place that thread in the Lazarus IDE. :) Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin Schreiber wrote: checking of the compiler output and the file search run in separate threads, so MSEide is fully functional while compiling or searching. Nice! :-) From the MSEide config dialog, I gather MSEide also launches the FPC compiler in a separate process and not built into the MSEide? Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Vincent Snijders wrote: Martin schreef: Just out of curiosity, what are we trying to solve? snip Or am I missing some crucial point? 3 idle cores on a quad core CPU. Why pay for a quad core, if Lazarus + FPC use only one? I am still waiting during a compile and those 3 other cores are idling. Let them to part of the work to get the job done faster. ;-) True, but from the point of using FPC and the point of faster compilation = this needs a multithreaded fpc. Running a single threaded fpc in it's own thread (never mind if compiled into lazarus, or loaded as process from a thread) will not do much (except for one core could do the updates of the message window A multi-threade fpc can still be used in a forked process. As for thinks happening inside lazarus (codetools, search in files,) different story Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin schrieb: Vincent Snijders wrote: Martin schreef: Just out of curiosity, what are we trying to solve? snip Or am I missing some crucial point? 3 idle cores on a quad core CPU. Why pay for a quad core, if Lazarus + FPC use only one? I am still waiting during a compile and those 3 other cores are idling. Let them to part of the work to get the job done faster. ;-) True, but from the point of using FPC and the point of faster compilation = this needs a multithreaded fpc. Running a single threaded fpc in it's own thread (never mind if compiled into lazarus, or loaded as process from a thread) will not do much Well, currently this does not help much either because FPC is mainly bound to memory throughput if the disk cache is already hot. Resistent FPC keeping units loaded would be the largest gain for projects like MSE or Lazarus. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin schrieb: and maybe add an application.idle(flase); too Lazalus nevel makes things flase! SCNL DoDi -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
2009/8/14 Martin Schreiber fp...@bluewin.ch: On Friday 14 August 2009 13:22:01 Florian Klaempfl wrote: It probably requires to rewrite FPC completely :) FPC chooses always maintainability and portability over speed and compared with gcc or even VS, FPC is still fast ;) True, but the distance will be less with every FPC release if compiling speed is not observed I fear. The possibility of Delphi to compile big projects in some seconds and small projects in less than a second on simple PC's has already gone in FPC. :-( The problem is that you really compare apples to pears. And, as Michael explained, resources (of the human kind) are limited. hopefully this will change when Delphi finally crumbles. Henry -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Friday 14 August 2009 13:41:30 Florian Klaempfl wrote: Well, currently this does not help much either because FPC is mainly bound to memory throughput if the disk cache is already hot. Resistent FPC keeping units loaded would be the largest gain for projects like MSE or Lazarus. Really? And why is Delphi 7 so much faster than FPC under same circumstances? I used dcc32 and not the Delphi IDE for comparison. Again, you are right, FPC is fast but it should not become much slower or it will loose a big benefit over gcc or other c++ compilers. Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin Schreiber wrote: From the MSEide config dialog, I gather MSEide also launches the FPC compiler in a separate process and not built into the MSEide? Yes. It can also call gcc and parse the error messages. Also possible So how did you resolve the async process issue that Lazarus is experiencing under Linux? Or does running the compiler process (TProcess style) via a separate thread bypass the issue of TAsyncProcess that Lazarus IDE is using. That means MSEide doesn't need a TAsyncProcess type component? If so... Mattias, could Lazarus IDE be modified to run the compiler TProcess in a separate thread. And then another thread which filters the compiler output to the Messages window. Or is that a huge task? Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Friday 14 August 2009 14:02:34 Graeme Geldenhuys wrote: Martin Schreiber wrote: Yes. It can also call gcc and parse the error messages. Also possible So how did you resolve the async process issue that Lazarus is experiencing under Linux? Or does running the compiler process (TProcess style) via a separate thread bypass the issue of TAsyncProcess that Lazarus IDE is using. That means MSEide doesn't need a TAsyncProcess type component? I don't know TAsyncProcess. What does it compared to TProcess? What are the Problems? I use the MSEgui version of exec() and the tpipereader class to run the compiler and read the messages. Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin Schreiber wrote: So does MSEide with the compile and the search in files function. The checking of the compiler output and the file search run in separate I just tried with a oldish MSEide 2.1 (unstable) and I monitored the thread usage for the mseide process. You seem to use a few threads for things. One for the main process, one for the editor (though I don't know what it does) and 'search in files' also uses a separate thread. Very nice to see that even with a long running search, the main ide still responded to all user interactions as normal. I could even compile a project while doing a search. All tested under Linux. Don't you want to move to Lazarus IDE development. ;-) Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Mattias Gärtner wrote: Zitat von Graeme Geldenhuys grae...@opensoft.homeip.net: Martin Schreiber wrote: From the MSEide config dialog, I gather MSEide also launches the FPC compiler in a separate process and not built into the MSEide? Yes. It can also call gcc and parse the error messages. Also possible So how did you resolve the async process issue that Lazarus is experiencing under Linux? Or does running the compiler process (TProcess style) via a separate thread bypass the issue of TAsyncProcess that Lazarus IDE is using. That means MSEide doesn't need a TAsyncProcess type component? If so... Mattias, could Lazarus IDE be modified to run the compiler TProcess in a separate thread. And then another thread which filters the compiler output to the Messages window. Or is that a huge task? No. Just write a ThreadedProcess. That would work pretty much like TAsynProcess. And I guess it will suffer the same bug. I don't know what you mean by ThreadedProcess unless you meant yes, but wanted to indicate a different form of implementation? IMHO, if the process running fpc is started from a different thread, then the IDE can be kept as functional, as it can with TAsyncProcess. In fact if the TProcess is in started from a thread of it's own, it does not want to be Async, it wants to be blocking, so you don't need to put a lot of sleeps into your code. As separate thread *can* also solve one more issue. Currently even the AsyncProcess is done in a loop. It does use the OnDataRead event, but only to fill a cache, which is then used in a loop. It should not be in a loop at all. It should be purely event driven. Then it would continue to run, even if a search in files is started while compiling If you create a class TThreaded Process (just to simulate an AsyncProcess within a (hidden) Thread), then you still must transform the current loop into an event driven model. However this 2 issues can be addressed on their own. even using a blocking thread, you can go (almost) event drive. Put the read on a timer (every 10 milliseconds) or OnIdle, The read will block, but after each read, it will return to the event loop = events will be processed normally (instead of ProcessMessages), and then the next read happens on timer, or idle In fact if you make everything eventdriven, then you could introduce some method to mark events as thread-save, and the event-loop could automatically deploy threads, if a lot of events occur = that would be cool. Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Thu, Aug 13, 2009 at 11:09:02PM +0200, Hans-Peter Diettrich wrote: Graeme Geldenhuys schrieb: Does Delphi or Kylix have the compilers built-in or do they work similar to Lazarus IDE? AFAIK the Delphi IDE comes with an built-in compiler. A dll. Not statically built in. Afaik cmdline and IDE use the same DLL This may change with the announced 64-bit/cross compiler, and may be the reason for the long release delay of that new feature. BTW, when the compiler has to read and write disk files, a separate thread will not speed up much. Then the compiler thread will be idle much time, waiting for disk I/O, and the main thread will be idle, waiting for the compiler. IMO. Yes, but basic units could be cached. Also the cache of dirsearches would persist. Moreover you win process startup time (which can be quite significant on windows) -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin Schreiber schrieb: On Friday 14 August 2009 13:22:01 Florian Klaempfl wrote: It probably requires to rewrite FPC completely :) FPC chooses always maintainability and portability over speed and compared with gcc or even VS, FPC is still fast ;) True, but the distance will be less with every FPC release if compiling speed is not observed I fear. The possibility of Delphi to compile big projects in some seconds and small projects in less than a second on simple PC's has already gone in FPC. :-( On my Core Duo , FPC itself (230k lines) builds in 5.5s, so this is ok imo ;) -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Fri, Aug 14, 2009 at 09:19:11AM +0200, Mattias Gaertner wrote: And having compiler and linker in separate (asynchronous) threads would be the same as starting them as a new process in asynchronous mode. It would open a can of worms because of synchronising problems. They only share files. The synchronising is easy. The are good reasons to not put the compiler into the same process: -a process can be aborted/killed easily -various compiler versions -compiler crashes do not effect the IDE Disadvantages: - compiler versions might not match (entirely) - Multiple processes that can only be made sequential over I/O. Potential for race conditions. - (some) platforms might not support multiple processes at all, or miss the relevant 100% primitives. (also e.g. faulty pipe implementations) -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin Schreiber schrieb: On Friday 14 August 2009 13:41:30 Florian Klaempfl wrote: Well, currently this does not help much either because FPC is mainly bound to memory throughput if the disk cache is already hot. Resistent FPC keeping units loaded would be the largest gain for projects like MSE or Lazarus. Really? And why is Delphi 7 so much faster than FPC under same circumstances? FPC simply needs memory bandwidth due to design? -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Friday 14 August 2009 14:23:36 Graeme Geldenhuys wrote: Martin Schreiber wrote: So does MSEide with the compile and the search in files function. The checking of the compiler output and the file search run in separate I just tried with a oldish MSEide 2.1 (unstable) and I monitored the thread usage for the mseide process. You seem to use a few threads for things. One for the main process, one for the editor (though I don't know what it does) You probably see the thread which reads messages from gdb. Syntax highlighting is done in onidle in packages of 100 lines. Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Fri, Aug 14, 2009 at 09:37:56AM +0200, Michael Van Canneyt wrote: -various compiler versions -compiler crashes do not effect the IDE Good enough for me NOT to do it. If the compiler was internal to the IDE, I could not easily compile for 32/64 bit or a different version, without including 2+ compilers in the IDE. As it is now, all I must do is set the compiler to use. Good enough to not do it before a good packages system is developed so it can be loaded dynamically. But that is something else then to chuck it all together. (or simply not accelerate cross-compiling with the built-in till packages emerge) -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Friday 14 August 2009 14:24:53 Mattias Gärtner wrote: I have some small programs with one thousand lines in the main source and using units with about 30k loc. Recompile and linking the program takes less than a second. With small I mean 100'000 lines. ;-) Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Friday 14 August 2009 14:51:04 Florian Klaempfl wrote: Martin Schreiber schrieb: True, but the distance will be less with every FPC release if compiling speed is not observed I fear. The possibility of Delphi to compile big projects in some seconds and small projects in less than a second on simple PC's has already gone in FPC. :-( On my Core Duo , FPC itself (230k lines) builds in 5.5s, so this is ok imo ;) MSEide (330k lines) needs 25s on Athlon 64 X2. Commandline: ppc386 -B apps/ide/mseide.pas -Fulib/common/* -Fulib/common/kernel/i386-linux -Fi/lib/common/kernel 25 s is too much to wait and too little to sleep. ;-) Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
2009/8/14 Martin Schreiber fp...@bluewin.ch: MSEide (330k lines) needs 25s on Athlon 64 X2. Commandline: ppc386 -B apps/ide/mseide.pas -Fulib/common/* -Fulib/common/kernel/i386-linux -Fi/lib/common/kernel 25 s is too much to wait and too little to sleep. ;-) How long does fpc -sh take? Henry -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Friday 14 August 2009 16:14:16 Henry Vermaak wrote: 2009/8/14 Martin Schreiber fp...@bluewin.ch: MSEide (330k lines) needs 25s on Athlon 64 X2. Commandline: ppc386 -B apps/ide/mseide.pas -Fulib/common/* -Fulib/common/kernel/i386-linux -Fi/lib/common/kernel 25 s is too much to wait and too little to sleep. ;-) How long does fpc -sh take? 21.8 s. Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin Schreiber wrote: 25 s is too much to wait and too little to sleep. ;-) I so agree with that! :-) Same command line params as yours - on a Intel P4 2.4Ghz. 322187 lines compiled, 33.6 sec And with the -sh parameter. 322187 lines compiled, 28.7 sec Martin, how does your results compare when you use Delphi's bcc32 compiler? Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin schreef: Mattias Gärtner wrote: No. Just write a ThreadedProcess. That would work pretty much like TAsynProcess. And I guess it will suffer the same bug. I don't know what you mean by ThreadedProcess unless you meant yes, but wanted to indicate a different form of implementation? IMHO, if the process running fpc is started from a different thread, then the IDE can be kept as functional, as it can with TAsyncProcess. In fact if the TProcess is in started from a thread of it's own, it does not want to be Async, it wants to be blocking, so you don't need to put a lot of sleeps into your code. As separate thread *can* also solve one more issue. Currently even the AsyncProcess is done in a loop. It does use the OnDataRead event, but only to fill a cache, which is then used in a loop. It should not be in a loop at all. It should be purely event driven. Then it would continue to run, even if a search in files is started while compiling I have looked a bit more why outputfilter is so slow at parsing a lot of compiler output (e.g. compiling a simple LCL app with -va). The pipe buffer is rather small, NumBytesAvailable is not bigger than 1280 bytes. So OnAsyncReadData reads only 1280 bytes at a time. OnAsyncReadData is only called after an application.handlemessage. So on line 319 of outputfilter.pas, the buffer is most times empty, but NumBytesAvailable = 1280, and we are sleeping until Application.HandleMessage is called. I had some success with the following patch. It does a sleep(0) to give the compiler the chance to write to the pipe again and tries to read it again. --- ide/outputfilter.pas +++ ide/outputfilter.pas @@ -1206,9 +1206,12 @@ var Count: LongWord; begin if fProcess=nil then exit; - Count:=TAsyncProcess(fProcess).NumBytesAvailable; - if Count0 then -FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count); + repeat +Count:=TAsyncProcess(fProcess).NumBytesAvailable; +if Count0 then + FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count); +sleep(0); + until Count=0; end; function TOutputFilter.CreateScanners(ScannerOptions: TStrings): boolean; If you create a class TThreaded Process (just to simulate an AsyncProcess within a (hidden) Thread), then you still must transform the current loop into an event driven model. However this 2 issues can be addressed on their own. even using a blocking thread, you can go (almost) event drive. Put the read on a timer (every 10 milliseconds) or OnIdle, The read will block, but after each read, it will return to the event loop = events will be processed normally (instead of ProcessMessages), and then the next read happens on timer, or idle In fact if you make everything eventdriven, then you could introduce some method to mark events as thread-save, and the event-loop could automatically deploy threads, if a lot of events occur = that would be cool. It is not clear to me why we can't check NumBytesAvailable (available since fpc 2.2.2 or earlier) on unix and windows and just forget about TAsyncProcess for compiling? Vincent -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Vincent Snijders wrote: I have looked a bit more why outputfilter is so slow at parsing a lot of compiler output (e.g. compiling a simple LCL app with -va). The pipe buffer is rather small, NumBytesAvailable is not bigger than 1280 bytes. So OnAsyncReadData reads only 1280 bytes at a time. OnAsyncReadData is only called after an application.handlemessage. So on line 319 of outputfilter.pas, the buffer is most times empty, but NumBytesAvailable = 1280, and we are sleeping until Application.HandleMessage is called. I had some success with the following patch. It does a sleep(0) to give the compiler the chance to write to the pipe again and tries to read it again. --- ide/outputfilter.pas +++ ide/outputfilter.pas @@ -1206,9 +1206,12 @@ var Count: LongWord; begin if fProcess=nil then exit; - Count:=TAsyncProcess(fProcess).NumBytesAvailable; - if Count0 then -FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count); + repeat +Count:=TAsyncProcess(fProcess).NumBytesAvailable; +if Count0 then + FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count); +sleep(0); + until Count=0; end; function TOutputFilter.CreateScanners(ScannerOptions: TStrings): boolean; If I understand it correctly the problem is that TAsyncProcess, currently only reads data, when the event is triggered, which is in a ProcessMessages. Ouch. Either TAsyncProcess.OnReadData is transferred into the main loop, in a similar way as for the normal TProcess. I have not tested this, but it should work, as NumBytesAvailable seems to be none blocking. = This could simplify the overall loop as the handling of the 2 projects would be more or less the same. Alternatively, before we sleep(3), we could check for data, and process it. end else begin // no new input, but process still running if TheAsyncProcess.NumBytesAvailable 0 then Application.ProcessMessages else Sleep(30); end; -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Fri, 14 Aug 2009, Graeme Geldenhuys wrote: Martin Schreiber wrote: 25 s is too much to wait and too little to sleep. ;-) I so agree with that! :-) Same command line params as yours - on a Intel P4 2.4Ghz. 322187 lines compiled, 33.6 sec And with the -sh parameter. 322187 lines compiled, 28.7 sec These numbers tell you exactly nothing. At best they give you an indication of how complex your project is. I have Delphi compile times of over 2 minutes on projects that have maybe 5.000 lines. It takes this long, because there are a lot of dependencies on external packages and whatnot, and Delphi takes a long time loading the packages and correctly resolving the symbols. Just for fun, I once managed to have delphi compiling 5 minutes (!) on a single form project. (granted, I did put up some messy dependencies :)) The only way to measure the compiler speed is to create 1 very large unit (say 500.000 lines) and let the compilers compile that single unit. Michael. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
The loop in outputfilter can be simplified quite a bit All the extra handling of the asyncprocess can be reduced to the following lines (the event handlers can be removed): if fProcess is TAsyncProcess then begin Count:=TAsyncProcess(fProcess).NumBytesAvailable; if Count 0 then Count:=fProcess.Output.Read(Buf[1], Min(Count, length(Buf))); end else // using a blocking TProcess Count:=fProcess.Output.Read(Buf[1], length(Buf)); if (Count=0) and (not fProcess.Running) then break else if count = 0 then sleep(30); Works on windows / shouldn't change anything on *nix Martin wrote: Vincent Snijders wrote: I have looked a bit more why outputfilter is so slow at parsing a lot of compiler output (e.g. compiling a simple LCL app with -va). The pipe buffer is rather small, NumBytesAvailable is not bigger than 1280 bytes. So OnAsyncReadData reads only 1280 bytes at a time. OnAsyncReadData is only called after an application.handlemessage. So on line 319 of outputfilter.pas, the buffer is most times empty, but NumBytesAvailable = 1280, and we are sleeping until Application.HandleMessage is called. I had some success with the following patch. It does a sleep(0) to give the compiler the chance to write to the pipe again and tries to read it again. --- ide/outputfilter.pas +++ ide/outputfilter.pas @@ -1206,9 +1206,12 @@ var Count: LongWord; begin if fProcess=nil then exit; - Count:=TAsyncProcess(fProcess).NumBytesAvailable; - if Count0 then -FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count); + repeat +Count:=TAsyncProcess(fProcess).NumBytesAvailable; +if Count0 then + FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count); +sleep(0); + until Count=0; end; function TOutputFilter.CreateScanners(ScannerOptions: TStrings): boolean; If I understand it correctly the problem is that TAsyncProcess, currently only reads data, when the event is triggered, which is in a ProcessMessages. Ouch. Either TAsyncProcess.OnReadData is transferred into the main loop, in a similar way as for the normal TProcess. I have not tested this, but it should work, as NumBytesAvailable seems to be none blocking. = This could simplify the overall loop as the handling of the 2 projects would be more or less the same. Alternatively, before we sleep(3), we could check for data, and process it. end else begin // no new input, but process still running if TheAsyncProcess.NumBytesAvailable 0 then Application.ProcessMessages else Sleep(30); end; -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Friday 14 August 2009 16:58:40 Graeme Geldenhuys wrote: Martin Schreiber wrote: 25 s is too much to wait and too little to sleep. ;-) I so agree with that! :-) Same command line params as yours - on a Intel P4 2.4Ghz. 322187 lines compiled, 33.6 sec And with the -sh parameter. 322187 lines compiled, 28.7 sec Martin, how does your results compare when you use Delphi's bcc32 compiler? http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg08029.html Martin -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Martin Schreiber wrote: Martin, how does your results compare when you use Delphi's bcc32 compiler? http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg08029.html Wow, that is a sizeable difference. I did not realize the Delphi compiler was that fast. Regards, - Graeme - fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Graeme Geldenhuys schreef: Hi, Does the Lazarus IDE use multi-threading for anything? Will it take advantage of a Quad Core processor? The IDE itself is single threaded. Examples where I think it could be used (if not already) - background parsing of units - fpdoc lookups - compile project in separate thread (currently the IDE is almost completely non responsive when compiling) Compiler runs in a separate process. On windows this process is run asynchronous, on linux there is some issue with TAsyncProcess, Mattias and Marc know more about it. Under windows this works not perfectly though, compile a project with -va, it takes a long time, but CPU is used very little (10%). I'm sure there are many other areas as well? Does anybody know if FPC takes multiple cores in consideration when compiling? AFAIK, not yet. Vincent -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Graeme Geldenhuys wrote: Hi, Does the Lazarus IDE use multi-threading for anything? Will it take advantage of a Quad Core processor? Examples where I think it could be used (if not already) - background parsing of units - fpdoc lookups - compile project in separate thread (currently the IDE is almost completely non responsive when compiling) Don't know about the others, but compiling is done by a separate process, so there is a chance that your os runs it in a different core. Marc -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Graeme Geldenhuys schreef: Marc Weustink wrote: Don't know about the others, but compiling is done by a separate process, so there is a chance that your os runs it in a different core. That's not the same as running it in a separate thread though - is it? Quoted from a Posix Thread tutorial I found on the internet. http://www.yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html Threads require less overhead than forking or spawning a new process because the system does not initialize a new system virtual memory space and environment for the process. So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Yes, it is more efficient. The fp ide also doesn't create a new process. The disadvantage of using the same process is that the coupling is much tighter: No easy switching between compiler versions, and crashing the compiler crashes the IDE. Vincent -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Thu, 13 Aug 2009, Graeme Geldenhuys wrote: Marc Weustink wrote: Don't know about the others, but compiling is done by a separate process, so there is a chance that your os runs it in a different core. That's not the same as running it in a separate thread though - is it? Quoted from a Posix Thread tutorial I found on the internet. http://www.yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html Threads require less overhead than forking or spawning a new process because the system does not initialize a new system virtual memory space and environment for the process. So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Efficient, yes; Desirable: absolutely not. You cannot do that except by including the compiler in the IDE. This opens up a whole other can of worms, finally resulting in a less stable IDE. As it is now, you can switch compilers on the fly; you can't do that if the compiler is compiled-in in the IDE. Michael. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Graeme Geldenhuys wrote: How can I see the thread-count per process / application under Linux? To answer my own question... $ watch -n.1 'ps -C thunderbird-bin -L -o pid,tid,pcpu,state' This will execute the ps command every second showing a live update of the threads for the 'thunderbird-bin' process. Pretty cool! :) For more such handy command line functions have a look at: http://www.pixelbeat.org/cmdline.html#monitor Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Michael Van Canneyt schrieb: So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Efficient, yes; As I pointed out on fpc-core a few days ago, I'd like to create an fpc which stays in memory and takes commands via pipes/sockets/ipc whatever to reduce process startup times. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
I published and provided source for Manager and Worker Threads. It includes advanced knowledge on how to design a highly efficient system with little to no wait-states. See: http://wiki.freepascal.org/Manager_Worker_Threads_System Let me know what you think, - Andrew On Thu, Aug 13, 2009 at 5:40 AM, Graeme Geldenhuysgrae...@opensoft.homeip.net wrote: Hi, Does the Lazarus IDE use multi-threading for anything? Will it take advantage of a Quad Core processor? Examples where I think it could be used (if not already) - background parsing of units - fpdoc lookups - compile project in separate thread (currently the IDE is almost completely non responsive when compiling) I'm sure there are many other areas as well? Does anybody know if FPC takes multiple cores in consideration when compiling? How can I see the thread-count per process / application under Linux? Under windows that's pretty easy with the Task Manager from what I remember. Ubuntu / Gnome's System Monitor doesn't have a thread count column. :-( Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Thu, 13 Aug 2009, Florian Klaempfl wrote: Michael Van Canneyt schrieb: So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Efficient, yes; As I pointed out on fpc-core a few days ago, I'd like to create an fpc which stays in memory and takes commands via pipes/sockets/ipc whatever to reduce process startup times. Aha. I found it. The thread title was rather confusing. If it is using IPC, then it should be quite a good solution. We have the necessary things in place, the simple IPC implementation I made is more than sufficient for this... Michael. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Graeme Geldenhuys schrieb: Quoted from a Posix Thread tutorial I found on the internet. http://www.yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html Threads require less overhead than forking or spawning a new process because the system does not initialize a new system virtual memory space and environment for the process. IMO it's not a good idea to let the IDE and the compiler share the same address space. Even if the compiler could reuse the data (files...), loaded into the IDE memory, the synchronized access can cost much runtime. So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Since Lazarus already includes an parser, a built-in compiler should be feasable. But I'm not sure how far the FPC developers will support a threaded version of the compiler, that would integrate neatly into Lazarus. DoDi -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Thu, Aug 13, 2009 at 02:01:38PM +0200, Florian Klaempfl wrote: Michael Van Canneyt schrieb: So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Efficient, yes; As I pointed out on fpc-core a few days ago, I'd like to create an fpc which stays in memory and takes commands via pipes/sockets/ipc whatever to reduce process startup times. Specially it would employ an unit cache. Still I'm not entirely convinced if a built-in compiler wouldn't be better. For the problems that you get, you also loose some, since the compiler is essentially fully synchronous to the calling Lazarus thread. The multiple compiler argument (for crosscompilation) could be resolved by using packages based plugin architecture in time. It would cause a tighter coupling between FPC and Lazarus, and require changes in project structure and releasing, for sure. But maybe it would be more desirable long term. 99% of the users strictly use as installed anyway, and the 1% is capable of compiling their own. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Marc Weustink wrote: Don't know about the others, but compiling is done by a separate process, so there is a chance that your os runs it in a different core. OK, after reading all the other responses I now understand the flexibility of keeping FPC and Lazarus separate, but why then is Lazarus non-responsive while the compiler is compiling in a separate process? Simply clicking on the editor tabs take 10-15 seconds before they react. I currently have a Intel P4 2.4Ghz with 1.5GB memory, running Ubuntu 8.04.2. Lazarus v0.9.27 from last week. Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
OK, after reading all the other responses I now understand the flexibility of keeping FPC and Lazarus separate, but why then is Lazarus non-responsive while the compiler is compiling in a separate process? because IDE is reading FPC output? and gives only a few .ProcessMessages for user interface? thanks, dmitry -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Vincent Snijders wrote: Yes, it is more efficient. The fp ide also doesn't create a new process. Interesting, I'll take a peak at that. The disadvantage of using the same process is that the coupling is much tighter: No easy switching between compiler versions, and crashing the compiler crashes the IDE. Can somebody that is more familiar with the internals of the Lazarus IDE tell me, would it be hard to implement the IDE with FPC built in, so that compiling can be done in a separate thread instead of separate process? I'll do the work of course - if it's not a major job. ;-) This would simply be an exercise to fulfill my own curiosity. Maybe if it works well and does not require to much code changes, it could be wrapped in IFDEF's for more advanced users that want to try it for themselves. Possible or not (in the Lazarus IDE), I'll take a look at the FP IDE to see how such an integration works. All in the name of learning multi-threading. ;-) PS: Does Delphi or Kylix have the compilers built-in or do they work similar to Lazarus IDE? Regards, - Graeme - fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Can somebody that is more familiar with the internals of the Lazarus IDE tell me, would it be hard to implement the IDE with FPC built in, so that compiling can be done in a separate thread instead of separate process? I'll do the work of course - if it's not a major job. ;-) The most hardest job, is that Lazarus treats compiler as a separate command tool. There's no TCompiler wrapper class, that would hide actually compiler implementation (i.e. calling for external process, or using built-in lazarus). AFAIK, there're two components that would require change: - IDE (building, compilation, compiler options) - CodeTools (they're using information about units used from the compiler). If you're able to modify the code, to use some abstract TFPCCompiler class. Then it would be possible to implement TFPCExternalCompiler and TFPCInternalCompiler, without much troubles. Of course, it would be even more efficient, if CodeTools share/re-use some information about sources with the internal compiler, but this could be done later. Because even now, Codetools are working great! (thanks Mattias)! thanks, dmitry -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Thu, 13 Aug 2009, Graeme Geldenhuys wrote: Vincent Snijders wrote: Yes, it is more efficient. The fp ide also doesn't create a new process. Interesting, I'll take a peak at that. The disadvantage of using the same process is that the coupling is much tighter: No easy switching between compiler versions, and crashing the compiler crashes the IDE. Can somebody that is more familiar with the internals of the Lazarus IDE tell me, would it be hard to implement the IDE with FPC built in, so that compiling can be done in a separate thread instead of separate process? I'll do the work of course - if it's not a major job. ;-) This would simply be an exercise to fulfill my own curiosity. Maybe if it works well and does not require to much code changes, it could be wrapped in IFDEF's for more advanced users that want to try it for themselves. Possible or not (in the Lazarus IDE), I'll take a look at the FP IDE to see how such an integration works. All in the name of learning multi-threading. ;-) The FP IDE does not use multi-threading. While the compiler is compiling, the IDE is stuck. PS: Does Delphi or Kylix have the compilers built-in or do they work similar to Lazarus IDE? As far as I know: They have the compiler in a DLL which is loaded when the IDE starts. Michael. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Graeme Geldenhuys schrieb: Vincent Snijders wrote: Yes, it is more efficient. The fp ide also doesn't create a new process. Interesting, I'll take a peak at that. The disadvantage of using the same process is that the coupling is much tighter: No easy switching between compiler versions, and crashing the compiler crashes the IDE. Can somebody that is more familiar with the internals of the Lazarus IDE tell me, would it be hard to implement the IDE with FPC built in, Probably simple. The compiler is started by the compile function in fpc/compiler/compiler.pas and takes simply a command line arguments. You need to set some hooks though to catch messages, see e.g. fpc/ide/fpcompile.pas (especially DoCompile starting at 868) and fpc/ide/fpintf.pas. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Thu, 13 Aug 2009 12:40:54 +0200 Graeme Geldenhuys grae...@opensoft.homeip.net wrote: Hi, Does the Lazarus IDE use multi-threading for anything? It is single threaded. Will it take advantage of a Quad Core processor? Not yet. Examples where I think it could be used (if not already) - background parsing of units Yes, this is planned. Some expensive parts can be done in worker threads. But it is no simple task. Keep in mind that the codetools do not parse in one run like the compiler and they update a lot of caches. - fpdoc lookups That should be easier. This already works in several separate parts executed on idle. - compile project in separate thread (currently the IDE is almost completely non responsive when compiling) That should be the easiest: The compiler is an external process and TAsyncProcess is already available. The only problem is that TAsyncProcess bites the debugger under Linux. Compile LCL and IDE with -dUseAsyncProcess. Run an application. Close the IDE. The gui will close, but lazarus is still running. I'm sure there are many other areas as well? I'm not so sure. For example: An expensive function is Find in files which could be multithreaded. But the bottleneck here is the memory system and the disk, not the cpu. Does anybody know if FPC takes multiple cores in consideration when compiling? AFAIK Florian did some tests years ago and the bottleneck was the memory system and the disk. Mattias -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Thu, 13 Aug 2009 14:01:38 +0200 Florian Klaempfl flor...@freepascal.org wrote: Michael Van Canneyt schrieb: So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Efficient, yes; As I pointed out on fpc-core a few days ago, I'd like to create an fpc which stays in memory and takes commands via pipes/sockets/ipc whatever to reduce process startup times. Hooray. \O/ Can I help somehow? Mattias -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Mattias Gaertner schrieb: On Thu, 13 Aug 2009 14:01:38 +0200 Florian Klaempfl flor...@freepascal.org wrote: Michael Van Canneyt schrieb: So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Efficient, yes; As I pointed out on fpc-core a few days ago, I'd like to create an fpc which stays in memory and takes commands via pipes/sockets/ipc whatever to reduce process startup times. Hooray. \O/ Can I help somehow? Well, first we have to decide which method to use. I even think a slightly modified fpc which can be controlled through stdin is the easiest. -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
On Thu, 13 Aug 2009 22:24:37 +0200 Florian Klaempfl flor...@freepascal.org wrote: Mattias Gaertner schrieb: On Thu, 13 Aug 2009 14:01:38 +0200 Florian Klaempfl flor...@freepascal.org wrote: Michael Van Canneyt schrieb: So wouldn't it be more efficient to create a new thread for the compiler instead of a new process? Efficient, yes; As I pointed out on fpc-core a few days ago, I'd like to create an fpc which stays in memory and takes commands via pipes/sockets/ipc whatever to reduce process startup times. Hooray. \O/ Can I help somehow? Well, first we have to decide which method to use. I even think a slightly modified fpc which can be controlled through stdin is the easiest. Ok. Don't forget proper encoding. ;) Mattias -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Michael Van Canneyt wrote: They have the compiler in a DLL which is loaded when the IDE starts. Clever. I would imagine their command line compiler also uses that DLL then. Simple code reuse. Regards, - Graeme - fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Florian Klaempfl wrote: Probably simple. The compiler is started by the compile function in fpc/compiler/compiler.pas and takes simply a command line arguments. You need to set some hooks though to catch messages, see e.g. fpc/ide/fpcompile.pas (especially DoCompile starting at 868) and fpc/ide/fpintf.pas. Thanks for the hints Florian, I'll investigate this further. I'll probably play around with FP IDE first. Regards, - Graeme - fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Re: [Lazarus] Multi-threading support in IDE
Mattias Gaertner wrote: Examples where I think it could be used (if not already) - background parsing of units Yes, this is planned. Some expensive parts can be done in worker Excellent. - fpdoc lookups That should be easier. This already works in several separate parts executed on idle. Nice. available. The only problem is that TAsyncProcess bites the debugger under Linux. Compile LCL and IDE with -dUseAsyncProcess. Run an application. Close the IDE. The gui will close, but lazarus is still running. I probably run my applications 1% of the times via the IDE. All other uses and testing is by running them from a command line. I few years ago when debugging via the IDE really sucked, I wrote various logging methods which I got so used to and always works. Logging gets done via a separate thread and I can log to various outputs. cached logging to file, cached logging to GUI or non-cached logging to console (stdout). I'll recompile tomorrow with-dUseAsyncProcess and see how it goes. For example: An expensive function is Find in files which could be multithreaded. But the bottleneck here is the memory system and the disk, not the cpu. That is an excellent candidate. We did some testing with a multi-threaded application that does CRC calculation on files, recursively through directories. It was one of the multi-threading entries to a Borland Multi-Threading competition held some years ago. We adapted the program for our needs and by using 2-4 threads on a multi-core system improved overall throughput a lot. The app did not work so well when the files were on CD-ROM though. :-) A major disk bottleneck, but hard drives or raid system is much faster. AFAIK Florian did some tests years ago and the bottleneck was the memory system and the disk. I believe GCC can use threads for parallel compiling, so there must be some advantage. With faster hard drives or SSD drives it should make a difference - I guess. But yes, I understand your point regarding disk IO being slower than in-memory computations. Regards, - Graeme - fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/ -- ___ Lazarus mailing list Lazarus@lists.lazarus.freepascal.org http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
