Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-14 Thread Hans-Peter Diettrich

Florian Klämpfl schrieb:


This is a very specific example which allows to explain rather simple
the slowness of 2.x: The reason is a decision geared by maintainability
and portability: 2.x uses a so-called graph colouring register allocator
while 1.x used a pretty simple register allocator specifically tailored
for i386.


Shouldn't we make the register allocator configurable, so that e.g. 
non-release builds can become faster, and several replacements can be 
tested easily?


The same for other parts of the compiler, where the time-per-task is the 
first information required to detect real bottlenecks, and to check 
alternative solutions.


DoDi

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-14 Thread Hans-Peter Diettrich

Florian Klämpfl schrieb:


Anyways, before this ends in an endless discussion: if anybody is
interested in improving FPC compilation speed (for my needs is
sufficient) and have a look at fillchar


IMO not FillChar is the bottleneck, instead it's the access to newly 
allocated memory in/around InitInstance, resulting in page faults.


DoDi

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Martin Schreiber

 Am 12.09.2010 00:20, schrieb Graeme Geldenhuys:

Now this is weird! Anybody else spotted the difference? Delphi seems
to compile +-28000 lines less that FPC! Florian, I presume it's the
same machine with the same MSEgui source code revision? What would be
the reason for that?

Would that (lines compiled) also account for the huge difference in
bytes data. What is the final executable size (not that this matters
much to me) generated by the two compilers?

Martin, do you get the same results for lines compiled?
MSEide Delphi version has no DB components. In order to switch off non 
Delphi components in FPC mode some defines must be set.

This is with MSEide+MSEgui SVN trunk rev.3910:

Delphi:
280496 lines, 4.49 seconds, 2127360 bytes code, 752073 bytes data.
12.09.2010  07:013'129'856 mseide.exe

FPC 2.4.0:
281604 lines compiled, 40.2 sec , 2136496 bytes code, 1541192 bytes data
12.09.2010  07:093'687'228 mseidefp.exe

FPC 2.4.0 with debug info:
281604 lines compiled, 54.4 sec , 2142896 bytes code, 1541336 bytes data
12.09.2010  07:16   41'759'691 mseidefp.exe

Commandline Delphi:
dcc32 -B -I..\..\lib\common\kernel -U..\..\lib\common\kernel 
-U..\..\lib\common\kernel\i386-win32 -U..\..\lib\common\image 
-U..\..\lib\common\widgets -U..\..\lib\common\designutils 
-U..\..\lib\common\sysutils -U..\..\lib\common\editwidgets 
-U..\..\lib\common\dialogs -U..\..\lib\common\regcomponents 
-U..\..\lib\common\serialcomm -U..\..\lib\common\printer 
-U..\..\lib\common\ifi -U..\..\lib\common\math -dmse_no_db 
-dmse_no_opengl mseide.pas


Commandline FPC:
ppc386.exe -O2 -CX -XX -Xs -B -I..\..\lib\common\kernel 
-Fu..\..\lib\common\kernel -Fu..\..\lib\common\kernel\i386-win32 
-Fu..\..\lib\common\image -Fu..\..\lib\common\widgets 
-Fu..\..\lib\common\designutils -Fu..\..\lib\common\sysutils 
-Fu..\..\lib\common\editwidgets -Fu..\..\lib\common\dialogs 
-Fu..\..\lib\common\regcomponents -Fu..\..\lib\common\serialcomm 
-Fu..\..\lib\common\printer -Fu..\..\lib\common\ifi 
-Fu..\..\lib\common\math -dmse_no_db -dmse_no_opengl -omseidefp.exe 
mseide.pas


Commandline FPC with ddebug info:
ppc386.exe -O2 -CX -XX -gl -B -I..\..\lib\common\kernel 
-Fu..\..\lib\common\kernel -Fu..\..\lib\common\kernel\i386-win32 
-Fu..\..\lib\common\image -Fu..\..\lib\common\widgets 
-Fu..\..\lib\common\designutils -Fu..\..\lib\common\sysutils 
-Fu..\..\lib\common\editwidgets -Fu..\..\lib\common\dialogs 
-Fu..\..\lib\common\regcomponents -Fu..\..\lib\common\serialcomm 
-Fu..\..\lib\common\printer -Fu..\..\lib\common\ifi 
-Fu..\..\lib\common\math -dmse_no_db -dmse_no_opengl -omseidefp.exe 
mseide.pas


Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Mattias Gaertner
On Sun, 12 Sep 2010 07:33:55 +0200
Martin Schreiber mse00...@gmail.com wrote:

 On Saturday, 11. September 2010 21.10:20 Florian Klämpfl wrote:
 
  Anyways, before this ends in an endless discussion: if anybody is
  interested in improving FPC compilation speed (for my needs is
  sufficient) and have a look at fillchar and, have a look at FPC's unit
  loading algorithm (not the actual i/o itself but how all the symbols
  classes etc. are restored). This two points are bottlenecks which might
  help when they are improved, though it's pretty unlikely that this will
  improve things more than a few percent.
 
 Agreed. My opinion is that before we start to implement difficult and 
 error-prone multi-threading into FPC we should find out why the hell Delphi 7 
 can compile so much faster and produces even better code?

Seeing that dcc is only 800K: 
maybe it fits into the cpu cache.

Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Florian Klämpfl
Am 12.09.2010 07:33, schrieb Martin Schreiber:
 On Saturday, 11. September 2010 21.10:20 Florian Klämpfl wrote:

 Anyways, before this ends in an endless discussion: if anybody is
 interested in improving FPC compilation speed (for my needs is
 sufficient) and have a look at fillchar and, have a look at FPC's unit
 loading algorithm (not the actual i/o itself but how all the symbols
 classes etc. are restored). This two points are bottlenecks which might
 help when they are improved, though it's pretty unlikely that this will
 improve things more than a few percent.
 
 Agreed. My opinion is that before we start to implement difficult and 
 error-prone multi-threading into FPC we should find out why the hell Delphi 7 
 can compile so much faster 

Because of the same reason why it seems to take years to port delphi to
64 bit: different design goals. It seems speed were the only design
goal, nobody took care about maintainability or portability.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Florian Klämpfl
Am 12.09.2010 01:20, schrieb Graeme Geldenhuys:
 On 11 September 2010 21:10, Florian Klämpfl flor...@freepascal.org wrote:
 
 FPC
 ---
 Linking mseidefp.exe
 308574 lines compiled, 10.6 sec , 2577952 bytes code, 1618920 bytes data

 
 Delphi
 -
 mseide.pas(63)
 280491 lines, 2.18 seconds, 2110568 bytes code, 752073 bytes data.
 
 
 Now this is weird!  Anybody else spotted the difference?  Delphi seems
 to compile +-28000 lines less that FPC! Florian, I presume it's the
 same machine with the same MSEgui source code revision? What would be
 the reason for that?

Ifdef'ed unit usage/includes?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Martin Schreiber
On Sunday, 12. September 2010 10.12:59 Florian Klämpfl wrote:
 
  Agreed. My opinion is that before we start to implement difficult and
  error-prone multi-threading into FPC we should find out why the hell
  Delphi 7 can compile so much faster

 Because of the same reason why it seems to take years to port delphi to
 64 bit: different design goals. It seems speed were the only design
 goal, nobody took care about maintainability or portability.

And that results in a discrepancy of factor 5..10? I can't believe it.

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Marco van de Voort
In our previous episode, Mattias Gaertner said:
  Agreed. My opinion is that before we start to implement difficult and 
  error-prone multi-threading into FPC we should find out why the hell Delphi 
  7 
  can compile so much faster and produces even better code?
 
 Seeing that dcc is only 800K: 
 maybe it fits into the cpu cache.

I assume dcc.exe uses more data than code :-)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Florian Klämpfl
Am 12.09.2010 10:21, schrieb Martin Schreiber:
 On Sunday, 12. September 2010 10.12:59 Florian Klämpfl wrote:

 Agreed. My opinion is that before we start to implement difficult and
 error-prone multi-threading into FPC we should find out why the hell
 Delphi 7 can compile so much faster

 Because of the same reason why it seems to take years to port delphi to
 64 bit: different design goals. It seems speed were the only design
 goal, nobody took care about maintainability or portability.
 
 And that results in a discrepancy of factor 5..10? I can't believe it.

Digging out 1.0.10 and using some extreme example:

C:\fpc\tests\webtbsc:\pp 1.0.10\bin\win32\ppc386.exe tw2242 -O2
Free Pascal Compiler version 1.0.10 [2003/06/27] for i386
Copyright (c) 1993-2003 by Florian Klaempfl
Target OS: Win32 for i386
Compiling tw2242.pp
Linking tw2242.exe
13083 Lines compiled, 0.8 sec

C:\fpc\tests\webtbsfpc tw2242 -O2
Free Pascal Compiler version 2.4.0 [2009/12/18] for i386
Copyright (c) 1993-2009 by Florian Klaempfl
Target OS: Win32 for i386
Compiling tw2242.pp
Linking tw2242.exe
13083 lines compiled, 4.7 sec , 301376 bytes code, 1864 bytes data
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Martin Schreiber
On Sunday, 12. September 2010 10.29:32 Florian Klämpfl wrote:
 
  And that results in a discrepancy of factor 5..10? I can't believe it.

 Digging out 1.0.10 and using some extreme example:

 C:\fpc\tests\webtbsc:\pp 1.0.10\bin\win32\ppc386.exe tw2242 -O2
 Free Pascal Compiler version 1.0.10 [2003/06/27] for i386
 Copyright (c) 1993-2003 by Florian Klaempfl
 Target OS: Win32 for i386
 Compiling tw2242.pp
 Linking tw2242.exe
 13083 Lines compiled, 0.8 sec

 C:\fpc\tests\webtbsfpc tw2242 -O2
 Free Pascal Compiler version 2.4.0 [2009/12/18] for i386
 Copyright (c) 1993-2009 by Florian Klaempfl
 Target OS: Win32 for i386
 Compiling tw2242.pp
 Linking tw2242.exe
 13083 lines compiled, 4.7 sec , 301376 bytes code, 1864 bytes data

Impressive. Now we can hook in. Where is the difference? What makes 2.4.0 so 
much slower?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Marco van de Voort
In our previous episode, Martin Schreiber said:
  Anyways, before this ends in an endless discussion: if anybody is
  interested in improving FPC compilation speed (for my needs is
  sufficient) and have a look at fillchar and, have a look at FPC's unit
  loading algorithm (not the actual i/o itself but how all the symbols
  classes etc. are restored). This two points are bottlenecks which might
  help when they are improved, though it's pretty unlikely that this will
  improve things more than a few percent.
 
 Agreed. My opinion is that before we start to implement difficult and 
 error-prone multi-threading into FPC we should find out why the hell Delphi 7 
 can compile so much faster and produces even better code?

I partially agree with you in the fact that the exact reasons are not known.

I'm no expert on profiling the compiler, but if I read the various threads
over the years I see defensive and conflicting statements:

In discussions with Hans, it is said that I/O is not a factor, since after
one run everything is cached anyway, and then in this thread I/O is to blame
for a huge difference in speed.

The same with the fact that we use shortstring for performance in many
places where delphi in fact allows longer mangled names and is faster.

That leaves the maintainability bit. I think that is certainly true,
and it probably can be tweaked to be a bit better. It won't be easy though
and is more likely to be a lot of things that add a little than few that
matter a lot. Which makes it a permanent positions instead of a one-off
effort. Any takers?

As far as the I/O performance discussion goes, some observations:

- To find something interesting, probably the test has to be done for
  various sizes of code. To see if maybe some startup factor (Rather than
  actual processing) is a cause.
- FPC has to read at least twice the number of files (.ppu/.o). So if
  opening or number of files in the dir path is a factor, it will do 
  worse by default. 
- Possibily, a defaultly installed windows searches through a larger unit
  path (more dirs, more file) than Delphi _default_
- For that to be found, more has to be known about dcc.cfg during the 
  tests.
- THis means that tests will have to be repeated with various sizes
  of unitpath trees (both dirs and files) to see if this is a discussion.
- Most profiling recently afaik has been done by Jonas, and thus not
  on Windows. Yet the delphi comparisons are on windows. 
- Actually linking should be avoided in the test. It might obscure
  things.



___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Marco van de Voort
In our previous episode, Florian Kl?mpfl said:
 Digging out 1.0.10 and using some extreme example:
 
 C:\fpc\tests\webtbsc:\pp 1.0.10\bin\win32\ppc386.exe tw2242 -O2
 Free Pascal Compiler version 1.0.10 [2003/06/27] for i386
 Copyright (c) 1993-2003 by Florian Klaempfl
 Target OS: Win32 for i386
 Compiling tw2242.pp
 Linking tw2242.exe
 13083 Lines compiled, 0.8 sec
 
 C:\fpc\tests\webtbsfpc tw2242 -O2
 Free Pascal Compiler version 2.4.0 [2009/12/18] for i386
 Copyright (c) 1993-2009 by Florian Klaempfl
 Target OS: Win32 for i386
 Compiling tw2242.pp
 Linking tw2242.exe
 13083 lines compiled, 4.7 sec , 301376 bytes code, 1864 bytes data

Do both numbers include linking ?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Jonas Maebe

On 12 Sep 2010, at 10:39, Martin Schreiber wrote:

 On Sunday, 12. September 2010 10.29:32 Florian Klämpfl wrote:
 
 And that results in a discrepancy of factor 5..10? I can't believe it.
 
 Digging out 1.0.10 and using some extreme example:
 
 C:\fpc\tests\webtbsc:\pp 1.0.10\bin\win32\ppc386.exe tw2242 -O2
 Free Pascal Compiler version 1.0.10 [2003/06/27] for i386
 Copyright (c) 1993-2003 by Florian Klaempfl
 Target OS: Win32 for i386
 Compiling tw2242.pp
 Linking tw2242.exe
 13083 Lines compiled, 0.8 sec
 
 C:\fpc\tests\webtbsfpc tw2242 -O2
 Free Pascal Compiler version 2.4.0 [2009/12/18] for i386
 Copyright (c) 1993-2009 by Florian Klaempfl
 Target OS: Win32 for i386
 Compiling tw2242.pp
 Linking tw2242.exe
 13083 lines compiled, 4.7 sec , 301376 bytes code, 1864 bytes data
 
 Impressive. Now we can hook in. Where is the difference? What makes 2.4.0 so 
 much slower?

In the above case: primarily the register allocator (which I mentioned before).


Jonas

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Florian Klämpfl
Am 12.09.2010 10:39, schrieb Martin Schreiber:
 On Sunday, 12. September 2010 10.29:32 Florian Klämpfl wrote:

 And that results in a discrepancy of factor 5..10? I can't believe it.

 Digging out 1.0.10 and using some extreme example:

 C:\fpc\tests\webtbsc:\pp 1.0.10\bin\win32\ppc386.exe tw2242 -O2
 Free Pascal Compiler version 1.0.10 [2003/06/27] for i386
 Copyright (c) 1993-2003 by Florian Klaempfl
 Target OS: Win32 for i386
 Compiling tw2242.pp
 Linking tw2242.exe
 13083 Lines compiled, 0.8 sec

 C:\fpc\tests\webtbsfpc tw2242 -O2
 Free Pascal Compiler version 2.4.0 [2009/12/18] for i386
 Copyright (c) 1993-2009 by Florian Klaempfl
 Target OS: Win32 for i386
 Compiling tw2242.pp
 Linking tw2242.exe
 13083 lines compiled, 4.7 sec , 301376 bytes code, 1864 bytes data
 
 Impressive. Now we can hook in. Where is the difference? What makes 2.4.0 so 
 much slower?

This is a very specific example which allows to explain rather simple
the slowness of 2.x: The reason is a decision geared by maintainability
and portability: 2.x uses a so-called graph colouring register allocator
while 1.x used a pretty simple register allocator specifically tailored
for i386.

The 2.x register allocator is more robust (no more internalerrors 10),
it is small (basically 2k lines, compiler/rgobj.pas) and it generates
reasonable register allocations on all types of CPUs (remember, FPC
supports CPUs with high register pressure like i386 as well as those
with a lot registers: PowerPC) we support, so we need only to maintain
one register allocator.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Jonas Maebe

On 12 Sep 2010, at 10:43, Marco van de Voort wrote:

 I'm no expert on profiling the compiler, but if I read the various threads
 over the years I see defensive and conflicting statements:
 
 In discussions with Hans, it is said that I/O is not a factor, since after
 one run everything is cached anyway, and then in this thread I/O is to blame
 for a huge difference in speed.

Disk throughput doesn't really matter. Reading directory contents, getting file 
information and opening/closing files is another matter.

 The same with the fact that we use shortstring for performance in many
 places where delphi in fact allows longer mangled names and is faster.

That's a non sequitur.

 - Possibily, a defaultly installed windows searches through a larger unit
  path (more dirs, more file) than Delphi _default_

I've also been thinking that.

 - Most profiling recently afaik has been done by Jonas, and thus not
  on Windows. Yet the delphi comparisons are on windows. 

There's a free profiler for Windows by AMD: 
http://developer.amd.com/cpu/codeanalyst/Pages/default.aspx


Jonas___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Jonas Maebe

On 12 Sep 2010, at 11:41, Jonas Maebe wrote:

 There's a free profiler for Windows by AMD: 
 http://developer.amd.com/cpu/codeanalyst/Pages/default.aspx

And by Microsoft: http://msdn.microsoft.com/en-us/performance/cc825801.aspx


Jonas___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Mattias Gaertner
On Sun, 12 Sep 2010 10:25:22 +0200 (CEST)
mar...@stack.nl (Marco van de Voort) wrote:

 In our previous episode, Mattias Gaertner said:
   Agreed. My opinion is that before we start to implement difficult and 
   error-prone multi-threading into FPC we should find out why the hell 
   Delphi 7 
   can compile so much faster and produces even better code?
  
  Seeing that dcc is only 800K: 
  maybe it fits into the cpu cache.
 
 I assume dcc.exe uses more data than code :-)

CPU caches do not work FIFO.
If FPC does not fit into the CPU cache, then the CPU has to constantly
load code mem additionally to the data.

Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Marco van de Voort
In our previous episode, Mattias Gaertner said:
   Seeing that dcc is only 800K: 
   maybe it fits into the cpu cache.
  
  I assume dcc.exe uses more data than code :-)
 
 CPU caches do not work FIFO.

I assume not, since the administration overhead would be too large.

 If FPC does not fit into the CPU cache, then the CPU has to constantly
 load code mem additionally to the data.

- only the hot track has to fit. It makes no sense to cache processing 
  parameter options.
- all caches are afaik unified nowadays, so both 
  code and data dynamically share the cache

It could be that the working set is larger for FPC, but I doubt you can see
this from the .exe size.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Martin Schreiber

 Am 12.09.2010 10:12, schrieb Florian Klämpfl:

The 2.x register allocator is more robust (no more internalerrors 10),
it is small (basically 2k lines, compiler/rgobj.pas) and it generates
reasonable register allocations on all types of CPUs (remember, FPC
supports CPUs with high register pressure like i386 as well as those
with a lot registers: PowerPC) we support, so we need only to maintain
one register allocator.

I replaced the += by := s +, testresults on the same machine as before:

Delphi 7:
E:\FPC\svn\fixes_2_4\tests\webtbfdcc32 tw2242x.pp
Borland Delphi Version 15.0
Copyright (c) 1983,2002 Borland Software Corporation
tw2242x.pp(20009)
20010 lines, 0.19 seconds, 311088 bytes code, 1801 bytes data.

E:\FPC\svn\fixes_2_4\tests\webtbf

FPC:
E:\FPC\svn\fixes_2_4\tests\webtbfppc386 tw2242x.pp
Free Pascal Compiler version 2.4.0 [2009/12/18] for i386
Copyright (c) 1993-2009 by Florian Klaempfl
Target OS: Win32 for i386
Compiling tw2242x.pp
tw2242x.pp(16386,7) Fatal: Procedure too complex, it requires too many 
registers


Fatal: Compilation aborted

Truncated at line 16380:
Delphi 7:
E:\FPC\svn\fixes_2_4\tests\webtbfdcc32 tw2242xtrunc.pp
Borland Delphi Version 15.0
Copyright (c) 1983,2002 Borland Software Corporation
tw2242xtrunc.pp(16382)
16383 lines, 0.16 seconds, 256684 bytes code, 1801 bytes data.

FPC:
E:\FPC\svn\fixes_2_4\tests\webtbfppc386 tw2242xtrunc.pp
Free Pascal Compiler version 2.4.0 [2009/12/18] for i386
Copyright (c) 1993-2009 by Florian Klaempfl
Target OS: Win32 for i386
Compiling tw2242xtrunc.pp
Linking tw2242xtrunc.exe
16381 lines compiled, 12.3 sec , 370736 bytes code, 1864 bytes data

Hmm. ;-)
Please take it with humor. :-)

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Sergei Gorelkin

Martin Schreiber пишет:


Truncated at line 16380:
Delphi 7:
E:\FPC\svn\fixes_2_4\tests\webtbfdcc32 tw2242xtrunc.pp
Borland Delphi Version 15.0
Copyright (c) 1983,2002 Borland Software Corporation
tw2242xtrunc.pp(16382)
16383 lines, 0.16 seconds, 256684 bytes code, 1801 bytes data.

FPC:
E:\FPC\svn\fixes_2_4\tests\webtbfppc386 tw2242xtrunc.pp
Free Pascal Compiler version 2.4.0 [2009/12/18] for i386
Copyright (c) 1993-2009 by Florian Klaempfl
Target OS: Win32 for i386
Compiling tw2242xtrunc.pp
Linking tw2242xtrunc.exe
16381 lines compiled, 12.3 sec , 370736 bytes code, 1864 bytes data

Hmm. ;-)
Please take it with humor. :-)

Does that happen because of the SSA? I mean, it looks like a new register is allocated for every 
statement until limit of 16384 is hit. At the same time, this procedure compiles into a sequence of 
calls, which doesn't require registers at all (except to place function arguments).


Regards,
Sergei
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Jonas Maebe

On 12 Sep 2010, at 16:15, Jonas Maebe wrote:

 On 12 Sep 2010, at 14:50, Martin Schreiber wrote:
 
 On 12 Sep 2010, at 16:10, Sergei Gorelkin wrote:
 
 Does that happen because of the SSA? I mean, it looks like a new register is 
 allocated for every statement until limit of 16384 is hit.
 
 No, that's unrelated to SSA (or even graph colouring). Also, I think the 
 limit is 65535 rather than 16384.

Well, not entirely unrelated (more virtual registers are used with SSA), but 
even without SSA you run out of virtual registers after a while.


Jonas___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Dimitri Smits

- Marco van de Voort mar...@stack.nl schreef:

 I partially agree with you in the fact that the exact reasons are not
 known.
 
 I'm no expert on profiling the compiler, but if I read the various
 threads
 over the years I see defensive and conflicting statements:
 
 In discussions with Hans, it is said that I/O is not a factor, since
 after
 one run everything is cached anyway, and then in this thread I/O is to
 blame
 for a huge difference in speed.

that may be the case for reading, not necessarily for the files being written. 
in ppu.pas, everything you put results in a blockwrite of x bytes. Wouldn't a 
cached memory stream be better, not resulting in those int21h calls or windows 
equivalent calls?
Haven't looked at .s creation (donno where to start looking, but I guess this 
is the same???)

 
 The same with the fact that we use shortstring for performance in
 many
 places where delphi in fact allows longer mangled names and is
 faster.
 
 That leaves the maintainability bit. I think that is certainly true,
 and it probably can be tweaked to be a bit better. It won't be easy
 though
 and is more likely to be a lot of things that add a little than few
 that
 matter a lot. Which makes it a permanent positions instead of a
 one-off
 effort. Any takers?
 
 As far as the I/O performance discussion goes, some observations:
 
 - To find something interesting, probably the test has to be done for
   various sizes of code. To see if maybe some startup factor (Rather
 than
   actual processing) is a cause.

agreed

 - FPC has to read at least twice the number of files (.ppu/.o). So if
   opening or number of files in the dir path is a factor, it will do 
   worse by default. 

actually, this is a false statement. The units that come with delphi (rtl and 
others) are packaged in .dcp (and .bpl). That means that not only those x2 
files, but a factor of those x2 can be done. Admitted that you probably load 
way to much in some cases as well as that it is not easy to know in what .dcp 
your needed unitinfo is included. Or at least isn't obvious.

 - Possibily, a defaultly installed windows searches through a larger
 unit
   path (more dirs, more file) than Delphi _default_

 - For that to be found, more has to be known about dcc.cfg during the
 
   tests.

 - THis means that tests will have to be repeated with various sizes
   of unitpath trees (both dirs and files) to see if this is a
 discussion.
 - Most profiling recently afaik has been done by Jonas, and thus not
   on Windows. Yet the delphi comparisons are on windows. 
 - Actually linking should be avoided in the test. It might obscure
   things.

in that case, all external tools should be avoided.

kind regards,
Dimitri Smits
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Florian Klämpfl
Am 12.09.2010 18:24, schrieb Dimitri Smits:
 
 - Marco van de Voort mar...@stack.nl schreef:
 
 I partially agree with you in the fact that the exact reasons are
 not known.
 
 I'm no expert on profiling the compiler, but if I read the various 
 threads over the years I see defensive and conflicting statements:
 
 In discussions with Hans, it is said that I/O is not a factor,
 since after one run everything is cached anyway, and then in this
 thread I/O is to blame for a huge difference in speed.
 
 that may be the case for reading, not necessarily for the files being
 written. in ppu.pas, everything you put results in a blockwrite of
 x bytes. Wouldn't a cached memory stream be better, not resulting in
 those int21h calls or windows equivalent calls?

The ppu writer uses a buffer of 16 kB which is enough. Believe me, it's
no so simple as just use a memory stream to improve performance. If it's
the case, we would have done it for years.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Florian Klämpfl
Am 12.09.2010 14:50, schrieb Martin Schreiber:

 
 FPC:
 E:\FPC\svn\fixes_2_4\tests\webtbfppc386 tw2242x.pp
 Free Pascal Compiler version 2.4.0 [2009/12/18] for i386
 Copyright (c) 1993-2009 by Florian Klaempfl
 Target OS: Win32 for i386
 Compiling tw2242x.pp
 tw2242x.pp(16386,7) Fatal: Procedure too complex, it requires too many
 registers

This could be fixed for another speed penalty :)

 Please take it with humor. :-)

As long as the compiler itself builds on a reasonable machine in less
than 10 seconds, I'am happy :)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Martin Schreiber
On Sunday, 12. September 2010 18.29:34 Florian Klämpfl wrote:

  Please take it with humor. :-)

 As long as the compiler itself builds on a reasonable machine in less
 than 10 seconds, I'am happy :)

Yup, I know. But there are people who use FPC for other tasks than compiling 
FPC and there are people who have no reasonable machine. ;-)

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Marco van de Voort
In our previous episode, Dimitri Smits said:
  after
  one run everything is cached anyway, and then in this thread I/O is to
  blame
  for a huge difference in speed.
 
 that may be the case for reading, not necessarily for the files being
 written.  in ppu.pas, everything you put results in a blockwrite of x
 bytes.  Wouldn't a cached memory stream be better, not resulting in those
 int21h calls or windows equivalent calls?  Haven't looked at .s creation
 (donno where to start looking, but I guess this is the same???)

It's worth checking out. (writing in general), but I don't expect the
difference to be shocking. If only because typically many more .ppu are read
than written.

  - FPC has to read at least twice the number of files (.ppu/.o). So if
opening or number of files in the dir path is a factor, it will do 
worse by default. 
 
 actually, this is a false statement. The units that come with delphi (rtl
 and others) are packaged in .dcp (and .bpl).  That means that not only
 those x2 files, but a factor of those x2 can be done.  Admitted that
 you probably load way to much in some cases as well as that it is not
 easy to know in what .dcp your needed unitinfo is included.  Or at least
 isn't obvious.

To my best knowledge that is incorrect. Delphi uses the .dcu's (debug or
not), and only preloads the .dcp that are in the runtime packages list
when compiling with packages is checked. 
 
of unitpath trees (both dirs and files) to see if this is a
  discussion.
  - Most profiling recently afaik has been done by Jonas, and thus not
on Windows. Yet the delphi comparisons are on windows. 
  - Actually linking should be avoided in the test. It might obscure
things.
 
 in that case, all external tools should be avoided.

For benchmarking: yes. It would actualy be interesting to learn such info,
since generally I've only regarded command like make all in the past.

There, Windows (time) : linux (time) is about 2:1, but I always blamed the
bulk on this on the slower startup time of windows exes. Such benchmarks
could confirm (or dispell) that assumption.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Marco van de Voort
In our previous episode, Jonas Maebe said:
  I'm no expert on profiling the compiler, but if I read the various threads
  over the years I see defensive and conflicting statements:
  
  In discussions with Hans, it is said that I/O is not a factor, since after
  one run everything is cached anyway, and then in this thread I/O is to blame
  for a huge difference in speed.
 
 Disk throughput doesn't really matter. Reading directory contents, getting
 file information and opening/closing files is another matter.

Good. That's what I wanted to say too. If one excludes linking, there is not
that much left. I assume the moments of timing of the FPC and Delphi compiler
of course could be different too.
 
  The same with the fact that we use shortstring for performance in many
  places where delphi in fact allows longer mangled names and is faster.
 
 That's a non sequitur.

I'm not creating a complot theory here. I just want to state some facts to
avoid sidediscussion obscuring the main problem.

I'm not expecting sb will go really deep and profiling, but that doesn't
mean a somewhat correct definition of the cause of the difference would
hurt. If only for the next discussion. 

  - Most profiling recently afaik has been done by Jonas, and thus not
   on Windows. Yet the delphi comparisons are on windows. 
 
 There's a free profiler for Windows by AMD:
 http://developer.amd.com/cpu/codeanalyst/Pages/default.aspx

I'll see if I can make/enhance the wiki page about profiling the coming
days.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Adem

 On 9/12/2010 12:41 PM, Jonas Maebe wrote:

In discussions with Hans, it is said that I/O is not a factor, since after
one run everything is cached anyway, and then in this thread I/O is to blame
for a huge difference in speed.

Disk throughput doesn't really matter. Reading directory contents, getting file 
information and opening/closing files is another matter.

My exeperience seems to confirm this.

I ran the same tests with Lazarus install on a NAS.

As the connection was 2xGigabit (port-trunked), read/write speeds were 
higher than my local HHD.


Yet, rebuild times doubled.

It seems latency plays a more significant role than thoughput.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Adem

 On 9/12/2010 12:53 PM, Jonas Maebe wrote:

On 12 Sep 2010, at 11:41, Jonas Maebe wrote:


There's a free profiler for Windows by AMD: 
http://developer.amd.com/cpu/codeanalyst/Pages/default.aspx

And by Microsoft: http://msdn.microsoft.com/en-us/performance/cc825801.aspx


It's a 2.5G download --running in the background as I write.

But, do you know if it takes into account processes (exe's) started by 
the process (exe) being profiled?


Also, does anyone know if Intel's vTune would be useful?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Jonas Maebe

On 12 Sep 2010, at 19:05, Adem wrote:

 On 9/12/2010 12:53 PM, Jonas Maebe wrote:
 On 12 Sep 2010, at 11:41, Jonas Maebe wrote:
 
 There's a free profiler for Windows by AMD: 
 http://developer.amd.com/cpu/codeanalyst/Pages/default.aspx
 And by Microsoft: http://msdn.microsoft.com/en-us/performance/cc825801.aspx
 
 It's a 2.5G download --running in the background as I write.
 
 But, do you know if it takes into account processes (exe's) started by the 
 process (exe) being profiled?

I know next to nothing about Windows development since I don't use Windows. I 
just googled for profiling windows and followed the links from the first 
result I got 
(http://stackoverflow.com/questions/67554/whats-the-best-free-c-profiler-for-windows-if-there-are).
 Besides, FPC on Windows does not start any other executables when compiling 
programs.

And note that even with a profiler you have to know what you should measure, 
what is relevant and what the results mean before you can draw any conclusions 
(just like with the disk benchmarking you did). If you don't know anything 
about that, read the manual/docs. For example, the profiling results I posted 
earlier to this list do not say anything about the influence of I/O since they 
were based on sampling the program code executing every 1 millisecond.


Jonas___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Dimitri Smits

- Mattias Gaertner nc-gaert...@netcologne.de schreef:

 On Sun, 12 Sep 2010 10:25:22 +0200 (CEST)
 mar...@stack.nl (Marco van de Voort) wrote:
 
  In our previous episode, Mattias Gaertner said:
Agreed. My opinion is that before we start to implement
 difficult and 
error-prone multi-threading into FPC we should find out why the
 hell Delphi 7 
can compile so much faster and produces even better code?
   
   Seeing that dcc is only 800K: 
   maybe it fits into the cpu cache.
  
  I assume dcc.exe uses more data than code :-)
 
 CPU caches do not work FIFO.
 If FPC does not fit into the CPU cache, then the CPU has to
 constantly
 load code mem additionally to the data.
 

in that case, can splitting up the .exe into .exe + more .dll's help?

aka a compiler package, a rtl, a assembler package, ...?

kind regards,
Dimitri Smits
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Florian Klämpfl
Am 12.09.2010 18:39, schrieb Martin Schreiber:
 On Sunday, 12. September 2010 18.29:34 Florian Klämpfl wrote:

 Please take it with humor. :-)

 As long as the compiler itself builds on a reasonable machine in less
 than 10 seconds, I'am happy :)
 
 Yup, I know. But there are people who use FPC for other tasks than compiling 
 FPC and there are people who have no reasonable machine. ;-)

Those can always use FPC 1.x or Delphi7 :) There is no free lunch ...
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Jonas Maebe

On 12 Sep 2010, at 19:15, Dimitri Smits wrote:

 - Mattias Gaertner nc-gaert...@netcologne.de schreef:
 
 CPU caches do not work FIFO.
 If FPC does not fit into the CPU cache, then the CPU has to
 constantly load code mem additionally to the data.
 
 in that case, can splitting up the .exe into .exe + more .dll's help?

Only the parts of the executable that are actually used (plus some surrounding 
bytes) are loaded into the caches. Splitting the used (or unused) code will not 
change that. You might want to read up on cpu caches: 
http://en.wikipedia.org/wiki/CPU_cache#Details_of_operation

Furthermore, splitting everything will make things worse, because then you get 
extra glue code and/or dynamic linker fix ups (i.e., it increases the code size 
and the program execution time).


Jonas___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Mattias Gaertner
On Sun, 12 Sep 2010 19:15:29 +0200 (CEST)
Dimitri Smits smi...@telenet.be wrote:

 
 - Mattias Gaertner nc-gaert...@netcologne.de schreef:
 
  On Sun, 12 Sep 2010 10:25:22 +0200 (CEST)
  mar...@stack.nl (Marco van de Voort) wrote:
  
   In our previous episode, Mattias Gaertner said:
 Agreed. My opinion is that before we start to implement
  difficult and 
 error-prone multi-threading into FPC we should find out why the
  hell Delphi 7 
 can compile so much faster and produces even better code?

Seeing that dcc is only 800K: 
maybe it fits into the cpu cache.
   
   I assume dcc.exe uses more data than code :-)
  
  CPU caches do not work FIFO.
  If FPC does not fit into the CPU cache, then the CPU has to
  constantly
  load code mem additionally to the data.
  
 
 in that case, can splitting up the .exe into .exe + more .dll's help?
 
 aka a compiler package, a rtl, a assembler package, ...?

No. Putting code into dll/so can even create slower code.
The cpu cache uses cache lines, so only the relevant parts of the
machine code is loaded, no matter if it is in a dll, exe, so or
whatever.

Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Adem

 On 9/12/2010 8:14 PM, Jonas Maebe wrote:

Besides, FPC on Windows does not start any other executables when compiling 
programs

You might be making a distinction (between compiling and building) here, 
but when I press 'rebuild lazarus' on that menu, here the list 
executables of executables called are below [numbers represent 'events'].


ppcx64.exe: 274,889
make.exe: 27,664
rm.exe: 17,968
fpc.exe: 8,992
gorc.exe: 6,718
lazarus.exe: 6,593
conhost.exe: 4,751
gdate.exe: 3,122
pwd.exe: 1,483
startlazarus.exe: 1,092
cmd.exe: 786
csrss.exe: 642

here, for example, the 'events' for make.exe:

QueryDirectory: 6,502
CreateFile: 5,800
CloseFile: 4,916
CreateFileMapping: 1,591
RegOpenKey: 1,483
QueryNameInformationFile: 727
QueryOpen: 675
RegCloseKey: 643
RegQueryValue: 591
QueryBasicInformationFile: 554
RegSetInfoKey: 552
ReadFile: 511
Load Image: 431
QueryStandardInformationFile: 367
RegQueryKey: 341
QueryFileInternalInformationFile: 318
SetBasicInformationFile: 318
QueryInformationVolume: 288
QueryAttributeInformationVolume: 276
QuerySecurityFile: 228
QueryAttributeTagFile: 205
FileSystemControl: 126
Process Create: 113
RegEnumKey: 26
RegEnumValue: 26
Process Exit: 13
Process Start: 13
Thread Create: 13
Thread Exit: 13
WriteFile: 2
SetDispositionInformationFile: 1
SetEndOfFileInformationFile: 1

I am not sure what all those do, but 'Load Image: 431' seems to mean 
'make.exe' is run 431 times.

And note that even with a profiler you have to know what you should measure, 
what is relevant and what the results mean before you can draw any conclusions 
(just like with the disk benchmarking you did). If you don't know anything 
about that, read the manual/docs. For example, the profiling results I posted 
earlier to this list do not say anything about the influence of I/O since they 
were based on sampling the program code executing every 1 millisecond.

ATM, I am mainly interested in finding out how many other external 
process Lazarus depends on (how many times does it call them) when 
compiling/building an exe. For that, a process explorer should be 
sufficient.

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-12 Thread Jonas Maebe

On 12 Sep 2010, at 23:01, Adem wrote:

 On 9/12/2010 8:14 PM, Jonas Maebe wrote:
 Besides, FPC on Windows does not start any other executables when compiling 
 programs
 
 You might be making a distinction (between compiling and building) here,
 but when I press 'rebuild lazarus' on that menu, here the list executables of 
 executables called are below [numbers represent 'events'].

That's indeed not FPC starting executables, that's Lazarus invoking make 
(which in turn invokes tons of other stuff).

 I am not sure what all those do, but 'Load Image: 431' seems to mean 
 'make.exe' is run 431 times.

make indeed works by recursively executing itself. And that is a known problem 
on Windows, because that platform is extremely slow at starting new processes 
for some reason. Some people have worked on alternatives (http://fastmake.org/, 
http://benjamin.smedbergs.us/pymake/), but afaik none of them can currently 
deal with everything that appears in FPC's makefiles.


Jonas___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Martin Schreiber
On Friday 10 September 2010 17:43:59 Adem wrote:
 Sometime ago, there was a brief mention of multi-threading FPC would be
 counter productive because compilation process was mostly disk IO bound
 --this is what I understood anyway.

 I wanted to check to see if disk IO was really limiting FPC/Lazarus
 compile performance.

Interesting is that Delphi 7 compiles about 10 times faster than FPC on the 
same machine.
http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg08029.html
Results with more code and FPC 2.4:
http://thread.gmane.org/gmane.comp.ide.mseide.user/18797
One would think Delphi and FPC need the same disk IO?

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Jonas Maebe

On 11 Sep 2010, at 08:55, Martin Schreiber wrote:

 Interesting is that Delphi 7 compiles about 10 times faster than FPC on the 
 same machine.
 http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg08029.html
 Results with more code and FPC 2.4:
 http://thread.gmane.org/gmane.comp.ide.mseide.user/18797
 One would think Delphi and FPC need the same disk IO?

First of all, they don't, unless Delphi's source/DCU searching and DCU loading 
logic is identical to FPC's.

Secondly, even *if* FPC (due to its design) is currently mainly limited in 
speed by I/O and *if* parellising would not help much because of that reason, 
then it can still also be slower than Delphi in other ways. Since Delphi 7 does 
not use parallel compilation (afaik), that's in fact a given.

So yes, FPC is slower than Delphi. Would parallelising FPC reduce the speed 
gap? Maybe (more likely for hot compiles), maybe not (more likely for cold 
compiles).


Jonas___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Martin Schreiber
On Saturday, 11. September 2010 11.32:38 Jonas Maebe wrote:

 So yes, FPC is slower than Delphi. Would parallelising FPC reduce the speed
 gap? 

Because the gap is so big I think not substantial.
Given that (please correct me if I am wrong):

- FPC bottleneck is disk IO and not compiler logic and calculation.
- Delphi is much faster than FPC.

- Delphi uses a file access approach which performs much better than the 
approach of FPC.

or it isn't true that FPC bottleneck is disk IO. Are we absolutely sure about 
the bottleneck?

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Juha Manninen (gmail)
On Saturday 11 September 2010 09:55:14 Martin Schreiber wrote:
 On Friday 10 September 2010 17:43:59 Adem wrote:
  Sometime ago, there was a brief mention of multi-threading FPC would be
  counter productive because compilation process was mostly disk IO bound
  --this is what I understood anyway.
  
  I wanted to check to see if disk IO was really limiting FPC/Lazarus
  compile performance.
 
 Interesting is that Delphi 7 compiles about 10 times faster than FPC on the
 same machine.
 http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg08029.html
 Results with more code and FPC 2.4:
 http://thread.gmane.org/gmane.comp.ide.mseide.user/18797
 One would think Delphi and FPC need the same disk IO?

I read the threads. My guess is also that the slowness comes from searching 
and writing many files in big directory structures. It is slow even if the 
files 
are cached. Also starting a new process is slow.
These OS kernel tasks are difficult to measure and process monitors don't give 
reliable results.

Suggestion:
Create an API for integrating FPC with IDEs and special make programs.
The API would pass info about exact file names and locations.
It could also pass the whole source memory buffers.

Then build FPC as a dynamic shared library. There would be 2 FPC binaries 
then: the traditional executable, and a shared library to be called from 
external programs.

For example Lazarus IDE already scans a lot of information about project's 
files and directories. That info could be easily passed to compiler.
Codetools in Lazarus already parses lots of code. The whole parsed interface 
section could be passed to compiler (symbol table and whatnot).
... but that is the next step, let's stick with file info now...

Then there would be a new dedicated build program which reads all project info 
first and then calls the compile function (not a separate process) in the 
shared lib for each source file.

No expensive process startups, no searching for include files again for each 
source file from huge directory structures.
I bet it would make a BIG difference in speed.
Delphi must be doing something like this although I don't know details.
After that it makes sense to make the compiler multi-threaded. It could scale 
almost linearly with CPU cores (maybe).

I haven't seen such ideas in these mailing lists. Is it possible I am the first 
one to have it? I don't believe because the idea is so obvious.
If there is already such development with the new make tools then sorry for my 
ignorance.


Juha
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Jonas Maebe

On 11 Sep 2010, at 12:23, Martin Schreiber wrote:

 or it isn't true that FPC bottleneck is disk IO. Are we absolutely sure about 
 the bottleneck?

I'm quite certain that there are many reasons that FPC compiles more slowly 
than Delphi. The bottlenecks probably also vary from platform to platform and 
from compilation scenario to compilation scenario. I'm quite sure there is not 
a single thing you can fix and suddenly get compilation speeds in the same 
ballpark as Delphi 7 (how does the compilation speed of current Delphi's 
compare to Delphi 7 btw? I read that Delphi XE compiles much faster than Delphi 
2010 in some cases, but I did not see comparisons to Delphi 7).

One thing that could be done is adding a linear scan register allocator (it 
would result in slightly worse code than the current register colouring, but it 
executes more quickly).

Going further, general restructuring of the compiler for compilation speed 
reasons would only be acceptable if it does not negatively impact the 
maintainability. There is a reason why we can support 6 architectures and 
umpteen OSes in the compiler with only a handful of people.


Jonas___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Martin Schreiber
On Saturday, 11. September 2010 12.25:14 Juha Manninen (gmail) wrote:
  One would think Delphi and FPC need the same disk IO?

 I read the threads. My guess is also that the slowness comes from searching
 and writing many files in big directory structures. It is slow even if the
 files are cached. Also starting a new process is slow.
 These OS kernel tasks are difficult to measure and process monitors don't
 give reliable results.

 Suggestion:
 Create an API for integrating FPC with IDEs and special make programs.
 The API would pass info about exact file names and locations.
 It could also pass the whole source memory buffers.

And why does the Delphi commandline compiler (dcc32) not need this IDE 
assistance?

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Juha Manninen (gmail)
On Saturday 11 September 2010 13:40:26 Martin Schreiber wrote:
 And why does the Delphi commandline compiler (dcc32) not need this IDE
 assistance?

My guess is that dcc32 works as an integrated make program + compiler and thus 
doesn't start external processes for each file.
Or, if it starts an external process then it can use some (hidden) temporary 
file with pre-scanned info of the project. So the compiler would only open one 
info file instead of scanning the whole search paths.

I used the IDE always when working with Delphi and don't really know dcc32. 
Guessing only.

Juha
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Sergei Gorelkin

Jonas Maebe пишет:

On 11 Sep 2010, at 12:23, Martin Schreiber wrote:

or it isn't true that FPC bottleneck is disk IO. Are we absolutely sure about 
the bottleneck?


I'm quite certain that there are many reasons that FPC compiles more slowly than Delphi. 
The bottlenecks probably also vary from platform to platform and from compilation 
scenario to compilation scenario. I'm quite sure there is not a single thing you can 
fix and suddenly get compilation speeds in the same ballpark as Delphi 7 (how 
does the compilation speed of current Delphi's compare to Delphi 7 btw? I read that 
Delphi XE compiles much faster than Delphi 2010 in some cases, but I did not see 
comparisons to Delphi 7).

One thing that could be done is adding a linear scan register allocator (it 
would result in slightly worse code than the current register colouring, but it 
executes more quickly).

Going further, general restructuring of the compiler for compilation speed 
reasons would only be acceptable if it does not negatively impact the 
maintainability. There is a reason why we can support 6 architectures and 
umpteen OSes in the compiler with only a handful of people.


One idea that comes at this point is to put PPU data directly into object files, so the number of 
output files is reduced plain twice. The PPU data could be placed into a section that is ignored by 
linker. However I don't know is this is possible for all platforms.


Regards,
Sergei
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Michael Van Canneyt



On Sat, 11 Sep 2010, Juha Manninen (gmail) wrote:


On Saturday 11 September 2010 13:40:26 Martin Schreiber wrote:

And why does the Delphi commandline compiler (dcc32) not need this IDE
assistance?


My guess is that dcc32 works as an integrated make program + compiler and thus
doesn't start external processes for each file.


No it does not. 
dcc32 compiles 1 file only, but does compile any additional units it needs.


You'll need makefiles as well if you use dcc32 (or any other build tool).

I have an extended build system using dcc32, and it takes easily up to 15
minutes to compile a 1.5 million lines project.

Michael.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Michael Van Canneyt



On Sat, 11 Sep 2010, Sergei Gorelkin wrote:


Jonas Maebe пишет:

On 11 Sep 2010, at 12:23, Martin Schreiber wrote:

or it isn't true that FPC bottleneck is disk IO. Are we absolutely sure 
about the bottleneck?


I'm quite certain that there are many reasons that FPC compiles more slowly 
than Delphi. The bottlenecks probably also vary from platform to platform 
and from compilation scenario to compilation scenario. I'm quite sure there 
is not a single thing you can fix and suddenly get compilation speeds in 
the same ballpark as Delphi 7 (how does the compilation speed of current 
Delphi's compare to Delphi 7 btw? I read that Delphi XE compiles much 
faster than Delphi 2010 in some cases, but I did not see comparisons to 
Delphi 7).


One thing that could be done is adding a linear scan register allocator (it 
would result in slightly worse code than the current register colouring, 
but it executes more quickly).


Going further, general restructuring of the compiler for compilation speed 
reasons would only be acceptable if it does not negatively impact the 
maintainability. There is a reason why we can support 6 architectures and 
umpteen OSes in the compiler with only a handful of people.



One idea that comes at this point is to put PPU data directly into object 
files, so the number of output files is reduced plain twice. The PPU data 
could be placed into a section that is ignored by linker. However I don't 
know is this is possible for all platforms.


Not each platform may support this, so if you want to go down this path, 
why not do the opposite, as Delphi ? Putting the .o data at the end allows 
the compiler to simply not read that data when reading the PPU. 
Only when actually linking do you need to extract all .o data from the ppus.


Michael.___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Mattias Gaertner
On Sat, 11 Sep 2010 16:02:52 +0200 (CEST)
Michael Van Canneyt mich...@freepascal.org wrote:

 
 
 On Sat, 11 Sep 2010, Juha Manninen (gmail) wrote:
 
  On Saturday 11 September 2010 13:40:26 Martin Schreiber wrote:
  And why does the Delphi commandline compiler (dcc32) not need this IDE
  assistance?
 
  My guess is that dcc32 works as an integrated make program + compiler and 
  thus
  doesn't start external processes for each file.
 
 No it does not. 
 dcc32 compiles 1 file only, but does compile any additional units it needs.
 
 You'll need makefiles as well if you use dcc32 (or any other build tool).
 
 I have an extended build system using dcc32, and it takes easily up to 15
 minutes to compile a 1.5 million lines project.

Maybe dcc32 likes the MSEgui sources.

Martin, can you give a comparison between win32 and Linux 32?


Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Sergei Gorelkin

Michael Van Canneyt пишет:



One idea that comes at this point is to put PPU data directly into 
object files, so the number of output files is reduced plain twice. 
The PPU data could be placed into a section that is ignored by linker. 
However I don't know is this is possible for all platforms.


Not each platform may support this, so if you want to go down this path, 
why not do the opposite, as Delphi ? Putting the .o data at the end 
allows the compiler to simply not read that data when reading the PPU. 
Only when actually linking do you need to extract all .o data from the 
ppus.


That is an option, too. But given that linking typically takes place at every build (as opposed to 
compiling), I am afraid that total number of file operations on platforms without builtin linker 
won't be noticeably less than it is currently.



Regards,
Sergei
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Graeme Geldenhuys
On 11 September 2010 16:12, Mattias Gaertner wrote:

 Martin, can you give a comparison between win32 and Linux 32?

Add to that Martin, I know MSEgui is compilable with FPC and
Delphi. Is MSEgui compilable with Kylix 3 too? Then one could do a
Delphi 7 vs Kylix 3 comparison as well - seeing that Kylix 3 is
supposed to be on-par with Delphi 7.  I've got Kylix 3 Enterprise
running in a VM here, I also have Delphi 7 available in a VM.

-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Martin Schreiber
On Saturday, 11. September 2010 18.37:49 Graeme Geldenhuys wrote:
 On 11 September 2010 16:12, Mattias Gaertner wrote:
  Martin, can you give a comparison between win32 and Linux 32?

 Add to that Martin, I know MSEgui is compilable with FPC and
 Delphi. Is MSEgui compilable with Kylix 3 too?

In theory yes, I did not test recently. Probably some modifications in Linux 
MSEgui are necessary now.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Martin Schreiber
On Saturday, 11. September 2010 16.12:10 Mattias Gaertner wrote:

 Maybe dcc32 likes the MSEgui sources.

Or maybe FPC does not like MSEgui sources. ;-)

 Martin, can you give a comparison between win32 and Linux 32?

I don't have a working Kylix 3 environment at the moment. IIRC dcc32 on Linux 
and Windows had about the same compiling performance.

Building MSEide with FPC fixes_2_4, about 366'000 lines:
Windows NTFS:
smartlink 50.4 sec
no smartlink 50.5 sec

Same machine Linux reiser FS:
smartlink 126 sec
no smartlink 62.7 sec.

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Florian Klämpfl
Am 11.09.2010 19:50, schrieb Martin Schreiber:
 On Saturday, 11. September 2010 16.12:10 Mattias Gaertner wrote:

 Maybe dcc32 likes the MSEgui sources.

 Or maybe FPC does not like MSEgui sources. ;-)
 
 Martin, can you give a comparison between win32 and Linux 32?

 I don't have a working Kylix 3 environment at the moment. IIRC dcc32 on Linux 
 and Windows had about the same compiling performance.
 
 Building MSEide with FPC fixes_2_4, about 366'000 lines:
 Windows NTFS:
 smartlink 50.4 sec
 no smartlink 50.5 sec

What machine? Because with hot disk cache, I just build MSEide in about
10 s (15 s cold) on W7 64 Bit:

...
Linking mseidefp.exe
308574 lines compiled, 10.6 sec , 2577952 bytes code, 1618920 bytes data
196 warning(s) issued
414 note(s) issued
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Mattias Gaertner
On Sat, 11 Sep 2010 19:50:41 +0200
Martin Schreiber mse00...@gmail.com wrote:

 On Saturday, 11. September 2010 16.12:10 Mattias Gaertner wrote:
 
  Maybe dcc32 likes the MSEgui sources.
 
 Or maybe FPC does not like MSEgui sources. ;-)
 
  Martin, can you give a comparison between win32 and Linux 32?
 
 I don't have a working Kylix 3 environment at the moment. IIRC dcc32 on Linux 
 and Windows had about the same compiling performance.
 
 Building MSEide with FPC fixes_2_4, about 366'000 lines:
 Windows NTFS:
 smartlink 50.4 sec
 no smartlink 50.5 sec
 
 Same machine Linux reiser FS:
 smartlink 126 sec
 no smartlink 62.7 sec.

Strange.
Here fpc is much faster, even on a 3 year old linux machine:

343520 lines compiled, 16.2 sec 
258 warning(s) issued
570 note(s) issued

real0m16.249s
user0m14.953s
sys 0m0.904s

time /usr/lib/fpc/2.4.0/ppc386 -omseide -Fu...msegui/lib/addon/*/ 
-Fi...msegui/lib/addon/*/ -Fu...msegui/lib/common/kernel/i386-linux/ 
-Fu...msegui/lib/common/kernel/ -Fi...msegui/lib/common/kernel/ 
-Fu...msegui/lib/common/*/ -l -Mobjfpc -Sh -gl -O- mseide.pas


Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Martin Schreiber
On Saturday 11 September 2010 20:27:46 Florian Klämpfl wrote:

 What machine? Because with hot disk cache, I just build MSEide in about
 10 s (15 s cold) on W7 64 Bit:

The same as for all other tests,
win2000, AMD Athlon XP 3000+, 1GB RAM

 ...
 Linking mseidefp.exe
 308574 lines compiled, 10.6 sec , 2577952 bytes code, 1618920 bytes data

Then Delphi7 probably uses about 1 sec on your machine. ;-)

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Florian Klämpfl
Am 11.09.2010 20:50, schrieb Martin Schreiber:
 On Saturday 11 September 2010 20:27:46 Florian Klämpfl wrote:

 What machine? Because with hot disk cache, I just build MSEide in about
 10 s (15 s cold) on W7 64 Bit:

 The same as for all other tests,
 win2000, AMD Athlon XP 3000+, 1GB RAM
 
 ...
 Linking mseidefp.exe
 308574 lines compiled, 10.6 sec , 2577952 bytes code, 1618920 bytes data
 
 Then Delphi7 probably uses about 1 sec on your machine. ;-)
 

...
mseide.pas(63)
280491 lines, 2.18 seconds, 2110568 bytes code, 752073 bytes data.

Anyways, before this ends in an endless discussion: if anybody is
interested in improving FPC compilation speed (for my needs is
sufficient) and have a look at fillchar and, have a look at FPC's unit
loading algorithm (not the actual i/o itself but how all the symbols
classes etc. are restored). This two points are bottlenecks which might
help when they are improved, though it's pretty unlikely that this will
improve things more than a few percent.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Florian Klämpfl
Am 11.09.2010 22:04, schrieb Jonas Maebe:
 
 On 11 Sep 2010, at 21:10, Florian Klämpfl wrote:
 
 Anyways, before this ends in an endless discussion: if anybody is 
 interested in improving FPC compilation speed (for my needs is 
 sufficient) and have a look at fillchar and, have a look at FPC's
 unit loading algorithm (not the actual i/o itself but how all the
 symbols classes etc. are restored). This two points are bottlenecks
 which might help when they are improved, though it's pretty
 unlikely that this will improve things more than a few percent.
 
 Note that in 2.5.1 it will already be somewhat faster because of
 r15604 (removes a lot of calls to fillchar from the register
 allocator) and r15515 (optimizations to sysfreemem_fixed).

True, but at least I fear the generic i386 fillchar is slower than the
MacOSX system routine __bzero. But I agree also that there won't be a
big improvement by improving fillchar.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Graeme Geldenhuys
On 11 September 2010 21:10, Florian Klämpfl flor...@freepascal.org wrote:

FPC
---
 Linking mseidefp.exe
 308574 lines compiled, 10.6 sec , 2577952 bytes code, 1618920 bytes data


Delphi
-
 mseide.pas(63)
 280491 lines, 2.18 seconds, 2110568 bytes code, 752073 bytes data.


Now this is weird!  Anybody else spotted the difference?  Delphi seems
to compile +-28000 lines less that FPC! Florian, I presume it's the
same machine with the same MSEgui source code revision? What would be
the reason for that?

Would that (lines compiled) also account for the huge difference in
bytes data. What is the final executable size (not that this matters
much to me) generated by the two compilers?

Martin, do you get the same results for lines compiled?


-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Dimitri Smits

- Martin Schreiber mse00...@gmail.com schreef:

 On Saturday, 11. September 2010 12.25:14 Juha Manninen (gmail) wrote:
   One would think Delphi and FPC need the same disk IO?
 
  I read the threads. My guess is also that the slowness comes from
 searching
  and writing many files in big directory structures. It is slow even
 if the
  files are cached. Also starting a new process is slow.
  These OS kernel tasks are difficult to measure and process monitors
 don't
  give reliable results.
 
  Suggestion:
  Create an API for integrating FPC with IDEs and special make
 programs.
  The API would pass info about exact file names and locations.
  It could also pass the whole source memory buffers.
 
 And why does the Delphi commandline compiler (dcc32) not need this IDE
 
 assistance?
 

it does. Delphi IDE passes extra assumptions/directories that the commandline 
tool does not know about (for instance $(DELPHI)/Projects/Bpl). 

Juha's idea is the way Borland did it with D7. That is: a shared lib with api  
a commandline tool. Unfortunately the DLL is not linked from the cmdline tool, 
so it seems those 2 are separately compiled and statically linked with 'the 
same' code.

Donno for sure if that still is the case with 2010/XE.

kind regards,
Dimitri Smits
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Dimitri Smits

- Juha Manninen (gmail) juha.mannine...@gmail.com schreef:

 On Saturday 11 September 2010 13:40:26 Martin Schreiber wrote:
  And why does the Delphi commandline compiler (dcc32) not need this
 IDE
  assistance?
 
 My guess is that dcc32 works as an integrated make program + compiler
 and thus 
 doesn't start external processes for each file.
 Or, if it starts an external process then it can use some (hidden)
 temporary 
 file with pre-scanned info of the project. So the compiler would only
 open one 
 info file instead of scanning the whole search paths.
 
 I used the IDE always when working with Delphi and don't really know
 dcc32. 
 Guessing only.
 

no, the bin directory of D7 is filled with extra .exe's (tlib, tasm32, ...) and 
the dcc32.exe is  800K big.

kind regards,
Dimitri Smits 
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Dimitri Smits

- Michael Van Canneyt mich...@freepascal.org schreef:

 On Sat, 11 Sep 2010, Juha Manninen (gmail) wrote:
 
  On Saturday 11 September 2010 13:40:26 Martin Schreiber wrote:
  And why does the Delphi commandline compiler (dcc32) not need this
 IDE
  assistance?
 
  My guess is that dcc32 works as an integrated make program +
 compiler and thus
  doesn't start external processes for each file.
 
 No it does not. 
 dcc32 compiles 1 file only, but does compile any additional units it
 needs.
 
 You'll need makefiles as well if you use dcc32 (or any other build
 tool).
 
 I have an extended build system using dcc32, and it takes easily up to
 15
 minutes to compile a 1.5 million lines project.
 

that is my experience as well. (using 'want.exe' aka windows 'ant')

it implies though that you can merely compile your .dpk and .dpr files to get 
everything compiled ;-)

kind regards,
Dimitri Smits
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Dimitri Smits

- Florian Klämpfl flor...@freepascal.org schreef:

 Am 11.09.2010 20:50, schrieb Martin Schreiber:
  On Saturday 11 September 2010 20:27:46 Florian Klämpfl wrote:
 
  What machine? Because with hot disk cache, I just build MSEide in
 about
  10 s (15 s cold) on W7 64 Bit:
 
  The same as for all other tests,
  win2000, AMD Athlon XP 3000+, 1GB RAM
  
  ...
  Linking mseidefp.exe
  308574 lines compiled, 10.6 sec , 2577952 bytes code, 1618920 bytes
 data
  
  Then Delphi7 probably uses about 1 sec on your machine. ;-)
  
 
 ...
 mseide.pas(63)
 280491 lines, 2.18 seconds, 2110568 bytes code, 752073 bytes data.
 
 Anyways, before this ends in an endless discussion: if anybody is
 interested in improving FPC compilation speed (for my needs is
 sufficient) and have a look at fillchar and, have a look at FPC's
 unit
 loading algorithm (not the actual i/o itself but how all the symbols
 classes etc. are restored). This two points are bottlenecks which
 might
 help when they are improved, though it's pretty unlikely that this
 will
 improve things more than a few percent.

I am not as intimately familiar with the compiler internals as most of you, but 
the writing/reading of those .ppu files looks to me like a prime candidate.

other things that I do not know if they are already implemented this way:
1) if the .ppu/.ppl + .o are merged and they give better results then why not 
go the extra mile and implement packages (or at least the .dcp)? Although I 
fail to see why (except for linking) both have to be opened during compilation. 
With packages I mean the .dcp/.bpl equivalent.
It is my understanding that .dcp is like a .lib + all the stuff in the unitinfo 
(.dcu / .ppu), and ofcourse the more simpler .bpl = .dll with extra's. That 
way, you need to dive into much less files.

2) when fetching / checking the files, does it happen according to the unit  
include paths and those paths appended by targetos en architectures for every 
file?
Wouldn't it be more efficient to fetch/cache some dir-info first for all files 
in those dirs and then fetch the necessary info from there? For instance: get 
filenames + size + timestamp for all .ppu files or .pas files first.

3) It is my understanding that the .ppu structures are written with 
size-footprint in mind, not processing efficiency? Maybe the structures need 
changing too?

kind regards,
Dimitri Smits


___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Martin Schreiber
On Sunday, 12. September 2010 01.31:43 Dimitri Smits wrote:
 
  And why does the Delphi commandline compiler (dcc32) not need this IDE
 
  assistance?

 it does. Delphi IDE passes extra assumptions/directories that the
 commandline tool does not know about (for instance $(DELPHI)/Projects/Bpl).

The comparisons I made were with dcc32 not Delphi IDE. The commands are 
here:
http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg08029.html

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-11 Thread Martin Schreiber
On Saturday, 11. September 2010 21.10:20 Florian Klämpfl wrote:

 Anyways, before this ends in an endless discussion: if anybody is
 interested in improving FPC compilation speed (for my needs is
 sufficient) and have a look at fillchar and, have a look at FPC's unit
 loading algorithm (not the actual i/o itself but how all the symbols
 classes etc. are restored). This two points are bottlenecks which might
 help when they are improved, though it's pretty unlikely that this will
 improve things more than a few percent.

Agreed. My opinion is that before we start to implement difficult and 
error-prone multi-threading into FPC we should find out why the hell Delphi 7 
can compile so much faster and produces even better code?

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-10 Thread Jonas Maebe


On 10 Sep 2010, at 17:43, Adem wrote:


   SSD: 103,060 ms (1 min 43 sec)
RAMDisk: 105,463 ms (1 min 46 sec)

This doesn't make sense. FPC/Lazarus compiles on the faster medium  
longer (albeit only 3 sec.).


Everything on your SSD is cached in RAM, so it's normal that both are  
about the same speed. The 3 seconds difference is probably just noise.



Jonas
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-10 Thread Adem

 On 2010-09-10 06:54 PM, Jonas Maebe wrote:


On 10 Sep 2010, at 17:43, Adem wrote:


   SSD: 103,060 ms (1 min 43 sec)
RAMDisk: 105,463 ms (1 min 46 sec)

This doesn't make sense. FPC/Lazarus compiles on the faster medium 
longer (albeit only 3 sec.).
I am sorry, but what you've just said doesn't make sense either --at 
least to me.


If that were true, wouldn't the disk benchmarks be also the same?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-10 Thread Jonas Maebe


On 10 Sep 2010, at 18:05, Adem wrote:


On 2010-09-10 06:54 PM, Jonas Maebe wrote:


On 10 Sep 2010, at 17:43, Adem wrote:


  SSD: 103,060 ms (1 min 43 sec)
RAMDisk: 105,463 ms (1 min 46 sec)

This doesn't make sense. FPC/Lazarus compiles on the faster medium  
longer (albeit only 3 sec.).
Everything on your SSD is cached in RAM, so it's normal that both  
are about the same speed. The 3 seconds difference is probably just  
noise.
I am sorry, but what you've just said doesn't make sense either --at  
least to me.


If that were true, wouldn't the disk benchmarks be also the same?


No. Disk benchmark specifically disable OS disk in order to be able to  
measure the actual disk throughput rather than just the speed of the  
OS buffer cache.



Jonas

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-10 Thread Marco van de Voort
In our previous episode, Adem said:
 I wanted to check to see if disk IO was really limiting FPC/Lazarus 
 compile performance.

 The only quick way I could devise to check this was to use two different 
 disks which are significantly different from one another in terms of 
 performance.

Both are atypical memory technology based devices which probably have low
seeking times, compared to harddisks.

With the total FPC/Lazarus sources being in the magnitude of 100MB, this
should be logical, doing this test that is based on raw read/write
bandwidth is not even necessary to determine that the bottleneck is not raw
read/write performance.

This becasue even a harddisk budget model still does 80-100MB/s, and overall
building is not in the 1-2s magnitude.

The I/O bottleneck is thus more the searching and opening of files, as well,
on Windows, executing programs. (.exe's)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-10 Thread Daniel
AFAIR the ATTO tool measures read and write bursts of single files X in size.

An interesting exercise is to transfer 1000 files to a USB memory
stick in 2 situations:

- Compacted in a single file, transfers at or near full USB speed.
- Spread out normally on a folder takes forever.

This happens because the time it takes to SWITCH between one file to
another is significant. Ending one operation (a single file transfer)
and begining another takes a time slice. Summing up all these start
and finish ops takes a significant time slice.

I guess this kind of behaviour is exactly what you're noticing. The
raw I/O transfer speed provided by your SSD is enough for your needs.
A faster channel won't help you. Some other factor is accounting for
your lags.

You can't really judge a storage medium just by a single benchmark
kind. The compiler will ask for your storage medium to seek files all
the time. Switching files all the time. This takes time. Much more
time than blazing through a single file like ATTO does.

On Fri, Sep 10, 2010 at 12:54, Jonas Maebe jonas.ma...@elis.ugent.be wrote:

 On 10 Sep 2010, at 17:43, Adem wrote:

   SSD: 103,060 ms (1 min 43 sec)
 RAMDisk: 105,463 ms (1 min 46 sec)

 This doesn't make sense. FPC/Lazarus compiles on the faster medium longer
 (albeit only 3 sec.).

 Everything on your SSD is cached in RAM, so it's normal that both are about
 the same speed. The 3 seconds difference is probably just noise.


 Jonas
 ___
 fpc-devel maillist  -  fpc-de...@lists.freepascal.org
 http://lists.freepascal.org/mailman/listinfo/fpc-devel

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel


Re: [fpc-devel] FPC/Lazarus Rebuild performance

2010-09-10 Thread Adem

 On 2010-09-10 07:16 PM, Daniel wrote:

This happens because the time it takes to SWITCH between one file to
another is significant. Ending one operation (a single file transfer)
and begining another takes a time slice. Summing up all these start
and finish ops takes a significant time slice.
I wonder if all this means compilation/build speeds cannot be improved 
much by introducing faster hardware.


To see what's happening in the bigger picture, I used 'Process Monitor' 
[ http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx ] and 
filtered out those I believed to be irrelevant to FPC/Lazarus build 
operation.


Here are the results I see on Process Monitor (An 'event' is 
Create/Open/Read/Write/Delete etc.).


I am pasting summaries below hoping it helps someone identify a 
bottleneck or something.


*Count of occurences by EventClass:*
File System: 324,958
Registry: 25,165
Process: 4,577

*Count of occurences by ProcessName: (Count: 12)*
ppcx64.exe: 275,003
make.exe: 27,825
rm.exe: 17,973
fpc.exe: 9,110
lazarus.exe: 6,742
gorc.exe: 6,719
conhost.exe: 5,034
gdate.exe: 3,122
pwd.exe: 1,483
startlazarus.exe: 1,102
csrss.exe: 946
cmd.exe: 786

*Count of occurences by Result:*
SUCCESS: 300,998
NO SUCH FILE: 26,919
NAME NOT FOUND: 15,446
BUFFER OVERFLOW: 3,110
REPARSE: 2,846
FILE LOCKED WITH ONLY READERS: 2,469
NO MORE FILES: 2,308
NAME INVALID: 667
PATH NOT FOUND: 538
NO MORE ENTRIES: 327
END OF FILE: 213
INVALID PARAMETER: 32
NOT REPARSE POINT: 10
NAME COLLISION: 7
IS DIRECTORY: 5

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel