Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Seth Hillbrand]

On 11/5/19 9:59 AM, Simon Richter wrote:

Hi,

I've made another quick run for Release builds:

 MakeNinja
Scripting ON5:19.66 5:03.91
 3336%   3565%
Scripting OFF   3:33.84 3:08.50
 4947%   5684%

That is with -j64 on 2 socket T2P9D01.

You can see that dependency generation makes about 15 seconds difference
(the CMake generator for Makefiles has that as a separate step, while the
generator for Ninja knows how to output dependency information during the
first build).


Wow.  I have nowhere near your horsepower but still run a 12-core system 
and don't see anywhere near this discrepancy.Must be in the extended 
parallelism.


For me,
    Make    Ninja
Scripting ON    8:04.92 7:25.43
    1043%   1127%
Scripting OFF   7:33.23 7:15.09
    1090%   1129%

That is to say, scripting adds about 10-30 seconds on a full build.  Do 
you see a difference between running '-j64' and '-j ' on 
the POWER9?


-S
--
KiCad Services Corporation KiCad Services Corporation Logo
Seth Hillbrand
*Lead Developer*
+1-530-302-5483‬ 
Davis, CA
www.kipro-pcb.com  i...@kipro-pcb.com 

https://twitter.com/KiProEDA  
https://www.linkedin.com/company/kicad 



___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Simon Richter]

Hi,

I've made another quick run for Release builds:

MakeNinja
Scripting ON5:19.66 5:03.91
3336%   3565%
Scripting OFF   3:33.84 3:08.50
4947%   5684%

That is with -j64 on 2 socket T2P9D01.

You can see that dependency generation makes about 15 seconds difference
(the CMake generator for Makefiles has that as a separate step, while the
generator for Ninja knows how to output dependency information during the
first build).

The Python scripting makes a huge difference, because the Swig header needs
to be generated before we can start on the pcbnew subdirectory, and both
this generator and compiling the wrappers take ages, so there is less
parallelism.

Even a factor of 56.84 is quite some distance from 64, so there may be some
potential left.

   Simon

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Tomasz Wlostowski]

On 30/10/2019 17:37, Seth Hillbrand wrote:
> 
> Two things can help there.  First, using the gold linker (if you are not
> already) and second, using Ninja.  The gold linker is substantially
> faster than the BFD linker. 

I can confirm the above, way faster!

> And Ninja is smarter than make about
> parallelizing actions.  Cmake will happily generate the Ninja files for
> you with -GNinja.

Didn't know about Ninja, thanks Seth!

Tom

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Seth Hillbrand]

On 2019-10-30 08:20, Simon Richter wrote:

That is more than 1100% CPU usage, with -j12, so very close to full 
usage.


How is that even possible, don't you have that two minute phase at the 
end
of building pcbnew_kiface where it's just building pcbnew_wrap.cxx.o 
and

everything else is done?

   Simon


Two things can help there.  First, using the gold linker (if you are not 
already) and second, using Ninja.  The gold linker is substantially 
faster than the BFD linker.  And Ninja is smarter than make about 
parallelizing actions.  Cmake will happily generate the Ninja files for 
you with -GNinja.


Here are the results for my testing with all scripting enabled (Debian 
Buster)


seth@cpu1 (master) % time ninja

 4050.50s user
 291.35s system
 1137% cpu
 6:21.85 total

Best-
Seth

Seth Hillbrand
KiCad Services Corporation
https://www.kipro-pcb.com
+1 530 302 5483 | +1 212 603 9372

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Tomasz Wlostowski]

On 30/10/2019 16:20, Simon Richter wrote:
> Hi,
> 
> On Tue, Oct 29, 2019 at 07:00:48PM +0100, Tomasz Wlostowski wrote:
> 
>> $ make -j12
> 
>> real 7m59.758s
>> user 86m44.231s
>> sys  5m9.724s
> 
> That is more than 1100% CPU usage, with -j12, so very close to full usage.
> 
> How is that even possible, don't you have that two minute phase at the end
> of building pcbnew_kiface where it's just building pcbnew_wrap.cxx.o and
> everything else is done?
> 

IIRC pcbnew_wrap.cxx is only generated when Kicad is built with Python
support. I never build it with Python enabled ;-).

Tom

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Simon Richter]

Hi,

On Tue, Oct 29, 2019 at 07:00:48PM +0100, Tomasz Wlostowski wrote:

> $ make -j12

> real  7m59.758s
> user  86m44.231s
> sys   5m9.724s

That is more than 1100% CPU usage, with -j12, so very close to full usage.

How is that even possible, don't you have that two minute phase at the end
of building pcbnew_kiface where it's just building pcbnew_wrap.cxx.o and
everything else is done?

   Simon

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Tim Hawkins]

I would have thought that any discussion of performance would have to take
the speed of the diskio into account

executing: (if you get command not found then install the sysstat package)

$iostat -h -x -d 2

whilst compiling would show the diskio utilization factor (last column), if
it is pegged at 100% then your compilation process is io bound and not cpu
bound, if it is less than 100% then you are cpu bound.

SSDs and disk oops would probably have an impact on compilation speed,
especially when searching deep include trees.

On Tue, Oct 29, 2019 at 1:51 PM Henner Zeller  wrote:

>
>
> On Mon, 28 Oct 2019 at 22:21, Andrew Lutsenko 
> wrote:
>
>> Hi all,
>>
>> This information may be useful to people that are considering an upgrade
>> for their workstations or for general information, feel free to skip this
>> email if you are not interested.
>>
>> I've been using my 6 year old Haswell i7 machine that served me well over
>> the years but kicad compilation times both under msys and in linux vm were
>> frustratingly long. That reason along with a few others pushed me to
>> upgrade.
>> If you've been following the hardware news lately AMD's Ryzen is all the
>> rage lately and for a very good reason. I've been rooting for them since
>> Athlon days but had to begrudgingly buy Intel a few times after that
>> because AMD products just weren't competitive.
>> This definitely has changed with Ryzen series.
>>
>> To not make this too long and so that you don't think of me as AMD shill
>> let the numbers speak for themselves and you decide whether it's worth it
>> to upgrade for you.
>>
>> My previous config: i7 4770 Haswell 4 core CPU, 16gb of DDR3 ram in dual
>> channel, sata ssd.
>> My new config: R9 3900x Matisse 12 core CPU, 32gb of DDR4 ram in dual
>> channel, nvme ssd.
>>
>> I set my master branch to same commit in both cases (846397c69) and did
>> "make clean". Then I compiled all targets to measure time of clean build.
>> Then switched to master~15 and compiled, then switched to master~5 and
>> compiled, and lastly back to master and compiled again. Thus I measured
>> more typical scenario of incremental compilation as well.
>> In case of my old system I did "make -j8" as that is the number of
>> threads that CPU has, in case of new one I did "make -j24" for same reason,
>> maximizing used threads.
>>
>
> For compilation uses, I recently upgraded to a 16 core 1950x and am also
> pretty impressed.
> I find that using a bit more parallelism than can even give a little more
> as not all compile threads might be using the CPU 100% but might be waiting
> on IO. On my 1950X, I use -j50 (on 32 hyper-threaded cores). Something like
> this might in particular work for your network storage connection.
>
> Also highly recommended (when in development with intermediate 'make
> clean') install ccache
>
> Also something of note is that for both configs linux vm was run from a
>> virtual disk located on a network storage device with 1Gbps link, not on
>> local ssd. And I didn't master~15 build for linux, just master~5 and master
>> again.
>>
>> clean master~15 master~5 master
>> i7 4770 msys 36m28.515s 26m51.178s 27m45.738s 14m21.653s
>> R9 3900x msys 6m40.112s 4m56.747s 4m55.271s 3m15.945s
>> i7 4770 linux 44m49.681s - 15m53.635s 15m53.098s
>> R9 3900x linux 6m35.732s - 3m26.773s 3m18.514s
>>
>> I kinda expected 2x maybe 3x decrease because not all computations scale
>> linearly with number of threads. I was pleasantly surprised by almost 6x
>> decrease in clean build time and 5x in incremental builds.
>>
>> Here are the numbers in google sheet in case table above is not readable
>>
>> https://docs.google.com/spreadsheets/d/1UI1aW-FTTI-TncTWhAdG_Ag__5nc2FF1626WaPiL3LE/edit?usp=sharing
>>
>> Regards,
>> Andrew
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Tomasz Wlostowski]

On 29/10/2019 15:40, Simon Richter wrote:
> We could probably shave off another two or three minutes of build time if
> we could make sure that we always make progress on the critical path. The
> dependency generation as a side effect pulls all the sources and headers
> into cache, which reduces the effects of I/O latency a bit during
> compilation, so parallelizing with more than the number of threads you
> actually have is probably counterproductive.

Hi,

Another idea: use precompiled headers. I experimented some years ago
with cotire for CMake on KiCad sources and it gave quite promising
results. Since I didn't maintain this code it's probably useless by now
given how new stuff has been added to KiCad tree but the savings in
build time by header precompilation will be still IMHO quite significant.

As for MSYS's GCC speed - personally I use Visual Studio under Windows,
not only because of faster build time but also because of a debugger
that actually works...

As a side comment:

$ make -j12 (latest KiCad master, i7-9750H, 32 GB DDR4 @ 2666 MHz,
Ubuntu 19.04)

real7m59.758s
user86m44.231s
sys 5m9.724s

Citing the classic: "not great, not terrible" ;-)

Cheers,
Tom

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Jon Evans]

Ha, I also keep around my Thinkpad T61 for occasional testing for this
purpose, although usually I avoid building on it as that takes a lonng
time.

On Tue, Oct 29, 2019 at 10:57 AM Drew Van Zandt 
wrote:

> Let me encourage some of y'all to stick with an older machine, or keep it
> around for testing... the latest build of KiCAD is noticeably laggy on my
> machine, which is not terribly fast (but also not that old).  It makes some
> of the bugs I have reported worse, and their fixes have been pushed out to
> 6.x (for perfectly good reasons).
>
> I am a bit manic, though, as Jon can attest; patience is last on my list
> of accessible virtues.
>
>
> *Drew Van Zandt*
>
>
> On Tue, Oct 29, 2019 at 10:40 AM Simon Richter 
> wrote:
>
>> Hi,
>>
>> On Mon, Oct 28, 2019 at 10:20:45PM -0700, Andrew Lutsenko wrote:
>>
>> > I kinda expected 2x maybe 3x decrease because not all computations scale
>> > linearly with number of threads. I was pleasantly surprised by almost 6x
>> > decrease in clean build time and 5x in incremental builds.
>>
>> FWIW, we have a number of bottlenecks in the build process:
>>
>>  - dependency generation of common doesn't start until the version header
>>is generated, which only happens after bitmaps and gal are complete
>>  - dependency generation of pcbnew_kiface doesn't start until the python
>>wrapper is generated, which takes ages itself.
>>  - the python wrapper is compiled as one of the last things, and the
>> linker
>>must wait for that.
>>
>> We could probably shave off another two or three minutes of build time if
>> we could make sure that we always make progress on the critical path. The
>> dependency generation as a side effect pulls all the sources and headers
>> into cache, which reduces the effects of I/O latency a bit during
>> compilation, so parallelizing with more than the number of threads you
>> actually have is probably counterproductive.
>>
>> Numbers with CPU time as percent of real time:
>>
>> clean   ~15 ~5  tip
>> E5-2620v4 (32)  7:04.87 5:18.74 5:16.64 3:40.92
>> 2243%   1854%   1854%   1330%
>> T2P9D01 (64)5:16.37 4:14.04 4:10.26 3:20.97
>> 3360%   2558%   2570%   1546%
>>
>>Simon
>>
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Drew Van Zandt]

Let me encourage some of y'all to stick with an older machine, or keep it
around for testing... the latest build of KiCAD is noticeably laggy on my
machine, which is not terribly fast (but also not that old).  It makes some
of the bugs I have reported worse, and their fixes have been pushed out to
6.x (for perfectly good reasons).

I am a bit manic, though, as Jon can attest; patience is last on my list of
accessible virtues.


*Drew Van Zandt*


On Tue, Oct 29, 2019 at 10:40 AM Simon Richter 
wrote:

> Hi,
>
> On Mon, Oct 28, 2019 at 10:20:45PM -0700, Andrew Lutsenko wrote:
>
> > I kinda expected 2x maybe 3x decrease because not all computations scale
> > linearly with number of threads. I was pleasantly surprised by almost 6x
> > decrease in clean build time and 5x in incremental builds.
>
> FWIW, we have a number of bottlenecks in the build process:
>
>  - dependency generation of common doesn't start until the version header
>is generated, which only happens after bitmaps and gal are complete
>  - dependency generation of pcbnew_kiface doesn't start until the python
>wrapper is generated, which takes ages itself.
>  - the python wrapper is compiled as one of the last things, and the linker
>must wait for that.
>
> We could probably shave off another two or three minutes of build time if
> we could make sure that we always make progress on the critical path. The
> dependency generation as a side effect pulls all the sources and headers
> into cache, which reduces the effects of I/O latency a bit during
> compilation, so parallelizing with more than the number of threads you
> actually have is probably counterproductive.
>
> Numbers with CPU time as percent of real time:
>
> clean   ~15 ~5  tip
> E5-2620v4 (32)  7:04.87 5:18.74 5:16.64 3:40.92
> 2243%   1854%   1854%   1330%
> T2P9D01 (64)5:16.37 4:14.04 4:10.26 3:20.97
> 3360%   2558%   2570%   1546%
>
>Simon
>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Simon Richter]

Hi,

On Mon, Oct 28, 2019 at 10:20:45PM -0700, Andrew Lutsenko wrote:

> I kinda expected 2x maybe 3x decrease because not all computations scale
> linearly with number of threads. I was pleasantly surprised by almost 6x
> decrease in clean build time and 5x in incremental builds.

FWIW, we have a number of bottlenecks in the build process:

 - dependency generation of common doesn't start until the version header
   is generated, which only happens after bitmaps and gal are complete
 - dependency generation of pcbnew_kiface doesn't start until the python
   wrapper is generated, which takes ages itself.
 - the python wrapper is compiled as one of the last things, and the linker
   must wait for that.

We could probably shave off another two or three minutes of build time if
we could make sure that we always make progress on the critical path. The
dependency generation as a side effect pulls all the sources and headers
into cache, which reduces the effects of I/O latency a bit during
compilation, so parallelizing with more than the number of threads you
actually have is probably counterproductive.

Numbers with CPU time as percent of real time:

clean   ~15 ~5  tip
E5-2620v4 (32)  7:04.87 5:18.74 5:16.64 3:40.92
2243%   1854%   1854%   1330%
T2P9D01 (64)5:16.37 4:14.04 4:10.26 3:20.97
3360%   2558%   2570%   1546%

   Simon

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Jon Evans]

Thanks for this data! I also have a Haswell system right now and have been
eyeing the new Ryzens... I too feel the pain of long kicad compilation
time.

-Jon

On Tue, Oct 29, 2019, 01:51 Henner Zeller  wrote:

>
>
> On Mon, 28 Oct 2019 at 22:21, Andrew Lutsenko 
> wrote:
>
>> Hi all,
>>
>> This information may be useful to people that are considering an upgrade
>> for their workstations or for general information, feel free to skip this
>> email if you are not interested.
>>
>> I've been using my 6 year old Haswell i7 machine that served me well over
>> the years but kicad compilation times both under msys and in linux vm were
>> frustratingly long. That reason along with a few others pushed me to
>> upgrade.
>> If you've been following the hardware news lately AMD's Ryzen is all the
>> rage lately and for a very good reason. I've been rooting for them since
>> Athlon days but had to begrudgingly buy Intel a few times after that
>> because AMD products just weren't competitive.
>> This definitely has changed with Ryzen series.
>>
>> To not make this too long and so that you don't think of me as AMD shill
>> let the numbers speak for themselves and you decide whether it's worth it
>> to upgrade for you.
>>
>> My previous config: i7 4770 Haswell 4 core CPU, 16gb of DDR3 ram in dual
>> channel, sata ssd.
>> My new config: R9 3900x Matisse 12 core CPU, 32gb of DDR4 ram in dual
>> channel, nvme ssd.
>>
>> I set my master branch to same commit in both cases (846397c69) and did
>> "make clean". Then I compiled all targets to measure time of clean build.
>> Then switched to master~15 and compiled, then switched to master~5 and
>> compiled, and lastly back to master and compiled again. Thus I measured
>> more typical scenario of incremental compilation as well.
>> In case of my old system I did "make -j8" as that is the number of
>> threads that CPU has, in case of new one I did "make -j24" for same reason,
>> maximizing used threads.
>>
>
> For compilation uses, I recently upgraded to a 16 core 1950x and am also
> pretty impressed.
> I find that using a bit more parallelism than can even give a little more
> as not all compile threads might be using the CPU 100% but might be waiting
> on IO. On my 1950X, I use -j50 (on 32 hyper-threaded cores). Something like
> this might in particular work for your network storage connection.
>
> Also highly recommended (when in development with intermediate 'make
> clean') install ccache
>
> Also something of note is that for both configs linux vm was run from a
>> virtual disk located on a network storage device with 1Gbps link, not on
>> local ssd. And I didn't master~15 build for linux, just master~5 and master
>> again.
>>
>> clean master~15 master~5 master
>> i7 4770 msys 36m28.515s 26m51.178s 27m45.738s 14m21.653s
>> R9 3900x msys 6m40.112s 4m56.747s 4m55.271s 3m15.945s
>> i7 4770 linux 44m49.681s - 15m53.635s 15m53.098s
>> R9 3900x linux 6m35.732s - 3m26.773s 3m18.514s
>>
>> I kinda expected 2x maybe 3x decrease because not all computations scale
>> linearly with number of threads. I was pleasantly surprised by almost 6x
>> decrease in clean build time and 5x in incremental builds.
>>
>> Here are the numbers in google sheet in case table above is not readable
>>
>> https://docs.google.com/spreadsheets/d/1UI1aW-FTTI-TncTWhAdG_Ag__5nc2FF1626WaPiL3LE/edit?usp=sharing
>>
>> Regards,
>> Andrew
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Henner Zeller]

On Mon, 28 Oct 2019 at 22:21, Andrew Lutsenko  wrote:

> Hi all,
>
> This information may be useful to people that are considering an upgrade
> for their workstations or for general information, feel free to skip this
> email if you are not interested.
>
> I've been using my 6 year old Haswell i7 machine that served me well over
> the years but kicad compilation times both under msys and in linux vm were
> frustratingly long. That reason along with a few others pushed me to
> upgrade.
> If you've been following the hardware news lately AMD's Ryzen is all the
> rage lately and for a very good reason. I've been rooting for them since
> Athlon days but had to begrudgingly buy Intel a few times after that
> because AMD products just weren't competitive.
> This definitely has changed with Ryzen series.
>
> To not make this too long and so that you don't think of me as AMD shill
> let the numbers speak for themselves and you decide whether it's worth it
> to upgrade for you.
>
> My previous config: i7 4770 Haswell 4 core CPU, 16gb of DDR3 ram in dual
> channel, sata ssd.
> My new config: R9 3900x Matisse 12 core CPU, 32gb of DDR4 ram in dual
> channel, nvme ssd.
>
> I set my master branch to same commit in both cases (846397c69) and did
> "make clean". Then I compiled all targets to measure time of clean build.
> Then switched to master~15 and compiled, then switched to master~5 and
> compiled, and lastly back to master and compiled again. Thus I measured
> more typical scenario of incremental compilation as well.
> In case of my old system I did "make -j8" as that is the number of threads
> that CPU has, in case of new one I did "make -j24" for same reason,
> maximizing used threads.
>

For compilation uses, I recently upgraded to a 16 core 1950x and am also
pretty impressed.
I find that using a bit more parallelism than can even give a little more
as not all compile threads might be using the CPU 100% but might be waiting
on IO. On my 1950X, I use -j50 (on 32 hyper-threaded cores). Something like
this might in particular work for your network storage connection.

Also highly recommended (when in development with intermediate 'make
clean') install ccache

Also something of note is that for both configs linux vm was run from a
> virtual disk located on a network storage device with 1Gbps link, not on
> local ssd. And I didn't master~15 build for linux, just master~5 and master
> again.
>
> clean master~15 master~5 master
> i7 4770 msys 36m28.515s 26m51.178s 27m45.738s 14m21.653s
> R9 3900x msys 6m40.112s 4m56.747s 4m55.271s 3m15.945s
> i7 4770 linux 44m49.681s - 15m53.635s 15m53.098s
> R9 3900x linux 6m35.732s - 3m26.773s 3m18.514s
>
> I kinda expected 2x maybe 3x decrease because not all computations scale
> linearly with number of threads. I was pleasantly surprised by almost 6x
> decrease in clean build time and 5x in incremental builds.
>
> Here are the numbers in google sheet in case table above is not readable
>
> https://docs.google.com/spreadsheets/d/1UI1aW-FTTI-TncTWhAdG_Ag__5nc2FF1626WaPiL3LE/edit?usp=sharing
>
> Regards,
> Andrew
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

[Kicad-developers] Benchmarking kicad compilation on CPUs released 6 years apart

[Andrew Lutsenko]

Hi all,

This information may be useful to people that are considering an upgrade
for their workstations or for general information, feel free to skip this
email if you are not interested.

I've been using my 6 year old Haswell i7 machine that served me well over
the years but kicad compilation times both under msys and in linux vm were
frustratingly long. That reason along with a few others pushed me to
upgrade.
If you've been following the hardware news lately AMD's Ryzen is all the
rage lately and for a very good reason. I've been rooting for them since
Athlon days but had to begrudgingly buy Intel a few times after that
because AMD products just weren't competitive.
This definitely has changed with Ryzen series.

To not make this too long and so that you don't think of me as AMD shill
let the numbers speak for themselves and you decide whether it's worth it
to upgrade for you.

My previous config: i7 4770 Haswell 4 core CPU, 16gb of DDR3 ram in dual
channel, sata ssd.
My new config: R9 3900x Matisse 12 core CPU, 32gb of DDR4 ram in dual
channel, nvme ssd.

I set my master branch to same commit in both cases (846397c69) and did
"make clean". Then I compiled all targets to measure time of clean build.
Then switched to master~15 and compiled, then switched to master~5 and
compiled, and lastly back to master and compiled again. Thus I measured
more typical scenario of incremental compilation as well.
In case of my old system I did "make -j8" as that is the number of threads
that CPU has, in case of new one I did "make -j24" for same reason,
maximizing used threads. Also something of note is that for both configs
linux vm was run from a virtual disk located on a network storage device
with 1Gbps link, not on local ssd. And I didn't master~15 build for linux,
just master~5 and master again.

clean master~15 master~5 master
i7 4770 msys 36m28.515s 26m51.178s 27m45.738s 14m21.653s
R9 3900x msys 6m40.112s 4m56.747s 4m55.271s 3m15.945s
i7 4770 linux 44m49.681s - 15m53.635s 15m53.098s
R9 3900x linux 6m35.732s - 3m26.773s 3m18.514s

I kinda expected 2x maybe 3x decrease because not all computations scale
linearly with number of threads. I was pleasantly surprised by almost 6x
decrease in clean build time and 5x in incremental builds.

Here are the numbers in google sheet in case table above is not readable
https://docs.google.com/spreadsheets/d/1UI1aW-FTTI-TncTWhAdG_Ag__5nc2FF1626WaPiL3LE/edit?usp=sharing

Regards,
Andrew
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp