Re: Compiling kernels faster (was Re: v4.10: kernel stack frame pointer .. has bad value (null))

2017-03-10 Thread Willy Tarreau
Hi Pavel,

On Fri, Mar 10, 2017 at 02:17:51PM +0100, Pavel Machek wrote:
> On Thu 2017-03-09 13:16:09, Geert Uytterhoeven wrote:
> > Hi Pavel,
> > 
> > On Thu, Mar 9, 2017 at 11:56 AM, Pavel Machek  wrote:
> > > On Thu 2017-03-09 10:38:46, Geert Uytterhoeven wrote:
> > >> On Wed, Mar 8, 2017 at 10:22 PM, Pavel Machek  wrote:
> > >> > Well, I have fast CPUs, but most of the time they just compile
> > >> > stuff. Especially bisect is compile-heavy. I suspect going back to
> > >> > gcc-3.2 would bring me bigger advantages than CPU upgrade...
> > >>
> > >> I hope you do use ccache or distcc?
> > >
> > > I tried to use distcc before, but it was rather hard to maintain. No
> > > ccache here. Hmm. I guess ccache really makes sense for bisect.
> > 
> > Yes it does. So if you're not using it yet, do the below, today, not 
> > tomorrow.
> > 
> > If your distro supports it, prepend /usr/lib/ccache/ to your $PATH.
> > Create symlinks from the names of your favorite cross-compilers
> > to /usr/bin/ccache, and make sure they are early in your $PATH.
> > 
> > That's it! Enjoy!
> 
> Hmm. Installed, and seems to work. OTOH, compilation now seems to
> produce 2-3MB/sec writing on spinning rust, and CPUs are no longer
> fully loaded. (make -j 7 on 2 core HT machine). Any io load sends the
> CPU utilization to cca 50% range... Compilation goes up from 9:13 to
> 11:40... to 23 minutes depending on situation. I guess it is still
> worth it for the bisect, but it looks like ccache really needs an ssd.

You need to put your cache in /dev/shm or some fast place not moving
heavy metallic heads.

That said, I have great success with distcc, I use it a lot with my build
farms (home and work) on some fanless machines [1]. I had to apply some
small updates recently to distcc because I noticed it would not delegate
files built with certain -Wa arguments as were recently added to the kernel.
I also found that by default it limits itself to 50 jobs which is often not
enough to keep your local machine busy.

The last 3.10.105-rc1 series I sent was build-tested this way, and it
builds allmodconfig in slightly less than 5 mn (4900 modules I believe),
with peaks up to 120 files per second. That's quite fast.

With distcc however you need a fast local machine because cpp is run
there, like a number of other tasks (eg: do not compress modules). You need
some RAM as well because you'll need a high parallel job count to keep your
machines busy (I build at -j60 which is the optimal value in my case). At
work only 4 small fanless ARM boards (cortex A17) cut my build time by 3
(local is a t430s with an i5-3320m) and at home 6 such boards cut the build
time by 2.5 (local is i7-6700K).

You cannot reasonably do that with too slow build nodes because you want
to limit the maximum build time for a single file. Here the cortex A17
at 1.8-2.0 GHz is perfect, it's the fastest fanless machine I found. Some
cheap x86 machines can work well also. In all case you must absolutely use
gigabit network or you'll constantly have some idle time as the traffic is
very bursty. I'm seeing up to 650 Mbps without compressing with LZO, and
between 70-150 Mbps with LZO, and it always builds faster with it.

> On the other hand, switching to -O1 is really easy, and gets 15% or so
> improvement.

I used to do such things in the past but certain level of optimizations
are useful to report warnings. For example I build haproxy daily at -O0,
but sometimes I discover later that I introduced some warnings that are
only detected at -O2.

> Hmm. And killing chromium matters a lot for a compile time. I hate
> modern web :-(.

killall -STOP. That's what I'm doing with firefox when I want some
resources.

Cheers,
Willy

---
[1] 
https://forum.mqmaker.com/t/miqi-based-build-farm-finally-up-and-running/605/24


Re: Compiling kernels faster (was Re: v4.10: kernel stack frame pointer .. has bad value (null))

2017-03-10 Thread Willy Tarreau
Hi Pavel,

On Fri, Mar 10, 2017 at 02:17:51PM +0100, Pavel Machek wrote:
> On Thu 2017-03-09 13:16:09, Geert Uytterhoeven wrote:
> > Hi Pavel,
> > 
> > On Thu, Mar 9, 2017 at 11:56 AM, Pavel Machek  wrote:
> > > On Thu 2017-03-09 10:38:46, Geert Uytterhoeven wrote:
> > >> On Wed, Mar 8, 2017 at 10:22 PM, Pavel Machek  wrote:
> > >> > Well, I have fast CPUs, but most of the time they just compile
> > >> > stuff. Especially bisect is compile-heavy. I suspect going back to
> > >> > gcc-3.2 would bring me bigger advantages than CPU upgrade...
> > >>
> > >> I hope you do use ccache or distcc?
> > >
> > > I tried to use distcc before, but it was rather hard to maintain. No
> > > ccache here. Hmm. I guess ccache really makes sense for bisect.
> > 
> > Yes it does. So if you're not using it yet, do the below, today, not 
> > tomorrow.
> > 
> > If your distro supports it, prepend /usr/lib/ccache/ to your $PATH.
> > Create symlinks from the names of your favorite cross-compilers
> > to /usr/bin/ccache, and make sure they are early in your $PATH.
> > 
> > That's it! Enjoy!
> 
> Hmm. Installed, and seems to work. OTOH, compilation now seems to
> produce 2-3MB/sec writing on spinning rust, and CPUs are no longer
> fully loaded. (make -j 7 on 2 core HT machine). Any io load sends the
> CPU utilization to cca 50% range... Compilation goes up from 9:13 to
> 11:40... to 23 minutes depending on situation. I guess it is still
> worth it for the bisect, but it looks like ccache really needs an ssd.

You need to put your cache in /dev/shm or some fast place not moving
heavy metallic heads.

That said, I have great success with distcc, I use it a lot with my build
farms (home and work) on some fanless machines [1]. I had to apply some
small updates recently to distcc because I noticed it would not delegate
files built with certain -Wa arguments as were recently added to the kernel.
I also found that by default it limits itself to 50 jobs which is often not
enough to keep your local machine busy.

The last 3.10.105-rc1 series I sent was build-tested this way, and it
builds allmodconfig in slightly less than 5 mn (4900 modules I believe),
with peaks up to 120 files per second. That's quite fast.

With distcc however you need a fast local machine because cpp is run
there, like a number of other tasks (eg: do not compress modules). You need
some RAM as well because you'll need a high parallel job count to keep your
machines busy (I build at -j60 which is the optimal value in my case). At
work only 4 small fanless ARM boards (cortex A17) cut my build time by 3
(local is a t430s with an i5-3320m) and at home 6 such boards cut the build
time by 2.5 (local is i7-6700K).

You cannot reasonably do that with too slow build nodes because you want
to limit the maximum build time for a single file. Here the cortex A17
at 1.8-2.0 GHz is perfect, it's the fastest fanless machine I found. Some
cheap x86 machines can work well also. In all case you must absolutely use
gigabit network or you'll constantly have some idle time as the traffic is
very bursty. I'm seeing up to 650 Mbps without compressing with LZO, and
between 70-150 Mbps with LZO, and it always builds faster with it.

> On the other hand, switching to -O1 is really easy, and gets 15% or so
> improvement.

I used to do such things in the past but certain level of optimizations
are useful to report warnings. For example I build haproxy daily at -O0,
but sometimes I discover later that I introduced some warnings that are
only detected at -O2.

> Hmm. And killing chromium matters a lot for a compile time. I hate
> modern web :-(.

killall -STOP. That's what I'm doing with firefox when I want some
resources.

Cheers,
Willy

---
[1] 
https://forum.mqmaker.com/t/miqi-based-build-farm-finally-up-and-running/605/24


Re: Compiling kernels faster (was Re: v4.10: kernel stack frame pointer .. has bad value (null))

2017-03-10 Thread Geert Uytterhoeven
Hi Pavel,

On Fri, Mar 10, 2017 at 2:17 PM, Pavel Machek  wrote:
> On Thu 2017-03-09 13:16:09, Geert Uytterhoeven wrote:
>> On Thu, Mar 9, 2017 at 11:56 AM, Pavel Machek  wrote:
>> > On Thu 2017-03-09 10:38:46, Geert Uytterhoeven wrote:
>> >> I hope you do use ccache or distcc?
>> >
>> > I tried to use distcc before, but it was rather hard to maintain. No
>> > ccache here. Hmm. I guess ccache really makes sense for bisect.
>>
>> Yes it does. So if you're not using it yet, do the below, today, not 
>> tomorrow.
>>
>> If your distro supports it, prepend /usr/lib/ccache/ to your $PATH.
>> Create symlinks from the names of your favorite cross-compilers
>> to /usr/bin/ccache, and make sure they are early in your $PATH.
>>
>> That's it! Enjoy!
>
> Hmm. Installed, and seems to work. OTOH, compilation now seems to
> produce 2-3MB/sec writing on spinning rust, and CPUs are no longer
> fully loaded. (make -j 7 on 2 core HT machine). Any io load sends the
> CPU utilization to cca 50% range... Compilation goes up from 9:13 to
> 11:40... to 23 minutes depending on situation. I guess it is still

I guess that was the first build, with a clean cache?
Now run "make clean", and try again ;-)

BTW, I tend not to do -j beyond the number of cores/threads (i.e. -j 8
on the i7-4770), unless you just want to compile, and not enjoy other
interactive work ;-)

> worth it for the bisect, but it looks like ccache really needs an ssd.

Adding an SSD never hurts.
Although I have been a happy user of ccache since long before I got an SSD.

> Hmm. And killing chromium matters a lot for a compile time. I hate
> modern web :-(.

Adding (freeing) RAM also never hurts ;-)

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds


Re: Compiling kernels faster (was Re: v4.10: kernel stack frame pointer .. has bad value (null))

2017-03-10 Thread Geert Uytterhoeven
Hi Pavel,

On Fri, Mar 10, 2017 at 2:17 PM, Pavel Machek  wrote:
> On Thu 2017-03-09 13:16:09, Geert Uytterhoeven wrote:
>> On Thu, Mar 9, 2017 at 11:56 AM, Pavel Machek  wrote:
>> > On Thu 2017-03-09 10:38:46, Geert Uytterhoeven wrote:
>> >> I hope you do use ccache or distcc?
>> >
>> > I tried to use distcc before, but it was rather hard to maintain. No
>> > ccache here. Hmm. I guess ccache really makes sense for bisect.
>>
>> Yes it does. So if you're not using it yet, do the below, today, not 
>> tomorrow.
>>
>> If your distro supports it, prepend /usr/lib/ccache/ to your $PATH.
>> Create symlinks from the names of your favorite cross-compilers
>> to /usr/bin/ccache, and make sure they are early in your $PATH.
>>
>> That's it! Enjoy!
>
> Hmm. Installed, and seems to work. OTOH, compilation now seems to
> produce 2-3MB/sec writing on spinning rust, and CPUs are no longer
> fully loaded. (make -j 7 on 2 core HT machine). Any io load sends the
> CPU utilization to cca 50% range... Compilation goes up from 9:13 to
> 11:40... to 23 minutes depending on situation. I guess it is still

I guess that was the first build, with a clean cache?
Now run "make clean", and try again ;-)

BTW, I tend not to do -j beyond the number of cores/threads (i.e. -j 8
on the i7-4770), unless you just want to compile, and not enjoy other
interactive work ;-)

> worth it for the bisect, but it looks like ccache really needs an ssd.

Adding an SSD never hurts.
Although I have been a happy user of ccache since long before I got an SSD.

> Hmm. And killing chromium matters a lot for a compile time. I hate
> modern web :-(.

Adding (freeing) RAM also never hurts ;-)

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds


Compiling kernels faster (was Re: v4.10: kernel stack frame pointer .. has bad value (null))

2017-03-10 Thread Pavel Machek
On Thu 2017-03-09 13:16:09, Geert Uytterhoeven wrote:
> Hi Pavel,
> 
> On Thu, Mar 9, 2017 at 11:56 AM, Pavel Machek  wrote:
> > On Thu 2017-03-09 10:38:46, Geert Uytterhoeven wrote:
> >> On Wed, Mar 8, 2017 at 10:22 PM, Pavel Machek  wrote:
> >> > Well, I have fast CPUs, but most of the time they just compile
> >> > stuff. Especially bisect is compile-heavy. I suspect going back to
> >> > gcc-3.2 would bring me bigger advantages than CPU upgrade...
> >>
> >> I hope you do use ccache or distcc?
> >
> > I tried to use distcc before, but it was rather hard to maintain. No
> > ccache here. Hmm. I guess ccache really makes sense for bisect.
> 
> Yes it does. So if you're not using it yet, do the below, today, not tomorrow.
> 
> If your distro supports it, prepend /usr/lib/ccache/ to your $PATH.
> Create symlinks from the names of your favorite cross-compilers
> to /usr/bin/ccache, and make sure they are early in your $PATH.
> 
> That's it! Enjoy!

Hmm. Installed, and seems to work. OTOH, compilation now seems to
produce 2-3MB/sec writing on spinning rust, and CPUs are no longer
fully loaded. (make -j 7 on 2 core HT machine). Any io load sends the
CPU utilization to cca 50% range... Compilation goes up from 9:13 to
11:40... to 23 minutes depending on situation. I guess it is still
worth it for the bisect, but it looks like ccache really needs an ssd.

On the other hand, switching to -O1 is really easy, and gets 15% or so
improvement.

Hmm. And killing chromium matters a lot for a compile time. I hate
modern web :-(.

Best regards,   Pavel


--- a/Makefile
+++ b/Makefile
@@ -639,9 +639,9 @@ ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
 KBUILD_CFLAGS  += -Os $(call cc-disable-warning,maybe-uninitialized,)
 else
 ifdef CONFIG_PROFILE_ALL_BRANCHES
-KBUILD_CFLAGS  += -O2 $(call cc-disable-warning,maybe-uninitialized,)
+KBUILD_CFLAGS  += -O1 $(call cc-disable-warning,maybe-uninitialized,)
 else
-KBUILD_CFLAGS   += -O2
+KBUILD_CFLAGS   += -O1
 endif
 endif

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) 
http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html


signature.asc
Description: Digital signature


Compiling kernels faster (was Re: v4.10: kernel stack frame pointer .. has bad value (null))

2017-03-10 Thread Pavel Machek
On Thu 2017-03-09 13:16:09, Geert Uytterhoeven wrote:
> Hi Pavel,
> 
> On Thu, Mar 9, 2017 at 11:56 AM, Pavel Machek  wrote:
> > On Thu 2017-03-09 10:38:46, Geert Uytterhoeven wrote:
> >> On Wed, Mar 8, 2017 at 10:22 PM, Pavel Machek  wrote:
> >> > Well, I have fast CPUs, but most of the time they just compile
> >> > stuff. Especially bisect is compile-heavy. I suspect going back to
> >> > gcc-3.2 would bring me bigger advantages than CPU upgrade...
> >>
> >> I hope you do use ccache or distcc?
> >
> > I tried to use distcc before, but it was rather hard to maintain. No
> > ccache here. Hmm. I guess ccache really makes sense for bisect.
> 
> Yes it does. So if you're not using it yet, do the below, today, not tomorrow.
> 
> If your distro supports it, prepend /usr/lib/ccache/ to your $PATH.
> Create symlinks from the names of your favorite cross-compilers
> to /usr/bin/ccache, and make sure they are early in your $PATH.
> 
> That's it! Enjoy!

Hmm. Installed, and seems to work. OTOH, compilation now seems to
produce 2-3MB/sec writing on spinning rust, and CPUs are no longer
fully loaded. (make -j 7 on 2 core HT machine). Any io load sends the
CPU utilization to cca 50% range... Compilation goes up from 9:13 to
11:40... to 23 minutes depending on situation. I guess it is still
worth it for the bisect, but it looks like ccache really needs an ssd.

On the other hand, switching to -O1 is really easy, and gets 15% or so
improvement.

Hmm. And killing chromium matters a lot for a compile time. I hate
modern web :-(.

Best regards,   Pavel


--- a/Makefile
+++ b/Makefile
@@ -639,9 +639,9 @@ ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
 KBUILD_CFLAGS  += -Os $(call cc-disable-warning,maybe-uninitialized,)
 else
 ifdef CONFIG_PROFILE_ALL_BRANCHES
-KBUILD_CFLAGS  += -O2 $(call cc-disable-warning,maybe-uninitialized,)
+KBUILD_CFLAGS  += -O1 $(call cc-disable-warning,maybe-uninitialized,)
 else
-KBUILD_CFLAGS   += -O2
+KBUILD_CFLAGS   += -O1
 endif
 endif

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) 
http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html


signature.asc
Description: Digital signature