Re: [gentoo-user] is multi-core really worth it?

[Kai Peter]

On 2017-12-06 16:34, Alan McKinnon wrote:


And just to round off a mostly pointless discussion with little real
merit, the really stupid thing about portage is why oh why are ports 
and

distfiles in /usr?
I'm really surprised that someone recognized this or may be does 
question this. Fortunately it is easy to change for the user on a per 
installation base, but not for the upstream because of the things which 
follows.


I'll tell you why, it's because that's where FreeBSD puts them, and
drobbins built Gentoo back in the day heavily borrowing from his
pleasant FreeBSD experience (he went there for 6 months recovering from
his departure from another distro, the one with the "toxic
personality"). And no-one ever bothered changing that initial decision 
-

a classic case of cargo cult
That you not aware of it doesn't indicate that it wasn't bothered. 
Perhaps people will not waste (any more) time with senseless discussions 
...


--
Sent with eQmail-1.10

Re: [gentoo-user] is multi-core really worth it?

[Wols Lists]

On 06/12/17 15:34, Alan McKinnon wrote:
> Those guidelines you mention about what /tmp and /var/tmp are "for" are
> probably from the FHS. On the whole, I tend to agree they are good ideas
> but the proper wording is more like this (from memory, being far too
> lazy after a day's work to actually look something up):
> 
> - contents of /tmp are not expected to survive the invocation of the
> program that created them

http://www.pathname.com/fhs/pub/fhs-2.3.html#TMPTEMPORARYFILES

> - contents of /var/tmp are not expected to survive a reboot
> 
http://www.pathname.com/fhs/pub/fhs-2.3.html#VARTMPTEMPORARYFILESPRESERVEDBETWEE

> Which is different from what you said.

Except that /var/tmp is exactly the opposite of what you said :-)

 Not surprisingly, if you follow
> that through, you can run rm -rf /tmp/* in a cron every minute and
> nothing should ever break. Or, every file in /tmp can be anonymous (just
> an inode without a dentry giving it a name)

Cheers,
Wol

Re: [gentoo-user] is multi-core really worth it?

[Wols Lists]

On 06/12/17 15:34, Alan McKinnon wrote:
> - contents of /tmp are not expected to survive the invocation of the
> program that created them
> - contents of /var/tmp are not expected to survive a reboot

That sounds completely wrong, actually.

The contents of /var/tmp are expected to survive a system crash, as that
is where vi, emacs, libreoffice et al are expected to store their
recovery logs.

Not much point putting the logs somewhere where they will be deleted by
the very occurrence they are intended to protect against ...

And yes, the rules for /tmp are "don't expect to find anything you put
there will be there a few minutes later ..." :-)

Cheers,
Wol

Re: [gentoo-user] is multi-core really worth it?

[Alan McKinnon]

On 06/12/2017 15:29, Wols Lists wrote:
> On 05/12/17 21:56, Neil Bothwick wrote:
>> On Tue, 05 Dec 2017 10:09:56 +, Peter Humphrey wrote:
>>
>>> $ grep tmpfs /etc/fstab
>>> tmpfs   /var/tmp/portage  tmpfs
>>> noatime,uid=portage,gid=portage,mode=0775  0 0
>>> tmpfs   /tmp  tmpfs
>>> noatime,nosuid,nodev,noexec,mode=1777   0 0
>>
>> Or you could set PORTAGE_TMPDIR to /tmp to save the second mount.
>>
> Dunno why portage puts this stuff in /var/tmp, rather than /tmp, but do
> be aware of what the standard says ...
> 
> Stuff in /tmp should be cleared at shutdown/boot.
> 
> Stuff in /var/tmp should survive a shutdown/boot.
> 
> Of course, if, like me you've put /var/tmp/portage as tmpfs, then of
> course it won't survive a reboot, contrary to spec ... :-)

Those guidelines you mention about what /tmp and /var/tmp are "for" are
probably from the FHS. On the whole, I tend to agree they are good ideas
but the proper wording is more like this (from memory, being far too
lazy after a day's work to actually look something up):

- contents of /tmp are not expected to survive the invocation of the
program that created them
- contents of /var/tmp are not expected to survive a reboot

Which is different from what you said. Not surprisingly, if you follow
that through, you can run rm -rf /tmp/* in a cron every minute and
nothing should ever break. Or, every file in /tmp can be anonymous (just
an inode without a dentry giving it a name)

The thing about standards, is that there are so many to choose from. And
the FHS has never been a standard that anyone paid much attention to.
It's also not a spec, it's a great example of a failed standard that few
if any distros ever bothered following.

Gentoo in particular never bothered following FHS explicitly; any
overlap is mostly accidental. And that is OK as Gentoo devs are
permitted to do whatever they feel like doing. Doubly so if they can
defend their decisions on technical merit.

On the whole, /var/tmp is a better place to put build files than /tmp
just in case someone does take FHS seriously - build files are
necessarily needed after the completion of the program that created them.

And just to round off a mostly pointless discussion with little real
merit, the really stupid thing about portage is why oh why are ports and
distfiles in /usr?

I'll tell you why, it's because that's where FreeBSD puts them, and
drobbins built Gentoo back in the day heavily borrowing from his
pleasant FreeBSD experience (he went there for 6 months recovering from
his departure from another distro, the one with the "toxic
personality"). And no-one ever bothered changing that initial decision -
a classic case of cargo cult


-- 
Alan McKinnon
alan.mckin...@gmail.com

Re: [gentoo-user] is multi-core really worth it?

[Wols Lists]

On 05/12/17 21:56, Neil Bothwick wrote:
> On Tue, 05 Dec 2017 10:09:56 +, Peter Humphrey wrote:
> 
>> $ grep tmpfs /etc/fstab
>> tmpfs   /var/tmp/portage  tmpfs
>> noatime,uid=portage,gid=portage,mode=0775  0 0
>> tmpfs   /tmp  tmpfs
>> noatime,nosuid,nodev,noexec,mode=1777   0 0
> 
> Or you could set PORTAGE_TMPDIR to /tmp to save the second mount.
> 
Dunno why portage puts this stuff in /var/tmp, rather than /tmp, but do
be aware of what the standard says ...

Stuff in /tmp should be cleared at shutdown/boot.

Stuff in /var/tmp should survive a shutdown/boot.

Of course, if, like me you've put /var/tmp/portage as tmpfs, then of
course it won't survive a reboot, contrary to spec ... :-)

Cheers,
Wol

Re: [gentoo-user] is multi-core really worth it?

[Neil Bothwick]

On Tue, 05 Dec 2017 10:09:56 +, Peter Humphrey wrote:

> $ grep tmpfs /etc/fstab
> tmpfs   /var/tmp/portage  tmpfs
> noatime,uid=portage,gid=portage,mode=0775  0 0
> tmpfs   /tmp  tmpfs
> noatime,nosuid,nodev,noexec,mode=1777   0 0

Or you could set PORTAGE_TMPDIR to /tmp to save the second mount.

> If a tmpfs fills up, the excess gets swapped out, but with 32GB RAM
> here I haven't yet seen any swap used at all - not even in an emerge -e
> world.

You can increase the memory available to it with the size option.
size=75% works for m, even if building chromium and LO at the same time.


-- 
Neil Bothwick

Tribble math: * + * = ***


pgp3DiDgOQhix.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] is multi-core really worth it?

[Peter Humphrey]

On Tuesday, 5 December 2017 13:57:38 GMT Wols Lists wrote:

> I've just had a long thread with someone on the SUSE list who refuses to
> believe that the "twice ram" rule ever existed.
> 
> This despite someone else actually describing the algorithm (from which
> one can see where the rule comes from), and me pointing out that (after
> Linus stripped out all the "awful" optimisation code) the early vanilla
> 2.4 kernels enforced this rule by crashing if you broke it.
> 
> Swap was rewritten as a result of that, but I've never heard whether the
> fundamental algorithm was changed, so I still provision my systems on
> the assumption it's true. Disk is cheap ... my 4TB drives cost about
> £110, so that makes 128GB for swap cost, what, £3? I'll probably never
> need it, but hey, at that price :-)

Ah, but it's a different kettle of fish if you're using LVMe SSD!  :)

Mine is 256GB and doesn't have an awful lot of spare capacity, what with 
BOINC and being the compile host for two other boxes.

-- 
Regards,
Peter.

Re: [gentoo-user] is multi-core really worth it?

[Wols Lists]

On 05/12/17 13:07, Peter Humphrey wrote:
>> (My new system when I get it working maxes out at 64GB ram so I'll have
>> > 256GB swap and (currently) 16GB ram)

> I've halved my original 4GB swap to 2GB since it never seems to be used. I'm 
> not brave enough to do away with it altogether though.

I've just had a long thread with someone on the SUSE list who refuses to
believe that the "twice ram" rule ever existed.

This despite someone else actually describing the algorithm (from which
one can see where the rule comes from), and me pointing out that (after
Linus stripped out all the "awful" optimisation code) the early vanilla
2.4 kernels enforced this rule by crashing if you broke it.

Swap was rewritten as a result of that, but I've never heard whether the
fundamental algorithm was changed, so I still provision my systems on
the assumption it's true. Disk is cheap ... my 4TB drives cost about
£110, so that makes 128GB for swap cost, what, £3? I'll probably never
need it, but hey, at that price :-)

Cheers,
Wol

Re: [gentoo-user] is multi-core really worth it?

[Peter Humphrey]

On Tuesday, 5 December 2017 10:46:43 GMT Wols Lists wrote:
> On 05/12/17 10:09, Peter Humphrey wrote:
> >> I assume using a ramdisk would help with this? I wouldn't want to do a
> >> 
> >> > SSD as I assume it would excessively wear by doing compiles.
> > 
> > I use tmpfs, like this:
> > 
> > $ grep tmpfs /etc/fstab
> > tmpfs   /var/tmp/portage  tmpfs  
> > noatime,uid=portage,gid=portage,mode=0775  0 0 tmpfs   /tmp
> >  tmpfs   noatime,nosuid,nodev,noexec,mode=1777   0 0
> > 
> > If a tmpfs fills up, the excess gets swapped out, but with 32GB RAM here
> > I haven't yet seen any swap used at all - not even in an emerge -e
> > world.
> Same here. Note that tmpfs defaults to half ram, so that would give you
> a 16GB /var/tmp/portage.

That's the starting size, yes. The kernel will expand it or shrink it with 
changes in the demands on the system.

> With 16GB ram here, that would probably cause things like emerging
> libreoffice or firefox or gcc to abort.

Not unless you run out of swap space - remember that tmpfs gets swapped like 
anything else in memory. Or unless one compile job requires a single 
temporary file bigger than your tmpfs file system, which hardly seems 
likely.

> My fstab has these lines ... note the SIZE option ...

Yes, I know about the size option. I haven't needed to use it on this box.

> # glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
> # POSIX shared memory (shm_open, shm_unlink).
> # (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
> #  use almost no memory if not populated with files)
> shm /dev/shmtmpfs
> nodev,nosuid,noexec 0 0

You shouldn't need that shm entry any more.

> portage /var/tmp/portagetmpfs
> size=30G,mode=0777  0 0
> tmp /tmptmpfs
> size=10G,mode=0777  0 0
> 
> My swap partitions are twice max ram, so I currently have two 32GB
> partitions giving me 80GB total ram and swap.

So you shouldn't find big compiler jobs aborting on out-of-memory.

> (My new system when I get it working maxes out at 64GB ram so I'll have
> 256GB swap and (currently) 16GB ram)

I've halved my original 4GB swap to 2GB since it never seems to be used. I'm 
not brave enough to do away with it altogether though.

-- 
Regards,
Peter.

Re: [gentoo-user] is multi-core really worth it?

[Mick]

On Tuesday, 5 December 2017 11:13:53 GMT Raffaele Belardi wrote:
> Wols Lists wrote:
> >> If a tmpfs fills up, the excess gets swapped out, but with 32GB RAM here
> >> I
> >> haven't yet seen any swap used at all - not even in an emerge -e world.
> > 
> > Same here. Note that tmpfs defaults to half ram, so that would give you
> > a 16GB /var/tmp/portage. With 16GB ram here, that would probably cause
> > things like emerging libreoffice or firefox or gcc to abort.
> 
> Not really, libreoffice and gcc compile slowly but fine here with 3Gb RAM
> and 3Gb spin-disk swap, and using PORTAGE_NICENESS=19 the system is still
> usable (rebuilding world for the profile switch right now). That's with an
> LXDE desktop, Gnome3 survived only a few months, _that_ was really
> unusable.
> 
> $ free
>   totalusedfree  shared  buff/cache  
> available Mem:3102960 1316120  964848  370488 
> 821992 1123260 Swap:   3076344   91648 2984696
> 
> $ df
> Filesystem  Size  Used Avail Use% Mounted on
> tmpfs   304M  1.9M  302M   1% /run
> none1.5G 0  1.5G   0% /dev/shm
> 
> $ eix -I libreoffice
> [I] app-office/libreoffice
>  Available versions:  5.4.2.2 (~)5.4.3.2
>  Installed versions:  5.4.3.2
> 
> $ gcc-config -l
>  [1] i686-pc-linux-gnu-7.2.0 *
> 
> $ qlop -gH libreoffice | tail -n 2
> libreoffice: Wed Nov 22 18:13:17 2017: 12 hours, 37 minutes, 9 seconds
> libreoffice: 13 times
> 
> $ qlop -gH gcc | tail -n 2
> gcc: Mon Dec  4 21:50:26 2017: 3 hours, 9 minutes, 7 seconds
> gcc: 80 times
> 
> $ uname -a
> Linux ws2912 4.14.0-gentoo #1 SMP Fri Nov 17 09:31:56 CET 2017 i686 Intel(R)
> Pentium(R) 4 CPU 3.40GHz GenuineIntel GNU/Linux

How does gcc-7.2.0 compare with 6.4.0 in terms of memory efficiency?

I'm asking because on a 1st gen i7 and 4G of RAM, for the first time ever, 
www-client/chromium-62.0.3202.94 failed to build because of an internal 
compiler error.  I'm guessing it had something to do with me always running 
MAKEOPTS="-j5 -l12.8", but I can't recall what formula I used to arrive at 
this load number ...

I'm trying again with emerge --load-average 4 to see if it compiles this time.  
When I switched to profile 17.0 I noticed a 2.5x increase in time it took to 
emerge the previous version of chromium, as part of 'emerge -e @world', but I 
hoped this was a one off mishap because I was using the PC at the time:

 Thu Nov  9 17:44:58 2017 >>> www-client/chromium-62.0.3202.89
   merge time: 8 hours, 12 minutes and 30 seconds.

 Mon Dec  4 11:39:36 2017 >>> www-client/chromium-62.0.3202.89
   merge time: 20 hours, 2 minutes and 4 seconds.

With only 4G RAM the poor thing was thrashing the disk swapping on and off 
forever, becoming completely unresponsive for minutes at a time.

-- 
Regards,
Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] is multi-core really worth it?

[Raffaele Belardi]

Wols Lists wrote:
>>
>> If a tmpfs fills up, the excess gets swapped out, but with 32GB RAM here I
>> haven't yet seen any swap used at all - not even in an emerge -e world.
> 
> Same here. Note that tmpfs defaults to half ram, so that would give you
> a 16GB /var/tmp/portage. With 16GB ram here, that would probably cause
> things like emerging libreoffice or firefox or gcc to abort.

Not really, libreoffice and gcc compile slowly but fine here with 3Gb RAM and 
3Gb
spin-disk swap, and using PORTAGE_NICENESS=19 the system is still usable 
(rebuilding world
for the profile switch right now). That's with an LXDE desktop, Gnome3 survived 
only a few
months, _that_ was really unusable.

$ free
  totalusedfree  shared  buff/cache   available
Mem:3102960 1316120  964848  370488  821992 1123260
Swap:   3076344   91648 2984696

$ df
Filesystem  Size  Used Avail Use% Mounted on
tmpfs   304M  1.9M  302M   1% /run
none1.5G 0  1.5G   0% /dev/shm

$ eix -I libreoffice
[I] app-office/libreoffice
 Available versions:  5.4.2.2 (~)5.4.3.2
 Installed versions:  5.4.3.2

$ gcc-config -l
 [1] i686-pc-linux-gnu-7.2.0 *

$ qlop -gH libreoffice | tail -n 2
libreoffice: Wed Nov 22 18:13:17 2017: 12 hours, 37 minutes, 9 seconds
libreoffice: 13 times

$ qlop -gH gcc | tail -n 2
gcc: Mon Dec  4 21:50:26 2017: 3 hours, 9 minutes, 7 seconds
gcc: 80 times

$ uname -a
Linux ws2912 4.14.0-gentoo #1 SMP Fri Nov 17 09:31:56 CET 2017 i686 Intel(R) 
Pentium(R) 4
CPU 3.40GHz GenuineIntel GNU/Linux

Re: [gentoo-user] is multi-core really worth it?

[Wols Lists]

On 05/12/17 10:09, Peter Humphrey wrote:
>> I assume using a ramdisk would help with this? I wouldn't want to do a
>> > SSD as I assume it would excessively wear by doing compiles.
> I use tmpfs, like this:
> 
> $ grep tmpfs /etc/fstab
> tmpfs   /var/tmp/portage  tmpfs   noatime,uid=portage,gid=portage,mode=0775  
> 0 0
> tmpfs   /tmp  tmpfs   
> noatime,nosuid,nodev,noexec,mode=1777   0 0
> 
> If a tmpfs fills up, the excess gets swapped out, but with 32GB RAM here I
> haven't yet seen any swap used at all - not even in an emerge -e world.

Same here. Note that tmpfs defaults to half ram, so that would give you
a 16GB /var/tmp/portage. With 16GB ram here, that would probably cause
things like emerging libreoffice or firefox or gcc to abort.

My fstab has these lines ... note the SIZE option ...

# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
#  use almost no memory if not populated with files)
shm /dev/shmtmpfs
nodev,nosuid,noexec 0 0
portage /var/tmp/portagetmpfs
size=30G,mode=0777  0 0
tmp /tmptmpfs
size=10G,mode=0777  0 0

My swap partitions are twice max ram, so I currently have two 32GB
partitions giving me 80GB total ram and swap.

(My new system when I get it working maxes out at 64GB ram so I'll have
256GB swap and (currently) 16GB ram)

Cheers,
Wol

Re: [gentoo-user] is multi-core really worth it?

[Peter Humphrey]

On Tuesday, 5 December 2017 04:11:17 GMT taii...@gmx.com wrote:
> On my 16 core opteron I have to do -j32 or sometimes -j64 to be using
> everything all the time, is this normal? If I don't do this it won't be
> pegged at 100% all the time.

On my 12-thread i7 I have -j24 -l60. Most times it's better not to limit the
number of jobs, just the load average; then portage loads up the CPU as high
as it can.

The exception, and even this is debatable, is when you're compiling a large
set of packages, say an emerge -e world, in which case so many jobs have
been started by the time they're all into compiling that the load soars to
silly heights - I've seen 80-odd here. But that's only about seven jobs
queued per CPU thread, so maybe it isn't too bad after all.

> I assume using a ramdisk would help with this? I wouldn't want to do a
> SSD as I assume it would excessively wear by doing compiles.

I use tmpfs, like this:

$ grep tmpfs /etc/fstab
tmpfs   /var/tmp/portage  tmpfs   noatime,uid=portage,gid=portage,mode=0775  0 0
tmpfs   /tmp  tmpfs   noatime,nosuid,nodev,noexec,mode=1777 
  0 0

If a tmpfs fills up, the excess gets swapped out, but with 32GB RAM here I
haven't yet seen any swap used at all - not even in an emerge -e world.

I've read that modern SSDs are far less prone to wear than earlier ones, as
R0b0t1 suggests.

-- 
Regards,
Peter.

Re: [gentoo-user] is multi-core really worth it?

[R0b0t1]

On Mon, Dec 4, 2017 at 10:11 PM, taii...@gmx.com  wrote:
> On my 16 core opteron I have to do -j32 or sometimes -j64 to be using
> everything all the time, is this normal? If I don't do this it won't be
> pegged at 100% all the time.
>

I use a ramdisk and anything over -j${NCPU} (incl. hyperthreaded
cores) causes performance degradation in general due mainly to the
compiling step. For some package sets I have noticed that higher build
parallelism helps due to some ebuild steps not being easily
parallelizable.

> I assume using a ramdisk would help with this? I wouldn't want to do a SSD
> as I assume it would excessively wear by doing compiles.
>

It should. While builds on an SSD will wear the SSD, comparing the
write volume to the expected drive lifetime will tell you the true
impact. It is (if I remember) still relatively small.

Cheers,
 R0b0t1

Re: [gentoo-user] is multi-core really worth it?

[taii...@gmx.com]

On my 16 core opteron I have to do -j32 or sometimes -j64 to be using 
everything all the time, is this normal? If I don't do this it won't be 
pegged at 100% all the time.


I assume using a ramdisk would help with this? I wouldn't want to do a 
SSD as I assume it would excessively wear by doing compiles.

Re: [gentoo-user] is multi-core really worth it?

[Neil Bothwick]

On Wed, 22 Nov 2017 09:57:29 +0100, David Haller wrote:

> Hm. What's this EMERGE_DEFAULT_OPTS though? ;)) And
> how can I override them if not wanted? .oO( gotta read up on that )Oo.

man make.conf
man emerge
emerge --ignore-default-opts


-- 
Neil Bothwick

The trouble with life is that you are halfway through it before you
realize it's a "do it yourself" thing.


pgpQ6sEEnLpbF.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] is multi-core really worth it?

[J. Roeleveld]

On Wednesday, November 22, 2017 3:34:56 PM CET Wols Lists wrote:
> On 22/11/17 14:11, Tsukasa Mcp_Reznor wrote:
> > You won't get build failures or dependency problems, portage is built to
> > handle emerging multiple packages that do not depend on each other
> > simultaneously.
> > it will not ever build a dependency and the main program at the same time.
> 
> Are you sure?
> 
> You *shouldn't* get problems, but I'm sure I have hit that problem, and
> I think it's down to buggy ebuilds.

buggy, as in, at least one dependency missed.

> Starting the emerge again fixes it, because cocked-up dependencies will
> sort themselves out first time round, and the second time the problem
> has gone away.

2nd time the dependency might have been installed already.
Sometime it hasn't.

This can also cause a failure when not using parallel builds for exactly the 
same reason.

--
Joost

Re: [gentoo-user] is multi-core really worth it?

[Wols Lists]

On 22/11/17 14:11, Tsukasa Mcp_Reznor wrote:
> 
> You won't get build failures or dependency problems, portage is built to
> handle emerging multiple packages that do not depend on each other
> simultaneously.
> it will not ever build a dependency and the main program at the same time.

Are you sure?

You *shouldn't* get problems, but I'm sure I have hit that problem, and
I think it's down to buggy ebuilds.

Starting the emerge again fixes it, because cocked-up dependencies will
sort themselves out first time round, and the second time the problem
has gone away.

Cheers,
Wol

Re: [gentoo-user] is multi-core really worth it?

[Tsukasa Mcp_Reznor]

From: Raffaele Belardi <raffaele.bela...@st.com>
Sent: Wednesday, November 22, 2017 8:12 AM
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] is multi-core really worth it?

Jeremi Piotrowski wrote:
> That being said: if you do a world rebuild you will have lots of packages
> that spend ~40 seconds doing their autoconf run, only to build 2-3 sources
> files. On an 8-core machine at work, I get good results using parallel
> emerge jobs (emerge -jX). For your 6-core AMD CPU (assuming it actually
> has 12 threads) I'd start with 'emerge -j3' and MAKEOPTS='-j12 -l16'.
> That should get you a nice speedup, but may require a bit more ram.

emerge -j3 is something I did not think of, I'll try it. But won't that break 
portage's
carefully crafted package dependencies? I suppose you could get occasional 
build failures?

I'm using MAKEOPTS=-j7, I thought 2 threads per CPU (hyperthreading?) was an 
Intel thing only.

raffaele


There are alot of good tweaks to get by long configure sections, I have an octa 
core, so for normal updates I use -
MAKEOPTS="-j8" # for 8 cores being used per package
EMERGE_DEFAULT_OPTS="--jobs 2 --load-average 4" # only starts 2nd job if load 
average is low.

You won't get build failures or dependency problems, portage is built to handle 
emerging multiple packages that do not depend on each other simultaneously.
it will not ever build a dependency and the main program at the same time.

PORTAGE_NICENESS="19" # best setting ever, I do some gaming while updating, you 
mainly only notice install phase if your are loading while the disk queue is 
full of writes.

Re: [gentoo-user] is multi-core really worth it?

[J. Roeleveld]

On Wednesday, November 22, 2017 2:12:31 PM CET Raffaele Belardi wrote:
> Jeremi Piotrowski wrote:
> > That being said: if you do a world rebuild you will have lots of packages
> > that spend ~40 seconds doing their autoconf run, only to build 2-3 sources
> > files. On an 8-core machine at work, I get good results using parallel
> > emerge jobs (emerge -jX). For your 6-core AMD CPU (assuming it actually
> > has 12 threads) I'd start with 'emerge -j3' and MAKEOPTS='-j12 -l16'.
> > That should get you a nice speedup, but may require a bit more ram.
> 
> emerge -j3 is something I did not think of, I'll try it. But won't that
> break portage's carefully crafted package dependencies? I suppose you could
> get occasional build failures?

No, it won't.
If it has multiple packages that can be built (taking dependencies into 
account) it will. If all packages in the queue are waiting for a single 
package to be installed, all those will wait till that package is finished.

> I'm using MAKEOPTS=-j7, I thought 2 threads per CPU (hyperthreading?) was an
> Intel thing only.

I thought so too.

Don't forget to add the "load-average" option to both, otherwise there is a 
very likely chance you will have 3 * 7 = 21 build processes running at the 
same time.

Also, I have had situations where using "-j" actually caused some issues where 
it was picked up by the wrong process or it has a different meaning for some 
other process. Specifying the long version (--jobs) has been reliable for me.

--
Joost

Re: [gentoo-user] is multi-core really worth it?

[Raffaele Belardi]

Jeremi Piotrowski wrote:
> That being said: if you do a world rebuild you will have lots of packages
> that spend ~40 seconds doing their autoconf run, only to build 2-3 sources
> files. On an 8-core machine at work, I get good results using parallel
> emerge jobs (emerge -jX). For your 6-core AMD CPU (assuming it actually
> has 12 threads) I'd start with 'emerge -j3' and MAKEOPTS='-j12 -l16'.
> That should get you a nice speedup, but may require a bit more ram.

emerge -j3 is something I did not think of, I'll try it. But won't that break 
portage's
carefully crafted package dependencies? I suppose you could get occasional 
build failures?

I'm using MAKEOPTS=-j7, I thought 2 threads per CPU (hyperthreading?) was an 
Intel thing only.

raffaele

Re: [gentoo-user] is multi-core really worth it?

[Peter Humphrey]

On Wednesday, 22 November 2017 08:57:29 GMT David Haller wrote:
> Hello,
> 
> On Wed, 22 Nov 2017, J. Roeleveld wrote:
> >On Wednesday, November 22, 2017 8:48:08 AM CET David Haller wrote:
> >> So, your emerge is mostly IO and "emerge"-threads bound. Solution:
> >> adjust your build-threads[1], and then adjust your emerge jobs! See
> >> '--jobs' in 'man emerge'. Can't find where to set a default in a
> >> config-file for that ATM, but you could always alias/function/script
> >
> >> emerge to something else, e.g. put this in your ~/.bashrc:
> >Instead of all these aliases, simply edit your make.conf ( /etc/portage/
> >make.conf )
> >
> >And edit (or add) the following lines:
> >MAKEOPTS="--jobs 24 --load-average 48"
> >EMERGE_DEFAULT_OPTS="--jobs 24 --load-average 48"
> 
> Well, that alias/function/script is mostly for all the other (default)
> options I want ;) Hm. What's this EMERGE_DEFAULT_OPTS though? ;)) And
> how can I override them if not wanted? .oO( gotta read up on that )Oo.

Options passed on the command line (as in an alias) override any others. I 
assume a function would be equivalent to an alias here, but I haven't 
checked as I don't use them.

There used to be debates on the efficiency of alias versus function, but if 
they're only taking nanoseconds to execute, what does it matter?

-- 
Regards,
Peter.

Re: [gentoo-user] is multi-core really worth it?

[David Haller]

Hello,

On Wed, 22 Nov 2017, J. Roeleveld wrote:
>On Wednesday, November 22, 2017 8:48:08 AM CET David Haller wrote:
>> So, your emerge is mostly IO and "emerge"-threads bound. Solution:
>> adjust your build-threads[1], and then adjust your emerge jobs! See
>> '--jobs' in 'man emerge'. Can't find where to set a default in a
>> config-file for that ATM, but you could always alias/function/script
>> emerge to something else, e.g. put this in your ~/.bashrc:
>
>Instead of all these aliases, simply edit your make.conf ( /etc/portage/
>make.conf )
>
>And edit (or add) the following lines:
>MAKEOPTS="--jobs 24 --load-average 48"
>EMERGE_DEFAULT_OPTS="--jobs 24 --load-average 48"

Well, that alias/function/script is mostly for all the other (default)
options I want ;) Hm. What's this EMERGE_DEFAULT_OPTS though? ;)) And
how can I override them if not wanted? .oO( gotta read up on that )Oo.

-dnh

-- 
   85: Hot plugable
  Glüht beim Einstecken auf (Martin Neumann)

Re: [gentoo-user] is multi-core really worth it?

[David Haller]

Hello,

On Wed, 22 Nov 2017, Jeremi Piotrowski wrote:
>That being said: if you do a world rebuild you will have lots of packages

that (even without the fetch) spend 40 seconds setting up the emerge
(unpack, prepare (plus eautoreconf))...

>that spend ~40 seconds doing their autoconf run, only to build 2-3 sources
>files.

Well, yes, autoreconf can be slow, configure itself less, but as one
writing both .spec files for rpms and .ebuilds, autotools is _by far_
the best both from a users and a packagers view.

Where's e.g. cmake's equvialent to "./configure --help"? Take defaults
or dig through whatnot spreadout cmake-files to find what -D arguments
to use... *bwah* Or qmake. Or whatnot. Yes. autotools sucks. A lot. 
But it is transparent and easily hackable[1], and the rest sucks a lot
worse.

-dnh

[1] do a "grep" on 'sed .. configure{,.ac,.in}' in ebuilds and patches
in ${FILESDIR}/ to configure{,.ac,.in} in the portage tree, and/or
sedding/patching *.m4 files ... That's definitely "hacking" the
autotools build process which should not be done (a autotools
tarball is "selfcontained"). But it's so easy, it is done quite
often, instead of patching up the result, moving files around etc. 
So easy and convenient, there's even "eautoreconf", even though
that takes (single-threaded) time to run. But rightly so. Upstream
is "notoriously" bad... The hoops you have to jump through for a
clean package with autotools are "walkthroughs" compared to the
stuff needed for cmake or qmake or ... I recently just gave up on
a (qmake) package, I just could not find what *.prf, *.pri
*whatnot file I needed to patch up to do as I needed... And I'm
not a newbie at patching that stuff.

And if you're adventurous, go and try patch up libreoffice or
mozillen ... ;) It's a pain. (BTW: I'm on the former, patching out
a lot of deps, like, when was the last time, if ever, you needed
to get a WordPerfect or Zoner Draw file to load?) Yes, all nicely
"useflagged", 406 lines of -U0 diff ATM with a 576 lines original
ebuild, but not ready yet ;) And yes, I'm even on the
"libstaroffice" stuff, and I _do_ have some of those file to load
and convert for myself...

-- 
>Someone from the society for prevention of cruelty to paper is sure to come
>after me if I suggest ramming it up a HR droid's ass.
Actually, it's that Lumber Cartel (TINLC) Directive (TAND) regarding
inappropriate use of their products.   -- in asr

Re: [gentoo-user] is multi-core really worth it?

[J. Roeleveld]

On Wednesday, November 22, 2017 8:48:08 AM CET David Haller wrote:
> Hello,
> 
> On Wed, 22 Nov 2017, Raffaele Belardi wrote:
> >rebuilding system and world with gcc-7.2.0 on a 6-core AMD CPU I have
> >the impression that most of the ebuilds limit parallel builds to 1, 2
> >or 3 threads.
> 
> Most should build with "many" cores, but some might be limited, but ...
> 
> >I'm aware it is only an impression, I did not spend the
> >night monitoring the process, but nevertheless every time I checked
> >the load was very low.
> 
> I guess you're mostly observing stalls because the (single-thread)
> overhead of emerge, configure (etc.), which can also be IO-bound...
> The actual builds probably goes so fast on your box, you hardly
> notice unless they're rather big(gish).
> 
> Some numbers for comparison, and I generally build with just one
> thread in the background on my 2 core "old" Athlon II X2 250:
> dev-libs/boost took a good 45mins to build with one thread. Divide by
> 6 for your cores, divide again by about 2-3 for your newer cores and
> you're in the 3:45min range for such a "biggie" as boost. Most other
> stuff won't even peg your meter.
> 
> So, your emerge is mostly IO and "emerge"-threads bound. Solution:
> adjust your build-threads[1], and then adjust your emerge jobs! See
> '--jobs' in 'man emerge'. Can't find where to set a default in a
> config-file for that ATM, but you could always alias/function/script
> emerge to something else, e.g. put this in your ~/.bashrc:

Instead of all these aliases, simply edit your make.conf ( /etc/portage/
make.conf )

And edit (or add) the following lines:
MAKEOPTS="--jobs 24 --load-average 48"
EMERGE_DEFAULT_OPTS="--jobs 24 --load-average 48"

Adjust the values to match your system, the above works fine on my desktop.
CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
And 32GB ram.

--
Joost

Re: [gentoo-user] is multi-core really worth it?

[Neil Bothwick]

On Wed, 22 Nov 2017 08:26:01 +0100, Jeremi Piotrowski wrote:

> That being said: if you do a world rebuild you will have lots of
> packages that spend ~40 seconds doing their autoconf run, only to build
> 2-3 sources files. On an 8-core machine at work, I get good results
> using parallel emerge jobs (emerge -jX). For your 6-core AMD CPU
> (assuming it actually has 12 threads) I'd start with 'emerge -j3' and
> MAKEOPTS='-j12 -l16'. That should get you a nice speedup, but may
> require a bit more ram.

Instead of specifying a number of jobs with "-j 3" you can let portage
work it out. On a similar system, I use 

EMERGE_DEFAULT_OPTS="--jobs --load-average 12"


-- 
Neil Bothwick

"An investment in knowledge always pays the best interest." - Benjamin
Franklin


pgpASw8EtmgsC.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] is multi-core really worth it?

[David Haller]

Hello,

On Wed, 22 Nov 2017, Raffaele Belardi wrote:
>rebuilding system and world with gcc-7.2.0 on a 6-core AMD CPU I have
>the impression that most of the ebuilds limit parallel builds to 1, 2
>or 3 threads.

Most should build with "many" cores, but some might be limited, but ...

>I'm aware it is only an impression, I did not spend the
>night monitoring the process, but nevertheless every time I checked
>the load was very low.

I guess you're mostly observing stalls because the (single-thread)
overhead of emerge, configure (etc.), which can also be IO-bound... 
The actual builds probably goes so fast on your box, you hardly
notice unless they're rather big(gish).

Some numbers for comparison, and I generally build with just one
thread in the background on my 2 core "old" Athlon II X2 250:
dev-libs/boost took a good 45mins to build with one thread. Divide by
6 for your cores, divide again by about 2-3 for your newer cores and
you're in the 3:45min range for such a "biggie" as boost. Most other
stuff won't even peg your meter.

So, your emerge is mostly IO and "emerge"-threads bound. Solution:
adjust your build-threads[1], and then adjust your emerge jobs! See
'--jobs' in 'man emerge'. Can't find where to set a default in a
config-file for that ATM, but you could always alias/function/script
emerge to something else, e.g. put this in your ~/.bashrc:


eworld() {
emerge --jobs 7 [..whatever_you_use_for_world..] @world
}


or use a "proper" script in ~/bin/ for that purpose.

You might want to "tune down" your "make" threads in make.conf though,
as your'll be building 7 packages in parallell ... Well, best try it
out, and have a guess at a sweet spot between emerge-jobs and
make-jobs. I'd hazard a guess of [2-4] emerge- and [3-5] make-jobs for
your six core should be nice.

Just experiment with 'emerge -j [1..7] ... @world'...

HTH,
-dnh

PS:[lines rebroken]


# type eworld 
eworld is a function
eworld () 
{ 
emerge --verbose --pretend --update --changed-use \
--newuse --deep --with-bdeps y "$@" @world;
beep -r 1 -l 50
}


I usually call it as: eworld -t [--verbose-conflicts].

I then "pick and build" from the result (from the bottom up, due to
"--tree"). Even with ccache, it'd be rather stupid starting a
libreoffice, firefox or icedtea build that'll take hours when I have
to leave in 10min and won't keep the box running...

[1] I think that's good practice when doing multithreaded build on
multicore: add (at least) one for IO. THREADS="$(( $N_CORES+1 ))"
You can add that in your make.conf and feed $THREADS to
MAKEOPTS="-j ${THREADS}" etc.

-- 
"I have always felt that violence was the last refuge of the incompetent,
 and emtpy threats the final sanctuary of the terminally inept."
-- the Marquis de Carabas

Re: [gentoo-user] is multi-core really worth it?

[Jeremi Piotrowski]

On Wed, Nov 22, 2017 at 07:52:54AM +0100, Raffaele Belardi wrote:
> Hi,
> 
> rebuilding system and world with gcc-7.2.0 on a 6-core AMD CPU I have the 
> impression that
> most of the ebuilds limit parallel builds to 1, 2 or 3 threads. I'm aware it 
> is only an
> impression, I did not spend the night monitoring the process, but 
> nevertheless every time
> I checked the load was very low.

multi-core is definitely worth it. Any non-trivial package
(wine,glibc,gcc,llvm,qt off the top of my head) will spend a long time
building many files that can be built in parallel with multiple make jobs.

That being said: if you do a world rebuild you will have lots of packages
that spend ~40 seconds doing their autoconf run, only to build 2-3 sources
files. On an 8-core machine at work, I get good results using parallel
emerge jobs (emerge -jX). For your 6-core AMD CPU (assuming it actually
has 12 threads) I'd start with 'emerge -j3' and MAKEOPTS='-j12 -l16'.
That should get you a nice speedup, but may require a bit more ram.

Re: [gentoo-user] is multi-core really worth it?

[R0b0t1]

Hello,

On Wed, Nov 22, 2017 at 12:52 AM, Raffaele Belardi
 wrote:
> Hi,
>
> rebuilding system and world with gcc-7.2.0 on a 6-core AMD CPU I have the 
> impression that
> most of the ebuilds limit parallel builds to 1, 2 or 3 threads. I'm aware it 
> is only an
> impression, I did not spend the night monitoring the process, but 
> nevertheless every time
> I checked the load was very low.
>

Assuming all of your compilation is on a RAM disk, there are two main
bottlenecks that are easy to spot: network access (downloading new
packages) and dependency chokepoints (packages must be compiled in a
chain). Other potential chokepoints like disk access are negligible in
my experience, though for one merge I did have two or three ebuilds
fighting for disk IO "lock up" a system.

If all dependencies have been satisfied on your system I invite you to
merge a bunch of packages at once (@world?), you should notice greater
parallelism.

> Does anyone have real-world statistics of CPU usage based on gentoo world 
> build?
>

I've considered ways to gather these statistics off and on over the
years, but it is easy to start sinking a lot of time into it. It is
possible the data you want exists, but I have not found any extant
solution involving portage that provides that detail of logging.*

Cheers,
 R0b0t1


* Someone will prove me wrong in 3... 2... 1...

[gentoo-user] is multi-core really worth it?

[Raffaele Belardi]

Hi,

rebuilding system and world with gcc-7.2.0 on a 6-core AMD CPU I have the 
impression that
most of the ebuilds limit parallel builds to 1, 2 or 3 threads. I'm aware it is 
only an
impression, I did not spend the night monitoring the process, but nevertheless 
every time
I checked the load was very low.

Does anyone have real-world statistics of CPU usage based on gentoo world build?

raffaele