Re: [gentoo-user] per package parallel build

[Khaosgrille]

Hi,

since some people recommended distcc i want to add icecream (or icecc) by SUSE 
https://github.com/icecc/icecream

It is based on distcc but can automaticly schedule things, do some networking 
and is system agnostic. Since it works perfectly fine with portage i have it as 
my default. When i dont have access to my compile machines it just uses my 
laptop and works fine.

Another thing that speeds up compile time significantly is putting your tmpdir 
in the RAM. https://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs

Khaosgrille

publickey - Khaosgrille@protonmail.com - 0xE78BC986.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: [gentoo-user] per package parallel build

[Neil Bothwick]

On Tue, 3 Dec 2019 16:26:58 -0500, John Blinka wrote:

> > But if you emerge --update libreoffice before the package that is
> > forcing the rebuild, why would libreoffice rebuild? I would expect it
> > to only rebuild libreoffice after the dependency had been changed.  
> 
> That’s exactly what happened.  I issued an emerge -DuNv —changed-deps
> libreoffice first.  That had the effect of 1) first upgrading several
> libreoffice dependencies, and 2) subsequently rebuilding libreoffice
> once the dependencies changed.  I’m guessing emerge is smart enough to
> trigger both activities and sequence them appropriately.  Operative
> word is guess - I don’t pretend to understand the inner workings.

That's good to know - that it works, i mean, not that you don't
understand how ;-)
> 
> > I'm not saying out wouldn't work some of the time, but I can see
> > situations where it wouldn't. Whereas
> >
> > emerge --opts @world --exclude memory-hogs...
> > emerge --opts --jobs 1 @world
> >
> > should always isolate them.  
> 
> Agreed that this technique should always work.  And it has the
> advantage of fewer invocations of emerge.  But at the cost, I suspect,
> of serializing the building of any memory hog dependencies that were
> excluded from consideration by the first invocation of emerge.

That's a fair point, but it would only apply if the hog had a new
dependency. Even then, it would only be significant if the dependency was
large.

> What situations do you see as not working?

The ones I mentioned before, but it seems that portage may be intelligent
to deal with them. It will be interesting to see if it continues to work
over time.


-- 
Neil Bothwick

Q: Why do PCs - even modern ones - have reset buttons on the front?
A: Because they come with Microsoft operating systems.


pgperF5jy_u5d.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] per package parallel build

[John Blinka]

On Tue, Dec 3, 2019 at 3:46 PM Neil Bothwick  wrote:

> On Tue, 3 Dec 2019 10:43:36 -0500, John Blinka wrote:
>
> > > > Couldn't you just have a script that "emerge --update"s each
> > > > package in sequence? If the package isn't due for update nothing
> > > > will happen. And then you could follow that with an "emerge world"
> > > > knowing that your hogs are already done.
> > >
> > > Sometimes the packages are rebuilt without an update, especially if
> > > you use --changed-use or --changed-deps, so it's not quite that
> > > simple.
> >
> >
> > But still pretty simple.  I’ve just used the “build in sequence” idea
> > for an update that forced a libreoffice rebuild.  It first upgraded a
> > few of libreoffice’s dependencies in parallel, and then rebuilt
> > libreoffice by itself afterwards. A subsequent emerge @world upgraded a
> > bunch of minor kde stuff.  I like this idea - seems to isolate the
> > “hogs” so they build one at a time, and it does so without any
> > intervention on my part.  Thanks!
>
> But if you emerge --update libreoffice before the package that is forcing
> the rebuild, why would libreoffice rebuild? I would expect it to only
> rebuild libreoffice after the dependency had been changed.


That’s exactly what happened.  I issued an emerge -DuNv —changed-deps
libreoffice first.  That had the effect of 1) first upgrading several
libreoffice dependencies, and 2) subsequently rebuilding libreoffice once
the dependencies changed.  I’m guessing emerge is smart enough to trigger
both activities and sequence them appropriately.  Operative word is guess -
I don’t pretend to understand the inner workings.

>
> I'm not saying out wouldn't work some of the time, but I can see
> situations where it wouldn't. Whereas
>
> emerge --opts @world --exclude memory-hogs...
> emerge --opts --jobs 1 @world
>
> should always isolate them.


Agreed that this technique should always work.  And it has the advantage of
fewer invocations of emerge.  But at the cost, I suspect, of serializing
the building of any memory hog dependencies that were excluded from
consideration by the first invocation of emerge.

What situations do you see as not working?

John Blinka

Re: [gentoo-user] per package parallel build

[Neil Bothwick]

On Tue, 3 Dec 2019 10:43:36 -0500, John Blinka wrote:

> > > Couldn't you just have a script that "emerge --update"s each
> > > package in sequence? If the package isn't due for update nothing
> > > will happen. And then you could follow that with an "emerge world"
> > > knowing that your hogs are already done.  
> >
> > Sometimes the packages are rebuilt without an update, especially if
> > you use --changed-use or --changed-deps, so it's not quite that
> > simple.  
> 
> 
> But still pretty simple.  I’ve just used the “build in sequence” idea
> for an update that forced a libreoffice rebuild.  It first upgraded a
> few of libreoffice’s dependencies in parallel, and then rebuilt
> libreoffice by itself afterwards. A subsequent emerge @world upgraded a
> bunch of minor kde stuff.  I like this idea - seems to isolate the
> “hogs” so they build one at a time, and it does so without any
> intervention on my part.  Thanks!

But if you emerge --update libreoffice before the package that is forcing
the rebuild, why would libreoffice rebuild? I would expect it to only
rebuild libreoffice after the dependency had been changed.

I'm not saying out wouldn't work some of the time, but I can see
situations where it wouldn't. Whereas 

emerge --opts @world --exclude memory-hogs...
emerge --opts --jobs 1 @world

should always isolate them.


-- 
Neil Bothwick

Hello, this is an extension to the famous signature virus, called spymail.
Could you please copy me into your signature and send back what you were
doing last night between 10pm and 3am?


pgp5pCpb7diPU.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] Encfs and mlocate

[tuxic]

karl:
Doesn't work here. Your $HOME is encryoted (see my initial mail)?

I can stat/md5sum the database file. But even a locate right 
in the directory containing the database file fails with

# lib/mlocate>l
total 1092
-rw--- 1 "uid"  users 1115645 2019-12-03 20:12 mlocate.db
solfire:lib/mlocate>locate -d mlocate.db test
locate: can not stat () `mlocate.db': Permission denied
[1]19339 exit 1 locate -d mlocate.db test

Regards,
Meino 

On 12/02 11:34, k...@aspodata.se wrote:
> mcc:
> > I want to index the contents of a ecryptfs-ed directory tree with
> > mlocate storing the resulting db in that directory tree
> 
> I do like this:
> 
>   Nightly run:
> PRUNEPATHS=$HOME/tmp
> updatedb -l 0 -U $HOME -o $HOME/.updatedb
> chmod 600 $HOME/.updatedb
> 
>   To find:
> locate -d $HOME/.updatedb "$@"
> 
> Works without problem here.
> 
> ...
> > Reading its contents with
> > 
> > locate -d var/lib/mlocate/mlocate.db whattofind
> > 
> > results in 
> > 
> > locate: can not stat () `var/lib/mlocate/mlocate.db': Permission denied
> > 
> > the file shows this permission:
> > 
> > -rw-r--r-- 1 me users 1319628 2019-12-02 04:25 var/lib/mlocate/mlocate.db
> > 
> > . Why can I create a file, which afterwards I am not allowed to read?
> 
> Even if you cannot read a file you should be able to stat it,
> like e.g. stat /etc/shadow.
> 
> It can be some problem with directory permissions, try to cd to your 
> var/lib/mlocate directory and then do a locate -d mlocate.db ...
> 
> Can you read your db file with something else, e.g. md5sum,
> to verify that you can read it in some ways.
> 
> Regards,
> /Karl Hammar
> 
> 
> 

Re: [gentoo-user] per package parallel build

[Wols Lists]

On 03/12/19 15:43, John Blinka wrote:
> 
> 
> On Sat, Nov 30, 2019 at 5:35 PM Neil Bothwick  > wrote:
> 
> On Sat, 30 Nov 2019 16:47:35 +, Wols Lists wrote:
> 
> > > There's no need to mess around adding and removing masks, just use
> > > the -
> > > - exclude option.
> > >
> > > Yep!  For some reason, that option doesn’t always occur to me, but
> > > that’s clearly a simpler way to do it.  Thanks for reminding me!
> > >   
> > Couldn't you just have a script that "emerge --update"s each
> package in
> > sequence? If the package isn't due for update nothing will happen. And
> > then you could follow that with an "emerge world" knowing that
> your hogs
> > are already done.
> 
> Sometimes the packages are rebuilt without an update, especially if you
> use --changed-use or --changed-deps, so it's not quite that simple.
> 
> 
> But still pretty simple.  I’ve just used the “build in sequence” idea
> for an update that forced a libreoffice rebuild.  It first upgraded a
> few of libreoffice’s dependencies in parallel, and then rebuilt
> libreoffice by itself afterwards. A subsequent emerge @world upgraded a
> bunch of minor kde stuff.  I like this idea - seems to isolate the
> “hogs” so they build one at a time, and it does so without any
> intervention on my part.  Thanks!
> 
Eggsackerley

If you do an "emerge memory-hog --normal-options" (whatever your normal
options are), then when you do an "emerge world" with those same
options, it should not see the need to emerge said memory-hog.

Cheers,
Wol

Re: [gentoo-user] per package parallel build

[John Blinka]

On Sat, Nov 30, 2019 at 5:35 PM Neil Bothwick  wrote:

> On Sat, 30 Nov 2019 16:47:35 +, Wols Lists wrote:
>
> > > There's no need to mess around adding and removing masks, just use
> > > the -
> > > - exclude option.
> > >
> > > Yep!  For some reason, that option doesn’t always occur to me, but
> > > that’s clearly a simpler way to do it.  Thanks for reminding me!
> > >
> > Couldn't you just have a script that "emerge --update"s each package in
> > sequence? If the package isn't due for update nothing will happen. And
> > then you could follow that with an "emerge world" knowing that your hogs
> > are already done.
>
> Sometimes the packages are rebuilt without an update, especially if you
> use --changed-use or --changed-deps, so it's not quite that simple.


But still pretty simple.  I’ve just used the “build in sequence” idea for
an update that forced a libreoffice rebuild.  It first upgraded a few of
libreoffice’s dependencies in parallel, and then rebuilt libreoffice by
itself afterwards. A subsequent emerge @world upgraded a bunch of minor kde
stuff.  I like this idea - seems to isolate the “hogs” so they build one at
a time, and it does so without any intervention on my part.  Thanks!

John Blinka