Re: Proposal: Build timers

[Liliana Marie Prikler]

Am Donnerstag, den 25.11.2021, 04:03 + schrieb Jacob Hrbek:
> Make it clear it's an estimate, or maybe even abstract away the time
> units so that there is no expectation of any particular time. --
> Vagrant
> 
> My theory is designed with tolerance of <5 min with max tolerance of
> =10 min with methods that I am confident will get us within <30 sec
> assuming sufficient amount of data to construct the variables.
You are courageous to assume a variance of 5 minutes, let alone 30
seconds in everyday systems.  Aside from the variables I've already
pointed out, you'd have to consider that certain builds outside of CI
will enter a state of constant swapping at some point, yet still finish
building if not hitting OOM limits.  Also the fact that Guix might not
be the only process hogging resources on some particular machine.

Re: Proposal: Build timers

[Jacob Hrbek]

Make it clear it's an estimate, or maybe even abstract away the time units so 
that there is no expectation of any particular time. -- Vagrant

My theory is designed with tolerance of <5 min with max tolerance of =10 min 
with methods that I am confident will get us within <30 sec assuming sufficient 
amount of data to construct the variables.

-- Jacob "Kreyren" Hrbek

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐

On Wednesday, November 24th, 2021 at 8:23 PM, Vagrant Cascadian 
 wrote:

> On 2021-11-24, zimoun wrote:
> 

> > On Tue, 23 Nov 2021 at 18:50, Julien Lepiller jul...@lepiller.eu wrote:
> > 

> > > Do we even care that much about accuracy? I don't really care that the
> > > 

> > > build takes 30 or 31 seconds, or even 1 minute, but I certainly care
> > > 

> > > whether it takes 30s or 3h. I think this is also what SBUs give you: a
> > > 

> > > rough estimate of which build is longer than the other. I think a
> > > 

> > > simple proportionality relation would work well enough in most common
> > > 

> > > cases. It might be quite off on a super computer, but who cares,
> > > 

> > > really?
> > 

> > What if it takes 3h and the prediction says 2h?
> 

> Those sound about "the same" for any kind of reasonable expectation...
> 

> I would guess you only want the correct order of magnitude... hours,
> 

> minutes, days, weeks, months, years... or maybe quick, fast, slow,
> 

> painful.
> 

> I do this soft of fuzzy estimation all the time when working on
> 

> Reproducible Builds in Debian; look at the past test history to get a
> 

> rough estimate of how long I might expect a build to take. This helps
> 

> me decide if I should start a build and get a $COFFEE, do some
> 

> $SWORDFIGHTING on the $OFFICECHAIRS, or sit and watch the progress bar
> 

> so I don't loose the mental state working on the problem becuase it will
> 

> be done $SOON.
> 

> Make it clear it's an estimate, or maybe even abstract away the time
> 

> units so that there is no expectation of any particular time.
> 

> I know there are people who would love to get a a value that was
> 

> consistently right but to be useful it only needs an estimate to be
> 

> mostly not completely wrong. At least to me. :)
> 

> live well,
> 

> vagrant

publickey - kreyren@rixotstudio.cz - 0x1677DB82.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: Proposal: Build timers

[Jacob Hrbek]

> The “pokémon-battle” model is a simple linear model
(cross-multiplication); using Jacob’s “notation” -- zimoun

It's not designed to be linear as the HP variable could be adjusted in real 
time e.g. recalculating it every X amount of time as needed which can include 
calculations for noise that influences the task.

It currently seems as linear as I developed it to be a workable platform on 
which we can develop more robust solution in case the simple linear 
calculations are not sufficiently accurate (which i think that it will be if we 
get sufficient amount of data to calculate it).

>  - HP: time to build on machine A  -- zimoun

Not time, but **COMPLEXITY** of the package as I see that as an important 
destiction since it's by design never meant to store time, but "time value" 
**that is converted in time**.

> based on some experiments.  Last, on machine B, knowing both nthread' and 
> cpufreq' for that machine B, you are expecting to evaluate HP' for that 
> machine B applying the formula:
>   HP' = a * nthread' * cpufreq' + b -- zimoun

In this context I would describe it as:

CPU strenght = nthread * cpufreq * "other things that make the CPU to deal more 
damage"
HP = "CPU strenght" * "time it took to build in sec"

Which is linear, but the components used to figure out this linear function are 
non-linear e.g. "RAM memory" will most likely  appear as exponential, but it 
will eventually hit constant when the CPU's requirement for the memory are 
satisfied.

Also the calculation should never contain systems values from systems a,b,c,.. 
, rather interpreting the hardware resources into an equasion that should be 
integrated to calculate unknowns

where the issue in that theory is figuring out the "time it took to build" and 
"CPU strenght" which i think can be mitigated by determining how the hardware 
affects the build by changing it's variables in two builds to determine e.g.

4 thread = 5 min
3 threads = 10 min
2 threads = 15 min

-> 1 threads will take 20 min.

So literally following a pokemon battle system alike:

Pokemon with 100 HP and you dealing 10 HP per turn -> it will take you 10 turns 
to win the battle.

---

Btw. The components capable of bottleneck such as the amount of RAM memory 
should be probably calculated as <0.0,1.0> so that they can be applied as 
**multipliers** to the "CPU strenght" since (following "the rule of 
compilation" using 2 GB of RAM per 1 thread) the CPU will function in a 
scenario of 4 threads with 4 GB of RAM function at half the efficiency (0.5) if 
it's requirements for fast memory are not satisfied.

-- Jacob "Kreyren" Hrbek

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐

On Wednesday, November 24th, 2021 at 11:35 AM, zimoun 
 wrote:

> Hi,
> 

> On Tue, 23 Nov 2021 at 14:39, Jacob Hrbek krey...@rixotstudio.cz wrote:
> 

> > > This approximation would not even be accurate enough for the same
> > > 

> > > machine. For instance, the test suite of the julia package runs
> > > 

> > > mainly sequential using one thread...
> > 

> > I am aware of this scenario and I adapted the equasion for it, but I
> > 

> > recognize that this exponentially increases the inaccuracy with more
> > 

> > threads and I don't believe that there is a mathematical way with the
> > 

> > provided values to handle that scenario so we would have to adjust the
> > 

> > calculation for those packages.
> 

> What I am trying to explain is that the model cannot work to be
> 

> predictable enough with what I consider a meaningful accuracy.
> 

> Obviously, relaxing the precision, it is easy to infer a rule of thumb;
> 

> a simple cross-multiplication fits the job. ;-)
> 

> The “pokémon-battle” model is a simple linear model
> 

> (cross-multiplication); using Jacob’s “notation”:
> 

> -   HP: time to build on machine A
> -   DPS = nthread * cpufreq : “power” of machine
> 

> Then it is expected to evaluate ’a’ and ’b’ on average such that:
> 

> HP = a * DPS + b
> 

> based on some experiments. Last, on machine B, knowing both nthread'
> 

> and cpufreq' for that machine B, you are expecting to evaluate HP' for
> 

> that machine B applying the formula:
> 

> HP' = a * nthread' * cpufreq' + b
> 

> Jacob, do I correctly understand the model?
> 

> In any case, that’s what LFS is doing, instead HP is named SBU. And
> 

> instead DPS, they use a reference package. And this normalization is
> 

> better, IMHO. Other said, for one specific package considered as
> 

> reference, they compute HP1 (resp. HP2) for machine A (resp. B), then
> 

> for machine A, they know HP for another package and they deduce,
> 

> HP' = HP2/HP1 * HP
> 

> All this is trivial. :-) The key is the accuracy, i.e., the error
> 

> between the prediction HP' and the real time. Here, the issue is that
> 

> HP1 and HP2 capture for one specific package the overall 

Re: Derivations differ between computers?

[Leo Famulari]

On Wed, Nov 24, 2021 at 08:55:31PM -0500, Leo Famulari wrote:
> Computer A:
> --
> $ guix time-machine --commit=c70eadeaed9367e45be3797a17d3a34e5f8f67cd -- 
> build --no-grafts hello -d
> /gnu/store/260bk0ch4np4h2yz5yqhf8hjbsyhwpmr-hello-2.10.drv
> --
> 
> Computer B:
> --
> $ guix time-machine --commit=c70eadeaed9367e45be3797a17d3a34e5f8f67cd -- 
> build --no-grafts hello -d
> /gnu/store/vs34g0npra395dsqyplrjrwlm417a1w4-hello-2.10.drv
> --

It's caused by $GUIX_DOWNLOAD_FALLBACK_TEST=none

I noticed by "climbing up" the dependency graph, comparing derivations
until I got to the XZ bootstrap seed derivation on computer B:

--
Derive([("out","/gnu/store/qc9b01x31ayxb36r0zw5cw28awisdq98-xz","r:sha256","1324eb9f74026653255d3a44da3103318e3b8e07f577115d7fa5ae1362eb3345")],[],["/gnu/store/3q889p5zj9kdnl7wfs8dah8wzxvq6j8j-no-content-addressed-mirrors","/gnu/store/chgbsnxxi0032ikjgwi14asmkm792yla-no-disarchive-mirrors","/gnu/store/hdprrnsaa40vl1k63hnq7l1f012jzq2y-mirrors"],"x86_64-linux","builtin:download",[],[("content-addressed-mirrors","/gnu/store/3q889p5zj9kdnl7wfs8dah8wzxvq6j8j-no-content-addressed-mirrors"),("disarchive-mirrors","/gnu/store/chgbsnxxi0032ikjgwi14asmkm792yla-no-disarchive-mirrors"),("executable","1"),("impureEnvVars","http_proxy
 https_proxy LC_ALL LC_MESSAGES LANG 
COLUMNS"),("mirrors","/gnu/store/hdprrnsaa40vl1k63hnq7l1f012jzq2y-mirrors"),("out","/gnu/store/qc9b01x31ayxb36r0zw5cw28awisdq98-xz"),("preferLocalBuild","1"),("url","(\"https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/i686-linux/xz?id=44f07d1dc6806e97c4e9ee3e6be883cc59dc666e\;
 
\"https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/xz?id=44f07d1dc6806e97c4e9ee3e6be883cc59dc666e\;
 
\"http://flashner.co.il/guix/bootstrap/i686-linux/xz?id=44f07d1dc6806e97c4e9ee3e6be883cc59dc666e\;
 
\"http://lilypond.org/janneke/guix/i686-linux/xz?id=44f07d1dc6806e97c4e9ee3e6be883cc59dc666e\;)")])
--

It includes all those empty mirror lists, which computer A does not
refer to.

The issue went away when I unset GUIX_DOWNLOAD_FALLBACK_TEST


signature.asc
Description: PGP signature

‘core-updates-frozen’: julia status

[zimoun]

Hi,

On Mon, 22 Nov 2021 at 15:13, Maxim Cournoyer  wrote:
> zimoun  writes:

>> Julia is annoying,  Because the test suite sometimes passes, sometimes
>> not.  Well, I am not even sure it is the same part that fails.  I
>> notice that "guix build julia --without-tests=julia" then "guix build
>> julia" tends to works. :-)
>>
>> Any idea to debug this?
>
> No idea how to debug this, but when faced with nondeterministic tests,
> here's how I proceed:
>
> 1. Report them upstream.
> 2. Disable the problematic tests.  You may need to disable a whole test suite 
> of
> similarly faulted tests, if they all behave the same.

After this hacking day, Julia builds deterministically* on
core-updates-frozen.  Along the way, now the test suite runs in parallel
which reduces a lot the feedback loop. :-) And a variant of the current
patch of julia source is upstream PR.

*deterministically: at least on 2 machines. ;-)

One or two failing tests randomly failing should be investigated and
probably reported upstream.

The test suite of julia-* packages supports also (more or less) some
parallelism.  From my understanding, the limitation comes from the
script 'test/runtests.jl' from each package; each implements more or
less some parallelism.  Now, the environment variable JULIA_CPU_THREADS
is set but Julia does not always honor it as its doc specifies – see the
patch fixing the parallelism of Julia test suite ;-).  Therefore, the
test suite of each package is launched with
'--procs=(parallel-job-count)' and Julia exploits at best the available
workers.

I am currently rebuilding all the julia-* packages.  It is high probable
some are now broken.  But it appears to me not a blocker for the merge.

On a side note, Julia is not currently reproducible [1] and
'julia-build-system' neither [2].  Therefore, probable non-reproducible
build introduced by parallelism is currently not the point, IMHO.



Thanks Maxim for the help and the good vibes to tackle that.

Cheers,
simon

1: 
2: 

Re: Proposal: Build timers

[zimoun]

Hi Vagrant,

On Wed, 24 Nov 2021 at 12:23, Vagrant Cascadian  wrote:
> On 2021-11-24, zimoun wrote:

>> What if it takes 3h and the prediction says 2h?
>
> Those sound about "the same" for any kind of reasonable expectation...

Ah, we are not speaking about the same thing thus. :-)


> I would guess you only want the correct order of magnitude... hours,
> minutes, days, weeks, months, years... or maybe quick, fast, slow,
> painful.

Well, an order of magnitude is relative to an expectation.  My engineer
side is fine with a low expectation: it should take 2h and it
effectively takes 6h (probably unhappy) or the contrary it should take
6h and it effectively takes 2h (really happy).  My scientist side is
less fine about this poorly defined expectation.  Anyway! :-)

I think it is easier to set quick, fast, slow, courage based on timings
from Berlin.  Similarly with master, staging, core-updates which set a
rough number of packages for package impact, why not have:

 - fast: t < X
 - quick: X < t < 3X
 - fast: 3X < t < 6X
 - slow: 6X < t < 36X
 - courage: 36X < t

where X could be arbitrarily picked as 10min on Berlin or Bayfront.
This data could be exposed with the package, displayed by Cuirass or the
Data Service or the website [1].  Well, all require some work though.

(fast = less couples of minutes, quick = less than half-hour, fast =
less than hour, slow = less than six hours, courage = wait for it; the
reference is Bayfront or Berlin, with a clear hardware specifications =
number of core/thread, cpufreq and probably couple of other relevant
parameters)

1: 


> I do this soft of fuzzy estimation all the time when working on
> Reproducible Builds in Debian; look at the past test history to get a
> *rough* estimate of how long I might expect a build to take. This helps
> me decide if I should start a build and get a $COFFEE, do some
> $SWORDFIGHTING on the $OFFICECHAIRS, or sit and watch the progress bar
> so I don't loose the mental state working on the problem becuase it will
> be done $SOON.

Yeah, me too. :-) I, more than often, do back-to-envelope computations
to estimate something.  From my point of view, there is a difference
between my personal estimation and a public official estimation.


> Make it clear it's an estimate, or maybe even abstract away the time
> units so that there is no expectation of any particular time.

I agree.  My point is: if the estimation providing a (even rough)
duration is not accurate enough, then it is not trusted (by user), i.e.,
not used; and all the effort does not worth it, IMHO.

Let back this claim by the example of ’relevance’ from «guix
search». :-) Because the accuracy, regarding the user expectations from
a query, is highly variable, the major behaviour I see is: iteration
over “guix package -A | grep” trying various package name.


Cheers,
simon

Re: default tar format for "make dist" and patch file length

[Vagrant Cascadian]

On 2021-11-19, Vagrant Cascadian wrote:
> On 2021-11-19, Philip McGrath wrote:
>> On 11/19/21 09:54, Ludovic Courtès wrote:
>>> Vagrant Cascadian  skribis:
 So, I guess I'm leaning towards making the guix lint check a little more
 lenient.

 Thoughts?
>>> 
>>> That sounds even better, I’m all for it (changing (guix lint) + fixing
>>> the two remaining issues)!
>
> Submitted the guix lint change as https://issues.guix.gnu.org/51775

Er, I meant: https://issues.guix.gnu.org/51985


>> It might also help to change the warning given by the check.
>>
>> When a program called "lint" tells me that something is too long, I 
>> understand that to mean that what I've done is generally considered bad 
>> style, but there might be a very good reason to do it in some specific 
>> case. For example, I might exceed a line length guideline to avoid 
>> inserting linebreaks into a URL.
>
> That's a good point!
>
>
>> If instead `guix lint` is telling us about a hard limit that will break 
>> things, I think it should say so clearly.
>
> Not sure how to convey succinctly, but here's an attempt at a patch
> (which ironically also probably makes the line a bit too long in the
> code):
>
> diff --git a/guix/lint.scm b/guix/lint.scm
> index ac2e7b3841..6464fb751a 100644
> --- a/guix/lint.scm
> +++ b/guix/lint.scm
> @@ -968,7 +968,7 @@ (define (starts-with-package-name? file-name)
>   max)
>(make-warning
> package
> -   (G_ "~a: file name is too long")
> +   (G_ "~a: file name is too long and may break 
> release tarball generation")
> (list (basename patch))
> #:field 'patch-file-names)
>#f))
> diff --git a/tests/lint.scm b/tests/lint.scm
> index 9a91dd5426..d4c3d62aaf 100644
> --- a/tests/lint.scm
> +++ b/tests/lint.scm
> @@ -509,7 +509,7 @@ (define hsab (string-append (assoc-ref inputs
> "hsab")
>  (test-equal "patches: file name too long"
>(string-append "x-"
>   (make-string 100 #\a)
> - ".patch: file name is too long")
> + ".patch: file name is too long and may break release 
> tarball generation")
>(single-lint-warning-message
> (let ((pkg (dummy-package
> "x"


live well,
  vagrant


signature.asc
Description: PGP signature

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[zimoun]

Hi Florian,

On Wed, 24 Nov 2021 at 15:40, "pelzflorian (Florian Pelz)" 
 wrote:

>>> I don't know if that convinces maintainers to change decisions.
>>
>> This decision is consistent with the analysis [1] done by Software
>> Conservancy Freedom, at least.
>
> I did not speak about one decision.

About which decision are you talking about?  I am sorry if I have
misread or misunderstood.  From my readings, the second part of that
thread is an appeal about previous clear decisions:

 1. distributing source of ZFS
 2. not-distributing substitutes of ZFS

Both are consistent with the legal analysis of SFC [1].

1: 


> What I meant is that maybe Denis argued “dynamic linking creates a
> derivative work” if and only if “ZFS source code is a derivative
> work of Linux”.

I have no opinion; because IANAL.  As Jelle jokingly said on IRC [2], it
is a typical WANAX session*. :-)  As I suggested here [3], because WANAL,
I do not understand what we are discussing and on which legal basis this
discussion tries to appeal the decision made by Guix long time ago about
ZFS based on [1], among other things.

The appeal cannot happen here but it has to be raised to FSF lawyers.
IMHO.  I mean, it appears sane to openly discuss any topic, for sure,
and freely rehash previous decisions.  However, here I miss what could
be the conclusion because a) it is legal speculations since the case
have never been pleaded in Court and b) many of us are not qualified to
parse all lengthy judicial documents – what a lawyer is daily doing,
IIUC friends’ job. :-)

2: 
3: 


Cheers,
simon

*pattern of WANAX session: «I am not a X but my strong opinion on
related-to-X is …». :-)

Re: Proposal: Build timers

[Vagrant Cascadian]

On 2021-11-24, zimoun wrote:
> On Tue, 23 Nov 2021 at 18:50, Julien Lepiller  wrote:
>> Do we even care that much about accuracy? I don't really care that the
>> build takes 30 or 31 seconds, or even 1 minute, but I certainly care
>> whether it takes 30s or 3h. I think this is also what SBUs give you: a
>> rough estimate of which build is longer than the other. I think a
>> simple proportionality relation would work well enough in most common
>> cases. It might be quite off on a super computer, but who cares,
>> really?
>
> What if it takes 3h and the prediction says 2h?

Those sound about "the same" for any kind of reasonable expectation...

I would guess you only want the correct order of magnitude... hours,
minutes, days, weeks, months, years... or maybe quick, fast, slow,
painful.

I do this soft of fuzzy estimation all the time when working on
Reproducible Builds in Debian; look at the past test history to get a
*rough* estimate of how long I might expect a build to take. This helps
me decide if I should start a build and get a $COFFEE, do some
$SWORDFIGHTING on the $OFFICECHAIRS, or sit and watch the progress bar
so I don't loose the mental state working on the problem becuase it will
be done $SOON.

Make it clear it's an estimate, or maybe even abstract away the time
units so that there is no expectation of any particular time.

I know there are people who would love to get a a value that was
consistently right but to be *useful* it only needs an estimate to be
mostly not completely wrong. At least to me. :)


live well,
  vagrant


signature.asc
Description: PGP signature

Re: ZFS part of Guix? RFC?

[Vagrant Cascadian]

On 2021-11-24, Denis 'GNUtoo' Carikli wrote:
> On Wed, 24 Nov 2021 13:03:18 +0100
> "pelzflorian (Florian Pelz)"  wrote:
>
>> On Wed, Nov 24, 2021 at 01:45:19AM +0100, Denis 'GNUtoo' Carikli
>> wrote:

https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/

> That article states that:
>> Pure distribution of source with no binaries is undeniably different.
>> When distributing source code and no binaries, requirements in those
>> sections of GPLv2 and CDDLv1 that cover modification and/or binary
>> (or “Executable”, as CDDLv1 calls it) distribution do not activate.
>> Therefore, the analysis is simpler, 
> So is it legal because zfs-on-linux is distributed as source and that
> the CDDL license incompatible requirements are waived when it is
> distributed as source?

Rather than "waived", they are simply not applicable. There is basically
an "if" statement in the CDDL that triggers the incompatibility, and in
the case of source-only distribution, the conflicting parts of the
licenses do not come into play.


> And that combining that work with GPLv2 code in
> source form is OK because GPLv2 is not violated because the
> incompatible CDDL requirements are only activated when distributed in
> executable form?
>
> If that's the case that would be the first explanation that
> doesn't undermine copyleft that I come across, and that is OK for me.

This is exactly the case, as I understand it...

It is precisely because the terms of the GPLv2 and CDDL licenses do not
conflict in terms of source code, the only conflict arises when you
actually distribute binaries.

It is by no means *ideal* by Free Software principles and goals. It is
so nuanced, non-obvious, tricky and complicated, which is why it keeps
getting rehashed over the years.

It is a bit obnoxious in my personal opinion, but as a layman/non-lawyer
reading it, seems technically and legally permissible, as long as you do
not distribute the compiled binaries.


The fact that GNU Guix can technically handle this issue correctly by
marking the package as not substitutable, in other words "do not
distribute binaries of this package", is a very interesting
workaround. Thankfully there are very few cases like this!


live well,
  vagrant


signature.asc
Description: PGP signature

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[Leo Famulari]

On Mon, Nov 22, 2021 at 07:10:48PM +0100, pelzflorian (Florian Pelz) wrote:
> Maybe there is consensus that adding ZFS is a legal risk.

I assume we are talking about the risk of litigation from Oracle (ZFS
owner), right?

Canonical decided in 2016 that the risk was low enough to take a chance
and include ZFS in Ubuntu:

https://ubuntu.com/blog/zfs-licensing-and-linux

Canonical earned $138 million in revenue in 2020, after earning more
than $100M in previous years as well [0]. So, they have a lot at stake
when taking that chance.

There are disagreements about whether or not the licenses in question
are compatible, but there is no definitive answer, only speculation and
rhetoric. Until then, all we can do is apply our values, balance our
options, and make a decision.

[0] Refer to "Group of companies' accounts made up to 31 December 2020":
https://find-and-update.company-information.service.gov.uk/company/06870835/filing-history

Re: Replacing polkit by polkit-duktape on core-updates-frozen ?

[Maxim Cournoyer]

Hi Josselin,

Josselin Poiret  writes:

> Maxim Cournoyer  writes:
>
> at it throws, after adding the missing polkit export:
>>
>> --8<---cut here---start->8---
>> $ ./pre-inst-env guix build  -s aarch64-linux gnome-control-center -n
>> guix build: error: gnu/packages/gnome.scm:5299:2: package
>> `colord-minimal@1.4.5' has an invalid input: ("polkit"
>> #)
>> --8<---cut here---end--->8---
>
> Since polkit is now syntax, it needs to be expanded before being
> compiled at all its uses, however guile (and our build system) doesn't
> automatically recompile all .scm files that use the changed definition.

That's the impression I got after realizing the fix.  Is there an open
issue about this shortcoming in Guile?  I think that's where a fix
should go.

> A simple `rm $(grep --include=\*.go -rl polkit .)` in the root of the
> checkout, followed by a `make` did it for me.

Thanks for the tip!

> As a side note, I'm also investigating that sddm error today.

Awesome!

Thank you,

Maxim

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[pelzflorian (Florian Pelz)]

On Wed, Nov 24, 2021 at 01:32:56PM +0100, pelzflorian (Florian Pelz) wrote:
> Sorry I misunderstood.  I think your claim is that the ZFS decisions
> listed by Ludo i.e. to disallow binary substitutes but to allow
> patches for a ZFS file-system object (once reviewed) are inconsistent.
> Right?
> 
> I don't know if that convinces maintainers to change decisions.
On Wed, Nov 24, 2021 at 01:51:08PM +0100, zimoun wrote:
> This decision is consistent with the analysis [1] done by Software
> Conservancy Freedom, at least.

I did not speak about one decision.

What I meant is that maybe Denis argued “dynamic linking creates a
derivative work” if and only if “ZFS source code is a derivative
work of Linux”.

Case 1, “ZFS source code is a derivative work of Linux” is true, then
it does not have a valid license, i.e. no free software license.

Case 2, “dynamic linking creates a derivative work” is false, then we
should offer binary substitutes as well.

It would follow that Guix’ decisions are inconsistent.

Then again, from Denis’ new mail,

On Wed, Nov 24, 2021 at 02:33:00PM +0100, Denis 'GNUtoo' Carikli wrote:
> The issue here is that I think that we need to find a valid and
> plausible explanation that makes the ZFS kernel module source code legal
> in a way that doesn't undermine copyleft.

Maybe Denis argued that any support for ZFS is at odds with the FSF
opinion that “dynamic linking creates a derivative work”.

On Wed, Nov 24, 2021 at 02:33:00PM +0100, Denis 'GNUtoo' Carikli wrote:
> So is it legal because zfs-on-linux is distributed as source and that
> the CDDL license incompatible requirements are waived when it is
> distributed as source? And that combining that work with GPLv2 code in
> source form is OK because GPLv2 is not violated because the
> incompatible CDDL requirements are only activated when distributed in
> executable form?

That the CDDL has terms specific to source code distribution
complicates things further and it is hard to tell if copyright law can
even impose all of CDDL’s terms.

Regards,
Florian

Donations: Set up monero

[Jacob Hrbek]

Set up monero wallet to enable donations from privacy concious supporters

-- Jacob "Kreyren" Hrbek

Sent with ProtonMail Secure Email.

publickey - kreyren@rixotstudio.cz - 0x1677DB82.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[Denis 'GNUtoo' Carikli]

On Wed, 24 Nov 2021 13:03:18 +0100
"pelzflorian (Florian Pelz)"  wrote:

> On Wed, Nov 24, 2021 at 01:45:19AM +0100, Denis 'GNUtoo' Carikli
> wrote:
> > If that's the case then it would also be legal to redistribute
> > binaries too as long as they are dynamically linked as the linking
> > happens at runtime.
> 
> The FSF is unable to have such a position.
What I didn't understand is what magically made source exempt of the
GPLv2 requirement while binaries aren't.

For instance if I make a module for Linux and uses symbols for Linux:
- The source code uses code from Linux. But if it's distributed
  separately both are not combined. To be compilable, the source needs
  to use headers or function definitions from Linux.
- Modules being linked dynamically (=m) also use code from Linux and
  during the compilation, as I understand it, it only uses the headers
  and/or functions definitions that I mentioned above. So this case
  is not very different from the source code.
- Binaries being statically linked (=y) are being included in the Linux
  binary.

So from the GPLv2 side, I don't see what could make source code and
dynamically linked module that different as to justify different
applications of the GPL.

That article states that:
> Pure distribution of source with no binaries is undeniably different.
> When distributing source code and no binaries, requirements in those
> sections of GPLv2 and CDDLv1 that cover modification and/or binary
> (or “Executable”, as CDDLv1 calls it) distribution do not activate.
> Therefore, the analysis is simpler, 
So is it legal because zfs-on-linux is distributed as source and that
the CDDL license incompatible requirements are waived when it is
distributed as source? And that combining that work with GPLv2 code in
source form is OK because GPLv2 is not violated because the
incompatible CDDL requirements are only activated when distributed in
executable form?

If that's the case that would be the first explanation that
doesn't undermine copyleft that I come across, and that is OK for me.

If it's not the case then we have an issue and I think that we still
need to find a valid explanation that doesn't undermine copyleft and/or
get rid of the ZFS kernel module.

It also states that it's risky legally:
> Nevertheless, there may be arguments for contributory and/or indirect
> copyright infringement in many jurisdictions. We present no specific
> analysis ourselves on the efficacy of a contributory infringement
> claim regarding source-only distributions of ZFS and Linux. However,
> in our GPL litigation experience, we have noticed that judges are
> savvy at sniffing out attempts to circumvent legal requirements, and
> they are skeptical about attempts to exploit loopholes. Furthermore,
> we cannot predict Oracle's view — given its past willingness to
> enforce copyleft licenses, and Oracle's recent attempts to adjudicate
> the limits of copyright in Court. Downstream users should consider
> carefully before engaging in even source-only distribution. 
But as long as the rationale behind doesn't undermine copyleft, I've no
issue with it.

> It seems unrelated to the FSDG, so GNU Guix need not care.
The issue here is that I think that we need to find a valid and
plausible explanation that makes the ZFS kernel module source code legal
in a way that doesn't undermine copyleft.

And in some cases, laws or interpretations of laws that are undermine
copyleft might be good if they also brings systemic advantages to free
software: for instance if new laws makes all software free in
theory and in practice too (by also making sure that we have the
corresponding source code, or because we have tools to derive it from
binaries).

But here if we don't have a rationale that doesn't undermine copyleft,
what we gain here is just the ability to use a filesystem in the kernel
instead of using it through FUSE, so it's an awful tradeoffs for free
software.

But if we do have a rationale that doesn't undermine copyleft, it just
brings some legal risk, but doesn't undermine copyleft, so I'm OK with
that.

In the past and present, distributions also had/have to deal with
patents, anti DRM circumvention legislation, and many other legal
risks, and the consensus in the FLOSS community (at least for patents
and anti DRM circumvention) is that the choice of distributing risky
packages is to be made by the individual distributions and not the whole
FLOSS and/or free software community at large.

Denis.


pgp21tTZwt_eJ.pgp
Description: OpenPGP digital signature

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[zimoun]

Hi Florian,

On Wed, 24 Nov 2021 at 13:32, "pelzflorian (Florian Pelz)" 
 wrote:

> I don't know if that convinces maintainers to change decisions.

This decision is consistent with the analysis [1] done by Software
Conservancy Freedom, at least.

I have not read a clear position by the FSF.  They only analyses [2] the
incompatibilities of licenses but their analysis is not fine enough to
conclude what is their position about distributing ZFS; although they
say «ZFS is free software», thus nothing prevent Guix to distribute
source code of ZFS.


1: 
2: 


Cheers,
simon

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[pelzflorian (Florian Pelz)]

On Wed, Nov 24, 2021 at 01:03:29PM +0100, pelzflorian (Florian Pelz) wrote:
> On Wed, Nov 24, 2021 at 01:45:19AM +0100, Denis 'GNUtoo' Carikli wrote:
> > If that's the case then it would also be legal to redistribute binaries
> > too as long as they are dynamically linked as the linking happens at
> > runtime.
> 
> The FSF is unable to have such a position.
> 
> It seems unrelated to the FSDG, so GNU Guix need not care.

Sorry I misunderstood.  I think your claim is that the ZFS decisions
listed by Ludo i.e. to disallow binary substitutes but to allow
patches for a ZFS file-system object (once reviewed) are inconsistent.
Right?

I don't know if that convinces maintainers to change decisions.

Regards,
Florian

build system option to allow CPU optimizations?

[Ricardo Wurmus]

Hi Guix,

currently we patch source code and build systems to ensure that no 
special instructions are used that would not be portable, 
e.g. AVX2, SSE4.1 etc.  What do you think of adding a build system 
option that would allow users to restore these optimizations?


The build phases that patch out these features would have to check 
for that build system option, much like they check the TESTS? 
option before attempting to run tests.


What do you think?

--
Ricardo

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[pelzflorian (Florian Pelz)]

On Wed, Nov 24, 2021 at 01:45:19AM +0100, Denis 'GNUtoo' Carikli wrote:
> If that's the case then it would also be legal to redistribute binaries
> too as long as they are dynamically linked as the linking happens at
> runtime.

The FSF is unable to have such a position.

It seems unrelated to the FSDG, so GNU Guix need not care.

Regards,
Florian

Re: Proposal: Build timers

[zimoun]

Hi,

On Tue, 23 Nov 2021 at 14:39, Jacob Hrbek  wrote:

>> This approximation would not even be accurate enough for the same
>> machine.  For instance, the test suite of the julia package runs
>> mainly sequential using one thread... 
>
> I am aware of this scenario and I adapted the equasion for it, but I
> recognize that this exponentially increases the inaccuracy with more
> threads and I don't believe that there is a mathematical way with the
> provided values to handle that scenario so we would have to adjust the
> calculation for those packages.

What I am trying to explain is that the model cannot work to be
predictable enough with what I consider a meaningful accuracy.
Obviously, relaxing the precision, it is easy to infer a rule of thumb;
a simple cross-multiplication fits the job. ;-)

The “pokémon-battle” model is a simple linear model
(cross-multiplication); using Jacob’s “notation”:

 - HP: time to build on machine A
 - DPS = nthread * cpufreq : “power” of machine

Then it is expected to evaluate ’a’ and ’b’ on average such that:

  HP = a * DPS + b

based on some experiments.  Last, on machine B, knowing both nthread'
and cpufreq' for that machine B, you are expecting to evaluate HP' for
that machine B applying the formula:

  HP' = a * nthread' * cpufreq' + b

Jacob, do I correctly understand the model?


In any case, that’s what LFS is doing, instead HP is named SBU.  And
instead DPS, they use a reference package.  And this normalization is
better, IMHO.  Other said, for one specific package considered as
reference, they compute HP1 (resp. HP2) for machine A (resp. B), then
for machine A, they know HP for another package and they deduce,

  HP' = HP2/HP1 * HP

All this is trivial. :-) The key is the accuracy, i.e., the error
between the prediction HP' and the real time.  Here, the issue is that
HP1 and HP2 capture for one specific package the overall time; which
depends on hidden parameters as nthread, cpufreq, IO, and other
parameters from hardware.  But that a strong assumption when considering
these hidden parameters (evaluated for one specific package) are equally
the same for any other package.

It is a strong assumption because the hidden parameters depends on
hardware specifications (nthread, cpufreq, etc.) *and* how the package
itself exploits them.

Therefore, the difference between the prediction and the real time is
highly variable, and thus personally I am not convince the effort is
worth; for local build.  That’s another story. ;-)

LSF is well-aware of the issue and it is documented [1,2].

The root of the issue is the model based on a strong assumption; both
(model and assumption) do not fit how the reality concrete works, IMHO.

One straightforward way — requiring some work though – for improving the
accuracy is to use statistical regressions.  We cannot do really better
to capture the hardware specification – noticing that the machine stress
(what the machine is currently doing when the build happens) introduces
a variability hard to estimate beforehand.  However, it is possible to
do better when dealing with packages.  Other said, exploit the data from
the build farms.


Well, I stop here because it rings a bell: model could be discussed at
length if it is never applied to concrete numbers. :-)

Let keep it pragmatic! :-)

Using the simple LFS model and SBU, what would be the typical error?

For instance, I propose that we collectively send the timings of
packages: bash, gmsh, julia, emacs, vim; or any other 5 packages for
x86_64 architecture.  Then we can compare typical errors between
prediction and real, i.e., evaluate “accuracy“ for SBU and then decide
if it is acceptable or not. :-)


Cheers,
simon


1: 
2: 

Re: Proposal: Build timers

[zimoun]

Hi,

On Tue, 23 Nov 2021 at 18:50, Julien Lepiller  wrote:
> Do we even care that much about accuracy? I don't really care that the
> build takes 30 or 31 seconds, or even 1 minute, but I certainly care
> whether it takes 30s or 3h. I think this is also what SBUs give you: a
> rough estimate of which build is longer than the other. I think a
> simple proportionality relation would work well enough in most common
> cases. It might be quite off on a super computer, but who cares,
> really?

What if it takes 3h and the prediction says 2h?

Which build is longer than the other is already provided by data in the
build farm.  I agree it is hard to find and one improvement could be
that Cuirass or Build Coordinator exposes this data (I think it is not
so easy because for instance Cuirass knows derivation and package,
somehow and IIUC :-)).

Who cares to know how longer it would locally take if the substitute is
available*? ;-) Because at some point, this SBU should be computed.

Anyway, what would be the typical error?

For instance, I propose that we collectively send the timings of
packages: bash, gmsh, julia, emacs, vim; or any other 5 packages for
x86_64 architecture.  Then we can compare typical errors between
prediction and real, i.e., evaluate “accuracy“ for SBU and then decide
if it is acceptable or not. :-)



*available: which is not the case for LFS, though.


Cheers,
simon

Re: Thursday 18th: ‘core-updates-frozen’ sprint!

[zimoun]

Hi,

On Mon, 22 Nov 2021 at 15:13, Maxim Cournoyer  wrote:

> 2. Disable the problematic tests.  You may need to disable a whole test suite 
> of
> similarly faulted tests, if they all behave the same.

I am becoming crazy…  For instance this test randomly* fails:

@test mul!(C, vf, transpose(vf), 2, 3) == 2vf*vf' .+ 3C0



Therefore, I add,

--8<---cut here---start->8---
(substitute* "stdlib/LinearAlgebra/test/matmul.jl" 
  (("@test mul\\!\\(C, vf, transpose\\(vf\\), 2, 3\\) == 2vf\\*vf' 
\\.+ 3C0") 
   "@test_broken @test mul!(C, vf, transpose(vf), 2, 3) == 2vf*vf' 
.+ 3C0")
(substitute* "test/math.jl" 
  (("@test isinf\\(log1p\\(-one\\(T\\)\\)\\)")
   "@test_broken isinf(log1p(-one(T)))"
--8<---cut here---end--->8---

So far, so good.  Then I get:

--8<---cut here---start->8---
Error During Test at 
/tmp/guix-build-julia-1.6.3.drv-0/julia-1.6.3/test/math.jl:278
 Unexpected Pass 
Expression: isinf(log1p(-(one(T
 Got correct result, please change to @test if no longer broken.
--8<---cut here---end--->8---

and thus the build is reported as failed.  Argh!!  Ok, because these 2
tests are random, I replace the snippet above by:

--8<---cut here---start->8---
(substitute* "stdlib/LinearAlgebra/test/matmul.jl" 
  (("@test mul\\!\\(C, vf, transpose\\(vf\\), 2, 3\\) == 2vf\\*vf' 
\\.+ 3C0") 
   "")
(substitute* "test/math.jl" 
  (("@test isinf\\(log1p\\(-one\\(T\\)\\)\\)")
   ""
--8<---cut here---end--->8---

And now, something magical happens, I get this error with parenthesis:

--8<---cut here---start->8---
Test Failed at 
/tmp/guix-build-julia-1.6.3.drv-0/julia-1.6.3/usr/share/julia/stdlib/v1.6/LinearAlgebra/test/matmul.jl:155
Expression: mul!(C, vf, transpose(vf), 2, 3) == (2vf) * vf' .+ 3C0
--8<---cut here---end--->8---

What?!  And AFAIU, it is not in the test suite.

Do I miss something with ’substitute*’?


Cheers,
simon

*randomly: because I do not know why, nothing is random with
 computer. ;-)

PS: Note that current Julia master contains this test:

@test mul!(C, vf, transpose(vf), 2, 3) ≈ 2vf*vf' .+ 3C0

https://github.com/JuliaLang/julia/blob/master/stdlib/LinearAlgebra/test/matmul.jl#L155

I am trying to ’substitute*’ with that.  Then I give up for now.

Update on bordeaux.guix.gnu.org

[Christopher Baines]

Hey!

It's been 3 months since I sent the last update [1]. This email was
meant to go out on Friday, but it seems opensmtpd was broken on my
machine, so it got stuck.

1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg00075.html

First, some good things:

I've been doing some performance tuning, submitting builds is now more
parallelised, a source of slowness when fetching builds has been
addressed, and one of the long queries involved in allocating builds has
been removed, which also improved handling of the WAL (Sqlite write
ahead log).

There's also a few new features. Agents can be deactivated which means
they won't get any builds allocated. The coordinator now checks the
hashes of outputs which are submitted, a safeguard which I added because
the coordinator now also supports resuming the uploads of outputs. This
is particularly important when trying to upload large (> 1GiB) outputs
over slow connections.

I also added a new x86_64 build machine. It's a 4 core Intel NUC that I
had sitting around, but I cleaned it up and got it building things. This
was particularly useful as I was able to use it to retry building
guile@3.0.7, which is extremely hard to build [2]. This was blocking
building the channel instance derivations for x86_64-linux.

2: 
https://data.guix.gnu.org/gnu/store/7k6s13bzbz5fd72ha1gx9rf6rrywhxzz-guile-3.0.7.drv

On the related subject of data.guix.gnu.org (which is the source of
derivations for bordeaux.guix.gnu.org, as well as a recipient of build
information), there have been a couple of changes. There was some web
crawler activity that was slowing data.guix.gnu.org down significantly,
NGinx now has some rate limiting configuration to prevent crawlers
abusing the service. The other change is that substitutes for the latest
processed revision of master will be queried on a regular basis, so this
page [3] should be roughly up to date, including for ci.guix.gnu.org.

3: 
https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-substitute-availability

Now for some not so good things:

Submitting builds wasn't working quite right for around a month, one of
the changes I made to speed things up led to some builds being
missed. This is now fixed, and all the missed builds have been
submitted, but this was more than 50,000 builds. This, along with all
the channel instance derivation builds that can now proceed mean that
there's a very large backlog of x86 and ARM builds which will probably
take at least another week to clear. While this backlog exists,
substitute availability for x86_64-linux will be lower than usual.

Space is running out on bayfront, the machine that runs the coordinator,
stores all the nars and build logs, and serves the substitutes. I knew
this was probably going to be an issue, bayfront didn't have much space
to begin with, but I had hoped I'd be further forward in developing some
way to allow moving the nars around between multiple machines, to remove
the need to store all of them on bayfront. I have got a plan, there's
some ideas I mentioned back in February [4], but I haven't got around to
implementing anything yet. The disk space usage trend is pretty much
linear, so if things continue without any change, I think it will be
necessary to pause the agents within a month, to avoid filling up
bayfront entirely.

4: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Replacing polkit by polkit-duktape on core-updates-frozen ?

[Development of GNU Guix and the GNU System distribution.]

Maxim Cournoyer  writes:

at it throws, after adding the missing polkit export:
>
> --8<---cut here---start->8---
> $ ./pre-inst-env guix build  -s aarch64-linux gnome-control-center -n
> guix build: error: gnu/packages/gnome.scm:5299:2: package 
> `colord-minimal@1.4.5' has an invalid input: ("polkit" # polkit>)
> --8<---cut here---end--->8---

Since polkit is now syntax, it needs to be expanded before being
compiled at all its uses, however guile (and our build system) doesn't
automatically recompile all .scm files that use the changed definition.

A simple `rm $(grep --include=\*.go -rl polkit .)` in the root of the
checkout, followed by a `make` did it for me.

As a side note, I'm also investigating that sddm error today.

Best,
Josselin Poiret