gnu-build-system not handling pkg-config properly in package definition

[Nathan Dehnel]

https://www.mail-archive.com/bug-guix@gnu.org/msg00180.html
According to this,  if you use gnu-build-system, all packages added to
inputs get added to PKG_CONFIG_PATH, but for some reason it's not
happening for util-linux, which causes a build error. During the build
PKG_CONFIG_PATH ends up not containing any of the util-linux packages.
Does anyone know how to fix this? Package below:

(define-module (gooby-channel packages bcache-tools)
  #:use-module (guix licenses)
  #:use-module (guix packages)
  #:use-module (gnu packages)
  #:use-module (guix build-system gnu)
  #:use-module (guix git-download)
  #:use-module (gnu packages linux)
  #:use-module (gnu packages pkg-config))

(define-public bcache-tools
  (package
(name "bcache-tools")
(version "1.1")
(source
 (origin
   (method git-fetch)
   (uri
(git-reference
 (url (string-append
"https://git.kernel.org/pub/scm/linux/kernel/git/colyli/; name
".git"))
 (commit (string-append name "-" version
   (file-name (git-file-name name version))
   (sha256
(base32
 "1vfqm3asx7h1sjwyq45dki4wk7lkipqrx2pw2hnqm4xir85qf67h"))
   (patches (search-patches "bcache-tools-noprobe.patch"
(build-system gnu-build-system)
(arguments
 `(;;#:tests? #f  ; no "check" target
   #:make-flags
 (list (string-append "PREFIX=" (assoc-ref %outputs "out")))
   #:phases
 (modify-phases %standard-phases
  (delete 'configure
(inputs
 `(("util-linux" ,util-linux)))
(native-inputs
 `(("pkg-config" ,pkg-config)))
(home-page "https://bcache.evilpiepirate.org/;)
(synopsis "These are the userspace tools required for bcache.")
(description "Bcache patches for the Linux kernel allow one to use
SSDs to cache other block devices. It's analogous to L2Arc for ZFS,
but Bcache also does writeback caching (besides just write through
caching), and it's filesystem agnostic. ")
(license gpl2)))

Re: Linux-Libre-LTS

[Raghav Gururajan]

Hello Efraim!

> I was waiting for the kernel code reorganization before adding it as a
> variable. The trick is to add also linux-libre-lts-source and all the
> others, and in a useful location. Now it's just taking the time to add
> it in somewhere.
> 
> Do you want to take a stab at it? I'm not sure when I'll get around to
> it.

Does the attached patch looks good?

The idea here is to update the version number that the variable
linux-libre-lts pointing to, whenever the new LTS version is released.

Regards,
RG.











From 883c805cf36a1b94fca45a3e3efbf851b7ea11ce Mon Sep 17 00:00:00 2001
From: Raghav Gururajan 
Date: Wed, 23 Dec 2020 22:43:10 -0500
Subject: [PATCH] gnu: Add Linux-Libre-LTS.

Enables the choice of using current LTS version of linux-libre in Guix System.

* gnu/packages/linux.scm (linux-libre-lts-version): New variable.
* gnu/packages/linux.scm (linux-libre-lts-pristine-source): New variable.
* gnu/packages/linux.scm (linux-libre-lts-source): New variable.
* gnu/packages/linux.scm (linux-libre-lts): New variable.
---
 gnu/packages/linux.scm | 8 
 1 file changed, 8 insertions(+)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 01f12f77d9..37727dcf94 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -931,6 +931,14 @@ It has been modified to remove all non-free binary blobs.")
 ("CONFIG_DEVPTS_MULTIPLE_INSTANCES" . #t))
   %default-extra-linux-options)))
 
+;; Linux-Libre-LTS means the *current* long-term support version of Linux-Libre.
+;; Reference: https://jxself.org/linux-libre/
+
+(define-public linux-libre-lts-version linux-libre-5.10-version)
+(define-public linux-libre-lts-pristine-source linux-libre-5.10-pristine-source)
+(define-public linux-libre-lts-source  linux-libre-5.10-source)
+(define-public linux-libre-lts linux-libre-5.10)
+
 
 ;;;
 ;;; Specialized kernel variants.
-- 
2.17.1



signature.asc
Description: OpenPGP digital signature

Re: Identical files across subsequent package revisions

[Mark H Weaver]

I wrote:
> FYI, here's the reason that IceCat is unusual among the projects sampled
> in Ludovic's analysis: the large collection of JavaScript source files
> and other auxiliary files, which are mostly unchanged from one release
> to the next, are bundled into a single zip file in lib/icecat/omni.ja.

Correction: there are actually two omni.ja files.  The second one is in
lib/icecat/browser/omni.ja.

  Mark

Re: Identical files across subsequent package revisions

[Jonathan Brielmaier]

On 23.12.20 23:06, Mark H Weaver wrote:

Hi Taylan,

Taylan Kammer  writes:

My second thought: it's surprising that IceCat supposedly changes so
much between releases.  I suppose the reason is that this analysis is on
a per-file basis, and IceCat is mostly a massive binary.


FYI, here's the reason that IceCat is unusual among the projects sampled
in Ludovic's analysis: the large collection of JavaScript source files
and other auxiliary files, which are mostly unchanged from one release
to the next, are bundled into a single zip file in lib/icecat/omni.ja.


And even more most of the changed files differ only via the different
hash in the output path. So the actual diff is a few hundred lines of
almost the same -> should get compressed very good...

Re: Staging branch

[Leo Famulari]

On Wed, Dec 23, 2020 at 12:27:22AM -0500, Leo Famulari wrote:
> ... and reply with any problems.

The package kwidgetsaddons fails its test suite like this:

--
* Start testing of KColumnResizerTest *
Config: Using QtTest library 5.15.2, Qt 5.15.2 (x86_64-little_endian-lp64 
shared (dynamic) release build; by GCC 7.5.0), unknown unknown
PASS   : KColumnResizerTest::initTestCase()
QWARN  : KColumnResizerTest::test(forms) This plugin does not support 
propagateSizeHints()
PASS   : KColumnResizerTest::test(forms)
QWARN  : KColumnResizerTest::test(grids) This plugin does not support 
propagateSizeHints()
PASS   : KColumnResizerTest::test(grids)
QWARN  : KColumnResizerTest::test(grid-and-form) This plugin does not support 
propagateSizeHints()
FAIL!  : KColumnResizerTest::test(grid-and-form) Compared values are not the 
same
   Actual   (widget1->x()): 145
   Expected (widget2x): 161
   Loc: 
[/tmp/guix-build-kwidgetsaddons-5.70.0.drv-0/kwidgetsaddons-5.70.0/autotests/kcolumnresizertest.cpp(83)]
PASS   : KColumnResizerTest::cleanupTestCase()
Totals: 4 passed, 1 failed, 0 skipped, 0 blacklisted, 18ms
* Finished testing of KColumnResizerTest *


95% tests passed, 1 tests failed out of 19

Total Test time (real) =   8.27 sec

The following tests FAILED:
 19 - kwidgetsaddons-kcolumnresizertest (Failed)
Errors while running CTest
make: *** [Makefile:110: test] Error 8

Test suite failed, dumping logs.
--

I tried using the newer version of this package (5.71) but it fails due
to incompatibilities with the rest of the KDE frameworks.

Any ideas?

Re: Identical files across subsequent package revisions

[Mark H Weaver]

Hi Taylan,

Taylan Kammer  writes:
> My second thought: it's surprising that IceCat supposedly changes so 
> much between releases.  I suppose the reason is that this analysis is on 
> a per-file basis, and IceCat is mostly a massive binary.

FYI, here's the reason that IceCat is unusual among the projects sampled
in Ludovic's analysis: the large collection of JavaScript source files
and other auxiliary files, which are mostly unchanged from one release
to the next, are bundled into a single zip file in lib/icecat/omni.ja.

  Mark

Re: [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

[Mark H Weaver]

Hi Efraim,

Efraim Flashner  writes:
> "regular powerpc", ie macppc/ppc32/powerpc-linux-gnu, does have some
> bootstrap binaries built but isn't near ready for merging. Go ahead and
> make any changes necessary.

I appreciate that, but if rebuilding the world on regular powerpc would
significantly add to the burden of even a single developer, then it's
probably not worth it.  I suggested fixing the powerpc64le case now only
because it was just added a few days ago, and more generally to raise
awareness about how best to run the 'patch' program in Guix.

If it's truly no extra burden, then you could change "--batch" to
"--force" on line 69 of libffi.c (in the "powerpc-*" case).

  Regards,
Mark

Re: [PATCH] gnu: libffi: Add unreleased patch to fix float128 on powerpc64le.

[Mark H Weaver]

Hi Chris,

Chris Marusich  writes:

> Mark H Weaver  writes:
>
>> Earlier, I wrote:
>>> When invoking 'patch' in Guix, you should *always* use "--force" instead
>>> of "--batch".
>>
>> (See  for my earlier message).
>
> Thank you for letting me know about this.  I didn't know about the
> difference between "--batch" and "--force".  I agree we should use
> "--force" instead of "--batch".  How do you recommend that I proceed?

Simply changing "--batch" to "--force" on line 79 (in the powerpc64le-*
case, i.e. the one that was just added) seems like the right thing.
That will force a rebuild of almost everything on the powerpc64le-*
architecture, but should not cause any rebuilds on other architectures.

>> Since writing the message above, I've found another problem in the same
>> commit (7eaa2f24ea77cddbb4bbc2d6a6905673a36f8f99): it searches for the
>> 'patch' program in 'inputs'.  This is a mistake, because when
>> cross-compiling, 'inputs' will contain software compiled to run on the
>> target system instead of the build system.
>
> Is it searching for the "patch" program, or is it searching for the
> patch file?  It looks to me like the code is searching for the patch
> file in inputs, not the "patch" program.

LOL, you're right, I got confused.  Please disregard my second email in
this thread, and apologies for that noise.

> Again, thank you for taking the time to bring these topics up.  I'm
> always trying to make sure I do things the best way I can in Guix, so I
> appreciate the feedback.

Thank you, Chris.

Warm regards,
Mark

Re: Identical files across subsequent package revisions

[Miguel Ángel Arruga Vivas]

Hi Julien and Simon,

Julien Lepiller  writes:

> Le 23 décembre 2020 09:07:23 GMT-05:00, zimoun  a 
> écrit :
>>Hi,
>>
>>On Wed, 23 Dec 2020 at 14:10, Miguel Ángel Arruga Vivas
>> wrote:
>>> Another idea that might fit well into that kind of protocol---with
>>> harder impact on the design, and probably with a high cost on the
>>> runtime---would be the "upgrade" of the deduplication process towards
>>a
>>> content-based file system as git does[2].  This way the a description
>>of
>>> the nar contents (size, hash) could trigger the retrieval only of the
>>> needed files not found in the current store.
>>
>>Is it not related to Content-Addressed Store?  i.e, «intensional
>>model»?
>>
>>Chap. 6: 
>>Nix FRC:
>>
>
> I think this is different, because we're talking about sub-element
> content-addressing. The intensional model is about content-addressing
> whole store elements. I think the idea would be to save individual
> files in, say, /gnu/store/.links, and let nar or narinfo files
> describe the files to retrieve. If we are missing some, we'd download
> them, then create hardlinks. This could even help our deduplication I
> think :)

Exactly.  My first approach would be a tree %links-prefix/hash/size, to
where all the internal contents of each store item would be hard linked:
mainly Git's approach with some touch here and there---some of them
probably have too much good will, the first approach isn't usually the
best. :-)

- The garbage collection process could check if there is any hard link
  to those files or remove them otherwise, deleting the hash folder when
  needed.

- The deduplication process could be in charge of moving the files and
  placing hard links instead, but hash collisions with the same size are
  always a possibility, therefore some mechanism is needed to treat
  these cases[1] and the vulnerabilities associated to them.

- The substitution process could retrieve from the server the
  information about the needed files, check which contents are already
  available and which ones must be retrieved, and ensure that the
  "generated nar" is the same as the one from the server.  This is quite
  related to the deduplication process and the mechanism used there[2].

Happy hacking!
Miguel

[1] Perhaps the usage of two different hash algorithms instead of one,
or different salts, could be enough for the "common" error case as a
collision on both cases are quite improbable.  They are possible anyway
with a size bigger than the hash size, therefore a final fallback to
actual bytes is probably needed.

[2] The folder could be hashed, even with a temporary salt agreed with
the server, to perform an independent/real-time check, but any issue
here has bigger consequences too, as no byte to byte comparison is
possible before the actual transmission.

Re: Identical files across subsequent package revisions

[Julien Lepiller]

Le 23 décembre 2020 09:07:23 GMT-05:00, zimoun  a 
écrit :
>Hi,
>
>On Wed, 23 Dec 2020 at 14:10, Miguel Ángel Arruga Vivas
> wrote:
>
>> Probably you're already aware of it, but I want to mention that
>> Tridgell's thesis[1] contains a very neat approach to this problem.
>
>This thesis is a must to read! :-)
>
>
>> A naive prototype would be copying of the latest available nar of the
>> package on the client side and using it as the destination for a copy
>> using rsync.  Either the protocol used by the rsync application, or a
>> protocol based on those ideas, could be implemented over the HTTP
>layer;
>> client and server implementation and cooperation would be needed
>> though.
>
>I could misunderstand and miss something, one part of the problem is
>how
>to detect “latest”; other said how to know it is different.  From my
>memories, and I have drunk couple of beers since I read the thesis :-),
>the ’rsync’ approach uses timestamp and size.  And if you switch to
>checksum instead, the performances are poor, because of IO.  Well, it
>depends on the number of files and their size, if this checksum are
>computed ahead, etc.
>
>
>> Another idea that might fit well into that kind of protocol---with
>> harder impact on the design, and probably with a high cost on the
>> runtime---would be the "upgrade" of the deduplication process towards
>a
>> content-based file system as git does[2].  This way the a description
>of
>> the nar contents (size, hash) could trigger the retrieval only of the
>> needed files not found in the current store.
>
>Is it not related to Content-Addressed Store?  i.e, «intensional
>model»?
>
>Chap. 6: 
>Nix FRC:
>

I think this is different, because we're talking about sub-element 
content-addressing. The intensional model is about content-addressing whole 
store elements. I think the idea would be to save individual files in, say, 
/gnu/store/.links, and let nar or narinfo files describe the files to retrieve. 
If we are missing some, we'd download them, then create hardlinks. This could 
even help our deduplication I think :)

>
>
>Cheers,
>simon

Re: Identical files across subsequent package revisions

[zimoun]

Hi,

On Wed, 23 Dec 2020 at 14:10, Miguel Ángel Arruga Vivas  
wrote:

> Probably you're already aware of it, but I want to mention that
> Tridgell's thesis[1] contains a very neat approach to this problem.

This thesis is a must to read! :-)


> A naive prototype would be copying of the latest available nar of the
> package on the client side and using it as the destination for a copy
> using rsync.  Either the protocol used by the rsync application, or a
> protocol based on those ideas, could be implemented over the HTTP layer;
> client and server implementation and cooperation would be needed
> though.

I could misunderstand and miss something, one part of the problem is how
to detect “latest”; other said how to know it is different.  From my
memories, and I have drunk couple of beers since I read the thesis :-),
the ’rsync’ approach uses timestamp and size.  And if you switch to
checksum instead, the performances are poor, because of IO.  Well, it
depends on the number of files and their size, if this checksum are
computed ahead, etc.


> Another idea that might fit well into that kind of protocol---with
> harder impact on the design, and probably with a high cost on the
> runtime---would be the "upgrade" of the deduplication process towards a
> content-based file system as git does[2].  This way the a description of
> the nar contents (size, hash) could trigger the retrieval only of the
> needed files not found in the current store.

Is it not related to Content-Addressed Store?  i.e, «intensional model»?

Chap. 6: 
Nix FRC: 



Cheers,
simon

Re: Identical files across subsequent package revisions

[Miguel Ángel Arruga Vivas]

Hi Ludo,

Just one interjection: wow! :-)

Ludovic Courtès  writes:

> Hello Guix!
>
> Every time a package changes, we end up downloading complete substitutes
> for itself and for all its dependents, even though we have the intuition
> that a large fraction of the files in those store items are unchanged.

It's great you're taking a look into these kind of optimizations, as
they also close the gap between only-binary distribution and the
substitutes system.

> [Awesome data collection omitted for brevity]
>
> Thoughts?  :-)

Probably you're already aware of it, but I want to mention that
Tridgell's thesis[1] contains a very neat approach to this problem.

A naive prototype would be copying of the latest available nar of the
package on the client side and using it as the destination for a copy
using rsync.  Either the protocol used by the rsync application, or a
protocol based on those ideas, could be implemented over the HTTP layer;
client and server implementation and cooperation would be needed though.

Another idea that might fit well into that kind of protocol---with
harder impact on the design, and probably with a high cost on the
runtime---would be the "upgrade" of the deduplication process towards a
content-based file system as git does[2].  This way the a description of
the nar contents (size, hash) could trigger the retrieval only of the
needed files not found in the current store.

Nonetheless, these are only thoughts, I'll ping back if and when I have
something more tangible. ;-)

Happy hacking!
Miguel

[1] https://rsync.samba.org/~tridge/phd_thesis.pdf
[2] https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Re: SJTUG's Guix mirror is online now

[Julien Lepiller]

Le 23 décembre 2020 03:39:54 GMT-05:00, zimoun  a 
écrit :
>Hi,
>
>On Wed, 23 Dec 2020 at 09:50, Peng Mei Yu  wrote:
>
 The substitute URL is: https://mirrors.sjtug.sjtu.edu.cn/guix
>
>Nice news!
>
>
>>> If you’re interested, you could write an article for the blog post
>(in
>>> English, but it’d be a good opportunity to start translating the web
>>> site into Chinese :-)) about your setup and the choices you made.
>>
>> I don't think there are enough things to write in a blog post but I
>will
>> consider translating the web site.
>
>Does it make sense to add the URL in the Chinese manual?

Many people prefer the original version despite the availability of a good 
translation. You might also live in China and not be a fluent speaker. I think 
it makes sense to add it to the English manual too :).

I think it's a bad idea to have different information in different languages.

>
>All the best,
>simon

Re: Word order in Guix l10n

[Miguel Ángel Arruga Vivas]

Hi Julien,

Julien Lepiller  writes:

> This specific syntax looks ok, but we need to limit ourself to the
> common syntax between guile and lisp, because that's what gettext
> supports.

The issue with Guile's format is explained here[1], as the used
implementation follows SRFI-28[2], but there are no difference between
the format from Common Lisp and the one from (ice-9 format)[3] on the
surface level: both implementations are compatible regarding numeric,
iteration, selection and jump directives, to name a few.

Other directives might be compatible, such as the plural directive ~P,
or not, although most of them shouldn't be used in any case: not because
they could have compatibility problems but because they don't fit into
internationalized messages correctly.

For example, most languages have irregular cases for plural formation,
some have more than two grammatical numeric cases, such as
singular/dual/plural, and some don't have an equivalent category, such
as Japanese.  That's exactly use case of ngettext---I've pointed out on
the other mail the pending issue on that area, which is related to the
omission of the numeric parameter but not its order, and applies both to
Common Lisp and (ice-9 format).

> We should use this kind of syntax everywhere we have more than one
> argument.

I don't see the advantage of using everywhere jumps on the msgids.
Nonetheless, a TRANSLATORS: comment placed on the first string appearing
on the POT file, pointing the section of the manual for (ice-9 format),
or even an explicit and detailed explanation of this syntax could be
very helpful for translators.  The attached patch does this, although
any suggestion or even a complete rewrite is welcome, because I don't
feel it quite inspired.

> Also thinking about rtl languages, it's probably important
> for them, though I'm not sure how gettext works for them.

gettext-family functions only see byte arrays and provide the
corresponding array, the bytes are always placed in increasing memory
locations.  Right-to-left handling is a responsibility of visualization
layer, which sometimes includes the final format, but that is an issue
even with left-to-right languages as French.

For example, this composition...

  (string-append translated ": " other-translated)

... produces weird results, or convoluted French translations, because
it isn't handled properly.  A format string must be used here too,
because it must include the white-space expected in French before the
colon:

  (format #f (_ "~a: ~a") translated other-translated)

Newlines are the only ones that are omitted sometimes from the
internationalized composition because the convention up-to-down is
followed, but this is a limitation of the teletype/terminal interface
though; graphic interfaces aren't composed with this limitation and
"whole widgets" should be the localization frame, which usually is the
case.

Happy hacking!
Miguel

[1] https://www.gnu.org/software/guile/manual/html_node/Simple-Output.html
[2] https://www.gnu.org/software/guile/manual/html_node/SRFI_002d28.html
[3] https://www.gnu.org/software/guile/manual/html_node/Formatted-Output.html
>From 2615934a2c377858dce2a0410982287faed754a9 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Miguel=20=C3=81ngel=20Arruga=20Vivas?=
 
Date: Wed, 23 Dec 2020 13:07:38 +0100
Subject: [PATCH] nls: Add comment about format directives.

* gnu.scm (%try-use-modules): Add comment for translations.  It should
be placed on the first string found by xgettext.
---
 gnu.scm | 13 +
 1 file changed, 13 insertions(+)

diff --git a/gnu.scm b/gnu.scm
index f139531ef3..0e87b10eb2 100644
--- a/gnu.scm
+++ b/gnu.scm
@@ -78,6 +78,19 @@
   (raise
(apply
 make-compound-condition
+;; TRANSLATORS: The scheme-format tag is used to identify
+;; strings that contain format directives as specified
+;; here:
+;; https://www.gnu.org/software/guile/manual/html_node/Formatted-Output.html
+;;
+;; The goto/jump directive can be used to alter the order
+;; of the arguments, either performing relative jumps with
+;; ~N* and ~N:* (forward and backwards respectively) or
+;; the absolute position of the argument can be used
+;; (starting from 0) with ~N@*.  When N isn't provided,
+;; it's understood to be 1 on the relative jumps (next and
+;; previous argument respectively) and 0 on the absolute
+;; jumps (first argument).
 (formatted-message (G_ "module ~a not found")
module)
 (condition
-- 
2.29.2

Re: wip-arm-bootstrap: Reduced binary seed bootstrap

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

> Jan Nieuwenhuizen  skribis:
>
>> I pushed an initial wip-arm-bootstrap that builds up to tcc-boot:
>>
>> ./pre-inst-env guix build -e '(@@ (gnu packages commencement) tcc-boot)'
>>
>>
>> Danny did a terrific job on armhf-linux support for GNU Mes and we are
>> getting close to release v0.23.  What's known to be missing is "setjmp"
>> support for tcc/gcc.  And, of course, for the Reduced binary seed
>> bootstrap to work on ARM!
>
> Woohoo, thumbs up for this great milestone, hackers! 

Thanks!!

>> Anyway, the problem I'm looking at now is that while tcc-boot builds on
>> an ARM box, it does not build on aarch64-linux, i.e. this
>>
>> ./pre-inst-env guix build --system=armhf-linux \
>>   -e '(@@ (gnu packages commencement) tcc-boot0)'
>>
>> fails on overdrive1.
>
> That’s “interesting”.  Did you eventually find a lead?

Yes, I added '|| BOOTSTRAP' in tccelf.c:

--8<---cut here---start->8---
#if defined (TCC_ARM_EABI) || BOOTSTRAP
ehdr.e_ident[EI_OSABI] = 0;
ehdr.e_flags = EF_ARM_EABI_VER4;
if (file_type == TCC_OUTPUT_EXE || file_type == TCC_OUTPUT_DLL)
ehdr.e_flags |= EF_ARM_HASENTRY;
if (s1->float_abi == ARM_HARD_FLOAT)
ehdr.e_flags |= EF_ARM_VFP_FLOAT;
else
ehdr.e_flags |= EF_ARM_SOFT_FLOAT;
#else
ehdr.e_ident[EI_OSABI] = ELFOSABI_ARM;
#endif
--8<---cut here---end--->8---

this should be investigated a bit further.  Compiling with TCC_ARM_EABI
fails (which should work, really), but for this linking stage we need it
set to produce an aarch64-linux-compatible ELF header.  Dunno!

> Thanks for the update, it’s exciting!

You're welcome!  Meanwhile, I built gcc-core-mesboot0 (2.95.3).
However, that's using bash from %bootstrap-coreutils

The last thing I did was to investigate into bash-2.05b and found that
bash depends on a working setjmp/longjmp for tinycc.  I tried several
versions of macro substitution on x86 instead of setjmp/longjmp, but
that doesn't seem possible: subshells and such.  So Danny, would you
like to look at that?

Greetings,
Janneke

-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Identical files across subsequent package revisions

[Christopher Baines]

Ludovic Courtès  writes:

> The reason I’m looking at this is to understand how much would be gained
> in terms of bandwidth usage if we were able to avoid downloading
> individual files already in the store.  It would seem to be rather
> encouraging.

Very cool! This might help guide the implementation of using things like
IPFS for substitutes.


signature.asc
Description: PGP signature

Re: SJTUG's Guix mirror is online now

[宋文武]

Peng Mei Yu  writes:

> Hello,
>
> After extensive cooperation with SJTUG's maintainers, we are happy to
> announce that SJTUG's Guix mirror is online now.
>
> The substitute URL is: https://mirrors.sjtug.sjtu.edu.cn/guix
>
> Asia residents, especially China residents, will greatly benefit from
> this mirror.  Average download speed is about 3MB/s within China, which
> is a 100 times increase compared to ci.guix.gnu.org.

Truely a life saver, thank you!

Re: Identical files across subsequent package revisions

[Pierre Neidhardt]

Taylan Kammer  writes:

> On 22.12.2020 23:01, Ludovic Courtès wrote:
>> 
>> Thoughts?  :-)
>> 
>
> My first thought: Neat, would love to see this implemented! :D

I second that! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Identical files across subsequent package revisions

[zimoun]

Hi again,

Sorry, I have not finished my previous email.  Wrong keystroke in the
wrong buffer because misuse of the Ludo’s code. :-)


On Wed, 23 Dec 2020 at 11:19, zimoun  wrote:

>> I wanted to evaluate that by looking at store items corresponding to
>> subsequent revisions of a package (be it different versions or rebuilds
>> induced by dependencies), and this is what the program below does.

What I wanted to illustrate is “revision” does not mean “new version“
and I do not know what is the typical usage by people; aside Guix
dev. :-)

How many times per week or month people are doing “guix pull && guix
upgrade” almost blindly without reviewing what is new?


Last, I was trying to understand what is the “revision” number
corresponding to which outputs.  For example r-minimal:



There are 3 outputs with 4.0.3, so I am expecting 3 similar revisions in
the chart.  Then 3 others with 4.0.2, then 2 others with 4.0.1.

In the 6 revisions in the chart, I was trying to figure out what does
mean the ’0’: r-minimal against which other one, i.e., which Guix commit
against which other one.  Idem for 1,2,3,4,5. :-)


BTW, really interesting topic!  Thanks Ludo. :-)

Cheers,
simon

Re: Identical files across subsequent package revisions

[zimoun]

Hi Ludo,

On Tue, 22 Dec 2020 at 23:01, Ludovic Courtès  wrote:

> I wanted to evaluate that by looking at store items corresponding to
> subsequent revisions of a package (be it different versions or rebuilds
> induced by dependencies), and this is what the program below does.
>
> Here are preliminary results for a few packages:

[...]

> The reason I’m looking at this is to understand how much would be gained
> in terms of bandwidth usage if we were able to avoid downloading
> individual files already in the store.  It would seem to be rather
> encouraging.

This is really interesting.  Especially through normal distribution and
co. lens.  The mean needs to be weighed: 10% of 180MB is not the same
than 55% of 1MB (icecat vs diffoscope, from “guix size” and I could
misread the numbers).  The variance between revisions is highly variable
(from 0 to significant).  The one between packages says how fat or fit
the normal distribution is.

A smarter distribution could lead to some bandwith saving.  The question
is: predict how many?  Is it worth to add a bit of complexity to save
corner cases or for all.

By corner cases, I mean that for example I am not updating my Emacs at
each revisions.






> Below is the program that does that.  It grabs revision history from
> data.guix.gnu.org, fetches nars from ci.guix.gnu.org, computes a
> “digest” (list of files along with their hash and size), compares
> package digests pairwise, and plots the result with Guile-Charting.
> Example REPL session:
>
> --8<---cut here---start->8---
> scheme@(similarities)>  (pairwise-similarities (package-archive-contents 
> "icecat" #:max 4))
> updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> $86 = (17363915/196387883 11380615/98152193)
> scheme@(similarities)> (map exact->inexact $86)
> $87 = (0.08841642740249916 0.11594865740799087)
>
> […]
>
> scheme@(similarities)> ,pp (at-most 10 (package-instances "r-minimal") )
> $100 = (#< version: "4.0.3" output: 
> "/gnu/store/vv3ca1r5zw5y35xgkix4r80hdnncx52b-r-minimal-4.0.3">
>  #< version: "4.0.3" output: 
> "/gnu/store/5dzad7nhhv3dvmap60d6gsj9ppflgzrd-r-minimal-4.0.3">
>  #< version: "4.0.3" output: 
> "/gnu/store/01xi3sig314wgwa1j9sxk37vl816mj74-r-minimal-4.0.3">
>  #< version: "4.0.2" output: 
> "/gnu/store/nv7lqhnm0mncqwdpkkhnlsgb562lcwff-r-minimal-4.0.2">
>  #< version: "4.0.2" output: 
> "/gnu/store/w0izbm8q26dmyndhv46xr7dgz1irai1z-r-minimal-4.0.2">
>  #< version: "4.0.2" output: 
> "/gnu/store/yd83ibzxjrb7cgcc6d4smx4pqcdl8r3p-r-minimal-4.0.2">
>  #< version: "4.0.1" output: 
> "/gnu/store/kpdh0lwlwcwfmmfzqhwbi6j7m4zzxlmn-r-minimal-4.0.1">
>  #< version: "4.0.1" output: 
> "/gnu/store/9x9nzzsiyn1q7g5myhgwjh0yx9js3nrj-r-minimal-4.0.1">
>  #< version: "4.0.0" output: 
> "/gnu/store/ahbm2gsqc3420a23pcwrxd4pdhl7rdpp-r-minimal-4.0.0">
>  #< version: "4.0.0" output: 
> "/gnu/store/0sgqhj2628js419wvw1vc1cw07wil7gr-r-minimal-4.0.0">)
> $101 = (#< version: "3.6.3" output: 
> "/gnu/store/gmx6p0wk3xbc9ylv83zfj855azgjxr0p-r-minimal-3.6.3">
>  #< version: "3.6.2" output: 
> "/gnu/store/dnb6fzp5295fcda66dnjk2y51mcva20f-r-minimal-3.6.2">
>  #< version: "3.6.1" output: 
> "/gnu/store/gd6sm42b6fr1qgyp6p1zp3z4aavxwyk2-r-minimal-3.6.1">
>  #< version: "3.6.1" output: 
> "/gnu/store/lpmfhxys3vsaqmqvj85r344ygfmlmlbg-r-minimal-3.6.1">
>  #< version: "3.6.1" output: 
> "/gnu/store/4413h13v7zrb7rp9lvyp2gfx2laj60wm-r-minimal-3.6.1">
>  #< version: "3.6.1" output: 
> "/gnu/store/zm5pl1y0wmh3c845498pbjvrzrm6sb07-r-minimal-3.6.1">
>  #< version: "3.6.1" output: 
> "/gnu/store/xrv7y4xgrdrsx5qba7144cpw69qc5f0j-r-minimal-3.6.1">
>  #< version: "3.6.0" output: 
> "/gnu/store/cbwhhqv69xi3k3g25dcfhwjkkf2427rp-r-minimal-3.6.0">
>  #< version: "3.6.0" output: 
> "/gnu/store/69k46yr70zkzzz9v18wl7sxasha5m0a5-r-minimal-3.6.0">
>  #< version: "3.6.0" output: 
> "/gnu/store/71w0383x0hay6ng1zaddz59by3g0gxaf-r-minimal-3.6.0">
>  #< version: "3.6.0" output: 
> "/gnu/store/m317mg8faxp9qn949dnv1xgsxyw57s3x-r-minimal-3.6.0">
>  #< version: "3.5.3" output: 
> "/gnu/store/33qdfplk4riig77vqg758m1zll16n6f0-r-minimal-3.5.3">
>  #< version: "3.5.3" output: 
> "/gnu/store/g8gkrcxn49yj8zjr59l7y4k7wgar5brq-r-minimal-3.5.3">
>  #< version: "3.5.1" output: 
> "/gnu/store/vrgbyvnx7b1sva2ij5a1gwrkbfwmg3lm-r-minimal-3.5.1">
>  #< version: "3.5.1" output: 
> "/gnu/store/4fzw7s0cv2zbixw4wb58zq6lkd97ghnf-r-minimal-3.5.1">
>  #< version: "3.5.1" output: 
> "/gnu/store/yb5048dr40nbmyfa1ar4hfiy3kd06v4c-r-minimal-3.5.1">)
> scheme@(similarities)> (similarity-chart '("icecat" "gimp" "openmpi" "emacs" 
> "diffoscope" "r-minimal") "/tmp/t.png" #:max 8)
> updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> $102 = #
> --8<---cut here---end--->8---
>
> Thoughts?  :-)
>
> Ludo’.
>
> ;;; Copyright © 2020 Ludovic Courtès 
> ;;; Hereby placed under the GNU General Public License, 

Re: Identical files across subsequent package revisions

[Taylan Kammer]

On 22.12.2020 23:01, Ludovic Courtès wrote:


Thoughts?  :-)



My first thought: Neat, would love to see this implemented! :D

My second thought: it's surprising that IceCat supposedly changes so 
much between releases.  I suppose the reason is that this analysis is on 
a per-file basis, and IceCat is mostly a massive binary.  That leads me 
to wonder: what about binary diffs for large files?


Perhaps for all files that are bigger than N bytes, we could check if 
the binary diff is X% or smaller of the total size, and if so, the build 
servers could host the diff file alongside the full file.  (Substitute N 
and X for sensible values, like maybe 5 MB and 50%.)


But that could be a second, separate step I suppose.

- Taylan

Re: SJTUG's Guix mirror is online now

[zimoun]

Hi,

On Wed, 23 Dec 2020 at 09:50, Peng Mei Yu  wrote:

>>> The substitute URL is: https://mirrors.sjtug.sjtu.edu.cn/guix

Nice news!


>> If you’re interested, you could write an article for the blog post (in
>> English, but it’d be a good opportunity to start translating the web
>> site into Chinese :-)) about your setup and the choices you made.
>
> I don't think there are enough things to write in a blog post but I will
> consider translating the web site.

Does it make sense to add the URL in the Chinese manual?

All the best,
simon