Re: Go importer and packages with version flags

[Sarah Morgensen]

Hi Katherine, Jack,

Katherine Cox-Buday  writes:

> Jack Hill  writes:
>
>> Hi Guix,
>
> Hey, Jack, a few thoughts.
>
>> While I was was working with the go importer today, it suggested I package
>> go-github-com-russross-blackfriday-v2. Fair enough, except we already have a
>> package for go-github-com-russross-blackfriday.
>
> I was poking around a rust code-base the other day and I noticed our crate
> importer, and thus a lot of crate packages, have major-version suffixes. I
> think one of the unique benefits of Guix is that it can simultaneously have
> multiple versions of libraries installed, and I think we should allow this
> for library packages.
>
> I know that leads to dependency graph explosion, but perhaps we only commit
> to substitutes for the latest version, and thus any packages using old
> versions. It should converge over time unless packages go unmaintained.
>
> I thought our current stance was to only allow one version at a time, but
> the crate packages made me question this. I'd like clarity too.

I think there's a bit of a difference between (our packages for) the Rust and
Go ecosystems here.

In the Go ecosystem, a module is uniquely identified by its module path,
e.g. "github.com/russross/blackfriday/v2".  According to Go's major version
suffix rules [0], "[s]tarting with major version 2, module paths must have a
major version suffix like /v2 that matches the major version."  Therefore,
each major version is logically a different module according to Go, and so I
think we should treat them as separate packages as well.  (Note that in many
cases we can use 'inherit' for brevity.)

Additionally, the major version suffix rules dictate that "[i]f an old package
and a new package have the same import path, the new package must be backwards
compatible with the old package."  Assuming upstream sources follow the rules,
we should be able to update each Go package within each major version without
breaking dependencies.

(A corollary to that is that packages often break if you try to use a v2 when
it is expecting a v1.)

I think this differs from Rust, where we have, for example, package-0.1 and
package-0.2 coexisting.  This difference should prevent dependency graph
explosion for Go.

There are some caveats with "major version suffixes":

* Major versions 0 and 1 don't get a version suffix (so no /v1)...

* ...except for module paths starting with "gopkg.in/", which always have
  a major version suffix, but theirs is styled ".v1" rather than "/v1".

* Modules may either be located in the repository root, or in a "/v2"
  subdirectory (for major version 2).  This makes things difficult for our
  importer, because we can't know whether the unpack path should include "/v2"
  without looking at the repository contents.

This is why Jack had to manually add "/v2" to the unpack path.  I recently
made some changes to the importer to better handle, for example,
"github.com/example/repository/subproject", but it doesn't yet discriminate
between "/subproject" and "/v2", so it treated "/v2" like a subdirectory of
the repository.  (Until we fix this properly, the importer should probably not
count major version suffixes when calculating the unpack path, since most
projects don't use a "/v2" subdirectory.)


All that to say... I think we should definitely maintain coexisting Go v2, v3,
etc. package definitions.  We should probably go the way of Rust though, so we
have them all in the same package, at different versions:

(define-public go-github-com-russross-blackfriday-v2
  (package
(name "go-github-com-russross-blackfriday")
(version "2.1.0")

instead of as different packages:

(define-public "go-github-com-russross-blackfriday-v2"
  (package
(name "go-github-com-russross-blackfriday-v2")
(version "2.1.0")

And of course, it should be policy to remove dependency packages with no
dependents.  (Perhaps we could write a new linter to warn if a "go-" package
has no inheriters and no dependents?)

Does that sound reasonable?

--
Sarah

Re: Linux-libre source code will be taken offline

[Jason Self]

On Mon, 27 Sep 2021 19:30:29 -0400
Leo Famulari  wrote:

> I didn't check that the files are bit-identical, but my understanding
> is that the "old" revisions of the deblobbing scripts are available
> here:
> 
> https://linux-libre.fsfla.org/pub/linux-libre/releases/old/

Yes. In gen6. They have been moved, not deleted.

The versioning and locations in terms of gnuN and genN are knowable and
predictable in advance. I wonder if there is, or could be made, a way to
leverage that so that future moving of files can be done without
causing problems, as long as the files themselves remain otherwise
identical. As an example, the current cleanup scripts might be found in
old/gen7 in the future. Although using git would probably be a better
choice as it would seem to eliminate URL hunting.


pgpiE2tI6auaf.pgp
Description: OpenPGP digital signature

Re: Go importer and packages with version flags

[Katherine Cox-Buday]

Jack Hill  writes:

> Hi Guix,

Hey, Jack, a few thoughts.

> While I was was working with the go importer today, it suggested I package
> go-github-com-russross-blackfriday-v2. Fair enough, except we already have a
> package for go-github-com-russross-blackfriday.

I was poking around a rust code-base the other day and I noticed our crate 
importer, and thus a lot of crate packages, have major-version suffixes. I 
think one of the unique benefits of Guix is that it can simultaneously have 
multiple versions of libraries installed, and I think we should allow this for 
library packages.

I know that leads to dependency graph explosion, but perhaps we only commit to 
substitutes for the latest version, and thus any packages using old versions. 
It should converge over time unless packages go unmaintained.

I thought our current stance was to only allow one version at a time, but the 
crate packages made me question this. I'd like clarity too.

> The packages differ in their version (2.1.0 verses 2.0.1), their imputs (the
> imported package definition is missing the inputs and native-inputs that we
> have in the Guix package), and the presense of the v2 versioning
> disambiguation tag.

The importer suggested package is correct. It looks like the listed 
dependencies dropped >= v1.5. See:

- 
https://pkg.go.dev/github.com/russross/blackfriday@v2.0.0+incompatible?tab=imports
vs
- https://pkg.go.dev/github.com/russross/blackfriday/v2?tab=imports

I wasn't sure if pkg.go listed test dependencies, so I ran this command to be 
sure:

#+begin_example
$ go list -deps -test -f '{{.ImportPath}} {{.Standard}}' |awk '{if ($2 == 
"false")  print $1}'
github.com/russross/blackfriday/v2
github.com/russross/blackfriday/v2.test
#+end_example

I think those were just dependencies incorrectly carried over as the package 
changed.

I hope that helps.

-- 
Katherine

Re: Code sharing between system and home services (was Re: On the naming of System and Home services modules.)

[Maxim Cournoyer]

Hi,

Xinglu Chen  writes:

> On Thu, Sep 23 2021, Ludovic Courtès wrote:
>
>> Hi,
>>
>> Xinglu Chen  skribis:
>>
>>> Some services might be useful to have in both Guix System and Guix Home;
>>> for instance, Guix System currently has a service for configuring
>>> Syncthing, and I think it makes sense to also have one for Guix Home,
>>> this would mean that people not using Guix System (me :-)) could also
>>> have Guix manage Syncthing.  With the current approach, we would have to
>>> copy and paste quite a bit of code, and if the Syncthing service for
>>> Guix System changes, then the one for Guix Home might have to change as
>>> well.
>>
>> Silly question, but why do we need to have two different configuration
>> record types in the first place?
>
> The problem is that the configuration records for system and home
> service don’t necessarily have the same fields.  The Syncthing service
> for Guix System has a ‘user’ and a ‘group’ field, which is not really of
> any use in Guix Home, as the only user would be the user invoking ‘guix
> home’.
>
>> Sharing configuration between Home and System sounds important to me: it
>> means users can easily move services from one to the other, which is
>> pretty big deal.  It also means we’d have much less code to maintain.
>
> Agreed, that’s what I would like to see as well.
>
>> Would that be feasible?  (Apologies if this has already been
>> discussed!)
>
> Since it might not make sense to have the same records fields for a
> system service and home service, I proposed (in the mail you replied to)
> a ‘define-configuration’ form that would generate a configuration record
> for a system service and optionally one for a home service, without
> having to maintain two records separately.
>
> (define-configuration syncthing-configuration
>   (package
>(package syncthing)
>"Syncthing package to use.")
>   (arguments
>(list-of-strings ’())
>"Command line arguments to pass to the Syncthing package.")
>   (log-flags
>(integer 0)
>"Sum of logging flags.")
>   (user
>(maybe-string 'disabled)
>"The user as which the Syncthing service is to be run."
>(home-service? #f))  ; not for Guix Home
>   (group
>(string "users")
>"The group as which the Syncthing service is to be run."
>(home-service? #f))  ; likewise ^^
>   (home
>(maybe-string 'disabled)
>"Common configuration and data directory.")
>   (home-service? #t))
>
> It would generate  and
> .  The only difference being that
>  doesn’t have a ‘user’ and a ‘group’
> field.

Interesting idea, although I'm a bit wary of adding yet more complexity
to this already relatively complex macro.  I'd favor the cleaner
inheritance idea proposed by Maxime Devos.  I wonder if some fields can
even be masked or removed through inheritance or some other record
transformation (haven't checked).

> It’s probably going to be quite complicated, so it would be good to get
> some feedback/thoughts on it.  Cc Maxim since he has done some work with
> (gnu services configuration).
>
> Also, it’s probably time to properly document (gnu services
> configuration) in the manual.  ;-)

Agreed!

Maxim

Re: Linux-libre source code will be taken offline

[zimoun]

(Sorry, for the typos.)

On Tue, 28 Sept 2021 at 02:13, zimoun  wrote:

> even if it happens, SWH does rebuild the exact same tarball (because

SWH does *NOT* rebuild the exact same tarball.

Re: Linux-libre source code will be taken offline

[zimoun]

Hi,

On Tue, 28 Sept 2021 at 01:34, Leo Famulari  wrote:

> Of course, adding this to the list of URIs in linux-libre-deblob-scripts
> won't help users of old Guix revisions... Software Heritage and other
> archives that support content-addressed lookups are the only solution
> for that.

I do not know how Software Heritage (SWH) handles this case.
Especially for these deblob scripts.  For informatio, the current
status* about SWH is:

  - all the git-fetch packages are "saved" by running manually "guix
lint -c archival" or click to the Save Button on SWH webpage. :-)
  - (almost) all the url-fetch are ingested by SWH using the
guix.gnu.org/sources.json

The content-adressed lookup works fine for Git repo (from channel to
package).  However, it is not ready yet for tarballs.  In short, at
package time, we have a checksum which is content+metadata; then SWH
archives only the content and drops the metadata; and this content is
content-addressed using SWH-ID.  At fallback time, the knowlegde of
checksum does not guarantee to be able to lookup for the content.  And
even if it happens, SWH does rebuild the exact same tarball (because
of the metadata drops), i.e., there is a high probably to have a
checksum mismatch somehow.

Therefore, here it is the aime of Disarchive.  It somehow builds this
map between the checksum and the SWH content-addressed content.

Bricks are there but missing glue. :-)

*current status about SWH: if I have not missed a recent feature. ;-)


An improvement is done by the patch to add these computed origins to
sources.json; see here:



and this patch is blocked by missing comment on this one:
.

However, if now upstream removed the material, bah it is another story
to save this very same material to SWH. :-)

All the best,
simon

Re: Linux-libre source code will be taken offline

[Leo Famulari]

On Mon, Sep 27, 2021 at 06:30:21PM -0400, Mark H Weaver wrote:
> If we wish to preserve Guix users' ability to reproduce older systems,
> we will need an 'origin' to fetch the Linux-libre deblob scripts from
> that has a policy of retaining older releases, unchanged and at a fixed
> location.  Apparently the HTTPS URLs at 
> that we currently use are not suitable for that purpose.

I didn't check that the files are bit-identical, but my understanding is
that the "old" revisions of the deblobbing scripts are available here:

https://linux-libre.fsfla.org/pub/linux-libre/releases/old/

Of course, adding this to the list of URIs in linux-libre-deblob-scripts
won't help users of old Guix revisions... Software Heritage and other
archives that support content-addressed lookups are the only solution
for that.

Re: Linux-libre source code will be taken offline

[Vagrant Cascadian]

On 2021-09-27, Mark H Weaver wrote:
> Leo Famulari  writes:
>
>> On Sat, Sep 04, 2021 at 01:32:16PM -0700, Jason Self wrote:
>>> The scripts are not being removed and my understanding is that Guix
>>> only uses the scripts anyway.
>>
>> Okay, that's great. We do use the scripts.
>
> Unfortunately, the older deblobbing scripts have now been removed from
> the HTTPS URLs that our linux-libre build recipes fetched them from, so
> I guess there will now be difficulties for users trying to reproduce any
> Guix system more than about a month old.
>
> If we wish to preserve Guix users' ability to reproduce older systems,
> we will need an 'origin' to fetch the Linux-libre deblob scripts from
> that has a policy of retaining older releases, unchanged and at a fixed
> location.  Apparently the HTTPS URLs at 
> that we currently use are not suitable for that purpose.
>
> The Linux-libre git repository *might* be suitable, but I haven't yet
> seen a commitment from the Linux-libre project that they will retain the
> tags for older flawed deblobbing scripts in their repository going
> forward.  Without tags protecting them, I guess the commits could be
> deleted by 'git-gc' even if we referenced them directly by their commit
> hashes.
>
> Perhaps the SWH fallback is the answer we need, if they are archiving
> the Linux-libre git repository.  Does anyone know if they are?

The most promising two I found were:

  
https://archive.softwareheritage.org/browse/origin/directory/?origin_url=git://linux-libre.fsfla.org/releases.git
  
https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://jxself.org/git/linux-libre.git

Not sure exactly how Software Heritage handles rebased branches...

live well,
  vagrant


signature.asc
Description: PGP signature

Re: Linux-libre source code will be taken offline

[Mark H Weaver]

Leo Famulari  writes:

> On Sat, Sep 04, 2021 at 01:32:16PM -0700, Jason Self wrote:
>> The scripts are not being removed and my understanding is that Guix
>> only uses the scripts anyway.
>
> Okay, that's great. We do use the scripts.

Unfortunately, the older deblobbing scripts have now been removed from
the HTTPS URLs that our linux-libre build recipes fetched them from, so
I guess there will now be difficulties for users trying to reproduce any
Guix system more than about a month old.

If we wish to preserve Guix users' ability to reproduce older systems,
we will need an 'origin' to fetch the Linux-libre deblob scripts from
that has a policy of retaining older releases, unchanged and at a fixed
location.  Apparently the HTTPS URLs at 
that we currently use are not suitable for that purpose.

The Linux-libre git repository *might* be suitable, but I haven't yet
seen a commitment from the Linux-libre project that they will retain the
tags for older flawed deblobbing scripts in their repository going
forward.  Without tags protecting them, I guess the commits could be
deleted by 'git-gc' even if we referenced them directly by their commit
hashes.

Perhaps the SWH fallback is the answer we need, if they are archiving
the Linux-libre git repository.  Does anyone know if they are?

Thoughts?

 Thanks,
   Mark

 Start of forwarded message 
From: Alexandre Oliva 
To: linux-li...@fsfla.org, info-...@gnu.org
Subject: GNU Linux-libre's nonfree releases removed
Date: Mon, 27 Sep 2021 14:08:01 -0300

https://linux-libre.fsfla.org/#retired-2021-09-27
2021-09-27 - Removing obsolete releases and repositories

We celebrate the 38th anniversary of the GNU Project and of the Free
Software movement by removing past releases found to contain nonfree
software.  Their cleaning-up scripts remain available from the git
repository, and from releases/old/gen6.

We're also removing long-obsolete repositories that still contained
builds of those and of even earlier releases.  Instead of freed-ebian
and planet, we recommend Freesh. Instead of rt, we recommend LibeRTy.  A
freeloong replacement might be added to Freesh if there's enough demand.

Freed-ora repositories for recent Fedora releases remain available, but
we're still looking for new maintainers for Freed-ora.

-- 
Alexandre Oliva, happy hackerhttps://FSFLA.org/blogs/lxo/
   Free Software Activist   GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about 

-- 
If you have a working or partly working program that you'd like
to offer to the GNU project as a GNU package,
see https://www.gnu.org/help/evaluation.html.
 End of forwarded message 

-- 
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about .

Go importer and packages with version flags

[Jack Hill]

Hi Guix,

While I was was working with the go importer today, it suggested I package 
go-github-com-russross-blackfriday-v2. Fair enough, except we already have 
a package for go-github-com-russross-blackfriday. The packages differ in 
their version (2.1.0 verses 2.0.1), their imputs (the imported package 
definition is missing the inputs and native-inputs that we have in the 
Guix package), and the presense of the v2 versioning disambiguation tag.


I was puzzled by this last bit since both packages seem to compatible with 
the v2 API. If I replace the exising package definition with the one 
suggested by the importer (with one small tweak, I needed to add a /v2 to 
the #:unpack-path) all the packages reported by `guix refresh -l 
go-github-com-russross-blackfriday` build sucessfully.


What's the path forward here? I'm thinking I should update our blackfriday 
backage to the 2.1.0 release, but leave the inputs and native inputs. I'm 
less sure about what to do about the v2 tag. Should I switch over the 
package and varialbe name? What do we usually do?


Best,
Jack

The package the importer suggested:

```
(define-public go-github-com-russross-blackfriday-v2
  (package
(name "go-github-com-russross-blackfriday-v2")
(version "2.1.0")
(source
  (origin
(method git-fetch)
(uri (git-reference
   (url "https://github.com/russross/blackfriday;)
   (commit (string-append "v" version
(file-name (git-file-name name version))
(sha256
  (base32 "0d1rg1drrfmabilqjjayklsz5d0n3hkf979sr3wsrw92bfbkivs7"
(build-system go-build-system)
(arguments
  '(#:import-path
"github.com/russross/blackfriday/v2"
#:unpack-path
"github.com/russross/blackfriday"))
(home-page "https://github.com/russross/blackfriday;)
(synopsis "Blackfriday")
(description "Package blackfriday is a markdown processor.
")
(license license:bsd-2)))
```

Our existing package:

```
(define-public go-github-com-russross-blackfriday
  (package
(name "go-github-com-russross-blackfriday")
(version "2.0.1")
(source
 (origin
   (method git-fetch)
   (uri (git-reference
 (url "https://github.com/russross/blackfriday;)
 (commit (string-append "v" version
   (file-name (git-file-name name version))
   (sha256
(base32
 "0nlz7isdd4rgnwzs68499hlwicxz34j2k2a0b8jy0y7ycd2bcr5j"
(build-system go-build-system)
(arguments
 '(#:import-path "github.com/russross/blackfriday"))
(propagated-inputs
 `(("go-github-com-shurcool-sanitized-anchor-name"
,go-github-com-shurcool-sanitized-anchor-name)))
(native-inputs
 `(("go-github-com-pmezard-go-difflib" ,go-github-com-pmezard-go-difflib)))
(home-page "https://github.com/russross/blackfriday;)
(synopsis "Markdown processor in Go")
(description "Blackfriday is a Markdown processor in Go.")
(license license:bsd-2)))
```

Re: branch core-updates-frozen updated: services: database: Change postgresql default socket.

[Xinglu Chen]

On Mon, Sep 27 2021, Mathieu Othacehe wrote:

> Hello,
>
>> The manual should also be updated to reflect this change.
>
> Fixed, thanks for the reminder.

You are welcome!  :-)

>> Also, is there any reason for making this change on
>> ‘core-updates-frozen’ rather than ‘master’?
>
> Yes, that's a follow-up of 86cf4c039631cdf15c4c428bf4ffe52d6cbd7d4c that
> was pushed on core-updates a while ago.

Ah, OK.


signature.asc
Description: PGP signature

Re: branch core-updates-frozen updated: services: database: Change postgresql default socket.

[Mathieu Othacehe]

Hello,

> The manual should also be updated to reflect this change.

Fixed, thanks for the reminder.

> Also, is there any reason for making this change on
> ‘core-updates-frozen’ rather than ‘master’?

Yes, that's a follow-up of 86cf4c039631cdf15c4c428bf4ffe52d6cbd7d4c that
was pushed on core-updates a while ago.

Thanks,

Mathieu

Re: branch core-updates-frozen updated: services: database: Change postgresql default socket.

[Xinglu Chen]

On 27 September 2021 15:26, guix-comm...@gnu.org wrote:

> commit 502925655d1a51aad544804c8ef492a5d24e1776
> Author: Mathieu Othacehe 
> AuthorDate: Mon Sep 27 19:22:56 2021 +
>
> services: database: Change postgresql default socket.
>
> Adapt to the postgresql default socket directory set to 
> /var/run/postgresql.
>
> * gnu/services/databases.scm 
> ()[socket-directory]: Set
> to /var/run/postgresql.
> (): Ditto.
> * gnu/tests/databases.scm (run-postgresql-test): Adapt it.
> ---
>  gnu/services/databases.scm | 4 ++--
>  gnu/tests/databases.scm| 3 +--
>  2 files changed, 3 insertions(+), 4 deletions(-)

The manual should also be updated to reflect this change.

Also, is there any reason for making this change on
‘core-updates-frozen’ rather than ‘master’?




signature.asc
Description: PGP signature

Building guix system

[Gh0sty]

Hello very cool project, please, can you describe how to build live 
image of guix?



With best wishes,

Gh0sty

Re: minification R: build-system

[Charles]

I think what would be ideal is packaging the Javascript dependencies 
separately, then including them as inputs. I wouldn't be suprised if someone 
has tried this, but I don't know the history.

‐‐‐ Original Message ‐‐‐

On Monday, September 27th, 2021 at 12:28 PM, zimoun  
wrote:

> Hi,
>
> Recently, the switch from uglify-js to uglifyjs raises a question if
>
> this minification should be part of the r-build-system; as suggested
>
> by Ricardo:
>
> http://logs.guix.gnu.org/guix-hpc/2021-09-22.log#160424
>
> What people think to this move? Because, currently the replacement of
>
> the minifier means grep and replace all by hand, i.e., prone-error.
>
> It should be nice to add a step for the R packages. WDYT?
>
> All the best,
>
> simon

minification R: build-system

[zimoun]

Hi,

Recently, the switch from uglify-js to uglifyjs raises a question if
this minification should be part of the r-build-system; as suggested
by Ricardo:



What people think to this move?  Because, currently the replacement of
the minifier means grep and replace all by hand, i.e., prone-error.
It should be nice to add a step for the R packages.  WDYT?


All the best,
simon