Re: Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

Ricardo Wurmus writes:

> I’m aware.  I lived with the firewall for ~7 years and would like to
> make sure that people who are subjected to the firewall can use Guix
> without being restricted.

Alas, why does this happen everywhere in this decade...

>> We hope there's a way like rsync to sync /gnu/store with upstream, so
>> that we can provide a faster cache for Chinese users.
>
> I think it may be best to just sync the cache of nars and narinfos.

Yes it's better to just sync the necessary parts.

> Would rsync over SSH be sufficient or does it have to be rsyncd on the
> default rsync port?  (SSH will be easier for me because then I don’t
> need to apply for another port to be opened at the institute firewall.)

I think SSH will be interfered frequently by the firewall, because many
people use SSH for anti-circumvention. It's better to use rsyncd.
Considering there's verification, so the encryption is unnecessary.

> I’ve also been looking into our options for restricting rsync access via
> chroot or namespaces.  Looks like the easiest way to do this is to have
> an rsync user account that is restricted to a chroot with access to the
> nar cache.

I think it can be configured as anonymouse read-only authority. And
open the necessary directories.


Best regards.

--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058


signature.asc
Description: PGP signature

IPv6 Guix

[dftxbs3e]

Hello,

It looks like ci.guix.gnu.org does not have  records.

I suggest they get added because IPv6-only networking is the future.

$ dig ci.guix.gnu.org 

; <<>> DiG 9.11.8-RedHat-9.11.8-1.fc30 <<>> ci.guix.gnu.org 
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4389
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ci.guix.gnu.org.        IN    

;; AUTHORITY SECTION:
guix.gnu.org.        3595    IN    SOA    ns.guix.gnu.org.
hostmaster.guix.gnu.org. 2019080102 172800 900 1209600 3600

I checked all other domain names in

and they all have  records.

Now if anything is downloaded by GNU Guix during operation, domain names
should also have  records but I think ci.guix.gnu.org mirrors
everything?

Thanks




signature.asc
Description: OpenPGP digital signature

Re: "python setup.py test" is deprecated

[Chris Marusich]

Marius Bakke  writes:

> Pythons setuptools are deprecating the "python setup.py test" command:
>
> https://github.com/pypa/setuptools/issues/1684
>
> As you may know, "python setup.py test" is what python-build-system does
> during the 'check' phase.
>
> I'm not sure what we should do about it, and there does not seem to be
> an established "community consensus" yet.  I suppose many packages will
> be migrating to Pytest and/or Tox?  Perhaps we could check package
> inputs and try to guess the correct test command?
>
> Something to keep in mind for future python-build-system improvements...
>
> PS: Apparently "python setup.py install" is going away too:
> https://github.com/pypa/setuptools/issues/1684#issuecomment-508302910

To my knowledge, the "targets" of the setup.py script are somewhat not
standardized, even though they seem like they ought to be.  At the very
least, you are right about the "test" target.  Even pytest dropped
support for it since last year:

https://docs.pytest.org/en/latest/changelog.html?highlight=%22setup.py%20test%22#id511

This is just one example of how a project can implement its own
idiosyncratic build logic.  Somebody uses tox, someone else uses a
Makefile with a "check" target (that runs pytest), somebody else just
runs pytest directly...  It's business as usual as far as I can tell.
Guix can accommodate all those idiosyncrasies via modifications to the
phases, and if the Python community starts using one method much more
often, we can just make that the default in the build system.

-- 
Chris


signature.asc
Description: PGP signature

Re: Support rsync to help Chinese users to setup mirrors

[Ricardo Wurmus]

Hi,

> There is the national firewall in China, so common users have difficult
> to download Guix packages in a fair speed. This may hardly understand by
> people outside China, but that is the fact.

I’m aware.  I lived with the firewall for ~7 years and would like to
make sure that people who are subjected to the firewall can use Guix
without being restricted.

> We hope there's a way like rsync to sync /gnu/store with upstream, so
> that we can provide a faster cache for Chinese users.

I think it may be best to just sync the cache of nars and narinfos.

Would rsync over SSH be sufficient or does it have to be rsyncd on the
default rsync port?  (SSH will be easier for me because then I don’t
need to apply for another port to be opened at the institute firewall.)

I’ve also been looking into our options for restricting rsync access via
chroot or namespaces.  Looks like the easiest way to do this is to have
an rsync user account that is restricted to a chroot with access to the
nar cache.

--
Ricardo

Re: Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

Hi Richardo!

Ricardo Wurmus writes:

> Guix clients fetch binaries in nar and narinfo format.  Mirroring them
> might be sufficient.  To create the cache requires running “guix
> publish” with a /gnu/store, which cannot easily be mirrored as it needs
> to be in sync with its database.

What I expect is to mirror nar/narinfo conveniently and fast, that is
the purpose why I raise this topic. And I know to run "guix publish" to
create a repo for users, the problem I want to solve is to sync
/gnu/store with the upstream conveniently and fast.

There is the national firewall in China, so common users have difficult
to download Guix packages in a fair speed. This may hardly understand by
people outside China, but that is the fact.

We hope there's a way like rsync to sync /gnu/store with upstream, so
that we can provide a faster cache for Chinese users.


Best regards.

--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058


signature.asc
Description: PGP signature

Re: bug#36855: guix system switch-generation doesn't

[Robert Vollmert]

On 8. Aug 2019, at 18:40, Chris Marusich  wrote:
> zerodaysford...@sdf.lonestar.org (Jakob L. Kreuze) writes:
> 
>> 'switch-to-system-generation' doesn't call out to
>> 'upgrade-shepherd-services'. I'm not sure if this was an intentional
>> decision or not
> 
> It is intentional, but only because there is currently no way to call
> upgrade-shepherd-services when switching system generations.

How does shepherd work on a non-guix system? Can’t be it be configured
like other daemons to read its configuration from a file, e.g. from

   /run/current-system/etc/shepherd.conf

and be told via signal to reload its configuration from disk?

…

(I feel a bit cheated right now. This behaviour makes Guix System entirely
unsuitable for server use. It shouldn’t be advertised as supporting
transactional upgrades and rollbacks if those require a reboot.)

Re: bug#36855: guix system switch-generation doesn't

[Chris Marusich]

Hi Jakob,

zerodaysford...@sdf.lonestar.org (Jakob L. Kreuze) writes:

> 'switch-to-system-generation' doesn't call out to
> 'upgrade-shepherd-services'. I'm not sure if this was an intentional
> decision or not

It is intentional, but only because there is currently no way to call
upgrade-shepherd-services when switching system generations.

Consider the procedure upgrade-shepherd-services: you must pass it an
 record.  When you are flipping from one generation to
another, how do you get the  record that was used for
the generation you're switching to?  Guix doesn't currently store the
operating system configuration file or its  record
anywhere, so we can't call upgrade-shepherd-services.

This was discussed in 2016 and we agreed we need to persist some
information to enable us to handle Shepherd services correctly.  This is
what Ludo suggested at the time:

https://lists.gnu.org/archive/html/guix-devel/2016-06/msg00173.html

"Maybe we could store in the system output (result of ‘guix system
build’) an sexp representation of (part of) our 
records:

  (shepherd-service
(provisions (x y z))
(requirements (a b c))
(start-script "/gnu/store/…-start-foo.scm")
(stop-script "/gnu/store/…-stop-foo.scm")
…)

Then ‘upgrade-shepherd-services’ could start from this simplified
representation instead of using the full-blown 
objects, and thus could work both when instantiating a new generation
and when rolling back."

Until that happens, you'll always have to reboot to complete the
switch.

FYI, last I checked (about 3 years ago), in NixOS they took a slightly
different approach: instead of storing state describing the previous
system generation and relying on the current system's logic to correctly
parse it and use it to revert the system to a prior configuration, they
just dump everything into a self-contained script that knows how to
update the entire system to one specific configuration.  That approach
is nice in some ways because switching generations is dead simple - you
just run the switching script belonging to the generation you want to
switch to - but it also has downsides.

For example, if the target generation is old enough compared to the
current system, then the target generation's old switching script might
not understand how to deal with the current system correctly.  Instead,
if you only store the bare minimum of state required to take the right
actions, and you implement the meat of the logic in the current Guix
installation, you are more likely to be able to switch generations even
to very old ones where the world was very different, since the current
Guix can be taught how to deal gracefully with the old world.  But it
seems more complicated.  It's all about trade-offs.

That said, I've never gone back to implement this.  If you want to give
it a try, you're more than welcome to do so!

-- 
Chris


signature.asc
Description: PGP signature

Re: Please merge wip-haskell-updates (Re: [bug#36807] remove obsolete broken haskell packages)

[Marius Bakke]

Timothy Sample  writes:

> Hi,
>
> Robert Vollmert  writes:
>
>> On 8. Aug 2019, at 15:12, Marius Bakke  wrote:
>>> I have one comment about the series: we've disabled tests on some
>>> packages that have been broken "forever" on i686.  It would be better to
>>> do so selectively on just the affected architectures.  I.e.:
>>> 
>>> #:tests? (if (string-prefix? "i686" (%current-system))
>>>  #f
>>>  #t)
>>> 
>>> Preferably with a comment about why they need to be disabled.  That way,
>>> we will still notice when something breaks on other architectures.  Can
>>> you try it Rob?
>>
>> I don’t intend to, because I think the effort is better spent elsewhere.
>> But do make the change if you like!
>>
>> The rough plan from my point of view would be to aim for an upgrade of
>> the haskell packages to build with ghc-8.6 from a recent stackage LTS
>> set, and reevaluate skipped tests across the set while doing that or
>> once that’s done.
>
> This was in the back of my mind, too.  Stackage LTS 14.0 (built on top
> of GHC 8.6.5) was released three days ago.  Upgrading will involve
> sweeping changes to the whole set of Haskell packages, giving us lots of
> opportunities to revisit these failing tests.  If we still have problems
> with the i686 tests, we can make them conditional then.
>
> Thanks for pointing this out though, Marius.  I had thought about making
> them conditional when reviewing, but I second guessed myself because we
> have a lot of packages with comments like “tests fail on architecture X”
> followed by an absolute “#:tests? #f”.  If this comes up in the future,
> I’ll just go ahead and make the tests conditional.

OK.  Thanks to both of you for clarifying :-)


signature.asc
Description: PGP signature

Re: Please merge wip-haskell-updates (Re: [bug#36807] remove obsolete broken haskell packages)

[Timothy Sample]

Hi,

Robert Vollmert  writes:

> On 8. Aug 2019, at 15:12, Marius Bakke  wrote:
>> I have one comment about the series: we've disabled tests on some
>> packages that have been broken "forever" on i686.  It would be better to
>> do so selectively on just the affected architectures.  I.e.:
>> 
>> #:tests? (if (string-prefix? "i686" (%current-system))
>>  #f
>>  #t)
>> 
>> Preferably with a comment about why they need to be disabled.  That way,
>> we will still notice when something breaks on other architectures.  Can
>> you try it Rob?
>
> I don’t intend to, because I think the effort is better spent elsewhere.
> But do make the change if you like!
>
> The rough plan from my point of view would be to aim for an upgrade of
> the haskell packages to build with ghc-8.6 from a recent stackage LTS
> set, and reevaluate skipped tests across the set while doing that or
> once that’s done.

This was in the back of my mind, too.  Stackage LTS 14.0 (built on top
of GHC 8.6.5) was released three days ago.  Upgrading will involve
sweeping changes to the whole set of Haskell packages, giving us lots of
opportunities to revisit these failing tests.  If we still have problems
with the i686 tests, we can make them conditional then.

Thanks for pointing this out though, Marius.  I had thought about making
them conditional when reviewing, but I second guessed myself because we
have a lot of packages with comments like “tests fail on architecture X”
followed by an absolute “#:tests? #f”.  If this comes up in the future,
I’ll just go ahead and make the tests conditional.


-- Tim

Re: We need your feedback of the documentation videos!

[Ricardo Wurmus]

Hi Laura,

> 01-installation-from-script:
> - at 01:15 the URL is broken in an odd manner.  This can be fixed in one
> of these ways:
>   a) use a shorter existing URL:
>   https://git.sv.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
>   b) realize that the URL is still too long and create an alias at
>   https://guix.gnu.org/install.sh and use that.
> I will try using both links, if a) is still too long will make you know so
> that we create b).

We now have https://guix.gnu.org/install.sh.  You are free to use it.

> - at 01:35 the output has been altered.  We are not using stars in the
>   logo.  What is the reason for altering the output?
> There was a kind of encoding issue, the actual logo was not being shown
> with the script so tried to fix it like that :/

Can you tell me how to reproduce this?  Prehaps it’s a problem with our
scripts?

> - at 02:15 the way “# yes” is input would not work in real life because
>   “# yes” is not “yes”.  Is this a limitation of the video generation
>   scripts?
> Don't get this very well, the # is just to show that the user is root, but
> we can remove it if it is confusing.

It is confusing, because the user is not supposed to execute the “yes”
command but just to answer “yes” to the question — on the same line.

> - at the same mark there is a series of dots, which is not produced by
>   Guix.  Why have they been added?
> This is done in most videos, they are used to kind of show that something
> goes in between but it is not relevant to show it. Do you have any other
> idea for that?

I’d prefer either a simple ellipsis (“…”) or the actual console output.

--
Ricardo

Re: We need your feedback of the documentation videos!

[Laura Lazzati]

Hi!
I will start fixing what we can from the videos :) Will be answering video
by video to see what we can change and what we cannot unless we record
again the transcript :/

01-installation-from-script:
- at 01:15 the URL is broken in an odd manner.  This can be fixed in one
of these ways:
  a) use a shorter existing URL:
  https://git.sv.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
  b) realize that the URL is still too long and create an alias at
  https://guix.gnu.org/install.sh and use that.
I will try using both links, if a) is still too long will make you know so
that we create b).

- at 01:20 the GPG key is fetched from the SKS servers, which expose
  users to attacks.  This should be replaced with the new method to
  fetch the GPG key.
- at 01:30 Ludo’s name is mangled.  Looks like an encoding problem.
Will install it from scratch again to fix the first issue, and as regards
Ludo's name what I show was the output of fetching the key. Will go back to
this after the installation.

- at 01:35 the output has been altered.  We are not using stars in the
  logo.  What is the reason for altering the output?
There was a kind of encoding issue, the actual logo was not being shown
with the script so tried to fix it like that :/

- at 02:00 the output looks odd… is the script really creating
  “” and then again “”?  If this has
  been edited: why?
This is really a mistake :) To fix!

- at 02:15 the way “# yes” is input would not work in real life because
  “# yes” is not “yes”.  Is this a limitation of the video generation
  scripts?
Don't get this very well, the # is just to show that the user is root, but
we can remove it if it is confusing.

- at 02:50 the command should probably be “guix install hello” instead
  of “guix package -i hello”.
Yes, the video is outdated since it was created.

- at the same mark there is a series of dots, which is not produced by
  Guix.  Why have they been added?
This is done in most videos, they are used to kind of show that something
goes in between but it is not relevant to show it. Do you have any other
idea for that?
- at 02:55 the environment variable hint is outdated.  Guix now prints
  something shorter.
Again outdated.
- at 3:10 the URL is printed in italics, which makes it harder to read.
  We should probably use “https://guix.gnu.org/manual”.
This is something to fix in all videos.

Regards :)
Laura

Re: How to reference external program used in shell-scripts?

[Ricardo Wurmus]

Hi Hartmut,

> Assume some program, shell-script, whatever is calling an external program.
> What is the correct way to reference this? Shall it become an absolute
> path, or just the basename.

this depends on what the user may reasonably expect from the script.  If
it’s a core feature of the tool then I would expect to be able to use it
without having to manually install other packages.

If it’s a rare or optional feature it could be fine to use the basename
and assume that the user has the tool installed.

Generally, I try to make sure that required tools are either propagated
or that references to tools are made to absolute file names.

There is no hard rule.

-- 
Ricardo

Re: Please merge wip-haskell-updates (Re: [bug#36807] remove obsolete broken haskell packages)

[Robert Vollmert]

On 8. Aug 2019, at 15:12, Marius Bakke  wrote:
> I have one comment about the series: we've disabled tests on some
> packages that have been broken "forever" on i686.  It would be better to
> do so selectively on just the affected architectures.  I.e.:
> 
> #:tests? (if (string-prefix? "i686" (%current-system))
>  #f
>  #t)
> 
> Preferably with a comment about why they need to be disabled.  That way,
> we will still notice when something breaks on other architectures.  Can
> you try it Rob?

I don’t intend to, because I think the effort is better spent elsewhere.
But do make the change if you like!

The rough plan from my point of view would be to aim for an upgrade of
the haskell packages to build with ghc-8.6 from a recent stackage LTS
set, and reevaluate skipped tests across the set while doing that or
once that’s done.

Cheers
Robert

How to reference external program used in shell-scripts?

[Hartmut Goebel]

Hi,

when I started (nut never finished) to package dtx some time ago, I
already ask similar. No I'm packaging debops, and I'm questioning the
answer I got last time.

Assume some program, shell-script, whatever is calling an external program.
What is the correct way to reference this? Shall it become an absolute
path, or just the basename.

In the DEB/RPM world, the basename is used, and package "pkgA"'s
meta-data ensure that the required package "pkgB" is installed in an
sufficient version. When "pkgB" is updated, "pkgA" does not need to be
rebuild or updated, as long as "pkgA"'s requirements are still
satisfied. Any installed "pkgA" will now use the new version of "pkgB".

In Guix, I was told to do so, the absolute path is used. This will
ensure the required package "pkgB" is installed in exact the version
used when building "pkgA". When the "pkgB" is updated, "pkgA" need to be
rebuild and updated. too. If not updated, existing installations of
"pkgA" will still use the *old* version of "pkgB".

I understand that in Guix package dependencies are not based ion
meta-data, but are tracked based on references found in the files. I
also understand that in many cases it is desirable to know exactly which
version of pkgB was used when running pkgA.

Yet in many cases the exact version of pkgB does not matter, a minimum
version might be enough. And if the version of pkgB was new enough when
bringing the pkgA to Guix, the version does not even really matter. E.g.
if the external program is less, git, encfs, or which.

So: What is the official recommendations? Is this already been stated in
to manual?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |




signature.asc
Description: OpenPGP digital signature

Re: Please merge wip-haskell-updates (Re: [bug#36807] remove obsolete broken haskell packages)

[Marius Bakke]

Timothy Sample  writes:

> Hi Robert and Marius,
>
> Marius Bakke  writes:
>
>> Robert Vollmert  writes:
>>
>>> Oh, I meant to ask:
>>>
>>> On 6. Aug 2019, at 06:29, Timothy Sample  wrote:
>>> 
 I think it makes sense to wait for the core-updates merge (which
 shouldn’t be too far out).
>>>
>>> Why? Shouldn’t substitutes be available since CI built the branch 
>>> successfully?
>>
>> core-updates is still waiting for a full-rebuild change, and the CI is
>> currently unusable, so merging might take a month or more still.
>>
>> Getting this in 'master' meanwhile would be great, especially for i686
>> users :-)
>
> I stand corrected!  :)  I cherry-picked these commits from core-updates.
>
> Thanks for chiming in, Marius.

Thanks for merging, and huge thanks to Rob for untangling the Haskell
packages.

I have one comment about the series: we've disabled tests on some
packages that have been broken "forever" on i686.  It would be better to
do so selectively on just the affected architectures.  I.e.:

 #:tests? (if (string-prefix? "i686" (%current-system))
  #f
  #t)

Preferably with a comment about why they need to be disabled.  That way,
we will still notice when something breaks on other architectures.  Can
you try it Rob?

Thanks,
Marius


signature.asc
Description: PGP signature

Re: Support rsync to help Chinese users to setup mirrors

[Ricardo Wurmus]

Nala Ginrut  writes:

> How do you create this cache? Maybe we can just follow you in the same
> way. We're not trying to fetch packages like regular Guix clients, we'd
> like to provide a mirror/cache to help users to accelerate downloading
> inside the firewall covered area.

Guix clients fetch binaries in nar and narinfo format.  Mirroring them
might be sufficient.  To create the cache requires running “guix
publish” with a /gnu/store, which cannot easily be mirrored as it needs
to be in sync with its database.

-- 
Ricardo

Re: Support rsync to help Chinese users to setup mirrors

[Nala Ginrut]

Hi Ricardo!

Ricardo Wurmus writes:

> Extra information would always be useful.  I’m assuming that a direct
> copy of the store would not be helpful, because that’s not what clients
> request.  They fetch nars and narinfos instead, which we bake on demand
> and then cache.  I’m planning to expose that cache.

How do you create this cache? Maybe we can just follow you in the same
way. We're not trying to fetch packages like regular Guix clients, we'd
like to provide a mirror/cache to help users to accelerate downloading
inside the firewall covered area.

Best regards.


--
GNU Powered it
GPL Protected it
GOD Blessed it
HFG - NalaGinrut
Fingerprint F53B 4C56 95B5 E4D5 6093 4324 8469 6772 846A 0058


signature.asc
Description: PGP signature