Re: guix-based installation of pigx-rnaseq - works

[Steffen Möller]

Am 05.03.21 um 18:10 schrieb Steffen Möller:
> Am 05.03.21 um 01:10 schrieb zimoun:
>> Hi,
>>
>> On Wed, 03 Mar 2021 at 20:45, Steffen Möller  wrote:
>>
 What would be nice for Debian instead of
 ?
>>> The updates of Debian's packages and those of guix are not synchronized.
>>> And once something is in a regular Debian release, the Debian package
>>> likely points to an older version of the same in guix. It would hence be
>>> preferable to have an unversioned page to point to - which then lists
>>> all the versions available for that package.
>> Something like that:
>>
>> 1: 
>> 
>>
>> ?
>>
> https://data.guix.gnu.org/repository/1/branch/master/package/vim/
>
> Seems just fine.
>
 And where could I find a an example on how the others do?
>>> https://tracker.debian.org/pkg/vim
>> Something equivalent is the Guix Data Service, as mentioned in [1].
> I'll add some refs in d/u/metadata over the upcoming weeks to get this
> forward - if you feel like adding some yourself for the med packages,
> just ping me with your salsa account, please.

https://salsa.debian.org/med-team/pigx-rnaseq/-/blob/master/debian/upstream/metadata

Registry:
- Name: OMICtools
Entry: OMICS_33677
- Name: conda:bioconda
Entry: NA
- Name: guix
Entry: pigx-rnaseq

Andreas already kindly offered to do the downstream adjustments from here.

Cheers,
Steffen

Re: guix-based installation of pigx-rnaseq - works

[Steffen Möller]

Am 05.03.21 um 01:10 schrieb zimoun:
> Hi,
>
> On Wed, 03 Mar 2021 at 20:45, Steffen Möller  wrote:
>
>>> What would be nice for Debian instead of
>>> ?
>> The updates of Debian's packages and those of guix are not synchronized.
>> And once something is in a regular Debian release, the Debian package
>> likely points to an older version of the same in guix. It would hence be
>> preferable to have an unversioned page to point to - which then lists
>> all the versions available for that package.
> Something like that:
>
> 1: 
> 
>
> ?
>
https://data.guix.gnu.org/repository/1/branch/master/package/vim/

Seems just fine.

>>> And where could I find a an example on how the others do?
>> https://tracker.debian.org/pkg/vim
> Something equivalent is the Guix Data Service, as mentioned in [1].

I'll add some refs in d/u/metadata over the upcoming weeks to get this
forward - if you feel like adding some yourself for the med packages,
just ping me with your salsa account, please.

Best,

Steffen

Upstream permission needed to package machine learning preconditions (Was: guix-based installation of pigx-rnaseq - works)

[Andreas Tille]

Hi

On Fri, Mar 05, 2021 at 01:18:37AM +0100, zimoun wrote:
> 
> Because packages are really hard to package (I have in mind Machine
> Learning stuff) and they will not be included soon, IMHO.  Then I could
> remove my Guix on the top of Debian and simply have Debian inside
> Guix. ;-)

Apropos hard to package Machine Learning stuff.  I recently intended to
package cython-blis[1] since it is needed by recent versions of
python-thinc.  It turns out that my packaging attemt[2] is not
specifically hard from a technical point of view (at least the package
builds and passes its test - it needs to be verified via benchmarking
whether it is as performant as possible) there is some non-technical
issue I would like you to read:

   https://github.com/explosion/cython-blis/issues/32 

I have collected opinions of other Debian Developers who have touched
that package before inside my ITP bug[1].  My intention is to open a new
discussion with upstream and it would be really great if you would join
me in this from a different perspective than "just another Debian
developer".  So my question is:  Would you join me in this discussion to
get "permission" to package cython-blis (which is strictly speaking not
needed license wise but my per Debian policy I should not package
anything if upstream does not want me to do so).

Kind regards

 Andreas.

[1] http://bugs.debian.org/984497
[2] https://salsa.debian.org/science-team/python-cython-blis

-- 
http://fam-tille.de

Re: guix-based installation of pigx-rnaseq - works

[Steffen Möller]

Am 05.03.21 um 01:18 schrieb zimoun:
> Hi Pjotr,
>
> On Wed, 03 Mar 2021 at 22:16, Pjotr Prins  wrote:
>
>> I love Debian, but Guix packages and containers are the cats whiskers!
>> It is brilliant to have them both. I have no need for conda.
> Now this works:
>
>   # apt install guix
>   $ guix install hello
>
> and since Vagrant sent the announce on Guix Devel, I have on my TODO to
> try the contrary: install Debian packages on Guix System,
>
>   # guix install dpkg
>   # dpkg 
This is darn cool.
> Because packages are really hard to package (I have in mind Machine
> Learning stuff) and they will not be included soon, IMHO.

https://tracker.debian.org/pkg/pytorch

But all the CI with GPUs are yet not clear if I got this right.

> Then I could
> remove my Guix on the top of Debian and simply have Debian inside
> Guix. ;-)
Could we make this somehow visible on popcon.debian.org?

Best,

Steffen

Re: guix-based installation of pigx-rnaseq - works

[zimoun]

Hi Pjotr,

On Wed, 03 Mar 2021 at 22:16, Pjotr Prins  wrote:

> I love Debian, but Guix packages and containers are the cats whiskers!
> It is brilliant to have them both. I have no need for conda.

Now this works:

  # apt install guix
  $ guix install hello

and since Vagrant sent the announce on Guix Devel, I have on my TODO to
try the contrary: install Debian packages on Guix System,

  # guix install dpkg
  # dpkg 

Because packages are really hard to package (I have in mind Machine
Learning stuff) and they will not be included soon, IMHO.  Then I could
remove my Guix on the top of Debian and simply have Debian inside
Guix. ;-)


Cheers,
simon

Re: guix-based installation of pigx-rnaseq - works

[zimoun]

Hi,

On Wed, 03 Mar 2021 at 20:45, Steffen Möller  wrote:

>> What would be nice for Debian instead of
>> ?
> The updates of Debian's packages and those of guix are not synchronized.
> And once something is in a regular Debian release, the Debian package
> likely points to an older version of the same in guix. It would hence be
> preferable to have an unversioned page to point to - which then lists
> all the versions available for that package.

Something like that:

1: 


?


>> And where could I find a an example on how the others do?
>
> https://tracker.debian.org/pkg/vim

Something equivalent is the Guix Data Service, as mentioned in [1].



Cheers,
simon

Re: guix-based installation of pigx-rnaseq - works

[Pjotr Prins]

On Tue, Feb 23, 2021 at 09:28:48PM +0100, Steffen Möller wrote:
> Hello,
> 
> sudo apt-get install guix
> guix install pigx-rnaseq

:)

> indeed installs the pigx-rnaseq library with its R dependencies. It
> installs it all in /gnu, which is somewhat inconvenient, as in "my root
> partition complained", and it takes painstakingly long, but it works. 
> 
> The system finds STAR and other executables in the regular system path,
> only pigx-rnaseq was installed:
> 
> $ guix package -l
> Generation 1    Feb 23 2021 18:27:42    (current)
>   pigx-rnaseq    0.0.10    out   
> /gnu/store/nlknrjmm2knbr8i5m5qj94788arfb14n-pigx-rnaseq-0.0.10
> 
> The full install "pigx" I have not tried, yet. Will do.
> 
> We keep mentioning conda also the time, while guix is somewhat left
> aside, but https://guix.gnu.org/packages/ is truly impressive. Somewhat
> annoying, if we ever decide to also reference guix packages in
> d/u/metadata, then there are only versioned web pages like
> https://guix.gnu.org/de/packages/bc-1.07.1/, so we would only have
> moving targets point to.
> 
> But otherwise, Debian's guix package has formidably done its job. It is
> about time we get the conda packages into a similar state, just maybe
> the / partition should be spared by default, but then again, it was only
> 6.5GB.
> 
> I had tried in vain to create a dockerfile with this setup which gets a
> permission error in the moment that the installation of the packages starts:
> 
> FROM debian:unstable
> ENV TERM=xterm
> RUN apt-get update -qq
> RUN apt-get install -y guix
> #RUN (/usr/bin/guix-daemon --build-users-group=_guixbuild & ) && guix
> install pigx-rnaseq
> #RUN (/usr/bin/guix-daemon & ) && guix install pigx-rnaseq # too big to fail
> RUN (/usr/bin/guix-daemon & ) && guix install vim # still takes long and
> fails

The guix-daemon needs full privileges by default, but it does not need
to run inside a container. Just build a docker or guix container and
you'll get a minimal dependency tree inside the container. I wrote
some stuff up here:

  https://github.com/pjotrp/guix-notes/blob/master/CONTAINERS.org

mostly because I don't trust my memory ;)

> Should this trigger any idea among those reading this - some RTFM plus a
> small pointer would be much appreciated.

Note that Guix is a rolling distribution. It can do that because
versions do not interfere with each other. You can have an unlimited
number of glibcs, pythons etc. 

The reason it installs software in /gnu/store is because of
reproducibility. All paths are hard coded in binaries and libs. There
is no search path to look up libs, for example. Install a package once
and you'll get identical paths between machines, containers etc. 

I love Debian, but Guix packages and containers are the cats whiskers!
It is brilliant to have them both. I have no need for conda.

Pj.

Re: guix-based installation of pigx-rnaseq - works

[Steffen Möller]

Hi Simon,

Am 02.03.21 um 14:52 schrieb zimoun:
> Hi,
>
> On Sat, 27 Feb 2021 at 19:55, Steffen Möller  wrote:
>
 We keep mentioning conda also the time, while guix is somewhat left
 aside, but https://guix.gnu.org/packages/ is truly impressive. Somewhat
 annoying, if we ever decide to also reference guix packages in
 d/u/metadata, then there are only versioned web pages like
 https://guix.gnu.org/de/packages/bc-1.07.1/, so we would only have
 moving targets point to.
>>> What do you mean exactly?
>> In d/u/metadata we give references to the bio.tools registry,
>> SciCrunch's RRIDs and conda. But we should also give one to guix if we
>> know about an "ortholog" package. These references point up as URLs on
>> our task pages for the various blends. There is however no page for a
>> package in guix that is not versioned if I get this right.
> Just to understand, since I am not totally familiar with the debian/
> folder, you are speaking about this file, for instance,
>
> https://salsa.debian.org/med-team/pigx-rnaseq/-/blob/master/debian/upstream/metadata
yes
> What would be nice for Debian instead of
> ?
The updates of Debian's packages and those of guix are not synchronized.
And once something is in a regular Debian release, the Debian package
likely points to an older version of the same in guix. It would hence be
preferable to have an unversioned page to point to - which then lists
all the versions available for that package.
> And where could I find a an example on how the others do?

https://tracker.debian.org/pkg/vim

https://anaconda.org/conda-forge/vim

>> The fun part is, and this is where this shot on guix comes in, that we
>> can compare a Debian based implementation with an image that upstream
>> provides. Or compare with conda/brew/...guix. We are currently team
>> building and collect ideas what we want to achieve.
> I am not sure how the image upstream provides is built, 
> probably with
> "guix pack -f docker":
>
> 

Ah, there is is. Many thanks. I knew I had seen this somewhere :)

>> Hm. Not so sure. My interpretation was some general incompatibility of
>> above docker-approach with GUIX, so I happily adopt the "guix pack".
> There is another level to build Docker images using Guix: "guix system
> docker-image" but it is more complicated and probably not the correct
> path to distribute packages in a container.
>
> 

Hm. Would be interesting to learn about differences between different
ways to come up with an image.

Best,

Steffen

Re: guix-based installation of pigx-rnaseq - works

[zimoun]

Hi,

On Sat, 27 Feb 2021 at 19:55, Steffen Möller  wrote:

> >> We keep mentioning conda also the time, while guix is somewhat left
> >> aside, but https://guix.gnu.org/packages/ is truly impressive. Somewhat
> >> annoying, if we ever decide to also reference guix packages in
> >> d/u/metadata, then there are only versioned web pages like
> >> https://guix.gnu.org/de/packages/bc-1.07.1/, so we would only have
> >> moving targets point to.
> >
> > What do you mean exactly?
>
> In d/u/metadata we give references to the bio.tools registry,
> SciCrunch's RRIDs and conda. But we should also give one to guix if we
> know about an "ortholog" package. These references point up as URLs on
> our task pages for the various blends. There is however no page for a
> package in guix that is not versioned if I get this right.

Just to understand, since I am not totally familiar with the debian/
folder, you are speaking about this file, for instance,

https://salsa.debian.org/med-team/pigx-rnaseq/-/blob/master/debian/upstream/metadata

What would be nice for Debian instead of
?
And where could I find a an example on how the others do?


> The fun part is, and this is where this shot on guix comes in, that we
> can compare a Debian based implementation with an image that upstream
> provides. Or compare with conda/brew/...guix. We are currently team
> building and collect ideas what we want to achieve.

I am not sure how the image upstream provides is built, probably with
"guix pack -f docker":



> Hm. Not so sure. My interpretation was some general incompatibility of
> above docker-approach with GUIX, so I happily adopt the "guix pack".

There is another level to build Docker images using Guix: "guix system
docker-image" but it is more complicated and probably not the correct
path to distribute packages in a container.




All the best,
simon

Re: guix-based installation of pigx-rnaseq - works

[Steffen Möller]

Am 26.02.21 um 22:25 schrieb zimoun:
> Hi,
>
> On Tue, 23 Feb 2021 at 21:28, Steffen Möller  
> wrote:
>
>> sudo apt-get install guix
>> guix install pigx-rnaseq
> Thanks Vagrant! :-)
> And if I may, I would recommend this Debconf track:
> 
Watched it! [hence the delayed reply]
>
>
>> We keep mentioning conda also the time, while guix is somewhat left
>> aside, but https://guix.gnu.org/packages/ is truly impressive. Somewhat
>> annoying, if we ever decide to also reference guix packages in
>> d/u/metadata, then there are only versioned web pages like
>> https://guix.gnu.org/de/packages/bc-1.07.1/, so we would only have
>> moving targets point to.
> What do you mean exactly?
In d/u/metadata we give references to the bio.tools registry,
SciCrunch's RRIDs and conda. But we should also give one to guix if we
know about an "ortholog" package. These references point up as URLs on
our task pages for the various blends. There is however no page for a
package in guix that is not versioned if I get this right.
>> I had tried in vain to create a dockerfile with this setup which gets a
>> permission error in the moment that the installation of the packages
>> starts:
> To create a Docker image, instead of going through Dockerfile, you could
> use:
>
>   guix pack -f docker pigx-rnaseq
>
> (plus maybe some other options, for reference:
> )
Will try that. Many thanks!
> Well, from my point of view, the issue with this snippet
>
>> FROM debian:unstable
>> ENV TERM=xterm
>> RUN apt-get update -qq
>> RUN apt-get install -y guix
>> #RUN (/usr/bin/guix-daemon --build-users-group=_guixbuild & ) && guix
>> install pigx-rnaseq
>> #RUN (/usr/bin/guix-daemon & ) && guix install pigx-rnaseq # too big to fail
> is that it is not reproducible, first because of ’apt-get update’ then
> second because the Guix version is not tracked.  Other said, it will be
> hard nor impossible to rebuild the exact same image.

This was only a pilot that mimics the install instructions for
pigx-rnaseq provided by upstream. Also, a controlled variation is
intentional as we will be using it all together with snaptshot.d.o. The
idea is to have the RNA-seq workflow as a template for other  (also
non-biological) workflows to come and then see how results change while
we travel through time. So we want to get away from "does it build and
autotest" to a quantitative comparison - and maybe find other
benchmarks. We'll see.

The fun part is, and this is where this shot on guix comes in, that we
can compare a Debian based implementation with an image that upstream
provides. Or compare with conda/brew/...guix. We are currently team
building and collect ideas what we want to achieve.

> Instead,
>
>  guix time-machine --commit=12345 -- pack -f docker pix-rnaseq 
> --save-provenance
>
> seems better, IMHO.  However, going via debuerreotype would be another
> option. :-)
>
>
> About the ’vim’ fails, see .
> The Guix CI does not have a concept of “stable” or “testing” and time to
> time a Sid commit breaks master. :-)

Hm. Not so sure. My interpretation was some general incompatibility of
above docker-approach with GUIX, so I happily adopt the "guix pack".

Many thanks!

Steffen

Re: guix-based installation of pigx-rnaseq - works

[zimoun]

Hi,

On Tue, 23 Feb 2021 at 21:28, Steffen Möller  
wrote:

> sudo apt-get install guix
> guix install pigx-rnaseq

Thanks Vagrant! :-)
And if I may, I would recommend this Debconf track:



> We keep mentioning conda also the time, while guix is somewhat left
> aside, but https://guix.gnu.org/packages/ is truly impressive. Somewhat
> annoying, if we ever decide to also reference guix packages in
> d/u/metadata, then there are only versioned web pages like
> https://guix.gnu.org/de/packages/bc-1.07.1/, so we would only have
> moving targets point to.

What do you mean exactly?


> I had tried in vain to create a dockerfile with this setup which gets a
> permission error in the moment that the installation of the packages
> starts:

To create a Docker image, instead of going through Dockerfile, you could
use:

  guix pack -f docker pigx-rnaseq

(plus maybe some other options, for reference:
)


Well, from my point of view, the issue with this snippet

> FROM debian:unstable
> ENV TERM=xterm
> RUN apt-get update -qq
> RUN apt-get install -y guix
> #RUN (/usr/bin/guix-daemon --build-users-group=_guixbuild & ) && guix
> install pigx-rnaseq
> #RUN (/usr/bin/guix-daemon & ) && guix install pigx-rnaseq # too big to fail

is that it is not reproducible, first because of ’apt-get update’ then
second because the Guix version is not tracked.  Other said, it will be
hard nor impossible to rebuild the exact same image.  Instead,

 guix time-machine --commit=12345 -- pack -f docker pix-rnaseq --save-provenance

seems better, IMHO.  However, going via debuerreotype would be another
option. :-)


About the ’vim’ fails, see .
The Guix CI does not have a concept of “stable” or “testing” and time to
time a Sid commit breaks master. :-)



All the best,
simon

guix-based installation of pigx-rnaseq - works

[Steffen Möller]

Hello,

sudo apt-get install guix
guix install pigx-rnaseq

indeed installs the pigx-rnaseq library with its R dependencies. It
installs it all in /gnu, which is somewhat inconvenient, as in "my root
partition complained", and it takes painstakingly long, but it works. 

The system finds STAR and other executables in the regular system path,
only pigx-rnaseq was installed:

$ guix package -l
Generation 1    Feb 23 2021 18:27:42    (current)
  pigx-rnaseq    0.0.10    out   
/gnu/store/nlknrjmm2knbr8i5m5qj94788arfb14n-pigx-rnaseq-0.0.10

The full install "pigx" I have not tried, yet. Will do.

We keep mentioning conda also the time, while guix is somewhat left
aside, but https://guix.gnu.org/packages/ is truly impressive. Somewhat
annoying, if we ever decide to also reference guix packages in
d/u/metadata, then there are only versioned web pages like
https://guix.gnu.org/de/packages/bc-1.07.1/, so we would only have
moving targets point to.

But otherwise, Debian's guix package has formidably done its job. It is
about time we get the conda packages into a similar state, just maybe
the / partition should be spared by default, but then again, it was only
6.5GB.

I had tried in vain to create a dockerfile with this setup which gets a
permission error in the moment that the installation of the packages starts:

FROM debian:unstable
ENV TERM=xterm
RUN apt-get update -qq
RUN apt-get install -y guix
#RUN (/usr/bin/guix-daemon --build-users-group=_guixbuild & ) && guix
install pigx-rnaseq
#RUN (/usr/bin/guix-daemon & ) && guix install pigx-rnaseq # too big to fail
RUN (/usr/bin/guix-daemon & ) && guix install vim # still takes long and
fails

Should this trigger any idea among those reading this - some RTFM plus a
small pointer would be much appreciated.

Best,

Steffen