Re: "Parameterized Packages: The Project Completion Update"

[Munyoki Kilyungi]

Lost in communication I guess ;)

Andy Tai  aliandika:

> no, not  my effort.  I just pointed out that report.  credit to the
> real author Sarthak Shah
>

I've actually met Sarthak on Matrix where he was
talking about his work to our group.  I used to
stick around to hear what he had up his plate.
So ...

> On Mon, Oct 2, 2023 at 1:13 PM Munyoki Kilyungi  
> wrote:
>>
>> Andy Tai  aliandika:
>>
>> > Not seen mentioned on this mailing list  yet, so probably of interest:
>> > GSOC project's final report:
>> > https://blog.lispy.tech/parameterized-packages-the-project-completion-update.html
>> >
>>
>> Thanks for the effort!

... thanks for the effort (from him) and you
posting this ;)

-- 
(Life is like a pencil that will surely run out,
but will leave the beautiful writing of life.)
(D4F09EB110177E03C28E2FE1F5BBAE1E0392253F
(hkp://keys.openpgp.org))


signature.asc
Description: PGP signature

Re: "Parameterized Packages: The Project Completion Update"

[Andy Tai]

no, not  my effort.  I just pointed out that report.  credit to the
real author Sarthak Shah

On Mon, Oct 2, 2023 at 1:13 PM Munyoki Kilyungi  wrote:
>
> Andy Tai  aliandika:
>
> > Not seen mentioned on this mailing list  yet, so probably of interest:
> > GSOC project's final report:
> > https://blog.lispy.tech/parameterized-packages-the-project-completion-update.html
> >
>
> Thanks for the effort!

Re: "Parameterized Packages: The Project Completion Update"

[Munyoki Kilyungi]

Andy Tai  aliandika:

> Not seen mentioned on this mailing list  yet, so probably of interest:
> GSOC project's final report:
> https://blog.lispy.tech/parameterized-packages-the-project-completion-update.html
>

Thanks for the effort!

-- 
(Life is like a pencil that will surely run out,
but will leave the beautiful writing of life.)
(D4F09EB110177E03C28E2FE1F5BBAE1E0392253F
(hkp://keys.openpgp.org))


signature.asc
Description: PGP signature

Re: Parameterized packages

[zimoun]

On Mon, 27 Jan 2020 at 12:51, Pierre Neidhardt  wrote:
>
> zimoun  writes:
>
> > Maybe I misread. :-)
> > I am still convinced that it is not the correct design and instead be
> > able to partially override the package definition seems more
> > appropriate.
>
> If you have another design in mind that supports parameter composition, please
> do share :)

Héhé, you are kidding, right? :-)

Maybe improving how to override a package... :-D
Maybe something similar to Nix... which claims composition. ;-)

https://nixos.org/nixos/nix-pills/override-design-pattern.html
https://nixos.org/nixos/nix-pills/nixpkgs-overriding-packages.html

Anyway.
I should misunderstand something and should miss a key point.
You could explain me IRL at Guix Days in the front of the beer you owe me. ;-)


Cheers,
simon

Re: Parameterized packages

[zimoun]

Hi Pierre,

On Mon, 27 Jan 2020 at 11:13, Pierre Neidhardt  wrote:
>
> zimoun  writes:
>
> > Maybe I misread something and/or I misunderstand other thing but the
> > symbols 'video-player' and 'python' need to be defined somewhere. And
> > with this proposal adding the field 'parameters', this somewhere is
> > outside the package. Therefore, I will end up to a big mess, IMHO.
>
> I think you misread indeed :p

Maybe I misread. :-)
I am still convinced that it is not the correct design and instead be
able to partially override the package definition seems more
appropriate.

At the end, code is law. :-)


Cheers,
simon

Re: Parameterized packages

[Pierre Neidhardt]

Hi John,

I believe the complexity will remain under control.

"Flags" are not replacing Scheme: they are Scheme.  Just like we have
support for multiple outputs now.

For newcomers, package parameters are not an issue since they are
completely optional.  You won't see them until you want them!

Regarding the command line: maybe we don't have to update it after all.
We could stick to manifests and scripts to leverage package parameters.

Thoughts on this, everyone?

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[zimoun]

Hi,

On Fri, 24 Jan 2020 at 23:06, ison  wrote:

> My understanding of the global definitions they're talking about is that
> they would just be meta objects, not global preferences.
> For example (maybe it won't look like this, but just a guess):
> Instead of passing the arguments "mpv" and "3.7" into your function you'd
> give the parameters:
> '((video-player . "mpv")
>   (python-version . "3.7"))
> to the package.
> video-player and python-version would be globally defined meta objects
> that just give descriptions about what these parameters should be used for
> and such (so users can read more information about package parameters when
> looking up packages). But the actual values that get used by the package
> are not global, they're passed into the package just like your function
> example.

Maybe I misread something and/or I misunderstand other thing but the
symbols 'video-player' and 'python' need to be defined somewhere. And
with this proposal adding the field 'parameters', this somewhere is
outside the package. Therefore, I will end up to a big mess, IMHO.

Let consider the packages 'foo' and 'bar'. They depend both on say the
'video-player' package.
Well, the definition looks like:

--8<---cut here---start->8---
(package
 (name "foo")
 (...)
 (parameters video-player)
 (...))

(package
 (name "bar")
 (...)
 (parameters video-player)
 (...))
--8<---cut here---end--->8---

Now, the user wants to install the both packages but:
 -  they requires that "foo" depends on vlc without any X support, say
"vlc-no-x"
 -  and they require that "bar" depends on full "vlc"

So how to do?
Where is defined 'video-player'?
How to install the both packages with the same transaction?


> I'm not sure with the function method how easy it would be to inform the
> user of all available options. For example when running `guix search`
> would it be able to programmatically collect all the options that a
> package has and display them there?

To me, the "official" packages in master should stay "simple" with no
parameter. Otherwise, it is will hard -- if not impossible -- to
ensure that all combinations work; or the number of parameters will be
very limited.

However, Guix should provide more tune-able rewriting package
functions. And these modified packages should live in channels;
possibly official ones to be built by Berlin (substitutes).


> Also it seems that whether or not there are global meta objects is a
> separate issue than whether to use parameters or functions. It looks like
> either method could use them or not. In the function approach you could
> still just pass in an alist as above.

Yes.
But the big differences are:

 1. the base package is untouched
 2. the arguments are locally defined
 3. everything is explicit

So less surprises IMHO.


Well, I really do not know and I do not have a strong opinion: I just
fear that the 'parameters' field would add complexity which would lead
to unmaintenable packages.


All the best,
simon

Re: Parameterized packages

[John Soo]

Hey all!

I’ve been following very roughly. I have a couple issues with parameterized 
packages. 

> On Jan 22, 2020, at 4:24 AM, zimoun  wrote:
> 
> Well, I am wanting the same thing: be able to modify the 'arguments'
> field but I am not convinced by the design you are proposing because I
> have the feeling that with such design then mess is coming. :-)

The maintenance and complexity does worry me.

Introducing more complexity may also make it harder to hack on guix itself. I 
love the hack ability of guix. (I correct me if I’m wrong, but isn’t hack 
ability a core design goal of the project?)

The other worry is that command line flags are close to another way of coding 
package modifications in guile. Maybe it can be more scriptable but I do want 
the guile code itself to continue to be the nice and fun place it is to work 
on. I’d rather flags not be a replacement for scheme.

So to sum up: not only could command line parameters potentially make it harder 
to work on guix code it might disincentivize actually doing any hacking.  I 
think there could be very nice ways of implementing flags that improve guix 
overall. I jut want to keep in mind the design goals of the project. 

-  John

Re: Parameterized packages

[ison]

On Wed, Jan 22, 2020 at 01:23:26PM +0100, zimoun wrote:
> > > --8<---cut here---start->8---
> > > (define (make-you-get VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
> > >   (package
> > >  (inherit you-get
> > > #:add-inputs
> > >  `(("PLAYER" ,VIDEO-PLAYER))
> > >,@(IF WITH-FFMPEG)
> > >  ;; FOR MULTI-PART AND >=1080P VIDEOS
> > >  `("FFMPEG" ,FFMPEG)
> > > #:replace-arguments ...
> > >#:add-phase ...
> > >  '(
> > >
> > > (define-public you-get-vlc (make-you-get 'vlc))
> > > --8<---cut here---end--->8---
> > >
> > >
> > > Something like that. And everything is more controlled,
> >
> > What you propose here is essentially the same as what I propose, the
> > difference is that you wrapped it around `make-you-get` instead of
> > declaring the parameters inside as a field.
> 
> The big differences are:
> 
>  1. the base package is untouched
>  2. the arguments are locally defined
>  3. everything is explicit
> 
> So less surprises IMHO.

My understanding of the global definitions they're talking about is that
they would just be meta objects, not global preferences.
For example (maybe it won't look like this, but just a guess):
Instead of passing the arguments "mpv" and "3.7" into your function you'd
give the parameters:
'((video-player . "mpv")
  (python-version . "3.7"))
to the package.
video-player and python-version would be globally defined meta objects
that just give descriptions about what these parameters should be used for
and such (so users can read more information about package parameters when
looking up packages). But the actual values that get used by the package
are not global, they're passed into the package just like your function
example.

I hope I'm not missing something here, but this is how I was reading the
conversation.

I'm not sure with the function method how easy it would be to inform the
user of all available options. For example when running `guix search`
would it be able to programmatically collect all the options that a
package has and display them there?
It almost seems like achieving that would require a parameters field in
the package anyway.

Also it seems that whether or not there are global meta objects is a
separate issue than whether to use parameters or functions. It looks like
either method could use them or not. In the function approach you could
still just pass in an alist as above.

Re: Parameterized packages

[zimoun]

Hi Pierre,

On Wed, 22 Jan 2020 at 10:54, Pierre Neidhardt  wrote:
> zimoun  writes:

> > To me, a package is:
> > "./configure && make && make check && make install"
> > so I understand why tweak the flags used by "./configure", for example
> > change "--with-vlc=" from the default 0 to the tuned 1. Or use another
> > compiling toolchain.
> > I understand also these flags could require different inputs.
>
> If we reuse your example, a package may very well have --with-vlc and
> --with-mpv.
>
> Those flags conflict, so the package definition must know about the
> corresponding "package parameters" at the same time to raise the
> appropriate error (or choose The Right Option automatically).

It is the job to './configure' to detect conflicting parameters and
not the job of the package definition, IMHO.
The well-known: crap in, crap out! :-)


> Instead, if you'd inherit you'd always overwrite the changes without the
> user noticing.  This removes a lot of control.

Yes, I agree with the current implementation. That's why I wrote: ;-)
<<
And again from my understanding, this is more or less cover by
'inherit'. Or some macrology should be improved in 'package/inherit'
or in 'records.scm', i.e., something more fine grained when
inheriting.
>>



> > --8<---cut here---start->8---
> > (define (make-you-get VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
> >   (package
> >  (inherit you-get
> > #:add-inputs
> >  `(("PLAYER" ,VIDEO-PLAYER))
> >,@(IF WITH-FFMPEG)
> >  ;; FOR MULTI-PART AND >=1080P VIDEOS
> >  `("FFMPEG" ,FFMPEG)
> > #:replace-arguments ...
> >#:add-phase ...
> >  '(
> >
> > (define-public you-get-vlc (make-you-get 'vlc))
> > --8<---cut here---end--->8---
> >
> >
> > Something like that. And everything is more controlled,
>
> What you propose here is essentially the same as what I propose, the
> difference is that you wrapped it around `make-you-get` instead of
> declaring the parameters inside as a field.

The big differences are:

 1. the base package is untouched
 2. the arguments are locally defined
 3. everything is explicit

So less surprises IMHO.


> > i.e., no mess with global parameters.
>
> I think there is a misunderstanding, there is no "mess with global
> parameters".

Where should be defined what you called VIDEO-PLAYER?
the one in ' (PARAMETERS VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)'.

Because the symbol needs to be defined somewhere, right?


> Tobias suggestion was to declare the parameters (name, doc and type)
> globally, just like we declare record globally.

It will end with a big mess because there is almost as many
"parameters" as packages.

And 'parameter' means somehow 'symbol', isn't it?


> > What you want to do is: add/remove/replace inputs/arguments so
> > #:add-inputs, #:remove-native-inputs, etc.
>
> Not just that, that's the point: we want the possibility to modify
> anything, in particular what's inside #:arguments.

I am suggesting the same thing. :-)

But instead I am suggesting to pass the modifications 'arguments' via
inheritance mechanism.


> > Hum? but months later your system is broken... so I am not convinced
> > it is powerful. :-)
> >
> > It is broken because composing complex system is an hard task. Typing
> > helps a bit to detect error at compile time (in this case at building
> > package time) but run-time errors will still remain and it will be
> > hard time to find them.
>
> Why would it be broken after months?  Packagers do the work once and it

I do not know but it is what Gentoo users are saying.
Well, I know a awesome guy wearing psychadelic T-shirt and former
Gentoo user that explained me that IRL. ;-)


> works.  Of course the package parameters will have to be tested on
> update of the package definition.  From the user's perspective, the

This adds a lot of workload to packagers, IMHO.

> worst case scenario is that a formerly support parameter goes
> unsupported for a given package, in which case we show a warning.
>
> What do you think?

Well, I am wanting the same thing: be able to modify the 'arguments'
field but I am not convinced by the design you are proposing because I
have the feeling that with such design then mess is coming. :-)


Cheers,
simon

Re: Parameterized packages

[Pierre Neidhardt]

zimoun  writes:

> I do not understand what you do mean by "does not compose".
>
>
> To me, a package is:
> "./configure && make && make check && make install"
> so I understand why tweak the flags used by "./configure", for example
> change "--with-vlc=" from the default 0 to the tuned 1. Or use another
> compiling toolchain.
> I understand also these flags could require different inputs.

If we reuse your example, a package may very well have --with-vlc and
--with-mpv.

Those flags conflict, so the package definition must know about the
corresponding "package parameters" at the same time to raise the
appropriate error (or choose The Right Option automatically).

Instead, if you'd inherit you'd always overwrite the changes without the
user noticing.  This removes a lot of control.

> --8<---cut here---start->8---
> (define (make-you-get VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
>   (package
>  (inherit you-get
> #:add-inputs
>  `(("PLAYER" ,VIDEO-PLAYER))
>,@(IF WITH-FFMPEG)
>  ;; FOR MULTI-PART AND >=1080P VIDEOS
>  `("FFMPEG" ,FFMPEG)
> #:replace-arguments ...
>#:add-phase ...
>  '(
>
> (define-public you-get-vlc (make-you-get 'vlc))
> --8<---cut here---end--->8---
>
>
> Something like that. And everything is more controlled,

What you propose here is essentially the same as what I propose, the
difference is that you wrapped it around `make-you-get` instead of
declaring the parameters inside as a field.

> i.e., no mess with global parameters.

I think there is a misunderstanding, there is no "mess with global
parameters".

Tobias suggestion was to declare the parameters (name, doc and type)
globally, just like we declare record globally.

> What you want to do is: add/remove/replace inputs/arguments so
> #:add-inputs, #:remove-native-inputs, etc.

Not just that, that's the point: we want the possibility to modify
anything, in particular what's inside #:arguments.

> Hum? but months later your system is broken... so I am not convinced
> it is powerful. :-)
>
> It is broken because composing complex system is an hard task. Typing
> helps a bit to detect error at compile time (in this case at building
> package time) but run-time errors will still remain and it will be
> hard time to find them.

Why would it be broken after months?  Packagers do the work once and it
works.  Of course the package parameters will have to be tested on
update of the package definition.  From the user's perspective, the
worst case scenario is that a formerly support parameter goes
unsupported for a given package, in which case we show a warning.

What do you think?

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[zimoun]

On Tue, 21 Jan 2020 at 14:13, Pierre Neidhardt  wrote:
>
> zimoun  writes:
>
> >> > --8<---cut here---start->8---
> >> > (define (make-me-a-package option1 option2)
> >> > (package
> >> >   …))
> >> > --8<---cut here---end--->8---
> >>
> >> The ellipsis is a bit vague here.  What is this trying to do?
> >
> > What you wrote below. :-)
>
> ?

--8<---cut here---start->8---
(define (make-me-a-package VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
   (package
[all your "messy" code using stuff]
  ))

(define-public you-get-vlc
(make-me-a-package 'vlc 'python #t))
--8<---cut here---start->8---



> > And my opinion is that you described is already possible (more or
> > less) using 'inherit'. Except the modification of the compiling
> > options (build-system).
>
> The crucial difference is the current approach with inherit /does not
> compose/.  This is where USE flags stand strong because you can tell
> what happens when both flags X and Y are set, e.g.

I do not understand what you do mean by "does not compose".


To me, a package is:
"./configure && make && make check && make install"
so I understand why tweak the flags used by "./configure", for example
change "--with-vlc=" from the default 0 to the tuned 1. Or use another
compiling toolchain.
I understand also these flags could require different inputs.

And again from my understanding, this is more or less cover by
'inherit'. Or some macrology should be improved in 'package/inherit'
or in 'records.scm', i.e., something more fine grained when
inheriting.


--8<---cut here---start->8---
(define (make-you-get VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
  (package
 (inherit you-get
#:add-inputs
 `(("PLAYER" ,VIDEO-PLAYER))
   ,@(IF WITH-FFMPEG)
 ;; FOR MULTI-PART AND >=1080P VIDEOS
 `("FFMPEG" ,FFMPEG)
#:replace-arguments ...
   #:add-phase ...
 '(

(define-public you-get-vlc (make-you-get 'vlc))
--8<---cut here---end--->8---


Something like that. And everything is more controlled, i.e., no mess
with global parameters.

What you want to do is: add/remove/replace inputs/arguments so
#:add-inputs, #:remove-native-inputs, etc.

And this composes in the same way function composes. And yes, it
composes explicitly which can be annoying, But implicit composition
will end up to the Gentoo USE flag mess, IMHO.



> >> - To let the user choose which video player to use.  This is a popular
> >>   USE flag on Gentoo (maybe with a different name).
> >
> > What I do not understand is: people who used Gentoo and especially USE
> > flag are saying that it ends with a big mess with broken packages.
> > Therefore, why does Guix want to reproduce the mess?
>
> Because it is extremely powerful: in a simple declaration like "no-X +
> pulseaudio + python3 + gtk2" you can build /your entire system/,
> according to the flags you've set, in just one command.

Hum? but months later your system is broken... so I am not convinced
it is powerful. :-)

It is broken because composing complex system is an hard task. Typing
helps a bit to detect error at compile time (in this case at building
package time) but run-time errors will still remain and it will be
hard time to find them.

Cheers,
simon

Re: Parameterized packages

[Pierre Neidhardt]

zimoun  writes:

>> > --8<---cut here---start->8---
>> > (define (make-me-a-package option1 option2)
>> > (package
>> >   …))
>> > --8<---cut here---end--->8---
>>
>> The ellipsis is a bit vague here.  What is this trying to do?
>
> What you wrote below. :-)

?

> And my opinion is that you described is already possible (more or
> less) using 'inherit'. Except the modification of the compiling
> options (build-system).

The crucial difference is the current approach with inherit /does not
compose/.  This is where USE flags stand strong because you can tell
what happens when both flags X and Y are set, e.g.

--8<---cut here---start->8---
  (if (and X Y)
 ; Do something when we have both X and Y activated.
 ; Else maybe just X, or just Y, or none.
 )
--8<---cut here---end--->8---

>> The point of declaring the parameters in advance is that it allows the
>> user to list all parameters used by a given package.
>
> I bet that the number of broken packages will increase.

Probably.  But if we keep the same standard for the default values of
the parameters, we would keep the same level of quality.  It should be OK.

>> - To let the user choose which video player to use.  This is a popular
>>   USE flag on Gentoo (maybe with a different name).
>
> What I do not understand is: people who used Gentoo and especially USE
> flag are saying that it ends with a big mess with broken packages.
> Therefore, why does Guix want to reproduce the mess?

Because it is extremely powerful: in a simple declaration like "no-X +
pulseaudio + python3 + gtk2" you can build /your entire system/,
according to the flags you've set, in just one command.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[zimoun]

Hi,

Thank you for the explanations.

On Mon, 20 Jan 2020 at 19:57, Pierre Neidhardt  wrote:

> > The solution of 2. and 3. seems to write, as Ludo mentioned:
> >
> > --8<---cut here---start->8---
> > (define (make-me-a-package option1 option2)
> > (package
> >   …))
> > --8<---cut here---end--->8---
>
> The ellipsis is a bit vague here.  What is this trying to do?

What you wrote below. :-)


> --8<---cut here---start->8---
> (define-public you-get
>   (package
> (name "you-get")
> (version "0.4.1355")
> (PARAMETERS VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
> (source (origin
>   (method git-fetch)
>   (uri (git-reference
> (url "https://github.com/soimort/you-get.git";)
> (commit (string-append "v" version
>   (file-name (git-file-name name version))
>   (sha256
>(base32
> "0xq7z04hvw3b3npiahlpzhbxsjvam9n9dynplyrkn84dx6k9ajbj"
> (build-system python-build-system)
> (inputs
>  `(("PLAYER" ,(DEREF-PARAM VIDEO-PLAYER))
>,@(IF (DEREF-PARAM WITH-FFMPEG)
>  ;; FOR MULTI-PART AND >=1080P VIDEOS
>  `("FFMPEG" ,FFMPEG)
>  '(
> (arguments
>  `(#:PYTHON ,(DEREF-PARAM PYTHON-VERSION)
>#:phases
>(modify-phases %standard-phases
>  ,(WHEN (DEREF-PARAM WITH-FFMPEG)
> (add-after 'unpack 'qualify-input-references
>   ;; Explicitly invoke the input ffmpeg, instead of whichever one
>   ;; happens to be in the user's $PATH at run time.
>   (lambda* (#:key inputs #:allow-other-keys)
> (let ((ffmpeg (string-append (assoc-ref inputs "ffmpeg")
>  "/bin/ffmpeg")))
>   (substitute* "src/you_get/processor/ffmpeg.py"
> ;; Don't blindly replace all occurrences of ‘'ffmpeg'’: 
> the
> ;; same string is also used when sniffing ffmpeg's output.
> (("(FFMPEG == |\\()'ffmpeg'" _ prefix)
>  (string-append prefix "'" ffmpeg "'")))
>   #t
>  (ADD-AFTER 'UNPACK 'TWEAK-PLAYER-SETTINGS
>(LAMBDA* (#:KEY INPUTS #:ALLOW-OTHER-KEYS)
>  (MATCH ,(DEREF-PARAM VIDEO-PLAYER)
>(VLC
> ;; DO SOMETHING WITH VLC.
> )
>(MPV
> ;; DO SOMETHING WITH MPV.
> )
>(_
> ;; ERROR OUT?
> )
>#:tests? #f)); XXX some tests need Internet access
> (synopsis "Download videos, audio, or images from Web sites")
> (description
>  "You-Get is a command-line utility to download media contents (videos,
> audio, images) from the Web.  It can use either mpv or vlc for playback.")
> (home-page "https://you-get.org/";)
> (license license:expat)))
> --8<---cut here---end--->8---
>
> In the above I've highlighted the changes in uppercase.

Welcome in a big and unmaintainable mess! :-)


And my opinion is that you described is already possible (more or
less) using 'inherit'. Except the modification of the compiling
options (build-system).


> On line (PARAMETERS ...) I've declared which parameters I'm going to use
> in my package declaration.  Those parameters must be defined globally
> somewhere in Guix.

This will end up with a big and unmaintainable mess, IMHO.


> The point of declaring the parameters in advance is that it allows the
> user to list all parameters used by a given package.

I bet that the number of broken packages will increase.



> - To let the user choose which video player to use.  This is a popular
>   USE flag on Gentoo (maybe with a different name).

What I do not understand is: people who used Gentoo and especially USE
flag are saying that it ends with a big mess with broken packages.
Therefore, why does Guix want to reproduce the mess?


All the best,
simon

Re: Parameterized packages

[Ludovic Courtès]

Hi,

ison  skribis:

> Just a thought, but could this be achieved using Guile's parameterization
> feature?
> https://www.gnu.org/software/guile/manual/html_node/Parameters.html

I don’t think so: these “parameters” (I like to write “SRFI-39
parameters” to avoid confusion, which refers to
) have little to do with
the kind of parameters discussed here.

Ludo’.

Re: Parameterized packages

[Pierre Neidhardt]

ison  writes:

> Just a thought, but could this be achieved using Guile's parameterization
> feature?
> https://www.gnu.org/software/guile/manual/html_node/Parameters.html
>
> Not sure which method will be best overall, but its something to consider
> and might avoid the need to define a bunch of extra functions. So basically
> all standard parameters could be defined globally using make-parameter, then
> in the parameters field of the package it lists the parameters they want
> which will pick up the default values, or be overridden by parameterization.
>
> Although I wonder if maybe it's better to not use global defaults for the
> parameters and just let packages set their own defaults?

Actually I don't know.  An important feature of package parameters / USE
flags is typing.  But Guile does not have much support for typing as far
as know.  So if we need to implement some sort of typing in Guile, what
we could do is create a Guile record with the following fields:

- value
- predicate

The value should not be accessible directly, instead we need some
function that will reference the value by first applying the predicate
on it and throw an error if it returns #f.  This way we can enforce some
run-time typing.  Not ideal but that could work.  Any better option?

Typed/Racket would be great here :)

Otherwise, we can ditch the typing and use Guile parameters as you suggested.

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[ison]

On Mon, Jan 20, 2020 at 07:57:30PM +0100, Pierre Neidhardt wrote:
> > The solution of 2. and 3. seems to write, as Ludo mentioned:
> >
> > --8<---cut here---start->8---
> > (define (make-me-a-package option1 option2)
> > (package
> >   …))
> > --8<---cut here---end--->8---

Just a thought, but could this be achieved using Guile's parameterization
feature?
https://www.gnu.org/software/guile/manual/html_node/Parameters.html

Not sure which method will be best overall, but its something to consider
and might avoid the need to define a bunch of extra functions. So basically
all standard parameters could be defined globally using make-parameter, then
in the parameters field of the package it lists the parameters they want
which will pick up the default values, or be overridden by parameterization.

Although I wonder if maybe it's better to not use global defaults for the
parameters and just let packages set their own defaults?

Re: Parameterized packages

[Pierre Neidhardt]

zimoun  writes:

> What is the final aim to have parametrized packages?
> What does it mean "parametrized"?

Easy and composable customization of packages.

> Does it mean extend the transformation options as Ludo described [2].

I think you forgot this reference.  If you meant "like Ludo described in
the original post" I'd say yes, more or less that.

> Could you define what do you mean by "parameters"?
> Sorry to be slow but I am not clear about what we are talking about
> precisely. :-)

No problem.  "Parameters" is a term Ludo came up with, I believe.  We
can always change the naming later.

In my understanding, a parameter is a "first class, globally defined
object -- that comes with a description and a type -- which value can be
used as condition in a package to modify its output.  A package may rely
on multiple parameters to modify its output."

> 1. Because the initial Ludo's message was about glic-locale which is
> somehow an input.

Yes, but in my opinion the example just happened to be an input
modification, it does not have to be.  Gentoo USE flags don't only
change the dependencies, they also change the compilation options, etc.

> 2. The thread spoke also about flags à la Gentoo USE flags.

Indeed, they are essentially the same, but recent discussions are
leading towards limiting Guix parameters a bit more to avoid
a combinatorial explosion in the maintenance complexity.

> 3. Other messages were about set X or Wayland or no-X for this or that 
> package.

Just like Gentoo "X" USE flag I think.

> 4. Another were about change the toolchain of the build systems.
> etc.

In my opinion, we don't need to change anything in the build system.

The point is that parameters can be used anywhere in the package
definition, and thus it can change #:arguments fields like #:python to
customize the python builder.

> All are "parameters". :-)
>
> The solution of 1. and 4. seems to be able to re-write all the implicit 
> inputs.

As Ison mentioned, this might not be reasonable because it might just
break too often.  A better approach could be to only customize the
direct inputs (and build system options).

> The solution of 2. and 3. seems to write, as Ludo mentioned:
>
> --8<---cut here---start->8---
> (define (make-me-a-package option1 option2)
> (package
>   …))
> --8<---cut here---end--->8---

The ellipsis is a bit vague here.  What is this trying to do?

> Well, could you clarify what do you have in mind about "parameters"?
> Just to stay on the wavelength and keep focus. :-)

OK, I'll try with one full example:

--8<---cut here---start->8---
(define-public you-get
  (package
(name "you-get")
(version "0.4.1355")
(PARAMETERS VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
(source (origin
  (method git-fetch)
  (uri (git-reference
(url "https://github.com/soimort/you-get.git";)
(commit (string-append "v" version
  (file-name (git-file-name name version))
  (sha256
   (base32
"0xq7z04hvw3b3npiahlpzhbxsjvam9n9dynplyrkn84dx6k9ajbj"
(build-system python-build-system)
(inputs
 `(("PLAYER" ,(DEREF-PARAM VIDEO-PLAYER))
   ,@(IF (DEREF-PARAM WITH-FFMPEG)
 ;; FOR MULTI-PART AND >=1080P VIDEOS
 `("FFMPEG" ,FFMPEG)
 '( 
(arguments
 `(#:PYTHON ,(DEREF-PARAM PYTHON-VERSION)
   #:phases
   (modify-phases %standard-phases
 ,(WHEN (DEREF-PARAM WITH-FFMPEG)
(add-after 'unpack 'qualify-input-references
  ;; Explicitly invoke the input ffmpeg, instead of whichever one
  ;; happens to be in the user's $PATH at run time.
  (lambda* (#:key inputs #:allow-other-keys)
(let ((ffmpeg (string-append (assoc-ref inputs "ffmpeg")
 "/bin/ffmpeg")))
  (substitute* "src/you_get/processor/ffmpeg.py"
;; Don't blindly replace all occurrences of ‘'ffmpeg'’: the
;; same string is also used when sniffing ffmpeg's output.
(("(FFMPEG == |\\()'ffmpeg'" _ prefix)
 (string-append prefix "'" ffmpeg "'")))
  #t
 (ADD-AFTER 'UNPACK 'TWEAK-PLAYER-SETTINGS
   (LAMBDA* (#:KEY INPUTS #:ALLOW-OTHER-KEYS)
 (MATCH ,(DEREF-PARAM VIDEO-PLAYER)
   (VLC
;; DO SOMETHING WITH VLC.
)
   (MPV
;; DO SOMETHING WITH MPV.
)
   (_
;; ERROR OUT?
)
   #:tests? #f)); XXX some tests need Internet access
(synopsis "Download videos, audio, or images from Web sites")
(description
 "You-Get is a command-line utility to downloa

Re: Parameterized packages

[zimoun]

On Mon, 20 Jan 2020 at 10:08, Pierre Neidhardt  wrote:
>
> Ludovic Courtès  writes:
>
> > I agree.  ‘package-input-rewriting’ gives us almost what you want, with
> > the limitation that implicit inputs are ignored (which is a good thing
> > sometimes, and a problem in cases where you want to experiment with
> > toolchains, as you write).
> >
> > What we’d need is a variant of ‘package-input-rewriting’ that somehow
> > walks through implicit dependencies.  It could do that by changing the
> > build system, I guess.
>
> To clarify, this does not cover the extent of what "parameters" could
> do, since parameters are not just about input rewriting.

As we discussed here [1]:
<<
> Should we touch implicit inputs, we would have to parse all the
> references and not just the explicit inputs like --with-input does.

My mind is not clear at all.

What is the final aim to have parametrized packages?
What does it mean "parametrized"?
Does it mean extend the transformation options as Ludo described [2].
Does it mean more?
>>

Could you define what do you mean by "parameters"?
Sorry to be slow but I am not clear about what we are talking about
precisely. :-)

1. Because the initial Ludo's message was about glic-locale which is
somehow an input.
2. The thread spoke also about flags à la Gentoo USE flags.
3. Other messages were about set X or Wayland or no-X for this or that package.
4. Another were about change the toolchain of the build systems.
etc.

All are "parameters". :-)

The solution of 1. and 4. seems to be able to re-write all the implicit inputs.

The solution of 2. and 3. seems to write, as Ludo mentioned:

--8<---cut here---start->8---
(define (make-me-a-package option1 option2)
(package
  …))
--8<---cut here---end--->8---


Well, could you clarify what do you have in mind about "parameters"?
Just to stay on the wavelength and keep focus. :-)


[1] https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00274.html




Cheers,
simon

Re: Parameterized packages

[zimoun]

Hi Pierre,

On Fri, 17 Jan 2020 at 17:56, Pierre Neidhardt  wrote:

> In this case, it's trivial to use parameters to influence which compiler
> the build system will use.

I am not sure that "trivial" is the correct word. ;-)


> For gnu-build-system (with gcc, clang, etc.) we can probably do similar
> things already by setting CC.

Yes, and about Rust, Ruby, Perl, etc.

I do think it is the right abstraction. :-)


> The solution to your problem in my opinion is simply to expose just the
> right amount of options through #:arguments for all build systems.
> Would that be satisfactory to you?

Maybe. I am not sure to see what do you mean by '#:arguments'.

If you mean something that does what does
`package-with-explicit-python' to rewrite the 'arguments' field, yes
it is something in this flavor. But generic enough.

For example compare `package-with-explicit-python' and
`package-with-explicit-ocaml'.

Well, something to tweak `lower'. And the question is: what is the
right abstraction?


All the best,
simon

Re: Parameterized packages

[zimoun]

Hi Ludo,

On Sun, 19 Jan 2020 at 21:34, Ludovic Courtès  wrote:

> > I feel something is lacking.
>
> I agree.  ‘package-input-rewriting’ gives us almost what you want, with
> the limitation that implicit inputs are ignored (which is a good thing
> sometimes, and a problem in cases where you want to experiment with
> toolchains, as you write).

Yes. And I do not know what should be exposed, i.e., at which level.

As you probably better know than me, in HPC world, tweaking the
toolchain and its options is an habit (bad or good, not the point :-))


> What we’d need is a variant of ‘package-input-rewriting’ that somehow
> walks through implicit dependencies.  It could do that by changing the
> build system, I guess.

It would be cool! :-)


Cheers,
simon

Re: Parameterized packages

[Pierre Neidhardt]

Ludovic Courtès  writes:

> I agree.  ‘package-input-rewriting’ gives us almost what you want, with
> the limitation that implicit inputs are ignored (which is a good thing
> sometimes, and a problem in cases where you want to experiment with
> toolchains, as you write).
>
> What we’d need is a variant of ‘package-input-rewriting’ that somehow
> walks through implicit dependencies.  It could do that by changing the
> build system, I guess.

To clarify, this does not cover the extent of what "parameters" could
do, since parameters are not just about input rewriting.

What do you think of my other answer on using parameters to tweak the
`arguments' field?  (E.g. the "python" argument for the python-build-system.)

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[Ludovic Courtès]

Hi,

zimoun  skribis:

> To be clear, the seed (bootstrap) fixes how the compilers OCaml@4.01,
> OCaml@4.07, OCaml@4.09, CPython@2.7, CPython@3.5, CPython@3.6, PyPy
> (not yet in Guix), GCC@8, GCC@9, Clang@8, Clang@9, etc are compiled.
> Then, it is difficult to use them to compile a package with one of
> them.
>
> Do we end with 'package-with-ocaml4.01', 'package-with-ocaml4.07',
> 'package-with-ocaml4.09', 'package-with-python2',
> 'package-with-python3.5', 'package-with-python3.6',
> 'package-with-pypy', 'package-with-gcc8', 'package-with-gcc9', etc.?
>
> Do the users have to write all these 'package-with-'?
> And if one wants to use CPython@3.5 and GCC@8, does they need to
> define the package using:
>
> (define-public my-foo
>   (package-with-python3.5
>(package-with-gcc8 foo)))
>
> ?
>
> I feel something is lacking.

I agree.  ‘package-input-rewriting’ gives us almost what you want, with
the limitation that implicit inputs are ignored (which is a good thing
sometimes, and a problem in cases where you want to experiment with
toolchains, as you write).

What we’d need is a variant of ‘package-input-rewriting’ that somehow
walks through implicit dependencies.  It could do that by changing the
build system, I guess.

Thanks,
Ludo’.

Re: Parameterized packages

[Ludovic Courtès]

Hi,

Nicolò Balzarotti  skribis:

> Hello! I've not followed the discussion with too much attention, so
> forgive me if I'm plain wrong.
> But, if I get this right, to me it seems we want something similar to
> what nix is already doing.

To me, what Nix is doing amounts to:

  (define (make-me-a-package option1 option2)
(package
  …))

I believe the extra bit that would be provided here is a bridge between
the API and the user interface.

Ludo’.

Re: Parameterized packages

[Pierre Neidhardt]

Hi Zimoun!

zimoun  writes:

> Well, it would ease comparison in the HPC world. :-)
>
>
> It is not related to the "parametrized package" in the sense of flag
> options. :-)
>
> And I do not know if it make sense. What do you think?

I think you are making a lot of sense and yes, parameters should
probably be able to customize the builder as well.

But as I understand it, all the "configurable part" of the builder is
expose through the #:arguments field.  A simple example of this is
python: the python-build-system has an argument which allows the package
to specify whether we use python-2 or python-3.

In this case, it's trivial to use parameters to influence which compiler
the build system will use.

For gnu-build-system (with gcc, clang, etc.) we can probably do similar
things already by setting CC.

The solution to your problem in my opinion is simply to expose just the
right amount of options through #:arguments for all build systems.
Would that be satisfactory to you?

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[Pierre Neidhardt]

Hi!

L  p R n  d n writes:

> (define-modificator headless
> `((xorg (modify-package xorg
> ...))

Thanks for sharing, this is an interesting approach!  But it may raise a
major challenge.  The point of parameters is that they must compose
(e.g. headless + static-lib), so can we do that with the "modificator"
approach?

Say for instance that 2 parameters in particular are not compatible for
a given package, how to we handle that?

This probably needs more thoughts! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[Pierre Neidhardt]

Hi Ricardo,

Ricardo Wurmus  writes:

> I think this is a sensible approach, though it would require agreement
> or at least coordination among package contributors about what
> parameters to use.  For example, one such parameter could be
> “#:headless” to disable any graphical user interface or desktop feature,
> but a case could just as well be made for “#:gui” or “#:graphical”…
>
> Coordination hasn’t been a problem so far because we managed to agree on
> build systems and utilities, but it seems to me that parameters to match
> on span a much larger range — I feel reminded of the problem of package
> tags or categories, which are a similarly large field that defies
> agreement.

I think Tobias suggestion is a good solution to this: make parameters
first class, global objects.  A package can only use globally declared
parameters.  This way we have a limited set of well-defined parameters
under control (understand: no duplicates, consistent naming).

As Tobias pointed out, global declaration of parameters allows us to
centralize their documentation and types.

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[Pierre Neidhardt]

Hi ison!

ison  writes:

> Maybe the current discussion is trying too hard to emulate Gentoo's USE
> flags and dependency graph concept (perhaps its my fault for bringing up
> global flags). But that feels like introducing "side effects", and maybe the
> whole idea should be treated more "functionally" in Guix.
>
> That is, simplify the problem to the mere concept of passing arguments to
> functions and nothing more. Take the headless server example: some parameter
> is passed to a package such as
> '("-X")
> That package would then be entirely responsible for what to do with them. If
> the package decides not to pass the same parameters to its inputs then the
> inputs are simply built without any parameters.

Yes, I am starting to come to a similar conclusion.
I think this makes more sense in terms of maintainability.

What that in mind, maintenance would still get more complex than it is
today: a package with N parameters would have 2^N possible outputs.
It quickly gets hard to test them all.  That said, we can probably assume
that in most cases parameters won't conflict with each other.

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[zimoun]

On Thu, 16 Jan 2020 at 20:06, ison  wrote:
> On Wed, Jan 15, 2020 at 02:54:25PM +0100, zimoun wrote:

> > And what I was thinking is a mechanism to easily set some arguments to
> > the build-system; for example changing the compiler toolchain (say
> > replacing GCC by Clang/LLVM).
> >
> > Well, as I said, I do not know if it is related to "parametrized
> > packages" because I am not sure to understand the final aim for these
> > "parametrized packages". :-)

[...]

> That is, simplify the problem to the mere concept of passing arguments to
> functions and nothing more. Take the headless server example: some parameter
> is passed to a package such as
> '("-X")
> That package would then be entirely responsible for what to do with them. If
> the package decides not to pass the same parameters to its inputs then the
> inputs are simply built without any parameters.

I agree.

What I was trying to suggest and/or discuss is to see the build-system
as a function where the compiler toolset is one argument.

Let consider these "compilers" (associated with complete toolset for
"compiling"): OCaml@4.01, OCaml@4.07, OCaml@4.09, CPython@2.7,
CPython@3.5, CPython@3.6, PyPy (not yet in Guix), GCC@8, GCC@9,
Clang@8, Clang@9, etc.

Today, for one particular build system, the compiler is fixed. To use
another compiler than the default one, the trick is to have
'package-with-explicit' and each build-system has to provide
'package-with-' where  is hard coded.

To be clear, the seed (bootstrap) fixes how the compilers OCaml@4.01,
OCaml@4.07, OCaml@4.09, CPython@2.7, CPython@3.5, CPython@3.6, PyPy
(not yet in Guix), GCC@8, GCC@9, Clang@8, Clang@9, etc are compiled.
Then, it is difficult to use them to compile a package with one of
them.

Do we end with 'package-with-ocaml4.01', 'package-with-ocaml4.07',
'package-with-ocaml4.09', 'package-with-python2',
'package-with-python3.5', 'package-with-python3.6',
'package-with-pypy', 'package-with-gcc8', 'package-with-gcc9', etc.?

Do the users have to write all these 'package-with-'?
And if one wants to use CPython@3.5 and GCC@8, does they need to
define the package using:

(define-public my-foo
  (package-with-python3.5
   (package-with-gcc8 foo)))

?

I feel something is lacking.

And do not take me wrong, I am not suggesting that Guix should
maintain and ensure it works for all the combinations. The default is
already enough! :-)
And yes, it means potentially recompiling the world. Try to plot "guix
graph -t bag-emerged foo". ;-)


Why do I want that?

For example, a couple of software have been used to simulate complex
physical phenomena. Using the commit A -- which provides the seed A
and all compiling tools in the version X -- I get some results. Then
using the commit B -- which provided the seed B and all the compiling
tools in the version Y -- I get different results. Where does come
from the difference? From my couple of software which are not
reproducible? From round-off errors and floating point arithmetic
mess? Well, the most probable... but can I reduce the search space of
the difference? And I would like to be able to fix the seed to A, then
used the compiling tools in the version X and compare using this very
same seed A with the compiling tools in the version Y. Do the same
with seed B.

Well, it would ease comparison in the HPC world. :-)


It is not related to the "parametrized package" in the sense of flag
options. :-)

And I do not know if it make sense. What do you think?


All the best,
simon

Re: Parameterized packages

[L p R n d n]

Hello!

Also followed the thread from a little far away. It's probably an
awfull idea (I don't even know if it's possible) but throwing in
different perspetives migh give birth to new ones so here I go.

Instead of thinking in terms of use flags or parameterized packages maybe
using something that I would call "modificators" might be an idea.
I see it as some kind of list of funtions modifying packages (If it's
clear?). The main differences would be that the package definitions
would be left untouched and that modificators' definitions would be
centralized keeping the complexity and the maintaining burden away.
Something like modificators maintainer might emerge then?

That could look like:

(define-modificator headless
`((xorg (modify-package xorg
...))

or something replacing a package directly without introducing new syntax:

  (xorg (package (inherit xorg)
...

Then givin a list of packages, apply a modificator to it and get a
modified list of packages?

Hope it helps,

Have a nice day,

L  p R n  d n

Re: Parameterized packages

[Nicolò Balzarotti]

Hello! I've not followed the discussion with too much attention, so
forgive me if I'm plain wrong.
But, if I get this right, to me it seems we want something similar to
what nix is already doing.

Taking handbrake[1] recipe as an example: there's the useGtk flag that
can be passed to the package definition, and that affects build
options. Since in guix dependencies build inputs are specified inside
the package definition, propagating those flags should be easy.  If we
standardize on flags names (don't have "-X", "noX", "disable-x"...), it
should be easy to incrementally add flags support to all package
dependencies.  I imagine (with an invalid syntax probably) something
like:

(inputs
`(,(unless (member build-flags 'no-x) ("libx11" libx11))
  ; this package always need x, don't pass flags
  ("input" ,input "out" (filter (lambda (x) (not (eq? x 'no-x))) build-flags))
  ("other-input" ,input))) ; flags are passed implicitly here?

This would probably increase build recipes complexity a bit (a better
understanding of programs' optional/required features, flasgs available
in guix & co. is required).

If I missed the point, please ignore this message :)

Thanks, Nicolò

[1] 
https://github.com/NixOS/nixpkgs/blob/9ec439e009098b3132848c525a033a602f74faed/pkgs/applications/video/handbrake/default.nix

ison  writes:

> On Wed, Jan 15, 2020 at 02:54:25PM +0100, zimoun wrote:
>> Everything is doable with Guix. ;-)
>> However, it is not clear to me what is the best/easiest way to go.
>> For example, here [2] I give a try.
>> 
>> [2] https://lists.gnu.org/archive/html/help-guix/2020-01/msg00087.html
>> 
>> 
>> And what I was thinking is a mechanism to easily set some arguments to
>> the build-system; for example changing the compiler toolchain (say
>> replacing GCC by Clang/LLVM).
>> 
>> Well, as I said, I do not know if it is related to "parametrized
>> packages" because I am not sure to understand the final aim for these
>> "parametrized packages". :-)
>
> Maybe the current discussion is trying too hard to emulate Gentoo's USE
> flags and dependency graph concept (perhaps its my fault for bringing up
> global flags). But that feels like introducing "side effects", and maybe the
> whole idea should be treated more "functionally" in Guix.
>
> That is, simplify the problem to the mere concept of passing arguments to
> functions and nothing more. Take the headless server example: some parameter
> is passed to a package such as
> '("-X")
> That package would then be entirely responsible for what to do with them. If
> the package decides not to pass the same parameters to its inputs then the
> inputs are simply built without any parameters.
>
> This would have the benefit of not breaking any existing packages regardless
> of what parameters are sent to them.
> And packages can be modified by maintainers to accept parameters and pass
> them on to their inputs as the need arises over time (e.g. packages typically
> installed by headless servers can be modified to pass the "-X" flag on to
> certain inputs).
>
> Additionally if someone wanted to take the risk and _force_ all inputs to
> use the same flags then they can use a function similar to
> "package-input-rewriting" (there could be a "package-use-flags-rewriting"),
> so it becomes similar to how other types of transformations are currently
> handled in Guix.
>
> Ideally packages can still be made to allow all flags to "trickle down" to
> their dependencies without input-rewriting simply by passing all parameters
> into their inputs in the package definition, but the difference here is that
> we don't force it. That's something that can be approached over time, or
> not. The idea is to just let packages do it if they want to, and let the
> package maintainers filter the list of flags before passing on to inputs if
> they need to, and the whole thing becomes more predictable and functional.
> This also avoids the problem of which types of inputs to modify (normal
> inputs, native-inputs, etc) because then it's just up to the package
> maintainer.

Re: Parameterized packages

[Ricardo Wurmus]

ison  writes:

> Maybe the current discussion is trying too hard to emulate Gentoo's USE
> flags and dependency graph concept (perhaps its my fault for bringing up
> global flags). But that feels like introducing "side effects", and maybe the
> whole idea should be treated more "functionally" in Guix.
>
> That is, simplify the problem to the mere concept of passing arguments to
> functions and nothing more. Take the headless server example: some parameter
> is passed to a package such as
> '("-X")
> That package would then be entirely responsible for what to do with them. If
> the package decides not to pass the same parameters to its inputs then the
> inputs are simply built without any parameters.

I think this is a sensible approach, though it would require agreement
or at least coordination among package contributors about what
parameters to use.  For example, one such parameter could be
“#:headless” to disable any graphical user interface or desktop feature,
but a case could just as well be made for “#:gui” or “#:graphical”…

Coordination hasn’t been a problem so far because we managed to agree on
build systems and utilities, but it seems to me that parameters to match
on span a much larger range — I feel reminded of the problem of package
tags or categories, which are a similarly large field that defies
agreement.

--
Ricardo

Re: Parameterized packages

[ison]

On Wed, Jan 15, 2020 at 02:54:25PM +0100, zimoun wrote:
> Everything is doable with Guix. ;-)
> However, it is not clear to me what is the best/easiest way to go.
> For example, here [2] I give a try.
> 
> [2] https://lists.gnu.org/archive/html/help-guix/2020-01/msg00087.html
> 
> 
> And what I was thinking is a mechanism to easily set some arguments to
> the build-system; for example changing the compiler toolchain (say
> replacing GCC by Clang/LLVM).
> 
> Well, as I said, I do not know if it is related to "parametrized
> packages" because I am not sure to understand the final aim for these
> "parametrized packages". :-)

Maybe the current discussion is trying too hard to emulate Gentoo's USE
flags and dependency graph concept (perhaps its my fault for bringing up
global flags). But that feels like introducing "side effects", and maybe the
whole idea should be treated more "functionally" in Guix.

That is, simplify the problem to the mere concept of passing arguments to
functions and nothing more. Take the headless server example: some parameter
is passed to a package such as
'("-X")
That package would then be entirely responsible for what to do with them. If
the package decides not to pass the same parameters to its inputs then the
inputs are simply built without any parameters.

This would have the benefit of not breaking any existing packages regardless
of what parameters are sent to them.
And packages can be modified by maintainers to accept parameters and pass
them on to their inputs as the need arises over time (e.g. packages typically
installed by headless servers can be modified to pass the "-X" flag on to
certain inputs).

Additionally if someone wanted to take the risk and _force_ all inputs to
use the same flags then they can use a function similar to
"package-input-rewriting" (there could be a "package-use-flags-rewriting"),
so it becomes similar to how other types of transformations are currently
handled in Guix.

Ideally packages can still be made to allow all flags to "trickle down" to
their dependencies without input-rewriting simply by passing all parameters
into their inputs in the package definition, but the difference here is that
we don't force it. That's something that can be approached over time, or
not. The idea is to just let packages do it if they want to, and let the
package maintainers filter the list of flags before passing on to inputs if
they need to, and the whole thing becomes more predictable and functional.
This also avoids the problem of which types of inputs to modify (normal
inputs, native-inputs, etc) because then it's just up to the package
maintainer.

Re: Parameterized packages

[zimoun]

On Wed, 15 Jan 2020 at 12:51, Pierre Neidhardt  wrote:
>
> zimoun  writes:
>
> > For example, be able to rebuild all the packages with GCC-8.3, or to
> > install Python packages with Python 3.5 instead of the current default
> > Python 3.7.
>
> I think this would tackle a different issue.  The point of parameters is
> to offer _composable_ and _package-specific_ build options to the user.
> A build system cannot know what it means to build with Pulseaudio or
> without Wayland.

I do not see the difference between using specific locale [1] (for a
package that does not explicitly depend on glibc) and using a specific
version of the "compiler" used by the build system.

Maybe I miss the point...

[1] https://lists.gnu.org/archive/html/guix-devel/2019-05/msg00285.html


> Can't we already do what you are saying by the way?  By means of
> settings some build-system-specific arguments from fold-package?

Everything is doable with Guix. ;-)
However, it is not clear to me what is the best/easiest way to go.
For example, here [2] I give a try.

[2] https://lists.gnu.org/archive/html/help-guix/2020-01/msg00087.html


And what I was thinking is a mechanism to easily set some arguments to
the build-system; for example changing the compiler toolchain (say
replacing GCC by Clang/LLVM).

Well, as I said, I do not know if it is related to "parametrized
packages" because I am not sure to understand the final aim for these
"parametrized packages". :-)


All the best,
simon

Re: Parameterized packages

[zimoun]

Hi Pierre,

On Wed, 15 Jan 2020 at 10:40, Pierre Neidhardt  wrote:

> - The source.
> - The explicit inputs.
> - the implicit inputs.
> - The build inputs.
> - The build system inputs.
> - Recursive inputs.
> - ...?
>
> Which one should we expose?  I don't know.  If we want the system to
> have some set of properties, I guess only the resulting packages matther
> and the build inputs don't.

But the resulting packages depends on the build inputs.
So if the resulting packages matter, then the build inputs too.

Well, you already described the issue here [1] (point 7.) -- if I
understand correctly.

[1] https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00026.html


> Should we touch implicit inputs, we would have to parse all the
> references and not just the explicit inputs like --with-input does.

My mind is not clear at all.

What is the final aim to have parametrized packages?
What does it mean "parametrized"?
Does it mean extend the transformation options as Ludo described [2].
Does it mean more?

[2] https://lists.gnu.org/archive/html/guix-devel/2019-05/msg00285.html


> > but be able to
> > recompile a matrix of combinaison would be really cool!
>
> Indeed.  But now I'm starting to wonder if it is really doable :p

What I was thinking of is to parametrize only the build systems.

For example, be able to rebuild all the packages with GCC-8.3, or to
install Python packages with Python 3.5 instead of the current default
Python 3.7.



Cheers,
simon

Re: Parameterized packages

[Pierre Neidhardt]

Hi Simon!

You are making a good point: there are different levels of
"accessibility" when it comes to inputs:

- The source.
- The explicit inputs.
- the implicit inputs.
- The build inputs.
- The build system inputs.
- Recursive inputs.
- ...?

Which one should we expose?  I don't know.  If we want the system to
have some set of properties, I guess only the resulting packages matther
and the build inputs don't.

Should we touch implicit inputs, we would have to parse all the
references and not just the explicit inputs like --with-input does.

> but be able to
> recompile a matrix of combinaison would be really cool!

Indeed.  But now I'm starting to wonder if it is really doable :p

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[zimoun]

Hi Pierre,

I do not not if it related.

Currently, the option "--with-input" does not re-write the implicit
inputs. Probably for good reasons. :-)

For example, say I would like to have the version 3.4.3 of R (2017
old). This version is really old and had disappeared. Other said, it
is older than the introduction of "inferior"s and so "guix
time-machine" does not help (but that another story).
At this point, it is not matter the GCC version that it is used to
compile this R version.

Well, it is not too complicated:

  git show cbe1314a7e:gnu/packages/statistics.scm > /tmp/stuff/old-stats.scm
  # edit old-stats.scm to tweak the modules
  guix build -L /tmp r-minimal@3.4.3

Now, I would like to build the package r-rcpp at the version 1.0.3
which is introduced recently using the old R version 3.4.3. Well, the
naive idea is:

   guix build -L /tmp --with-inputs=r-minimal=r-minimal@3.4.3 r-rcpp

but because r-minimal is an implicit inputs, it is not re-written.

This behaviour is expected because the bag is really huge. To be
convinced, try: "guix graph -t bag r-rcpp | dot -Tps >
r-rcpp-bag.ps"... and wait... :-)


>From a practical point of view, re-write any implicit inputs is
touchy.  And I do not know if the "parametrized package" is related or
not at all -- I do not see real difference between re-compiling with
this flags and re-compiling with this version -- but be able to
recompile a matrix of combinaison would be really cool!


All the best,
simon

Re: Parameterized packages

[Pierre Neidhardt]

Hi Ludo!

Ludovic Courtès  writes:

> The way I see it, we’re still toying with the idea and its pros and
> cons—discussions about CLI syntax can come later.  ;-)

Sure thing!

> The added flexibility of package parameters is definitely nice, but
> really, maintainability is a big concern.  The example Tobias gave (a
> parameter to enable/disable X11 support) is interesting because it shows
> that parameters can quickly become ubiquitous and get “out of control”,
> from a maintenance viewpoint.

What I understood from Tobias comment on the X11 support parameter is
that we need to store the parameter metadata centrally.  Why do you
think this suggests it will get "out of control"?


> That’s another good example of a problem that would arise.  :-/
> It doesn’t seem reasonable to me to add complex logic in (guix packages)
> to deal with this issue; I would very much prefer to leave input
> handling unchanged.
>
> I’m worried about the maintenance cost of parameters.  Having the
> feature is one thing, but being able to guarantee that the package
> combinations we offer all work is another one.
>
> We could have the feature and use it very seldom in Guix itself, but I’m
> guessing that that’s not what you have in mind.

Indeed, the ultimate goal would be to do things like "install a headless
system" and "use static libraries everywhere", etc.

> Also, for CI purposes, we would need a way to enumerate all the possible
> combinations…

Considering the number of outputs would explode as parameters get added,
it does not seem reasonable to try building everything.  Maybe an easy
option would be:

- Only build the default packages (i.e. using the default values of the
  parameters).
- If a non-default output is requested often enough (to be defined) then
  build it.

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[Ludovic Courtès]

Hello!

The way I see it, we’re still toying with the idea and its pros and
cons—discussions about CLI syntax can come later.  ;-)

The added flexibility of package parameters is definitely nice, but
really, maintainability is a big concern.  The example Tobias gave (a
parameter to enable/disable X11 support) is interesting because it shows
that parameters can quickly become ubiquitous and get “out of control”,
from a maintenance viewpoint.

Pierre Neidhardt  skribis:

> 7. Dependency management.
>Also known as the USE flag nightmare among Gentoo users...
>This is where hell breaks loose! :p
>
>The problem: the user wants to specify a parameter to use globally where it
>applies, on all installed packages and all their inputs, recursively.
>
>For instance, use guile-2.2.4 instead of guile for all guile libraries, or
>use pulseaudio everywhere, including in dependencies that are not 
> explicitly
>installed to the user profile.
>
>The obstacle: A package may require inputs with a specific parameter.
>For instance, BAR depends on a FOO package built with ":audio
>pulseaudio".  What happens if the user seta ":audio alsa" globally
>and installs BAR?  BAR needs to specify explicitly that its FOO input
>requires pulseaudio.  Otherwise BAR would fail to build.
>
>To specify that a package input depends on a specific parameter, we could
>extend the syntax of the (inputs ...) and (native-inputs ...) fields like 
> so:
>
>(input
> `(("foo" ,foo "(:audio pulseaudio)")))
>
>A bigger problem is that the parameter compatibility issue is 
> combinatorial:
>"Which parameter combination does BAR support?"  It's hard to know it in
>advance.  Any idea how to tackle this?

That’s another good example of a problem that would arise.  :-/
It doesn’t seem reasonable to me to add complex logic in (guix packages)
to deal with this issue; I would very much prefer to leave input
handling unchanged.

I’m worried about the maintenance cost of parameters.  Having the
feature is one thing, but being able to guarantee that the package
combinations we offer all work is another one.

We could have the feature and use it very seldom in Guix itself, but I’m
guessing that that’s not what you have in mind.

Also, for CI purposes, we would need a way to enumerate all the possible
combinations…

Apologies for spoiling the party :-), but I think these concerns need
all our attention!

Thanks,
Ludo’.

Re: Parameterized packages

[Marius Bakke]

Pierre Neidhardt  writes:

> Any thought on this, anyone?

Having used Gentoo for some years, I'm not sure a "global" parameter
system is desirable.  It could encourage enabling feature "foo", even
for packages that have experimental or broken support for "foo".

But I see how it could be useful to parameterize packages nevertheless.
I imagine adding a 'parameters' field to the package type, and write
definitions along these lines.

(package
  (parameters
   '((pulseaudio . #t)))
  (arguments
   `(#:configure-flags ,(match parameters
 ((('pulseaudio . #t)
  '("--enable-pulseaudio" "--enable-dbus"))
 (_ '()
  (inputs
   `(("foo" ,foo)
 ,@(match parameters
 ((('pulseaudio . #t)
   `(("pulseaudio" ,pulseaudio
 (_ '()))

And perhaps a "with-parameters" procedure for toggling it:

(inputs
  `(("foo" ,foo))
("bar" ,(with-parameters ((pulseaudio . #f)) bar)))

We could have a similar procedure that recursively toggles supported
parameters of all given packages.

Actually we can abuse the 'properties' field for this purpose without
having to change anything.

Well, food for thought, thanks for driving this :-)


signature.asc
Description: PGP signature

Re: Parameterized packages

[Pierre Neidhardt]

Hi!

Bringing this topic back to life now that I'm starting working on this.

1. Everyone seems to agree that we need a dedicated field in the package
   declaration.

   For now, 3 names were proposed:
   - parameters
   - options
   - argument-overrides

   I find "argument-overrides" a bit too verbose :p
   I'll go with "parameters" for now since it's the title of this thread.

2. As Tobias and Ison suggested, we need to centralize the declaration of
   parameters, so that we can list them and centralize metadata like the
   description & types.  Any suggestion on how to implement this in practice?
   Where do we store them?  How?

3. From Ludo's comment:

> • We’d need to store more info in manifest entries so that
>   transformation options are preserved upon ‘guix upgrade’.
  > ...
> • We might want to make the CLI option less verbose too, but how?

  We'd also need a way to specify the parameters in the manifest specs and from
  command line.
  Now we have "@" and ":" for versions and outputs respectively.  What about
  parentheses?  E.g.

"foo@2.0(:locale fr_FR.UTF-8 :audio pulseaudio :gui X)"

  I think a plist like above would be less verbose than an alist.

  To apply parameters to the rest of the command line, we could also do:

guix install --with-parameters="(:locale ...)" package-foo package-bar

4. From Ludo's comment:
   > • We might need to expose the package parameters somehow, so that if
   >one types ‘--with-argument=foobar=baz’, they get a correct error
   >message saying that “foobar” is not a known parameter.

   I think we should only display a warning so that we can emulate global USE
   flags as Ison suggested.

5. Global parameter storage

   When using "guix install" it can be convenient not to have to repeat the
   parameter specification.  So it'd be nice to store it somewhere.  What about
   a file in ~/.config/guix/parameters.scm with something like

   (parameters
 :locale fr_FR.UTF-8
 :audio pulseaudio
 ...)

   ?

6. Leverage build systems to automate parameterization when possible.

   Some build systems use well-known flags.  For instance, for the
   gnu-build-system if the user sets the parameter "(:audio pulseaudio)" we 
could
   automatically pass --enable-pulseaudio to the "configure" flags, even
   for packages that didn't specify their own well-known parameters.

   It would be nice if build systems had a way to report if a flag is supported
   or not.  Any clue?

   To implement this, we could do the following:

   (define-parameter audio
 "Some description"
 (types ...)
 (argument
   (lambda (pkg)
 (match (package-build-system pkg)
   (gnu-build-system
 (add-to-configure-flags "--enable-pulseaudio"))
   (cmake-build-system
 (...))

7. Dependency management.
   Also known as the USE flag nightmare among Gentoo users...
   This is where hell breaks loose! :p

   The problem: the user wants to specify a parameter to use globally where it
   applies, on all installed packages and all their inputs, recursively.

   For instance, use guile-2.2.4 instead of guile for all guile libraries, or
   use pulseaudio everywhere, including in dependencies that are not explicitly
   installed to the user profile.

   The obstacle: A package may require inputs with a specific parameter.
   For instance, BAR depends on a FOO package built with ":audio
   pulseaudio".  What happens if the user seta ":audio alsa" globally
   and installs BAR?  BAR needs to specify explicitly that its FOO input
   requires pulseaudio.  Otherwise BAR would fail to build.

   To specify that a package input depends on a specific parameter, we could
   extend the syntax of the (inputs ...) and (native-inputs ...) fields like so:

   (input
`(("foo" ,foo "(:audio pulseaudio)")))

   A bigger problem is that the parameter compatibility issue is combinatorial:
   "Which parameter combination does BAR support?"  It's hard to know it in
   advance.  Any idea how to tackle this?



   The easy way around this is to not propagate the global parameters by
   default, but this seriously limits the usefulness of global parameters.

   Maybe a better workaround would be to automatically fall back to the default
   value of all parameters when a build fails.
   
Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Parameterized packages

[ison]

On Fri, May 17, 2019 at 02:15:29PM -0400, Mark H Weaver wrote:
> Also, at present, the current 'properties' field can be changed without
> changing the derivation, and I think that's quite useful.  It's nice to
> be able to do things like increase the build timeouts on a core package,
> for the sake of a slow architecture, without forcing rebuilds of
> everything on top of that package on other architectures.
> 
> So, I would prefer to see a different package field used for this.

Just a suggestion: Could the field allow extra arguments without throwing
an error, and also without causing a rebuild if the arguments aren't used?
If so then users could emulate _global_ USE flags in their system config by
simply mapping all packages through a function to modify the parameters.

Such an idea also seems to call for some standardization of parameters
across packages.

Re: Parameterized packages

[Chris Marusich]

Mark H Weaver  writes:

>> Ludovic Courtès wrote:
>>> While thinking about  and
>>> looking
>>> for ways to allow users to install just the locales they need right
>>> from
>>> ‘guix package’, I realized that “parameterized packages” are a
>>> low-hanging fruit
>>
>> Wow.  Unexpected turn…
>>
>> I'm still thinking about this, so all this is just me doing it out
>> loud:
>>
>>>   (package
>>> (inherit glibc-utf8-locales)
>>> (properties `((locales . ("zh_CN.utf8")
>>>
>>> and tadaam! we have a parameterized package.
>>>
>>> There’s a couple of gotchas:
>>>
>>>   • We’d need to store more info in manifest entries so that
>>> transformation options are preserved upon ‘guix upgrade’.
>>>
>>>   • We might need to expose the package parameters somehow, so
>>> that if
>>> one types ‘--with-argument=foobar.z’, they get a correct
>>> error
>>> message saying that “foobar” is not a known parameter.
>>
>> Interesting idea to piggy-back on the properties field, and it might
>> save some code (at least initially), but I don't see the overlap here.
>>
>> None of the current properties (superseded, upstream-name, *-variant,
>> cpe-name, hidden?) make much sense to expose in this way; they're
>> semimmutable facts about the package.
>
> Also, at present, the current 'properties' field can be changed without
> changing the derivation, and I think that's quite useful.  It's nice to
> be able to do things like increase the build timeouts on a core package,
> for the sake of a slow architecture, without forcing rebuilds of
> everything on top of that package on other architectures.
>
> So, I would prefer to see a different package field used for this.

I agree with Mark; using a custom package field seems better here
because the arguments change the derivation (don't they?), but
properties do not.  Maybe "argument-overrides"?

-- 
Chris


signature.asc
Description: PGP signature

Re: Parameterized packages

[Mark H Weaver]

Hi,

Tobias Geerinckx-Rice  writes:

> Ludovic Courtès wrote:
>> While thinking about  and
>> looking
>> for ways to allow users to install just the locales they need right
>> from
>> ‘guix package’, I realized that “parameterized packages” are a
>> low-hanging fruit
>
> Wow.  Unexpected turn…
>
> I'm still thinking about this, so all this is just me doing it out
> loud:
>
>>   (package
>> (inherit glibc-utf8-locales)
>> (properties `((locales . ("zh_CN.utf8")
>>
>> and tadaam! we have a parameterized package.
>>
>> There’s a couple of gotchas:
>>
>>   • We’d need to store more info in manifest entries so that
>> transformation options are preserved upon ‘guix upgrade’.
>>
>>   • We might need to expose the package parameters somehow, so
>> that if
>> one types ‘--with-argument=foobar=baz’, they get a correct
>> error
>> message saying that “foobar” is not a known parameter.
>
> Interesting idea to piggy-back on the properties field, and it might
> save some code (at least initially), but I don't see the overlap here.
>
> None of the current properties (superseded, upstream-name, *-variant,
> cpe-name, hidden?) make much sense to expose in this way; they're
> semimmutable facts about the package.

Also, at present, the current 'properties' field can be changed without
changing the derivation, and I think that's quite useful.  It's nice to
be able to do things like increase the build timeouts on a core package,
for the sake of a slow architecture, without forcing rebuilds of
everything on top of that package on other architectures.

So, I would prefer to see a different package field used for this.

 Thanks,
   Mark

Re: Parameterized packages

[Tobias Geerinckx-Rice]

Ludo',

Ludovic Courtès wrote:
While thinking about  and 
looking
for ways to allow users to install just the locales they need 
right from

‘guix package’, I realized that “parameterized packages” are a
low-hanging fruit


Wow.  Unexpected turn…

I'm still thinking about this, so all this is just me doing it out 
loud:



  (package
(inherit glibc-utf8-locales)
(properties `((locales . ("zh_CN.utf8")

and tadaam! we have a parameterized package.

There’s a couple of gotchas:

  • We’d need to store more info in manifest entries so that
transformation options are preserved upon ‘guix upgrade’.

  • We might need to expose the package parameters somehow, so 
  that if
one types ‘--with-argument=foobar=baz’, they get a correct 
error

message saying that “foobar” is not a known parameter.


Interesting idea to piggy-back on the properties field, and it 
might save some code (at least initially), but I don't see the 
overlap here.


None of the current properties (superseded, upstream-name, 
*-variant, cpe-name, hidden?) make much sense to expose in this 
way; they're semimmutable facts about the package.


While a dedicated field can provide richer metadata, such as 
description, basic ‘types’ & sane/permitted values, to be used by 
the UI.


Looking at Exherbo/ntoo, this will inevitably lead to factoring 
out very common option metadata so one doesn't have to keep 
writing ‘x11 bool "Add support for the X Window System"’ in 
every. single. package.


Then come the people asking how to set CFLAGS for all packages and 
you'll know you've made it.


  • We might want to make the CLI option less verbose too, but 
  how?


Possibly relevant: do you see package options¹ being added to 
package specifications?  That seems to be the classical approach.


Kind regards,

T G-R

[1]: I use the term ‘option’ because it's what my last FHS 
distribution called them, and it was a good distribution, and 
because ‘arguments’ is already an unrelated field in package 
expressions, and flags are boolean, but substitute whatever name 
you prefer.


signature.asc
Description: PGP signature