Re: [Nix-dev] Auto-generated expressions for applications

[zimbatm]

On Wed, 31 May 2017, 17:23 Benno Fünfstück, 
wrote:

> Profpatsch  schrieb am Mi., 31. Mai 2017 um 16:01 Uhr:
>
>> On 17-05-31 08:25am, Benno Fünfstück wrote:
>> > A package set
>> > is a consistent set of packages of a given language.
>>
>> exactly that is not possible with e.g. npm or golang packages.
>>
>
> Yes, those should be dealth with differently. Is sharing of deps between
> different applications in these languages common? (sorry, i'm not very
> familar with either)
>

With npm, a single application might contain multiple versions of the same
library. There is no forcing function that nudges developers to deal with
API compatibilities.

With golang, each project dependency is usually pinned to a single commit
ID. There is no notion of semver that can help build a constraint solver.

>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[Benno Fünfstück]

Profpatsch  schrieb am Mi., 31. Mai 2017 um 16:01 Uhr:

> On 17-05-31 08:25am, Benno Fünfstück wrote:
> > A package set
> > is a consistent set of packages of a given language.
>
> exactly that is not possible with e.g. npm or golang packages.
>

Yes, those should be dealth with differently. Is sharing of deps between
different applications in these languages common? (sorry, i'm not very
familar with either)

Otherwise, I think a rust-like approach is best for these langs: split the
build phase of applications into "fetch" and "build", where "fetch" has
network access and produces some deterministic output, and "build" then
uses "fetch" to build the package. Dependencies are built in the same
derivation as the application. This means that our build process does not
need any upstream-unsupported solutions. If the package manager does not
support this two-way build, then we should work with upstream to get that
feature supported.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[Profpatsch]

On 17-05-31 08:25am, Benno Fünfstück wrote:
> A package set
> is a consistent set of packages of a given language.

exactly that is not possible with e.g. npm or golang packages.

-- 
Proudly written in Mutt with Vim on NixOS.
Q: Why is this email five sentences or less?
A: http://five.sentenc.es
May take up to five days to read your message. If it’s urgent, call me.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[Benno Fünfstück]

I think that as a first step, we have to separate two use cases for nixpkgs:

* creating development environments to use with nix-shell etc
* installing applications

Package sets are key here. An application is built against a specific
package set, and a development environment is based on a package set. A
package set
is a consistent set of packages of a given language.

For nixpkgs itself, we should focus on providing access to *applications*.
It is not necessary to provide the latest and greatest of every library on
the planet, as long as we have no application that requires it.

Developers can use the tools we use for nixpkgs to generate package sets to
get access to the libraries they need. We may also have some "presets",
like for example "all of stackage" for Haskell.

The abstraction of a "package set" should be unified for all languages that
require consistent package sets (= no more than one version of the same
package in the dep chain. This includes Python, Haskell, .. but not for
example NodeJS, as npm supports "private" dependencies). There should be
*no* differences in how the functions are called, as this creates
confusion. They should use the same technique.

IMO, it is fine if each application uses is build against a package set
that is generated just for this application, and includes just the
dependencies for this app. However,
if we find that there is much duplication here, we could also adopt the
haskell approach which is:

* have one "superset" that includes all the needed package expressions.
this is not a package set by itself, as it may include multiple versions of
a dependencies
* then, for each application, take a consistent subset of packages out of
this set.
* we may also have a default package set if many packages share the same
versions of some dependencies. each application would then only need to
override
  a small number of dependencies in this default set.

Note that overrides on package sets must be deep overrides, so that if I
replace `foo-x` with `foo-y`, then all libraries previously depending on
`foo-x` should now also depend on `foo-y`. This ensures consistency.

If we had proper support for import-from-derivation, we could even support
the following syntax (as used in the haskell set):

packageSet.override { overrides = self: super: {
some-package = self.callHackage "some-package" "1.2.43"
}};

so the needed non-default packages can be generated on the fly.

My opinion is that for the future, we should strive to auto-generate as
much as possible in nixpkgs. I view `nix` more as a assembly language, a
common format that auto-generators can use. Wouldn't it be nice if we
could, for each package, just `anything2nix package-version` and it would
spit out a derivation for that package? Then we can focus more on the
composition of those packages. In a way, I believe that security could even
be better with such an approach as we have more time to focus on package
versions and don't need to do as much "grunt" work like figuring out easy
dependencies of packages etc


I hope this wasn't too long of an email :),

Benno

Profpatsch  schrieb am Mi., 31. Mai 2017 um 06:03 Uhr:

> On 17-05-30 08:02am, Wout Mertens wrote:
> > This actually ties into my question about nodePackages. It seems to me
> that
> > for these large packaging systems, we should have separate repos that
> > update from their source, and you can then include them into your nixpkgs
> > configuration.
>
> nodePackages is a bad example, because “node dependencies”
> make no sense on their own. Each application brings with
> it its own set of dependencies.
>
> >
> > Only packages that are useful by themselves should get a derivation in
> > nixpkgs, the rest would be in these add-on repos. For nodejs, that would
> be
> > e.g. yarn.
> > No idea how to accomplish this though.
>
> Exactly.
> As long as we keep generated formats in nice, diffable formats
> they shouldn’t impose too much noise to the system.
> Unfortunalely, the current output of node2nix is quite verbose.
>
> As far as CVEs go, one would have to integrate sites like
> https://snyk.io/vuln somehow.
>
>
> --
> Proudly written in Mutt with Vim on NixOS.
> Q: Why is this email five sentences or less?
> A: http://five.sentenc.es
> May take up to five days to read your message. If it’s urgent, call me.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[Profpatsch]

On 17-05-30 08:02am, Wout Mertens wrote:
> This actually ties into my question about nodePackages. It seems to me that
> for these large packaging systems, we should have separate repos that
> update from their source, and you can then include them into your nixpkgs
> configuration.

nodePackages is a bad example, because “node dependencies”
make no sense on their own. Each application brings with
it its own set of dependencies.

> 
> Only packages that are useful by themselves should get a derivation in
> nixpkgs, the rest would be in these add-on repos. For nodejs, that would be
> e.g. yarn.
> No idea how to accomplish this though.

Exactly.
As long as we keep generated formats in nice, diffable formats
they shouldn’t impose too much noise to the system.
Unfortunalely, the current output of node2nix is quite verbose.

As far as CVEs go, one would have to integrate sites like
https://snyk.io/vuln somehow.


-- 
Proudly written in Mutt with Vim on NixOS.
Q: Why is this email five sentences or less?
A: http://five.sentenc.es
May take up to five days to read your message. If it’s urgent, call me.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[lewo]

Freddy Rietdijk  writes:

> Hi,
>
> At several places in Nixpkgs we use auto-generated data, mostly for the
> larger package sets like Haskell. Sometimes we also use auto-generated sets
> for applications that may need different versions than are offered in the
> main package sets. In the past months several issues/PR's have been opened
> to add rather large Python applications to Nixpkgs, generated using
> `pypi2nix`.
>
> Using such tool to generate the expressions has some very big advantages.
> However, there's also a disadvantage. The expressions shouldn't only be
> updated when there's a new release of the application, but also whenever
> there's security updates in any of its generated dependencies, which the
> application maintainer now has to keep track of. Therefore, I find it quite
> a risk to have separate package sets. At the same time, we'll also likely
> run behind in the main package sets every once in a while.

Regarding the security updates, if we are able to know which versions of
libraries are used in the `pythonPackage` set and in all 'application
pythonPackage sets', in case of a CVE on a particular package version,
we could know which applications are impacted. We then have to fix them
or mark them as broken.
I don't really know how it is hard to get all python module versions
used by applications and if this would be feasable.

> What do you think of this issue? Any suggestions how we can improve this?
> Maybe we could have a server/bot that runs update scripts and opens a PR
> whenever there's an actual diff?
>
> Freddy
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[Freddy Rietdijk]

> Current approach seems to be doing the job except notifying people when
dependency is updated.

What do you base that on? With the generated data sets for applications we
have another issue which I didn't mention, and that is the sometimes overly
conservative pinning of versions by the application developer. Aside from
reporting such issues upstream, how to handle that?

Other distributions face similar issues but, at least with Python, on a
smaller scale, because they cannot have multiple versions of packages and
therefore having separate package sets is just not an option for them.

Maybe I should have written in the initial mail that I'm looking for a
policy on how we're going to deal with these ki nd of situations.

On Tue, May 30, 2017 at 2:10 PM, Tomasz Czyż  wrote:

> Current approach seems to be doing the job except notifying people when
> dependency is updated. Previously we had monitor to do some similar stuff,
> and I think vulnix can check that without much effort so I wouldn't worry
> about having duplicated packages for apps.
> I think focusing on improving CI process and security notifications
> process is the way to go.
>
> Probably we could set another process/job in hydra to check all apps for
> security issues/updates. (I'm not sure if security team doesn't have that
> already).
>
> 2017-05-30 10:17 GMT+01:00 Marc Weber :
>
>> Let met try to sum up what I remember:
>>
>> - There 3+ solutions to update "sources" documented on the wiki
>>   somewhere - ideas from comparing versions with other distributions up
>>   to adding scrapers getting latest version from web sites if I recall
>>   correctly
>>
>> - Putting automatically generated code into nixpkgs doesn't solve all
>>   issues, for corner cases you have to duplicate dependencies using
>>   different version constraints.
>>
>>   -> overhead
>>
>> - Its not always quite clear how stable the user wants to be
>>   (gimp/inskscape) case, master sometimes has new features.
>>
>>   So which versions to support ?
>>
>> - Whatever we do, we don't solve anything for other distros (which
>>   suffer the same problem), unless we switch point of view:
>>
>>   The solution would be a cross platform cross language dependency
>>   management system allowing to declare dependencies in a file so that
>>   you can even install from gihtub automatically.
>>
>>   systemPackages = [ (fromGithub "user/package" "HEAD") ] # sort rest out
>> on your own, thanks
>>
>> package A could be working, package B could be working, but [A B] in the
>> same environment not (because they both depend on executable C)
>>
>> After all we want nixpkgs to be at least stable enough to have broken
>> packages marked as broken and expect everything else to at least
>> compile/install.
>>
>> Which are the best short/middle/long term solutions ?
>>
>> Marc Weber
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>>
>
>
>
> --
> Tomasz Czyż
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[Tomasz Czyż]

Current approach seems to be doing the job except notifying people when
dependency is updated. Previously we had monitor to do some similar stuff,
and I think vulnix can check that without much effort so I wouldn't worry
about having duplicated packages for apps.
I think focusing on improving CI process and security notifications process
is the way to go.

Probably we could set another process/job in hydra to check all apps for
security issues/updates. (I'm not sure if security team doesn't have that
already).

2017-05-30 10:17 GMT+01:00 Marc Weber :

> Let met try to sum up what I remember:
>
> - There 3+ solutions to update "sources" documented on the wiki
>   somewhere - ideas from comparing versions with other distributions up
>   to adding scrapers getting latest version from web sites if I recall
>   correctly
>
> - Putting automatically generated code into nixpkgs doesn't solve all
>   issues, for corner cases you have to duplicate dependencies using
>   different version constraints.
>
>   -> overhead
>
> - Its not always quite clear how stable the user wants to be
>   (gimp/inskscape) case, master sometimes has new features.
>
>   So which versions to support ?
>
> - Whatever we do, we don't solve anything for other distros (which
>   suffer the same problem), unless we switch point of view:
>
>   The solution would be a cross platform cross language dependency
>   management system allowing to declare dependencies in a file so that
>   you can even install from gihtub automatically.
>
>   systemPackages = [ (fromGithub "user/package" "HEAD") ] # sort rest out
> on your own, thanks
>
> package A could be working, package B could be working, but [A B] in the
> same environment not (because they both depend on executable C)
>
> After all we want nixpkgs to be at least stable enough to have broken
> packages marked as broken and expect everything else to at least
> compile/install.
>
> Which are the best short/middle/long term solutions ?
>
> Marc Weber
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>



-- 
Tomasz Czyż
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[Marc Weber]

Let met try to sum up what I remember:

- There 3+ solutions to update "sources" documented on the wiki
  somewhere - ideas from comparing versions with other distributions up
  to adding scrapers getting latest version from web sites if I recall
  correctly

- Putting automatically generated code into nixpkgs doesn't solve all
  issues, for corner cases you have to duplicate dependencies using
  different version constraints.

  -> overhead

- Its not always quite clear how stable the user wants to be
  (gimp/inskscape) case, master sometimes has new features.

  So which versions to support ?

- Whatever we do, we don't solve anything for other distros (which
  suffer the same problem), unless we switch point of view:

  The solution would be a cross platform cross language dependency
  management system allowing to declare dependencies in a file so that
  you can even install from gihtub automatically.

  systemPackages = [ (fromGithub "user/package" "HEAD") ] # sort rest out on 
your own, thanks

package A could be working, package B could be working, but [A B] in the
same environment not (because they both depend on executable C)

After all we want nixpkgs to be at least stable enough to have broken
packages marked as broken and expect everything else to at least
compile/install.

Which are the best short/middle/long term solutions ?

Marc Weber
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[lewo]

Freddy Rietdijk  writes:

> Hi,
>
> At several places in Nixpkgs we use auto-generated data, mostly for the
> larger package sets like Haskell. Sometimes we also use auto-generated sets
> for applications that may need different versions than are offered in the
> main package sets. In the past months several issues/PR's have been opened
> to add rather large Python applications to Nixpkgs, generated using
> `pypi2nix`.
>
> Using such tool to generate the expressions has some very big advantages.
> However, there's also a disadvantage. The expressions shouldn't only be
> updated when there's a new release of the application, but also whenever
> there's security updates in any of its generated dependencies, which the
> application maintainer now has to keep track of. Therefore, I find it quite
> a risk to have separate package sets. At the same time, we'll also likely
> run behind in the main package sets every once in a while.

As an example, `openstackclient` needs a version of the `requests`
library which is older than our `pythonPackages.requests`. We then need
to add another version but we don't want to mix different versions in
the python packages set (to avoid conflicts). In this context, it seems
the only way to bring `openstackclient` was to provide its dependencies
as 'private' dependencies.

I think several `golang` applications are also packaged in `nixpkgs` by
"vendorizing" their dependencies.
How dependencies of these applications are managed?

> What do you think of this issue? Any suggestions how we can improve this?
> Maybe we could have a server/bot that runs update scripts and opens a PR
> whenever there's an actual diff?

I don't know how but I think it would be nice to be able to know the
python module versions that are currently used by all applications.

> Freddy
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[Wout Mertens]

This actually ties into my question about nodePackages. It seems to me that
for these large packaging systems, we should have separate repos that
update from their source, and you can then include them into your nixpkgs
configuration.

Only packages that are useful by themselves should get a derivation in
nixpkgs, the rest would be in these add-on repos. For nodejs, that would be
e.g. yarn.

No idea how to accomplish this though.

On Tue, May 30, 2017 at 9:31 AM Freddy Rietdijk 
wrote:

> Hi,
>
> At several places in Nixpkgs we use auto-generated data, mostly for the
> larger package sets like Haskell. Sometimes we also use auto-generated sets
> for applications that may need different versions than are offered in the
> main package sets. In the past months several issues/PR's have been opened
> to add rather large Python applications to Nixpkgs, generated using
> `pypi2nix`.
>
> Using such tool to generate the expressions has some very big advantages.
> However, there's also a disadvantage. The expressions shouldn't only be
> updated when there's a new release of the application, but also whenever
> there's security updates in any of its generated dependencies, which the
> application maintainer now has to keep track of. Therefore, I find it quite
> a risk to have separate package sets. At the same time, we'll also likely
> run behind in the main package sets every once in a while.
>
> What do you think of this issue? Any suggestions how we can improve this?
> Maybe we could have a server/bot that runs update scripts and opens a PR
> whenever there's an actual diff?
>
> Freddy
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Auto-generated expressions for applications

[Freddy Rietdijk]

Hi,

At several places in Nixpkgs we use auto-generated data, mostly for the
larger package sets like Haskell. Sometimes we also use auto-generated sets
for applications that may need different versions than are offered in the
main package sets. In the past months several issues/PR's have been opened
to add rather large Python applications to Nixpkgs, generated using
`pypi2nix`.

Using such tool to generate the expressions has some very big advantages.
However, there's also a disadvantage. The expressions shouldn't only be
updated when there's a new release of the application, but also whenever
there's security updates in any of its generated dependencies, which the
application maintainer now has to keep track of. Therefore, I find it quite
a risk to have separate package sets. At the same time, we'll also likely
run behind in the main package sets every once in a while.

What do you think of this issue? Any suggestions how we can improve this?
Maybe we could have a server/bot that runs update scripts and opens a PR
whenever there's an actual diff?

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev