Re: [Nix-dev] Simplify node packages?
On Tue, Apr 25, 2017 at 3:21 AM, Sander van der Burg wrote: Now yarn (and other approaches such as ied) do obviously a much better job > than NPM in some aspects. Unfortunately, as of today, NPM remains the > de-facto standard for Node.js package deployment, and I'm not sure if this > will ever change. > It's changing now and quite quickly. A google search for "yarn add" on npmjs.com yields about 5K results[2]. Here's one random example[2] not affiliated with Facebook . The installation instructions tell people to use yarn. Or try searching for "yarn.lock" on Github[3. There are 128K results. There are a few reasons to expect this to continue: - Yarn is developed and supported by Facebook. As the developers of React, Facebook plays a huge role in the Node community and yarn will get a lot of adoption just because it's the tool "blessed" by Facebook. Further, Facebook has the resources and incentive to push yard forward quickly. - NPM has been foundering for a while, and Yarn is just better. It's faster and more reliable and easy to adopt. - At some point, NPM Inc. will embrace yarn in order to keep control of the package registry, which is where they make their money. This shift will take time, of course, and most developers are still using npm for now. But it won't be long before "I can't install your package with yarn" is a show-stopper bug that destroys the viability of a node project. Ideally, whenever an NPM project works, node2nix should work as well. And this is why the node-community shift towards yarn matters to us nixers. Increasingly, node developers are going to care more about whether it works yarn, not npm, and projects are going to built with and tested against yarn. So if we want node packages to "just work" with nix, basing it on yarn's behaviour will be the easiest way to do that. Colin [1] https://encrypted.google.com/search?hl=en&as_q=&as_epq=yarn+add&as_oq=&as_eq=&as_nlo=&as_nhi=&lr=&cr=&as_qdr=all&as_sitesearch=npmjs.com&as_occt=any&safe=images&as_filetype=&as_rights=#q=%22yarn+add%22+site:npmjs.com&hl=en&as_qdr=all&start=20 [2] https://www.npmjs.com/package/miku [3] https://github.com/search?q=yarn.lock&type=Code&utf8=%E2%9C%93 ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
On Tue, Apr 25, 2017 at 12:51 AM, Profpatsch wrote: > yarn is really just a 1:1 rewrite of node with lockfiles. > I don’t know why people celebrate it so much, > because it does exactly nothing differently apart from > creating the lockfiles. > Don’t even know why a rewrite was necessary. > There are 2 improvements: - lock file - caching of downloads These address two huge pain points in the node world, not coincidentally the ones that nix also fixes. Colin ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
On 17-04-25 12:21pm, Sander van der Burg wrote: > I'm not sure if you have read my blog articles correctly, but what node2nix > currently does is one derivation per application and statically bundles all > NPM dependencies with it. The old approach that npm2nix used to implement > was a derivation per dependency and symlinking between them. As explained > in this blog post: > http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html > this works for many packages, but does not reliably work in all cases. > Moreover, it does not work with packages having cyclic dependencies. Apart from cyclic dependencies and leaving out “flattened” trees, are there any other deal-breakers you can think of? > Now yarn (and other approaches such as ied) do obviously a much better job > than NPM in some aspects. Unfortunately, as of today, NPM remains the > de-facto standard for Node.js package deployment, and I'm not sure if this > will ever change. yarn is pretty much a drop-in replacement for npm. Instead of `npm install && node2nix` you say `yarn install && yarn2nix` and you get a local `node_modules` and in yarn’s case also a `yarn.lock`. So as far as packaging goes, adoption should not matter imho. > The latter aspect can only be solved if NPM makes their dependency > addressing and composition facilities better. +1 -- Proudly written in Mutt with Vim on NixOS. Q: Why is this email five sentences or less? A: http://five.sentenc.es May take up to five days to read your message. If it’s urgent, call me. ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
We (IOHK) are going to use yarn2nix in the future as well. Maybe we could have a call where things stand and how to proceed? On Tue, Apr 25, 2017 at 12:11 PM, Maarten Hoogendoorn wrote: > I've been working with zimbatim and manveru on a yarn2nix as well, see > https://github.com/moretea/yarn2nix > > I'm not actively using it myself though. I would be interested in planning > a few hours to hack on this together ;) > > 2017-04-25 10:37 GMT+02:00 Profpatsch : > >> On 17-04-25 08:20am, Benno Fünfstück wrote: >> > If we get upstream to support enough for our use case, the solution >> should >> > be much more stable. >> >> Upstream support might be helpful, >> but that’s a wholly different beast. >> >> > Hmm, so perhaps in we can unpack the tarballs already in `phase 1` and >> tell >> > yarn to just look in $DIR for unpacked tarballs? Yarn should already >> have >> > most of the code that is required for correct symlinking, no? >> >> The code for correct symlinking is <4 lines, >> maybe a few more for manpages and binaries. >> >> I’d argue integrating it into yarn is even more frickle: >> 1. With yarn there’s now two implementations of npm packages. >> 2. We have to invest into lobbying so our integration >>isn’t broken by upstream sometimes. Then lobby again. >> 3. You’d definitely have to change yarn to have >>an offline mode where it doesn’t try to download stuff. >>I’m pretty sure upstream doesn’t care about that very much. >> >> These are the problem points I see. >> >> -- >> Proudly written in Mutt with Vim on NixOS. >> Q: Why is this email five sentences or less? >> A: http://five.sentenc.es >> May take up to five days to read your message. If it’s urgent, call me. >> ___ >> nix-dev mailing list >> nix-dev@lists.science.uu.nl >> https://mailman.science.uu.nl/mailman/listinfo/nix-dev >> > > > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > https://mailman.science.uu.nl/mailman/listinfo/nix-dev > > ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
> > I have read the articles from Sander van der Burg, but they don't seem to > explain why we went with "one derivation per library", which requires us to > manually symlink things into the node_modules folder. > I'm not sure if you have read my blog articles correctly, but what node2nix currently does is one derivation per application and statically bundles all NPM dependencies with it. The old approach that npm2nix used to implement was a derivation per dependency and symlinking between them. As explained in this blog post: http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html this works for many packages, but does not reliably work in all cases. Moreover, it does not work with packages having cyclic dependencies. * A simple approach to packaging NPM applications would be to split the > package for an NPM application into two: > - fetch: let NPM download all the dependencies. it should(?) be possible > to make this a fixed-output derivation, as long as NPM downloads the same > set of dependencies each time > - build: use the cache from the fetch phase, should require no network > access > This might be an alternative approach worth investigating. However, dealing with NPM's conflicting dependency management is not that simple -- for example, there are version specifiers always triggering network communication (such as version specifiers referring to tags or HTTP URLs), regardless whether a version has already been provided or not. Furthermore, there are dependencies that bundle C code with them requiring them to be source compiled e.g. with gcc. Running NPM outside a Nix builder environment first and "importing" them into Nix through a fixed-output derivation does not reliably work with such packages. Well, perhaps that's the algorithm it uses right now, but I feel that we >> depend very much on implementation details of npm here, which IMO is not a >> good solution. >> If we get upstream to support enough for our use case, the solution >> should be much more stable. >> > It would obviously be great if we could convince the NPM developers to change/reconsider some of their "design choices", but I think this will be quite difficult. One of the reasons I have given a talk at FOSDEM 2017 about this subject is to explain some of the undesired implications of NPM in the hope that somebody would look at it. However, as with many open source projects, it is difficult to change someone's mind/opinion. Moreover, even if there are people willing to change, there is still the argument that legacy applications need to be supported. I think this is the best way forward. Yarn is *much* better than npm and is > being enthusiastically adopted by the node community. Whatever > compatibility issues that currently exist with yarn will soon be resolved. > (Nix users have basically no influence, but package maintainers can't > afford to ignore yarn users.) > Now yarn (and other approaches such as ied) do obviously a much better job than NPM in some aspects. Unfortunately, as of today, NPM remains the de-facto standard for Node.js package deployment, and I'm not sure if this will ever change. The purpose of node2nix is not to be "the better NPM" -- it's only objective is to integrate NPM packages and development projects into the Nix ecosystem with minimal disruptions. Ideally, whenever an NPM project works, node2nix should work as well. (Obviously: this experience is not always perfect, but we try as hard as we can). Finally, I'd like to note that the some of the concepts that NPM (and in some extent yarn) implement are fundamentally different and somewhat conflicting with how Nix does things. For example, what we ideally like to see is that dependencies can be shared. For example, when a project depends on lodash-4.17.4 we ideally only want one copy of it in the store that all dependent projects can use. Unfortunately, the way NPM composes packages is context-dependent (e.g. in some scenario's lodash's dependencies resolve to other versions depending on which common dependencies are already used in the project). As a result, sharing NPM package dependencies among projects/application will not work. Because this is an NPM/CommonJS limitation, it is a limitation in Nix as well. The latter aspect can only be solved if NPM makes their dependency addressing and composition facilities better. On Tue, Apr 25, 2017 at 10:37 AM, Profpatsch wrote: > On 17-04-25 08:20am, Benno Fünfstück wrote: > > If we get upstream to support enough for our use case, the solution > should > > be much more stable. > > Upstream support might be helpful, > but that’s a wholly different beast. > > > Hmm, so perhaps in we can unpack the tarballs already in `phase 1` and > tell > > yarn to just look in $DIR for unpacked tarballs? Yarn should already have > > most of the code that is required for correct symlinking, no? > > The code for correct symlinking is <4 lines, > maybe a few more for manpages and b
Re: [Nix-dev] Simplify node packages?
I've been working with zimbatim and manveru on a yarn2nix as well, see https://github.com/moretea/yarn2nix I'm not actively using it myself though. I would be interested in planning a few hours to hack on this together ;) 2017-04-25 10:37 GMT+02:00 Profpatsch : > On 17-04-25 08:20am, Benno Fünfstück wrote: > > If we get upstream to support enough for our use case, the solution > should > > be much more stable. > > Upstream support might be helpful, > but that’s a wholly different beast. > > > Hmm, so perhaps in we can unpack the tarballs already in `phase 1` and > tell > > yarn to just look in $DIR for unpacked tarballs? Yarn should already have > > most of the code that is required for correct symlinking, no? > > The code for correct symlinking is <4 lines, > maybe a few more for manpages and binaries. > > I’d argue integrating it into yarn is even more frickle: > 1. With yarn there’s now two implementations of npm packages. > 2. We have to invest into lobbying so our integration >isn’t broken by upstream sometimes. Then lobby again. > 3. You’d definitely have to change yarn to have >an offline mode where it doesn’t try to download stuff. >I’m pretty sure upstream doesn’t care about that very much. > > These are the problem points I see. > > -- > Proudly written in Mutt with Vim on NixOS. > Q: Why is this email five sentences or less? > A: http://five.sentenc.es > May take up to five days to read your message. If it’s urgent, call me. > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > https://mailman.science.uu.nl/mailman/listinfo/nix-dev > ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
On 17-04-25 08:20am, Benno Fünfstück wrote: > If we get upstream to support enough for our use case, the solution should > be much more stable. Upstream support might be helpful, but that’s a wholly different beast. > Hmm, so perhaps in we can unpack the tarballs already in `phase 1` and tell > yarn to just look in $DIR for unpacked tarballs? Yarn should already have > most of the code that is required for correct symlinking, no? The code for correct symlinking is <4 lines, maybe a few more for manpages and binaries. I’d argue integrating it into yarn is even more frickle: 1. With yarn there’s now two implementations of npm packages. 2. We have to invest into lobbying so our integration isn’t broken by upstream sometimes. Then lobby again. 3. You’d definitely have to change yarn to have an offline mode where it doesn’t try to download stuff. I’m pretty sure upstream doesn’t care about that very much. These are the problem points I see. -- Proudly written in Mutt with Vim on NixOS. Q: Why is this email five sentences or less? A: http://five.sentenc.es May take up to five days to read your message. If it’s urgent, call me. ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
> > > This would only need a feature from upstream to use cached tarballs. > > npm uses the following algorithm: > 1. resolve dependencies > 2. move dependencies as far up the tree as possible to enable sharing > 3. unpack the tarballs into the correct dirs > 4. link the "bin" files into a .bin directory and the man files > > That’s it. > Well, perhaps that's the algorithm it uses right now, but I feel that we depend very much on implementation details of npm here, which IMO is not a good solution. If we get upstream to support enough for our use case, the solution should be much more stable. I would like nixpkgs to move away from the approach of "mocking" things for stupid package managers (creating files in certain directories) and instead try to work with upstream to just implement the features we need. I'd even volunteer to work on this myself, but I thought I'd ask if there are any principal problems here first. > > phase 1: build a directory full of tarballs (just symlink the result of > > fetchurl without unpacking) > > phase 2: let yarn assemble the dependency tree (thus it will deal with > > cyclic dependencies for us) > > Right now it takes around 4–5 minutes to unpack the tarballs > for my project with node2nix. That is on every dep change. > Hmm, so perhaps in we can unpack the tarballs already in `phase 1` and tell yarn to just look in $DIR for unpacked tarballs? Yarn should already have most of the code that is required for correct symlinking, no? ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
On 17-04-25 07:26am, Benno Fünfstück wrote: > > > > The most interesting property is, that all dependency trees > > are linked together with symlinked, so they are cached by nix > > on a package level: > > https://github.com/Profpatsch/yarn2nix/blob/master/buildNodePackage.nix#L33 > > > Does this caching really gain us that much? The disadvantage of doing this > is that we have to maintain our own scripts for doing it, since upstream > does not support it. Depends by who you mean by upstream. yarn: only the parser could break, but that’s easily fixable npm: they could change something in how packages work, but that’s unlikely. > phase 1: build a directory full of tarballs (just symlink the result of > fetchurl without unpacking) > phase 2: let yarn assemble the dependency tree (thus it will deal with > cyclic dependencies for us) Right now it takes around 4–5 minutes to unpack the tarballs for my project with node2nix. That is on every dep change. > This would only need a feature from upstream to use cached tarballs. npm uses the following algorithm: 1. resolve dependencies 2. move dependencies as far up the tree as possible to enable sharing 3. unpack the tarballs into the correct dirs 4. link the "bin" files into a .bin directory and the man files That’s it. We can skip step 2, because we don’t copy, but symlink already extracted packages (and filename length is not a problem on modern ext/btrfs/zfs) Step 4 becomes trivial once we skip 2. yarn is really just a 1:1 rewrite of node with lockfiles. I don’t know why people celebrate it so much, because it does exactly nothing differently apart from creating the lockfiles. Don’t even know why a rewrite was necessary. Maybe I missed something, but that’s the plan. -- Proudly written in Mutt with Vim on NixOS. May take up to five days to read your message. If it’s urgent, call me. ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
> > The most interesting property is, that all dependency trees > are linked together with symlinked, so they are cached by nix > on a package level: > https://github.com/Profpatsch/yarn2nix/blob/master/buildNodePackage.nix#L33 Does this caching really gain us that much? The disadvantage of doing this is that we have to maintain our own scripts for doing it, since upstream does not support it. Perhaps a better way would be to work with upstream to allow the following interface: phase 1: build a directory full of tarballs (just symlink the result of fetchurl without unpacking) phase 2: let yarn assemble the dependency tree (thus it will deal with cyclic dependencies for us) This would only need a feature from upstream to use cached tarballs. ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
On 17-04-23 01:10pm, Benno Fünfstück wrote: > * A simple approach to packaging NPM applications would be to split the > package for an NPM application into two: > - fetch: let NPM download all the dependencies. it should(?) be possible > to make this a fixed-output derivation, as long as NPM downloads the same > set of dependencies each time > - build: use the cache from the fetch phase, should require no network > access > > The fetch phase should be deterministic if we freeze the NPM index (is such > a thing possible?) and use shrinkwrap. > Or is this not the case? In what cases will it fail to be deterministic? > > * Alternatively, `yarn` claims to be more deterministic. Can we use `yarn` > to build all NPM packages? Or are there NPM packages that are incompatible? I’m in the process of writing yarn2nix. After resolving dependencies yarn writes a lockfile, example: https://github.com/facebook/react/blob/master/yarn.lock As you can observe this has everything we need (even the hash!) Sadly it is not a known format, so I wrote a simple-stupid parser: https://hackage.haskell.org/package/yarn-lock-0.1.0/docs/Yarn-Lock.html The pre-alpha-version of converting yarn.lock files to nix expressions can be found here: https://github.com/Profpatsch/yarn2nix nix-shell $ hpack $ cabal build $ ./dist/build/yarn2nix/yarn2nix should do the trick. It works for simple yarn.lock files, but the generated output cannot be nicely checked in yet, since it’s way too verbose. I need to change the internal data representation a bit, to a multi-keyed map. When trying to build our production (ember) app I stumbled on another major problem: npm package dependencies and yarn.lock files have cycles. No sane package manager would allow cycles, but here we are. So a cycle-breaking function is still needed, not very much work but not trivial either. The most interesting property is, that all dependency trees are linked together with symlinked, so they are cached by nix on a package level: https://github.com/Profpatsch/yarn2nix/blob/master/buildNodePackage.nix#L33 I expect yarn2nix to be ready in a few days, most certainly in the next two weeks. -- Proudly written in Mutt with Vim on NixOS. May take up to five days to read your message. If it’s urgent, call me. ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
On Sun, Apr 23, 2017 at 6:10 AM, Benno Fünfstück < benno.fuenfstu...@gmail.com> wrote: > * Alternatively, `yarn` claims to be more deterministic. Can we use `yarn` > to build all NPM packages? Or are there NPM packages that are incompatible? > I think this is the best way forward. Yarn is *much* better than npm and is being enthusiastically adopted by the node community. Whatever compatibility issues that currently exist with yarn will soon be resolved. (Nix users have basically no influence, but package maintainers can't afford to ignore yarn users.) Node2nix focusses on replicating the typical npm workflow so that "normal" node developers experience minimal disruption. I think that's a worthy goal, but it should be balanced against the need to fully exploit the benefits of nix. For example, it seems to be impossible to override an npm derivation to add a native dependency. That to, to me, is one of key features of nix, but at the moment we can't use it. As yarn fixes more and more of the more pedestrian issue with npm, integration with the nix ecosystem and will become the only real advantage to using nix with node. -Colin ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
William Casarin schrieb am So., 23. Apr. 2017 um 16:46 Uhr: > > The fetch phase should be deterministic if we freeze the NPM index (is > such > > a thing possible?) and use shrinkwrap. > > Or is this not the case? In what cases will it fail to be deterministic? > > This is effectively what node2nix[1] does, which is the most reliable > mechanism for deterministic node packages. It looks like this: > This is not what node2nix does. Node2nix completely replaces `npm`'s dependency resolution, therefore it is very dependent on the implementation of npm. If instead, we would let npm do the resolution but make it do it determistically by letting it resolve against the same npm index state we wouldn't have that problem. Cheers, Benno ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Simplify node packages?
Benno Fünfstück writes: > * A simple approach to packaging NPM applications would be to split the > package for an NPM application into two: > - fetch: let NPM download all the dependencies. it should(?) be possible > to make this a fixed-output derivation, as long as NPM downloads the same > set of dependencies each time AFAIK npm isn't deterministic, so this wouldn't be a good idea. > The fetch phase should be deterministic if we freeze the NPM index (is such > a thing possible?) and use shrinkwrap. > Or is this not the case? In what cases will it fail to be deterministic? This is effectively what node2nix[1] does, which is the most reliable mechanism for deterministic node packages. It looks like this: https://github.com/jb55/npm-repo-proxy/blob/81182f25cb783a986d7b7ee4a63f0ca6ca9c8989/node-packages.nix and then I use it like so: https://github.com/jb55/nix-files/blob/1c33031e5678c732dad3f0c76131a4575b7e7bd4/machines/charon/default.nix#L11-L16 It has been working great for my node servers. I haven't had much luck with the nixpkgs nodejs machinery. Cheers, [1] https://github.com/svanderburg/node2nix -- https://jb55.com ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Simplify node packages?
Hello, the nodejs infrastructure in nixpkgs looks quite complex to me. I'm sure that there are good reasons for this. In particular, can someone answer the following questions, as I am not intimately familar with the full NPM ecosystem: Both of the following approaches change the concept from "one derivation per npm library" to "one or two derivations per npm *application/tool*" (libraries are build together with the application, in the same derivation). * A simple approach to packaging NPM applications would be to split the package for an NPM application into two: - fetch: let NPM download all the dependencies. it should(?) be possible to make this a fixed-output derivation, as long as NPM downloads the same set of dependencies each time - build: use the cache from the fetch phase, should require no network access The fetch phase should be deterministic if we freeze the NPM index (is such a thing possible?) and use shrinkwrap. Or is this not the case? In what cases will it fail to be deterministic? * Alternatively, `yarn` claims to be more deterministic. Can we use `yarn` to build all NPM packages? Or are there NPM packages that are incompatible? I have read the articles from Sander van der Burg, but they don't seem to explain why we went with "one derivation per library", which requires us to manually symlink things into the node_modules folder. Kind regards, Benno ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev