Re: [Nix-dev] How do you work on big packages?
Just to add to this, something I haven't seen neatly documented but I personally find very helpful (if a bit rough) is to run "genericBuild" from within nix-shell, which will run the phases from the derivation... and usually fail to install during installPhase. I often start this and then just interrupt it during the build so I can poke at the source or config.log or whatever. Anyway, it is an easy way to say "just do what nix-build does" for unknown derivations. Doesn't let you re-use the bits and install into the nix store, but usually that's much less of an issue--once I have a thing working I don't mind rebuilding (in fact, it's /good/ to rebuild from scratch before saying "ship it") one final time. ~Will PS: nix-shell doesn't set NIX_BUILD_CORES so I often do something like 'NIX_BUILD_CORES=8 genericBuild' (or whatever) On Fri, Mar 17, 2017 at 2:26 PM, Profpatschwrote: > On 17-03-17 06:04pm, Volth wrote: >> "nix-shell" would be a super option here if it could handle >> "installPhase" (this seems easy to fix) and .nix files less trivial >> than "hello.nix" (this seems not easy to fix; for example "nix-shell >> '' -A linux_4_4" has no "configurePhase", and there are >> similar problems with almost every of the big projects; nix-shell >> launches "make" when "nix-build" launches "cmake" or vice-versa, etc) > > That’s a pretty common stumbling block. > > If someone defines his own `installPhase` for example, > the `installPhase` shell function is just the standard > stdenv `installPhase`. What you want to call is rather > the contents of `$installPhase` (the variable), since > that contains the phase you defined in `mkDerivation`. > > nix-shell > $ unpackPhase > unpacking … > $ configurePhase > configuring … > $ buildPhase > … > $ $installPhase > running your installPhase > > > There is not much abstraction. Every nix attribute within > `derivation` (and by extension `mkDerivation`) will end up > as a bash shell variable in your shell (and your build env). > > -- > Proudly written in Mutt with Vim on NixOS. > Q: Why is this email five sentences or less? > A: http://five.sentenc.es > May take up to five days to read your message. If it’s urgent, call me. > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
On Tue, Mar 21, 2017 at 10:01 AM, Kosyrev Sergewrote: > > Dmitry Kalinkin writes: >> On 17 Mar 2017, at 17:44, Vladimír Čunát wrote: >> >> I do believe the intention was for "SW distribution" etc, at least >> primarily, and the suitability for development is a by-product due to >> some properties, e.g. easy (non-)mixing of development and stable >> versions/configs. Marc can surely remember the earlier days of NixOS. >> >> So nix has “nix copy” functionality to do the distribution part. But the >> rest of the nix system allows to describe a >> relatively general computation process in Unix-like environment. I use nix >> derivations to run a numerical >> calculation code and store intermediate steps in outputs. I imagine, there >> are some “SW distribution”-oriented >> users facing more resistance when using nix than me doing my thing. I also >> think that the nix/NixOS community >> will have lots of people who appreciate generality of nix as a tool. >> >> It's even possible to use nix-build instead of make to compile >> individual files, but there it just doesn't seem to be very suitable… >> >> Yes. Like https://github.com/edolstra/nix-make . I wonder what didn’t work >> out. > > Garbage management becomes complicated from a human perspective, I guess. > > Perhaps, Nix could adopt a generational GC approach, to avoid mixing > long-lived packages-related-files and short-lived build-related files.. > Yes. I think a good way to address this would be to have nix support external stores. That way nix-make can store its artefacts separately. Also this would allow to store data files on a separate volume optimised for big files. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
Dmitry Kalinkin writes: > On 17 Mar 2017, at 17:44, Vladimír Čunátwrote: > > I do believe the intention was for "SW distribution" etc, at least > primarily, and the suitability for development is a by-product due to > some properties, e.g. easy (non-)mixing of development and stable > versions/configs. Marc can surely remember the earlier days of NixOS. > > So nix has “nix copy” functionality to do the distribution part. But the rest > of the nix system allows to describe a > relatively general computation process in Unix-like environment. I use nix > derivations to run a numerical > calculation code and store intermediate steps in outputs. I imagine, there > are some “SW distribution”-oriented > users facing more resistance when using nix than me doing my thing. I also > think that the nix/NixOS community > will have lots of people who appreciate generality of nix as a tool. > > It's even possible to use nix-build instead of make to compile > individual files, but there it just doesn't seem to be very suitable… > > Yes. Like https://github.com/edolstra/nix-make . I wonder what didn’t work > out. Garbage management becomes complicated from a human perspective, I guess. Perhaps, Nix could adopt a generational GC approach, to avoid mixing long-lived packages-related-files and short-lived build-related files.. -- с уважениeм / respectfully, Косырев Сергей -- “Most deadly errors arise from obsolete assumptions.” -- Frank Herbert, Children of Dune ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
Another option is to add a powerful remote builder on the local network. It doesn't beat incremental rebuilds but helps with having the laptop taking off and trying to burn my legs. Old desktop machines from ebay are not too expensive and still beat the laptop capacity. On Sat, 18 Mar 2017 at 02:19 Tuomas Tynkkynenwrote: > A while ago I wrote a wrapper around nix-shell that helps with running > the build steps in correct manner and order (among with other niceties > like automatically creating a temporary build directory and making > $out point to a path in /tmp/): > https://github.com/dezgeg/nix-debug-shell > > I mainly wrote it for debugging build steps for existing packages, but > I guess the same ideas/code could be used for local development as > well. > > It is rigorously undocumented, so here's a short primer: in a nixpkgs > tree, run /path/to/nix-debug-shell -A hello. That places you under > /tmp/nds-build-hello/hello-2.10 with the sources unpacked. Run > 'nd-step' as many times you want to step through the build phases > (it's named after the 'step' command in GDB). When installPhase is > done, it installs to /tmp/, e.g. /tmp/nds-install-hello/bin/hello. > > Hope anyone finds this useful. > > 2017-03-17 21:26 GMT+02:00 Profpatsch : > > On 17-03-17 06:04pm, Volth wrote: > >> "nix-shell" would be a super option here if it could handle > >> "installPhase" (this seems easy to fix) and .nix files less trivial > >> than "hello.nix" (this seems not easy to fix; for example "nix-shell > >> '' -A linux_4_4" has no "configurePhase", and there are > >> similar problems with almost every of the big projects; nix-shell > >> launches "make" when "nix-build" launches "cmake" or vice-versa, etc) > > > > That’s a pretty common stumbling block. > > > > If someone defines his own `installPhase` for example, > > the `installPhase` shell function is just the standard > > stdenv `installPhase`. What you want to call is rather > > the contents of `$installPhase` (the variable), since > > that contains the phase you defined in `mkDerivation`. > > > > nix-shell > > $ unpackPhase > > unpacking … > > $ configurePhase > > configuring … > > $ buildPhase > > … > > $ $installPhase > > running your installPhase > > > > > > There is not much abstraction. Every nix attribute within > > `derivation` (and by extension `mkDerivation`) will end up > > as a bash shell variable in your shell (and your build env). > > > > -- > > Proudly written in Mutt with Vim on NixOS. > > Q: Why is this email five sentences or less? > > A: http://five.sentenc.es > > May take up to five days to read your message. If it’s urgent, call me. > > ___ > > nix-dev mailing list > > nix-dev@lists.science.uu.nl > > http://lists.science.uu.nl/mailman/listinfo/nix-dev > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev > ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
A while ago I wrote a wrapper around nix-shell that helps with running the build steps in correct manner and order (among with other niceties like automatically creating a temporary build directory and making $out point to a path in /tmp/): https://github.com/dezgeg/nix-debug-shell I mainly wrote it for debugging build steps for existing packages, but I guess the same ideas/code could be used for local development as well. It is rigorously undocumented, so here's a short primer: in a nixpkgs tree, run /path/to/nix-debug-shell -A hello. That places you under /tmp/nds-build-hello/hello-2.10 with the sources unpacked. Run 'nd-step' as many times you want to step through the build phases (it's named after the 'step' command in GDB). When installPhase is done, it installs to /tmp/, e.g. /tmp/nds-install-hello/bin/hello. Hope anyone finds this useful. 2017-03-17 21:26 GMT+02:00 Profpatsch: > On 17-03-17 06:04pm, Volth wrote: >> "nix-shell" would be a super option here if it could handle >> "installPhase" (this seems easy to fix) and .nix files less trivial >> than "hello.nix" (this seems not easy to fix; for example "nix-shell >> '' -A linux_4_4" has no "configurePhase", and there are >> similar problems with almost every of the big projects; nix-shell >> launches "make" when "nix-build" launches "cmake" or vice-versa, etc) > > That’s a pretty common stumbling block. > > If someone defines his own `installPhase` for example, > the `installPhase` shell function is just the standard > stdenv `installPhase`. What you want to call is rather > the contents of `$installPhase` (the variable), since > that contains the phase you defined in `mkDerivation`. > > nix-shell > $ unpackPhase > unpacking … > $ configurePhase > configuring … > $ buildPhase > … > $ $installPhase > running your installPhase > > > There is not much abstraction. Every nix attribute within > `derivation` (and by extension `mkDerivation`) will end up > as a bash shell variable in your shell (and your build env). > > -- > Proudly written in Mutt with Vim on NixOS. > Q: Why is this email five sentences or less? > A: http://five.sentenc.es > May take up to five days to read your message. If it’s urgent, call me. > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
nix-shell on its installPhase has no rights to write to /nix/store, installPhase always fails. I am thinking about something like converting a derivation (a .drv file) into two files: 1. a .sh file, which would do unpackPhase, patchPhase, configurePhase, buildPhase using the current directory, not something in /tmp. nix-shell fits very well here. Ideally, this script just launches nix-shell to perform these phases. 2. a .nix file with src= pointing to the build directory made by the previous script and performs buildPhase, checkPhase, installPhase, fixupPhase. this .nix file could be used substitute a .nix file in all-packages.nix during troubleshooting. So buildPhase is performed twice (we assume it is to be idempotent, isn't it?), second time quickly because all .o files are already compiled. And it would automatically recompile only changed sources. In practice, it is not so beautiful. For example, kernel derivation sets environment variables in preUnpack then uses in buildPhase so it would be a trouble to simply skip all the phases before buildPhase. On 3/17/17, Profpatschwrote: > On 17-03-17 06:04pm, Volth wrote: >> "nix-shell" would be a super option here if it could handle >> "installPhase" (this seems easy to fix) and .nix files less trivial >> than "hello.nix" (this seems not easy to fix; for example "nix-shell >> '' -A linux_4_4" has no "configurePhase", and there are >> similar problems with almost every of the big projects; nix-shell >> launches "make" when "nix-build" launches "cmake" or vice-versa, etc) > > That’s a pretty common stumbling block. > > If someone defines his own `installPhase` for example, > the `installPhase` shell function is just the standard > stdenv `installPhase`. What you want to call is rather > the contents of `$installPhase` (the variable), since > that contains the phase you defined in `mkDerivation`. > > nix-shell > $ unpackPhase > unpacking … > $ configurePhase > configuring … > $ buildPhase > … > $ $installPhase > running your installPhase > > > There is not much abstraction. Every nix attribute within > `derivation` (and by extension `mkDerivation`) will end up > as a bash shell variable in your shell (and your build env). > > -- > Proudly written in Mutt with Vim on NixOS. > Q: Why is this email five sentences or less? > A: http://five.sentenc.es > May take up to five days to read your message. If it’s urgent, call me. > ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
On 17-03-17 05:34pm, Dmitry Kalinkin wrote: > Also it is funny how your statement is followed by some good advice on how to > turn nix into even better dev platform. I will only add that one could also > use ccache to speedup builds: > > my_cool_package.override { stdenv = pkgs.ccacheStdenv; } Caches will burn you. You should never mix non-deterministic caching with deployment. There’s just no way it will work out in all cases. -- Proudly written in Mutt with Vim on NixOS. Q: Why is this email five sentences or less? A: http://five.sentenc.es May take up to five days to read your message. If it’s urgent, call me. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
What I meant in the original question is more about troubleshooting than development. That development is incorporated in the deployment chain all other steps are already powered by nix: build a (patched) kernel, then build system closures with it, then test-driver scripts, then deploy/run them... On 3/17/17, Vladimír Čunátwrote: > On 03/17/2017 10:34 PM, Dmitry Kalinkin wrote: >> Also it is funny how your statement is followed by some good advice >> on how to turn nix into even better dev platform. I will only add >> that one could also use ccache to speedup builds: > > I do believe the intention was for "SW distribution" etc, at least > primarily, and the suitability for development is a by-product due to > some properties, e.g. easy (non-)mixing of development and stable > versions/configs. Marc can surely remember the earlier days of NixOS. > > It's even possible to use nix-build instead of make to compile > individual files, but there it just doesn't seem to be very suitable... > > ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
On 03/17/2017 07:04 PM, Volth wrote: > "nix-shell" would be a super option here if it could handle > "installPhase" For my paid work (upstream) I'm perfectly comfortable with nix-shell on an expression that is slightly modified from the nixpkgs version. I solve the installing just by specifying prefix to ./install, so `make install` does what I need. Almost all of the modifications are boring - just for easy switching of various options, such as compiler flag sets, gcc/clang, CI tests, etc. Maybe I'm just lucky with the packages I worked with, and they're no big beasts like kernel or chromium, too :-) --Vladimir smime.p7s Description: S/MIME Cryptographic Signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
Nix is 'software distribution platform', not a dev platform. Depending on the task other solutions might be better. For instance restarting nginx/ apache the nixos way is also slower than edit/restart cycle on 'debian'. So use the right tool / env for your job. There was a hack to impurely rewrite some dependencies for security reasons - forgot about it - so maybe you can force impurity to speed up your workflow. In the kernel case hat takes most time will be 'building it', thus you can try 'get compiled data' from your directory (binary distribution) like, then use something like nixos build-vm or such (disk caching reasons - no restart required). You can also try creating links to your 'dev' directory and switch off sandoxing and add a 'dummy env vars' which you increment to force rebuilding. Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How do you work on big packages?
On 17/03/17 16:00, Volth wrote: > What could be done here? nix-shell, and the upstream recommended build procedure. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] How do you work on big packages?
For example, you are working on patch for kernel or chromium or something huge like this. src= in your x.nix points to a local working directory, so every change would result in rebuilding of the deviation and its dependants. Nice so far. The problem is any change implies full rebuild, because .o files are not preserved between the builds, and in case of big projects it takes hours instead of seconds. What could be done here? An nix-build option to forcible preserve tmp directories (--just-keep-and-reuse-tmp-i-know-what-i-am-doing) ? ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev