Re: [Nix-dev] How do you work on big packages?

[Will Dietz]

Just to add to this, something I haven't seen neatly documented but I
personally find very helpful (if a bit rough) is to run "genericBuild"
from within nix-shell, which will run the phases from the
derivation... and usually fail to install during installPhase.  I
often start this and then just interrupt it during the build so I can
poke at the source or config.log or whatever.

Anyway, it is an easy way to say "just do what nix-build does" for
unknown derivations.  Doesn't let you re-use the bits and install into
the nix store, but usually that's much less of an issue--once I have a
thing working I don't mind rebuilding (in fact, it's /good/ to rebuild
from scratch before saying "ship it") one final time.

~Will

PS: nix-shell doesn't set NIX_BUILD_CORES so I often do something like
'NIX_BUILD_CORES=8 genericBuild' (or whatever)

On Fri, Mar 17, 2017 at 2:26 PM, Profpatsch  wrote:
> On 17-03-17 06:04pm, Volth wrote:
>> "nix-shell" would be a super option here if it could handle
>> "installPhase" (this seems easy to fix) and .nix files less trivial
>> than "hello.nix" (this seems not easy to fix; for example "nix-shell
>> '' -A linux_4_4" has no "configurePhase", and there are
>> similar problems with almost every of the big projects; nix-shell
>> launches "make" when "nix-build" launches "cmake" or vice-versa, etc)
>
> That’s a pretty common stumbling block.
>
> If someone defines his own `installPhase` for example,
> the `installPhase` shell function is just the standard
> stdenv `installPhase`. What you want to call is rather
> the contents of `$installPhase` (the variable), since
> that contains the phase you defined in `mkDerivation`.
>
> nix-shell
> $ unpackPhase
> unpacking …
> $ configurePhase
> configuring …
> $ buildPhase
> …
> $ $installPhase
> running your installPhase
>
>
> There is not much abstraction. Every nix attribute within
> `derivation` (and by extension `mkDerivation`) will end up
> as a bash shell variable in your shell (and your build env).
>
> --
> Proudly written in Mutt with Vim on NixOS.
> Q: Why is this email five sentences or less?
> A: http://five.sentenc.es
> May take up to five days to read your message. If it’s urgent, call me.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[Dmitry Kalinkin]

On Tue, Mar 21, 2017 at 10:01 AM, Kosyrev Serge  wrote:
>
> Dmitry Kalinkin writes:
>>  On 17 Mar 2017, at 17:44, Vladimír Čunát  wrote:
>>
>>  I do believe the intention was for "SW distribution" etc, at least
>>  primarily, and the suitability for development is a by-product due to
>>  some properties, e.g. easy (non-)mixing of development and stable
>>  versions/configs. Marc can surely remember the earlier days of NixOS.
>>
>> So nix has “nix copy” functionality to do the distribution part. But the 
>> rest of the nix system allows to describe a
>> relatively general computation process in Unix-like environment. I use nix 
>> derivations to run a numerical
>> calculation code and store intermediate steps in outputs. I imagine, there 
>> are some “SW distribution”-oriented
>> users facing more resistance when using nix than me doing my thing. I also 
>> think that the nix/NixOS community
>> will have lots of people who appreciate generality of nix as a tool.
>>
>>  It's even possible to use nix-build instead of make to compile
>>  individual files, but there it just doesn't seem to be very suitable…
>>
>> Yes. Like https://github.com/edolstra/nix-make . I wonder what didn’t work 
>> out.
>
> Garbage management becomes complicated from a human perspective, I guess.
>
> Perhaps, Nix could adopt a generational GC approach, to avoid mixing
> long-lived packages-related-files and short-lived build-related files..
>
Yes. I think a good way to address this would be to have nix support
external stores. That way nix-make can store its artefacts separately.
Also this would allow to store data files on a separate volume
optimised for big files.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[Kosyrev Serge]

Dmitry Kalinkin writes:
>  On 17 Mar 2017, at 17:44, Vladimír Čunát  wrote:
>
>  I do believe the intention was for "SW distribution" etc, at least
>  primarily, and the suitability for development is a by-product due to
>  some properties, e.g. easy (non-)mixing of development and stable
>  versions/configs. Marc can surely remember the earlier days of NixOS.
>
> So nix has “nix copy” functionality to do the distribution part. But the rest 
> of the nix system allows to describe a
> relatively general computation process in Unix-like environment. I use nix 
> derivations to run a numerical
> calculation code and store intermediate steps in outputs. I imagine, there 
> are some “SW distribution”-oriented
> users facing more resistance when using nix than me doing my thing. I also 
> think that the nix/NixOS community
> will have lots of people who appreciate generality of nix as a tool.
>
>  It's even possible to use nix-build instead of make to compile
>  individual files, but there it just doesn't seem to be very suitable…
>
> Yes. Like https://github.com/edolstra/nix-make . I wonder what didn’t work 
> out.

Garbage management becomes complicated from a human perspective, I guess.

Perhaps, Nix could adopt a generational GC approach, to avoid mixing
long-lived packages-related-files and short-lived build-related files..

-- 
с уважениeм / respectfully,
Косырев Сергей
--
“Most deadly errors arise from obsolete assumptions.”
  -- Frank Herbert, Children of Dune
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[zimbatm]

Another option is to add a powerful remote builder on the local network. It
doesn't beat incremental rebuilds but helps with having the laptop taking
off and trying to burn my legs. Old desktop machines from ebay are not too
expensive and still beat the laptop capacity.

On Sat, 18 Mar 2017 at 02:19 Tuomas Tynkkynen 
wrote:

> A while ago I wrote a wrapper around nix-shell that helps with running
> the build steps in correct manner and order (among with other niceties
> like automatically creating a temporary build directory and making
> $out point to a path in /tmp/):
> https://github.com/dezgeg/nix-debug-shell
>
> I mainly wrote it for debugging build steps for existing packages, but
> I guess the same ideas/code could be used for local development as
> well.
>
> It is rigorously undocumented, so here's a short primer: in a nixpkgs
> tree, run /path/to/nix-debug-shell -A hello. That places you under
> /tmp/nds-build-hello/hello-2.10 with the sources unpacked. Run
> 'nd-step' as many times you want to step through the build phases
> (it's named after the 'step' command in GDB). When installPhase is
> done, it installs to /tmp/, e.g. /tmp/nds-install-hello/bin/hello.
>
> Hope anyone finds this useful.
>
> 2017-03-17 21:26 GMT+02:00 Profpatsch :
> > On 17-03-17 06:04pm, Volth wrote:
> >> "nix-shell" would be a super option here if it could handle
> >> "installPhase" (this seems easy to fix) and .nix files less trivial
> >> than "hello.nix" (this seems not easy to fix; for example "nix-shell
> >> '' -A linux_4_4" has no "configurePhase", and there are
> >> similar problems with almost every of the big projects; nix-shell
> >> launches "make" when "nix-build" launches "cmake" or vice-versa, etc)
> >
> > That’s a pretty common stumbling block.
> >
> > If someone defines his own `installPhase` for example,
> > the `installPhase` shell function is just the standard
> > stdenv `installPhase`. What you want to call is rather
> > the contents of `$installPhase` (the variable), since
> > that contains the phase you defined in `mkDerivation`.
> >
> > nix-shell
> > $ unpackPhase
> > unpacking …
> > $ configurePhase
> > configuring …
> > $ buildPhase
> > …
> > $ $installPhase
> > running your installPhase
> >
> >
> > There is not much abstraction. Every nix attribute within
> > `derivation` (and by extension `mkDerivation`) will end up
> > as a bash shell variable in your shell (and your build env).
> >
> > --
> > Proudly written in Mutt with Vim on NixOS.
> > Q: Why is this email five sentences or less?
> > A: http://five.sentenc.es
> > May take up to five days to read your message. If it’s urgent, call me.
> > ___
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[Tuomas Tynkkynen]

A while ago I wrote a wrapper around nix-shell that helps with running
the build steps in correct manner and order (among with other niceties
like automatically creating a temporary build directory and making
$out point to a path in /tmp/):
https://github.com/dezgeg/nix-debug-shell

I mainly wrote it for debugging build steps for existing packages, but
I guess the same ideas/code could be used for local development as
well.

It is rigorously undocumented, so here's a short primer: in a nixpkgs
tree, run /path/to/nix-debug-shell -A hello. That places you under
/tmp/nds-build-hello/hello-2.10 with the sources unpacked. Run
'nd-step' as many times you want to step through the build phases
(it's named after the 'step' command in GDB). When installPhase is
done, it installs to /tmp/, e.g. /tmp/nds-install-hello/bin/hello.

Hope anyone finds this useful.

2017-03-17 21:26 GMT+02:00 Profpatsch :
> On 17-03-17 06:04pm, Volth wrote:
>> "nix-shell" would be a super option here if it could handle
>> "installPhase" (this seems easy to fix) and .nix files less trivial
>> than "hello.nix" (this seems not easy to fix; for example "nix-shell
>> '' -A linux_4_4" has no "configurePhase", and there are
>> similar problems with almost every of the big projects; nix-shell
>> launches "make" when "nix-build" launches "cmake" or vice-versa, etc)
>
> That’s a pretty common stumbling block.
>
> If someone defines his own `installPhase` for example,
> the `installPhase` shell function is just the standard
> stdenv `installPhase`. What you want to call is rather
> the contents of `$installPhase` (the variable), since
> that contains the phase you defined in `mkDerivation`.
>
> nix-shell
> $ unpackPhase
> unpacking …
> $ configurePhase
> configuring …
> $ buildPhase
> …
> $ $installPhase
> running your installPhase
>
>
> There is not much abstraction. Every nix attribute within
> `derivation` (and by extension `mkDerivation`) will end up
> as a bash shell variable in your shell (and your build env).
>
> --
> Proudly written in Mutt with Vim on NixOS.
> Q: Why is this email five sentences or less?
> A: http://five.sentenc.es
> May take up to five days to read your message. If it’s urgent, call me.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[Volth]

nix-shell on its installPhase has no rights to write to /nix/store,
installPhase always fails.


I am thinking about something like converting a derivation (a .drv
file) into two files:

1. a .sh file, which would do unpackPhase, patchPhase, configurePhase,
buildPhase using the current directory, not something in /tmp.
nix-shell fits very well here.
Ideally, this script just launches nix-shell to perform these phases.

2. a .nix file with src= pointing to the build directory made by the
previous script and performs buildPhase, checkPhase, installPhase,
fixupPhase. this .nix file could be used substitute a .nix file in
all-packages.nix during troubleshooting.

So buildPhase is performed twice (we assume it is to be idempotent,
isn't it?), second time quickly because all .o files are already
compiled. And it would automatically recompile only changed sources.

In practice, it is not so beautiful. For example, kernel derivation
sets environment variables in preUnpack then uses in buildPhase so it
would be a trouble to simply skip all the phases before buildPhase.

On 3/17/17, Profpatsch  wrote:
> On 17-03-17 06:04pm, Volth wrote:
>> "nix-shell" would be a super option here if it could handle
>> "installPhase" (this seems easy to fix) and .nix files less trivial
>> than "hello.nix" (this seems not easy to fix; for example "nix-shell
>> '' -A linux_4_4" has no "configurePhase", and there are
>> similar problems with almost every of the big projects; nix-shell
>> launches "make" when "nix-build" launches "cmake" or vice-versa, etc)
>
> That’s a pretty common stumbling block.
>
> If someone defines his own `installPhase` for example,
> the `installPhase` shell function is just the standard
> stdenv `installPhase`. What you want to call is rather
> the contents of `$installPhase` (the variable), since
> that contains the phase you defined in `mkDerivation`.
>
> nix-shell
> $ unpackPhase
> unpacking …
> $ configurePhase
> configuring …
> $ buildPhase
> …
> $ $installPhase
> running your installPhase
>
>
> There is not much abstraction. Every nix attribute within
> `derivation` (and by extension `mkDerivation`) will end up
> as a bash shell variable in your shell (and your build env).
>
> --
> Proudly written in Mutt with Vim on NixOS.
> Q: Why is this email five sentences or less?
> A: http://five.sentenc.es
> May take up to five days to read your message. If it’s urgent, call me.
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[Profpatsch]

On 17-03-17 05:34pm, Dmitry Kalinkin wrote:
> Also it is funny how your statement is followed by some good advice on how to
> turn nix into even better dev platform. I will only add that one could also
> use ccache to speedup builds:
> 
> my_cool_package.override { stdenv = pkgs.ccacheStdenv; }

Caches will burn you.
You should never mix non-deterministic caching with deployment.
There’s just no way it will work out in all cases.

-- 
Proudly written in Mutt with Vim on NixOS.
Q: Why is this email five sentences or less?
A: http://five.sentenc.es
May take up to five days to read your message. If it’s urgent, call me.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[Volth]

What I meant in the original question is more about troubleshooting
than development.
That development is incorporated in the deployment chain all other
steps are already powered by nix: build a (patched) kernel, then build
system closures with it, then test-driver scripts, then deploy/run
them...


On 3/17/17, Vladimír Čunát  wrote:
> On 03/17/2017 10:34 PM, Dmitry Kalinkin wrote:
>> Also it is funny how your statement is followed by some good advice
>> on how to turn nix into even better dev platform. I will only add
>> that one could also use ccache to speedup builds:
>
> I do believe the intention was for "SW distribution" etc, at least
> primarily, and the suitability for development is a by-product due to
> some properties, e.g. easy (non-)mixing of development and stable
> versions/configs.  Marc can surely remember the earlier days of NixOS.
>
> It's even possible to use nix-build instead of make to compile
> individual files, but there it just doesn't seem to be very suitable...
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[Vladimír Čunát]

On 03/17/2017 07:04 PM, Volth wrote:
> "nix-shell" would be a super option here if it could handle
> "installPhase"

For my paid work (upstream) I'm perfectly comfortable with nix-shell on
an expression that is slightly modified from the nixpkgs version.  I
solve the installing just by specifying prefix to ./install, so `make
install` does what I need.

Almost all of the modifications are boring - just for easy switching of
various options, such as compiler flag sets, gcc/clang, CI tests, etc.
Maybe I'm just lucky with the packages I worked with, and they're no big
beasts like kernel or chromium, too :-)

--Vladimir




smime.p7s
Description: S/MIME Cryptographic Signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[Marc Weber]

Nix is 'software distribution platform', not a dev platform.

Depending on the task other solutions might be better.

For instance  restarting nginx/ apache the nixos way is also slower than
edit/restart cycle on 'debian'. So use the right tool / env for your
job.

There was a hack to impurely rewrite some dependencies for security
reasons - forgot about it - so maybe you can force impurity to speed up
your workflow.

In the kernel case hat takes most time will be 'building it', thus you
can try 'get compiled data' from your directory (binary distribution)
like, then use something like nixos build-vm or such (disk caching
reasons - no restart required).

You can also try creating links to your 'dev' directory and switch off
sandoxing and add a 'dummy env vars' which you increment to force
rebuilding.

Marc Weber
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How do you work on big packages?

[Linus Heckemann]

On 17/03/17 16:00, Volth wrote:
> What could be done here?
nix-shell, and the upstream recommended build procedure.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev