Re: [Nix-dev] break purity

[Aycan iRiCAN]

This is a clever optimization. Thank you Oliver.

--
aycan

> On 22 Apr 2016, at 21:33, Oliver Charles  wrote:
> 
> It's certainly possible, because I just did it with Haskell ;)
> 
> https://gist.github.com/ocharles/cbd5d7ce63bb570abb86e655f36435ab
> 
>> On Fri, Apr 22, 2016 at 7:16 PM stewart mackenzie  wrote:
>> Yeah, I'm not so sure it's possible because one cannot copy from a
>> precompiled derivation output to a new derivation output, ie copy
>> cross derivation.
>> 
>> As these multiple outputs (outputs = [x y z]) are seen as different
>> derivations, this cannot happen right?
>> 
>> Have you actually managed to make progress beyond the copying part of
>> your example (preConfigure = "cp ${artefacts}/* .'';) in your
>> experimentation?
>> 
>> (I'm still thought experimenting)
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] break purity

[stewart mackenzie]

You will save me hours of waiting in the future, definitely beer worthy!

Much appreciated Oliver!
On 23 Apr 2016 02:33, "Oliver Charles"  wrote:

> It's certainly possible, because I just did it with Haskell ;)
>
> https://gist.github.com/ocharles/cbd5d7ce63bb570abb86e655f36435ab
>
> On Fri, Apr 22, 2016 at 7:16 PM stewart mackenzie 
> wrote:
>
>> Yeah, I'm not so sure it's possible because one cannot copy from a
>> precompiled derivation output to a new derivation output, ie copy
>> cross derivation.
>>
>> As these multiple outputs (outputs = [x y z]) are seen as different
>> derivations, this cannot happen right?
>>
>> Have you actually managed to make progress beyond the copying part of
>> your example (preConfigure = "cp ${artefacts}/* .'';) in your
>> experimentation?
>>
>> (I'm still thought experimenting)
>>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] break purity

[Oliver Charles]

It's certainly possible, because I just did it with Haskell ;)

https://gist.github.com/ocharles/cbd5d7ce63bb570abb86e655f36435ab

On Fri, Apr 22, 2016 at 7:16 PM stewart mackenzie 
wrote:

> Yeah, I'm not so sure it's possible because one cannot copy from a
> precompiled derivation output to a new derivation output, ie copy
> cross derivation.
>
> As these multiple outputs (outputs = [x y z]) are seen as different
> derivations, this cannot happen right?
>
> Have you actually managed to make progress beyond the copying part of
> your example (preConfigure = "cp ${artefacts}/* .'';) in your
> experimentation?
>
> (I'm still thought experimenting)
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] break purity

[stewart mackenzie]

Yeah, I'm not so sure it's possible because one cannot copy from a
precompiled derivation output to a new derivation output, ie copy
cross derivation.

As these multiple outputs (outputs = [x y z]) are seen as different
derivations, this cannot happen right?

Have you actually managed to make progress beyond the copying part of
your example (preConfigure = "cp ${artefacts}/* .'';) in your
experimentation?

(I'm still thought experimenting)
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] break purity

[stewart mackenzie]

Ah, okay, I get your drift, I'll think how to make it succinct and
tidy in the code :-)
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] break purity

[Oliver Charles]

Well you don't tie the knot in the expression itself. It's more that the
pre-existing artefacts are a function input, and a function output:

{ stdenv, gcc, etc, artefacts ? null }:
{
  preConfigure = "cp ${artefacts}/* .'';
  outputs = [ "out" "artefacts" ];
}

Then you would nix-build to produce binaries and an artefacts directory.
Subsequent nix-builds would then use nix-build --arg 'artefacts'
'./results-artefacts'.

Handwaving, but hopefully that helps clarify.
ocharles

On Fri, Apr 22, 2016 at 5:35 PM stewart mackenzie 
wrote:

> Interesting Oliver, though this sounds like it'll throw a recursive
> error. I'll investigate! cheers!
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] break purity

[stewart mackenzie]

Interesting Oliver, though this sounds like it'll throw a recursive
error. I'll investigate! cheers!
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python & installation

[Domen Kožar]

http://nixos.org/nixpkgs/manual/ should answer all your questions regarding
Python :)

On Fri, Apr 22, 2016 at 4:59 PM, Marc Weber  wrote:

> The question I have is which is the recommended way to install "scrapy
> along with dependencies", the following fixes it by making
> propagatedUserEnvPkgs equal to propagatedBuildInputs.. is there another
> way?
>
>   diff --git a/pkgs/development/python-modules/generic/default.nix
> b/pkgs/development/python-modules/generic/default.nix
>   index 1fdbd4f..458d187 100644
>   --- a/pkgs/development/python-modules/generic/default.nix
>   +++ b/pkgs/development/python-modules/generic/default.nix
>   @@ -70,6 +70,8 @@ python.stdenv.mkDerivation (builtins.removeAttrs attrs
> ["disabled" "doCheck"] //
>  # propagate python/setuptools to active setup-hook in nix-shell
>  propagatedBuildInputs = propagatedBuildInputs ++ [ python setuptools
> ];
>
>   +  propagatedUserEnvPkgs = propagatedBuildInputs ++ [ python setuptools
> ];
>   +
>  pythonPath = pythonPath;
>
>  configurePhase = attrs.configurePhase or ''
>
> Then installing scrapy and setting PYTHONPATH or PYTHONUSERBASE seems to
> enough.
>
> Full patch adding the missing packages:
> http://mawercer.de/tmp/scrapy-diff.patch
>
> Is there an automated maintained way to do such python packaging work I
> missed? Sorry for asking such stupid question - I didn't look at python
> for a long time.
>
> Marc Weber
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] break purity

[Oliver Charles]

Something I'm experimenting with here is to actually have these
intermediate artefacts be explicitly captured. I would use multiple outputs
to have both the normal complete binaries, and another output for .o files
and the like. Then you can feed this back in to future builds.

Does that help? Might give you some food for thought...
ocharles

On Thu, 21 Apr 2016, 4:42 p.m. stewart mackenzie, 
wrote:

> Hi,
>
> I've got a bunch of artifacts being built in the
> /tmp/nix-build-component_name.drv-0
>
> now I want to point the build manager, in this case cargo using the
> CARGO_TARGET_DIR env var to put the artifacts into a directory I make
> directly in the /tmp/target_${name}
>
> This way any further compilations will reuse the artifacts and they
> won't need to be rebuilt.
>
> obviously nix doesn't allow me to do this.
>
> Is there a dirty hack to allow creating and writing artifacts directly
> to /tmp/target_${name}?
>
> Why? We're using nix as a replacement for make, one major drawback is:
> every time we change a line of code the entire component + trans deps
> needs to be rebuilt. This takes a long time when transitive deps go
> deep.
>
> /sjm
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Python & installation

[Marc Weber]

The question I have is which is the recommended way to install "scrapy
along with dependencies", the following fixes it by making
propagatedUserEnvPkgs equal to propagatedBuildInputs.. is there another
way?

  diff --git a/pkgs/development/python-modules/generic/default.nix 
b/pkgs/development/python-modules/generic/default.nix
  index 1fdbd4f..458d187 100644
  --- a/pkgs/development/python-modules/generic/default.nix
  +++ b/pkgs/development/python-modules/generic/default.nix
  @@ -70,6 +70,8 @@ python.stdenv.mkDerivation (builtins.removeAttrs attrs 
["disabled" "doCheck"] //
 # propagate python/setuptools to active setup-hook in nix-shell
 propagatedBuildInputs = propagatedBuildInputs ++ [ python setuptools ];
   
  +  propagatedUserEnvPkgs = propagatedBuildInputs ++ [ python setuptools ];
  +
 pythonPath = pythonPath;
   
 configurePhase = attrs.configurePhase or ''

Then installing scrapy and setting PYTHONPATH or PYTHONUSERBASE seems to
enough.

Full patch adding the missing packages:
http://mawercer.de/tmp/scrapy-diff.patch

Is there an automated maintained way to do such python packaging work I
missed? Sorry for asking such stupid question - I didn't look at python
for a long time.

Marc Weber
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Monitoring by default

[Svein Ove Aas]

Thanks for the responses so far!

Let me see...

- I actually agree with Tomas about naming. I know I wrote
"services.monitoring.enable", but I hadn't put a lot of thought into that
sentence; "services.monitoring.prometheus" seems like a better namespace.

- I'd add battery life to the list of things we collect, but I don't have a
laptop to run NixOS on at present, and the "over the years" part of that
suggestion doesn't fit well with Prometheus, which runs up against
Prometheus being the only monitoring system I really know. Still, if I
write the monitoring console right, it should be possible to include other
services.

- I think "Generate instructions for enabling it in configuration.nix" is
the consensus so far. Let's do that. One question might be what to do about
already-installed systems, but I suppose I can write a blog post once this
is ready for use.

- I didn't know about systemd-cgtop. Neat.

- I'll try to keep pointless metric collection to a minimum, but some of
them aren't pointless. E.g. GPU memory use; there's really no call for
alerting based on that, ever, but it'd still be useful for informing
purchase decisions. (They tend to swap into main memory, which won't
register.)

The problem with turning on collection only once you have a problem, is
that it may turn out you really needed the data from ten hours ago; e.g.
imagine seeing a graph where disk usage takes a sharp spike after the
latest system upgrade. That's trivial for humans to spot, but moderately
difficult for computers. (And it won't happen if we don't collect.)

The data isn't going to make its way off the computer. My main concern is
memory usage... for the moment, I'm aiming for a ballpark figure of 100MB
for the service overall. Of course, it can write metrics out to disk; say
1GB there. Those will be configurable, but there's only so much you can cut
before alerts stop working.

- For "System software too old", what I'm really aiming at is an alert
saying "Hey, you haven't upgraded your system in half a year, and there's
probably some security flaws in there by now." The details are TBD; since
we don't have people watching all the security lists and keeping Nix up to
date, using age-of-installed-channel as a proxy is probably the best we can
do.

A lot of the things I'd like to alert on, on my own system, would be mildly
questionable on others. I'll try to include config options where it makes
sense to me, but... this sort of detailed discussion is really for the PRs.

- And I don't suppose I can deny being at Google, no, but this isn't a
Google project; I've just ended up using NixOS *everywhere* in my personal
life, so now I want monitoring. ;-)

On Thu, Apr 21, 2016 at 9:41 PM Layus  wrote:

> I like the idea too.
>
> It seems to me that distributions really lack metrics collection and
> data analysis.
> For example, it would be nice to have automatic gathering of the battery
> usage (charge/discharge/capacity) and an easy access to compiled
> historical data like the capacity loss over the years.
>
> I know this is far less ambitious than what you describe, but it would
> be a great entry point for new users.
> If they like it, they may want to gather wore statistics.
>
> Anyway, I think it makes total sense to have such a feature in NixOS.
>
> -- Layus.
>
> On 20/04/16 11:49, Rok Garbas wrote:
> > +1 for the initiative. i don't believe personally enabling monitoring
> > by default should be the right way to go (since we all use nixos in
> > different contexts), but having a commented instructions in generated
> > configurations.nix would be the way to go.
> >
> > it would be nice if systemd monitoring stuff could be used as well:
> > https://github.com/garbas/dotfiles/blob/master/nixos/rok.nix#L236
> > above line makes systemd-cgtop showing numbers.
> >
> >
> >
> > On Wed, Apr 20, 2016 at 8:44 AM, Alexei Robyn  wrote:
> >> Seems interesting. You mention alerts for "System software too old.",
> but
> >> the only vaguely-universal definition of "too old" I can think of would
> be
> >> "missing security updates", and that's both debatable and an area where
> >> NixOS is currently fairly lacking in infrastructure and tooling.
> >>
> >> Default collection of metrics beyond what is necessary to provide useful
> >> alerts is a bad idea. Alerts have essentially universal usefulness,
> >> statistics less so - they're unnecessary for most desktops and small
> >> servers. At least until you have issues, so of course it'd be nice if
> they
> >> were easy to switch on :p.
> >>
> >> - Alexei
> >>
> >>
> >> On Wed, Apr 20, 2016, at 12:40 AM, Svein Ove Aas wrote:
> >>
> >> Hi all,
> >>
> >> People who are not interested in reliability or monitoring can stop
> reading
> >> now.
> >>
> >> --
> >>
> >> I've written up a "design doc" (statement of intent?) for how we might
> do
> >> monitoring-by-default. Once I think there is a reasonable level of
> consensus
> >> about how we should do this, 

Re: [Nix-dev] Monitoring by default

[Tomasz Czyż]

That's interesting, however I don't think this should be part of
"monitoring" service.

I'm using prometheus daily and I'm following its development and I don't
think it's stable enough (for example backend/storage changes quite often),
and prometheus is far from 1.0 (stable).

I don't agree that something like "monitoring" should be tied to specific
software. I personally like debian-way better than redhat-way ("apache2"
service instead of "httpd" service which is apache or nginx).
So I think I would like to have prometheus service and I would build my
system using those blocks. On that level I can use nixops to make a
"monitoring" service which consists of many other applications (let's say
prometheus + some dashboard + other modules around or I can use totally
different software instead, but still it will be monitoring service).

2016-04-21 21:41 GMT+01:00 Layus :

> I like the idea too.
>
> It seems to me that distributions really lack metrics collection and
> data analysis.
> For example, it would be nice to have automatic gathering of the battery
> usage (charge/discharge/capacity) and an easy access to compiled
> historical data like the capacity loss over the years.
>
> I know this is far less ambitious than what you describe, but it would
> be a great entry point for new users.
> If they like it, they may want to gather wore statistics.
>
> Anyway, I think it makes total sense to have such a feature in NixOS.
>
> -- Layus.
>
> On 20/04/16 11:49, Rok Garbas wrote:
> > +1 for the initiative. i don't believe personally enabling monitoring
> > by default should be the right way to go (since we all use nixos in
> > different contexts), but having a commented instructions in generated
> > configurations.nix would be the way to go.
> >
> > it would be nice if systemd monitoring stuff could be used as well:
> > https://github.com/garbas/dotfiles/blob/master/nixos/rok.nix#L236
> > above line makes systemd-cgtop showing numbers.
> >
> >
> >
> > On Wed, Apr 20, 2016 at 8:44 AM, Alexei Robyn  wrote:
> >> Seems interesting. You mention alerts for "System software too old.",
> but
> >> the only vaguely-universal definition of "too old" I can think of would
> be
> >> "missing security updates", and that's both debatable and an area where
> >> NixOS is currently fairly lacking in infrastructure and tooling.
> >>
> >> Default collection of metrics beyond what is necessary to provide useful
> >> alerts is a bad idea. Alerts have essentially universal usefulness,
> >> statistics less so - they're unnecessary for most desktops and small
> >> servers. At least until you have issues, so of course it'd be nice if
> they
> >> were easy to switch on :p.
> >>
> >> - Alexei
> >>
> >>
> >> On Wed, Apr 20, 2016, at 12:40 AM, Svein Ove Aas wrote:
> >>
> >> Hi all,
> >>
> >> People who are not interested in reliability or monitoring can stop
> reading
> >> now.
> >>
> >> --
> >>
> >> I've written up a "design doc" (statement of intent?) for how we might
> do
> >> monitoring-by-default. Once I think there is a reasonable level of
> consensus
> >> about how we should do this, I'll go ahead and implement what's in the
> >> document, but I'd like to make sure we're all on the same page first;
> >> especially as I want this to be on by default.
> >>
> >> So I'd like your input. Can you take a look?
> >>
> >> --
> >> Svein Ove Aas
> >> ___
> >> nix-dev mailing list
> >> nix-dev@lists.science.uu.nl
> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >>
> >>
> >>
> >> ___
> >> nix-dev mailing list
> >> nix-dev@lists.science.uu.nl
> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >>
> >
> >
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>



-- 
Tomasz Czyż
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev