Re: [Nix-dev] Rust prebuilt package overlay.

[William Casarin]

Nicolas Pierron  writes:

> On Tue, Mar 7, 2017 at 1:41 PM, Profpatsch  wrote:
>> On 17-03-04 08:34pm, Nicolas Pierron wrote:
>>> The *.toml manifest file is then parsed (yes, in Nix [3]) to extract
>>> [3] https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix
>>
>> what have you done
>
> I made tool to avoid updating this repository every time there is a
> new nightly version of rustc.
> This Nix expression is able to parse a 200 kB toml file in less than 1 second.

apparently you have angered the hacker gods on macos:

nixpkgs (efe12d665812e2c5f7c20513a829edff5b8b1977)
nixpkgs-mozilla (4779fb7776c3d38d78b5ebcee62165e6d1350f74)
nix-env (Nix) 1.11.8

$ nix-shell -p rustChannels.stable.rust

error: while evaluating the attribute ‘nativeBuildInputs’ of the derivation 
‘shell’ at /Users/jb55/dev/nixpkgs/pkgs/build-support/trivial-builders.nix:7:14:
while evaluating ‘getOutput’ at 
/Users/jb55/dev/nixpkgs/lib/attrsets.nix:453:23, called from undefined position:
while evaluating ‘getCrossDrv’ at 
/Users/jb55/dev/nixpkgs/pkgs/stdenv/generic/default.nix:133:21, called from 
undefined position:
while evaluating the attribute ‘stable.rust’ at 
/Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:154:5:
while evaluating ‘fromManifest’ at 
/Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:85:28, called from 
/Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:154:15:
while evaluating ‘flip’ at /Users/jb55/dev/nixpkgs/lib/trivial.nix:109:16, 
called from /Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:87:21:
while evaluating ‘mapAttrs’ at /Users/jb55/dev/nixpkgs/lib/attrsets.nix:198:17, 
called from /Users/jb55/dev/nixpkgs/lib/trivial.nix:109:19:
while evaluating ‘fromTOML’ at 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:133:14, called from 
/Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:86:16:
while evaluating ‘zipAttrs’ at 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:137:18, called from 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:149:7:
while evaluating ‘parser’ at 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:130:12, called from 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:135:19:
while evaluating ‘closeSection’ at 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:91:18, called from 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:131:5:
while evaluating ‘tokenizer’ at 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:41:15, called from 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:131:53:
while evaluating ‘tokenizer_rec’ at 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:6:46, called from 
/Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:41:20:
compiling pattern ‘([
]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")[
 
[... same line repeated 4094 times ...]
]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")).*’:
 out of memory



Looks like some kind of infinite recursion bug?
Seems to work fine on my nixos box though 樂

Cheers,
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Rust prebuilt package overlay.

[Nicolas Pierron]

On Sun, Apr 16, 2017 at 2:57 PM, William Casarin  wrote:
> Nicolas Pierron  writes:
>
>> On Fri, Apr 14, 2017 at 4:40 PM, William Casarin  wrote:
>>>
>>> $ nix-shell -p rustChannels.stable.rust
>>>
>>> [...]
>>> while evaluating ‘tokenizer_rec’ at 
>>> /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:6:46, called from 
>>> /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:41:20:
>>> compiling pattern ‘([
>>> ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")[
>>> [... same line repeated 4094 times ...]
>>> ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")).*’:
>>>  out of memory
>>>
>>> Looks like some kind of infinite recursion bug?
>>> Seems to work fine on my nixos box though 樂
>>
>> There is a constant[1] that you can modify in the overlay, by
>> increasing it, it would make a smaller pattern and consume less
>> memory, but it would make take more time to parse the content of the
>> file.
>
> This worked. Increasing it to 180 fixed it.

Thanks for testing, feel free to submit a patch ;)

> I was watching activity
> monitor and noticed spikes of 14GB memory usage (12GB compressed) so
> about ~2-3GB real. It looks like generatePatterns is creating 14GB+
> of strings!?

No, it generates a string which is a few ~100KB, but apparently the
C++ std::regex_match implementation probably do so when compiling the
string into an automaton.

-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Rust prebuilt package overlay.

[William Casarin]

Nicolas Pierron  writes:

> On Fri, Apr 14, 2017 at 4:40 PM, William Casarin  wrote:
>>
>> $ nix-shell -p rustChannels.stable.rust
>>
>> [...]
>> while evaluating ‘tokenizer_rec’ at 
>> /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:6:46, called from 
>> /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:41:20:
>> compiling pattern ‘([
>> ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")[
>> [... same line repeated 4094 times ...]
>> ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")).*’:
>>  out of memory
>>
>> Looks like some kind of infinite recursion bug?
>> Seems to work fine on my nixos box though 樂
>
> There is a constant[1] that you can modify in the overlay, by
> increasing it, it would make a smaller pattern and consume less
> memory, but it would make take more time to parse the content of the
> file.

This worked. Increasing it to 180 fixed it. I was watching activity
monitor and noticed spikes of 14GB memory usage (12GB compressed) so
about ~2-3GB real. It looks like generatePatterns is creating 14GB+
of strings!?

Thanks for the help,

-- 
https://jb55.com
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Rust prebuilt package overlay.

[Profpatsch]

On 17-03-08 01:58am, Nicolas Pierron wrote:
> On Tue, Mar 7, 2017 at 1:41 PM, Profpatsch  wrote:
> > On 17-03-04 08:34pm, Nicolas Pierron wrote:
> >> The *.toml manifest file is then parsed (yes, in Nix [3]) to extract
> >> [3] 
> >> https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix
> >
> > what have you done
>
> If you want more detail, maybe I should present it at the next NixCon.

Yes, please, in the recreational hacking section. :)

-- 
Proudly written in Mutt with Vim on NixOS.
Q: Why is this email five sentences or less?
A: http://five.sentenc.es
May take up to five days to read your message. If it’s urgent, call me.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Rust prebuilt package overlay.

[Nicolas Pierron]

On Tue, Mar 7, 2017 at 1:41 PM, Profpatsch  wrote:
> On 17-03-04 08:34pm, Nicolas Pierron wrote:
>> The *.toml manifest file is then parsed (yes, in Nix [3]) to extract
>> [3] https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix
>
> what have you done

I made tool to avoid updating this repository every time there is a
new nightly version of rustc.
This Nix expression is able to parse a 200 kB toml file in less than 1 second.

To make it short:
 - The tokenizer abuse ` builtin.match ` ability to generate a list of
strings out of the captured regex, and avoid deep recursions by
approximating down the number of expected tokens.
 - The parser uses ` builtin.foldl' `, to make some kind of for-loop
which does not consume space and mutate the parser state to
reconstruct the corresponding attribtue set of the *.toml file

If you want more detail, maybe I should present it at the next NixCon.

-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Rust prebuilt package overlay.

[Profpatsch]

On 17-03-04 08:34pm, Nicolas Pierron wrote:
> The *.toml manifest file is then parsed (yes, in Nix [3]) to extract
> [3] https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix

what have you done

-- 
Proudly written in Mutt with Vim on NixOS.
Q: Why is this email five sentences or less?
A: http://five.sentenc.es
May take up to five days to read your message. If it’s urgent, call me.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Rust prebuilt package overlay.

[Matthias Beyer]

Same from me.

I added a PR for documenting this (and the rest of our rust 
"ecosystem"): https://github.com/NixOS/nixpkgs/pull/23510

Feel free to review and suggest things!

On 05-03-2017 17:17:14, stewart mackenzie wrote:
> Great stuff Nicolas, much appreciated!
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev

-- 
Mit freundlichen Grüßen,
Kind regards,
Matthias Beyer

Consider switching to free software.
It adds value to your life.
https://www.gnu.org/


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Rust prebuilt package overlay.

[stewart mackenzie]

Great stuff Nicolas, much appreciated!
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Rust prebuilt package overlay.

[Nicolas Pierron]

Hi list,

# Motivation

I saw an issue on github the other day about Rust packaging #23291.  I
have faced the problem that Rust nightly version was not enough for
building rustc out of the repository.

Rust nightly is probably changing too frequently to be maintained
within Nixpkgs #8697.  Also, Firefox developer, I will need to follow
Rust versions as soon as they are packaged and updated in Firefox.

The recommended way for developing Firefox is to periodically run
rustup (every ~6 weeks), which is sad from my point of view.  As I
rely on nixpkgs-mozilla [1] repository to generate the build
environment I use, I decided to tackle the issue inherent of having
recent version of rustc & cargo.

# What

I created an overlay which mimic rustup behaviour, and allow you to
spawn a nix-shell with the latest nightly, beta, or stable version of
rustc & cargo.

To use this overlay, checkout nixpkgs-mozilla [1], and create a
symbolic link to the file rust-overlay.nix [2] in the
~/.config/nixpkgs/overlays

  $ cd ~/
  $ git clone https://github.com/mozilla/nixpkgs-mozilla.git
  $ mkdir -p  ~/.config/nixpkgs/overlays
  $ ln -s ~/nixpkgs-mozilla/rust-overlay.nix
~/.config/nixpkgs/overlays/rust-overlay.nix

Once installed, you can install the latest versions with the following commands:

  $ nix-env -Ai nixos.rustChannels.stable.rust

Or use this attribute to build a nix-shell environment which pull the
latest version of rust for you when you enter it.

You can substitute the "stable" channel by "nightly" and "beta", or
use the function provided by this overlay to pull a version based on a
build date.

# How

This overlay should auto-update it-self as if you were running rustup
each time you go through the rustChannels attributes.  It works by
using the fetchurl builtin function to pull the same file as rustup do
through https.

The *.toml manifest file is then parsed (yes, in Nix [3]) to extract
the sha256 and the location of all the packages indexed by the
manifest file.  Then, some logic is used [2] to convert the *.toml
file into proper derivations which are used to pull the prebuilt
binaries and to change the interpreter of the binaries using patchelf.

If you encounter issue, feel free to report them on the
nixpkgs-mozilla repository [1], and to fix them as well ;)

Have fun and enjoy.

#23291: https://github.com/NixOS/nixpkgs/issues/23291
#8697: https://github.com/NixOS/nixpkgs/issues/8697
[1] https://github.com/mozilla/nixpkgs-mozilla
[2] https://github.com/mozilla/nixpkgs-mozilla/blob/master/rust-overlay.nix
[3] https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix

-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev