Re: [Nix-dev] Rust prebuilt package overlay.
Nicolas Pierron writes: > On Tue, Mar 7, 2017 at 1:41 PM, Profpatsch wrote: >> On 17-03-04 08:34pm, Nicolas Pierron wrote: >>> The *.toml manifest file is then parsed (yes, in Nix [3]) to extract >>> [3] https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix >> >> what have you done > > I made tool to avoid updating this repository every time there is a > new nightly version of rustc. > This Nix expression is able to parse a 200 kB toml file in less than 1 second. apparently you have angered the hacker gods on macos: nixpkgs (efe12d665812e2c5f7c20513a829edff5b8b1977) nixpkgs-mozilla (4779fb7776c3d38d78b5ebcee62165e6d1350f74) nix-env (Nix) 1.11.8 $ nix-shell -p rustChannels.stable.rust error: while evaluating the attribute ‘nativeBuildInputs’ of the derivation ‘shell’ at /Users/jb55/dev/nixpkgs/pkgs/build-support/trivial-builders.nix:7:14: while evaluating ‘getOutput’ at /Users/jb55/dev/nixpkgs/lib/attrsets.nix:453:23, called from undefined position: while evaluating ‘getCrossDrv’ at /Users/jb55/dev/nixpkgs/pkgs/stdenv/generic/default.nix:133:21, called from undefined position: while evaluating the attribute ‘stable.rust’ at /Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:154:5: while evaluating ‘fromManifest’ at /Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:85:28, called from /Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:154:15: while evaluating ‘flip’ at /Users/jb55/dev/nixpkgs/lib/trivial.nix:109:16, called from /Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:87:21: while evaluating ‘mapAttrs’ at /Users/jb55/dev/nixpkgs/lib/attrsets.nix:198:17, called from /Users/jb55/dev/nixpkgs/lib/trivial.nix:109:19: while evaluating ‘fromTOML’ at /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:133:14, called from /Users/jb55/etc/nixpkgs-mozilla/rust-overlay.nix:86:16: while evaluating ‘zipAttrs’ at /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:137:18, called from /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:149:7: while evaluating ‘parser’ at /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:130:12, called from /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:135:19: while evaluating ‘closeSection’ at /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:91:18, called from /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:131:5: while evaluating ‘tokenizer’ at /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:41:15, called from /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:131:53: while evaluating ‘tokenizer_rec’ at /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:6:46, called from /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:41:20: compiling pattern ‘([ ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")[ [... same line repeated 4094 times ...] ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")).*’: out of memory Looks like some kind of infinite recursion bug? Seems to work fine on my nixos box though 🤔 Cheers, ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Rust prebuilt package overlay.
On Sun, Apr 16, 2017 at 2:57 PM, William Casarin wrote: > Nicolas Pierron writes: > >> On Fri, Apr 14, 2017 at 4:40 PM, William Casarin wrote: >>> >>> $ nix-shell -p rustChannels.stable.rust >>> >>> [...] >>> while evaluating ‘tokenizer_rec’ at >>> /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:6:46, called from >>> /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:41:20: >>> compiling pattern ‘([ >>> ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")[ >>> [... same line repeated 4094 times ...] >>> ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")).*’: >>> out of memory >>> >>> Looks like some kind of infinite recursion bug? >>> Seems to work fine on my nixos box though 🤔 >> >> There is a constant[1] that you can modify in the overlay, by >> increasing it, it would make a smaller pattern and consume less >> memory, but it would make take more time to parse the content of the >> file. > > This worked. Increasing it to 180 fixed it. Thanks for testing, feel free to submit a patch ;) > I was watching activity > monitor and noticed spikes of 14GB memory usage (12GB compressed) so > about ~2-3GB real. It looks like generatePatterns is creating 14GB+ > of strings!? No, it generates a string which is a few ~100KB, but apparently the C++ std::regex_match implementation probably do so when compiling the string into an automaton. -- Nicolas Pierron http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Rust prebuilt package overlay.
Nicolas Pierron writes: > On Fri, Apr 14, 2017 at 4:40 PM, William Casarin wrote: >> >> $ nix-shell -p rustChannels.stable.rust >> >> [...] >> while evaluating ‘tokenizer_rec’ at >> /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:6:46, called from >> /Users/jb55/etc/nixpkgs-mozilla/lib/parseTOML.nix:41:20: >> compiling pattern ‘([ >> ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")[ >> [... same line repeated 4094 times ...] >> ]+(=|[[][[][a-zA-Z0-9_."*-]+[]][]]|[[][a-zA-Z0-9_."*-]+[]]|[a-zA-Z0-9_-]+|"[^"]*")).*’: >> out of memory >> >> Looks like some kind of infinite recursion bug? >> Seems to work fine on my nixos box though 🤔 > > There is a constant[1] that you can modify in the overlay, by > increasing it, it would make a smaller pattern and consume less > memory, but it would make take more time to parse the content of the > file. This worked. Increasing it to 180 fixed it. I was watching activity monitor and noticed spikes of 14GB memory usage (12GB compressed) so about ~2-3GB real. It looks like generatePatterns is creating 14GB+ of strings!? Thanks for the help, -- https://jb55.com ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Rust prebuilt package overlay.
On 17-03-08 01:58am, Nicolas Pierron wrote: > On Tue, Mar 7, 2017 at 1:41 PM, Profpatsch wrote: > > On 17-03-04 08:34pm, Nicolas Pierron wrote: > >> The *.toml manifest file is then parsed (yes, in Nix [3]) to extract > >> [3] > >> https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix > > > > what have you done > > If you want more detail, maybe I should present it at the next NixCon. Yes, please, in the recreational hacking section. :) -- Proudly written in Mutt with Vim on NixOS. Q: Why is this email five sentences or less? A: http://five.sentenc.es May take up to five days to read your message. If it’s urgent, call me. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Rust prebuilt package overlay.
On Tue, Mar 7, 2017 at 1:41 PM, Profpatsch wrote: > On 17-03-04 08:34pm, Nicolas Pierron wrote: >> The *.toml manifest file is then parsed (yes, in Nix [3]) to extract >> [3] https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix > > what have you done I made tool to avoid updating this repository every time there is a new nightly version of rustc. This Nix expression is able to parse a 200 kB toml file in less than 1 second. To make it short: - The tokenizer abuse ` builtin.match ` ability to generate a list of strings out of the captured regex, and avoid deep recursions by approximating down the number of expected tokens. - The parser uses ` builtin.foldl' `, to make some kind of for-loop which does not consume space and mutate the parser state to reconstruct the corresponding attribtue set of the *.toml file If you want more detail, maybe I should present it at the next NixCon. -- Nicolas Pierron http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Rust prebuilt package overlay.
On 17-03-04 08:34pm, Nicolas Pierron wrote: > The *.toml manifest file is then parsed (yes, in Nix [3]) to extract > [3] https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix what have you done -- Proudly written in Mutt with Vim on NixOS. Q: Why is this email five sentences or less? A: http://five.sentenc.es May take up to five days to read your message. If it’s urgent, call me. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Rust prebuilt package overlay.
Same from me. I added a PR for documenting this (and the rest of our rust "ecosystem"): https://github.com/NixOS/nixpkgs/pull/23510 Feel free to review and suggest things! On 05-03-2017 17:17:14, stewart mackenzie wrote: > Great stuff Nicolas, much appreciated! > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev -- Mit freundlichen Grüßen, Kind regards, Matthias Beyer Consider switching to free software. It adds value to your life. https://www.gnu.org/ signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Rust prebuilt package overlay.
Great stuff Nicolas, much appreciated! ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Rust prebuilt package overlay.
Hi list, # Motivation I saw an issue on github the other day about Rust packaging #23291. I have faced the problem that Rust nightly version was not enough for building rustc out of the repository. Rust nightly is probably changing too frequently to be maintained within Nixpkgs #8697. Also, Firefox developer, I will need to follow Rust versions as soon as they are packaged and updated in Firefox. The recommended way for developing Firefox is to periodically run rustup (every ~6 weeks), which is sad from my point of view. As I rely on nixpkgs-mozilla [1] repository to generate the build environment I use, I decided to tackle the issue inherent of having recent version of rustc & cargo. # What I created an overlay which mimic rustup behaviour, and allow you to spawn a nix-shell with the latest nightly, beta, or stable version of rustc & cargo. To use this overlay, checkout nixpkgs-mozilla [1], and create a symbolic link to the file rust-overlay.nix [2] in the ~/.config/nixpkgs/overlays $ cd ~/ $ git clone https://github.com/mozilla/nixpkgs-mozilla.git $ mkdir -p ~/.config/nixpkgs/overlays $ ln -s ~/nixpkgs-mozilla/rust-overlay.nix ~/.config/nixpkgs/overlays/rust-overlay.nix Once installed, you can install the latest versions with the following commands: $ nix-env -Ai nixos.rustChannels.stable.rust Or use this attribute to build a nix-shell environment which pull the latest version of rust for you when you enter it. You can substitute the "stable" channel by "nightly" and "beta", or use the function provided by this overlay to pull a version based on a build date. # How This overlay should auto-update it-self as if you were running rustup each time you go through the rustChannels attributes. It works by using the fetchurl builtin function to pull the same file as rustup do through https. The *.toml manifest file is then parsed (yes, in Nix [3]) to extract the sha256 and the location of all the packages indexed by the manifest file. Then, some logic is used [2] to convert the *.toml file into proper derivations which are used to pull the prebuilt binaries and to change the interpreter of the binaries using patchelf. If you encounter issue, feel free to report them on the nixpkgs-mozilla repository [1], and to fix them as well ;) Have fun and enjoy. #23291: https://github.com/NixOS/nixpkgs/issues/23291 #8697: https://github.com/NixOS/nixpkgs/issues/8697 [1] https://github.com/mozilla/nixpkgs-mozilla [2] https://github.com/mozilla/nixpkgs-mozilla/blob/master/rust-overlay.nix [3] https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix -- Nicolas Pierron http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev