Re: GSoC 2020
Hi Gábor, Gábor Boskovits skribis: > The organization application is now open for GSoC 2020. I expect that > we soon hear something from the GNU coordinator. I will keep you > updated. Thanks a lot for keeping an eye on it in addition to Outreachy! Ludo’.
Re: Proposal for a blog contribution on reproducible computations
Hi, I had missed that message. Konrad Hinsen skribis: > Ludovic Courtès writes: > >> Another thing that comes to mind: would it make sense to mention ‘guix >> graph’ in the part where you pipe the output of ‘guix show’ to ‘recsel’, >> etc.? > > Forgot that one, sorry. Yes, it would make sense, though I'd place it a > bit later in the text. But I'd have to figure out first how how the > various options of "guix graph" relate exactly to what I am writing. > > ‘package’ >This is the default type used in the example above. It shows the >DAG of package objects, excluding implicit dependencies. It is >concise, but filters out many details. > > Are "implicit dependencies" those added by the build system? If yes, > this edges in this graph would correspond to package-direct-inputs. Exactly. > ‘bag’ >Similar to ‘bag-emerged’, but this time including all the bootstrap >dependencies. > > And that is package-closure with arrows defined by bag-direct-inputs, right? Yes. Thanks, Ludo’.
Re: Package file indexing
Hello, Pierre Neidhardt skribis: > Thanks Nicolò, your feedback was very useful! > > So it's not program-not-found but command-not-found and it's define > here: > > > https://github.com/NixOS/nixpkgs/tree/master/nixos/modules/programs/command-not-found > > Then I found this > > https://github.com/NixOS/nixos-channel-scripts > > All this is pretty clear and simple. > > The main thing I wonder at this point is when the > "generate-programs-index.cc" file is run. > > What would be the good entry point in substitute servers to populate > such a database? The database could be updated upon reception of a build-completion notification from build machines or from the Data Service, like Chris proposed. (Or, if it were ‘guix publish’, it could do that on demand, the first time a narinfo is requested.) Ludo’.
Re: Presentation BlueHats (french workshop)
Hi zimoun, zimoun skribis: > Attached 2 patches for the repo 'maintenance'. > 1. Fixing broken links in talks/ > 2. My slides These had fallen through the holiday cracks, but I’ve finally pushed it! > This talk was in French with a slot of 5-7 minutes, questions included. It > was > taken in a full day satellite to Paris Open Source Summit. The initiative was > lead by Bastien Guerry from https://www.etalab.gouv.fr/. More information of > the programme > [[https://forum.etalab.gouv.fr/t/journee-bluehats-lors-du-paris-open-source-summit-le-11-decembre-2019/4614][here]]. > > The slot was very short and the audience very heterogeneous; especially about > the day-to-day concerns. As an engineer working in an institute doing > research > in biology, I have tried to explain what is the Reproducible Science challenge > in the modern age of data. > > In short, today a scientific result is an experiment producing data *and* a > numerical processing. From what I am seeing, the experimental part is more or > less well described, or let say that people in labs are aware of its > importance > because they have already several decades (even more) of collective learning. > > However, not enough people take care about the numerical processing. Mainly, > in > my opinion, because we are living a scientific paradigm shift. From what I am > seeing, more than often, it is not understood that more scientific value is in > the numerical process than really in the data itself (or how they are > produced). > Even if I am fully biased because computing is my job and I understand nothing > about labs. It’s nice you were able to talk at POSS. I suppose the audience was not necessarily familiar with reproducible science, right? > To guarantee Reproducible Science in the modern age of data, we need to > guarantee several items, especially: > 1. Open Articles > 2. Open Data > 3. Open Source > 4. Controlled computing environment (open, too) > Today, initiatives have been starting, to name some, about 1. > [[http://rescience.github.io/][ReScience journal]] > or french specific [[https://hal.archives-ouvertes.fr/][HAL]], 2. > [[https://zenodo.org/][Zenodo]] and 3. > [[https://www.softwareheritage.org/][Software Heritage]]. Yup! Not a fan of “open” which I find confusing here, but definitely a fan of putting all this in perspective! Thanks for sharing! Ludo’.
Re: [Proposal] The Formal Methods in GNU Guix Working Group
Hi Amin & Brett, Amin Bandali skribis: >> It’s fine to host the repo on Savannah: we can ask for a new repo under >> the Guix umbrella, the downside being that access control will be the >> same as for the other repos (we can only grant access to all the repos >> or none of them.) If you plan to open it more to formal methods people >> that do not yet contribute to Guix, it might be easier to use a separate >> repo. You tell us! >> > > Right. Thinking about this, as I see it right now I think our use cases > for repos fall roughly into two categories: > > - Closely Guix-related or small standalone things: this could be things > like the Haunt sources for our site, or a Guix channel for additional > package definitions, or anything closely related to Guix and/or small > enough to fit under the Guix umbrella just fine. For these, we should > be able to get by with a very small number of repos in the short and > long term. Initially, we will only have one such repository, say, > guix/guix-fm.git or guix/formal-methods.git, with its purpose being > mainly to keep the sources for the site. > > For these repos we’ll happily accept patches from folks who aren’t > Guix contributors via mailing list. And I’d imagine once they have > contributed enough patches, we could work out getting them commit > access, especially if their gathered knowledge/experience extends to > Guix directly (e.g. in form of familiarity with package definitions > and writing them). Sounds good to me. > - Larger projects or ones that don’t quite fit the scope of Guix: for > these, we might indeed consider registering separate Savannah projects > rather than putting them under the Guix project. I think the proposed > bootstrapping ML compiler could be an example of such project. Yes. > All that said, I do wish Savannah supported finer access control at the > project level. I just asked a fellow Savannah hacker for his opinion on > whether implementing that would be possible and feasible with the > current underlying infrastructure in mind. I suppose it would be hard and not necessarily advisable given that Savane is no longer actively developed, AIUI. >> As for the domain name: I think it would be fine to use >> formal-methods.guix.gnu.org as long as the web site follows GNU and Guix >> policy, which mostly means referring only to free software, avoiding the >> phrase “open source” to describe it, and probably avoiding institution >> logos and such (I don’t think there’s any written policy but I would >> personally find it out of place on gnu.org.) Anyway, the two of you are >> webmasters so you probably know this better than I do. IOW, if you want >> to flatter your employers and labs, you might want to opt for a separate >> web site. :-) >> > > Most certainly; I wouldn’t expect anything less. :-) > > As for institution logos, agreed. If it ever comes such time that we > absolutely “have to” consider that, I’ll be sure to check with you and > the other Guix maintainers, fellow GNU webmasters, and of course rms. Sure, sounds good to me! (Note that rms doesn’t have a say here, though.) > As for the domain name, I think formal-methods.guix.gnu.org is a bit of > a mouthful to type or say on a regular basis, and I think an abbreviated > fm.guix.gnu.org would be more convenient; à la ci.guix.gnu.org. For > what it’s worth, I’ve seen the FM abbreviation for Formal Methods used > fairly commonly around the community. Alright, I like expressive phrases like ‘call-with-current-continuation’ but I’m fine with “fm” nonetheless. ;-) > Lastly, I think it would be nice to have a guix...@gnu.org address for > Guix-FM. Rather than a full-fledged Mailman list, I think a simple > alias, like with guix-...@gnu.org, will suffice. Thoughts? Yeah, I guess a simple alias is enough for things that don’t fit on guix-devel (guix-...@gnu.org is very low traffic, mostly for when one needs to reach out to the usual HPC suspects.) Let us know when you have a web page ready. Then you’re welcome to send a patch against maintenance.git for ‘bayfront.scm’ (DNS config and probably web site as well.) Thanks! Ludo’.
Re: Inverted index to accelerate guix package search
Hi Ricardo, On Wed, 15 Jan 2020 at 22:03, Ricardo Wurmus wrote: > We could build and install the index when Guix is built and installed > and then use it for the search. I can’t think of a downside to adopting > an index compared to what we have now. I agree that it is an easy move that improve the current search experience. :-) So, yes work in progress... using the code that Arun sent and the code already available: sets.scm and pull.scm etc.. And how to adapt the lookup to the current fold-packages+regexp-matching. Do you think `vhash' is better than `hash-table'? However, I am not clear about how "guix pull" works so how to update the index. Still trying to figure out. :-) Well I am not clear about what is cached... I mean my folder ~/.cache/guix is crowdy. ;-) Cheers, simon
Re: Inverted index to accelerate guix package search
Arun Isaac writes: > Indeed, I found the bug and fixed it. Please find attached the updated > code. Also, the inverted index hash table stores package objects instead > of package names and all comparisons are now done with eq?. This gets us > an even higher speedup of around 300x. > > Built index in 8.556596040725708 seconds > Brute force search took 0.4107058048248291 seconds > Inverted index search took 0.0012979507446289062 seconds Neat! We could build and install the index when Guix is built and installed and then use it for the search. I can’t think of a downside to adopting an index compared to what we have now. -- Ricardo
OCaml 4.09
Hey all, Short message! I am trying to get our OCaml development system up to shape, as it is currently lagging some versions behind. I have opened a wip-ocaml4.09 branch for this work. Here are some of the things that need work that I have identified. -- The opam importer needs revised and updated. -- Basically all of the Jane Street components need reimported and modified. -- The dune build system probably needs to be able to handle two versions of dune without complication. -- We need to create some unison packages for things like dune-configuration. -- Basically all OCaml dependent packages need to get rebuilt and tested, ensuring that test cases are also appropriately handled (some are disabled that could easily be enabled.) -- Get our OCaml development components up to date and working, like ocamlformat, merlin, tuareg, the ocaml-lsp system (forked off of merlin). -- Potentially add a Coq subimporter for opam. -- Be able to have variant compilers for ocaml-flambda (and the upcoming ocaml-multicore) switches. If any of these tasks interest you, I challenge you to help! :). I am trying my best to do it myself so far, but I am running on a very limited amount of contribution time these next couple of weeks. This is an important task, too, for us Formal Methods people in the Guix community, as a lot of formal methods utilities run on OCaml. Carry on! Thanks. -- Brett M. Gilio GNU Guix, Contributor | GNU Project, Webmaster [DFC0 C7F7 9EE6 0CA7 AE55 5E19 6722 43C4 A03F 0EEE]
Re: Package file indexing
Thanks Nicolò, your feedback was very useful! So it's not program-not-found but command-not-found and it's define here: https://github.com/NixOS/nixpkgs/tree/master/nixos/modules/programs/command-not-found Then I found this https://github.com/NixOS/nixos-channel-scripts All this is pretty clear and simple. The main thing I wonder at this point is when the "generate-programs-index.cc" file is run. What would be the good entry point in substitute servers to populate such a database? -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Package file indexing
Hi Pierre, on NixOS, if you try to run the name of a program that you don't have installed (eg: $ endlessh) you get: The program ‘endlessh’ is currently not installed. You can install it by typing: nix-env -iA nixos.endlessh program-not-found is a perl script that uses an sqlite file placed under: /nix/var/nix/profiles/per-user/root/channels/nixos/programs.sqlite I don't know how this database is created. Table structure: CREATE TABLE Programs ( nametext not null, system text not null, package text not null, primary key (name, system, package) ); like kbdinfo|i686-linux|kbd (a nice thing I just found reading it: if you set NIX_AUTO_INSTALL=1 it automatically spawns a nix-shell with the required package and starts the program) Nicolò Pierre Neidhardt writes: > Ludovic Courtès writes: > >> You should look at how NixOS does it for its ‘command-not-found’ support >> (I think it’s part of NixOS, not Nix). IIRC they distribute an SQLite >> database, but it’s a pretty ad-hoc mechanism without authentication. > > I haven't found this yet, but I found this instead: > > https://github.com/bennofs/nix-index/pulls > > Tobias, are you sure Nix has such a feature? > > -- > Pierre Neidhardt > https://ambrevar.xyz/
Re: Inverted index to accelerate guix package search
Hi Giovanni, On Wed, 15 Jan 2020 at 12:53, Giovanni Biscuolo wrote: > zimoun writes: > > If Guix goes to Xapian, all the Git history of all the packages should > > be indexed. Today, it is really hard to find the right commit > > corresponding to the right version -- the only solution I have is "git > > log"+grep; not really friendly. > > indexing the all the history could be very interesting, but it's enough > interesting to me having a system to query (and tag if needed) Guix info > from a point in time, i.e. index creation Currently, "guix search" already does a good job (modulo the scoring already discussed in length :-)). If you read the doc about Guile regexp and recutils then you can query (almost) all you want; or explain what you are not able to query. :-) However, the current implementation cannot scale for all the packages; other said find removed packages, or old version. And it is very frustrating, especially when you use "guix time-machine". A concrete example is given here [1] and "git log --grep" done in [2] is not convenient at all. Another concrete example, see point 1. in [3]. [1] https://lists.gnu.org/archive/html/help-guix/2019-06/msg00094.html [2] https://lists.gnu.org/archive/html/help-guix/2019-06/msg00098.html [3] https://lists.gnu.org/archive/html/help-guix/2020-01/msg00087.html > thinking about implementation, IMHO indexing the output of "guix search" > is doable but indexing the commit logs of git is complex, probably too > much complex I do not want to index the commit log message. But today, AFAIK, use these commit logs is the only way to find the commit providing the version an user want. And it is not cool, IMHO. The whishlist is: be able to search through all the packages you can manipulate with Guix. Other said index all the packages after the big overhaul (inferior). (Note that we are talking about packages but it is the same thing for services.) > me too, as I said I'd like something like notmuch - guixmuch :-) - that > indexes my mailbox; in Guix terms I see "my mailbox" [1] as > packages/services/machines and all other useful metadata I have in all > my channels/profiles What do you mean by "other useful metadata"? > ...then I have muchsync to sync the notmuch database across different > machines, and a similar feature whould be nice for guixmuch :-O It is already possible using manifests. > [1] where IMAP is replaced by git (in various repos, for > packages/config/channels) and offlineimap is replaced by guix pull What you want already exist: manifest. :-) And "guix publish" to expose your own substitutes if the package is not build in the Guix farm and you have already built it. > > However, note that index all the packages of all the Guix history > > using guile-git+fold-packages is not straightforward: resource > > consuming and piece of well-thought data structures. > > > > Well, now "guix time-machine" is here, Guix is lacking tools to browse > > the history of all the packages. > > we are asking too much to Guix, but it is an interesting feature I do not think so it is "too much". :-) All the best, simon
Re: Inverted index to accelerate guix package search
Hi Pierre, On Wed, 15 Jan 2020 at 10:12, Pierre Neidhardt wrote: > > What about the "build time" size? Other said, all one needs to build > > to have xapian, i.e., the size of the full bag: starting from the > > bootstrap seed and having Guix+Xapian (vs. Guix alone). > > As shows with guix graph? > > This yields a small graph: > --8<---cut here---start->8--- > guix graph --type=bag-emerged xapian | dot -Tpdf > dag.pdf > --8<---cut here---end--->8--- > > This yields a big graph: > --8<---cut here---start->8--- > guix graph --type=bag xapian | dot -Tpdf > dag.pdf > --8<---cut here---end--->8--- > > I don't know how it compares to other packages. I do not know neither. But I guess that Xapian does not add extra dependencies then itself. I do not know if this filter is correct. --8<---cut here---start->8--- guixdot=/tmp/guix.dot guix graph --type=bag guix > $guixdot for what in `guix graph --type=bag xapian | grep '@' | cut -d'[' -f1`; do grep '@' $guixdot | grep $what > /dev/null if [ $? -eq 0 ] then echo "OK -- $what" else echo "KO -- $what" fi done --8<---cut here---end--->8--- Roughly, it compares the '/gnu/store' hashes of the nodes. And all the nodes of Xapian are already in the graph of Guix, except Xapian itself. So there is not extra cost (really low). I mean if I understand well. Well, guile-xapian bindings is the blocking point. Any taker? ;-) All the best, simon
Re: Inverted index to accelerate guix package search
On Wed, 15 Jan 2020 at 10:06, Pierre Neidhardt wrote: > We can always keep our current regexp search (which is trivial) for > those who really want it. I believe that Xapian is much more usable > than regexps on a daily basis. What do you mean by trivial? The command "guix search" supports the full Guile regexp engine, if I remember well. > > On the other hand, I can extend the inverted index implementation > > to support regular expression searches. Personally, I don't use regular > > expression based search queries, and don't think they are very useful > > especially if we make use of xapian's stemming. What do people think? > > Agreed! > > I see this with my emails (Notmuch): I type whatever words I remember > and whoever names was involved in a thread and I systematically find > it. I've used it for months and it never missed! :) The key point here is the scoring. Type whatever words you remember in the GNU mailing search engine. https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=load-path=Search%21=guix-devel=20=normal=score All the best, simon
Re: Inverted index to accelerate guix package search
Hi, On Wed, 15 Jan 2020 at 06:44, Arun Isaac wrote: > > Well, the issue is 'scoring' a query. > > I think the issue of whether to use an inverted index is orthogonal to > the quest to improve the relevance of the search results. Implementing > tf-idf like you suggested could greatly benefit from having fast > searches. I think it is not so orthogonal. In general, a fast and good system is a combination of relevant scoring adapted to the good data structure, IMHO. However, I agree that adding an inverted index will improve the current situation of "guix search" -- keeping the current scoring function -- and ease the end-user experience. > Pierre Neidhardt writes: > > > By the way, what about using Xapian in Guix? > > > > https://en.wikipedia.org/wiki/Xapian > > > > If it's relevant, maybe we can follow up with a discussion in a new > > thread. > > I feel xapian is too much work (considering that we don't yet have guile > bindings) compared to our own simple implementation of an inverted > index. But, of course, I am biased since I wrote the inverted index > code! :-) It depends on how long run we are talking. :-) Xapian avoids to reinvent the wheel. ;-) > But, on a more serious note, if we move to xapian, we will not be able > to support regular expression based search queries that we support > today. I am not convinced... > On the question of whether xapian is too heavy, I think we should make > it an optional dependency of Guix so that it remains possible to build > and use a more minimalistic Guix for those interested in such things. Guix comes with SQLite and it is ok. The question is: how Xapian is minimalist. :-) (need some investigation) All the best, simon
Re: Inverted index to accelerate guix package search
Hi Simon and all other developers first and foremost: I'm sorry I still cannot help in developing this "xapian feature" or even explore pros and cons of this feature for now mine it's just a user and possible beta-tester POV... let's call it food for thought :-) zimoun writes: [...] >> Actually I would love to search (tag?!?) for packages/services/machines >> (profiles?) the same way I query my email database with notmuch: it >> would give me a fourth dimension in "DevOps" :-) > > Currently, you can output all the packages with 'guix search ""' and > index them using the Python API. It is not so nice but it should give > an idea about what to expose first to Guile. interesting approach for an indexer external to Guix, let's call it a proof of concept... I'd really like to experiment this directly in Guile but I'm not able to help with Guile bindings >> I cannot find Guile in Xapian bindings [1] but NotDeft [2] have some >> Emacs Lisp code to interact with Xapian > > There is some work. IMHO. for sure :-) this is why i defined this low-pri in the general Guix plans > If Guix goes to Xapian, all the Git history of all the packages should > be indexed. Today, it is really hard to find the right commit > corresponding to the right version -- the only solution I have is "git > log"+grep; not really friendly. indexing the all the history could be very interesting, but it's enough interesting to me having a system to query (and tag if needed) Guix info from a point in time, i.e. index creation thinking about implementation, IMHO indexing the output of "guix search" is doable but indexing the commit logs of git is complex, probably too much complex > Some discussion spoke about using an external index, fetched for > example on the Guix Data Service. But I personally do not like to rely > on external service accessible through the network. me too, as I said I'd like something like notmuch - guixmuch :-) - that indexes my mailbox; in Guix terms I see "my mailbox" [1] as packages/services/machines and all other useful metadata I have in all my channels/profiles ...then I have muchsync to sync the notmuch database across different machines, and a similar feature whould be nice for guixmuch :-O [1] where IMAP is replaced by git (in various repos, for packages/config/channels) and offlineimap is replaced by guix pull > However, note that index all the packages of all the Guix history > using guile-git+fold-packages is not straightforward: resource > consuming and piece of well-thought data structures. > > Well, now "guix time-machine" is here, Guix is lacking tools to browse > the history of all the packages. we are asking too much to Guix, but it is an interesting feature Thanks! Gio' [...] -- Giovanni Biscuolo Xelera IT Infrastructures signature.asc Description: PGP signature
Re: Parameterized packages
Hi Pierre, On Wed, 15 Jan 2020 at 10:40, Pierre Neidhardt wrote: > - The source. > - The explicit inputs. > - the implicit inputs. > - The build inputs. > - The build system inputs. > - Recursive inputs. > - ...? > > Which one should we expose? I don't know. If we want the system to > have some set of properties, I guess only the resulting packages matther > and the build inputs don't. But the resulting packages depends on the build inputs. So if the resulting packages matter, then the build inputs too. Well, you already described the issue here [1] (point 7.) -- if I understand correctly. [1] https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00026.html > Should we touch implicit inputs, we would have to parse all the > references and not just the explicit inputs like --with-input does. My mind is not clear at all. What is the final aim to have parametrized packages? What does it mean "parametrized"? Does it mean extend the transformation options as Ludo described [2]. Does it mean more? [2] https://lists.gnu.org/archive/html/guix-devel/2019-05/msg00285.html > > but be able to > > recompile a matrix of combinaison would be really cool! > > Indeed. But now I'm starting to wonder if it is really doable :p What I was thinking of is to parametrize only the build systems. For example, be able to rebuild all the packages with GCC-8.3, or to install Python packages with Python 3.5 instead of the current default Python 3.7. Cheers, simon
Re: Parameterized packages
Hi Simon! You are making a good point: there are different levels of "accessibility" when it comes to inputs: - The source. - The explicit inputs. - the implicit inputs. - The build inputs. - The build system inputs. - Recursive inputs. - ...? Which one should we expose? I don't know. If we want the system to have some set of properties, I guess only the resulting packages matther and the build inputs don't. Should we touch implicit inputs, we would have to parse all the references and not just the explicit inputs like --with-input does. > but be able to > recompile a matrix of combinaison would be really cool! Indeed. But now I'm starting to wonder if it is really doable :p -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
GSoC 2020
Hello guix, The organization application is now open for GSoC 2020. I expect that we soon hear something from the GNU coordinator. I will keep you updated. Best regards, g_bor -- OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21
Core updates evaluation failure (10060, 227fab3)
Hey I pushed to core-updates, but the evaluation on ci.guix.gnu.org seems to have failed. Nothing stands out to me in the log [1], but the Guix Data Service also failed to load this revision [2], and the failure there was building Python [3]. 1: http://ci.guix.gnu.org/eval/10060/log/raw 2: http://data.guix.gnu.org/revision/227fab3ed8842b22337196ee21c79143fc093855 3: http://data.guix.gnu.org/job/10692#bottom I tried building that same derivation again, and I got the same failure. Thanks, Chris signature.asc Description: PGP signature
