Re: GSoC update
> Could you expound a bit? That’s a very short summary for all the sweat > you’ve put in it. :-) My apologies, at the time I sent the mail in a hurry. Basically now instead of converting unit files to services individually it happens in bulk so that it can check if there is a corresponding .socket file per service file (and the reverse). If there is a corresponding .socket file then the input and output of the .service will be redirected to the result of (accept) on the socket corresponding to the .socket file. It also makes the select that waits for the commands from herd to also wait for the sockets. > Also, what is the patch against? It’s not against ‘master’; I suppose > it’s against the previous state of your own branch, do you have a copy > of your repo on-line? It's against the previous patch that I sent, but I can put the branch online. > It’s OK that the thing doesn’t quite work—we knew it was not an easy > task. What’s disappointing though is that you didn’t come to us to > discuss the issues until now. GSoC is not about working in all > loneliness separately from the rest of the group; it’s about becoming > part of the group. > > On IRC Jelle and I (and possibly others) offered help on the ‘signalfd’ > issue; I also outlined reasonable milestones (first, only use > signalfd(2) instead of SIGCHLD, then discuss together what it would take > to Fiberize the whole thing.) It’s sad that you remained stuck instead > of taking this opportunity to discuss it with us. Until now (in general, not only during the gsoc) I tried to solve any issues that had arisen when I was programming by myself, so it was a bit difficult to change that mindset - I will try to be more communicative after this however. > The patch changes lots of things and unfortunately, without > explanations, I do not understand what to do with it. Like what’s the > new feature? How is it used? What implementation choices were made? > What’s left to be done?… The new feature is initial support of .socket unit files. It is used like: (let* ((port1 (open-input-file "/systemd/test.service")) (port2 (open-input-file "/systemd/test.socket"))) (apply register-services (unit-files->services `(("/systemd/test.service" "test" service ,(read-unit-file port1)) ("/systemd/test.socket" "test" socket ,(read-unit-file port2) (close-port port1) (close-port port2)) The things that are left are supporting more systemd options, and making it work properly.
Re: GSoC update
2018-07-11 0:40 GMT+02:00 Ludovic Courtès : > Hi Ioannis, > > Ioannis Panagiotis Koutsidis skribis: > > > This patch adds initial support for .socket unit files. It does not > > currently work but is near completion. > > Could you expound a bit? That’s a very short summary for all the sweat > you’ve put in it. :-) > > Also, what is the patch against? It’s not against ‘master’; I suppose > it’s against the previous state of your own branch, do you have a copy > of your repo on-line? > > > During the past month I also worked on a patch that adds signalfd and > > fiber support but these are currently way too unstable and for that > > reason I have not included them in this patch. > > It’s OK that the thing doesn’t quite work—we knew it was not an easy > task. What’s disappointing though is that you didn’t come to us to > discuss the issues until now. GSoC is not about working in all > loneliness separately from the rest of the group; it’s about becoming > part of the group. > > On IRC Jelle and I (and possibly others) offered help on the ‘signalfd’ > issue; I also outlined reasonable milestones (first, only use > signalfd(2) instead of SIGCHLD, then discuss together what it would take > to Fiberize the whole thing.) It’s sad that you remained stuck instead > of taking this opportunity to discuss it with us. > Ioannis, could you perhaps share some of your w.i.p. code regarding signalfd-based signal handling in guile? Adding to what Ludo' mentioned, I imagine you are running into some peculiarities regarding guile's way of handling signals, so I would recommend to start lurking on #guile if you did not do this before now, so you can interact with the folks with the most expertise regarding the problems you might be facing :-) > > > From cd260ae65056b53749e7c03f2498a28af2525934 Mon Sep 17 00:00:00 2001 > > From: Ioannis Panagiotis Koutsidis > > Date: Tue, 10 Jul 2018 20:03:21 +0300 > > Subject: [PATCH] .socket units > > > > --- > > modules/shepherd.scm | 44 +++-- > > modules/shepherd/service.scm | 170 ++--- > > modules/shepherd/systemd.scm | 354 --- > > 3 files changed, 368 insertions(+), 200 deletions(-) > > The patch changes lots of things and unfortunately, without > explanations, I do not understand what to do with it. Like what’s the > new feature? How is it used? What implementation choices were made? > What’s left to be done?… > > Thank you, > Ludo’. > >
Re: GSoC update
Hi Ioannis, Ioannis Panagiotis Koutsidis skribis: > This patch adds initial support for .socket unit files. It does not > currently work but is near completion. Could you expound a bit? That’s a very short summary for all the sweat you’ve put in it. :-) Also, what is the patch against? It’s not against ‘master’; I suppose it’s against the previous state of your own branch, do you have a copy of your repo on-line? > During the past month I also worked on a patch that adds signalfd and > fiber support but these are currently way too unstable and for that > reason I have not included them in this patch. It’s OK that the thing doesn’t quite work—we knew it was not an easy task. What’s disappointing though is that you didn’t come to us to discuss the issues until now. GSoC is not about working in all loneliness separately from the rest of the group; it’s about becoming part of the group. On IRC Jelle and I (and possibly others) offered help on the ‘signalfd’ issue; I also outlined reasonable milestones (first, only use signalfd(2) instead of SIGCHLD, then discuss together what it would take to Fiberize the whole thing.) It’s sad that you remained stuck instead of taking this opportunity to discuss it with us. > From cd260ae65056b53749e7c03f2498a28af2525934 Mon Sep 17 00:00:00 2001 > From: Ioannis Panagiotis Koutsidis > Date: Tue, 10 Jul 2018 20:03:21 +0300 > Subject: [PATCH] .socket units > > --- > modules/shepherd.scm | 44 +++-- > modules/shepherd/service.scm | 170 ++--- > modules/shepherd/systemd.scm | 354 --- > 3 files changed, 368 insertions(+), 200 deletions(-) The patch changes lots of things and unfortunately, without explanations, I do not understand what to do with it. Like what’s the new feature? How is it used? What implementation choices were made? What’s left to be done?… Thank you, Ludo’.
Re: [GSoC update] Npm & guix
Quick reply from my phone, but thanks for the feedback. On Jul 29, 2016 16:53, "Catonano"wrote: > 2016-07-25 23:26 GMT+02:00 Ludovic Courtès : > >> Hello! >> >> Jelle Licht skribis: >> >> > On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it >> > on my npm importer. After leaving the importer running for a quite >> > some hours (and making it more robust in the face of inconsistent npm >> > information), it turns out that jQuery has a direct or indirect >> > dependcy on about everything. We are talking pretty much all of the >> > build systems, all of the testing frameworks and all of the test >> > runners. Literally thousands of packages, and multiple (conflicting) >> > versions of most. >> >> I’m really impressed that your importer can already grovel this much! >> In itself, that’s already a significant achievement, despite the >> frustration of not getting “guix package -i jquery” right away. >> >> Do you have figures on the number of vertices and edges on this graph? >> Could it be that the recursive importer keeps revisiting the same nodes >> over and over again? :-) >> > > > >> I would suggest publishing the code somewhere so others can try to >> import their favorite JS library and give feedback. >> >> > I'd like to indicate that in the Guix code base there's a function > visiting the graph of the dependencies of a package with the concern of > covering it all AND of not considering the same node more than one time > > It's in guix/packages.scm on line 552, it's called "transitive-inputs" and > takes as an arguments the inputs of a single package. > > It could be used as a blueprint for such task of dependencies graphs > covering. > > The main observation that I can do is that both versions do check whether > a package being considered is already been seen in a previous passage. > > So, it doesn't seem to me that Jelle's version could revisit the same > package (node) over and over > > Also, the "official" version clearly distinguishes between the current > depth level in the graph and the next one, using 2 different variables for > containing the packages (nodes) at the current depth and those in the next > level ("inputs" and "propagated") as it does a breadth first visit. > > Instead the Jelle's version uses the same variable for the current AND the > next level and it's a list and it conses the nodes for the next level on > the current list > > So it seems to me that it does a depth first visit (I might be wrong, I > didn't do a complete trace) > > If anything, the "official" version uses a VHash to check if a package has > already been "seen" while Jelle's uses a plain list (again) > > So the cost of this check is constant in the official version and linear > in Jelle's version. > > Further, in Jelle's version, every package gets checked against the list > of already imported packages (as a plain list) AND against the packages > already present in the store. > > Both checks at every iteration. It seems to me that there's space for > improving here ;-) > > In fact, in Jelle's guix/import/npm.scm, on line 462 the "and" could be > switched to an "or". It would work anyway and it could save a handful of > checks on a complete run of the algorithm > You are totally right here! The quick and dirty way of doing it, which I am currently doing, is to wrap these checks in a memoized function. I'll take some of your points into account to properly rewrite this part. > > Anyway, I tried to "import" a random package and it has a ton of > dependencies. > Story of my life these past few weeks :-) > > It seems to me that a more systematic approach (like that of a real data > scientist ;-) ) could help here > > > The whole graph should be imported in some database and then graph > algorithms should be run on it > > For example: which are the packages with less or no dependencies (and a > lot of dependants) ? > Because those should be imported first, in my opinion. > > Jelle came to the notion that testing frameworks are a dependencies for > almost all the packages but as far as I understand that's a quite empirical > knowledge. > ^ This, I like. Does anyone have any suggestions on tools that could help me do this in guile? I know of projects like [1] that already do something similar, although I do no have any experience with constructing and querying graphs using free software. > > >> > I am currently hovering between version 0.6 and 0.7, which can >> > properly recompile itself and slightly more contemporary >> > version. Getting to version 1.6 from June 2013 should be doable using >> > this exact same approach. This will allow us to package a 2014 >> > version of the Mocha testing framework. >> >> Impressive. >> > > Indeed. Impressive. > > Ok, so these are the first impressions that I came up with. I'd like my > observations on the algorithms to be vetted by more experienced people. > > Just to help me verify if I can
Re: [GSoC update] Npm & guix
2016-07-25 23:26 GMT+02:00 Ludovic Courtès: > Hello! > > Jelle Licht skribis: > > > On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it > > on my npm importer. After leaving the importer running for a quite > > some hours (and making it more robust in the face of inconsistent npm > > information), it turns out that jQuery has a direct or indirect > > dependcy on about everything. We are talking pretty much all of the > > build systems, all of the testing frameworks and all of the test > > runners. Literally thousands of packages, and multiple (conflicting) > > versions of most. > > I’m really impressed that your importer can already grovel this much! > In itself, that’s already a significant achievement, despite the > frustration of not getting “guix package -i jquery” right away. > > Do you have figures on the number of vertices and edges on this graph? > Could it be that the recursive importer keeps revisiting the same nodes > over and over again? :-) > > I would suggest publishing the code somewhere so others can try to > import their favorite JS library and give feedback. > > I'd like to indicate that in the Guix code base there's a function visiting the graph of the dependencies of a package with the concern of covering it all AND of not considering the same node more than one time It's in guix/packages.scm on line 552, it's called "transitive-inputs" and takes as an arguments the inputs of a single package. It could be used as a blueprint for such task of dependencies graphs covering. The main observation that I can do is that both versions do check whether a package being considered is already been seen in a previous passage. So, it doesn't seem to me that Jelle's version could revisit the same package (node) over and over Also, the "official" version clearly distinguishes between the current depth level in the graph and the next one, using 2 different variables for containing the packages (nodes) at the current depth and those in the next level ("inputs" and "propagated") as it does a breadth first visit. Instead the Jelle's version uses the same variable for the current AND the next level and it's a list and it conses the nodes for the next level on the current list So it seems to me that it does a depth first visit (I might be wrong, I didn't do a complete trace) If anything, the "official" version uses a VHash to check if a package has already been "seen" while Jelle's uses a plain list (again) So the cost of this check is constant in the official version and linear in Jelle's version. Further, in Jelle's version, every package gets checked against the list of already imported packages (as a plain list) AND against the packages already present in the store. Both checks at every iteration. It seems to me that there's space for improving here ;-) In fact, in Jelle's guix/import/npm.scm, on line 462 the "and" could be switched to an "or". It would work anyway and it could save a handful of checks on a complete run of the algorithm Anyway, I tried to "import" a random package and it has a ton of dependencies. It seems to me that a more systematic approach (like that of a real data scientist ;-) ) could help here The whole graph should be imported in some database and then graph algorithms should be run on it For example: which are the packages with less or no dependencies (and a lot of dependants) ? Because those should be imported first, in my opinion. Jelle came to the notion that testing frameworks are a dependencies for almost all the packages but as far as I understand that's a quite empirical knowledge. > > I am currently hovering between version 0.6 and 0.7, which can > > properly recompile itself and slightly more contemporary > > version. Getting to version 1.6 from June 2013 should be doable using > > this exact same approach. This will allow us to package a 2014 > > version of the Mocha testing framework. > > Impressive. > Indeed. Impressive. Ok, so these are the first impressions that I came up with. I'd like my observations on the algorithms to be vetted by more experienced people. Just to help me verify if I can actually understand someone's else scheme code Thanks
Re: [GSoC update] Npm & guix
Hello! Jelle Lichtskribis: > On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it > on my npm importer. After leaving the importer running for a quite > some hours (and making it more robust in the face of inconsistent npm > information), it turns out that jQuery has a direct or indirect > dependcy on about everything. We are talking pretty much all of the > build systems, all of the testing frameworks and all of the test > runners. Literally thousands of packages, and multiple (conflicting) > versions of most. I’m really impressed that your importer can already grovel this much! In itself, that’s already a significant achievement, despite the frustration of not getting “guix package -i jquery” right away. Do you have figures on the number of vertices and edges on this graph? Could it be that the recursive importer keeps revisiting the same nodes over and over again? :-) I would suggest publishing the code somewhere so others can try to import their favorite JS library and give feedback. > This makes it IMHO a worthwhile goal bootstrap to a working test > framework, with of course at first tests disabled for the dependencies > of this test framework. Test frameworks all have an (indirect) > dependency on the Coffeescript compiler, of which the first version > was written in Ruby. Using this initial (alpha) compiler, and the > awesome git-bisect command, I was able to subsequently compile and use > the more modern (but still old) Coffeescript-in-coffeescript > compilers. > > I am currently hovering between version 0.6 and 0.7, which can > properly recompile itself and slightly more contemporary > version. Getting to version 1.6 from June 2013 should be doable using > this exact same approach. This will allow us to package a 2014 > version of the Mocha testing framework. Impressive. > For the people more knowledgeable in these matters, how would you deal > with deprecated functionality in languages such as python, ruby etc? > Because npm packages are so interdependent, I simply need to start > somewhere, by packaging things back when they did not have as many > dependencies. Currently, I have a file containing implementations of > old Node (pre 1.0) functionality on Node 6.0. I was thinking of > releasing this 'hack' as an npm package and then package it in Guix. > > The alternative would be to package each version of Node that was used > to build these ancient packages. For bootstrapping Coffeescript, this > already forces us to have 3~4 versions of Node, although it is > conceptually a lot cleaner. > > So my current view of our options: * Backport ancient node features to > a contemporary node version * Package a significant variety of node > versions Please let me know if anyone has some thoughts, critiques or > silver bullets for this problem. People have looked at bootstrapping compilers using their historical ancestor, and it often looked hard first to find what each implementation’s ancestor was, and then to actually chain them (I remember Ricardo going back to an old Haskellish implementation in Scheme at one point.) Of the two options you list, packaging several Node versions sounds like the simplest one. There may be other stumbling blocks though, so be prepared to stop the packaging recursion before you get mad. ;-) In practice, we’ve never managed to get this far for the other compilers we have. Thanks for the status update! Ludo’.
Re: [GSoC update] Npm & guix
Jelle Licht writes: > On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it on my > npm > importer. After leaving the importer running for a quite some hours (and > making > it more robust in the face of inconsistent npm information), it turns out that > jQuery has a direct or indirect dependcy on about everything. We are talking > pretty much all of the build systems, all of the testing frameworks and all of > the test runners. Literally thousands of packages, and multiple (conflicting) > versions of most. Have you asked the jQuery developers for their view on the number of dependencies you found and some of the conflicts you see? Thanks for your great effort! Greetings, Jan -- Jan Nieuwenhuizen| GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.nl
Re: [GSoC update] Npm & guix
On Sun 24 Jul 2016 03:06, Jelle Lichtwrites: > On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it > on my npm importer. After leaving the importer running for a quite > some hours (and making it more robust in the face of inconsistent npm > information), it turns out that jQuery has a direct or indirect > dependcy on about everything. We are talking pretty much all of the > build systems, all of the testing frameworks and all of the test > runners. Literally thousands of packages, and multiple (conflicting) > versions of most. Wow, how frustrating! > This makes it IMHO a worthwhile goal bootstrap to a working test > framework, with of course at first tests disabled for the dependencies > of this test framework. Test frameworks all have an (indirect) > dependency on the Coffeescript compiler, of which the first version > was written in Ruby. Using this initial (alpha) compiler, and the > awesome git-bisect command, I was able to subsequently compile and use > the more modern (but still old) Coffeescript-in-coffeescript > compilers. For what it's worth, to me as a bystander this looks like great progress :) Andy
Re: [GSoC update] Porting Guix to GNU/Hurd
Hi! Manolis Ragkousis manolis...@gmail.com skribis: 1) Guix can successfully cross-build any package for the Hurd and produce the bootstrap-tarballs to build packages with Guix natively on such a system. 2) Guix can build the native final toolchain. 3) Guix can build packages natively using the final toolchain. Even though most of the work is already present in the wip-hurd branch, there are still some patches to be reviewed/merged. This will happen asap. But nevertheless we can safely say we have ported Guix to Hurd. :-) Congratulations on that! That’s a great achievement, and I know there were some bumps on the road, and the project covers several very different pieces of software, so thumbs up! 2) guix/build/syscalls (mount, umount, setns ) Those aren't available on Hurd, so building Guix would fail. It seems turning them into procedures is enough to work around this. This module is only used by GuixSD anyway (not Guix itself), so no problem. In the interim, we should perhaps arrange to not build that module when, say, ‘mount’ is missing (using an ‘AM_CONDITIONAL’.) 4) Perl could not be build because of a problem with memmove in hurd's glibc. Together with Samuel and Justus we traced the problem to a not properly tested part of glibc. Fixed. Pheew. :-) 5) glibc-intermediate could not find the gnumach/hurd headers because of unsetting the path during the build process. It seems that glibc's sysdeps/mach and sysdeps/mach/hurd configure weren't using the --with-headers variable to get the headers. Fixed that and sent the patch to bug-hurd :-). OK. 6) Binaries produced from gcc-boot0 had problems with their runpath and validate-runpath? was failing. It seems ld wasn't passing -rpath to them. Added an ld-wrapper, solved. Interesting, will look at the patch hopefully soon. 7) glibc-final's debug output refers to %glibc-bootstrap. I think this happens indirectly through the headers, currently solving that. Would be nice to understand why that happens. In the meantime you could disable #:allowed-references altogether for that package when on GNU/Hurd, if you want. Also, I am near finishing sys/mount.h for Hurd. Will finish it as soon as possible. Great! I get the feeling I forgot some things so please remind/ask me anything. For the next two days I will wrap up patches and push them to wip-hurd. After that I will continue working on getting the full GuixSD experience on Hurd. And one of these days we will deploy guix on darnassus together with rbraunr. I will inform you when that happens. Excellent. Thanks to the Hurd and Guix people for giving you a hand! Cheers, Ludo’.
Re: [GSoC update] Porting Guix to GNU/Hurd
Amirouche Boubekki writes: Le 2015-08-19 22:27, taylanbayi...@gmail.com a écrit: Manolis Ragkousis manolis...@gmail.com writes: [...] But nevertheless we can safely say we have ported Guix to Hurd. :-) [...] Amazing news! :-) Thanks so much for working on this. I haven't tried out Hurd so far but am meaning to do so eventually; sounds like my first Hurd system will be GuixSD at the same time. So much awesome in one system. I second that. Thanks :) +1 Congratulations on your good progress, Manolis! Alex
Re: [GSoC update] Porting Guix to GNU/Hurd
On 2015-08-19 17:27, taylanbayi...@gmail.com wrote: Manolis Ragkousis manolis...@gmail.com writes: [...] But nevertheless we can safely say we have ported Guix to Hurd. :-) [...] Amazing news! :-) Thanks so much for working on this. I haven't tried out Hurd so far but am meaning to do so eventually; sounds like my first Hurd system will be GuixSD at the same time. So much awesome in one system. Taylan Amazing! GuixSD + Hurd = 3:D Thanks Guixs 3:) 3:) = GNU Head kkk -- Daniel Pimentel (d4n1 3:)
Re: [GSoC update] Porting Guix to GNU/Hurd
On 2015-08-19 16:23, Manolis Ragkousis wrote: Hello everyone, As the hard pencil down date approaches, I believe it's time for a detailed report on my progress till now. I will start with what we have right now and then continue with what problems I had and what I did. First let's see what we have now, working. 1) Guix can successfully cross-build any package for the Hurd and produce the bootstrap-tarballs to build packages with Guix natively on such a system. 2) Guix can build the native final toolchain. 3) Guix can build packages natively using the final toolchain. Even though most of the work is already present in the wip-hurd branch, there are still some patches to be reviewed/merged. This will happen asap. But nevertheless we can safely say we have ported Guix to Hurd. :-) Now problems I have faced during my work. 1) CHROOT_ENABLED was evaluated to false in the daemon nix/libstore/build.cc. Mark pointed out the correct solution and I will send the patch shortly. 2) guix/build/syscalls (mount, umount, setns ) Those aren't available on Hurd, so building Guix would fail. It seems turning them into procedures is enough to work around this. 3) We had a problem with binaries not being able to find and link against libmachuser libhurduser It seems adding the two to libc.so's search path was enough. 4) Perl could not be build because of a problem with memmove in hurd's glibc. Together with Samuel and Justus we traced the problem to a not properly tested part of glibc. Fixed. 5) glibc-intermediate could not find the gnumach/hurd headers because of unsetting the path during the build process. It seems that glibc's sysdeps/mach and sysdeps/mach/hurd configure weren't using the --with-headers variable to get the headers. Fixed that and sent the patch to bug-hurd :-). 6) Binaries produced from gcc-boot0 had problems with their runpath and validate-runpath? was failing. It seems ld wasn't passing -rpath to them. Added an ld-wrapper, solved. 7) glibc-final's debug output refers to %glibc-bootstrap. I think this happens indirectly through the headers, currently solving that. I think those are most of the problems I faced. Well those are the ones that were worked on and solved after Ludo left for vacations. I avoid referring to problems discussed and solved in mails earlier than that. Also, I am near finishing sys/mount.h for Hurd. Will finish it as soon as possible. I get the feeling I forgot some things so please remind/ask me anything. For the next two days I will wrap up patches and push them to wip-hurd. After that I will continue working on getting the full GuixSD experience on Hurd. And one of these days we will deploy guix on darnassus together with rbraunr. I will inform you when that happens. Please feel free to ask me anything. Thank you, Manolis Congratulation Manolis, Ludo and Guixs 3:) Thanks, -- Daniel Pimentel (d4n1 3:)
Re: [GSoC update] Porting Guix to GNU/Hurd
Hello, Congrats!! Samuel
Re: [GSoC update] Porting Guix to GNU/Hurd
Le 2015-08-19 22:27, taylanbayi...@gmail.com a écrit : Manolis Ragkousis manolis...@gmail.com writes: [...] But nevertheless we can safely say we have ported Guix to Hurd. :-) [...] Amazing news! :-) Thanks so much for working on this. I haven't tried out Hurd so far but am meaning to do so eventually; sounds like my first Hurd system will be GuixSD at the same time. So much awesome in one system. I second that. Thanks :) -- Amirouche ~ amz3 ~ http://www.hyperdev.fr