Hello Catonano, 2017-01-19 21:48 GMT+01:00 Catonano <caton...@gmail.com>:
> I made a crawler and I let it loose on the jquery 3.1.1 dependencies on > registry.npmjs.com > It recursevely fetched the dependencies of jquery 3.1.1, then the > dependencies of the dependencies, then the dependencies of the dependencies > of the dependencies... and so on > > Until there were no more dependencies to fetch. > > It stored all such dependencies in a graph database made by the amazing > amz3 for their project, Culturia. They wrote about such project several > times on the guile-users mailing list. > > It took days to download. And it took me months to produce a working > version of this code. Because I don't know Guile very well and maybe I'm > not that smart overall. > > Anyway, now I have a COMPLETE graph of the dependencies of jquery 3.1.1 > These pictures are very informative indeed. I will try to be brief, but I quickly wanted to share that I find your efforts (and results!) amazing. > > It's made of > 47311 vertices and > 324569 edges > > I made a graph of a subset (graphviz chocked a bit on this ;-) ) > It's here > http://catonano.altervista.org/grafo.svg > > There are 448 "broken" packages. Probably I should explain how I > classified packages as "broken". Maybe not in this email. > > Anyway, these broken packages pose a challenge to the mission of porting > Jquery into Guix, in my opinion, and they should be considered with some > attention. > Here's the list > http://catonano.altervista.org/broken_packages.txt > Something of note: a big chunk of the packages you classified as being 'broken' are making me recall some (unpleasant) memories; From my own crawling experiments, which were not nearly as complete as this one, I also ran into a lot of the same show-stoppers. In a very real sense, resolving node dependencies quickly devolves into resolving dependencies for most of the popular build systems as well as plugins like broccoli-funnel. What I am trying to get at here is that fixing and packaging these 448 packages will likely contribute a lot towards making an npm-enriched guix possible. > > > There are 1314 packages with NO dependencies that could be used as > starting points in porting Jquery into Guix. > Here's the list > http://catonano.altervista.org/broken_packages.txt > These could probably make use of the npm importer I worked on earlier. Do you make use of your own? Otherwase I'll get to rebasing my version on the current guix master branch. > > > If there's anyone interested, I can give you the data folder so you can > try all the queries you want on these data without having to to run this > thing for a bunch of hours > If possible, yes please. What would be the most convenient way you to share this data? > > In the future, I'd like to run this thing on some other package and merge > the graphs so I will be able to investigate which are the common > fundamental dependencies for SEVERAL important packages in Nodejs. > > So if someone wants to dedicate time to porting Nodejs stuff in Guix they > will be able to select most urgent packages to start from. > > The same could be said of broken packages taht affect several important > packages. > > The porting of Nodejs in Guix cannot be done with brute strength. A data > oriented approach can help, in my opinion. > > Indeed. > The ideal would be to have something that, like bitcoin, coordinates a > swarm in such a way that every node can contribute a tiniy bit of data to a > common data structure, so all the nodes would have a complete copy of the > database. > > Collecting a mantaining of datasets should be freed of the client server > model too. Not only the social media. > > I have no idea what you are referring to. Could you please elaborate a bit at a later point in time? > But that's more than I can handle, anyway. > I am already thankful > > I'd like to talk about the stumbling blocks I run into to discuss Guile > and my knowledge of it. > > For example, I can't use that thing in the autotools that processes > configure.am files so I just forked amz3's project and added my files in > there. As guests. Thanks amz3 ! > > I'd also like to describe te screw ups in the format I put the data into. > I realized my mistakes when hours of crunching had already been done. > > They can be migrated to a better format though > > If you don't mind, I will discuss these issue in the future, not now. > > The code is here > https://gitlab.com/humanitiesNerd/Culturia > > One last fun fact: while I was watching the output flowing in my terminal, > I saw a package called > > "broccoli-funnel" > > No, really. It's here > https://registry.npmjs.org/broccoli-funnel > > Ok, that's all for now. > Thanks again for your efforts on this. I am looking forward to working with your data. Regards, Jelle Licht
