Re: Packaging problems
Still, using the proposed high-level categorisation of GNUnet components would be handy for documentation purposes, even if the repo structure only vaguely reflects it, as a lot of service names aren't very indicative of their function or importance to the uninitiated. I'm taking after the Python standard library docs in that respect, where modules are categorised into broad domains first, before being sorted within those categories. On 03/06/2022 20:44, Christian Grothoff wrote: Having many packages doesn't usually make it easier for packagers, it just means that now they have to deal with even more sources, and create more package specifications. Moreover, build times go up, as you now need to run configure many times. Worse, you then need to find out in which order to build things, and what are dependencies. It basically makes it worse in all aspects. Another big issue is that right now, I at least notice if I break the build of an application and can fix it. Same if I run analysis tools: they at least get to see the entire codebase, and can warn us if something breaks. If we move those out-of-tree, they'll be even more neglected. What we can (and do do) is mark really badly broken applications as 'experimental' and require --with-experimental to build those. That's IMO better than moving stuff out of tree. Also, you probably don't want to split things as you proposed: GNS depends on VPN and SETU! SET is supposed to become obsolete, but consensus still needs it until SETU is extended to match the SET capabilities. Finally, as for build times, have you even tried 'make -j 16' or something like that? Multicore rules ;-). Happy hacking! Christian On 6/2/22 17:29, Willow Liquorice wrote: Right. Perhaps the onus is on the developers (i.e. us) to make things a bit easier, then? To be honest, I barely understand how the GNUnet project is put together on a source code level, let alone how packaging is done. One of the things I'm going to do with the Sphinx docs is provide a high-level overview of how the main repo is structured. On the subject of complexity, I attempted to disentangle that awful internal dependency graph a while ago, to get a better idea of how GNUnet works. I noticed that it's possible to divide the subsystems up into closely-related groups: * a "backbone" (CADET, DHT, CORE, and friends), * a VPN suite, * a GNS suite, * and a set operations suite (SET, SETI, SETU). A bunch of smaller "application layer" things (psyc+social+secushare, conversation, fs, secretsharing+consensus+voting) then rest on top of one or more of those suites. I seem to recall that breaking up the main repo has been discussed before, and I think it got nowhere because no agreement was reached on where the breaks should be made. My position is that those "applications" (which, IIRC, are in various states of "barely maintained") should be moved to their own repos, and the main repo be broken up into those four software suites. As Maxime says, GNUnet takes a long time to compile (when it actually does - I'm having problems with that right now), and presumably quite a while to test too. The obvious way to reduce those times is to simply *reduce the amount of code being compiled and tested*. Breaking up the big repo would achieve that quite nicely. More specifically related to packaging, would it be a good idea to look into CD (Continuous Delivery) to complement our current CI setup? It could make things easier on package maintainers. Looks like Debian has a CI system we might be able to make use of, and all we'd need to do is point out the test suite in the package that goes to the Debian archive.
Re: Packaging problems
On 6/3/22 22:37, Willow Liquorice wrote: > Alright, fair point. > > Still, more automated testing/packaging isn't a bad thing. What exactly > does the CI do, right now? I looked in .buildbot in the main repo, and I > guess it just tries to build and install GNUnet from source on whatever > OS hosts Buildbot? Couldn't see that much automated testing/packaging. > > I'll say again that not having GNUnet running on Debian's CI is a big > missed opportunity. Being able to deploy and test on Debian Unstable > automatically would surely make it easier to keep the Debian package up > to date. Absolutely. Having the CI do more would be great. > I'm not sure about the exact process, but I get the impression from > reading about the subject that it could just be a matter of creating a > new version, which could trigger building Debian packages, which then go > to Debian Unstable, and are then used with autopkgtest on ci.debian.net. Who can do this? Anyone, or only a DD? Our DD has been somewhat, eh, distracted recently... Cheers! Christian > Best wishes, > Willow > > On 03/06/2022 20:44, Christian Grothoff wrote: >> Having many packages doesn't usually make it easier for packagers, it >> just means that now they have to deal with even more sources, and >> create more package specifications. Moreover, build times go up, as >> you now need to run configure many times. Worse, you then need to find >> out in which order to build things, and what are dependencies. It >> basically makes it worse in all aspects. >> >> Another big issue is that right now, I at least notice if I break the >> build of an application and can fix it. Same if I run analysis tools: >> they at least get to see the entire codebase, and can warn us if >> something breaks. If we move those out-of-tree, they'll be even more >> neglected. What we can (and do do) is mark really badly broken >> applications as 'experimental' and require --with-experimental to >> build those. That's IMO better than moving stuff out of tree. >> >> Also, you probably don't want to split things as you proposed: GNS >> depends on VPN and SETU! SET is supposed to become obsolete, but >> consensus still needs it until SETU is extended to match the SET >> capabilities. >> >> Finally, as for build times, have you even tried 'make -j 16' or >> something like that? Multicore rules ;-). >> >> Happy hacking! >> >> Christian >> >> >> On 6/2/22 17:29, Willow Liquorice wrote: >>> Right. Perhaps the onus is on the developers (i.e. us) to make things >>> a bit easier, then? >>> >>> To be honest, I barely understand how the GNUnet project is put >>> together on a source code level, let alone how packaging is done. One >>> of the things I'm going to do with the Sphinx docs is provide a >>> high-level overview of how the main repo is structured. >>> >>> On the subject of complexity, I attempted to disentangle that awful >>> internal dependency graph a while ago, to get a better idea of how >>> GNUnet works. I noticed that it's possible to divide the subsystems >>> up into closely-related groups: >>> * a "backbone" (CADET, DHT, CORE, and friends), >>> * a VPN suite, >>> * a GNS suite, >>> * and a set operations suite (SET, SETI, SETU). >>> >>> A bunch of smaller "application layer" things (psyc+social+secushare, >>> conversation, fs, secretsharing+consensus+voting) then rest on top of >>> one or more of those suites. >>> >>> I seem to recall that breaking up the main repo has been discussed >>> before, and I think it got nowhere because no agreement was reached >>> on where the breaks should be made. My position is that those >>> "applications" (which, IIRC, are in various states of "barely >>> maintained") should be moved to their own repos, and the main repo be >>> broken up into those four software suites. >>> >>> As Maxime says, GNUnet takes a long time to compile (when it actually >>> does - I'm having problems with that right now), and presumably quite >>> a while to test too. The obvious way to reduce those times is to >>> simply *reduce the amount of code being compiled and tested*. >>> Breaking up the big repo would achieve that quite nicely. >>> >>> More specifically related to packaging, would it be a good idea to >>> look into CD (Continuous Delivery) to complement our current CI >>> setup? It could make things easier on package maintainers. Looks like >>> Debian has a CI system we might be able to make use of, and all we'd >>> need to do is point out the test suite in the package that goes to >>> the Debian archive. >>> >>> >> >
Re: Packaging problems
On 6/4/22 09:43, Schanzenbach, Martin wrote: > > >> On 3. Jun 2022, at 21:44, Christian Grothoff wrote: >> >> Having many packages doesn't usually make it easier for packagers, it just >> means that now they have to deal with even more sources, and create more >> package specifications. Moreover, build times go up, as you now need to run >> configure many times. Worse, you then need to find out in which order to >> build things, and what are dependencies. It basically makes it worse in all >> aspects. > > Well. I would think this suggests a very badly designed packaging tool. > Even the extremely old RPM format allows to build once and then make packages > from any subset of the built binaries. > That is how our gnunet rpm in the tree works. Oh, I'm talking about having many TGZ, not about generating multiple binary packages from one master source. We do that already for the GNUnet Debian packages for which we have rules in-tree.
Re: Packaging problems
> On 3. Jun 2022, at 21:44, Christian Grothoff wrote: > > Having many packages doesn't usually make it easier for packagers, it just > means that now they have to deal with even more sources, and create more > package specifications. Moreover, build times go up, as you now need to run > configure many times. Worse, you then need to find out in which order to > build things, and what are dependencies. It basically makes it worse in all > aspects. Well. I would think this suggests a very badly designed packaging tool. Even the extremely old RPM format allows to build once and then make packages from any subset of the built binaries. That is how our gnunet rpm in the tree works. > > Another big issue is that right now, I at least notice if I break the build > of an application and can fix it. Same if I run analysis tools: they at least > get to see the entire codebase, and can warn us if something breaks. If we > move those out-of-tree, they'll be even more neglected. What we can (and do > do) is mark really badly broken applications as 'experimental' and require > --with-experimental to build those. That's IMO better than moving stuff out > of tree. > That is the big issue I agree. BR > Also, you probably don't want to split things as you proposed: GNS depends on > VPN and SETU! SET is supposed to become obsolete, but consensus still needs > it until SETU is extended to match the SET capabilities. > > Finally, as for build times, have you even tried 'make -j 16' or something > like that? Multicore rules ;-). > > Happy hacking! > > Christian > > > On 6/2/22 17:29, Willow Liquorice wrote: >> Right. Perhaps the onus is on the developers (i.e. us) to make things a bit >> easier, then? >> To be honest, I barely understand how the GNUnet project is put together on >> a source code level, let alone how packaging is done. One of the things I'm >> going to do with the Sphinx docs is provide a high-level overview of how the >> main repo is structured. >> On the subject of complexity, I attempted to disentangle that awful internal >> dependency graph a while ago, to get a better idea of how GNUnet works. I >> noticed that it's possible to divide the subsystems up into closely-related >> groups: >> * a "backbone" (CADET, DHT, CORE, and friends), >> * a VPN suite, >> * a GNS suite, >> * and a set operations suite (SET, SETI, SETU). >> A bunch of smaller "application layer" things (psyc+social+secushare, >> conversation, fs, secretsharing+consensus+voting) then rest on top of one or >> more of those suites. >> I seem to recall that breaking up the main repo has been discussed before, >> and I think it got nowhere because no agreement was reached on where the >> breaks should be made. My position is that those "applications" (which, >> IIRC, are in various states of "barely maintained") should be moved to their >> own repos, and the main repo be broken up into those four software suites. >> As Maxime says, GNUnet takes a long time to compile (when it actually does - >> I'm having problems with that right now), and presumably quite a while to >> test too. The obvious way to reduce those times is to simply *reduce the >> amount of code being compiled and tested*. Breaking up the big repo would >> achieve that quite nicely. >> More specifically related to packaging, would it be a good idea to look into >> CD (Continuous Delivery) to complement our current CI setup? It could make >> things easier on package maintainers. Looks like Debian has a CI system we >> might be able to make use of, and all we'd need to do is point out the test >> suite in the package that goes to the Debian archive. > signature.asc Description: Message signed with OpenPGP
