> On 10. Feb 2019, at 10:36, Florian Dold <florian.d...@gmail.com> wrote: > > On 2/10/19 1:55 PM, Schanzenbach, Martin wrote: > >>> An example for such >>> tooling would be Googles's Repo tool >>> (https://source.android.com/setup/develop / >>> https://source.android.com/setup/develop/repo). >> >> >> Actually, google is an example for a proponents of monorepos. So your point >> is moot here. >> They need all this tooling _because_ they use a single repo. > > Not quite. While most of Google's code is in a monorepo (that doesn't > use git), Android is split over multiple git repos. > > Also they need all this tooling because they have many projects in one > monorepo. What I'm saying here is that we shouldn't split up one > *project* too much. Then we don't need that extra tooling! > >> Did you even read my last comment? Do you really consider all of the >> applications as one "GNUnet" that every >> user (and developer!) actually cares about? >> I can tell you the number of times I used / developed something for fs / >> social: 0. >> And, of course smaller repos make CI faster. It will result in smaller >> builds (and, more importantly, builds which actually build things that have >> changed). >> And please no arguments for stateful builds/runners. I hope we can at least >> agree that tests and builds should be done in clean environments every time. >> Else you will not catch a lot of stuff that can go wrong (I experienced this >> myself when I setup my docker builds for GNUnet which, unlike BB, actually >> build from scratch) > > Is tracking dependencies between repos really that much easier than > tracking dependencies between subdirectories? > > You are (at least I think) making the assumption that high-level GNUnet > repos would depend on stable versions of GNUnet? > > Otherwise, the cost for CI would be the same. If I make a change in > GNUnet-base, of course I'll also have to run tests for everything that > depends on it. Are you suggesting to just not do integration testing > anymore?
I don't see why this is so hard to understand. Of course still do testing. And a successful build of the code _plus_ a successful test of the core should trigger the build+tests of depending apps. The gist is: 1. It only triggers if the build+test of the core actually pass 2. The triggered component (e.g. reclaim) does not care about failing builds/tests in fs. It will still work without it. > >> How would that happen? Can you give a _concrete_ example in >> fs/social/reclaim where this is true? >> It is exactly the point that it is completely unclear what effects a >> configuration switch on what components. >> If we separate this, we might get _some_ overhead in the configure.ac's but >> from my experience I expect this to be very little. >> And our configure.ac is not something I would consider "developer friendly" >> through its sheer size and complexity. > > Let's say I want to enable verbose logging. Then I need to re-configure > and re-compile all of the repos I depend on, in the right order. That > sucks! > > (Re-linking doesn't suffice, as logging is done via CPP macros.) Not really. If I build reclaim, I will probably use a docker image, say gnunet:latest for my debugs builds and gunet:<version> for my release builds which have logging enabled/disabled. This is pretty easy to ensure. As I said, you are thinking in too much in "every component in own repo". > >> And think about it this way: If a new developer decides to write a new >> service / application on top of GNUnet, what will he be faced with? >> Image this app needs only GNS and maybe CADET. >> The dev will need to integrate and understand the full build and test in >> order to properly setup this project. >> If we had a few separate examples of how this can be done in a separate >> repo, this would go a long way. > > I don't see any issues with "incubating" new GNUnet > applications/libraries in their own repo. Maybe some of them can then > be later integrated with the main gnunet.git. At this point, there is not benefit in integrating it back. > > (I just saw that in your latest email, you're suggesting exactly this > for reclaimid. That's is fine, but of course now makes it hard to see > for other GNUnet devs locally whether they are breaking your code. > Well, at least the CI should eventually complain.) That is also the point. They should not care. Do you really think Gtk+ devs care if they break API/ABI and gnunet-gtk fails to build? > > - Florian > >>> On 2/10/19 1:02 AM, Amirouche Boubekki wrote: >>>> I think splitting the codebase will be a pain for gnunet. >>>> >>>> The only *good* reasons for manyrepos are social or ego politics "this >>>> is my lawn" or legal. The only one that applies to gnunet is legal >>>> because one needs to fill a gnu form to be able to contribute. >>>> >>>> I am biased toward monorepo by experience dealing with big project >>>> (100k+ SLOC) and the only time it made sens to split the project into >>>> many repositories because it was different teams / workflow (social) and >>>> different legal terms for the various services/daemons, at previous >>>> $WORK, they had to fork gentoo to make it work. >>>> >>>> Otherwise, each time I saw another repository it was a source of pain: >>>> >>>> - Need to manage several versions >>>> - git submodule workflow is not good enough, it doesn't track branch, I >>>> personally I never remember how to know the branch of a commit, plus it >>>> requires some more git-fu to bump the submodule. >>>> - refactoring anyone? >>>> - generally speaking manyrepos at small scale is more work >>>> >>>> And again, it requires somehow to track down every versions (what works >>>> with what) and you end up with another repository (or distribution) with >>>> another build system that puts everything together. Continuous >>>> Integration can do that? Where is the code of the CI? Another repo? More >>>> versions, more git clone more grep across repositories / directories not >>>> even in sync. >>>> >>>> Popularity arguments: >>>> >>>> a) Ok, everybody know GAFAM love monorepos and that is a also a source >>>> of pain (dedicated team and software). That said, gnunet is not the size >>>> of any GAFAM, hence it will not suffer from monorepo pain points. >>>> >>>> b) Github and Javascript made the manyrepos popular for various ego >>>> reasons and because JavaScript is not good. I won't take inspiration >>>> from that part of the JavaScript noosphere. gnunet-leftpad anyone? >>>> >>>> c) Now, there is GNOME. GNOME is famous for its bazaar model of >>>> development and also famous for the adoption of meson (maybe even its >>>> inception) or its previous incarnation jhbuild. Anyway, even if GNOME >>>> and GNU (which is also a bazaar) success is appealing, gnunet is not GNU >>>> or GNOME. From my point of view the bazaar development model scales >>>> better / more easily in a socially distributed setting. Also why Linux >>>> is still a single repository? >>>> >>>> Le sam. 9 févr. 2019 à 18:16, Schanzenbach, Martin >>>> <mschanzenb...@posteo.de <mailto:mschanzenb...@posteo.de>> a écrit : >>>> >>>> >>>> >>>>> On 9. Feb 2019, at 17:13, Christian Grothoff <groth...@gnunet.org >>>> <mailto:groth...@gnunet.org>> wrote: >>>>> >>>>> On 2/9/19 5:04 PM, Schanzenbach, Martin wrote: >>>>>> I have some inline comments as well below, but let us bring this >>>> discussion down to a more practical consensus maybe. >>>>>> I think we are arguing too much in the extremes and that is not >>>> helpful. I am not saying we should compartmentalise >>>>>> GNUnet into the tiniest possible components. >>>>>> It's just that I think it is becoming a bit bloated. >>>>>> >>>>>> That being said, _most_ of what is in GNUnet today is perfectly >>>> fine in a single repo and package. >>>>>> For now, at least let us not add another one (gtk) as well? >>>>>> >>>>>> Then, we remain with >>>>>> >>>>>> - reclaim (+the things reclaim needs wrt libraries) >>>>>> - conversation (+X) >>>>>> - secureshare (+X) >>>>>> - fs (+X) >>>>>> >>>>>> as components/services on my personal "list". >>>>>> I suggest that _if_ I find the time, I could extract reclaim into >>>> a separate repo as soon as we have a CI and I can >>>>>> test how it works and we can learn from the experience. >>>>>> Then, we can discuss if we want to do the same with other >>>> components, one at a time, if there is consensus and a person that >>>>>> would be willing to take ownership (I am pretty sure we talked >>>> about this concept last summer as well). >>>>> >>>>> Maybe you could start with extracting the SecuShare components? That >>>>> should do for a first "experience", and be a bit more effective at >>>>> reducing bloat as well ;-). >>>> >>>> Well, I could, but our secushare people are quite active so maybe >>>> there are volunteers (if they agree with the proposal at all). >>>> Regarding "bloat". If we want to effectively eliminate bloat than >>>> let's look at numbers: >>>> >>>> File Sharing: >>>> src/fs: 36918 (!) LOC in .c files >>>> src/datastore/cache: ~15k LOC in .c files >>>> >>>> Conversation: >>>> src/conversation: 10538 LOC in .c files >>>> >>>> SecuShare: >>>> src/psyc* : ~17000 LOC in .c files (altough I am not sure about this >>>> because theoretically psyc is a general use protocol, no?) >>>> src/social: 9447 LOC in .c files >>>> src/multicast: 5633 LOC in .c files >>>> >>>> Reclaim: >>>> src/reclaim* : ~6500 LOC in .c files >>>> >>>> Now, considering that fs is practically always built for everybody >>>> and SecuShare and reclaim are experimental, it hurts the most for >>>> devs that actually compile from source. >>>> Everything combined are 110000+ LOC which is 22% of the codebase >>>> (~500k, oO). Considering that there is a significant redundancy in >>>> transport/ (75k) at the moment, this number is probably closer to 25%. >>>> Granted, this is a lot less than I expected ;), but maybe >>>> illustrates the dimensions. >>>> >>>> >>>>> >>>>> That said, splitting of reclaim seems also much less problematic than >>>>> fs/conversation, and if you then integrate reclaim with the libgabe >>>>> tree, the overall number of downloads/installation for reclaim >>>> wouldn't >>>>> go up, so that would certainly kill my argument of making the >>>>> installation more complex (might indeed simplify it, as one >>>> doesn't have >>>>> to remember to install libgabe before GNUnet to get reclaim). >>>> >>>> Could do, but libgabe has some nasty additional deps (libpbc and >>>> gmp) which we _might_ eventually get rid of completely by >>>> implementing GNS-based encryption. >>>> >>>> >>>> _______________________________________________ >>>> GNUnet-developers mailing list >>>> GNUnet-developers@gnu.org <mailto:GNUnet-developers@gnu.org> >>>> https://lists.gnu.org/mailman/listinfo/gnunet-developers >>>> >>>> >>>> _______________________________________________ >>>> GNUnet-developers mailing list >>>> GNUnet-developers@gnu.org >>>> https://lists.gnu.org/mailman/listinfo/gnunet-developers >>>> >>> >>> _______________________________________________ >>> GNUnet-developers mailing list >>> GNUnet-developers@gnu.org >>> https://lists.gnu.org/mailman/listinfo/gnunet-developers >> >
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ GNUnet-developers mailing list GNUnet-developers@gnu.org https://lists.gnu.org/mailman/listinfo/gnunet-developers