On Monday 17 January 2005 19:23, Stephen P. Becker wrote: > Dan Armak wrote: > > On Monday 17 January 2005 17:18, Stephen P. Becker wrote: > >>Adding some 15mb of ebuilds to portage just for one desktop environment > >>isn't bloat? > > > > Any 350 ebuilds would take that much. Nothing I can do about it. Most of > > them don't even have function defs thanks to eclasses, so they can't > > become any smaller. > > And that is my point. 350 ebuilds that do the same thing that 12 did > before is bloat. They do more than the monolithic ebuilds. The major new features are: - Being able to install selected apps (see below about why DO_NOT_COMPILE can't really do this) - Having to recompile anywhere between 1/5 and 1/2 of KDE on a bugfix version, as opposed to all of it - Having to recompile far less for any new revisions of our own, which is especially important for rapid testing and installation of security fixes - More flexibility with USE flags (which are due for a big overhaul) - in monolithic ebuilds, we'd have to make a lot of USE flags affect each other, which is bad
We didn't go to all this trouble just for the fun of it... > > >>Particularly when the ebuilds already support splitting > >>themselves in a different way via an environment variable? > > > > They DO NOT support any such thing. The split ebuilds were made so that > > people wouldn't have to use the horrible DO_NOT_COMPILE. It's at least > > halfway to installing manually from source. > > Why is DO_NOT_COMPILE so horrible? 1. It completely breaks portage dep tracking. Sure you can track manually what you have installed, but then what's a package manager for? 2. It forces the user to know exactly what subdirs exist so that he can disable them. Imagine the average user learning the names and functions of all those subdirs, many of them libraries or even staic-only libraries that you never see installed. Next release there'll be a new app and he won't know about it and compile it by mistake. Sure, he can track all that manually too, but then what are packagers for? :-) 3. It wasn't meant to be used from the env to begin with, so it can break things. Fex., subdirs can have dependencies between them or need a particular build order. Sure, you could ask your users to spend a few hours on studying KDE's build system, but again, that's a bit like telling them to compile from source... 4. If I installed kmail yesterday and want to add korn today, I can't. I'll have to recompile kmail again together with korn. Ditto for removing apps. This alone kills any speed advantage, unless you always decide up ahead just what you want. 5. DO_NOT_COMPILE can't selectively disable second-level subdirs from compilation. 6. DO_NOT_COMPILE can't be stored per-package, so you have to do that manually. You can't set a systemwide value because some KDE packages have subdirs with identical names but different contents. (At least I think there are such... But even if I'm wrong here, a systemwide value for this is ugly.) 7. You can't make precompiled packages with it. Well you can, but you wouldn't be able to tell what each package provided using standard portage tools. And you couldn't distinguish between two packages. And you couldn't provide a separate package for every app, because the tarballs would have the same name. I could probably go on if I had to... > With that, people had to track down > which programs they didn't want and add them to that variable. Yes, and that's very bad. > With > split ebuilds, people will have to track down which ebuilds they don't > want, bypass the meta-ebuild that installs everything, and then either > merge what they want one-by-one or maintain their own meta-ebuild. Merging one-by-one is a lot easier than maintaining DO_NOT_COMPILE lists. Portage remembers what you've got installed, and doesn't try to install new things when they appear as part of a new release. The real solution to this, though, should be portage-side not ebuild-side: portage 'sets' support (I hope I'm not mixing up GLEPs; I mean the feature that would let us create lightweight lists of ebuilds instead of -meta ebuilds). If a -meta ebuild can be created that is smaller than kde-meta and makes sense for your users, go ahead and create it. If you can't, and the choice is highly personal, there's nothing we can do. To my way of thinking, it makes a lot more sense (and is more Gentooish) for people to manage a list of what they want, than a list of they do not want. > real 6m50.818s > user 2m34.113s > sys 0m50.345s > > Now, multiply that by 40 for kdebase, or by 350 for the whole taco, and > you can see why I'm concerned. Yes, but my calculation below (0.5MB * 350 packages) gives far more reasonable times. > >>Or, have you figured out a way to get around that? > > > > Well, we only extract the necessary files from the tarballs each time. > > I guess there isn't a way to come up with some sort of portage hackery > whereby the portage tmpdir doesn't get cleaned out until the last of the > split ebuilds is merged? Certainly not from the ebuild side. And the portage people were completely uninterested when I asked them long ago :-) But it's an interesting problem. If the portage side provided eg readonly access to an unpacked kde source tree, the ebuilds could build everything out-of-tree. Or we could create a tree of hardlinks and build inside that, since we don't have write access to those hardlinks. (This speedup would apply to the monolithic ebuilds, too.) I'd be willing to try implementing it if I was convinced that the unpacking was the main overhead introduced by increasing the amount of ebuilds. Right now the main overhead is running configure multiple times (how long does that take you?), so I want to see confcache get here first, and meanwhile work on other speedups like unsermake, and when all that's done I'll try out this idea. > > > Of course we could break up every KDE tarball on release and distribute > > lots of small tarballs on mirror://gentoo. I don't particularly mind if > > the extra mirror load isn't a problem and if it will help you. There'll > > always be some overhead, but not too much I think. Suppose an average of > > 0.5MB .tar.bz2 with 100 files per package (to take filesystem costs into > > account), with 350 packages total - how much would all the unpacking (and > > deleting each package workdir) take on an average mips? (I estimate these > > numbers are all somewhat higher than the real averages.) > > This would be better than unpacking the huge kde source tarballs every > time, but then it would bloat the mirrors somewhat. Not much. The tarballs for a KDE release (except i18n) are a total of 130-140MB in size. I remember hearing the total size of distfiles/ on a mirror is well over 100GB, so a few hundred MBs isn't a big increase. > > > BTW, what kind of RAM size do mips boxes have? If by any chance it's > > large compared to their CPU power - if we consider x86 the norm - then > > this is a good moment to mention I wanted to try using enable-final again > > :-) But if not, we'll just disable it on your arch. > > We tell people not to bother with gentoo unless they have at least 128mb > of RAM. Indys max out at 256mb, and some of the more powerful > machines like O2 can have up to 1gb now (thanks to iluxa). enable-final isn't really doable on anything under 512MB, and a gigabyte is nice. IIRC there's an eclass now that checks these kind of things. Well, we'll get to that later... -- Dan Armak Gentoo Linux developer (KDE) Public GPG key: http://dev.gentoo.org/~danarmak/danarmak-gpg-public.key Fingerprint: DD70 DBF9 E3D4 6CB9 2FDD 0069 508D 9143 8D5F 8951
pgpJQz7n2PaD5.pgp
Description: PGP signature
