Re: [gentoo-dev] Re: better support for binary packages
For the record: Fabio (lxnay) and I had a chat on IRC and cleared some things up. He also reproduced the problem I had encountered with mixed portage/entropy usage, and concluded that it was a bug in entropy. I'm glad we got to a constructive point. So things turn out not to be as bad as I was initially led to believe, and I'll give the entropy approach the benefit of the doubt. Cheers, -- Ben de Groot Gentoo Linux developer (qt, media, lxde, desktop-misc) Gentoo Linux Release Engineering PR liaison __
Re: [gentoo-dev] Re: better support for binary packages
On Tue, May 26, 2009 at 7:08 PM, Ben de Groot wrote: lx...@sabayonlinux.org wrote: On Tue, May 26, 2009 at 9:05 AM, Philipp Riegger wrote: See also the discussion about mixing package managers between Gentoo and Sabayon. I do not want these problems. incorrect. Give it a spin ;) Problems we have were *only* related to Portage world file handling, fixed some time ago. I am sorry to say that the issue reported here doesn't seem to be valid. Of course, if you mix both, you need to pay attention to not change USE flags (for eg.) that trigger libraries compilation, but that's a known binary-world problem. So are we going to discuss this or not? To quote your own words back at you: This is gentoo-dev and you are OFF TOPIC. Next time, please post Sabayon specific stuff on our ML/com. channels. ... My answer was completely in-topic. Even because it was just an answer. Your rants were totally OT, OTOH. *If* you want to promote entropy/equo (or other sabayon work) as a possible solution here, you should be open to discuss its shortcomings. If not, then you should refrain from bringing it up again. If you are just trying to annoy me, give up. It's not working ;) I was just answering, my friend. To get back on topic: I think portage's current binary support works reasonably well, based on some experience I have with building packages for a second, slower machine. But I can see there are shortcomings, mainly in the described problem of storing multiple versions of a binpkg (with different useflags etc.). I agree with Duncan that we do not want a change of focus away from being a source-based distribution, but then that is not a change you would be able to "sell" to the current developers anyway. That said, there could very well be a Gentoo project, or people contributing to portage development, to try and improve binary package support. -- Ben de Groot Gentoo Linux developer (qt, media, lxde, desktop-misc) Gentoo Linux Release Engineering PR liaison __ -- Fabio Erculiani signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: better support for binary packages
lx...@sabayonlinux.org wrote: > On Tue, May 26, 2009 at 9:05 AM, Philipp Riegger > wrote: >> See also the discussion about mixing package managers >> between Gentoo and Sabayon. I do not want these problems. > > incorrect. Give it a spin ;) > Problems we have were *only* related to Portage world file handling, > fixed some time ago. I am sorry to say that the issue reported here > doesn't seem to be valid. > Of course, if you mix both, you need to pay attention to not change USE > flags (for eg.) that trigger libraries compilation, but that's a known > binary-world problem. So are we going to discuss this or not? To quote your own words back at you: > This is gentoo-dev and you are OFF TOPIC. > Next time, please post Sabayon specific stuff on our ML/com. channels. ... *If* you want to promote entropy/equo (or other sabayon work) as a possible solution here, you should be open to discuss its shortcomings. If not, then you should refrain from bringing it up again. To get back on topic: I think portage's current binary support works reasonably well, based on some experience I have with building packages for a second, slower machine. But I can see there are shortcomings, mainly in the described problem of storing multiple versions of a binpkg (with different useflags etc.). I agree with Duncan that we do not want a change of focus away from being a source-based distribution, but then that is not a change you would be able to "sell" to the current developers anyway. That said, there could very well be a Gentoo project, or people contributing to portage development, to try and improve binary package support. -- Ben de Groot Gentoo Linux developer (qt, media, lxde, desktop-misc) Gentoo Linux Release Engineering PR liaison __
[gentoo-dev] Re: better support for binary packages
Philipp Riegger posted 1243335643.9661.46.ca...@hspc30.informatik.uni-stuttgart.de, excerpted below, on Tue, 26 May 2009 13:00:43 +0200: > Bit it seems to be quite an uninteresting topic, since the people most > affected by it (Gentoo developers) did not join the conversation, yet. > Maybe I should take this to gentoo-server@ and gentoo-portage@, it might > fit there. Agreed on the participation observation and taking it elsewhere, both. I'd think the gentoo-portage-dev list (which I also read) would be a good place for hopefully more discussion, with people actually interested. I still think it's likely better, at least at first, as a separate "helper" app, but a number of such helpers have ultimately been integrated into either portage itself, or into gentoolkit over time. Also, by doing it that way rather than by trying to change Gentoo as a whole, you avoid the prospect of /years/ of debate that has occurred over GLEP55 and with it 54, which also set about to change the package naming conventions, in this case for ebuilds. And given that PMS specifically defines binary package formats as out of its domain, I really do see that as the more practical approach... unless of course you /want/ to debate it for /years/ before anything gets done. =:^\ Then as it proves its value, it'll ultimately become the de-facto standard and be integrated into some future version of PMS or whatever. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Re: [gentoo-dev] Re: better support for binary packages
On Tue, 2009-05-26 at 08:48 +, Duncan wrote: > Philipp Riegger posted > 1243321504.9661.14.ca...@hspc30.informatik.uni-stuttgart.de, excerpted > below, on Tue, 26 May 2009 09:05:03 +0200: > > > I don't see the connection between the email Fabio wrote and your > > answer. Do you want to say, that you agree that he's doing what i > > described and that it works the way i described it? I doubt it. If you > > really care, could you answer my first email and state there the > > problems you see with the approach and why you think this is making > > Gentoo worse? > > I agree that an independent approach is the way to go. Gentoo (or > rather, its PMs, package managers, all three of them, of which portage is > only one) doesn't do binaries really well, certainly, but I really don't > see that changing within Gentoo itself, except at the expense of from- > source, which it does quite well indeed. Again, I see this as completely independent "features". Binary packages would only support the building from source and binary packages could never be created without ebuilds. This might even improve the quality of the ebuilds in the tree, because they would get built from time to time, build failures would get reported and all that. > > But how would you make it worse? It already has the functionality to add > > repositories for binpackages, why not improve it? Why leave it the way > > it is? > > Sure, improve it, but what you are talking about isn't a simple > improvement, but a massive rearchitecting. That's not easily doable > without a chance of focus. It's that change of focus that would > eventually kill the quality from-source support we have and that > continues to develop, now. Are you sure? How code would it take to look for a binary package in a different path than now? 1 additional line or 1 line exchanged? How much code would it take to create this packed bit vector out of the USE-flags? 5-10 lines? Changing portage to _also_ support the new kind of binary packages might be done in 1 day. And this is the first step. > >> That said, I could envision an eselect like "binary profile" switcher, > >> that subject to settings in a config file, changes USE flags, CFLAGS, > >> gcc- configs an appropriate gcc version, etc, changing PKGDIR > >> appropriately as well, so one could easily enough create as many > >> "binary profiles" as desired and as the use case dictated. > > > Not sure, what the binary profile switcher really would change here. > > What I was talking about were mostly USE-flags and packages, and I guess > > your binary profile switcher doesn't change too much, there. > > Sure it does, but incrementally. Basically, your proposal laid out a > complicated tree based on metadata, a tree the package manager would have > to understand and support, what I'm proposing is to allow the same thing, > but not by adding all that complication to the package manager, but > rather, by using a newly created application to switch among the branches > of that tree on request. Which might work, but would lead to a really strange package tree with really big restrictions and disadvantages (no package sharing, no way of finding out if an update is necessary,...). > Portage (or other PM) would use its configured > PKGDIR and wouldn't understand the tree, but it wouldn't need to, because > the switcher would manage switching between the branches according to its > configuration. And there the flexibility goes. If you want to emerge PHP and there is a binary package with the same USE-flage but also cli enabled or something like that, it would really get complicated to inform you (the user) that a simple and probably not too important change for you might save you an hour of compilation. > The result would be that in a large enough deployment to have build- > servers, the build server(s) could build a particular set of CFLAGS/USE- > flags/gcc-version/arch/whatever, then switch to another branch and build > that set. Individual package clients could simply point at the > appropriate tree and get most of their packages, at least the common > ones, that way. > > Now this wouldn't directly give you the flexibility of the package name > changes you proposed, but it could do so indirectly, putting that > information in the directory tree if configured to do so. Clients may > need to use the binprofile switcher as well, for individual packages they > chose to build outside their normal USE profile, but the process could be > automated once properly configured, using PM hooks as appropriate. And as soon as you use PM hooks, you ask yourself, why you did it that way, if not that change had made it easier, if it would have been better to discuss the issue with portage/pkgcore/paludis developers and all that. Since I really like PM hooks, I'm not sure if they are really helpfull for users. They are too cryptic. Maybe it would be wise to create some eselect like thing for those hooks prov
Re: [gentoo-dev] Re: better support for binary packages
On Tue, 2009-05-26 at 11:27 +0200, lx...@sabayonlinux.org wrote: > On Tue, May 26, 2009 at 9:05 AM, Philipp Riegger > wrote: > > And all this layer thing Fabio was talking about. I did not try it and I > > did not read the code, but I think it makes things much more > > complicated. See also the discussion about mixing package managers > > between Gentoo and Sabayon. I do not want these problems. > incorrect. Give it a spin ;) I'll do, if i find the time. > Problems we have were *only* related to Portage world file handling, > fixed some time ago. I am sorry to say that the issue reported here > doesn't seem to be valid. Of course, if you mix both, you need to pay > attention to not change USE flags (for eg.) that trigger libraries > compilation, but that's a known binary-world problem. You're talking about binary-world problems here that Gentoo as a source based distribution does not have, I assume. This is a strength of Gentoo and I think we can keep it for binary packages by a good design. If you emerge a package in Gentoo it gets build from source. If you use a binary distribution, you cannot influence, what flags were used for building the package. But with a hybrid approach (I was aming for that one in my proposal), you would simply have the choice to either install a binary package and be restricted, if it's not available in the way you want it, or install it from source. And it would work together, because you don't have 2 package managers that need to interface, talk, share, work together, whatever, but you have 1 package manager that does it all and can keep it consistent. > I agree with you that there could be some more room for improvements > here and there (especially in kernel module ebuilds), but with EAPI=2 > we're going in the right direction. Kernel packages and kernel modules are not really of interest for me. I would keep them as source packages. My aim is not to make thigns easier, but to provide the user with the tools to save 8 hours of compiling openoffice or something like that. Not a binary distribution, but some kind of -bin packages, just packaged by Gentoo and better. Philipp
Re: [gentoo-dev] Re: better support for binary packages
On Tue, May 26, 2009 at 9:05 AM, Philipp Riegger wrote: Hi Duncan, And all this layer thing Fabio was talking about. I did not try it and I did not read the code, but I think it makes things much more complicated. See also the discussion about mixing package managers between Gentoo and Sabayon. I do not want these problems. incorrect. Give it a spin ;) Problems we have were *only* related to Portage world file handling, fixed some time ago. I am sorry to say that the issue reported here doesn't seem to be valid. Of course, if you mix both, you need to pay attention to not change USE flags (for eg.) that trigger libraries compilation, but that's a known binary-world problem. I agree with you that there could be some more room for improvements here and there (especially in kernel module ebuilds), but with EAPI=2 we're going in the right direction. Philipp -- Fabio Erculiani signature.asc Description: OpenPGP digital signature
[gentoo-dev] Re: better support for binary packages
Philipp Riegger posted 1243321504.9661.14.ca...@hspc30.informatik.uni-stuttgart.de, excerpted below, on Tue, 26 May 2009 09:05:03 +0200: > I don't see the connection between the email Fabio wrote and your > answer. Do you want to say, that you agree that he's doing what i > described and that it works the way i described it? I doubt it. If you > really care, could you answer my first email and state there the > problems you see with the approach and why you think this is making > Gentoo worse? I agree that an independent approach is the way to go. Gentoo (or rather, its PMs, package managers, all three of them, of which portage is only one) doesn't do binaries really well, certainly, but I really don't see that changing within Gentoo itself, except at the expense of from- source, which it does quite well indeed. > But how would you make it worse? It already has the functionality to add > repositories for binpackages, why not improve it? Why leave it the way > it is? Sure, improve it, but what you are talking about isn't a simple improvement, but a massive rearchitecting. That's not easily doable without a chance of focus. It's that change of focus that would eventually kill the quality from-source support we have and that continues to develop, now. >> That said, I could envision an eselect like "binary profile" switcher, >> that subject to settings in a config file, changes USE flags, CFLAGS, >> gcc- configs an appropriate gcc version, etc, changing PKGDIR >> appropriately as well, so one could easily enough create as many >> "binary profiles" as desired and as the use case dictated. > Not sure, what the binary profile switcher really would change here. > What I was talking about were mostly USE-flags and packages, and I guess > your binary profile switcher doesn't change too much, there. Sure it does, but incrementally. Basically, your proposal laid out a complicated tree based on metadata, a tree the package manager would have to understand and support, what I'm proposing is to allow the same thing, but not by adding all that complication to the package manager, but rather, by using a newly created application to switch among the branches of that tree on request. Portage (or other PM) would use its configured PKGDIR and wouldn't understand the tree, but it wouldn't need to, because the switcher would manage switching between the branches according to its configuration. The result would be that in a large enough deployment to have build- servers, the build server(s) could build a particular set of CFLAGS/USE- flags/gcc-version/arch/whatever, then switch to another branch and build that set. Individual package clients could simply point at the appropriate tree and get most of their packages, at least the common ones, that way. Now this wouldn't directly give you the flexibility of the package name changes you proposed, but it could do so indirectly, putting that information in the directory tree if configured to do so. Clients may need to use the binprofile switcher as well, for individual packages they chose to build outside their normal USE profile, but the process could be automated once properly configured, using PM hooks as appropriate. > It would be ok to do all this stuff in an extra project, without Gentoo. The proposal above wouldn't be without Gentoo. It'd simply be a package level thing rather than a distribution level thing. But either that or the independent distribution based on Gentoo route that lxnay has taken, either one works, without defocusing Gentoo on its meta-distrib-wide from- source vision. For that matter, Gentoo already has three package managers, and binary support (or the lack thereof) has been deliberately left up to the package manager at this point -- it's NOT part of PMS. It'd be equally possible to either take one of the current PMs and add your enhanced binary package scheme to it, or to start an independent forth package manager, and have it focus on binaries rather than from-source. (It could even take the existing portage binpkgs and rename or otherwise manage them as necessary, thereby avoiding the from-source side entirely, if so desired.) > Well, the last is a little bit impossible outside of Gentoo > (I don't want to fork the tree) and I also don't want to fork portage, > so the first is complicated, too. You mention portage, but don't mention the other PMs at all. As mentioned, binary packages aren't part of PMS (the Package Management Specification, the app-doc/pms package, the standard that allows PMs besides portage, currently, paludis and pkgcore) at all. Gentoo really doesn't have an official binary package format at all (see PMS, Appendix B, Unspecified Items), only individual package managers like portage do, and at present they may conflict with each other. That's both a good and a bad thing. As it's not specified, you're free to define it as necessary to meet your needs for any tool y
Re: [gentoo-dev] Re: better support for binary packages
Hi Duncan, I don't see the connection between the email Fabio wrote and your answer. Do you want to say, that you agree that he's doing what i described and that it works the way i described it? I doubt it. If you really care, could you answer my first email and state there the problems you see with the approach and why you think this is making Gentoo worse? On Mon, 2009-05-25 at 11:43 +, Duncan wrote: > Gentoo is in general a from-source metadistribution. There's limited > support for binary packages in at least one of the package managers > (portage), but honestly, that's not what Gentoo's best at, and I don't > believe that will ever change without making it significantly worse at > what it IS best at now, the normal from-source Gentoo we know and love. But how would you make it worse? It already has the functionality to add repositories for binpackages, why not improve it? Why leave it the way it is? > Better to leave the serious distribution level binary repackaging to the > Gentoo-based distributions like Sabayon. Let them do what they do best, > and let Gentoo continue doing what it does best. By definition, binary > packaging isn't broken and doesn't need fixed, as that's not part of what > defines Gentoo, so don't fix what ain't broken! =:^) Well actually, some time ago Gentoo was by definition running a linux kernel or a BSD kernel and now it runs in a prefix on Windows and AIX and Solaris. Some time ago there was some guy called drobbins who was kind of the leader of Gentoo and now we have a council. If you really don't want changes, you could stop running emerge --sync. > That said, I could envision an eselect like "binary profile" switcher, > that subject to settings in a config file, changes USE flags, CFLAGS, gcc- > configs an appropriate gcc version, etc, changing PKGDIR appropriately as > well, so one could easily enough create as many "binary profiles" as > desired and as the use case dictated. It's likely various reasonably > large Gentoo deployments are already doing something like this as it > could certainly be scripted, but an emergable package to make it easy for > ordinary joe Gentoo user would be useful, and I believe appreciated by > many. > > (Here, I'd put it to use when testing new gcc versions, making it easy to > swap out PKGDIRs and revert to the old version either per-package or > system-wide, if the new version was breaking too much.) > > So here: No to the whole big complicated let's fix Gentoo binaries > thing. There's already Gentoo-based binary solutions like Sabayon for > those so interested, and I can't see Gentoo doing better than they're > doing, at least not without breaking the from-source that Gentoo's good > at. But yes to anyone interested in developing a nice new "binary > profile" switcher to make managing binary package sets easier for those > Gentoo admins that would find such a thing useful. Whether they then > start making those profiles public and whether anyone else chooses to use > them is entirely up to the individual admins whose systems would be > affected. Not sure, what the binary profile switcher really would change here. What I was talking about were mostly USE-flags and packages, and I guess your binary profile switcher doesn't change too much, there. It would be ok to do all this stuff in an extra project, without Gentoo. But there are some downsides: You are not Gentoo anymore. The communication channels get longer and more complicated. In my first post i described some things that would need to be changed. Either in portage or in the policy how packages are dealt with. Well, the last is a little bit impossible outside of Gentoo (I don't want to fork the tree) and I also don't want to fork portage, so the first is complicated, too. And all this layer thing Fabio was talking about. I did not try it and I did not read the code, but I think it makes things much more complicated. See also the discussion about mixing package managers between Gentoo and Sabayon. I do not want these problems. Philipp
[gentoo-dev] Re: better support for binary packages
lx...@sabayonlinux.org posted fv50wm3klntqyetcw4uyaxe124vaj_fire...@mail.gmail.com, excerpted below, on Mon, 25 May 2009 12:16:06 +0200: > This is what I am doing in Sabayon, creating a new layer over Portage => > Entropy. I'm almost done, just need to work out some documentation and > apidocs. http://gitweb.sabayon.org/?p=entropy.git;a=summary Agreed. Gentoo is in general a from-source metadistribution. There's limited support for binary packages in at least one of the package managers (portage), but honestly, that's not what Gentoo's best at, and I don't believe that will ever change without making it significantly worse at what it IS best at now, the normal from-source Gentoo we know and love. Better to leave the serious distribution level binary repackaging to the Gentoo-based distributions like Sabayon. Let them do what they do best, and let Gentoo continue doing what it does best. By definition, binary packaging isn't broken and doesn't need fixed, as that's not part of what defines Gentoo, so don't fix what ain't broken! =:^) That said, I could envision an eselect like "binary profile" switcher, that subject to settings in a config file, changes USE flags, CFLAGS, gcc- configs an appropriate gcc version, etc, changing PKGDIR appropriately as well, so one could easily enough create as many "binary profiles" as desired and as the use case dictated. It's likely various reasonably large Gentoo deployments are already doing something like this as it could certainly be scripted, but an emergable package to make it easy for ordinary joe Gentoo user would be useful, and I believe appreciated by many. (Here, I'd put it to use when testing new gcc versions, making it easy to swap out PKGDIRs and revert to the old version either per-package or system-wide, if the new version was breaking too much.) So here: No to the whole big complicated let's fix Gentoo binaries thing. There's already Gentoo-based binary solutions like Sabayon for those so interested, and I can't see Gentoo doing better than they're doing, at least not without breaking the from-source that Gentoo's good at. But yes to anyone interested in developing a nice new "binary profile" switcher to make managing binary package sets easier for those Gentoo admins that would find such a thing useful. Whether they then start making those profiles public and whether anyone else chooses to use them is entirely up to the individual admins whose systems would be affected. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
