Re: [gentoo-dev] Re: GLEP 55 Version 2
Ulrich Mueller wrote: Let's assume for the moment that we change from .ebuild to .eb. Then we obviously cannot change all ebuilds in the tree to .eb, otherwise old Portage versions would see an empty tree and there would be no upgrade path. Or am I missing something? That is a good point. Although things would probably break more gracefully than having old portage versions attempting to parse new ebuilds (maybe, maybe not). That actually makes me wonder - if we didn't change the extension at all could we somehow poison the portage tree so that old versions of portage would abort before doing anything dumb with a meaningful error message? As far as an upgrade path goes - we could provide a one-time tarball that will update portage (and its essential dependencies) to a version that can get users out of this bind. If a user has a system THAT old then they might just want to extract a stage1 tarball (manually - without overwriting /etc without care) and go from there. I'm not sure that gentoo generally supports graceful upgrades from very ancient systems to modern ones without keeping up to date. Other distros can do it since they do ~annual releases and users could just apply those sequentially. For portage we don't keep around all the files needed to do a sequential upgrade like this - if a user were to try to upgrade to a 3-year-old version of some package most likely it wouldn't be mirrored and upstream might not have it either. We obviously need to give some thought to not breaking old versions of portage, but given that portage will be only one of many problems if a user doesn't do an emerge -u world for 5 years I'm not sure we need a bulletproof solution...
Re: [gentoo-dev] Re: GLEP 55 Version 2
On Sunday 07 June 2009 11:34:12 Ulrich Mueller wrote: On Sun, 07 Jun 2009, Steven J Long wrote: I'd just like to know what the implications would be for users if we kept the .ebuild extension, and a new PMS were rolled out stating that the mangler were allowed to find the EAPI without sourcing (and giving the restrictions) once portage 2.2 was stable, or the ability to handle this backported to 2.1.6, and issued in a release? Even if we do only a one-time change of the file extension, can we ever get rid of the old extension? Or are we then stuck with two extensions in the tree until the end of time? Let's assume for the moment that we change from .ebuild to .eb. Then we obviously cannot change all ebuilds in the tree to .eb, otherwise old Portage versions would see an empty tree and there would be no upgrade path. Or am I missing something? I come to the same conclusion. Which means that the easiest way to get things migrated will be a one-time change of the _sync_ location so that users have a chance to get to an upgrade-ready state on their system, then change sync location, then upgrade to the current state. In which case we actually do not have to change the file name anymore ... Of course things aren't always that easy, but if you add a generation file somewhere in the rsync checkout it is easy to see if your current rsync location is old and deprecated or new and improved. And if you really absolutely have to do that you can change the sync location on every disruptive change, but (imo) that should be avoided. Less disruptive changes is better :) hth, Patrick
Re: [gentoo-dev] Re: GLEP 55 Version 2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ulrich Mueller wrote: On Sun, 07 Jun 2009, Steven J Long wrote: I'd just like to know what the implications would be for users if we kept the .ebuild extension, and a new PMS were rolled out stating that the mangler were allowed to find the EAPI without sourcing (and giving the restrictions) once portage 2.2 was stable, or the ability to handle this backported to 2.1.6, and issued in a release? Even if we do only a one-time change of the file extension, can we ever get rid of the old extension? unfortunately, no. Or are we then stuck with two extensions in the tree until the end of time? Let's assume for the moment that we change from .ebuild to .eb. better put this new ebuild type in a new tree; such a massive change to the tree its not recommended. Then we obviously cannot change all ebuilds in the tree to .eb, otherwise old Portage versions would see an empty tree and there would be no upgrade path. leaving actual .ebuilds as they are now (good healthy state :)) and making new development of .eb ebuilds happen in a new tree (I said new tree, but it could be any way that hides those new ebuild to OLD package managers) would help. only newer versions of package managers are required to support this, that is they will look for .eb (in new or current tree, not sure what's best) and then for .ebuilds, and ideally this should be transparent to old package managers and allow an upgrade path. - -- mescali...@g.o -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkortVIACgkQV/B5axfzrPsTiACeJCJb3F8Up/+CjHIwC3Slhn/6 yZgAoLcJgNn2d3W/JeZPkK85arUPW9vV =fR4T -END PGP SIGNATURE-
Re: [gentoo-dev] Re: GLEP 55 Version 2
Patrick Lauer wrote: And if you really absolutely have to do that you can change the sync location on every disruptive change, but (imo) that should be avoided. If mirroring and other practical concerns weren't an issue what you're essentially describing is just moving to a CVS/git/etc repository and using such a tool to do all the syncs (ie not using rsync at all). Then all you need is an update page that has a list of commands like: emerge --sync --date 2008-01-01 emerge -u world emerge --sync --date 2008-08-10 Follow this xorg update doc: link emerge --sync --date 2034-04-02 emerge -u portage so that you have support for the finally-approved glep55 And so on... :) That actually gives you a best-of-both-worlds option: continue to use rsync for normal use to ease distribution, but have a cvs tree that anybody can read from which could be used in upgrade guides for out-of-date users.
Re: [gentoo-dev] Re: GLEP 55 Version 2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2009.06.07 10:34, Ulrich Mueller wrote: On Sun, 07 Jun 2009, Steven J Long wrote: I'd just like to know what the implications would be for users if we kept the .ebuild extension, and a new PMS were rolled out stating that the mangler were allowed to find the EAPI without sourcing (and giving the restrictions) once portage 2.2 was stable, or the ability to handle this backported to 2.1.6, and issued in a release? Even if we do only a one-time change of the file extension, can we ever get rid of the old extension? Or are we then stuck with two extensions in the tree until the end of time? Let's assume for the moment that we change from .ebuild to .eb. Then we obviously cannot change all ebuilds in the tree to .eb, otherwise old Portage versions would see an empty tree and there would be no upgrade path. Or am I missing something? Ulrich First, lets be clear that the upgrade path related problems are driven by the EAPI change, not the .ebuild to .eb change. That serves only to allow the new EAPI to be introduced immedately and hide the change from older package managers The upgrade path issue remains even if we do the usual Gentoo introduce new features to the package manager then wait a year or so before they are deployed. Without the .ebuild to .eb change. late upgraders will get the error messages in Appendix A of the GLEP when these features are relesed. To keep an upgrade path, package managers and their dependencies need to use EAPI 0, 1 or 2. (EAPI 3 is not yet out) How far back should the upgrade path extend? As you suggest, even with .ebuild to .eb change.its not a clean break with the past but would happen over time, with ebuilds for new package versions being migrated to the new format. For example we would have foo-2.1-r1.ebuild foo-3.0.eb portage-2.3.ebuild portage-3.0.eb Portage-2.3 will understand the later EAPI but will itself, use only EAPI, 0, 1 or 2. With the .ebuild to .eb change. users who do not upgrade portage will not see newer versions of foo. Without it, they will get the error messages in Appendix A of the GLEP. This will have the side effect of driving the adoption of the newer package managers. It is not suffcient to have portage-2.3.ebuild as understanding later EAPI files. All of its dependencies need to keep to EAPI 0, 1 or 2. These must be the last packages to migrate to later EAPIs. Thats just portage. The same reasoning applies to any other package manager and there are at least three. This begs the question of how friendly do we want to be to derivitive distros that use our tree but their own package manager? Upgrade path issues need to be addressed in the GLEP. I will add something like the above in the next version. - -- Regards, Roy Bamford (NeddySeagoon) a member of gentoo-ops forum-mods treecleaners trustees -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.11 (GNU/Linux) iEYEARECAAYFAkoryrUACgkQTE4/y7nJvashtQCgkO+feHG2BBGLOTObrwq72iOx nI4AoNZ67Mhq4yEKYTzfBOPmu98Py9Mc =GB62 -END PGP SIGNATURE-
Re: [gentoo-dev] Re: GLEP 55: another approach: display pretty messages with old PMs
On Fri, 2009-05-29 at 10:42 +, Duncan wrote: Michael Haubenwallner ha...@gentoo.org posted on Fri, 29 May 2009 10:14:46 +0200: Ohw, the latter would be necessary here, or '4.ebuild' would not be found. s/4.ebuild/4.eclass/ I assume. Indeed. Except... since an ebuild must presently be sourced to (properly) determine EAPI, it still doesn't work for changes that break sourcing or other pre-EAPI processing (like parsing PN and PVR out of the filename), so the changes allowed with a simple EAPI change are still rather limited. As long as the *whole* ebuild content (including the filename) needs to be successfully interpreted just for EAPI detection, we will have to change the extension or wait the extended period for each incompatible EAPI change. So this full interpretation for EAPI detection doesn't feel like a good way to go at all. That's why the change to GLEP55 or alternative, whether in-filename or in-file-itself, will again require either an extended wait after introduction (the old way) or at minimum, a one-time change to extension such that old PM versions don't even see the currently EAPI incompatible changes. Wouldn't it be possible to avoid both the extension change and another extended wait period for new incompatible(*) EAPIs, when we do this early and silent exit hack for unsupported ebuilds with old PMs that still do full interpretation for EAPI detection? And after another extended wait period(**), we can start dropping the silent exit hack, so we finally have switched to EAPI detection without full interpretation, but still have the .ebuild extension. (*) The incompatibility of EAPIs must not begin (meaning the bytewise ebuild content) before the end of both the ebuild's EAPI value definition and the silent exit hack. But this IMO is an acceptable compromise. (**) After this wait period, the incompatibility of EAPIs can start after the end of the ebuild's EAPI value definition. Thanks! /haubi/ -- Michael Haubenwallner Gentoo on a different level
Re: [gentoo-dev] Re: GLEP 55 updated
On Mon, 18 May 2009 16:07:20 +0100 Steven J Long sl...@rathaus.eclipse.co.uk wrote: I missed the clamour of developers complaining about this oh-so-burdensome restriction that they've been dealing with for at least 5 years. Why do you think I wrote the awful hack that is versionator? Anything that finally lets us kill that off has to be good... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55 updated
Ciaran McCreesh wrote: On Mon, 18 May 2009 16:07:20 +0100 Steven J Long sl...@rathaus.eclipse.co.uk wrote: I missed the clamour of developers complaining about this oh-so-burdensome restriction that they've been dealing with for at least 5 years. Why do you think I wrote the awful hack that is versionator? Anything that finally lets us kill that off has to be good... I have to disagree. As Steve said, the fact that Gentoo has a strict way to specify versions brings clarity and uniformity to our tree, regardless of the myriad upstream conventions. I do not think that allowing all of those upstream conventions in our filenames is a benefit. It is actually quite ugly and would make the tree harder to comprehend. Someone looking at various packages in our tree would need to learn each specific upstream format in order to make sense of the filename content. The current consistency in versions in the tree is a great feature, IMHO. Using versionator and $MY_PV is, as I see it, a translation method. It gives us the best of both worlds: the ability to deal with upstream versions and a consistent Gentoo version format. These mechanisms could certainly be improved upon, of course, and handling some cases is currently difficult, as is handing the case in which upstream's tarball file does not contain the version, but these are fixable issues. -Joe
Re: [gentoo-dev] Re: GLEP 55 updated
2009/5/18 Steven J Long sl...@rathaus.eclipse.co.uk: David Leverton wrote: 2009/5/17 Ben de Groot yng...@gentoo.org: I think the way eapi-2 was introduced into the tree wasn't particularly problematic. I think there might be a misunderstanding here. Ciaran means functions provided by the package manager that ebuilds can call during metadata generation, for example built-in versionator-like functionality, not new phase functions like src_prepare and src_configure. Ugh. Firstly versionator is a piece of bloated crap that should have died a long time ago; all it does is stop Gentoo newbs learning the basics[1] of their implementation language, which leaves them open to nonsensical arguments about printf -v (glad you finally learnt that one, btw.) Yes, it should have died a long time ago, that's why we're talking about adding equivalent built-in functionality. Please try to keep up. (And it's always amusing to see your bizarre delusions about what I do and don't know.) Secondly, please explain fully and clearly, but concisely, why we can't simply state that EAPI=NN allows the ebuild to use the EAPI functions in global scope. Because by the time the package manager knows the EAPI and is in a position to provide the appropriate functions, the ebuild will have already tried to use them. Bear in mind that you have to deal with the case of the mangler which can get EAPI from an ebuild without sourcing, as portage currently can, I believe. Interesting Relaxing that restriction strikes me as much saner than relaxing all the other restrictions which make interoperability easier. (Frankly, I'm amazed at having to state that to people trusted to write a specification. Follow up to -project on this point please, as it's about process, not technique.) You're amazed that people trusted to write a specification don't already know what strikes you as much saner? Believe it or not, the world does not revolve around you and your opinions.
Re: [gentoo-dev] Re: GLEP 55 updated
2009/5/17 Ryan Hill dirtye...@gentoo.org: On Sun, 17 May 2009 17:56:06 +0200 Piotr Jaroszyński pe...@gentoo.org wrote: Hello, I have just updated GLEP 55 [1], hopefully making it a bit clearer. Just FYI, my order of preference of solutions is: 1. EAPI-suffixed ebuilds (obviously) 2. EAPI in the filename with one-time extension change 3. Easily fetchable EAPI inside the ebuild and one-time extension change I can live with any of these if that's what it takes to move forward. I'd like 2 if we could have multiple same-versioned ebuilds of different EAPI. 3 is good enough for me. That's covered in the GLEP: Note that it is still not permitted to have more than one ebuild with equal category, package name, and version. Although it would have the advantage of allowing authors to provide backwards compatible ebuilds, it would introduce problems too. The first is the requirement to have strict EAPI ordering, the second is ensuring that all the ebuilds for a single category/package-version are equivalent, i.e. installing any of them has exactly the same effect on a given system. I don't see a way to overcome these problems in a sensible way. -- Best Regards, Piotr Jaroszyński
Re: [gentoo-dev] Re: GLEP 55 updated
On Sun, 17 May 2009 12:15:24 -0600 Ryan Hill dirtye...@gentoo.org wrote: I'd like 2 if we could have multiple same-versioned ebuilds of different EAPI. 3 is good enough for me. We couldn't. Allowing multiple equal but different ebuilds gets highly crazy -- EAPIs aren't orderable, so it's not obvious which one the package manager should select. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
Fernando J. Pereda wrote: On 10 Jun 2008, at 15:33, Joe Peterson wrote: Luca Barbato wrote: Check if exists a line EAPI=*$, if does and the rest of the string matches an understood eapi, go on sourcing, otherwise ignore/mask it... And placing it out-of-band (like # EAPI=...) avoids any sourcing errors, makes parsing faster, etc. No, it doesn't make parsing faster. Had you bothered to profile any package manager you'd know that. How much is parsing speed relevant when users in majority of cases already have the metadata cache from the rsync servers? For devs (or users) hacking on an ebuild they have to source it anyway so the addition pre-parsing is negligible... VB -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On 2008/06/11, Ciaran McCreesh [EMAIL PROTECTED] wrote: You're missing the cases where the cache isn't usable. I was not talking about generating cache entries, and neither were you. I've replied to you because you were suggesting that the EAPI in ebuilds contents solution had extra cost when _using_ valid cache entries (need to extract the EAPI from the ebuild before reading this cache entry), which i think can be easily avoided. On 2008/06/10, Ciaran McCreesh [EMAIL PROTECTED] wrote: Currently we don't touch the ebuild's content *at all* for metadata operations, except where there's no or stale metadata cache (which is rare). We can get away with this currently because 0 and 1 have identical cache layouts and PMS has some (necessary) weasel wording; future EAPIs likely won't, so we're back to the chicken / egg problem. So... We either have the EAPI from the extension (which we already have, since we have to read the dir to know what versions are available), in which case we know how to read the metadata cache file. Or we have to open up a file that would otherwise not be opened, just to parse one line so we know how to read the cache file. -- TGL. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Wed, 11 Jun 2008 08:31:45 +0200 Thomas de Grenier de Latour [EMAIL PROTECTED] wrote: On 2008/06/11, Ciaran McCreesh [EMAIL PROTECTED] wrote: You're missing the cases where the cache isn't usable. I was not talking about generating cache entries, and neither were you. I've replied to you because you were suggesting that the EAPI in ebuilds contents solution had extra cost when _using_ valid cache entries (need to extract the EAPI from the ebuild before reading this cache entry), which i think can be easily avoided. And it does, since EAPI has to be known to use the cache contents. The way it's handled currently is a hack that doesn't work with future EAPIs defining new metadata. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mike Kelly wrote: | Wrong. Thanks for that. I'm gonna assume you meant I think you're wrong. | Sure, because of how the algorithm works, people could potentially do | both, but the GLEP makes it pretty clear that they shouldn't. It doesn't just say they shouldn't, it *says what happens if they do*: pkg-4.ebuild-2, EAPI=1 ~pre-source EAPI is 2, post-source EAPI is 1. EAPI 1 is used So to live up to the GLEP specification, package managers must source the ebuild to determine the correct EAPI. | Basically, you don't set the post-source EAPI. It is only supposed to | be used when the pre-source EAPI cannot be determined. If that's what was meant, the GLEP needs changing to say that. Something like: pkg-4.ebuild-2, EAPI=X ~Pre-source EAPI is 2, post-source EAPI is X. In this instance the post-source EAPI is ignored, since a pre-source EAPI is set, so EAPI 2 is used. It's not a big deal, but it will need a change to the GLEP in its current form, if that's what's meant. Mike 5:) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkhPdRUACgkQu7rWomwgFXq1twCgqnFzBqQI8vl63faZ1cq27MF7 7ekAn0aA73nic1v/ovwAUuHsaL44MrTE =StC2 -END PGP SIGNATURE- -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Wed, Jun 11, 2008 at 04:14:58AM +0100, Ciaran McCreesh wrote: On Tue, 10 Jun 2008 19:14:11 +0200 Olivier Galibert [EMAIL PROTECTED] wrote: On Tue, Jun 10, 2008 at 03:02:28PM +0100, Ciaran McCreesh wrote: Except that currently, the ebuild file isn't opened for read. So it's not in memory at all. Why would you need the EAPI before the time when you actually want to interpret the contents? You need the EAPI before you use the metadata. But you don't need the ebuild to get the metadata in the common case. The metadata should carry its version indicator too. OG. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Tue, Jun 10, 2008 at 03:14:47PM +0100, Ciaran McCreesh wrote: !-- EAPI=3 -- *Then* would be the time to change the extension. As long as the ebuild is bash-parseable with an appropriate environment, it doesn't make sense to change the extension because a env-variable set or a comment are more natural methods. If/when the format changes to something not parseable by bash, then it will be time to change the extension. And then how to mark (sub-)version will depend on the chosen new format, in case of xml that would be the dtd information. I suspect the rejection of the extension change will be there as long as the fundamental format (bash script) doesn't change. OG. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Wed, Jun 11, 2008 at 12:06 PM, Ciaran McCreesh [EMAIL PROTECTED] wrote: On Wed, 11 Jun 2008 08:31:45 +0200 Thomas de Grenier de Latour [EMAIL PROTECTED] wrote: On 2008/06/11, Ciaran McCreesh [EMAIL PROTECTED] wrote: You're missing the cases where the cache isn't usable. I was not talking about generating cache entries, and neither were you. I've replied to you because you were suggesting that the EAPI in ebuilds contents solution had extra cost when _using_ valid cache entries (need to extract the EAPI from the ebuild before reading this cache entry), which i think can be easily avoided. And it does, since EAPI has to be known to use the cache contents. The way it's handled currently is a hack that doesn't work with future EAPIs defining new metadata. Fix that, then. And I understand that the code is already there in both portage and pkgcore to store the cache as key-value pairs rather than one-slot-per-key, and would be relatively trivial to add to paludis. Regards, -- Arun Raghavan (http://nemesis.accosted.net) v2sw5Chw4+5ln4pr6$OFck2ma4+9u8w3+1!m?l7+9GSCKi056 e6+9i4b8/9HTAen4+5g4/8APa2Xs8r1/2p5-8 hackerkey.com -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Olivier Galibert [EMAIL PROTECTED] wrote: On Tue, Jun 10, 2008 at 03:14:47PM +0100, Ciaran McCreesh wrote: !-- EAPI=3 -- *Then* would be the time to change the extension. As long as the ebuild is bash-parseable with an appropriate environment, it doesn't make sense to change the extension because a env-variable set or a comment are more natural methods. If/when the format changes to something not parseable by bash, then it will be time to change the extension. And then how to mark (sub-)version will depend on the chosen new format, in case of xml that would be the dtd information. I suspect the rejection of the extension change will be there as long as the fundamental format (bash script) doesn't change. Well said. This is something that I've heard bandied about on IRC now and then, and sounds to me (notably *not* a package manager developer) like a fairly reasonable compromise. To the proponents of GLEP55: Is there some reason that GLEP55 is preferable to this? Are there reasons why a particular filename extension could not apply to a range of EAPIs? Why not just bump the filename suffix when it is required to support a new EAPI that breaks the sourcing rules of previous EAPIs? Or will backwards-incompatible changes be happening so frequently that the package suffix will have to change for every EAPI bump anyway, which would make this proposal equivalent to GLEP55? -- Jim Ramsay Gentoo/Linux Developer (rox,gkrellm) signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
On Wed, Jun 11, 2008 at 12:05 PM, Olivier Galibert [EMAIL PROTECTED] wrote: On Tue, Jun 10, 2008 at 03:14:47PM +0100, Ciaran McCreesh wrote: !-- EAPI=3 -- *Then* would be the time to change the extension. As long as the ebuild is bash-parseable with an appropriate environment, it doesn't make sense to change the extension because a env-variable set or a comment are more natural methods. If/when the format changes to something not parseable by bash, then it will be time to change the extension. And then how to mark (sub-)version will depend on the chosen new format, in case of xml that would be the dtd information. I suspect the rejection of the extension change will be there as long as the fundamental format (bash script) doesn't change. If the extension was based on the fact that ebuilds are bash scripts, they'd have .bash extension. The .ebuild extension means a specific kind of bash script and it doesn't seem wrong to change the extension if that specific kind changes, even if bash is still the interpreter. Even if we switched to sh or zsh I doubt we'd use the .sh or .zsh extension. Regards, -- Santiago M. Mola Jabber ID: [EMAIL PROTECTED] -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Thu, Jun 12, 2008 at 4:07 AM, Jim Ramsay [EMAIL PROTECTED] wrote: Why not just bump the filename suffix when it is required to support a new EAPI that breaks the sourcing rules of previous EAPIs? Or will backwards-incompatible changes be happening so frequently that the package suffix will have to change for every EAPI bump anyway, which would make this proposal equivalent to GLEP55? That works. Although, we'd have to keep track of two parameters when setting our EAPI. One being the EAPI itself and the suffix needed for that EAPI. So this doesn't seem to make the problem simpler. Regards, -- Santiago M. Mola Jabber ID: [EMAIL PROTECTED] -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On 2008-06-10 15:50, Fernando J. Pereda uttered these thoughts: On 10 Jun 2008, at 15:46, Joe Peterson wrote: Also, I'm not sure reading XML is a problem at all - python has good libs for this already. Reading XML files is easy, but it makes certain codepaths much much slower. Not a good 'feature'. ... AND there's no requirement for the PM to be written in python or any other language currently in use. -- () The ASCII Ribbon Campaign - against HTML Email /\ and proprietary formats. pgpuqnCKU4eqn.pgp Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
Santiago M. Mola [EMAIL PROTECTED] wrote: On Thu, Jun 12, 2008 at 4:07 AM, Jim Ramsay [EMAIL PROTECTED] wrote: Why not just bump the filename suffix when it is required to support a new EAPI that breaks the sourcing rules of previous EAPIs? Or will backwards-incompatible changes be happening so frequently that the package suffix will have to change for every EAPI bump anyway, which would make this proposal equivalent to GLEP55? That works. Although, we'd have to keep track of two parameters when setting our EAPI. One being the EAPI itself and the suffix needed for that EAPI. So this doesn't seem to make the problem simpler. I agree that it doesn't make things simpler. Though it doesn't make things that much more complex. That said, I'm becoming more convinced that a lot of the changes that really need to be made will indeed break EAPI sourcing rules, so maybe the latter part of my original quote above will indeed be the case - This would be equivalent to GLEP55. I should add that I am not personally opposed to having the EAPI in the filename, but this may find slightly more acceptance from the folks who think that solution is incorrect. -- Jim Ramsay Gentoo Developer (rox/fluxbox/gkrellm) signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 10:33:34 +0200 Tiziano Müller [EMAIL PROTECTED] wrote: Another ugly solution: Having the EAPI on a per-package (like $portagedir/cat/package-1) or per-tree basis ($portagedir/profiles/eapi) and start providing our tree as overlays of more than one tree (will end up in a mess of dependencies, but it would still be nice to specify the EAPI for a complete overlay instead of having the name all the ebuilds like .eapi-X :-). In addition: it wouldn't be possible to identify the EAPI of an ebuild by just looking at it... Kills the upgrade path completely. No good. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
Tiziano Müller wrote: Another ugly solution: Having the EAPI on a per-package (like $portagedir/cat/package-1) or per-tree basis ($portagedir/profiles/eapi) I like the per tree basis and I already asked about that (since makes things clearer and older portage can be tricked by rsync. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Ciaran McCreesh wrote: Kills the upgrade path completely. No good. Not really, you change the rsync paths and older portage will pick a repo that just has the necessary to upgrade to the next portage. This kind of things would work better using an scm supporting branches and tags a bit better. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 11:40:35 +0200 Luca Barbato [EMAIL PROTECTED] wrote: Ciaran McCreesh wrote: Kills the upgrade path completely. No good. Not really, you change the rsync paths and older portage will pick a repo that just has the necessary to upgrade to the next portage. This kind of things would work better using an scm supporting branches and tags a bit better. picard_facepalm.jpg -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
Ciaran McCreesh a écrit : Kills the upgrade path completely. No good. Lemme sum this up in layman's terms : 1) EAPI _has_ to be known before sourcing an ebuild. There's no way to avoid that for various reasons, all 100% valid. 2) Putting the EAPI in the filename : + it solves 1) + it keeps backward compatibility because old PM won't recognize the filenames - it's not very pretty 3) Putting the EAPI in metadata.xml or in another external file + it solves 1) + it keeps pretty file names - it breaks backwards compatibility - (specific to metadata.xml) PM will have to learn to read XML (yuck) That's about it, right? -- Rémi Cardona LRI, INRIA [EMAIL PROTECTED] [EMAIL PROTECTED] -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Tiziano Müller wrote:
Joe Peterson wrote:
Ciaran McCreesh wrote:
And a file extension is far less obscurely complex than enforcing
arbitrary syntax restrictions upon ebuilds.
I disagree. One is exposed to devs only as ebuild syntax; the other is
exposed in an inappropriate location to everyone looking at the portage
tree.
No it can't. EAPI has to be known before the source can start. Bash
doesn't provide traps for executing code upon changed variables.
Doing it out-of-band solve this.
No, it's only needed once per non-trivial change. So we might as well
just change it for every EAPI.
Huh? If the new portage knows how to determine the EAPI definitively
(and that would be defined), it can deal with the differences.
And then how do we deal with EAPI 3, where the syntax changes again?
Portage (or whatever PM) reads the EAPI, determines it is 3, and goes
from there. If you change the way you declare EAPI each time, yeah,
that's a problem, but I'm not sure why that would ne necessary.
No, that is not the problem.
Example:
In EAPI 42 we define that the package manager must provide a global function
extract_depend_from_setup_py() such that it is callable at a global level
in an ebuild like this
*snip start*
# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
EAPI=42
DESCRIPTION=A library aiming to support agile and test-driven python
development on various levels.
SRC_URI=http://codespeak.net/download/${PN}/${P}.tar.gz;
HOMEPAGE=http://codespeak.net/py/;
KEYWORDS=~amd64 ~x86
SLOT=0
LICENSE=MIT
IUSE=
extract_depend_from_setup_py
*snip end*
Now, a package manager (or a tool) not knowing EAPI 42 will fail when it
tries to source the above ebuild to determine the EAPI version (as it is
being currently done as far as I understood it) because
extract_depend_from_setup_py is undefined.
So it won't even be able to read out the EAPI.
With the EAPI in the filename tools now knowing EAPI-42 will either ignore
the above foo-1.0.ebuild-42 or mask it because they may identify the
EAPI-version without sourcing the ebuild.
Check if exists a line EAPI=*$, if does and the rest of the string
matches an understood eapi, go on sourcing, otherwise ignore/mask it...
lu
--
Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero
--
gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On 10 Jun 2008, at 12:30, Rémi Cardona wrote: Ciaran McCreesh a écrit : picard_facepalm.jpg I don't think any of us are completely thrilled by either proposals, but the EAPI-in-a-separate-file does have the potential for more flexibility, ie package-wide EAPI. And it does keep filenames simple enough. And it is not backwards compatible. - ferdy-- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Rémi Cardona wrote: Ciaran McCreesh a écrit : picard_facepalm.jpg I don't think any of us are completely thrilled by either proposals, but the EAPI-in-a-separate-file does have the potential for more flexibility, ie package-wide EAPI. And it does keep filenames simple enough. +1 -Joe -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Luca Barbato wrote: Check if exists a line EAPI=*$, if does and the rest of the string matches an understood eapi, go on sourcing, otherwise ignore/mask it... And placing it out-of-band (like # EAPI=...) avoids any sourcing errors, makes parsing faster, etc. -Joe -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On 10 Jun 2008, at 15:33, Joe Peterson wrote: Luca Barbato wrote: Check if exists a line EAPI=*$, if does and the rest of the string matches an understood eapi, go on sourcing, otherwise ignore/mask it... And placing it out-of-band (like # EAPI=...) avoids any sourcing errors, makes parsing faster, etc. No, it doesn't make parsing faster. Had you bothered to profile any package manager you'd know that. - ferdy -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Rémi Cardona wrote: Ciaran McCreesh a écrit : Kills the upgrade path completely. No good. Lemme sum this up in layman's terms : 1) EAPI _has_ to be known before sourcing an ebuild. There's no way to avoid that for various reasons, all 100% valid. 2) Putting the EAPI in the filename : + it solves 1) + it keeps backward compatibility because old PM won't recognize the filenames - it's not very pretty I'd say the problems go way beyond just being not pretty. That longish email I wrote yesterday has a bunch of reasons I don't like it. And pretty makes the issue sound unimportant or superficial. 3) Putting the EAPI in metadata.xml or in another external file + it solves 1) + it keeps pretty file names - it breaks backwards compatibility - (specific to metadata.xml) PM will have to learn to read XML (yuck) That's about it, right? Good summary, except I think we can find ways to deal with compatibility (several ideas have been put forth already: giving time for PM to be upgraded, or a one-time tree or extension bump - and I'm sure there are even better ones yet to be discussed). I do not believe that the filename mangling solution is the one and only way as some people are insisting. Also, I'm not sure reading XML is a problem at all - python has good libs for this already. -Joe -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Fernando J. Pereda wrote: No, it doesn't make parsing faster. Had you bothered to profile any package manager you'd know that. Do you have any number to share? lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Fernando J. Pereda wrote: On 10 Jun 2008, at 15:33, Joe Peterson wrote: Luca Barbato wrote: Check if exists a line EAPI=*$, if does and the rest of the string matches an understood eapi, go on sourcing, otherwise ignore/mask it... And placing it out-of-band (like # EAPI=...) avoids any sourcing errors, makes parsing faster, etc. No, it doesn't make parsing faster. Had you bothered to profile any package manager you'd know that. No, I have not profiled PMs to try this, but you are saying that reading the first few lines of a file is not faster than sourcing the whole thing with bash? Remember that it could abort the minute it sees a non '#' or blank line, which would be after the first few. -Joe -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On 10 Jun 2008, at 15:46, Joe Peterson wrote: Also, I'm not sure reading XML is a problem at all - python has good libs for this already. Reading XML files is easy, but it makes certain codepaths much much slower. Not a good 'feature'. - ferdy -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Rémi Cardona wrote: Ciaran McCreesh a écrit : Kills the upgrade path completely. No good. Lemme sum this up in layman's terms : 1) EAPI _has_ to be known before sourcing an ebuild. There's no way to avoid that for various reasons, all 100% valid. 2) Putting the EAPI in the filename : + it solves 1) + it keeps backward compatibility because old PM won't recognize the filenames - it's not very pretty 3) Putting the EAPI in metadata.xml or in another external file + it solves 1) + it keeps pretty file names - it breaks backwards compatibility - (specific to metadata.xml) PM will have to learn to read XML (yuck) That's about it, right? You missed some options - I'll try to be honest about their pros/cons: 4) Putting EAPI inside the ebuild, but in a manner that does not require sourcing using bash (ie comment at top of file). + it solves 1) + it keeps pretty file names + syntax/implementation is trivial - it breaks backwards compatibility (eventually - hacks might delay this) - it does force future ebuilds to have the EAPI line in it 5) Putting EAPI inside the ebuild as in #4, but do a one-time-only change of the filenames. + it solves 1) + it sort-of-keeps pretty file names (two pretty extensions instead of one) + syntax/implementation is trivial + it preserves backwards compatibility - it does force future ebuilds to have the EAPI line in it 6) Putting EAPI inside the ebuild as in #4, and do a one-time rsync location change + it solves 1) + the filenames don't change at all + syntax is trivial, future ebuilds are trivial + it preserves backwards compatibility - it does force future ebuilds to have the EAPI line in it - it seems like quite a hack - and you end up with a dummy portage tree sitting out there for some period of time Personally, I tend to lean towards either just putting the EAPI in the filename, or putting it in the first line and breaking backwards compatibility. The whole reason we're in this mess is the ebuild file format does not include a version field or the equivalent of extensible headers used in other sorts of files (tar, etc). We've certainly broken backwards-compatibility before with other changes within the distro. All we need to do is make it easy to update to a package manager that fixes the break and post instructions prominently, and make sure that package managers support the new process for a few months before taking advantage of it. Some object to parsing out the EAPI without sourcing the ebuild (only bash can source bash). I disagree with this argument - every time you run a shell script it is sourced by something other than bash - the kernel has to figure out what script interpreter to use by parsing the first line. There is no reason we can't use a magic number in the same way with the EAPI. That isn't reason enough on its own to put the EAPI in the filename, but it is a start. Most software packages store version information internal to a file format. I'm actually not aware of many that put it in the filename. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On 10 Jun 2008, at 15:48, Luca Barbato wrote: Fernando J. Pereda wrote: No, it doesn't make parsing faster. Had you bothered to profile any package manager you'd know that. Do you have any number to share? What number are you interested in? - ferdy -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Joe Peterson wrote: No, I have not profiled PMs to try this, but you are saying that reading the first few lines of a file is not faster than sourcing the whole thing with bash? Remember that it could abort the minute it sees a non '#' or blank line, which would be after the first few. Actually, if you specified that the EAPI goes on the first line, then you'd only need to read the first line. And to the extent that the rest got read into memory anyway by the kernel, well then it is already in the cache when whatever mechanism is invoked to parse the rest of the file. I don't think that filename-vs-first-line is going to make a big difference in practical performance. Any time lost in determining EAPI would be time saved in parsing the file. And even though you could ignore unknown EAPIs, I think that in a typical case users would be using an up-to-date package manager that would just end up parsing everything all the time anyway. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 07:49:31 -0600 Joe Peterson [EMAIL PROTECTED] wrote: No, it doesn't make parsing faster. Had you bothered to profile any package manager you'd know that. No, I have not profiled PMs to try this, but you are saying that reading the first few lines of a file is not faster than sourcing the whole thing with bash? Remember that it could abort the minute it sees a non '#' or blank line, which would be after the first few. The package manager does not currently source the whole thing with bash to get the EAPI, nor does it open the ebuild file at all for metadata. You're talking doubling the number of file operations here, and going from extremely good filesystem locality (which means very few seeks) to jumping backwards and forwards all over the place. This is basic stuff you really need to know before you can comment sensibly on a discussion about package metadata. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 09:49:04 -0400 Richard Freeman [EMAIL PROTECTED] wrote: 4) Putting EAPI inside the ebuild, but in a manner that does not require sourcing using bash (ie comment at top of file). + it solves 1) + it keeps pretty file names + syntax/implementation is trivial - it breaks backwards compatibility (eventually - hacks might delay this) - it does force future ebuilds to have the EAPI line in it - it doubles the number of file reads necessary during resolution. - it heavily restricts future syntax and meaning of EAPIs - it makes comments have meaning Most software packages store version information internal to a file format. I'm actually not aware of many that put it in the filename. Most software doesn't have to care about backwards / forwards compatibility. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 09:56:18 -0400 Richard Freeman [EMAIL PROTECTED] wrote: Joe Peterson wrote: No, I have not profiled PMs to try this, but you are saying that reading the first few lines of a file is not faster than sourcing the whole thing with bash? Remember that it could abort the minute it sees a non '#' or blank line, which would be after the first few. Actually, if you specified that the EAPI goes on the first line, then you'd only need to read the first line. And to the extent that the rest got read into memory anyway by the kernel, well then it is already in the cache when whatever mechanism is invoked to parse the rest of the file. Except that currently, the ebuild file isn't opened for read. So it's not in memory at all. I don't think that filename-vs-first-line is going to make a big difference in practical performance. It's about a factor of five difference in cold-cache resolution performance for Paludis. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
On Tue, Jun 10, 2008 at 7:30 PM, Ciaran McCreesh [EMAIL PROTECTED] wrote: [...] - it doubles the number of file reads necessary during resolution. The first read will cause the file to be cached for subsequent reads anyway, so the performance hit boils down to an additional read() call (which will probably be buffered by your file I/O library anyway, so it's unlikely to even result in a context switch). And even without, it is well worth the lack of fugliness in the ebuild name. - it heavily restricts future syntax and meaning of EAPIs Not by much. It's just a header. - it makes comments have meaning Just as much as #!/bin/bash and # vim: ... do Regards, -- Arun Raghavan (http://nemesis.accosted.net) v2sw5Chw4+5ln4pr6$OFck2ma4+9u8w3+1!m?l7+9GSCKi056 e6+9i4b8/9HTAen4+5g4/8APa2Xs8r1/2p5-8 hackerkey.com -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Tue, Jun 10, 2008 at 7:32 PM, Ciaran McCreesh [EMAIL PROTECTED] wrote: [...] I don't think that filename-vs-first-line is going to make a big difference in practical performance. It's about a factor of five difference in cold-cache resolution performance for Paludis. Could you please share more details on the experiment that showed this kind of performance degradation and the numbers, if possible? Thanks, -- Arun Raghavan (http://nemesis.accosted.net) v2sw5Chw4+5ln4pr6$OFck2ma4+9u8w3+1!m?l7+9GSCKi056 e6+9i4b8/9HTAen4+5g4/8APa2Xs8r1/2p5-8 hackerkey.com -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Fernando J. Pereda wrote: On 10 Jun 2008, at 15:48, Luca Barbato wrote: Fernando J. Pereda wrote: No, it doesn't make parsing faster. *Had you bothered to profile any package manager you'd know that.* Do you have any number to share? What number are you interested in? Profiling numbers... lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Ciaran McCreesh a écrit : The package manager does not currently source the whole thing with bash to get the EAPI, nor does it open the ebuild file at all for metadata. You're talking doubling the number of file operations here, and going from extremely good filesystem locality (which means very few seeks) to jumping backwards and forwards all over the place. Yeah, grepping through the ebuild is just bound to get a massive performance hit, I don't have numbers but that just looks obvious. Ciaran, what other pitfalls do you see for using one EAPI file per package (except for the broken compatibility, I know it's a big one) ? This is basic stuff you really need to know before you can comment sensibly on a discussion about package metadata. Then, thanks for explaining those things :) We are learning stuff as we go. -- Rémi Cardona LRI, INRIA [EMAIL PROTECTED] [EMAIL PROTECTED] -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 16:11:49 +0200 Rémi Cardona [EMAIL PROTECTED] wrote: Ciaran McCreesh a écrit : The package manager does not currently source the whole thing with bash to get the EAPI, nor does it open the ebuild file at all for metadata. You're talking doubling the number of file operations here, and going from extremely good filesystem locality (which means very few seeks) to jumping backwards and forwards all over the place. Yeah, grepping through the ebuild is just bound to get a massive performance hit, I don't have numbers but that just looks obvious. No no. Doing the seek to open a file in a different directory and then seeking back to your original directory over and over when otherwise you'd be doing nice linear opens on adjacent inodes in a single directory is where the performance hit is. Paludis is pretty much seek bound in a lot of cases. Ciaran, what other pitfalls do you see for using one EAPI file per package (except for the broken compatibility, I know it's a big one) ? It's moving the relevant information further and further away from where it's supposed to be. It doubles the number of files a developer has to check in order to do simple ebuild maintenance. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 19:54:33 +0530 Arun Raghavan [EMAIL PROTECTED] wrote: Well, most file systems have a local structure for this data (= block group), so it's not going to be a seek that's very far. Secondly, how many ebuilds do you need to read directly to get this data in a typical case? Isn't this what the metadata cache is for? It's a nice local structure when you're just using a) a nice, linear, readahead-friendly-ish slurp of the ebuild directory and b) a nice, linear slurp of files in the metadata cache, which is how things are now. When you move that to having to make alternating use of the metadata cache and the ebuild files, you're suddenly seeking backwards and forwards between two places over and over. - it heavily restricts future syntax and meaning of EAPIs Not by much. It's just a header. !-- EAPI=3 -- Do we want to keep the spec so wide open that we support any format under the Sun that we fancy? Seems like overgeneralizing to me. We want to keep it open enough that we're not going to have to go through yet another backwards compatibility breaking change. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
Ciaran McCreesh a écrit : No no. Doing the seek to open a file in a different directory and then seeking back to your original directory over and over when otherwise you'd be doing nice linear opens on adjacent inodes in a single directory is where the performance hit is. Paludis is pretty much seek bound in a lot of cases. Ok It's moving the relevant information further and further away from where it's supposed to be. It doubles the number of files a developer has to check in order to do simple ebuild maintenance. I said one file per package, not per ebuild. It only doubles the number of files if there's only one ebuild in a given package :) -- Rémi Cardona LRI, INRIA [EMAIL PROTECTED] [EMAIL PROTECTED] -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 19:40:22 +0530 Arun Raghavan [EMAIL PROTECTED] wrote: On Tue, Jun 10, 2008 at 7:32 PM, Ciaran McCreesh [EMAIL PROTECTED] wrote: [...] I don't think that filename-vs-first-line is going to make a big difference in practical performance. It's about a factor of five difference in cold-cache resolution performance for Paludis. Could you please share more details on the experiment that showed this kind of performance degradation and the numbers, if possible? Shove an open and a getline in when doing what would otherwise be a successful cache read. It adds a couple of thousand new open()s on files that would otherwise be left alone, and changes the nice linear slurp up things in this directory in order until we don't need anything else pattern into bounce backwards and forwards between lots of files in two different directories. On a cold cache, which is the most common use case, this is very very nasty. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
On Tue, Jun 10, 2008 at 7:44 PM, Ciaran McCreesh [EMAIL PROTECTED] wrote: [...] The first read will cause the file to be cached for subsequent reads anyway, so the performance hit boils down to an additional read() call (which will probably be buffered by your file I/O library anyway, so it's unlikely to even result in a context switch). And even without, it is well worth the lack of fugliness in the ebuild name. No, it results in a new open() on a file that's elsewhere on disk, which results in two new seeks. You get about fifty seeks per second. Well, most file systems have a local structure for this data (= block group), so it's not going to be a seek that's very far. Secondly, how many ebuilds do you need to read directly to get this data in a typical case? Isn't this what the metadata cache is for? - it heavily restricts future syntax and meaning of EAPIs Not by much. It's just a header. !-- EAPI=3 -- Do we want to keep the spec so wide open that we support any format under the Sun that we fancy? Seems like overgeneralizing to me. Regards, -- Arun Raghavan (http://nemesis.accosted.net) v2sw5Chw4+5ln4pr6$OFck2ma4+9u8w3+1!m?l7+9GSCKi056 e6+9i4b8/9HTAen4+5g4/8APa2Xs8r1/2p5-8 hackerkey.com -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On 10 Jun 2008, at 16:14, Ciaran McCreesh wrote: On Tue, 10 Jun 2008 19:38:52 +0530 Arun Raghavan [EMAIL PROTECTED] wrote: On Tue, Jun 10, 2008 at 7:30 PM, Ciaran McCreesh [EMAIL PROTECTED] wrote: [...] - it doubles the number of file reads necessary during resolution. The first read will cause the file to be cached for subsequent reads anyway, so the performance hit boils down to an additional read() call (which will probably be buffered by your file I/O library anyway, so it's unlikely to even result in a context switch). And even without, it is well worth the lack of fugliness in the ebuild name. No, it results in a new open() on a file that's elsewhere on disk, which results in two new seeks. You get about fifty seeks per second. Plus path resolution, which isn't exactly free - ferdy -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Ciaran McCreesh wrote: On Tue, 10 Jun 2008 19:40:22 +0530 Arun Raghavan [EMAIL PROTECTED] wrote: On Tue, Jun 10, 2008 at 7:32 PM, Ciaran McCreesh [EMAIL PROTECTED] wrote: [...] I don't think that filename-vs-first-line is going to make a big difference in practical performance. It's about a factor of five difference in cold-cache resolution performance for Paludis. Could you please share more details on the experiment that showed this kind of performance degradation and the numbers, if possible? Shove an open and a getline in when doing what would otherwise be a successful cache read. It adds a couple of thousand new open()s on files that would otherwise be left alone, and changes the nice linear slurp up things in this directory in order until we don't need anything else pattern into bounce backwards and forwards between lots of files in two different directories. On a cold cache, which is the most common use case, this is very very nasty. I'm curious as to what operation in particular we're looking at. Let's say I type in paludis --sync: Obviously the first step is to rsync the portage tree. Then we find all modified files (already output by rsync) and update the caches. We already need to fully parse every file to create a dependency tree, so why is it slow to cache the EAPI for each file while we're at it? Next, suppose I type in paludis -pi world: Why wouldn't everything you need not already be in the cache? And if something wasn't, then why is reading in the EAPI any slower than reading in (R)DEPEND/KEYWORDS/IUSE/etc? What specific operation (from an end-user perspective) is improved by putting EAPI in the filename? I'm not interested in theoretical operations like given a portage tree, find the EAPI of every file - users don't do those operations routinely in isolation, but as part of larger operations where doing an open and a getline now just speeds up the open and getline that would be executed 500 lines later anyway. Again, I'm not completely opposed to the idea of putting EAPI in the filename, but I'm not convinced that the arguments in favor of it are as clear-cut as some are suggesting. If everybody was open about the pros/cons of each solution and not suggesting that one solution is near-perfect and the others are total non-starters then this whole debate would probably be a whole lot less contentious. When people perceive that somebody isn't honest about the shortcomings of their own proposal then they're less likely to trust them - it is just a human thing... -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 11:08:21 -0400 Richard Freeman [EMAIL PROTECTED] wrote: I'm curious as to what operation in particular we're looking at. Let's say I type in paludis --sync: paludis --sync doesn't use metadata. Next, suppose I type in paludis -pi world: If it's straight after a --sync, then yes, some things are pre-loaded by rsync. Similarly, if it's straight after other operations, then yes, some things are pre-loaded. But we don't plan for just the use cases that make our life easy. Why wouldn't everything you need not already be in the cache? And if something wasn't, then why is reading in the EAPI any slower than reading in (R)DEPEND/KEYWORDS/IUSE/etc? Currently we don't touch the ebuild's content *at all* for metadata operations, except where there's no or stale metadata cache (which is rare). We can get away with this currently because 0 and 1 have identical cache layouts and PMS has some (necessary) weasel wording; future EAPIs likely won't, so we're back to the chicken / egg problem. So... We either have the EAPI from the extension (which we already have, since we have to read the dir to know what versions are available), in which case we know how to read the metadata cache file. Or we have to open up a file that would otherwise not be opened, just to parse one line so we know how to read the cache file. What specific operation (from an end-user perspective) is improved by putting EAPI in the filename? I'm not interested in theoretical operations like given a portage tree, find the EAPI of every file - users don't do those operations routinely in isolation, but as part of larger operations where doing an open and a getline now just speeds up the open and getline that would be executed 500 lines later anyway. paludis -pi anything on a cold cache. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ciaran McCreesh wrote: | No, it results in a new open() on a file that's elsewhere on disk, which | results in two new seeks. You get about fifty seeks per second. The speed issues aren't really a concern, since the GLEP suggests that the ebuild must be sourced anyway. The GLEP allows for a .ebuild-2 file to contain EAPI=1. This requires the package manager to source the ebuild to find out exactly what EAPI should be being used. The GLEP doesn't strictly outline what happens to a .ebuild-1 containing EAPI=2 for a PM that supports EAPI=2 (which needs to be addressed before the GLEP gets put into action). It does say a developer should not specify both a pre- and post- source EAPI, but the GLEP says that if both exist the post-source one is used. ~ That means all package managers would have to source the ebuilds. Personally, as a developer, I don't want to be messing around with yet more numbers in the ebuild filenames, and I think it looks ugly. Not great arguments, granted. It also feels like a bad idea to separate information required to process the contents of the file from the contents of the file itself. P/PN/PV variables are used in clearly defined locations of the file, and the script could run under a different name and version (as proved by every version bump around). The suggestion seems to be that an ebuild could not run under a different EAPI. In that case, the EAPI version should be embedded in the contents of the file. As people have pointed out though, there's no need to rush this decision. We're simply not going to achieve backwards compatibility forever (we haven't in the past), so why not choose a time period to allow for upgrades (6 months, a year?) and implement a new EAPI definition mechanism that's present in the file and offers a slightly better compromise between the various parties than the current suggestion? It will require a little more patience, but it offers time for a thought out and well designed choice. What's the rush? Mike 5:) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkhOpDEACgkQu7rWomwgFXoFvgCeMxSfRiQLEZr3QyT+Bx4b5aNe /9EAnicrcCQTXxsliAeM4mdisgjO7abg =tGD8 -END PGP SIGNATURE- -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Tue, Jun 10, 2008 at 12:01:00PM +0200, Rémi Cardona wrote: Ciaran McCreesh a écrit : Kills the upgrade path completely. No good. Lemme sum this up in layman's terms : 1) EAPI _has_ to be known before sourcing an ebuild. There's no way to avoid that for various reasons, all 100% valid. sourcing != reading the first line/n first lines/first block and grepping. OG. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Olivier Galibert a écrit : On Tue, Jun 10, 2008 at 12:01:00PM +0200, Rémi Cardona wrote: Ciaran McCreesh a écrit : Kills the upgrade path completely. No good. Lemme sum this up in layman's terms : 1) EAPI _has_ to be known before sourcing an ebuild. There's no way to avoid that for various reasons, all 100% valid. sourcing != reading the first line/n first lines/first block and grepping. IO-wise, it's the same, as most ebuilds fit inside a kernel memory page (ie, 4KB on most arches). So with a cold cache, the time required to actually run bash to source the ebuild won't matter much. But we're getting lost on this thread, let's let it cool off for a little while :) Cheers, Rémi -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Joe Peterson ha scritto:
It was mentioned that comments are to be ignored, but you point out a
perfect and very fundamental example of where this is not true:
#!/usr/bin/env bash
Putting another line close to this one with:
#EAPI=42
or
#!EAPI=42
if you like (conforms more to the shell script specifier), is not too
muchh of a stretch.
The so-called shebang; very good in my opinion!
Works very well for true shell scripts. why it can't work for ebuilds?
First, I think of two alternatives:
1) the interpreter is the EAPI (with /usr/bin/ebuild being EAPI=0)
this will be the first line of a EAPI=0 ebuild:
#!/usr/bin/ebuild
this will be a EAPI=paludis ebuild:
#!/usr/bin/ebuild-paludis
apparently it's the same mechanism the shell uses to execute a shell
script, except for the fact that:
* ebuild is not an executable file
* an external program is used to determine the EAPI (i.e.: extract the
shebang)
2) fixed interpreter name, use arguments for switching EAPIs
this is another option, wich carries itself a GREAT power of leaving an
open door when in the future will happen something similar to what we
have now, where switching the EAPI (or replace EAPI with something else)
is critical and breaks compatibility.
example - EAPI=0:
#!/usr/bin/ebuild
...
another eapi:
#!/usr/bin/ebuild -eapi paludis
This allows virtually unlimited cases similar to this; for example when
switching to another-feature-that-breaks-things, another switch will
be used:
#!/usr/bin/ebuild -eapi foo -fluffy bar
Advantages of both solutions:
* easy to parse
* doesn't break compatibility
What it does not address:
* doesn't break compatibility
In fact, it should break compatibility.
again I can think of two solutions:
1) change the ebuild extension to a different value, once and
(hopefully) forever, e.g. from .ebuild to .xbuild, and leave the minimal
set of .ebuild-s for the upgrade path
2) use two repositories: the legacy one, needed for the upgrade path,
and the shyny-new-repository with the new ebuilds format
best regards,
Federico Ferri
--
#include stdio.h
main(){printf(%x,4275974592);}
-BEGIN GEEK CODE BLOCK-
Version: 3.12
GCS d-(--) a- C++$ UL+++$ L+++$ E--- W++@ w--@
PE PGP+ R- tv-- DI+ D++ h+
--END GEEK CODE BLOCK--
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-dev] Re: GLEP 55
On Tue, Jun 10, 2008 at 10:20 PM, Federico Ferri [EMAIL PROTECTED] wrote: The so-called shebang; very good in my opinion! Works very well for true shell scripts. why it can't work for ebuilds? This option was already discussed when GLEP 55 was proposed, and in my opinion it's totally discarded. The concept of shebang doesn't apply here at all. Plus /usr/bin/ebuild is a binary provided by portage which has nothing to do with the process of sourcing ebuilds nor even with the internals of portage (not to speak of other package managers). Regards, -- Santiago M. Mola Jabber ID: [EMAIL PROTECTED] -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Santiago M. Mola wrote: On Tue, Jun 10, 2008 at 10:20 PM, Federico Ferri [EMAIL PROTECTED] wrote: The so-called shebang; very good in my opinion! Works very well for true shell scripts. why it can't work for ebuilds? This option was already discussed when GLEP 55 was proposed, and in my opinion it's totally discarded. The concept of shebang doesn't apply here at all. Plus /usr/bin/ebuild is a binary provided by portage which has nothing to do with the process of sourcing ebuilds nor even with the internals of portage (not to speak of other package managers). I had thought of this option as well - I agree that it has a lot of issues. I agree that ebuild wouldn't be the right tool, but in the right framework I could see how something like this might work. Here is how a portage tree might be handled by a future package manager that uses shebangs to implement any number of EAPIs that have almost no restrictions on file contents other than the shebang: When new ebuilds are synced into the repository, they are executed. If the interpreter doesn't exist, then it is an unsupported EAPI and the failure is silently ignored (or logged or whatever). If the interpreter does exist it will interpret the file properly - perhaps using a EAPI command-line option from the shebang line or whatever is appropriate for that EAPI which the interpreter obviously understands. If the interpreter determines that the file uses an EAPI it doesn't know, it aborts execution and skips the package. That makes the scheme always backwards compatible (well, at least until the switch to this approach - current package managers wouldn't be compatible). When the new ebuild is executed, it will call appropriate functions to register itself into the local repository. That registration might include callbacks of some kind or whatever - the whole way the ebuild works could vary by EAPI - since the interpreter used by the EAPI could change. The only restriction for compatibility is a standard shebang line. Each ebuild would only be executed upon a change when it is synced, so the overhead isn't huge. At that point all the data is stored in a local cache and the ebuilds themselves don't get touched until they are installed. Installation will work in an EAPI-dependent way - perhaps by executing the ebuild with a particular parameter or environment setting or whatever. The package manager will know since it has already been established that this is a supported EAPI by the fact that the ebuild got processed when it was synced in. This kind of system also makes it easy to support some scenarios that are currently hard: 1. Download a random ebuild off the internet and want to add it to your repository? Rather than having to put it in a local portage tree you could just execute it from anywhere and it would register itself in its present location - and act like any other package in the tree. 2. No need for utilities like ebuild to manipulate the install process - instead just execute the ebuild with the appropriate parameters or whatever to get it to do the install process. On the other hand, this is a big change from the present, and I'm not convinced that it will actually be a big improvement over some of the other EAPI ideas being floated around. However, it is a potentially-neat idea... -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
Richard Freeman wrote: On the other hand, this is a big change from the present, and I'm not convinced that it will actually be a big improvement over some of the other EAPI ideas being floated around. However, it is a potentially-neat idea... Rich, interesting thoughts! But yeah, I agree that for now that is a lot. My idea for the #! thing was much smaller-scale and is a way to add simple syntax to allow declaration of the EAPI in the file with no sourcing and no filename extension mangling (plus, no pre-source/post-source issues): Just have one shebang-like line in the header before any real bash code (#!EAPI=3, #EAPI=3, or whatever is agreed-upon). This is out-of-band meta info, so it's OK that it's in a comment. It's ignored by bash, but it can be read trivially without sourcing. To accelerate things for the tree (I've seen others mention this idea too), store the EAPI in the portage cache when it is generated. -Joe -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On 2008/06/10, Ciaran McCreesh [EMAIL PROTECTED] wrote: Currently we don't touch the ebuild's content *at all* for metadata operations, except where there's no or stale metadata cache (which is rare). We can get away with this currently because 0 and 1 have identical cache layouts and PMS has some (necessary) weasel wording; future EAPIs likely won't, so we're back to the chicken / egg problem. So... We either have the EAPI from the extension (which we already have, since we have to read the dir to know what versions are available), in which case we know how to read the metadata cache file. Or we have to open up a file that would otherwise not be opened, just to parse one line so we know how to read the cache file. Or you apply to future EAPI's cache formats one of the solutions that have been proposed for the ebuild side of the very same chicken / egg problem: for instance, you could use $EAPI as cache filename extension. And that's it, you can get EAPI from the cache, hence no more extra reading of ebuild files, and no more perfs issues. It sounds so simple, am I missing something obvious? -- TGL. -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Wed, 11 Jun 2008 01:36:01 +0200 Thomas de Grenier de Latour [EMAIL PROTECTED] wrote: Or you apply to future EAPI's cache formats one of the solutions that have been proposed for the ebuild side of the very same chicken / egg problem: for instance, you could use $EAPI as cache filename extension. And that's it, you can get EAPI from the cache, hence no more extra reading of ebuild files, and no more perfs issues. It sounds so simple, am I missing something obvious? You're missing the cases where the cache isn't usable. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
On Tue, 10 Jun 2008 19:14:11 +0200 Olivier Galibert [EMAIL PROTECTED] wrote: On Tue, Jun 10, 2008 at 03:02:28PM +0100, Ciaran McCreesh wrote: Except that currently, the ebuild file isn't opened for read. So it's not in memory at all. Why would you need the EAPI before the time when you actually want to interpret the contents? You need the EAPI before you use the metadata. But you don't need the ebuild to get the metadata in the common case. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: GLEP 55
On Wed, Jun 11, 2008 at 8:44 AM, Ciaran McCreesh [EMAIL PROTECTED] wrote: [...] Why would you need the EAPI before the time when you actually want to interpret the contents? You need the EAPI before you use the metadata. But you don't need the ebuild to get the metadata in the common case. Does the cache format _really_ need to be extensible the extent that we're jumping hoops to support arbitrary cache formats? -- Arun Raghavan (http://nemesis.accosted.net) v2sw5Chw4+5ln4pr6$OFck2ma4+9u8w3+1!m?l7+9GSCKi056 e6+9i4b8/9HTAen4+5g4/8APa2Xs8r1/2p5-8 hackerkey.com -- gentoo-dev@lists.gentoo.org mailing list
Re: [gentoo-dev] Re: GLEP 55
On Wed, 11 Jun 2008 09:52:17 +0530 Arun Raghavan [EMAIL PROTECTED] wrote: On Wed, Jun 11, 2008 at 8:44 AM, Ciaran McCreesh [EMAIL PROTECTED] wrote: [...] Why would you need the EAPI before the time when you actually want to interpret the contents? You need the EAPI before you use the metadata. But you don't need the ebuild to get the metadata in the common case. Does the cache format _really_ need to be extensible the extent that we're jumping hoops to support arbitrary cache formats? The cache format needs to be able to have keys added and removed from it. But the cache format is largely irrelevant here. There aren't any non-trivial EAPI related problems introduced by cache that don't already exist without cache. -- Ciaran McCreesh signature.asc Description: PGP signature
