Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Wed, 7 Mar 2012 21:41:02 +0100 Ulrich Mueller u...@gentoo.org wrote: Hi all, The way how we currently specify the EAPI in ebuilds has some problems. For example, there is no sane way to allow usage of features of a new bash version in a new EAPI. So we are currently stuck with bash 3.2. Also changes of global scope behaviour, like addition of new global scope functions (similar to inherit) are not possible. These flaws are outlined in GLEP 55 [1]: | In order to get the EAPI the package manager needs to source the | ebuild, which itself needs the EAPI in the first place. Otherwise it | imposes a serious limitation, namely every ebuild, using any of the | future EAPIs, will have to be source'able by old package managers | [...] The council has voted down GLEP 55 more than a year ago, but at the same time requested that a solution for the mentioned issues should be found. [2] However, there was no progress since then. The issue arose again in bug 402167 [3] where several solutions have been discussed. Below, I try to summarise the possible options resulting from that discussion. *** Proposal 1: Parse the EAPI assignment statement *** This first proposal would require that the syntax of the EAPI assignment statement in ebuilds matches a well defined regular expression. A scan of the Portage tree shows that the statement only occurs in the following variations (using EAPI 4 as example): EAPI=4 EAPI=4 EAPI='4' Sometimes this is followed by whitespace or a comment (starting with a # sign). Also, with very few exceptions the EAPI assignment occurs within the first few lines of the ebuild. For the vast majority of ebuilds it is in line 5. Written in a more formal way, appropriate for a specification: - Ebuilds must contain at most one EAPI assignment statement. - It must occur within the first N lines of the ebuild (N=10 and N=30 have been suggested). - The statement must match the following regular expression (extended regexp syntax): ^[ \t]*EAPI=([']?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$ Note: The first and the third point are already fulfilled by all ebuilds in the Portage tree. The second point will require very few ebuilds to be changed (9 packages for N=10, or 2 packages for N=30). The package manager would determine the EAPI by parsing the assignment with above regular expression. A sanity check would be added. Citing Zac Medico in [3]: The fact that we can compare the probed EAPI to the actual EAPI variable after the ebuild is sourced seems like a perfect sanity check. We could easily detect inconsistencies and flag such ebuilds as invalid, providing a reliable feedback mechanism to ebuild developers. This proposal comes in two variants: 1a) The change is applied retroactively for all EAPIs. 1b) It is only applied for EAPI 5 and later (which means that the result of the EAPI parsing would be discarded for earlier EAPIs). *** Proposal 2: EAPI in header comment *** A different approach would be to specify the EAPI in a specially formatted comment in the ebuild's header. No syntax has been suggested yet, but I believe that the following would work as a specification: - The EAPI must be declared in a special comment in the first line of the ebuild's header, as follows: - The first line of the ebuild must contain the word ebuild, followed by whitespace, followed by the EAPI, followed by end-of-line or whitespace. Again, the proposal comes in two variants: 2a) It is combined with a one time change of the file extension, like .ebuild - .eb. 2b) The usual EAPI assignment statement in the ebuild is still required, at least for a transition period. In the 2a case, the EAPI variable could be made read-only in bash before sourcing the ebuild. In the 2b case, a sanity check similar to the one mentioned above would be added. What do you think? (I really hope for a constructive discussion here. So, if you want to comment that all of the above proposals suck and GLEP 55 is much superior, then please open a new thread for it.) Ulrich Currently 5 proposals are listed on the wiki. [4] While all of them have some temptations the actual goal is to make obtaining the EAPI the very first step so everything else can be defined in terms of EAPI and so immediately deployable in future. This are changes in atom syntax like needed for GLEP 54 or those bash feature often mentioned besides many other things one can think of. GLEP 55 requires changing ebuild extensions on a regular basis but doesn't impose any limit on the ebuild format or atom syntax, only the file extensions would be imposed. The ebuild extensions for GLEP 55 would likely always be ebuild-integer as integers are reserved for future use by Gentoo. While for example .ebuild-5 is still recognised as an ebuild; .eb .ebld .ebd .bld .dliube .dlbe .be are not. This brings me to the point if not GLEP 55 then only if we
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Sun, Mar 18, 2012 at 3:23 AM, Ralph Sennhauser s...@gentoo.org wrote: The ebuild extensions for GLEP 55 would likely always be ebuild-integer as integers are reserved for future use by Gentoo. Looking at GLEP 55 the proposal was ebuild-EAPI - not ebuild-integer. I don't believe there is any restriction that EAPIs be integers or incremental in nature. Don't mean to nitpick, but perhaps not all realize this... Rich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Sun, 18 Mar 2012, Ralph Sennhauser wrote: If we want to keep .ebuild but avoid the compat issue another variant would be EAPI in header comment and one-time change of ebuild location or more formal: 6 EAPI in header comment and one-time change of ebuild location: - add a directory $CATEGORY/$PN/ebuilds to ebuild repositories. - all files in $CATEGORY/$PN/ebuilds are ebuilds and are using a well defined first line to denote the EAPI. - For practical reasons the header should be a bash comment. PMs shouldn't have to remove or skip first line from file for further processing of ebuilds supporting bash comments. - the .ebuild extension can be kept but could be changed if ever desired. This due to the filename only having meaning if the EAPI of the file is known. Similar ideas have been discussed before, in 2007 and 2009: http://archives.gentoo.org/gentoo-dev/msg_fe3f0b9d050ead86aed9b42ce7ec93b0.xml http://archives.gentoo.org/gentoo-dev/msg_2e41942be33d8595cf7152aa91417fbe.xml Comparing this with GLEP 55 then this allows us to keep .ebuild in return of some overhead with roughly the same pros and cons otherwise, right? From a technical point of view, it's the same pros and cons. There are however non-technical aspects for all these propositions. For example, we would have to live with ebuilds in different directories for a long transition period (as we would have to live with two file extensions for a long time if we changed that). Ulrich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, Mar 12, 2012 at 09:05:26AM -0700, Zac Medico wrote: On 03/12/2012 01:36 AM, Brian Harring wrote: On Sun, Mar 11, 2012 at 09:08:24PM -0700, Zac Medico wrote: 1) User downloads an overlay that doesn't provide cache. We want the package manager to give a pretty EAPI unsupported message, rather than spit out some bash syntax errors. This criticsm pretty much applies *strictly to the existing implementation*. It's disenguous busting it in this fashion. EAPI as a function explicitly gives it an out before hitting any of that, eliminating your entire critique. Same goes for parsing it out of the ebuild, or renaming the extension. You're assuming that the ebuild calls your eapi() function before it uses any syntax that's unsupported by the user's installed version of bash. A bit, although that's a pretty valid assumption frankly. For current bash syntax, the only thing they could do that would cause issues is bash regex's for example- which if they have regex's prior to the eapi invocation, they're doing something stupid anyways. Regardless, detecting and suppressing isn't too hard- start sourcing w/ `set -e`, disabling that once `eapi` has been invoked for example. Pretty sure people will scream that's horrible, but it's surprisingly effective. 1) PM still doesn't support that EAPI, looks at the cache/ebuild: checksums are the same, thus the stored EAPI is trustable, leading to the PM knowing it still can't process that ebuild and masking it appropriately. You're assuming that cache is provided by the repo, Sigh. I'm making no such assumptions, nor would I; it's a stupid line of thought. All of this has to function in the absence of a cache, and that's a core usage scenario. which is not guaranteed, depending on the source. Even if the cache does exist, then you're assuming it's in a format that the package manager can reliably parse the EAPI from, even though that EAPI may not be supported. That may or may not reliable assumption, and having a pre-defined protocol to directly obtain the EAPI without using the cache is much more reliable. This is a nonstatement. To deal w/ the cache (validate it) you have to be able to reliably pull EAPI out of the ebuild. That's what this whole fucking discussion is about; really not sure why you're trying to argue this point against eapi as a function. As I already laid out, it can deal w/it, same as the rest. Importantly, the approach can also work across the transition period preventing current-day PMs (using current EAPI mechanisms) from breaking when used against later EAPIs that were released via `eapi as a function`. What I'd like to see, is accuracy in this discussion. Skip the handwavey complexity! complexity! complexity! crap, same for selective robustness definitions. Past attempts at this discussion mostly failed due to people pulling crap like this and frankly it just pisses people off. It's just a symptom of people not abiding by the KISS principle. When you start talking about an approach such as the eapi() function approach which introduces lots of unnecessary complexity, it naturally makes the whole discussion more complex and hand-wavey. With respect; you're proposing we go gum up version parsing via shoving EAPI directly into it. Literally, make what is already a complex mess, worse. Apply some KISS to your proposal please. ;) Just hammering the point home; compatibility *is* complex. Claiming otherwise is naive. Case in point: your proposal breaks the shit out of any current-day package manager that saw such a filename. I really have a hard time reading your posts when basic issues like that aren't paid attention to, but you've no problems claiming complexity/brokenness in other proposals. As I said, I'd like to see some accuracy; not hand wavy buzzwords. I'd much more like to see prototypes of peoples proposals in addition since at least that way they would flush out the breakages in their proposals (potentially dropping it in the process since some of these are pretty half baked). ~brian
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/13/2012 07:01 PM, Brian Harring wrote: With respect; you're proposing we go gum up version parsing via shoving EAPI directly into it. Literally, make what is already a complex mess, worse. Apply some KISS to your proposal please. ;) Just hammering the point home; compatibility *is* complex. Claiming otherwise is naive. Case in point: your proposal breaks the shit out of any current-day package manager that saw such a filename. I'm not really interested in GLEP 55 or variants of it. I was just saying that if we do go with a GLEP 55 variant, then I'd prefer one that's similar to the EAPI in the filename with one-time extension change option [1]. There are plenty of ways to do that without making it difficult to separate the EAPI from the version part. [1] http://www.gentoo.org/proj/en/glep/glep-0055.html#eapi-in-the-filename-with-one-time-extension-change -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 12 March 2012 15:20, Rich Freeman ri...@gentoo.org wrote: On Sun, Mar 11, 2012 at 10:03 PM, Brian Harring ferri...@gmail.com wrote: Pragmatic reality, the eapi function actually would work fine. As pointed out elsewhere, bash parses as it goes- which isn't going to change. Unless the ebuild isn't written in bash... How do you source the ebuild if you don't know what to use to source it? How do you know what to use to source it if you don't know the EAPI? Right now all the existing EAPIs use bash, but there is no reason the file couldn't be xml, or python, or just about anything else. A convention that is often used in this scenario is to combine a hashbang call with that specific sourcer stripping that hashbang prior to the parse. ie: foo.ebuild: #!/usr/bin/env eapi-xml-5 would send the current file ( foo.ebuild ) to the process eapi-xml-5 ( as defined by the current $PATH setting ) This process can easily then strip the #! stanza before sending the content to an XML parser. ( ps. while I wouldn't actually use XML, its a good example case because standard shell-script style commenting is illegal syntax in XML ) The benefits of this approach seem obvious, but there are also obvious downsides. 1. PRO: Unlike /usr/bin/eapi , this style is PMS Agnostic, and only requires the PMS have a process named eapi-xml-5 *somewhere* in the system under $PATH 2. CON: Unlike /usr/bin/eapi , you're limited by what you can send it , /usr/bin/env eapi xml-5 would be preferable syntax imo, but it doesn't work, because its parsed as [ '/usr/bin/env' , 'eapi xml-5' ] which then yeilds a permission denied due to no command with that name existing. 3. PRO: There's not /much/ risk of a user trying to run it directly, mostly because there's no +x bit 4. CON: This syntax is going to conflict with whatever language we are proceeding, which while this caveat is dealt with by the command that the file is dispatched to, its a lot of work getting everything else to play ball ( editors won't understand, linters wont understand, various other tools that work strictly with the native forms of that language wont understand ) and this problem exists for *every* case where the coding system we're targeting doesn't natively support whatever magic indicators we're trying to stuff in the file. 5. PRO: detecting this notation being used in a file is cheap-ish , you only have to read 2 characters into the file to know its using this notation. As a result of all these, it seems to me if we ever want to support non-bash or non-scripting sources ( ie: YAML, JSON, XML , etc ) or any other language which doesn't support # is a comment , we're going to be pretty much forced to declare the EAPI outside the file somehow. Whether this is via the filename ( yes, I've seen that debate ) or via some other metadata file ( which seems more poison than cure ) would be the real issue. Having to store this in a metadata file that maps *.ebuild to EAPI would seem like a solution that would only increase the filesize of portage, and slow down runtime substantially, unless of course you had a per-repository index that sped this up and was generated during the metadata/ generation phase. Then $PMS could just load that file from each repository , and assuming the cache was still valid, it could easily know the EAPI for any .ebuild it would source in advance. -- Kent perl -e print substr( \edrgmaM SPA NOcomil.ic\\@tfrken\, \$_ * 3, 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 12 March 2012 15:24, Alec Warner anta...@gentoo.org wrote: I will stab the next person who suggests 'xml-like ebuilds.' State-fully coded ebuilds, while perhaps not to your liking, for some code-types can be incredibly useful. For example, 9/10 perl-module ebuilds don't need any code at all in the ebuild itself, it could do most of the work via structured data, with the rest of the work done by an e-class. However, bash has serious problems when you want to represent hierarchical data. -- Kent perl -e print substr( \edrgmaM SPA NOcomil.ic\\@tfrken\, \$_ * 3, 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/11/2012 11:50 PM, Kent Fredric wrote: #!/usr/bin/env eapi-xml-5 would send the current file ( foo.ebuild ) to the process eapi-xml-5 ( as defined by the current $PATH setting ) All we need is a way for the package manager to probe the EAPI. Spawning a process to do that is just overkill. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 12 March 2012 20:08, Zac Medico zmed...@gentoo.org wrote: On 03/11/2012 11:50 PM, Kent Fredric wrote: #!/usr/bin/env eapi-xml-5 would send the current file ( foo.ebuild ) to the process eapi-xml-5 ( as defined by the current $PATH setting ) All we need is a way for the package manager to probe the EAPI. Spawning a process to do that is just overkill. Yeah, but if you read the rest of my message, you'll see my argument that hinges around the fact most proposals so far tend to focus around the fact the current format for ebuild is bash, and bash only, and I think it very hard to implement a language agnostic way to specify the EAPI in a way that will work on languages where you a) Can't use # to mark comments b) May not even be working with the file directly, and perhaps generating a .json file from a data-structure being serialised, and the key named EAPI may be unpredictably placed near the end of the .json file. Case b is entirely plausible, and even realistic, because non-script-based data-structures often don't bear importance on the order of elements. -- Kent perl -e print substr( \edrgmaM SPA NOcomil.ic\\@tfrken\, \$_ * 3, 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, 12 Mar 2012 00:08:46 -0700 Zac Medico zmed...@gentoo.org wrote: On 03/11/2012 11:50 PM, Kent Fredric wrote: #!/usr/bin/env eapi-xml-5 would send the current file ( foo.ebuild ) to the process eapi-xml-5 ( as defined by the current $PATH setting ) All we need is a way for the package manager to probe the EAPI. Spawning a process to do that is just overkill. And we could just use a good regex for that instead. Something like: [eE][aA][pP][iI] whitespace-or-symbols [a-z0-9-+]+ and just require users for this to be the first thing declared in an ebuild. Of course, this could make problems with stuff like: # EAPI 4 because of foobarbaz EAPI=4 (on the other hand, in this particular case it will fetch '4' anyway). And this will work as well with: eapi15-xml/eapi and - eapi: 15-yaml -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, 12 Mar 2012 09:27:11 +0100 Michał Górny mgo...@gentoo.org wrote: eapi15-xml/eapi and - eapi: 15-yaml You're carefully concocting your examples to make it look like it should work. If you go the XML route, though, the EAPI would either be in a DTD or as eapi value=15 /. Part of the point of all of this is that we shouldn't have to guess what future EAPIs will look like. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Sun, Mar 11, 2012 at 09:08:24PM -0700, Zac Medico wrote: On 03/11/2012 06:55 PM, Brian Harring wrote: On Sat, Mar 10, 2012 at 08:06:50AM -0800, Zac Medico wrote: Yeah. Another way of putting it is that the requirement to spawn a bash process and source the ebuild adds a ridiculous amount of unnecessary complexity, in violation of the KISS principle [1]. This statement is incorrect. Even if EAPI could be parsed via some non sourcing approach, we *still* have to source the ebuild to get the metadata for when the EAPI is supported (the vast majority of usage). That complexity is there one way or another- we wouldn't be trying to extract the EAPI from the ebuild unless the cache was invalid/missing. There are a couple of other cases worth considering: 1) User downloads an overlay that doesn't provide cache. We want the package manager to give a pretty EAPI unsupported message, rather than spit out some bash syntax errors. This criticsm pretty much applies *strictly to the existing implementation*. It's disenguous busting it in this fashion. EAPI as a function explicitly gives it an out before hitting any of that, eliminating your entire critique. Same goes for parsing it out of the ebuild, or renaming the extension. 2) You're assuming that a package manager can validate cache for an EAPI that it doesn't necessarily support. Actually, I'm not. That's a somewhat fragile assumption, given the complexities of cache validation, which involve verification all files that affect metadata and those files may vary depending on the EAPI. This is a fair bit of handwavey bullshit. The same complexities of cache validation we have to by default deal with for valid/supported EAPIs; the case for unsupported EAPIs is actually simpler than for supported EAPis, and up until 7ddb7d30, was likely working fine in portage. Tweaking portage to restore that support is pretty easy. Also note that with the sole exception of g55, every implementation has to deal with this specific issue, even g55 isn't fully exempt from it since in cleansing a cache, there is the open question of pruning cache entries for ebuilds known, but not understood by the local PM. Either way, the algo is as follows: PM pulls EAPI from the ebuild (regardless of method); PM recognizes that it can't handle it- thus it stores EAPI:-$EAPI, and the normal cache ebuild checksumming; whether it be mtime, md5 for md5-dict, or whatever form new caches choose to use. For attempts to load, the cache subsystem checks that validation before trusting the cache entry- this is in place and has been for a very, very long time. This isn't anything new. So... the entry point is the ebuild; we have checksums for it, and a prefixed versions of it's EAPI stored. The sole deal breaker here is if we were to decide to allow eclasses to set EAPI (rather than the current you're not supposed to, although we don't explicitly block it- which could be tightened to be disallowed at the implementation level for EAPI4). The following scenarios exist: 1) PM still doesn't support that EAPI, looks at the cache/ebuild: checksums are the same, thus the stored EAPI is trustable, leading to the PM knowing it still can't process that ebuild and masking it appropriately. 2) PM still doesn't support that EAPI, looks, sees that the checksums no longer match the ebuild. Re-pulls metadata/EAPI; if the ebuild has been changed to a supported EAPI, continues on its way. If not, stores the checksum/negated EAPI, and masks appropriately. 3) PM now supports that EAPI, looks, sees the negated eapi and recognizes it as one it knowns, and forces regeneration. All of those scenarios are there, and easy enough to handle- this applies for the existing EAPI implementation additionally. Now, this isn't to say that someone can't go and try to structure new features explicitly to defeat the existing validations. They could. The question here is whether or not new features would actually *break* that w/out intentionally going out of their way to do so (aka, skip the strawman arguments, actual examples required). Phrasing it more bluntly: you can only avoid the sourcing step if you can isolate that the EAPI is unsupported (which is extremely rare in terms of actual user experience). For the rest of the time (well past the 99% mark) sourcing is the immediate step following. For the sake of being robust in all possible cases, it's just a lot simpler if we can obtain the EAPI simply and reliably, without spawning bash to source the ebuild. This still is arguable; any performant PM is going to have to do the cache dance I mentioned above, or maintain an in memory list of ebuilds it knows it can't handle (which if you want to talk robust, fails miserably if the API isn't designed for sync invalidation of that global list). Complexity, oh my. That's the kicker from where I'm sitting; you're so focused
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 12 March 2012 21:27, Michał Górny mgo...@gentoo.org wrote: And we could just use a good regex for that instead. Something like: [eE][aA][pP][iI] whitespace-or-symbols [a-z0-9-+]+ and just require users for this to be the first thing declared in an ebuild. Of course, this could make problems with stuff like: # EAPI 4 because of foobarbaz EAPI=4 (on the other hand, in this particular case it will fetch '4' anyway). And this will work as well with: eapi15-xml/eapi and - eapi: 15-yaml Also, remember the proposal is to read it only from the first 10-30 lines of the file, and if you're *generating* YAML/XML , then this is not necessarily guaranteed. Some generation tools emit keys in alphanumeric ordering, others psuedo-randomly, and the EAPI declaration line in some formats could easily be on the very last line of the file. And then the regexp could falsely detect something in another key the original author had not intended as an EAPI definition, but merely a comment, ... ie: generateddata comment This is a user comment, hurr, I hate EAPI 5 /comment eapi value=15 / /generateddata -- Kent perl -e print substr( \edrgmaM SPA NOcomil.ic\\@tfrken\, \$_ * 3, 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, 12 Mar 2012 08:30:19 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Mon, 12 Mar 2012 09:27:11 +0100 Michał Górny mgo...@gentoo.org wrote: eapi15-xml/eapi and - eapi: 15-yaml You're carefully concocting your examples to make it look like it should work. Or I am just printing the first thing that comes into my head. If you go the XML route, though, the EAPI would either be in a DTD Like .../gentoo/eapi/15-xml.dtd ? That would match my regex as well, unless we allow dots in EAPI. or as eapi value=15 /. No, definitely not. That's not the XML style. Part of the point of all of this is that we shouldn't have to guess what future EAPIs will look like. I'm just suggesting a way which will support a little more than bash-based solutions. We could also assume that if a file doesn't match the regexp at all, it's a unsupported EAPI. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, 12 Mar 2012 21:39:52 +1300 Kent Fredric kentfred...@gmail.com wrote: On 12 March 2012 21:27, Michał Górny mgo...@gentoo.org wrote: And we could just use a good regex for that instead. Something like: [eE][aA][pP][iI] whitespace-or-symbols [a-z0-9-+]+ and just require users for this to be the first thing declared in an ebuild. Of course, this could make problems with stuff like: # EAPI 4 because of foobarbaz EAPI=4 (on the other hand, in this particular case it will fetch '4' anyway). And this will work as well with: eapi15-xml/eapi and - eapi: 15-yaml Also, remember the proposal is to read it only from the first 10-30 lines of the file, and if you're *generating* YAML/XML , then this is not necessarily guaranteed. Some generation tools emit keys in alphanumeric ordering, others psuedo-randomly, and the EAPI declaration line in some formats could easily be on the very last line of the file. You should ensure your tool does the right thing, I'd say. XML with DTD often requires specific element order, so such a tool is simply bound to fail with XML. For yaml, it's a fair point. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 12 March 2012 22:09, Michał Górny mgo...@gentoo.org wrote: or as eapi value=15 /. No, definitely not. That's not the XML style. Sure, but these examples are just examples after all. And XML is only being used for an example use case, but there are many more structured formats than XML. Some of us are mostly just worried that the proposals as they stand won't be resilient enough to allow a future that isn't bash. Part of the point of all of this is that we shouldn't have to guess what future EAPIs will look like. I'm just suggesting a way which will support a little more than bash-based solutions. We could also assume that if a file doesn't match the regexp at all, it's a unsupported EAPI. I just find a top-down regexp solution dangerously naive, as its infering that the first line that matches the regexp *is* the EAPI requirement field, when depending on the actual format used, that may not be the case. If for example, a format is machine generated, and the EAPI declaration accidentally comes after something that *isnt* an EAPI declaration but by the regexp, LOOKS like one, then the probing mechanism will resolve the WRONG value. And that doesn't strike me as being very resilient. -- Kent perl -e print substr( \edrgmaM SPA NOcomil.ic\\@tfrken\, \$_ * 3, 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, 12 Mar 2012, Kent Fredric wrote: I just find a top-down regexp solution dangerously naive, as its infering that the first line that matches the regexp *is* the EAPI requirement field, when depending on the actual format used, that may not be the case. There's little danger if we require the EAPI specification to be in the first line of the ebuild. Of course the regexp should be general enough to account for a non-bash comment syntax. Ulrich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 12 March 2012 22:48, Ulrich Mueller u...@gentoo.org wrote: On Mon, 12 Mar 2012, Kent Fredric wrote: There's little danger if we require the EAPI specification to be in the first line of the ebuild. Of course the regexp should be general enough to account for a non-bash comment syntax. There's the obvious case of compiled-binaries where that might not be possible, but thats definately strawman stuff and I wouldn't support that sort of nonsense anyway. Compiled binaries for ebuilds can gtfo. -- Kent perl -e print substr( \edrgmaM SPA NOcomil.ic\\@tfrken\, \$_ * 3, 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, Mar 12, 2012 at 6:12 AM, Kent Fredric kentfred...@gmail.com wrote: There's the obvious case of compiled-binaries where that might not be possible, but thats definately strawman stuff and I wouldn't support that sort of nonsense anyway. Compiled binaries for ebuilds can gtfo. Why do I feel like a similar debate must have happened on some list back when the SMTP message terminator was worked out? The problem is that we're running into is the use of in-band control information. An advantage of sticking the EAPI in the filename is that it is completely out-of-band. Other options like an external file or POSIX attributes have the same advantage (with the disadvantage that they are harder to keep in-sync or support in general). If you want to stick the EAPI inside the file then we have to go with one of two routes: 1. Make the file directly interpret-able by some existing tool like bash, xml parsers, or (gtfo) ld-linux.so. This requires making the EAPI in-band info, and thus if the file format changes there is risk that any existing code that interprets EAPIs could break, to the degree that we weren't infinitely clever in designing the spec. 2. Make the file require pre-processing before being fed to another interpreter. This breaks syntax-highlighting and such, and is generally more of a pita. However, this allows in-file EAPI info to remain out-of-band. An ebuild might be an xml file with EAPI and other meta-data, and might contain sections that get ripped out and handed over to bash, python, or ld-linux.so to the extent that you want to be told to gtfo. In my thinking the option with the fewest drawbacks and the greatest extent of future-proofing is to go with GLEP 55 and stick it in the filename. Every filesystem and rsync tool handles filenames, they're impossible to separate from file contents, and every interpreter out there ignores them. If we don't want to stick it in the filename then either we have to decide that whatever we come up with could break, or that we really have to change what ebuilds look like. Rich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, 12 Mar 2012 01:36:12 -0700 Brian Harring ferri...@gmail.com wrote: Also note that with the sole exception of g55 ... and does so at the same robustness as everything sans g55 ... G55 is the sole exception. Interesting pattern, huh? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/12/2012 01:36 AM, Brian Harring wrote: On Sun, Mar 11, 2012 at 09:08:24PM -0700, Zac Medico wrote: 1) User downloads an overlay that doesn't provide cache. We want the package manager to give a pretty EAPI unsupported message, rather than spit out some bash syntax errors. This criticsm pretty much applies *strictly to the existing implementation*. It's disenguous busting it in this fashion. EAPI as a function explicitly gives it an out before hitting any of that, eliminating your entire critique. Same goes for parsing it out of the ebuild, or renaming the extension. You're assuming that the ebuild calls your eapi() function before it uses any syntax that's unsupported by the user's installed version of bash. 1) PM still doesn't support that EAPI, looks at the cache/ebuild: checksums are the same, thus the stored EAPI is trustable, leading to the PM knowing it still can't process that ebuild and masking it appropriately. You're assuming that cache is provided by the repo, which is not guaranteed, depending on the source. Even if the cache does exist, then you're assuming it's in a format that the package manager can reliably parse the EAPI from, even though that EAPI may not be supported. That may or may not reliable assumption, and having a pre-defined protocol to directly obtain the EAPI without using the cache is much more reliable. What I'd like to see, is accuracy in this discussion. Skip the handwavey complexity! complexity! complexity! crap, same for selective robustness definitions. Past attempts at this discussion mostly failed due to people pulling crap like this and frankly it just pisses people off. It's just a symptom of people not abiding by the KISS principle. When you start talking about an approach such as the eapi() function approach which introduces lots of unnecessary complexity, it naturally makes the whole discussion more complex and hand-wavey. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, 12 Mar 2012 09:05:26 -0700 Zac Medico zmed...@gentoo.org wrote: It's just a symptom of people not abiding by the KISS principle. Abiding by the KISS principle is what got us into this mess in the first place. EAPI as a metadata variable is too simple to allow us to do what we want to do. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/12/2012 09:12 AM, Ciaran McCreesh wrote: On Mon, 12 Mar 2012 09:05:26 -0700 Zac Medico zmed...@gentoo.org wrote: It's just a symptom of people not abiding by the KISS principle. Abiding by the KISS principle is what got us into this mess in the first place. EAPI as a metadata variable is too simple to allow us to do what we want to do. If our goal is to obtain the EAPI before the ebuild is sourced, then GLEP 55 is more in alignment with the KISS principle. The EAPI as metadata variable approach simply wasn't designed with our current goal in mind. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/12/2012 02:16 AM, Kent Fredric wrote: I just find a top-down regexp solution dangerously naive, as its infering that the first line that matches the regexp *is* the EAPI requirement field, when depending on the actual format used, that may not be the case. If for example, a format is machine generated, and the EAPI declaration accidentally comes after something that *isnt* an EAPI declaration but by the regexp, LOOKS like one, then the probing mechanism will resolve the WRONG value. And that doesn't strike me as being very resilient. It would be very fragile without the sanity check / feedback mechanism that's already been suggested [1]. The idea is to compare the probed EAPI with the result that's obtained from bash, and treat the ebuild as invalid if they are not identical. This would allow ebuilds that don't fit our EAPI probing mechanism to be immediately detected so that the ebuild developer could fix them. In practice, this will not be much of an issue, since the EAPI assignment will be required to be very close to the top of the file, and the package manager will stop searching as soon as it finds the first match. [1] https://bugs.gentoo.org/show_bug.cgi?id=402167#c18 -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, Mar 12, 2012 at 1:01 PM, Zac Medico zmed...@gentoo.org wrote: It would be very fragile without the sanity check / feedback mechanism that's already been suggested. Another obvious check is to have repoman run a grep with the regexp and give an error if there is not exactly one match. Rich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/12/2012 10:30 AM, Rich Freeman wrote: On Mon, Mar 12, 2012 at 1:01 PM, Zac Medico zmed...@gentoo.org wrote: It would be very fragile without the sanity check / feedback mechanism that's already been suggested. Another obvious check is to have repoman run a grep with the regexp and give an error if there is not exactly one match. If we want to handle every possible screwup, including stray EAPI assignments inside inherited eclasses, we still need to compare the probed value to the value that's obtained from bash. I guess you're hinting at using a non-bash ebuild format? Even if in that case, the the package manager should simply use whatever interpreter is appropriate for the probed EAPI. So, if EAPI 7 is a posix-shell format, and the probed EAPI is 7, then the package manager should source the ebuild with it's posix-shell instead of bash. If the probed EAPI is not a supported EAPI, then it should skip the sourcing entirely, and report the ebuild as having an unsupported EAPI. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mon, Mar 12, 2012 at 1:46 PM, Zac Medico zmed...@gentoo.org wrote: If we want to handle every possible screwup, including stray EAPI assignments inside inherited eclasses, we still need to compare the probed value to the value that's obtained from bash. Well, I wasn't intending to suggest that the repoman check need be the only one. However, preventing problems is at least as useful as detecting them. That said, it would probably best to have exactly one way to determine the official EAPI. If that is to parse the filename, then parse the filename. If it is to grep for a regexp and expect exactly one hit and parse it in a certain way, then do that. It might be that the one official way is to grep for a regexp and if you find it, use it, otherwise assume 0, 1, 2, 3, or 4 and source the ebuild for it. That still gives you only a single answer (well, except in situations where the current way is already broken). If people want to abuse the EAPI syntax I suppose we can generate an error, but ignoring it might be just as valid behavior. I'm not sure what happens if you define PN/etc in ebuild besides things breaking in a horrible manner. I'd put changing EAPI in the same category. Rich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, Mar 09, 2012 at 06:52:40PM +0100, Micha?? G??rny wrote: On Fri, 09 Mar 2012 12:31:24 -0500 Michael Orlitzky mich...@orlitzky.com wrote: On 03/09/12 12:11, Ulrich Mueller wrote: On Fri, 09 Mar 2012, Michael Orlitzky wrote: What if bash starts to parse the script completely and barfs at 'syntax error' before it starts executing stuff? It doesn't parse the script completely, it executes line-by-line, so we can bail out early. How can you tell that this behaviour won't be changed in a future bash version? Who's to say that in the future my computer won't be made out of delicious ice cream, eliminating the need for EAPIs entirely? Chances are, this would break thousands of scripts, so we hope they wouldn't do it. If it does happen, we either deal with it then, or don't upgrade to that version of bash -- the same as we would do with any other massive breaking change. Thousands of scripts? So... you're saying that people actually use thousands of scripts which have invalid syntax... Just a note; you need to look into how aliases work. That right there unfortunately means bash isn't going to pre-parse, not as long as aliases are supported. Back to your arguing... ~brian
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Sat, Mar 10, 2012 at 08:06:50AM -0800, Zac Medico wrote: On 03/09/2012 11:20 AM, Ciaran McCreesh wrote: On Fri, 09 Mar 2012 11:49:44 -0500 Michael Orlitzky mich...@orlitzky.com wrote: isnt the whole point of the proposal to get eapi without sourcing ? so that we can use new bash features at local or global scope without risking that people with an old bash get syntax errors trying to get the eapi Right. Michael has lost sight of the goal and is moving off on a tangent. The point was to be able to get the EAPI without crashing if the ebuild uses newer features. No, it's not. There's more to it than that. Some EAPIs really require defining certain environment variables, shell options, sandbox things etc *before* the sourcing starts. It's a massive pain in the ass to try to handle setting that kind of thing on the fly once the sourcing has already started. Knowing the EAPI before having to spawn a bash process isn't just about performance, it's also about making ebuilds much less difficult to deal with. Yeah. Another way of putting it is that the requirement to spawn a bash process and source the ebuild adds a ridiculous amount of unnecessary complexity, in violation of the KISS principle [1]. This statement is incorrect. Even if EAPI could be parsed via some non sourcing approach, we *still* have to source the ebuild to get the metadata for when the EAPI is supported (the vast majority of usage). That complexity is there one way or another- we wouldn't be trying to extract the EAPI from the ebuild unless the cache was invalid/missing. Phrasing it more bluntly: you can only avoid the sourcing step if you can isolate that the EAPI is unsupported (which is extremely rare in terms of actual user experience). For the rest of the time (well past the 99% mark) sourcing is the immediate step following. Also, stop referencing wikipedia. People know what trivial objection and KISS is. Pointing at random wikipedia links when people object is just a form of fallacious argument from authority ( http://en.wikipedia.org/wiki/Argument_from_authority ). :P ~brian
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, Mar 09, 2012 at 09:47:50AM -0800, Zac Medico wrote: On 03/09/2012 09:31 AM, Michael Orlitzky wrote: On 03/09/12 12:11, Ulrich Mueller wrote: On Fri, 09 Mar 2012, Michael Orlitzky wrote: What if bash starts to parse the script completely and barfs at 'syntax error' before it starts executing stuff? It doesn't parse the script completely, it executes line-by-line, so we can bail out early. How can you tell that this behaviour won't be changed in a future bash version? Who's to say that in the future my computer won't be made out of delicious ice cream, eliminating the need for EAPIs entirely? Chances are, this would break thousands of scripts, so we hope they wouldn't do it. If it does happen, we either deal with it then, or don't upgrade to that version of bash -- the same as we would do with any other massive breaking change. Ulrich is talking about extensions which require a newer version of bash. These kinds of extensions are quite common and don't cause massive breaking because people simply have to upgrade bash in order to use the new extensions, and their old scripts continue to run because the new extensions don't interfere with backward compatibility. Your eapi() function proposal is especially fragile in this context because it assumes that the installed version of bash will be able to execute a script that may target a newer version of bash. This is a special case that is typically not a problem, although it is a major problem under the specific conditions that your eapi() function approach induces. Pragmatic reality, the eapi function actually would work fine. As pointed out elsewhere, bash parses as it goes- which isn't going to change. If someone invokes 'eapi happy-meal' and it's not supported, the sourcing is stopped immediately, cache gets -happy-meal for the EAPI, and the pm continues to ignore the ebuild till either the ebuilds mtime changes (which case it redoes the metadata regen) or the PM now supports that EAPI, and... you guessed it, redoes the metadata regen. The cache behaviour is basically the same regardless of the EAPI mechanism. Now, this isn't to say everyone views the function as *optimal*. People can argue about that as much as they'd like. The point however is that it *would* work. Anyone claiming fragility/otherwise needs to put forth actual code examples. Anyway, lets focus on our main goal, which is to decide on a way to obtain the EAPI _without_ sourcing the ebuild. Nitpicking, but the point needs be made; this is *your* requirement. The requirement is to be able to deploy new globals/bash requirements/whatever. There is a difference. ~brian
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Sun, Mar 11, 2012 at 10:03 PM, Brian Harring ferri...@gmail.com wrote: Pragmatic reality, the eapi function actually would work fine. As pointed out elsewhere, bash parses as it goes- which isn't going to change. Unless the ebuild isn't written in bash... How do you source the ebuild if you don't know what to use to source it? How do you know what to use to source it if you don't know the EAPI? Right now all the existing EAPIs use bash, but there is no reason the file couldn't be xml, or python, or just about anything else. If we want to allow for that kind of flexibility, then it might make sense to go ahead and state that our convention is to stick EAPI=5 in one of the first few lines of the ebuild, or inside a comment, but also go a step further and state that the text EAPI= cannot appear elsewhere in the ebuild (or perhaps within the first 10 lines). Just about any file format we might use would allow us to make EAPI= appear in it, but not all could guarantee that it would occur at the start of a line, or at the start of a line immediately after a #. In any case, I can really see the KISS value in a very rigid syntax that is trivial to parse. Stuff like this almost makes me wish our ebuilds already were xml files or such, with bash embedded inside sections. Finding a particular tag in an xml file is trivial as the fundamentals haven't changed in 15 years. Rich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Sun, Mar 11, 2012 at 7:20 PM, Rich Freeman ri...@gentoo.org wrote: On Sun, Mar 11, 2012 at 10:03 PM, Brian Harring ferri...@gmail.com wrote: Pragmatic reality, the eapi function actually would work fine. As pointed out elsewhere, bash parses as it goes- which isn't going to change. Unless the ebuild isn't written in bash... I'd opt for a different extension in that case actually. How do you source the ebuild if you don't know what to use to source it? How do you know what to use to source it if you don't know the EAPI? Right now all the existing EAPIs use bash, but there is no reason the file couldn't be xml, or python, or just about anything else. If we want to allow for that kind of flexibility, then it might make sense to go ahead and state that our convention is to stick EAPI=5 in one of the first few lines of the ebuild, or inside a comment, but also go a step further and state that the text EAPI= cannot appear elsewhere in the ebuild (or perhaps within the first 10 lines). Just about any file format we might use would allow us to make EAPI= appear in it, but not all could guarantee that it would occur at the start of a line, or at the start of a line immediately after a #. In any case, I can really see the KISS value in a very rigid syntax that is trivial to parse. Stuff like this almost makes me wish our ebuilds already were xml files or such, with bash embedded inside sections. Finding a particular tag in an xml file is trivial as the fundamentals haven't changed in 15 years. I will stab the next person who suggests 'xml-like ebuilds.' -A Rich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/11/2012 07:03 PM, Brian Harring wrote: Pragmatic reality, the eapi function actually would work fine. As pointed out elsewhere, bash parses as it goes- which isn't going to change. If someone invokes 'eapi happy-meal' and it's not supported, the sourcing is stopped immediately, cache gets -happy-meal for the EAPI, and the pm continues to ignore the ebuild till either the ebuilds mtime changes (which case it redoes the metadata regen) or the PM now supports that EAPI, and... you guessed it, redoes the metadata regen. The cache behaviour is basically the same regardless of the EAPI mechanism. Now, this isn't to say everyone views the function as *optimal*. People can argue about that as much as they'd like. The point however is that it *would* work. Anyone claiming fragility/otherwise needs to put forth actual code examples. Suppose that EAPI 5 requires bash-5. There may be bash-5 syntax in the ebuild prior to your eapi() function call, causing a fatal syntax error. Anyway, lets focus on our main goal, which is to decide on a way to obtain the EAPI _without_ sourcing the ebuild. Nitpicking, but the point needs be made; this is *your* requirement. The requirement is to be able to deploy new globals/bash requirements/whatever. Well, I think most people would tend to accept my requirement, and that it falls within the realm of common-sense if avoiding unnecessary complexity is one of our goals. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/11/2012 06:55 PM, Brian Harring wrote: On Sat, Mar 10, 2012 at 08:06:50AM -0800, Zac Medico wrote: Yeah. Another way of putting it is that the requirement to spawn a bash process and source the ebuild adds a ridiculous amount of unnecessary complexity, in violation of the KISS principle [1]. This statement is incorrect. Even if EAPI could be parsed via some non sourcing approach, we *still* have to source the ebuild to get the metadata for when the EAPI is supported (the vast majority of usage). That complexity is there one way or another- we wouldn't be trying to extract the EAPI from the ebuild unless the cache was invalid/missing. There are a couple of other cases worth considering: 1) User downloads an overlay that doesn't provide cache. We want the package manager to give a pretty EAPI unsupported message, rather than spit out some bash syntax errors. 2) You're assuming that a package manager can validate cache for an EAPI that it doesn't necessarily support. That's a somewhat fragile assumption, given the complexities of cache validation, which involve verification all files that affect metadata and those files may vary depending on the EAPI. Phrasing it more bluntly: you can only avoid the sourcing step if you can isolate that the EAPI is unsupported (which is extremely rare in terms of actual user experience). For the rest of the time (well past the 99% mark) sourcing is the immediate step following. For the sake of being robust in all possible cases, it's just a lot simpler if we can obtain the EAPI simply and reliably, without spawning bash to source the ebuild. Also, stop referencing wikipedia. People know what trivial objection and KISS is. You can't assume that. On this list we've got potential to have readers and responders with lots of different backgrounds. Your insistence on using bash to obtain the EAPI would make me wonder if you understood the KISS principle, if I didn't know you better. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Sat, 10 Mar 2012 14:06:49 +1300
Kent Fredric kentfred...@gmail.com wrote:
HOMEPAGE Package's homepage. If you are unable to locate an
official one, try to provide a link to freshmeat.net or a similar
package tracking site. Never refer to a variable name in the string;
include only raw text.
I'm not sure if that is a good example of things that are weird like
EAPI
You can set HOMEPAGE in an eclass for instance, using anything you can
do with bash and build it out of substrings.
perl-module.eclass does just that.
[[ -z ${HOMEPAGE} ]] \
HOMEPAGE=http://search.cpan.org/dist/${MY_PN:-${PN}}/;
Not really comparable in entirety to EAPI.
The point is that there's a silly QA policy saying not to do that. It's
not a PMS rule.
--
Ciaran McCreesh
signature.asc
Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 11:20 AM, Ciaran McCreesh wrote: On Fri, 09 Mar 2012 11:49:44 -0500 Michael Orlitzky mich...@orlitzky.com wrote: isnt the whole point of the proposal to get eapi without sourcing ? so that we can use new bash features at local or global scope without risking that people with an old bash get syntax errors trying to get the eapi Right. Michael has lost sight of the goal and is moving off on a tangent. The point was to be able to get the EAPI without crashing if the ebuild uses newer features. No, it's not. There's more to it than that. Some EAPIs really require defining certain environment variables, shell options, sandbox things etc *before* the sourcing starts. It's a massive pain in the ass to try to handle setting that kind of thing on the fly once the sourcing has already started. Knowing the EAPI before having to spawn a bash process isn't just about performance, it's also about making ebuilds much less difficult to deal with. Yeah. Another way of putting it is that the requirement to spawn a bash process and source the ebuild adds a ridiculous amount of unnecessary complexity, in violation of the KISS principle [1]. [1] http://en.wikipedia.org/wiki/KISS_principle -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
* Zac Medico schrieb am 08.03.12 um 17:30 Uhr: On 03/08/2012 01:42 AM, Marc Schiffbauer wrote: * Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr: Such constructs also cannot be used with any of the other proposed solutions. And in fact, nobody is using such things in practice. _All_ ebuilds in the Portage tree can be successfully parsed with the regexp proposed. Ebuilds are bash scripts. I think introducing exceptions or constraints here is not straightforward. Given that ebuilds already have to conform to a vast number of constraints that ordinary bash scripts do not. I think that it's perfectly reasonable for ebuilds to have a constrained syntax for EAPI assignments. There are constraints in ebuilds, right. But its an *ebuild* in the end, not an ordinary shell script. Thats true. So if EAPI is very special, and I am now convinced it is, then well, this might be the most important contraint in an ebuild at all. If that is true I would vote to keep this as simple as possible: * EAPI *must* *be* the first non-Argument / shell code in an ebuild * The value of EAPI in the assignment *MUST* *NOT* contain any other variables or other shell substitutions. Period. Done. * That would be the least invasive change. * Could easily be checked by repoman * Is easy parsable by other programs (python code) -Marc -- 8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134 pgpFVfsBDxzOc.pgp Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/12 00:51, Zac Medico wrote: On 03/08/2012 09:35 PM, Michael Orlitzky wrote: The function can do any crazy thing you want. We don't need a function. We need to know the EAPI before we source the ebuild, and a function doesn't give us that. Surely we can source one or two lines of the ebuild safely, like the example shows?
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 06:42 AM, Michael Orlitzky wrote: On 03/09/12 00:51, Zac Medico wrote: On 03/08/2012 09:35 PM, Michael Orlitzky wrote: The function can do any crazy thing you want. We don't need a function. We need to know the EAPI before we source the ebuild, and a function doesn't give us that. Surely we can source one or two lines of the ebuild safely, like the example shows? Why would we though, when sourcing is a relatively costly operation, and there are much more efficient ways to get the EAPI? Also, you've got a chicken and egg problem, since you're assuming that your eapi() function is going to be available in the global environment when the ebuild is sourced. The whole point of knowing the EAPI in advance is so that we will be able to make changes to the global environment, such as add, remove, or change the behavior of _any_ function. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/12 10:05, Zac Medico wrote: Surely we can source one or two lines of the ebuild safely, like the example shows? Why would we though, when sourcing is a relatively costly operation, and there are much more efficient ways to get the EAPI? There do not seem to be any that people agree on =) I mean, it's /easy/ to get the EAPI. There's a list of ways on the GLEP, but all of them have trade-offs. The header comment seems to be the preference in this thread. The advantage that the eapi function has over a comment is that it's not magic -- it's just normal bash syntax. So we've addressed that issue at a small performance cost (we're really only sourcing the ebuild up to 'exit'). Whether or not that trade-off is worth it, who knows. I would prefer to make it work before we make it fast; but new ideas can't hurt considering nobody has agreed on an old one. Also, you've got a chicken and egg problem, since you're assuming that your eapi() function is going to be available in the global environment when the ebuild is sourced. The whole point of knowing the EAPI in advance is so that we will be able to make changes to the global environment, such as add, remove, or change the behavior of _any_ function. Nah, at the top of the ebuild, you would have, EAPI=5 eapi 5 until package managers catch up, just like we would have to do with the comment. Once they've caught up, we can drop the EAPI= assignment, and begin using new features after the call to 'eapi'.
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 07:21 AM, Michael Orlitzky wrote: The advantage that the eapi function has over a comment is that it's not magic -- it's just normal bash syntax. So we've addressed that issue at a small performance cost (we're really only sourcing the ebuild up to 'exit'). Also consider the case where a user syncs after not having updated for a couple of months, and the tree contains some ebuilds with EAPIs that are not supported by the currently installed package manager. In this case, when resolving dependencies and filtering ebuilds based on whether or not their EAPI is supported, spawning bash once per ebuild is much more costly than the alternatives. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, 09 Mar 2012 07:41:09 -0800 Zac Medico zmed...@gentoo.org wrote: On 03/09/2012 07:21 AM, Michael Orlitzky wrote: The advantage that the eapi function has over a comment is that it's not magic -- it's just normal bash syntax. So we've addressed that issue at a small performance cost (we're really only sourcing the ebuild up to 'exit'). Also consider the case where a user syncs after not having updated for a couple of months, and the tree contains some ebuilds with EAPIs that are not supported by the currently installed package manager. In this case, when resolving dependencies and filtering ebuilds based on whether or not their EAPI is supported, spawning bash once per ebuild is much more costly than the alternatives. isnt the whole point of the proposal to get eapi without sourcing ? so that we can use new bash features at local or global scope without risking that people with an old bash get syntax errors trying to get the eapi
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 09/03/12 10:41 AM, Zac Medico wrote: On 03/09/2012 07:21 AM, Michael Orlitzky wrote: The advantage that the eapi function has over a comment is that it's not magic -- it's just normal bash syntax. So we've addressed that issue at a small performance cost (we're really only sourcing the ebuild up to 'exit'). Also consider the case where a user syncs after not having updated for a couple of months, and the tree contains some ebuilds with EAPIs that are not supported by the currently installed package manager. IIRC we get this already, when the EAPI isn't supported by the version of portage installed -- upgrading really old systems won't allow an emerge of python-2.7 due to a too-new EAPI, and python-2.7 is needed to upgrade to the newer portage. I don't see how the EAPI check itself failing and thereby excluding an ebuild is much different than the specified EAPI excluding it..? Either way, the end user is going to have issues if they don't keep their portage up to date. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) iF4EAREIAAYFAk9aJ0QACgkQAJxUfCtlWe2tTAEA7iUgDOCaGoQhz1dXukQ/a3lY rsdqewd2DYZWtsv+3XoA/iRVe+qf4HXdkWTchFRHlolaTJechz6AZCzKY/sNdu4w =1e/8 -END PGP SIGNATURE-
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 07:51 AM, Alexis Ballier wrote: On Fri, 09 Mar 2012 07:41:09 -0800 Zac Medico zmed...@gentoo.org wrote: On 03/09/2012 07:21 AM, Michael Orlitzky wrote: The advantage that the eapi function has over a comment is that it's not magic -- it's just normal bash syntax. So we've addressed that issue at a small performance cost (we're really only sourcing the ebuild up to 'exit'). Also consider the case where a user syncs after not having updated for a couple of months, and the tree contains some ebuilds with EAPIs that are not supported by the currently installed package manager. In this case, when resolving dependencies and filtering ebuilds based on whether or not their EAPI is supported, spawning bash once per ebuild is much more costly than the alternatives. isnt the whole point of the proposal to get eapi without sourcing ? so that we can use new bash features at local or global scope without risking that people with an old bash get syntax errors trying to get the eapi Right. Michael has lost sight of the goal and is moving off on a tangent. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 07:52 AM, Ian Stakenvicius wrote: On 09/03/12 10:41 AM, Zac Medico wrote: On 03/09/2012 07:21 AM, Michael Orlitzky wrote: The advantage that the eapi function has over a comment is that it's not magic -- it's just normal bash syntax. So we've addressed that issue at a small performance cost (we're really only sourcing the ebuild up to 'exit'). Also consider the case where a user syncs after not having updated for a couple of months, and the tree contains some ebuilds with EAPIs that are not supported by the currently installed package manager. IIRC we get this already, when the EAPI isn't supported by the version of portage installed -- upgrading really old systems won't allow an emerge of python-2.7 due to a too-new EAPI, and python-2.7 is needed to upgrade to the newer portage. You're talking about a very specific (unfortunate) upgrade path breakage. We should keep the discussion focused on more generic cases that apply to future upgrade scenarios, rather than focusing on past breakages (though it's obvious that we should also try to avoid such upgrade path breakages in the future). Assuming that there is still a viable upgrade path, the package manager needs resolve dependencies and filter ebuilds based on whether or not their EAPI is supported. There are basically two ways to do the EAPI filtering: 1) Access the EAPI from the metadata cache. This assumes that the package manager is able to validate metadata cache entries for EAPIs that it does not support, which is a somewhat fragile assumption given the complexities of cache validation, which involves verification of ebuilds and additional sources of metadata which may include eclasses and possibly other files in the future. 2) Parse the ebuild from the ebuild using a pre-defined protocol (parse EAPI assignment, parse comment, or parse file name as in GLEP 55...). I don't see how the EAPI check itself failing and thereby excluding an ebuild is much different than the specified EAPI excluding it..? Either way, the end user is going to have issues if they don't keep their portage up to date. They may or may not have issues. Our goal is to minimize our vulnerability to these kinds of issues as much as possible. Being able to obtain the ebuild EAPI without the expense of sourcing it is one small step toward this goal. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, 09 Mar 2012 00:35:14 -0500 Michael Orlitzky mich...@orlitzky.com wrote: On 03/09/2012 12:04 AM, Michał Górny wrote: This is of course isomorphic to requiring a specific EAPI=4 format, but does allow you to do stupid things like x=`seq 4 4`; eapi $x; if you want. What advantage does it give us? We still can't change ebuild syntax in global scope because bash will barf. Not in EAPI=5, no, but once all PMs are using the eapi function we could. The function can do any crazy thing you want. Right now, we need to source the entire ebuild to get at its environment. Before we source it for real, we just want to know the value of $EAPI. Since eapi() will be the first function called, it can be our interface to this variable. What if bash starts to parse the script completely and barfs at 'syntax error' before it starts executing stuff? -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, Mar 09, 2012 at 08:15:11AM -0800, Zac Medico wrote: They may or may not have issues. Our goal is to minimize our vulnerability to these kinds of issues as much as possible. Being able to obtain the ebuild EAPI without the expense of sourcing it is one small step toward this goal. EAPI is metadata and is best treated as such. In other words, history aside, it does not belong inside an ebuild. Making EAPI info part of the filename does look like a reasonable solution - similar to seen/replied flags in the filenames in maildir directories. Heck, even version numbers in an ebuild filename is similar. I don't understand why there is a strong objection to it. But anyway, it is Friday night and I am out of here. Have fun. -- Eray Aslan e...@gentoo.org pgpgziRXSe880.pgp Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 08:33 AM, Eray Aslan wrote: On Fri, Mar 09, 2012 at 08:15:11AM -0800, Zac Medico wrote: They may or may not have issues. Our goal is to minimize our vulnerability to these kinds of issues as much as possible. Being able to obtain the ebuild EAPI without the expense of sourcing it is one small step toward this goal. EAPI is metadata and is best treated as such. In other words, history aside, it does not belong inside an ebuild. Making EAPI info part of the filename does look like a reasonable solution - similar to seen/replied flags in the filenames in maildir directories. Heck, even version numbers in an ebuild filename is similar. I don't understand why there is a strong objection to it. I have a mild preference for the parse EAPI assignment approach, simply because it's the least invasive. That said, ultimately it doesn't make much difference to me whether we parse it from inside the ebuild or from its file name. From my perspective, arguing between these approaches boils down to hair-splitting [1]. [1] http://en.wikipedia.org/wiki/Trivial_objections -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/12 10:58, Zac Medico wrote: On 03/09/2012 07:51 AM, Alexis Ballier wrote: On Fri, 09 Mar 2012 07:41:09 -0800 Zac Medico zmed...@gentoo.org wrote: On 03/09/2012 07:21 AM, Michael Orlitzky wrote: The advantage that the eapi function has over a comment is that it's not magic -- it's just normal bash syntax. So we've addressed that issue at a small performance cost (we're really only sourcing the ebuild up to 'exit'). Also consider the case where a user syncs after not having updated for a couple of months, and the tree contains some ebuilds with EAPIs that are not supported by the currently installed package manager. In this case, when resolving dependencies and filtering ebuilds based on whether or not their EAPI is supported, spawning bash once per ebuild is much more costly than the alternatives. isnt the whole point of the proposal to get eapi without sourcing ? so that we can use new bash features at local or global scope without risking that people with an old bash get syntax errors trying to get the eapi Right. Michael has lost sight of the goal and is moving off on a tangent. The point was to be able to get the EAPI without crashing if the ebuild uses newer features. If you can get the EAPI without sourcing, that obviously accomplishes the goal. But there are other goals, too, like not limiting the syntax of the EAPI assignment. I was just trying to think up something that addresses them all. In any case, yeah, it would crash and burn if someone synced his tree with an ancient version of portage. But so would the comment solution. If you want to fix that, we either have to rename everything (and hope we get it right this time) or reconsider GLEP 55.
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/12 11:29, Michał Górny wrote:
What if bash starts to parse the script completely and barfs at 'syntax
error' before it starts executing stuff?
It doesn't parse the script completely, it executes line-by-line, so we
can bail out early.
This returns 1:
exit 1
QWE*$)@#$%IT@$KTRGV{PO#J$%$M@#$_)TVI!%K@B)$YJMV$%())@K#@)
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 08:49 AM, Michael Orlitzky wrote: The point was to be able to get the EAPI without crashing if the ebuild uses newer features. If you can get the EAPI without sourcing, that obviously accomplishes the goal. But there are other goals, too, like not limiting the syntax of the EAPI assignment. I was just trying to think up something that addresses them all. Yeah, but you compromised on what can be considered our primary goal, which is to obtain the EAPI without sourcing the ebuild. In any case, yeah, it would crash and burn if someone synced his tree with an ancient version of portage. But so would the comment solution. Don't forget the possibility of bash syntax extensions triggering a syntax error [1]. The comment and bash assignment solutions are immune to that because bash is exceedingly unlikely to break backward compatibility in the comment and bash assignment syntax. [1] http://archives.gentoo.org/gentoo-dev/msg_2c1fbbb32f5bc47756303ca18055bee4.xml -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, 09 Mar 2012, Michael Orlitzky wrote: What if bash starts to parse the script completely and barfs at 'syntax error' before it starts executing stuff? It doesn't parse the script completely, it executes line-by-line, so we can bail out early. How can you tell that this behaviour won't be changed in a future bash version? Ulrich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/12 12:11, Ulrich Mueller wrote: On Fri, 09 Mar 2012, Michael Orlitzky wrote: What if bash starts to parse the script completely and barfs at 'syntax error' before it starts executing stuff? It doesn't parse the script completely, it executes line-by-line, so we can bail out early. How can you tell that this behaviour won't be changed in a future bash version? Who's to say that in the future my computer won't be made out of delicious ice cream, eliminating the need for EAPIs entirely? Chances are, this would break thousands of scripts, so we hope they wouldn't do it. If it does happen, we either deal with it then, or don't upgrade to that version of bash -- the same as we would do with any other massive breaking change. At any rate, I'm now convinced that we all want GLEP 55, but with a different name.
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 09:31 AM, Michael Orlitzky wrote: On 03/09/12 12:11, Ulrich Mueller wrote: On Fri, 09 Mar 2012, Michael Orlitzky wrote: What if bash starts to parse the script completely and barfs at 'syntax error' before it starts executing stuff? It doesn't parse the script completely, it executes line-by-line, so we can bail out early. How can you tell that this behaviour won't be changed in a future bash version? Who's to say that in the future my computer won't be made out of delicious ice cream, eliminating the need for EAPIs entirely? Chances are, this would break thousands of scripts, so we hope they wouldn't do it. If it does happen, we either deal with it then, or don't upgrade to that version of bash -- the same as we would do with any other massive breaking change. Ulrich is talking about extensions which require a newer version of bash. These kinds of extensions are quite common and don't cause massive breaking because people simply have to upgrade bash in order to use the new extensions, and their old scripts continue to run because the new extensions don't interfere with backward compatibility. Your eapi() function proposal is especially fragile in this context because it assumes that the installed version of bash will be able to execute a script that may target a newer version of bash. This is a special case that is typically not a problem, although it is a major problem under the specific conditions that your eapi() function approach induces. Anyway, lets focus on our main goal, which is to decide on a way to obtain the EAPI _without_ sourcing the ebuild. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, 09 Mar 2012 12:31:24 -0500 Michael Orlitzky mich...@orlitzky.com wrote: On 03/09/12 12:11, Ulrich Mueller wrote: On Fri, 09 Mar 2012, Michael Orlitzky wrote: What if bash starts to parse the script completely and barfs at 'syntax error' before it starts executing stuff? It doesn't parse the script completely, it executes line-by-line, so we can bail out early. How can you tell that this behaviour won't be changed in a future bash version? Who's to say that in the future my computer won't be made out of delicious ice cream, eliminating the need for EAPIs entirely? Chances are, this would break thousands of scripts, so we hope they wouldn't do it. If it does happen, we either deal with it then, or don't upgrade to that version of bash -- the same as we would do with any other massive breaking change. Thousands of scripts? So... you're saying that people actually use thousands of scripts which have invalid syntax... Well, one thing I can think of now is makeself and similar. Those are indeed a quite good argument. But the main point here is that at some point someone may want to use a non-bash syntax for ebuilds. Or some kind of optional bash extension. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 9 March 2012 17:31, Michael Orlitzky mich...@orlitzky.com wrote: At any rate, I'm now convinced that we all want GLEP 55, but with a different name. I think that moving the data to the filename is probably a better approach than semi- or repeat parsing, but I prefer preserving the .ebuild extension, and think that eapi should be specified similarly to ebuild revision, as a suffix. for instance: app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the new schema) app-foo/bar-1.0.0-r1-e1.ebuild app-foo/bar-1.0.0-r1-e99.ebuild
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/12 12:47, Zac Medico wrote: Ulrich is talking about extensions which require a newer version of bash. These kinds of extensions are quite common and don't cause massive breaking because people simply have to upgrade bash in order to use the new extensions, and their old scripts continue to run because the new extensions don't interfere with backward compatibility. Your eapi() function proposal is especially fragile in this context because it assumes that the installed version of bash will be able to execute a script that may target a newer version of bash. This is a special case that is typically not a problem, although it is a major problem under the specific conditions that your eapi() function approach induces. Well, you wouldn't need to execute the script targeting a newer version of bash. You would need to source it, which involves maybe setting EAPI=5, and then calling the 'eapi' function which will immediately exit. The new features aren't a problem because you never get to them. (If you try to use some new bash extension to compute your EAPI value, well, you shouldn't have done that.) Anyway, lets focus on our main goal, which is to decide on a way to obtain the EAPI _without_ sourcing the ebuild. Agreed. I'm forced to agree with Ciaran: can we just ignore the previous council's decision, and reconsider the merits of GLEP 55?
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, 9 Mar 2012 18:02:51 + James Broadhead jamesbroadh...@gmail.com wrote: On 9 March 2012 17:31, Michael Orlitzky mich...@orlitzky.com wrote: At any rate, I'm now convinced that we all want GLEP 55, but with a different name. I think that moving the data to the filename is probably a better approach than semi- or repeat parsing, but I prefer preserving the .ebuild extension, and think that eapi should be specified similarly to ebuild revision, as a suffix. for instance: app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the new schema) app-foo/bar-1.0.0-r1-e1.ebuild app-foo/bar-1.0.0-r1-e99.ebuild if you want to keep .ebuild you need to keep current naming, afaik package managers fail on invalid names
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 10:24 AM, Alexis Ballier wrote: On Fri, 9 Mar 2012 18:02:51 + James Broadhead jamesbroadh...@gmail.com wrote: On 9 March 2012 17:31, Michael Orlitzky mich...@orlitzky.com wrote: At any rate, I'm now convinced that we all want GLEP 55, but with a different name. I think that moving the data to the filename is probably a better approach than semi- or repeat parsing, but I prefer preserving the .ebuild extension, and think that eapi should be specified similarly to ebuild revision, as a suffix. for instance: app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the new schema) app-foo/bar-1.0.0-r1-e1.ebuild app-foo/bar-1.0.0-r1-e99.ebuild if you want to keep .ebuild you need to keep current naming, afaik package managers fail on invalid names Invalid names like those should only trigger warnings. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/12 13:02, James Broadhead wrote: On 9 March 2012 17:31, Michael Orlitzky mich...@orlitzky.com wrote: At any rate, I'm now convinced that we all want GLEP 55, but with a different name. I think that moving the data to the filename is probably a better approach than semi- or repeat parsing, but I prefer preserving the .ebuild extension, and think that eapi should be specified similarly to ebuild revision, as a suffix. for instance: app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the new schema) app-foo/bar-1.0.0-r1-e1.ebuild app-foo/bar-1.0.0-r1-e99.ebuild One of the benefits of GLEP 55 naming is that old package managers won't try to parse them. So, for example, if we put new features in, app-foo/bar-1.0.0.ebuild-5 portage from 2003 won't try to source it.
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/2012 10:33 AM, Michael Orlitzky wrote: On 03/09/12 13:02, James Broadhead wrote: On 9 March 2012 17:31, Michael Orlitzky mich...@orlitzky.com wrote: At any rate, I'm now convinced that we all want GLEP 55, but with a different name. I think that moving the data to the filename is probably a better approach than semi- or repeat parsing, but I prefer preserving the .ebuild extension, and think that eapi should be specified similarly to ebuild revision, as a suffix. for instance: app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the new schema) app-foo/bar-1.0.0-r1-e1.ebuild app-foo/bar-1.0.0-r1-e99.ebuild One of the benefits of GLEP 55 naming is that old package managers won't try to parse them. So, for example, if we put new features in, app-foo/bar-1.0.0.ebuild-5 portage from 2003 won't try to source it. Every software product has an end of life. I think if a system hasn't been updated in the last 2 years or so, then it's fair to assume that it will never be updated. So, all relevant versions of portage should simply show a warning message if the encounter an ebuild name such as app-foo/bar-1.0.0-r1-e99.ebuild. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, Mar 9, 2012 at 12:47 PM, Zac Medico zmed...@gentoo.org wrote: Anyway, lets focus on our main goal, which is to decide on a way to obtain the EAPI _without_ sourcing the ebuild. Agreed. Plus, an approach that either uses the filename or something like a comment line is also going to be much more flexible if bash syntax changes substantially, or if we want to support ebuilds-in-python or some other approach. That is the main merit I see to the shebang approach - you could define some kind of API that actually involves executing the ebuild. Sticking the EAPI in the filename is a little less complicated and it gives you the same flexibility - I'm not aware of any file format that is fussy about the content of the filename. Even approaches like putting EAPI=5 in the file (even inside a comment) might break if the file is meant to be generally interpreted by some program that has rigid syntax rules.Most scripting languages could probably handle this somehow, but if you ever wanted ELF ebuilds that wouldn't be so likely to fly unless you could embed \nEAPI=ELF8\n in something near the start of the file. Sure, I doubt we'll ever want ELF ebuilds (I'd consider non-programatic ebuilds for more trivial packages more likely), but the point is that we should prefer options that offer more flexibility down the road over ones that solve the immediate need but just leave us with the same debate two years from now. Rich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, 09 Mar 2012 11:49:44 -0500 Michael Orlitzky mich...@orlitzky.com wrote: isnt the whole point of the proposal to get eapi without sourcing ? so that we can use new bash features at local or global scope without risking that people with an old bash get syntax errors trying to get the eapi Right. Michael has lost sight of the goal and is moving off on a tangent. The point was to be able to get the EAPI without crashing if the ebuild uses newer features. No, it's not. There's more to it than that. Some EAPIs really require defining certain environment variables, shell options, sandbox things etc *before* the sourcing starts. It's a massive pain in the ass to try to handle setting that kind of thing on the fly once the sourcing has already started. Knowing the EAPI before having to spawn a bash process isn't just about performance, it's also about making ebuilds much less difficult to deal with. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Fri, 09 Mar 2012 10:56:03 -0800 Zac Medico zmed...@gentoo.org wrote: Every software product has an end of life. I think if a system hasn't been updated in the last 2 years or so, then it's fair to assume that it will never be updated. So, all relevant versions of portage should simply show a warning message if the encounter an ebuild name such as app-foo/bar-1.0.0-r1-e99.ebuild. It's worth noting that showing a warning message will result in zillions of freaking out users, and that we want to be able to introduce new EAPIs without having a stable Portage version that handles it around. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/09/12 13:56, Zac Medico wrote: On 03/09/2012 10:33 AM, Michael Orlitzky wrote: On 03/09/12 13:02, James Broadhead wrote: On 9 March 2012 17:31, Michael Orlitzky mich...@orlitzky.com wrote: At any rate, I'm now convinced that we all want GLEP 55, but with a different name. I think that moving the data to the filename is probably a better approach than semi- or repeat parsing, but I prefer preserving the .ebuild extension, and think that eapi should be specified similarly to ebuild revision, as a suffix. for instance: app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the new schema) app-foo/bar-1.0.0-r1-e1.ebuild app-foo/bar-1.0.0-r1-e99.ebuild One of the benefits of GLEP 55 naming is that old package managers won't try to parse them. So, for example, if we put new features in, app-foo/bar-1.0.0.ebuild-5 portage from 2003 won't try to source it. Every software product has an end of life. I think if a system hasn't been updated in the last 2 years or so, then it's fair to assume that it will never be updated. So, all relevant versions of portage should simply show a warning message if the encounter an ebuild name such as app-foo/bar-1.0.0-r1-e99.ebuild. I was just repeating your point against the eapi function =) And there is a non-zero benefit to it, I've had to rescue systems that haven't seen an update in years. The best reason I can think of for using ebuild-EAPI is simply semantic. The basename of the ebuild refers to the package, the extension decides what interprets it. That is at least consistent with every other file extension.
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 9 March 2012 06:30, Jeroen Roovers j...@gentoo.org wrote:
On Thu, 8 Mar 2012 17:14:58 +
Ciaran McCreesh ciaran.mccre...@googlemail.com wrote:
Having a different, special rule for something that looks exactly like
lots of other things that do not have that different, special rule is
hardly hair splitting. This rule would have to be documented and have
special code to carefully enforce it. That's a big deal.
Exactly like HOMEPAGE is a big deal?
HOMEPAGE Package's homepage. If you are unable to locate an
official one, try to provide a link to freshmeat.net or a similar
package tracking site. Never refer to a variable name in the string;
include only raw text.
I'm not sure if that is a good example of things that are weird like EAPI
You can set HOMEPAGE in an eclass for instance, using anything you can
do with bash and build it out of substrings.
perl-module.eclass does just that.
[[ -z ${HOMEPAGE} ]] \
HOMEPAGE=http://search.cpan.org/dist/${MY_PN:-${PN}}/;
Not really comparable in entirety to EAPI.
--
Kent
perl -e print substr( \edrgmaM SPA NOcomil.ic\\@tfrken\, \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Wed, Mar 7, 2012 at 11:27 PM, Ulrich Mueller u...@gentoo.org wrote:
On Wed, 7 Mar 2012, Alec Warner wrote:
*** Proposal 1: Parse the EAPI assignment statement ***
[...]
I don't like this idea because the sane way should be easy and
straightforward. Mixing a constant declaration with bash assignment
just confuses users who think the assignment is full bash when in
fact it is not.
EAPI=$(somefunc)
EAPI=${SOMEVAR%%-*}
and so forth all don't meet the regex (and would be flagged
invalid.) However a naive author might think they work.
Such constructs also cannot be used with any of the other proposed
solutions. And in fact, nobody is using such things in practice.
_All_ ebuilds in the Portage tree can be successfully parsed with the
regexp proposed.
I'm not saying they are valid EAPI syntax; but they are all valid
bash. I tend to assume all authors are as...ignorant as myself. Lets
not give them the rope to hang themselves.
The obvious sanity check, i.e. comparing the EAPI obtained by parsing
and by sourcing, could be added to repoman, which would prevent such
non-conforming ebuilds from being committed to the tree.
*** Proposal 2: EAPI in header comment ***
[...]
Overloading is bad.
There is no real difference between:
#!/usr/bin/ebuild --eapi 5
# EAPI=5
The problem is that the former is also the way to specify how to
execute the ebuild; so unless you plan to make ebuilds executable and
having /usr/bin/ebuild provide the ebuild environment, using that
syntax is confusing to users.
I agree with this point.
Many file formats are identifying themselves with a magic token (as
it is used by sys-apps/file), but it is not necessarily a shebang.
Ulrich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
* Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr:
On Wed, 7 Mar 2012, Alec Warner wrote:
*** Proposal 1: Parse the EAPI assignment statement ***
[...]
I don't like this idea because the sane way should be easy and
straightforward. Mixing a constant declaration with bash assignment
just confuses users who think the assignment is full bash when in
fact it is not.
EAPI=$(somefunc)
EAPI=${SOMEVAR%%-*}
and so forth all don't meet the regex (and would be flagged
invalid.) However a naive author might think they work.
Such constructs also cannot be used with any of the other proposed
solutions. And in fact, nobody is using such things in practice.
_All_ ebuilds in the Portage tree can be successfully parsed with the
regexp proposed.
Ebuilds are bash scripts. I think introducing exceptions or
constraints here is not straightforward.
I think the only relevant part whether EAPI is set correctly or not
should be the outcome of $EAPI.
I would vote for a solution in a bash comment where repoman would
have to check for its existance and equality to $EAPI.
-Marc
--
8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134
pgpiOirLF4AWG.pgp
Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Wed, 07 Mar 2012 17:14:13 -0500 Michael Orlitzky mich...@orlitzky.com wrote: On 03/07/2012 03:41 PM, Ulrich Mueller wrote: *** Proposal 2: EAPI in header comment *** A different approach would be to specify the EAPI in a specially formatted comment in the ebuild's header. No syntax has been suggested yet, but I believe that the following would work as a specification: - The EAPI must be declared in a special comment in the first line of the ebuild's header, as follows: - The first line of the ebuild must contain the word ebuild, followed by whitespace, followed by the EAPI, followed by end-of-line or whitespace. Someone suggested using a standard shebang the last time this came up, and if I remember correctly it was one of the least-disagreeable solutions proposed. We could of course define our own custom format, but I think something like, #!/usr/bin/eapi5 would be perfect if we could hand off the interpretation of the ebuild to that program. That solves the problem with new bash features, too, since you could point that command at a specific version. And what would /usr/bin/eapi5 do? Are you suggesting misusing shebang or making ebuilds PM-centric? -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Wed, 7 Mar 2012 20:12:25 -0800
Alec Warner anta...@gentoo.org wrote:
On Wed, Mar 7, 2012 at 12:41 PM, Ulrich Mueller u...@gentoo.org
wrote:
Hi all,
The way how we currently specify the EAPI in ebuilds has some
problems. For example, there is no sane way to allow usage of
features of a new bash version in a new EAPI. So we are currently
stuck with bash 3.2. Also changes of global scope behaviour, like
addition of new global scope functions (similar to inherit) are
not possible.
These flaws are outlined in GLEP 55 [1]:
| In order to get the EAPI the package manager needs to source the
| ebuild, which itself needs the EAPI in the first place. Otherwise
it | imposes a serious limitation, namely every ebuild, using any
of the | future EAPIs, will have to be source'able by old package
managers | [...]
The council has voted down GLEP 55 more than a year ago, but at the
same time requested that a solution for the mentioned issues should
be found. [2] However, there was no progress since then.
The issue arose again in bug 402167 [3] where several solutions have
been discussed. Below, I try to summarise the possible options
resulting from that discussion.
*** Proposal 1: Parse the EAPI assignment statement ***
This first proposal would require that the syntax of the EAPI
assignment statement in ebuilds matches a well defined regular
expression. A scan of the Portage tree shows that the statement only
occurs in the following variations (using EAPI 4 as example):
EAPI=4
EAPI=4
EAPI='4'
Sometimes this is followed by whitespace or a comment (starting with
a # sign). Also, with very few exceptions the EAPI assignment occurs
within the first few lines of the ebuild. For the vast majority of
ebuilds it is in line 5.
Written in a more formal way, appropriate for a specification:
- Ebuilds must contain at most one EAPI assignment statement.
- It must occur within the first N lines of the ebuild (N=10 and
N=30 have been suggested).
- The statement must match the following regular expression
(extended regexp syntax):
^[ \t]*EAPI=([']?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$
Note: The first and the third point are already fulfilled by all
ebuilds in the Portage tree. The second point will require very few
ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).
The package manager would determine the EAPI by parsing the
assignment with above regular expression. A sanity check would be
added. Citing Zac Medico in [3]: The fact that we can compare the
probed EAPI to the actual EAPI variable after the ebuild is sourced
seems like a perfect sanity check. We could easily detect
inconsistencies and flag such ebuilds as invalid, providing a
reliable feedback mechanism to ebuild developers.
This proposal comes in two variants:
1a) The change is applied retroactively for all EAPIs.
1b) It is only applied for EAPI 5 and later (which means that the
result of the EAPI parsing would be discarded for earlier EAPIs).
I don't like this idea because the sane way should be easy and
straightforward. Mixing a constant declaration with bash assignment
just confuses users who think the assignment is full bash when in
fact it is not.
EAPI=$(somefunc)
EAPI=${SOMEVAR%%-*}
and so forth all don't meet the regex (and would be flagged invalid.)
However a naive author might think they work.
And they all should be invalid due to our policies. The most important
ebuild variables like EAPI should be readable on sight, without having
to lookup random variables, functions etc.
--
Best regards,
Michał Górny
signature.asc
Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/08/2012 12:13 AM, Alec Warner wrote: On Wed, Mar 7, 2012 at 11:27 PM, Ulrich Mueller u...@gentoo.org wrote: Such constructs also cannot be used with any of the other proposed solutions. And in fact, nobody is using such things in practice. _All_ ebuilds in the Portage tree can be successfully parsed with the regexp proposed. I'm not saying they are valid EAPI syntax; but they are all valid bash. I tend to assume all authors are as...ignorant as myself. Lets not give them the rope to hang themselves. Something like DEPEND=foo bar is also valid bash, and yet we don't allow that either because foo bar does not contain valid dependency atoms. Also, keep in mind that we're not allowing people to hang themselves with invalid EAPI assignments. We'll simply give them a reasonable error message so that they can quickly and easily correct their mistake. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/08/2012 07:03 AM, Michał Górny wrote: Someone suggested using a standard shebang the last time this came up, and if I remember correctly it was one of the least-disagreeable solutions proposed. We could of course define our own custom format, but I think something like, #!/usr/bin/eapi5 would be perfect if we could hand off the interpretation of the ebuild to that program. That solves the problem with new bash features, too, since you could point that command at a specific version. And what would /usr/bin/eapi5 do? Are you suggesting misusing shebang or making ebuilds PM-centric? I was saying that I'd prefer a more-standard use of the shebang (if possible), rather than defining our own header comment syntax. Either way I think the second option is cleaner than regular expressions. Right now, we're guaranteed the features of bash-3.2. I guess we actually use whatever is executing ebuild.sh to source them. But we need to interpret the ebuild file with something: we might as well put *that* in the shebang, since that's what it's for. So if we were to do this with an ebuild right now, we'd add, #!/usr/bin/eapi4 to the header, and instead of sourcing the ebuild with whatever ebuild.sh is using, we would run it with 'eapi4' and pass whatever we need back and forth. Or maybe ebuild.sh would reload itself using 'eapi4'. If any of that makes sense, the PMS would just need to specify some requirements on the shebang command.
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/07/2012 03:41 PM, Ulrich Mueller wrote: *** Proposal 1: Parse the EAPI assignment statement *** There's also libbash now: http://www.gentoo.org/proj/en/libbash/index.xml Anyone know how close we are to being able to use it to parse the EAPI?
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Mar 8, 2012 3:29 PM, Zac Medico zmed...@gentoo.org wrote: Something like DEPEND=foo bar is also valid bash, and yet we don't allow that either because foo bar does not contain valid dependency atoms. There's a bit of a difference between caring about the value of a variable and caring about what syntax was used to assign it
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/08/2012 08:11 AM, David Leverton wrote: On Mar 8, 2012 3:29 PM, Zac Medicozmed...@gentoo.org wrote: Something like DEPEND=foo bar is also valid bash, and yet we don't allow that either because foo bar does not contain valid dependency atoms. There's a bit of a difference between caring about the value of a variable and caring about what syntax was used to assign it Maybe that sort of distinction truly makes a difference to some people, but to me it just seems like hair-splitting [1]. [1] http://en.wikipedia.org/wiki/Trivial_objections -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 08 Mar 2012 08:21:53 -0800 Zac Medico zmed...@gentoo.org wrote: Maybe that sort of distinction truly makes a difference to some people, but to me it just seems like hair-splitting [1]. So just to get this straight, you think that the following two restrictions are effectively equivalent? * The variable DEPEND's value must be set according to the following rules: * The EAPI variable assignment must not use full bash syntax. Instead, it must be assigned according to the following rules: And you believe that having exactly one place inside ebuild text where there are different whitespace, quoting and indenting rules for something that otherwise looks exactly like any other metadata variable isn't going to cause problems? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/08/2012 01:42 AM, Marc Schiffbauer wrote: * Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr: Such constructs also cannot be used with any of the other proposed solutions. And in fact, nobody is using such things in practice. _All_ ebuilds in the Portage tree can be successfully parsed with the regexp proposed. Ebuilds are bash scripts. I think introducing exceptions or constraints here is not straightforward. Given that ebuilds already have to conform to a vast number of constraints that ordinary bash scripts do not. I think that it's perfectly reasonable for ebuilds to have a constrained syntax for EAPI assignments. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 08 Mar 2012 08:30:57 -0800 Zac Medico zmed...@gentoo.org wrote: On 03/08/2012 01:42 AM, Marc Schiffbauer wrote: * Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr: Such constructs also cannot be used with any of the other proposed solutions. And in fact, nobody is using such things in practice. _All_ ebuilds in the Portage tree can be successfully parsed with the regexp proposed. Ebuilds are bash scripts. I think introducing exceptions or constraints here is not straightforward. Given that ebuilds already have to conform to a vast number of constraints that ordinary bash scripts do not. I think that it's perfectly reasonable for ebuilds to have a constrained syntax for EAPI assignments. ...and only EAPI assignments? Not for any other metadata variable? Doesn't that sort of suggest that EAPI shouldn't be a metadata variable? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Wed, Mar 7, 2012 at 3:41 PM, Ulrich Mueller u...@gentoo.org wrote: Again, the proposal comes in two variants: 2a) It is combined with a one time change of the file extension, like .ebuild - .eb. 2b) The usual EAPI assignment statement in the ebuild is still required, at least for a transition period. Just throwing my opinion in: I like proposal 2 better than proposal 1. Placing a regex-based constraint on a bash variable assignment doesn't feel right to me. I slightly prefer 2a over 2b because it feels cleaner. I'm sure it will break some tools, but I don't have a good feel for the scope of that breakage.
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/08/2012 08:29 AM, Ciaran McCreesh wrote: On Thu, 08 Mar 2012 08:21:53 -0800 Zac Medicozmed...@gentoo.org wrote: Maybe that sort of distinction truly makes a difference to some people, but to me it just seems like hair-splitting [1]. So just to get this straight, you think that the following two restrictions are effectively equivalent? * The variable DEPEND's value must be set according to the following rules: * The EAPI variable assignment must not use full bash syntax. Instead, it must be assigned according to the following rules: Yeah, I think they're reasonably/roughly equivalent for how we use them in the current context. And you believe that having exactly one place inside ebuild text where there are different whitespace, quoting and indenting rules for something that otherwise looks exactly like any other metadata variable isn't going to cause problems? Yes, with the stipulation that we implement a sanity check / feedback mechanism which guarantees that the probed EAPI is identical with the result obtained from bash [1]. [1] https://bugs.gentoo.org/show_bug.cgi?id=402167#c18 -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 08 Mar 2012, Michael Orlitzky wrote: On 03/07/2012 03:41 PM, Ulrich Mueller wrote: *** Proposal 1: Parse the EAPI assignment statement *** There's also libbash now: http://www.gentoo.org/proj/en/libbash/index.xml Looks like complete overkill to me, considering the simple task at hand. Ulrich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 2012-03-08 at 16:29 +, Ciaran McCreesh wrote: And you believe that having exactly one place inside ebuild text where there are different whitespace, quoting and indenting rules for something that otherwise looks exactly like any other metadata variable isn't going to cause problems? In light of the fact that all 29758 ebuilds in portage already satisfy the proposed whitespace, quoting, and indenting constrains on EAPI assignment, the probability of problems appears to be vanishingly small. And vanishingly small and can be reduced to zero by simply adding a check to repoman. Can you come up with a sane scenario under which an ebuild writer would want to use a different syntax for setting EAPI? -Alexandre
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, Mar 8, 2012 at 11:51 AM, Ulrich Mueller u...@gentoo.org wrote: On Thu, 08 Mar 2012, Michael Orlitzky wrote: There's also libbash now: Looks like complete overkill to me, considering the simple task at hand. Plus, wasn't the whole point that we can't guarantee that the bash installed on a user's system can parse the ebuild until we've checked the EAPI? If we use libbash doesn't that just keep the same constraint but on a different package? Is the current stable libbash guaranteed to be able to parse a bash v12 script? If you just parse the file with a defined set of rules without regard to how bash might parse the file, then you can determine the EAPI and then decide how to source it. For all we know EAPI G will turn ebuilds into python scripts so trying to read the thing with bash or libbash will be doomed to failure. Rich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 08 Mar 2012 11:59:33 -0500 Alexandre Rostovtsev tetrom...@gentoo.org wrote: On Thu, 2012-03-08 at 16:29 +, Ciaran McCreesh wrote: And you believe that having exactly one place inside ebuild text where there are different whitespace, quoting and indenting rules for something that otherwise looks exactly like any other metadata variable isn't going to cause problems? In light of the fact that all 29758 ebuilds in portage already satisfy the proposed whitespace, quoting, and indenting constrains on EAPI assignment, the probability of problems appears to be vanishingly small. And vanishingly small and can be reduced to zero by simply adding a check to repoman. Because they were recently changed, presumably... https://bugs.gentoo.org/show_bug.cgi?id=402167#c36 We had this discussion the last time around too, and people were told to assign in a particular way. As you can see, it didn't work. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/08/2012 08:35 AM, Ciaran McCreesh wrote: On Thu, 08 Mar 2012 08:30:57 -0800 Zac Medicozmed...@gentoo.org wrote: On 03/08/2012 01:42 AM, Marc Schiffbauer wrote: * Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr: Such constructs also cannot be used with any of the other proposed solutions. And in fact, nobody is using such things in practice. _All_ ebuilds in the Portage tree can be successfully parsed with the regexp proposed. Ebuilds are bash scripts. I think introducing exceptions or constraints here is not straightforward. Given that ebuilds already have to conform to a vast number of constraints that ordinary bash scripts do not. I think that it's perfectly reasonable for ebuilds to have a constrained syntax for EAPI assignments. ...and only EAPI assignments? Not for any other metadata variable? It's only needed for the EAPI, since that's the only value defined by the ebuild that we intend to use to control how the global environment is initialized prior to sourcing of the ebuild. Doesn't that sort of suggest that EAPI shouldn't be a metadata variable? It's a very special metadata variable. Of course, it could also be implemented in many different ways that do not involve bash variable assingments. Maybe the differences between the various possible ways truly make a difference to some people, but to me it's just hair-splitting [1]. [1] http://en.wikipedia.org/wiki/Trivial_objections -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 08 Mar 2012 09:07:18 -0800 Zac Medico zmed...@gentoo.org wrote: It's a very special metadata variable. Of course, it could also be implemented in many different ways that do not involve bash variable assingments. Maybe the differences between the various possible ways truly make a difference to some people, but to me it's just hair-splitting [1]. [1] http://en.wikipedia.org/wiki/Trivial_objections Having a different, special rule for something that looks exactly like lots of other things that do not have that different, special rule is hardly hair splitting. This rule would have to be documented and have special code to carefully enforce it. That's a big deal. Having something break because you add an unrelated comment to the top of a file is weird. Having something break because you indent it, where nothing else breaks if you do the same thing, is weird. Having something break because you make full use of bash syntax, where nothing else breaks if you do the same thing, is weird. There are already a whole pile of subtle traps for ebuild writers and complications for people learning the system. We should be aiming to reduce these, not add to them. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 08 Mar 2012 10:56:21 -0500 Michael Orlitzky mich...@orlitzky.com wrote: On 03/08/2012 07:03 AM, Michał Górny wrote: Someone suggested using a standard shebang the last time this came up, and if I remember correctly it was one of the least-disagreeable solutions proposed. We could of course define our own custom format, but I think something like, #!/usr/bin/eapi5 would be perfect if we could hand off the interpretation of the ebuild to that program. That solves the problem with new bash features, too, since you could point that command at a specific version. And what would /usr/bin/eapi5 do? Are you suggesting misusing shebang or making ebuilds PM-centric? I was saying that I'd prefer a more-standard use of the shebang (if possible), rather than defining our own header comment syntax. Either way I think the second option is cleaner than regular expressions. Right now, we're guaranteed the features of bash-3.2. I guess we actually use whatever is executing ebuild.sh to source them. But we need to interpret the ebuild file with something: we might as well put *that* in the shebang, since that's what it's for. So if we were to do this with an ebuild right now, we'd add, #!/usr/bin/eapi4 to the header, and instead of sourcing the ebuild with whatever ebuild.sh is using, we would run it with 'eapi4' and pass whatever we need back and forth. Or maybe ebuild.sh would reload itself using 'eapi4'. If any of that makes sense, the PMS would just need to specify some requirements on the shebang command. And something will need to provide that /usr/bin/eapi4 thing. And that introduces new problems: 1) how are we going to support multiple package managers? will we need to install eapi4 thing as a smart wrapper choosing the right PM? 2) what about Prefix? #!/usr/bin/env eapi4 then, or proactive updating of shebangs in synced ebuilds? and then regenerating the whole cache (guess how long does it take to update it), 3) what should happen if user executes ebuild? the ebuild should merge itself? with dependencies or without? -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 8 Mar 2012 17:14:58 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: Having a different, special rule for something that looks exactly like lots of other things that do not have that different, special rule is hardly hair splitting. This rule would have to be documented and have special code to carefully enforce it. That's a big deal. Exactly like HOMEPAGE is a big deal? HOMEPAGE Package's homepage. If you are unable to locate an official one, try to provide a link to freshmeat.net or a similar package tracking site. Never refer to a variable name in the string; include only raw text. http://devmanual.gentoo.org/ebuild-writing/variables/index.html jer
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 8 Mar 2012 18:30:47 +0100 Jeroen Roovers j...@gentoo.org wrote: On Thu, 8 Mar 2012 17:14:58 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: Having a different, special rule for something that looks exactly like lots of other things that do not have that different, special rule is hardly hair splitting. This rule would have to be documented and have special code to carefully enforce it. That's a big deal. Exactly like HOMEPAGE is a big deal? HOMEPAGE Package's homepage. If you are unable to locate an official one, try to provide a link to freshmeat.net or a similar package tracking site. Never refer to a variable name in the string; include only raw text. http://devmanual.gentoo.org/ebuild-writing/variables/index.html That's a QA thing, not a spec thing, and it's there because some old-school developers don't know how to use their package manager properly, and want to copy-paste a string from an ebuild into their browser. It's a silly rule. And if you take a count of how many ebuilds and eclasses break that rule, you'll see exactly why such rules are a problem. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/08/2012 12:28 PM, Michał Górny wrote: And something will need to provide that /usr/bin/eapi4 thing. And that introduces new problems: I'm just parroting someone else's suggestion; I don't really know enough about the details to answer these properly. Not that that will stop me. 1) how are we going to support multiple package managers? will we need to install eapi4 thing as a smart wrapper choosing the right PM? 2) what about Prefix? #!/usr/bin/env eapi4 then, or proactive updating of shebangs in synced ebuilds? and then regenerating the whole cache (guess how long does it take to update it), Wouldn't #!/use/bin/env eapi4 handle both (1) and (2)? You might have to eselect package-manager or something first if you want to use two PMs at once. 3) what should happen if user executes ebuild? the ebuild should merge itself? with dependencies or without? If a user marks the ebuild executable and does ./foo-x.y.ebuild, the eapi4 wrapper can decide what to do. Nothing at all, print larry the cow, or crash (what we do now) are all fine with me =)
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Wed, 7 Mar 2012 21:41:02 +0100 Ulrich Mueller u...@gentoo.org wrote: *** Proposal 1: Parse the EAPI assignment statement *** [...] Written in a more formal way, appropriate for a specification: - Ebuilds must contain at most one EAPI assignment statement. - It must occur within the first N lines of the ebuild (N=10 and N=30 have been suggested). - The statement must match the following regular expression (extended regexp syntax): ^[ \t]*EAPI=([']?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$ I'd make the regexp less strict -- at least allow whitespace around '='. If the intent is to not rely on a specific bash version for a global scope, why should we limit it to the (current) bash syntax? And it may be also a good idea to not rely on a specific line format, so e.g. 'dnl EAPI=4' will work as well. 1b) It is only applied for EAPI 5 and later (which means that the result of the EAPI parsing would be discarded for earlier EAPIs). Err... so what happens if 'new parsing' detects EAPI 4 and 'old parsing' detects EAPI 5? Or more exactly, how does it know when an older EAPI is used if it is supposed to not use the value it uses to detect EAPI? *** Proposal 2: EAPI in header comment *** A different approach would be to specify the EAPI in a specially formatted comment in the ebuild's header. No syntax has been suggested yet, but I believe that the following would work as a specification: - The EAPI must be declared in a special comment in the first line of the ebuild's header, as follows: - The first line of the ebuild must contain the word ebuild, followed by whitespace, followed by the EAPI, followed by end-of-line or whitespace. What if we ever decide to use a language which would would have another requirements for first line? Again, the proposal comes in two variants: 2a) It is combined with a one time change of the file extension, like .ebuild - .eb. And we're going to retroactively migrate the tree or have random file suffixes intermixed? Not to mention we're either keeping two different variants for a longer while, or disregarding backwards compatibility with older package managers for no actual benefit. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 08 Mar 2012 12:48:51 -0500 Michael Orlitzky mich...@orlitzky.com wrote: On 03/08/2012 12:28 PM, Michał Górny wrote: And something will need to provide that /usr/bin/eapi4 thing. And that introduces new problems: I'm just parroting someone else's suggestion; I don't really know enough about the details to answer these properly. Not that that will stop me. It probably should. Although in the early days the model for ebuilds was that they were scripts that were executed, nowadays there's so much support required that it's better to think of ebuilds as being data. If you did have a /usr/bin/eapi5, it would have to be implemented as something that invoked the package manager, not as a direct interpreter. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 8 Mar 2012 18:52:13 +0100 Michał Górny mgo...@gentoo.org wrote: Again, the proposal comes in two variants: 2a) It is combined with a one time change of the file extension, like .ebuild - .eb. And we're going to retroactively migrate the tree or have random file suffixes intermixed? Not to mention we're either keeping two different variants for a longer while, or disregarding backwards compatibility with older package managers for no actual benefit. No, you'd just switch extensions for EAPI 5 onwards, and keep the old rules for EAPI 4. Same idea and impact as GLEP 55, except without as much future proofing. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/08/2012 09:52 AM, Michał Górny wrote: On Wed, 7 Mar 2012 21:41:02 +0100 Ulrich Muelleru...@gentoo.org wrote: 1b) It is only applied for EAPI 5 and later (which means that the result of the EAPI parsing would be discarded for earlier EAPIs). Err... so what happens if 'new parsing' detects EAPI 4 and 'old parsing' detects EAPI 5? Or more exactly, how does it know when an older EAPI is used if it is supposed to not use the value it uses to detect EAPI? We can simply detect this case and treat it as an error. The purpose of the discarded for earlier EAPIs part is to allow more variance for older EAPIs, and treating this case as an error will probably affect zero or a negligible number of older ebuilds in practice. -- Thanks, Zac
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On 03/08/2012 12:53 PM, Ciaran McCreesh wrote: On Thu, 08 Mar 2012 12:48:51 -0500 Michael Orlitzkymich...@orlitzky.com wrote: On 03/08/2012 12:28 PM, Michał Górny wrote: And something will need to provide that /usr/bin/eapi4 thing. And that introduces new problems: I'm just parroting someone else's suggestion; I don't really know enough about the details to answer these properly. Not that that will stop me. It probably should. Although in the early days the model for ebuilds was that they were scripts that were executed, nowadays there's so much support required that it's better to think of ebuilds as being data. If you did have a /usr/bin/eapi5, it would have to be implemented as something that invoked the package manager, not as a direct interpreter. Fair enough, but aren't you arguing the opposite point with Zac? If ebuilds are data, fine, we write EAPI=4 somewhere and be done with it. Anything not having that format is out-of-spec. If they're code, they're code, and we need to execute them somehow. And the reason for the proposal in the first place was that the way we do it now ain't so great, eh?
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 08 Mar 2012 13:37:09 -0500 Michael Orlitzky mich...@orlitzky.com wrote: It probably should. Although in the early days the model for ebuilds was that they were scripts that were executed, nowadays there's so much support required that it's better to think of ebuilds as being data. If you did have a /usr/bin/eapi5, it would have to be implemented as something that invoked the package manager, not as a direct interpreter. Fair enough, but aren't you arguing the opposite point with Zac? If ebuilds are data, fine, we write EAPI=4 somewhere and be done with it. Anything not having that format is out-of-spec. The problem is that right now there's no way to determine the format of the data until you already know the format of the data. We hack around this by not allowing drastic format changes, where drastic includes using things in newer versions of bash and not adding new global scope commands. The question under discussion is whether we a) keep what format the data is in as being part of the data, but impose some strange and arbitrary conditions on it, b) make a one-time change to have some kind of 'header' inside the file describing its format that isn't really part of the data itself, or c) admit that GLEP 55 already solved the problem and we might as well just fix the issue properly once and for all, even if GLEP 55's author is considered by some to be one of Satan's little minions. If they're code, they're code, and we need to execute them somehow. The notion of execute them somehow that's used doesn't fit in with the #! interpreter model. You aren't executing ebuilds via an interpreter. You're performing an action that involves using the data and code in an ebuild multiple times and in multiple different ways, and that may also involve doing the same to an installed package that is being replaced. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 8 Mar 2012, Michał Górny wrote: *** Proposal 1: Parse the EAPI assignment statement *** [...] Written in a more formal way, appropriate for a specification: - Ebuilds must contain at most one EAPI assignment statement. - It must occur within the first N lines of the ebuild (N=10 and N=30 have been suggested). - The statement must match the following regular expression (extended regexp syntax): ^[ \t]*EAPI=([']?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$ I'd make the regexp less strict -- at least allow whitespace around '='. If the intent is to not rely on a specific bash version for a global scope, why should we limit it to the (current) bash syntax? This could certainly be done ... And it may be also a good idea to not rely on a specific line format, so e.g. 'dnl EAPI=4' will work as well. ... but loosening it too much looks a little dangerous, because it would probably match comments and could produce unintended matches. If we allow for a general syntax, we should restrict it to the first line of the ebuild. But that is proposal 2 then. 1b) It is only applied for EAPI 5 and later (which means that the result of the EAPI parsing would be discarded for earlier EAPIs). Err... so what happens if 'new parsing' detects EAPI 4 and 'old parsing' detects EAPI 5? This cannot happen for a legal ebuild: - If the ebuild is EAPI 4, then sourcing it (old parsing) must detect EAPI 4. - If the ebuild is EAPI 5, then new parsing must detect EAPI 5. Or more exactly, how does it know when an older EAPI is used if it is supposed to not use the value it uses to detect EAPI? If new parsing detects EAPI 5 or later, then use this value (and do a sanity check after the ebuild has been sourced). Otherwise, discard the value determined by new parsing and detect the EAPI in the traditional way by sourcing the ebuild. Ulrich
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 8 Mar 2012, Ciaran McCreesh wrote: On Thu, 08 Mar 2012 11:59:33 -0500 Alexandre Rostovtsev tetrom...@gentoo.org wrote: In light of the fact that all 29758 ebuilds in portage already satisfy the proposed whitespace, quoting, and indenting constrains on EAPI assignment, the probability of problems appears to be vanishingly small. And vanishingly small and can be reduced to zero by simply adding a check to repoman. Because they were recently changed, presumably... https://bugs.gentoo.org/show_bug.cgi?id=402167#c36 We had this discussion the last time around too, and people were told to assign in a particular way. As you can see, it didn't work. Sorry, but this is nonsense (or rather FUD). Indeed we had 3 ebuilds (0.01%) in the Portage tree where parsing resulted in an EAPI different from the one in metadata. In one of them, removal of the old assignment statement had simply been forgotten [1]. For the other two, the EAPI had been assigned by an eclass [2], which we consider illegal anyway. Ulrich [1] http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-ml/bin-prot/bin-prot-2.0.3.ebuild?r1=1.1r2=1.2 [2] http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/eclass/apache-2.eclass?r1=1.26r2=1.27
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 8 Mar 2012 20:17:41 +0100 Ulrich Mueller u...@gentoo.org wrote: In one of them, removal of the old assignment statement had simply been forgotten [1]. For the other two, the EAPI had been assigned by an eclass [2], which we consider illegal anyway. ...and yet people do it. That and the violations of the HOMEPAGE rule tell you a lot about what happens when something is made syntactically valid but supposedly not legal. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] RFD: EAPI specification in ebuilds
On Thu, 8 Mar 2012 20:04:55 +0100 Ulrich Mueller u...@gentoo.org wrote: Err... so what happens if 'new parsing' detects EAPI 4 and 'old parsing' detects EAPI 5? This cannot happen for a legal ebuild: - If the ebuild is EAPI 4, then sourcing it (old parsing) must detect EAPI 4. the problem here may be when old parsing will wrongly parse a malformed EAPI 42 assignment (so that new parsing doesnt detect an EAPI and assumes its 0) as EAPI 4. this is a purely hypothetical scenario that will be easily detected if it ever happens, though :) anyway, new parsing != old parsing should trigger a repoman warning for old EAPIs. A.
