Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 19:37:11 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Tue, 24 Feb 2009 20:28:43 +0100 Alexis Ballier aball...@gentoo.org wrote: On Tue, 24 Feb 2009 18:24:16 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Tue, 24 Feb 2009 18:16:54 +0100 Luca Barbato lu_z...@gentoo.org wrote: You're doubling the number of files that have to be read for an operation that's almost purely i/o bound. On top of that, you're introducing a whole bunch of disk seeks in what's otherwise a nice linear operation. I see words, not numbers. Number: double. That's a '2 times'. That only means you're increasing the constant factor in the complexity of the thing... which may very well be completely negligible unless someone provides real benchmarks. In the most common case where metadata is valid, around half of the time it takes Paludis to work out a resolution set is spent grabbing metadata for ebuilds. That sounds like an implementation detail that you could solve by using something else than a flat file database for metadata if open()/read() calls are the slow part. I would be very surprised if that 2 times factor happens to be true, because finding a string in a file is an order of magnitude simpler than sourcing said file with bash. Moreover this doesn't take into account disk and os cache. No no no. *Opening* the file is the slow part, not searching. The file wouldn't otherwise be opened at all. Thus the only drawback is when you open a file, see there that you can't handle the eapi, then close it and open an older one. So you're doing useless things only in that case which in practice will have a ratio far lower than 2. Moreover Luca's benchs show that searching for file names is a negligible factor faster than grepping them while you're just stating that it isn't. Alexis. signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Wed, 25 Feb 2009 07:34:41 +0100 Tiziano Müller dev-z...@gentoo.org wrote: Well, you could theoretical consider everything in the cache valid within the current scope, find the eapi within the cache or the ebuild and then reconsider things. You can't even do that, because new EAPIs might change how cache entries are validated. An ID might have been moved to a new EAPI using some new EAPI thing (such as changing how inherit works) that doesn't end up altering mtimes in a way that can be caught. Would it be possible to change the cache-format with with G55? Meaning: Have the current cache-format for the current *.ebuild and another for *.ebuild-N (where I mean by cache-format the contents of the cache-files)? If you have GLEP 55 you don't need it. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Wed, 25 Feb 2009 09:33:44 +0100 Alexis Ballier aball...@gentoo.org wrote: That sounds like an implementation detail that you could solve by using something else than a flat file database for metadata if open()/read() calls are the slow part. Metadata's shipped with the tree. It's a PMS detail. If we didn't care about package manager performance, we wouldn't be shipping metadata with the tree at all... I would be very surprised if that 2 times factor happens to be true, because finding a string in a file is an order of magnitude simpler than sourcing said file with bash. Moreover this doesn't take into account disk and os cache. No no no. *Opening* the file is the slow part, not searching. The file wouldn't otherwise be opened at all. Thus the only drawback is when you open a file, see there that you can't handle the eapi, then close it and open an older one. Uh. No. The drawback is that you're opening around ten thousand files that would otherwise not be opened. That's a huge cost. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Wed, 25 Feb 2009 04:04:46 +0100 Luca Barbato lu_z...@gentoo.org wrote: given that the simplest thing is hacking ebuild.sh and extract eapi with a simple C program (you can use pcre or ragel if you want) exactly before the ebuild source: That you're bringing ebuild.sh into this shows you still haven't worked out how the process works. There is no need to use ebuild.sh (which is a very good thing, because launching bash is slow) when there's valid metadata. So you have patches for Portage? Please show them. Up there what's enough to check the viability for the solution. No, it's completely wrong. ebuild.sh has nothing to do with this. unknown isn't unsupported? Huh? Please explain what you mean. mv cat/pkg-version.ebuild cat/pkg-version_foo.ebuild emerge -vp pkg portage will warn about not knowing pkg-version_foo and will ignore it. Yes, it will warn noisily. This is unacceptable, since stable users will have months and months of noise when new rules come along. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh wrote: On Wed, 25 Feb 2009 04:04:46 +0100 Luca Barbato lu_z...@gentoo.org wrote: given that the simplest thing is hacking ebuild.sh and extract eapi with a simple C program (you can use pcre or ragel if you want) exactly before the ebuild source: That you're bringing ebuild.sh into this shows you still haven't worked out how the process works. There is no need to use ebuild.sh (which is a very good thing, because launching bash is slow) when there's valid metadata. it there is a valid metadata you have everything there already and you don't have to parse the ebuild so... So you have patches for Portage? Please show them. Up there what's enough to check the viability for the solution. No, it's completely wrong. ebuild.sh has nothing to do with this. Is there another place in portage that extracts the EAPI value? Yes, it will warn noisily. This is unacceptable, since stable users will have months and months of noise when new rules come along. unacceptable... as in it's ugly to see... lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Wed, Feb 25, 2009 at 04:56:04PM +0100, Luca Barbato wrote: Yes, it will warn noisily. This is unacceptable, since stable users will have months and months of noise when new rules come along. unacceptable... as in it's ugly to see... No, it's unacceptable because stable users do not need that kind of stuff thrown at them. Stable users use stable because they want a very predictable workflow. Noisy errors that shouldn't affect them(they are in the stable branch) *is* unacceptable, and not just because it's ugly, though that's certainly part of it. -- - Thomas Anderson Gentoo Developer / Areas of responsibility: AMD64, Secretary to the Gentoo Council -
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Wed, 25 Feb 2009 16:56:04 +0100 Luca Barbato lu_z...@gentoo.org wrote: That you're bringing ebuild.sh into this shows you still haven't worked out how the process works. There is no need to use ebuild.sh (which is a very good thing, because launching bash is slow) when there's valid metadata. it there is a valid metadata you have everything there already and you don't have to parse the ebuild so... Except you can't tell whether the metadata is valid unless you already know the EAPI. As you have already been told. So you have patches for Portage? Please show them. Up there what's enough to check the viability for the solution. No, it's completely wrong. ebuild.sh has nothing to do with this. Is there another place in portage that extracts the EAPI value? Yup. The part where it reads the metadata cache. Yes, it will warn noisily. This is unacceptable, since stable users will have months and months of noise when new rules come along. unacceptable... as in it's ugly to see... No, as in it'll result in zillions of users wondering what's going on and why their screen is getting spammed, and zillions of bug reports and forum posts. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh wrote: On Wed, 25 Feb 2009 09:33:44 +0100 Alexis Ballier aball...@gentoo.org wrote: That sounds like an implementation detail that you could solve by using something else than a flat file database for metadata if open()/read() calls are the slow part. Metadata's shipped with the tree. It's a PMS detail. If we didn't care about package manager performance, we wouldn't be shipping metadata with the tree at all... I would be very surprised if that 2 times factor happens to be true, because finding a string in a file is an order of magnitude simpler than sourcing said file with bash. Moreover this doesn't take into account disk and os cache. No no no. *Opening* the file is the slow part, not searching. The file wouldn't otherwise be opened at all. Thus the only drawback is when you open a file, see there that you can't handle the eapi, then close it and open an older one. Uh. No. The drawback is that you're opening around ten thousand files that would otherwise not be opened. That's a huge cost. Huge cost... emerge -uDp world (cold os cache) real1m10.353s user0m17.077s sys 0m0.440s with eapitool getting the eapi before sourcing. real1m10.636s user0m16.941s sys 0m0.368s cold cache, no metadata available real6m23.106s user3m32.141s sys 1m50.855s with eapitool real6m26.564s user3m31.853s sys 1m50.511s I'd rather see more people backing their ideas with numbers... lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Thomas Anderson wrote: On Wed, Feb 25, 2009 at 04:56:04PM +0100, Luca Barbato wrote: Yes, it will warn noisily. This is unacceptable, since stable users will have months and months of noise when new rules come along. unacceptable... as in it's ugly to see... No, it's unacceptable because stable users do not need that kind of stuff thrown at them. Stable users use stable because they want a very predictable workflow. Noisy errors that shouldn't affect them(they are in the stable branch) *is* unacceptable, and not just because it's ugly, though that's certainly part of it. You have quite a good point here. It's purely cosmetical, but since it's an unexpected behavior and it's annoying and could alienate our users it's better to avoid it with all our strength. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Wed, 25 Feb 2009 17:17:29 +0100 Luca Barbato lu_z...@gentoo.org wrote: I'd rather see more people backing their ideas with numbers... I already told you your numbers are nonsense. Of course opening the file when you've already opened it isn't going to make any difference. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Tue, 24 Feb 2009 12:25:27 -0500 Jim Ramsay l...@gentoo.org wrote: ...and it means we can't change name or version rules. ...and it means over doubling the best possible time to work out a dependency tree in the common case where the metadata cache is valid. ...and it means we can't make arbitrary format changes. Those would all land in the category of backwards compatibility mentioned below, as they would all break current sourcing rules. No, they're also future issues. Without glep 55, every time they come up we have to go through the whole mess again. This minor/major EAPI scheme is exactly equivalent to glep 55 in the ways that it addresses the non-implementation-specific details - It could even be added as a caveat to glep-55 that says something like: You should not change filename extension (ie, major-EAPI version, or EPARSE version, or whatever we want to call it...) except when you *have to* because of changes such as name or version rules, arbitrary format changes, or anything else that breaks the sourcing rules of the existing filename extensions. Simpler feature improvements can be done using whatever internal minor-EAPI version is defined by the major-EAPI version. This doesn't prevent you from changing the filename extension when you have to do so, it just suggests that maybe you don't have to do it for every single feature-set you may want to implement. Developers already have to stop and think and consult the conveniently available table of features for EAPIs. By splitting the EAPI concept in two you're doubling the amount of data to be learnt. I would think that this is a very small cost, and the benefit would be (I hope) that more people would agree on the solution and then we can go forward. Is that not a valid consideration? I'd expect to see changes that would warrant a major bump in every other EAPI or so anyway, so it's not really worth the complexity. If that is indeed the case, then adding this caveat to glep-55 is basically a nop. If every EAPI includes a non-backwards-compatible change that breaks existing PMs, the filename extension will be changed every time. But when you say worth the complexity, I'm not exactly sure what your standards of worthiness are. I don't think the human cognition of a 2-level versioning scheme is that tricky, so I assume you must mean complexity in the internals of package managers - but this should just be a Simple Matter Of Programming. I'll further qualify this response by mentioning that I am not a package manager maintainer. I don't know beans about metadata and cacheing and what the tradeoffs may be between a two-level EAPI and a single-level EAPI stored in the filename. I understand that parsing two-level EAPI is more expensive than a single-level stored in the filename. I don't however know how to figure out if it is too expensive, or whose subjective scale we should use to measure this. I personally feel the complexity that you say is too costly is a fair tradeoff for a proposal that people will accept. (Of course I have no idea if people actually would accept a two-level EAPI any more than glep-55 as-is... I just think it addresses the concerns I've heard in this thread in a way that does not break the valid solutions to real problems presented in glep-55) -- Jim Ramsay Gentoo Developer (rox/fluxbox/gkrellm/vim) signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 14:08:45 -0500 Jim Ramsay l...@gentoo.org wrote: But when you say worth the complexity, I'm not exactly sure what your standards of worthiness are. I don't think the human cognition of a 2-level versioning scheme is that tricky, so I assume you must mean complexity in the internals of package managers - but this should just be a Simple Matter Of Programming. People are struggling with the one level scheme we have now. We're already having to produce fancy tables and summaries for new EAPIs because people can't keep track of when features came along. Two levels just means no-one will remember any of it. For the package manager, it's just a bit of added mess, not any major difficulty. (Of course I have no idea if people actually would accept a two-level EAPI any more than glep-55 as-is... I just think it addresses the concerns I've heard in this thread in a way that does not break the valid solutions to real problems presented in glep-55) People are opposed to 55 because of a knee-jerk reaction against changing file extensions and against doing anything that comes from the great Satan and all his little minions... If you're going to throw an equivalent but supposedly compromising solution at people, go for '.eapi3.eb' instead. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 18:24:16 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Tue, 24 Feb 2009 18:16:54 +0100 Luca Barbato lu_z...@gentoo.org wrote: You're doubling the number of files that have to be read for an operation that's almost purely i/o bound. On top of that, you're introducing a whole bunch of disk seeks in what's otherwise a nice linear operation. I see words, not numbers. Number: double. That's a '2 times'. That only means you're increasing the constant factor in the complexity of the thing... which may very well be completely negligible unless someone provides real benchmarks. I would be very surprised if that 2 times factor happens to be true, because finding a string in a file is an order of magnitude simpler than sourcing said file with bash. Moreover this doesn't take into account disk and os cache. Alexis. signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 20:28:43 +0100 Alexis Ballier aball...@gentoo.org wrote: On Tue, 24 Feb 2009 18:24:16 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Tue, 24 Feb 2009 18:16:54 +0100 Luca Barbato lu_z...@gentoo.org wrote: You're doubling the number of files that have to be read for an operation that's almost purely i/o bound. On top of that, you're introducing a whole bunch of disk seeks in what's otherwise a nice linear operation. I see words, not numbers. Number: double. That's a '2 times'. That only means you're increasing the constant factor in the complexity of the thing... which may very well be completely negligible unless someone provides real benchmarks. In the most common case where metadata is valid, around half of the time it takes Paludis to work out a resolution set is spent grabbing metadata for ebuilds. I would be very surprised if that 2 times factor happens to be true, because finding a string in a file is an order of magnitude simpler than sourcing said file with bash. Moreover this doesn't take into account disk and os cache. No no no. *Opening* the file is the slow part, not searching. The file wouldn't otherwise be opened at all. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: People are struggling with the one level scheme we have now. We're already having to produce fancy tables and summaries for new EAPIs because people can't keep track of when features came along. Two levels just means no-one will remember any of it. I disagree with your assertion that people are struggling - I think things are very nicely documented in PMS and the devmanual, which are where all EAPI changes should be documented in the future, regardless if you specify the EAPI in the file, the extension, or both. Two levels really just means that any fancy tables will have to have one extra row (or perhaps a series of fancy tables) and any summaries will have to have an extra section added whenever a new filename extension becomes necessary. For the package manager, it's just a bit of added mess, not any major difficulty. This is good to know, thank you for the clarification. People are opposed to 55 because of a knee-jerk reaction against changing file extensions and against doing anything that comes from the great Satan and all his little minions... If you're going to throw an equivalent but supposedly compromising solution at people, go for '.eapi3.eb' instead. I can't speak to anyone's motivations or religious beliefs other than my own here, but the opposition I have heard most often in this thread is something like: I don't like it when the file extension changes so often. Some people site historical president or the way other software does things, or whatever - doesn't really matter. What does matter is that some people don't like it when the file extension changes very often. I think my solution is a valid compromise because it balances, in my opinion, the two camps, whose arguments I summarize as: glep-55'ers: I don't care if the file extension changes all the time, I just want a solution that works and is reasonably future-proof Anti-55'ers: I don't want the file extension to change ever, but I would agree that for major-enough changes it may be required sometimes If I understand the '.eapi3.eb' to which you make passing reference, this is just a fancy hand-wavy way to say Look, the true .eb extension won't ever change, just the .eapi3 part which isn't technically the extension... which isn't a compromise at all - It's an attempt to (cleverly?) obfuscate where in the filename the EAPI is stored. -- Jim Ramsay Gentoo Developer (rox/fluxbox/gkrellm) signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 15:07:29 -0500 Jim Ramsay l...@gentoo.org wrote: Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: People are struggling with the one level scheme we have now. We're already having to produce fancy tables and summaries for new EAPIs because people can't keep track of when features came along. Two levels just means no-one will remember any of it. I disagree with your assertion that people are struggling - I think things are very nicely documented in PMS and the devmanual, which are where all EAPI changes should be documented in the future, regardless if you specify the EAPI in the file, the extension, or both. They only ended up nicely documented after people moaned a lot that they were having a hard time keeping track of EAPIs... Two levels really just means that any fancy tables will have to have one extra row (or perhaps a series of fancy tables) and any summaries will have to have an extra section added whenever a new filename extension becomes necessary. It'll mean people will carry on having to use the tables, and won't start remembering things as time goes on. If I understand the '.eapi3.eb' to which you make passing reference, this is just a fancy hand-wavy way to say Look, the true .eb extension won't ever change, just the .eapi3 part which isn't technically the extension... which isn't a compromise at all - It's an attempt to (cleverly?) obfuscate where in the filename the EAPI is stored. Yup. And yet there're people who are perfectly happy with .eapi3.eb who hate GLEP 55. That should tell you all you need to know about what's going on here... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh wrote: ..and it means we can't change name or version rules. Why? Just parse the EAPI out of the file before you interpret the version and name from the filename. Indeed - you could have a future EAPI remove the name and version from the filename entirely. If a package manager doesn't understand the EAPI in a file it shouldn't do anything at all with it. ..and it means over doubling the best possible time to work out a dependency tree in the common case where the metadata cache is valid. I can see why it takes an extra pass - but does that mean a doubling of time? Couldn't the EAPI be cached as well to reduce disk access? ..and it means we can't make arbitrary format changes. Well, you would need to preserve the EAPI in the header, but other than that you could actually turn an ebuild into an otherwise completely binary file, or whatever. Just how much more flexibility than that is needed?
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 15:17:01 -0500 Richard Freeman ri...@gentoo.org wrote: Why? Just parse the EAPI out of the file before you interpret the version and name from the filename. Indeed - you could have a future EAPI remove the name and version from the filename entirely. If a package manager doesn't understand the EAPI in a file it shouldn't do anything at all with it. Then you get into the mess of deciding what is or is not an ebuild... Currently it's well defined; if you start making the package manager look inside files things get very confusing... ..and it means over doubling the best possible time to work out a dependency tree in the common case where the metadata cache is valid. I can see why it takes an extra pass - but does that mean a doubling of time? Couldn't the EAPI be cached as well to reduce disk access? It means opening a file that would otherwise not be opened at all. And if the EAPI is in the file, you have to fish inside that file to pull it out before you can work out whether the cache entry that might contain the EAPI already is valid. (We don't have to do this currently because inherit hasn't changed behaviour at all.) ..and it means we can't make arbitrary format changes. Well, you would need to preserve the EAPI in the header, but other than that you could actually turn an ebuild into an otherwise completely binary file, or whatever. Just how much more flexibility than that is needed? I remember hearing that years ago, except it was well you can't change global scope behaviour for EAPIs, but just how much more flexibility than that is needed?. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Tue, 24 Feb 2009 15:07:29 -0500 Jim Ramsay l...@gentoo.org wrote: I think things are very nicely documented in PMS and the devmanual, which are where all EAPI changes should be documented in the future, regardless if you specify the EAPI in the file, the extension, or both. They only ended up nicely documented after people moaned a lot that they were having a hard time keeping track of EAPIs... You can't possibly be suggesting that everyone will be able to keep an ever-increasing number of feature sets in his or her mind, or that changing from a two-level to a one-level EAPI definition will remove the need for documentation going forward, so I'm not sure what you mean by this. Two levels really just means that any fancy tables will have to have one extra row (or perhaps a series of fancy tables) and any summaries will have to have an extra section added whenever a new filename extension becomes necessary. It'll mean people will carry on having to use the tables, and won't start remembering things as time goes on. See comment above. The need for documentation will only increase going forward as new and varied EAPI definitions are created. If I understand the '.eapi3.eb' to which you make passing reference, this is just a fancy hand-wavy way to say Look, the true .eb extension won't ever change, just the .eapi3 part which isn't technically the extension... which isn't a compromise at all - It's an attempt to (cleverly?) obfuscate where in the filename the EAPI is stored. Yup. And yet there're people who are perfectly happy with .eapi3.eb who hate GLEP 55. That should tell you all you need to know about what's going on here... Seriously? That's scary. But hey, if that's actually going to get more people behind this, let's do it instead. -- Jim Ramsay Gentoo Developer (rox/fluxbox/gkrellm/vim) signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 15:37:36 -0500 Jim Ramsay l...@gentoo.org wrote: They only ended up nicely documented after people moaned a lot that they were having a hard time keeping track of EAPIs... You can't possibly be suggesting that everyone will be able to keep an ever-increasing number of feature sets in his or her mind, or that changing from a two-level to a one-level EAPI definition will remove the need for documentation going forward, so I'm not sure what you mean by this. That's exactly what I mean. Developers can probably just about keep up with the two or three EAPIs they'll ever be working with on a regular basis, but they probably can't keep up with that if you double it. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Tue, 24 Feb 2009 15:37:36 -0500 Jim Ramsay l...@gentoo.org wrote: They only ended up nicely documented after people moaned a lot that they were having a hard time keeping track of EAPIs... You can't possibly be suggesting that everyone will be able to keep an ever-increasing number of feature sets in his or her mind, or that changing from a two-level to a one-level EAPI definition will remove the need for documentation going forward, so I'm not sure what you mean by this. That's exactly what I mean. Developers can probably just about keep up with the two or three EAPIs they'll ever be working with on a regular basis, but they probably can't keep up with that if you double it. Well, if you're assuming only two or three EAPIs in 'mental cache' at any one time under glep-55, I'm not sure how this changes wrt. a two-level system. A two-level system doesn't change the number of EAPIs in the tree or available to developers, it just changes how they are named. Regardless, this does not remove the need for documentation. All the EAPIs should be documented in both the PMS and the devmanual. This makes it possible for new developers to learn about the current features available, and also helps existing devs who may need to recover from 'mental page faults' from time to time. -- Jim Ramsay Gentoo Developer (rox/fluxbox/gkrellm/vim) signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh wrote: On Tue, 24 Feb 2009 15:17:01 -0500 Richard Freeman ri...@gentoo.org wrote: Why? Just parse the EAPI out of the file before you interpret the version and name from the filename. Indeed - you could have a future EAPI remove the name and version from the filename entirely. If a package manager doesn't understand the EAPI in a file it shouldn't do anything at all with it. Then you get into the mess of deciding what is or is not an ebuild... Currently it's well defined; if you start making the package manager look inside files things get very confusing... an ebuild is something ending with .ebuild ... It means opening a file that would otherwise not be opened at all. And if the EAPI is in the file, you have to fish inside that file to pull it out before you can work out whether the cache entry that might contain the EAPI already is valid. Keeping in mind that: - if the cache is present you won't do it (so normal users aren't touched) - you just need a way to upgrade portage and nothing else. You: - have to open them on regen, no matter what (you are adding it to portage) - the cache entry has already the eapi value so there it is. ..and it means we can't make arbitrary format changes. Well, you would need to preserve the EAPI in the header, but other than that you could actually turn an ebuild into an otherwise completely binary file, or whatever. Just how much more flexibility than that is needed? I remember hearing that years ago, except it was well you can't change global scope behaviour for EAPIs, but just how much more flexibility than that is needed?. Given that the fixed header gives you ALL the flexibility. You may give provision to consider the next bytes as any kind of serialization... lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 22:46:17 +0100 Luca Barbato lu_z...@gentoo.org wrote: It means opening a file that would otherwise not be opened at all. And if the EAPI is in the file, you have to fish inside that file to pull it out before you can work out whether the cache entry that might contain the EAPI already is valid. Keeping in mind that: - if the cache is present you won't do it (so normal users aren't touched) - you just need a way to upgrade portage and nothing else. Not true. You don't know whether the cache is valid until you know what the EAPI is. You: - have to open them on regen, no matter what (you are adding it to portage) - the cache entry has already the eapi value so there it is. Can't use the cache until you know what the EAPI is. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh wrote: Not true. You don't know whether the cache is valid until you know what the EAPI is. If you are on the user scenario the cache is valid. If the eapi changes the cache meaning you can always put the new cache in another place older portage won't look into. You: - have to open them on regen, no matter what (you are adding it to portage) - the cache entry has already the eapi value so there it is. Can't use the cache until you know what the EAPI is. The current cache holds all the current portage needs to know what to ignore, providing the cache in such format will make portage ignore any future change. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 23:48:27 +0100 Luca Barbato lu_z...@gentoo.org wrote: Ciaran McCreesh wrote: Not true. You don't know whether the cache is valid until you know what the EAPI is. If you are on the user scenario the cache is valid. Uh. Wrong. Can't use the cache until you know what the EAPI is. The current cache holds all the current portage needs to know what to ignore, providing the cache in such format will make portage ignore any future change. Uh. Wrong. The information used to validate a cache entry is only usable if you know the behaviour of 'inherit' that was used to create the entry. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh wrote:
On Tue, 24 Feb 2009 18:16:54 +0100
Luca Barbato lu_z...@gentoo.org wrote:
You're doubling the number of files that have to be read for an
operation that's almost purely i/o bound. On top of that, you're
introducing a whole bunch of disk seeks in what's otherwise a nice
linear operation.
I see words, not numbers.
Number: double. That's a '2 times'.
given that the simplest thing is hacking ebuild.sh and extract eapi with
a simple C program (you can use pcre or ragel if you want) exactly
before the ebuild source:
Index: bin/ebuild.sh
===
--- bin/ebuild.sh (revision 12704)
+++ bin/ebuild.sh (working copy)
@@ -1848,6 +1848,7 @@
# eclasses, they need to be unset before this process of
# interaction begins.
unset DEPEND RDEPEND PDEPEND IUSE
+ EAPI=$(eapitool ${EBUILD})
source ${EBUILD} || die error sourcing ebuild
if [ ${EBUILD_PHASE} != depend ] ; then
I think your numbers are a bit pessimistic, this is when you get EAPI in
portage, post source ${EBUILD}, opening the file before source would
just put in the cache one line earlier.
You don't know whether the cache is valid until you know the EAPI. It
only works currently because EAPIs don't change inherit behaviour.
There were already discussions about switching cache format, if we want
to change the inherit behaviour we could just switch at the same time
the cache format and leave dummy entry for compatibility with legacy
portage.
So you have patches for Portage? Please show them.
Up there what's enough to check the viability for the solution.
the bash subst I wrote before could be used instead of the call to get
the eapi in extension behaviour.
unknown isn't unsupported?
Huh? Please explain what you mean.
mv cat/pkg-version.ebuild cat/pkg-version_foo.ebuild
emerge -vp pkg
portage will warn about not knowing pkg-version_foo and will ignore it.
lu
--
Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Luca Barbato wrote: Luca Barbato wrote: Ciaran McCreesh wrote: Because your proposal addresses none of the underlying problems which GLEP 55 was created to solve. let's get some numbers to have an idea of the dimension of the problem. I just don't think those numbers tell us anything and that should be obvious from anyone who has read GLEP 55[1], we ain't really attempting to solve a problem that exists within the tree currently (well the bash issue does, in a way ). We are trying to solve issues that ware stopping the tree moving forward. Lets evaluate GLEP 55 in the problems it is attempting to solve. [1] Problem The current way of specifying the EAPI in ebuilds is flawed. In order to get the EAPI the package manager needs to source the ebuild, which itself needs the EAPI in the first place. Otherwise it imposes a serious limitation, namely every ebuild, using any of the future EAPIs, will have to be source'able by old package managers and hence there is no way to do any of the following: * Change the behaviour of inherit in any way (for example, to extend or change eclass functionality). * Add new global scope functions in any sane way. * Extend versioning rules in an EAPI - for example, addition of the scm suffix - GLEP54 [1].
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
(Ok this thread grew too long, so I gotta chime in :)) We could start using extended attributes or mandate reiser4 for portage dir or some other special in between (the inside of file and its name) feature.. Sorry for the noise and insane implementation suggestion :).. George PS Actually, come to think of it, reizer4 might be just the right tool for the job (for keeping the volatile info split into bunch of small entries), hypothetically of course.. Tuesday, 24. February 2009, Luca Barbato Ви написали: Luca Barbato wrote: Ciaran McCreesh wrote: Because your proposal addresses none of the underlying problems which GLEP 55 was created to solve. let's get some numbers to have an idea of the dimension of the problem. [skip] Please come up with other numbers or saner implementations to compare. lu
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Alistair Bush wrote: I just don't think those numbers tell us anything and that should be obvious from anyone who has read GLEP 55[1], we ain't really attempting to solve a problem that exists within the tree currently (well the bash issue does, in a way ). We are trying to solve issues that ware stopping the tree moving forward. Lets evaluate GLEP 55 in the problems it is attempting to solve. I'm afraid you missed the whole point... - what is in the proposal is a solution looking for a problem: nobody updated the glep with the required sections, nobody put up a list of bugs/rfe from bugzilla it helps to solve. Vague leading to the future change declaration aren't what I'd expect. - Assuming there is an actual reason to move forward (by digging bugzilla yourself or deciding to do so as academic exercise) you could think about the problem and its solutions (my the email starting this thread on dev) - Given all you need is just to have a way to get the information about EAPI before you actually parse the ebuild since the eapi defines how you parse it, you can come up with various solutions, the simplest being first extract the eapi, being it in a fixed place, and then do the parse. - Extracting such information could have different costs depending on where to place it. - I started to check if the proposal about having the fixed position as the end of the filename is really much more viable than having it at the top of the file. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
George Shapovalov wrote: (Ok this thread grew too long, so I gotta chime in :)) We could start using extended attributes or mandate reiser4 for portage dir or some other special in between (the inside of file and its name) feature.. No. 1) I wouldn't use reiser4 so that would be the end of that. 2) how well do rsync and cvs support xattr's. How about linux support verses bsd, or windows even. 3) It is just a bad solution Sorry for the noise and insane implementation suggestion :).. At least you realise it :)
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Luca Barbato wrote: Alistair Bush wrote: I just don't think those numbers tell us anything and that should be obvious from anyone who has read GLEP 55[1], we ain't really attempting to solve a problem that exists within the tree currently (well the bash issue does, in a way ). We are trying to solve issues that ware stopping the tree moving forward. Lets evaluate GLEP 55 in the problems it is attempting to solve. I'm afraid you missed the whole point... - what is in the proposal is a solution looking for a problem: nobody updated the glep with the required sections, nobody put up a list of bugs/rfe from bugzilla it helps to solve. Vague leading to the future change declaration aren't what I'd expect. So im mean't to start committing ebuilds into the tree that expect a certain unimplemented functionality, only to file bugs against portage for it not supporting them? Or can we be smart enough to realise that there are limitation to the current standard and then attempt to fix them before they become a problem. Plus we already know of at least one case where we will encounter a problem in the future, why? because we have already. Not sure if there is a open bug about it tho. This actually eats at me, your basically saying GLEP's should only be reactive. Why don't we all just roll over and die. - Assuming there is an actual reason to move forward (by digging bugzilla yourself or deciding to do so as academic exercise) you could think about the problem and its solutions (my the email starting this thread on dev) I have already considered the problems, and believe GLEP 55 is the **best** solution to them. Is it perfect, no. But I have yet to see anything better. - Given all you need is just to have a way to get the information about EAPI before you actually parse the ebuild since the eapi defines how you parse it, you can come up with various solutions, the simplest being first extract the eapi, being it in a fixed place, and then do the parse. Yes exactly, you need to know the EAPI before you __parse__ the ebuild. At least we agree that nothing should have to read the contents of the file to determine EAPI (doing so would be parsing now wouldn't it). So seeing that we agree with that, where should we stick the EAPI. m 1) How about in a flat txt file: That would become a developers nightmare. 2) In an xml file. Package managers would have to support xml. Not the best thing in the world. also could be a nightmare, adding an entry for every ebuild. 3) As an xattr. Well this idea is novel. I bet it would make the tree nice and stable too. Lets not forget how annoying it will be for devs. 4) Parsing the ebuild. But what are we parsing?, lets not limit ourselves to bash, we might want to change languages completely. If it is bash, what version, what if EAPI is set multiple times, what if its set in an eclass. 5) M...On the file name sounds like a good idea. especially as an extension. provides information to a package manager, person, script/program without them needing to open anything. identifies the contents just like .txt, .c, .o, .jpeg, etc - Extracting such information could have different costs depending on where to place it. I believe it being on the filename would be the least costly, in terms of processor/io at least.
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Alistair Bush wrote: Luca Barbato wrote: Alistair Bush wrote: I just don't think those numbers tell us anything and that should be obvious from anyone who has read GLEP 55[1], we ain't really attempting to solve a problem that exists within the tree currently (well the bash issue does, in a way ). We are trying to solve issues that ware stopping the tree moving forward. Lets evaluate GLEP 55 in the problems it is attempting to solve. I'm afraid you missed the whole point... - what is in the proposal is a solution looking for a problem: nobody updated the glep with the required sections, nobody put up a list of bugs/rfe from bugzilla it helps to solve. Vague leading to the future change declaration aren't what I'd expect. So im mean't to start committing ebuilds into the tree that expect a certain unimplemented functionality, only to file bugs against portage for it not supporting them? Apparently you missed rfe or that it does mean =\ Plus we already know of at least one case where we will encounter a problem in the future, why? because we have already. Not sure if there is a open bug about it tho. Do you know that a problem means nothing, bug #number means something? Do you know that improvement and enhancement can be requested on bugzilla as well? This actually eats at me, your basically saying GLEP's should only be reactive. Why don't we all just roll over and die. I'm afraid you are getting quite emotional for no reason. I have already considered the problems, and believe GLEP 55 is the **best** solution to them. Is it perfect, no. But I have yet to see anything better. YOU, other did and consider what is proposed on that trash. Mediation is needed apparently. What is sure is that the glep proposal is lacking lots of details and apparently none of the proponents are even willing to try to cope with that. Yes exactly, you need to know the EAPI before you __parse__ the ebuild. You have to extract the eapi before doing the parsing the eapi defines, but you can parse the ebuild just to get the eapi and then do something or nothing depending on that value... 1) How about in a flat txt file: That would become a developers nightmare. 2) In an xml file. Package managers would have to support xml. Not the best thing in the world. also could be a nightmare, adding an entry for every ebuild. 3) As an xattr. Well this idea is novel. I bet it would make the tree nice and stable too. Lets not forget how annoying it will be for devs. 4) Parsing the ebuild. But what are we parsing?, lets not limit ourselves to bash, we might want to change languages completely. If it is bash, what version, what if EAPI is set multiple times, what if its set in an eclass. What if is exactly something you cannot use, it's a slippery slope that leads to qbits frozen objects from the outher space or other insane stuff that may or may not happen. 5) M...On the file name sounds like a good idea. especially as an extension. provides information to a package manager, person, script/program without them needing to open anything. identifies the contents just like .txt, .c, .o, .jpeg, etc So for normal multimedia file I'd have to have myfile.mov-aac-h264-ass as extension? strange, mplayer is perfectly fine even if it is called myfile and file(1) usually can tell me what's inside it in term of codec and sometimes even it's params. - Extracting such information could have different costs depending on where to place it. I believe it being on the filename would be the least costly, in terms of processor/io at least. try yourself, I did and that's what I found regarding the case of cache regen (that, as I wrote earlier, is the interesting case) is in one of the previous emails as well... btw it's also quite easy plant both proposals in portage and see what happen if you really like, but I preferred give something everybody can try in bash. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 08:08:23 +0100 Luca Barbato lu_z...@gentoo.org wrote: Is there any technical merit in putting eapi in the file extension while we could restrict the format the same way in file and have about the same, negligible, performance hit? (I used warm cache since you need the file anyway so you don't spend time to look it up twice or put it in cache twice) Uh, your benchmarks are nonsense. That is not how metadata checks work. By parsing the ebuilds you're talking doubling the number of file reads required to get the job done, and massively increasing the number of seeks required. But that isn't even the main issue. The main issue is that even if you retroactively pretend that all ebuilds are in a format they're not, and ignore the breakage, and then wait for a year for package managers to try to parse your new format, you *still* can't change name or versioning rules. Again, these are all things that have been discussed at length previously. Please either come up with a legitimate technical objection, or admit that you've seen the light. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Alistair Bush wrote: 4) Parsing the ebuild. But what are we parsing?, lets not limit ourselves to bash, we might want to change languages completely. If it is bash, what version, what if EAPI is set multiple times, what if its set in an eclass. How do you do this if you're getting EAPI from the filename? How do you set it multiple times? How do you set it in an eclass if you're getting it from the filename? It seems like when we're talking about just putting the EAPI in a comment line on line x of the ebuild we're barraged with 47 ways that it will limit us, but when we're talking about EAPI in the filename suddenly we're not concerned with those limitations. If it helps maybe we need to split EAPI into two records - one that deals with how to fundamentally parse the file and find out the EAPI, and the other that implements everything else the EAPI does. I will certainly concede that putting it inside the ebuild potentially breaks compatibility with existing package managers. That is certainly a downside to this approach. However, none of the other objections that have been raised appear to hold water. An EAPI in a filename is a blob of text that needs to be parsed out in one particular way with one set of system calls. An EAPI embedded in the file is a blob of text that needs to be parsed out in one particular way with one set of system calls. And if backwards compatibility were a serious issue you could define a new .ebuild2 file spec that incorporates the EAPI inside the file and current package managers would ignore it. Then you're not changing the file extension every time a new EAPI comes along, and the need to do so could be handled via future GLEPs. Or we could just delay implementation and clean up existing package managers and tell users to migrate.
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 10:46:30 -0500 Richard Freeman ri...@gentoo.org wrote: I will certainly concede that putting it inside the ebuild potentially breaks compatibility with existing package managers. That is certainly a downside to this approach. However, none of the other objections that have been raised appear to hold water. ...and it means we can't change name or version rules. ...and it means over doubling the best possible time to work out a dependency tree in the common case where the metadata cache is valid. ...and it means we can't make arbitrary format changes. And if backwards compatibility were a serious issue you could define a new .ebuild2 file spec that incorporates the EAPI inside the file and current package managers would ignore it. Then you're not changing the file extension every time a new EAPI comes along, and the need to do so could be handled via future GLEPs. Developers already have to stop and think and consult the conveniently available table of features for EAPIs. By splitting the EAPI concept in two you're doubling the amount of data to be learnt. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh wrote: On Tue, 24 Feb 2009 08:08:23 +0100 Uh, your benchmarks are nonsense. Provide your nonsensical ones. That is not how metadata checks work. Explain how they work, regen works that way... By parsing the ebuilds you're talking doubling the number of file reads required to get the job done, and massively increasing the number of seeks required. Apparently it doesn't impact anything. But that isn't even the main issue. The main issue is that even if you retroactively pretend that all ebuilds are in a format they're not, and ignore the breakage, and then wait for a year for package managers to try to parse your new format, you *still* can't change name or versioning rules. why? when portage would breanch if I put an ebuild with a wacky version AND there is a valid cache for that telling its eapi 99 ? Again, these are all things that have been discussed at length previously. Please either come up with a legitimate technical objection, or admit that you've seen the light. the glep doesn't show any of those nor reference to it, as I said before, do your homework and probably more people will be happier with your proposals. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 17:04:28 +0100 Luca Barbato lu_z...@gentoo.org wrote: Ciaran McCreesh wrote: On Tue, 24 Feb 2009 08:08:23 +0100 Uh, your benchmarks are nonsense. Provide your nonsensical ones. You're doubling the number of files that have to be read for an operation that's almost purely i/o bound. On top of that, you're introducing a whole bunch of disk seeks in what's otherwise a nice linear operation. That is not how metadata checks work. Explain how they work, regen works that way... If metadata is valid, ebuilds aren't opened at all. An optimal implementation can slurp up the entire directory in one go and then start pulling out cache entries as it needs them, not having to go back to the ebuild directory or read its contents at all. Then it can open and read cache files in a carefully selected order to avoid having to do any more opens than necessary. By parsing the ebuilds you're talking doubling the number of file reads required to get the job done, and massively increasing the number of seeks required. Apparently it doesn't impact anything. Please show the patch you created (for Paludis, since Portage doesn't yet do a lot of the optimisations it could here) that demonstrates this. But that isn't even the main issue. The main issue is that even if you retroactively pretend that all ebuilds are in a format they're not, and ignore the breakage, and then wait for a year for package managers to try to parse your new format, you *still* can't change name or versioning rules. why? when portage would breanch if I put an ebuild with a wacky version AND there is a valid cache for that telling its eapi 99 ? Because it has to parse that version. Also, the package manager can't tell whether or not a cache entry is valid if it doesn't recognise the EAPI in the cache entry. Again, these are all things that have been discussed at length previously. Please either come up with a legitimate technical objection, or admit that you've seen the light. the glep doesn't show any of those nor reference to it, as I said before, do your homework and probably more people will be happier with your proposals. Why should it? The C++ standard doesn't explain why you should use it instead of Java... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, Feb 24, 2009 at 9:26 PM, Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: ...and it means we can't change name or version rules. And has such a case come to light recently where it was *essential* to do so? Why solve problems that don't exist? ...and it means over doubling the best possible time to work out a dependency tree in the common case where the metadata cache is valid. This is a valid cause. Perhaps the only valid cause. ...and it means we can't make arbitrary format changes. What? Why are we over-engineering this? Does anyone seriously want to convert ebuilds to XML? I honestly think anything beyond incremental changes is not relevant for Gentoo Developers already have to stop and think and consult the conveniently available table of features for EAPIs. By splitting the EAPI concept in two you're doubling the amount of data to be learnt. That's a documentation problem. -- ~Nirbheek Chauhan
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 21:59:39 +0530 Nirbheek Chauhan nirbh...@gentoo.org wrote: On Tue, Feb 24, 2009 at 9:26 PM, Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: ...and it means we can't change name or version rules. And has such a case come to light recently where it was *essential* to do so? Why solve problems that don't exist? Because they do exist, which is why name and version rules have been changed the hard way at least twice previously. The version format is still considerably less flexible than what upstreams use, and a lot of the current limitations on its format are purely historical. ...and it means we can't make arbitrary format changes. What? Why are we over-engineering this? Does anyone seriously want to convert ebuilds to XML? I honestly think anything beyond incremental changes is not relevant for Gentoo You appear to be confusing arbitrary format changes with doing a Zynot. The two are not the same. Developers already have to stop and think and consult the conveniently available table of features for EAPIs. By splitting the EAPI concept in two you're doubling the amount of data to be learnt. That's a documentation problem. No, it's a design problem. Good design looks for ways to minimise the amount of unnecessary arbitrary information the user has to remember. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 12:25:27 -0500 Jim Ramsay l...@gentoo.org wrote: ...and it means we can't change name or version rules. ...and it means over doubling the best possible time to work out a dependency tree in the common case where the metadata cache is valid. ...and it means we can't make arbitrary format changes. Those would all land in the category of backwards compatibility mentioned below, as they would all break current sourcing rules. No, they're also future issues. Without glep 55, every time they come up we have to go through the whole mess again. Developers already have to stop and think and consult the conveniently available table of features for EAPIs. By splitting the EAPI concept in two you're doubling the amount of data to be learnt. I would think that this is a very small cost, and the benefit would be (I hope) that more people would agree on the solution and then we can go forward. Is that not a valid consideration? I'd expect to see changes that would warrant a major bump in every other EAPI or so anyway, so it's not really worth the complexity. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
On Tue, 24 Feb 2009 18:16:54 +0100 Luca Barbato lu_z...@gentoo.org wrote: You're doubling the number of files that have to be read for an operation that's almost purely i/o bound. On top of that, you're introducing a whole bunch of disk seeks in what's otherwise a nice linear operation. I see words, not numbers. Number: double. That's a '2 times'. That is not how metadata checks work. Explain how they work, regen works that way... If metadata is valid, ebuilds aren't opened at all. An optimal implementation can slurp up the entire directory in one go and then start pulling out cache entries as it needs them, not having to go back to the ebuild directory or read its contents at all. Then it can open and read cache files in a carefully selected order to avoid having to do any more opens than necessary. So? if the cache is valid then you don't have to source them at all. If you have to regen it, well you have to read everything. You don't know whether the cache is valid until you know the EAPI. It only works currently because EAPIs don't change inherit behaviour. By parsing the ebuilds you're talking doubling the number of file reads required to get the job done, and massively increasing the number of seeks required. Apparently it doesn't impact anything. Please show the patch you created (for Paludis, since Portage doesn't yet do a lot of the optimisations it could here) that demonstrates this. Paludis isn't portage. So you have patches for Portage? Please show them. But that isn't even the main issue. The main issue is that even if you retroactively pretend that all ebuilds are in a format they're not, and ignore the breakage, and then wait for a year for package managers to try to parse your new format, you *still* can't change name or versioning rules. why? when portage would breanch if I put an ebuild with a wacky version AND there is a valid cache for that telling its eapi 99 ? Because it has to parse that version. Also, the package manager can't tell whether or not a cache entry is valid if it doesn't recognise the EAPI in the cache entry. unknown isn't unsupported? Huh? Please explain what you mean. Again, these are all things that have been discussed at length previously. Please either come up with a legitimate technical objection, or admit that you've seen the light. the glep doesn't show any of those nor reference to it, as I said before, do your homework and probably more people will be happier with your proposals. Why should it? The C++ standard doesn't explain why you should use it instead of Java... In fact many people do wonderful things with java and many more just do over engineered mess with C++? Your trolling is going rapidly downhill. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Luca Barbato wrote:
Ciaran McCreesh wrote:
Because your proposal addresses none of the underlying problems which
GLEP 55 was created to solve.
let's get some numbers to have an idea of the dimension of the problem.
domino portage # wc -l /dev/shm/eapi_files.list
2854 /dev/shm/eapi_files.list
domino portage # ls *-*/*/*.ebuild | wc -l
25761
domino portage # grep -l EAPI eclass/*.eclass | wc -l
22
domino portage # ls eclass/*.eclass | wc -l
240
there aren't eclasses setting EAPI directly.
eapi is set either using EAPI=X or EAPI=X
domino portage # time grep EAPI *-*/*/*.ebuild /dev/shm/eapi_files.list
real0m1.019s
user0m0.608s
sys 0m0.412s
domino portage # time (for a in *-*/*/*.ebuild*; do echo ${A##*.ebuild};
done) /dev/null
real0m0.916s
user0m0.764s
sys 0m0.152s
domino portage # time emerge --regen /dev/shm/regen
real0m9.308s
user0m7.648s
sys 0m1.664s
Restricting eapi so it could surely parsed using something as
complex as grep would and using a two stage parsing would increase to
about 1/9
Using a dumb way to extract the eapi from extension seems to take 1/10
Is there any technical merit in putting eapi in the file extension while
we could restrict the format the same way in file and have about the
same, negligible, performance hit? (I used warm cache since you need the
file anyway so you don't spend time to look it up twice or put it in
cache twice)
Please come up with other numbers or saner implementations to compare.
lu
--
Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero
[gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)
Ciaran McCreesh wrote: Because your proposal addresses none of the underlying problems which GLEP 55 was created to solve. As said long ago the glep doesn't tell enough: The current way of specifying the EAPI in ebuilds is flawed. In order to get the EAPI the package manager needs to source the ebuild, which itself needs the EAPI in the first place. Otherwise it imposes a serious limitation, namely every ebuild, using any of the future EAPIs, will have to be source'able by old package managers and hence there is no way to do any of the following: * Change the behaviour of inherit in any way (for example, to extend or change eclass functionality). * Add new global scope functions in any sane way. * Extend versioning rules in an EAPI - for example, addition of the scm suffix - GLEP54 Let's try to start with a common workflow for the user: - an user with an ancient version of portage syncs - it requires a package - it looks at the cache ($portdir/metadata/cache/) - picks the best entry from the ones showing an eapi it understands - keeps going. Apparently we do not have any issue... Now problems: 1- the cache could get a non compatible change 2- the user triggers a metadata cache regeneration - the ebuild is sourced - portage could fail or do something unpredictable 3- overlays do not provide metadata cache 4- A package manager different from portage do not use the provided cache. Solutions: 1- move the incompatible cache out of ancient portage scope (like in a separate directory) 2- The user will get unpredictable behavior, but portage tell you when upgrading is needed... 3- you'd have to disable them 4- unsupported. Apparently for this side we don't have much to do if we get a valid cache. Ebuilds have to be added to portage so here the workflow for the developer: - new ebuild is sourced - cache is generated - manifest is built In this case we have a problem if the source step is a single one, portage won't know in advance how to behave. So the first step has to be split in two: - first portage discovers which is the eapi version - then behave as defined by the eapi The problem is that right now sourcing is done by having an instructed bash. So the simplest way to get the first step done is parsing the ebuild file with something different like file(1) and then instruct bash and do the parsing. This will solve the issue for the developer. What is proposed in glep-55 seems to aim to solve both issues at the same time (it isn't stated) by switching file extension every time the eapi is changed. This is slightly against the principle of the least surprise and apparently is disliked by enough people to lead the situation to be discussed in the council. The fact the glep itself is too much terse doesn't help acknowledging the problems it aims to solve and the fact it fails to state actual issues that may need a solution doesn't make it worth the effort and disruption it would lead. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
