[gentoo-dev] QA last rites for www-apps/Embperl
# Diego E. Pettenò flamee...@gentoo.org (25 Nov 2009) # on behalf of QA team # # Fails to build, bug #248474, open since November 2008. # # Removal on 2010-01-24 www-apps/Embperl
Re: [gentoo-dev] autoconf-2.64 moved to unstable
Mike Frysinger wrote: just a heads up. most packages are fixed now, so time to get wider testing. -mike Thanks! It's about time. :) Also note to everyone: If KDE3 based application is failing because of this, realistically don't expect it to be fixed by nothing else than committing a Qt4/KDE4 replacement or version bump to tree.
[gentoo-dev] QA last rites for www-client/rabbitticker
# Diego E. Pettenò flamee...@gentoo.org (25 Nov 2009) # on behalf of QA team # # Fails to build (bug #276675, July 2009); imported in 2004, # never bumped, no maintainer, upstream homepage unavailable. # # Removal on 2010-01-24 www-client/rabbitticker
[gentoo-dev] Re: mtime preservation
A quick note to tell you that I have tried to look for examples of breakage due to how mtime preservation is currently implemented in portage. Unfortunately I didn't find anything, maybe because I'm not knowledgeable enough or because I can't afford to spend any more time on this. If any of you can provide an example then please do so. In case nobody could show any sign of such breakage I would have to add to the list of two propositions to vote on: 3- Do nothing. We can always point to code that shows the implementation is suboptimal but we have to understand that some of us who are less proactive at fixing issues may want to procrastinate until breakage actually happens. Denis.
[gentoo-dev] Next council meeting on 7 Dec 2009 at 1900UTC
The next council meeting will be on 7 Dec 2009 at 1900UTC. If you want us to discuss things please let us know in reply to this email. What is already known is we'll talk about mtime preservation and prefix. You can find threads about those at: http://archives.gentoo.org/gentoo-dev/msg_a9e26414f2278275bdfa08baf839704f.xml http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml Denis.
[gentoo-dev] Re: mtime preservation
Ciaran McCreesh posted on Wed, 25 Nov 2009 21:27:18 + as excerpted: Portage uses two ways of merging a file: os.rename() and the manual way. os.rename() correctly preserves mtimes. Python's os.utime call, which is what Portage uses to preserve mtimes for files that it installs the hard way, and which is not a wrapper for the Unix utime call, uses an IEEE 754 double to handle timestamps. But a double only gives sixteen accurate decimal digits, and you've got ten to the left of the decimal point, leaving only six reliably preserved, with the remaining three digits being corrupted. Thus, packages can end up being installed with some files having accurately-preserved mtimes, and some having corrupted mtimes, which will lead to mtime comparisons giving incorrect results. That's a great explanation (thanks, I now know the details to the degree I'd be interested), but what was asked for was examples of breakage, aka actual bugs. I seem to remember either you or someone else mentioning that there had been a couple issues already, and I'll take the claim at face value, but actual bug numbers would be nice. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] Re: mtime preservation
On Wed, 25 Nov 2009 21:52:00 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: That's a great explanation (thanks, I now know the details to the degree I'd be interested), but what was asked for was examples of breakage, aka actual bugs. Why? You can easily look and see that it's broken. Examples will merely be dismissed as one-off cases that can be worked around, or as relying upon a string of coincidences that will obviously never really happen, right up until they do, at which point they'll be dismissed with a WORKSFORME. What you have is a proof that it's broken, which is far better than an example. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds
Things seem to be progressing nicely on this front. We have answers to the questions people had and they look satisfactory to me. One thing that I think would be valuable is a document that explains the average dev how to make his/her ebuilds prefix compliant with links to more details when necessary. I understand that there's the trivial situations and the less trivial ones. In the latter case it would be nice to explain why the case isn't trivial and how to fix it. Using python as an example could be one way to do it. I'm thinking of something practical that could possibly be patched into devmanual. If such a document already exists then please just point us to it. On Fri, Nov 20, 2009 at 2:03 AM, Fabian Groffen grob...@gentoo.org wrote: I thought I asked Fabian to work on that at the end of the meeting. In case I didn't then consider this as me officially asking him if he can take care of it. Fabian is this OK with you? Yes, I agreed coming up with some patch. I admit I haven't yet even looked into it. Great, thanks. If you can have it ready some time before the meeting so that all devs can get a chance to review it before the council members vote on it that will be even better. If you need help don't hesitate to contact me. I'll try and look for the right people to help you depending on what you need. Technically, Portage trunk already contains the most important bits that allow us to continue since yesterday. The rest is waiting for a formal approval of the council, and then it will most probably be me and Zac fighting to get the prefix branch merged into trunk. Good. Next to that, it is part of the Prefix team's job to make sure that whatever is installed, does not reference the host system when this is not absolutely necessary. Could you give some examples of when it is absolutely necessary? Denis.
Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds
It looks like this question is still unanswered: On Thu, Nov 19, 2009 at 5:26 PM, Denis Dupeyron calc...@gentoo.org wrote: How are dynamically linked set*id programs going to work? Denis.
Re: [gentoo-dev] mtime preservation
On Wed, 25 Nov 2009, Ciaran McCreesh wrote: On Wed, 25 Nov 2009 21:52:00 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: That's a great explanation (thanks, I now know the details to the degree I'd be interested), but what was asked for was examples of breakage, aka actual bugs. Why? You can easily look and see that it's broken. Only for a suitable definition of broken. Examples will merely be dismissed as one-off cases that can be worked around, or as relying upon a string of coincidences that will obviously never really happen, right up until they do, at which point they'll be dismissed with a WORKSFORME. Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing that could be easily dismissed or worked around. Both issues are fixed with Portage since a long time. I don't know of any example where non-preservation of nanosecond timestamps would cause problems. What you have is a proof that it's broken, which is far better than an example. So we have a proven theorem, but unfortunately the cases where it is applicable form an empty set. ;-) Ulrich [1] http://bugs.gentoo.org/83877#c36 [2] http://bugs.gentoo.org/263387
Re: [gentoo-dev] mtime preservation
On Wed, 25 Nov 2009 23:59:45 +0100 Ulrich Mueller u...@gentoo.org wrote: Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing that could be easily dismissed or worked around. Both issues are fixed with Portage since a long time. Yes, those are examples of packages relying upon something that is undefined behaviour, and that behaves differently depending upon the Portage version you use. I don't know of any example where non-preservation of nanosecond timestamps would cause problems. Not non-preservation. Partial and inconsistent corruption. srcfile gets installed with its mtime preserved, because it is installed using rename. objfile, whose mtime should be a bit later than srctime's, gets installed with its mtime corrupted to be slightly less than it should be, and less than srctime's, because it is installed the slow way. A program checks that objfile's mtime is greater than or equal to srcfile's mtime. That check will fail, and reinstalls will randomly fix and break it depending upon which way the corruption goes. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] mtime preservation
Ciaran McCreesh wrote: On Wed, 25 Nov 2009 23:59:45 +0100 Ulrich Mueller u...@gentoo.org wrote: Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing that could be easily dismissed or worked around. Both issues are fixed with Portage since a long time. Yes, those are examples of packages relying upon something that is undefined behaviour, and that behaves differently depending upon the Portage version you use. I don't know of any example where non-preservation of nanosecond timestamps would cause problems. Not non-preservation. Partial and inconsistent corruption. Wouldn't loss of precision be a more accurate description? Of the known packages which require timestamp preservation, do any of them use sub-second precision in their timestamp comparisons? -- Thanks, Zac
Re: [gentoo-dev] mtime preservation
On Wed, Nov 25, 2009 at 04:49:17PM -0800, Zac Medico wrote: Ciaran McCreesh wrote: On Wed, 25 Nov 2009 23:59:45 +0100 Ulrich Mueller u...@gentoo.org wrote: Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing that could be easily dismissed or worked around. Both issues are fixed with Portage since a long time. Yes, those are examples of packages relying upon something that is undefined behaviour, and that behaves differently depending upon the Portage version you use. I don't know of any example where non-preservation of nanosecond timestamps would cause problems. Not non-preservation. Partial and inconsistent corruption. Wouldn't loss of precision be a more accurate description? Of the known packages which require timestamp preservation, do any of them use sub-second precision in their timestamp comparisons? This discussion in generall is daft. No package can rely on nanonsecond resolution for installation because the most common FS out there (ext3) does *second* level resolution only. As such, I can pretty much gurantee there is *zero* packages out there that require nanosecond resolution for installation. It's an academic discussion, and pointless. We don't mandate the filesystems PMS implementations are run on- as such we cannot make a gurantee to ebuilds that nanosecond resolution is available. It's daft to encode in the spec NS resolution when it's essentially impossible to gurantee it- I'm honestly not sure why we're having this discussion beyond the python sucks angle. ~brian pgp89cUgiTHJ1.pgp Description: PGP signature
Re: [gentoo-dev] mtime preservation
On Tue, Nov 24, 2009 at 10:21:06PM +, Ciaran McCreesh wrote:
On Mon, 23 Nov 2009 15:19:00 -0800
Brian Harring ferri...@gmail.com wrote:
Someone mind explaining to me why we're making mtime preservation so
nasty? Having to enumerate every pathway that requires mtime
preservation is pretty arduous for the ebuild dev, meaning it's
unlikely they'll get it right, leading to bugs.
It's not in the least bit nasty. It's requiring people to be explicit
about special requirements.
I honestly wish that explicitness you're pushing for mtime were
applied to all parts of mtime.
Why is this one special? Two out of three do this already, and it
works. Paludis doesn't preserve mtime- and it's approach to randomly
resetting mtimes is known to cause issues (bug 263387, python eclass
having to do pyc/pyo compilation in postinst, etc).
The thing I'm not understanding here is that pkgcore since day one
has preserved mtime- I've yet to see any complaints about that nor
any issues caused by it. Portage shifted over a year or two back,
same thing, haven't heard complaints.
You can't have been listening very hard, then. The complaint is that it
results in files with dumb mtimes being merged to /.
As far as I can tell, no one but you is complaining about the mtimes.
Your complaints primarily have been that tarball'd mtimes can pass
through to the fs- I've not seen any comments from you in bug 264130
that actually enumerate spots were this is anything but an aesthetic
complaint. Specifically your aesthetics- the pkgs in question have
functioned fine w/ odd mtimes passing through.
I know it won't fly, but realistically stating the package manager
must preserve mtime- if there are instances where it does not (due to
some feature, perhaps splitdebug or some form of compression) it is
the package managers responsibility to ensure this does not break the
ebuilds resultant merge/runtime invocation.
Via such wording an exemption is created and it's made clear it's the
managers responsibility to keep things playing nice... if the manager
can't do that, then the feature/functionality that is changing the
mtime (resulting in the pkg on disk failing) must be fixed or
disabled.
The nice thing about this wording is that it basically matches what
the case is now, and what has worked for quite a few years.
Wording such as that just isn't suitable for a specification. It
requires implementers to guess what future ebuilds are going to
rely upon (and ebuilds regularly do rely upon weird quirks), and makes
it impossible to produce a package manager that can be shown to be
compliant.
That wording is explicit that PMS cannot lock down all potentials.
This is no different then PMS's stance on VDB (and similar in
reasoning)- ebuilds are reliant on vdb internals yet PMS explicitly
ducked documenting it. Double standards suck.
Further, drop the specificity BS. You've intentionally left parts of
PMS vague- trailing '/' for ebuild vars is a good example (ssmtp
breaks if ${D} vs ${D}/, perl likely has the same breakage). These
lead to actual incompatibilities, and because PMS isn't explicit,
there is no 'right' implementation for it.
The daft thing about you picking at this wording is that the scenarios
you can come up with are all hypothetical. Yes, a manager could
randomize mtimes when it's doing splitdebug.
The manager wouldn't however because the only reason to do things like
that is to intentionally cause issues.
~brian
pgpIgEdcLNmnw.pgp
Description: PGP signature
Re: [gentoo-dev] Next council meeting on 7 Dec 2009 at 1900UTC
On Wed, Nov 25, 2009 at 02:50:22PM -0700, Denis Dupeyron wrote: The next council meeting will be on 7 Dec 2009 at 1900UTC. If you want us to discuss things please let us know in reply to this email. What is already known is we'll talk about mtime preservation and prefix. You can find threads about those at: http://archives.gentoo.org/gentoo-dev/msg_a9e26414f2278275bdfa08baf839704f.xml http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml I'd like http://archives.gentoo.org/gentoo-dev/msg_6b3e00049a1bf35fbf7a5e66d1449553.xml to be discussed, specifically zacs form of forced mtime updating of /var/db/pkg on vdb modifications Reiterating the reasoning, it'll be used for enabling the managers to manage caches, specifically invalidating said caches if an alternative manager modifies the vdb. End result, if we can build better caches for vdb, vdb access can be heavily sped up. At this point, pkgcore/portage both have it implemented. Not sure if portage has released it yet, but =pkgcore-0.5.2 is in the tree w/ said updating support. ~harring pgpFs1M8KJwVB.pgp Description: PGP signature
Re: [gentoo-dev] Next council meeting on 7 Dec 2009 at 1900UTC
Brian Harring wrote: At this point, pkgcore/portage both have it implemented. Not sure if portage has released it yet, but =pkgcore-0.5.2 is in the tree w/ said updating support. It's in =portage-2.1.7.2 (and 2.1.7.x should be going stable in a month or so). -- Thanks, Zac
[gentoo-dev] Re: mtime preservation
Ciaran McCreesh posted on Wed, 25 Nov 2009 22:13:27 + as excerpted: Examples will merely be dismissed as one-off cases that can be worked around, or as relying upon a string of coincidences that will obviously never really happen, right up until they do, at which point they'll be dismissed with a WORKSFORME. What you have is a proof that it's broken, which is far better than an example. Actually, that dismissed with WORKSFORME strikes a chord, here. There was a very strange parallel make bug that I filed that was closed with that. I'd have really liked to see someone with some skill tackle it, as that was the only one I've ever seen that had striped fail and working zones, and I've have loved to see some logic as to why... (If -j10 failed, -j3 and -j15 might succeed, -l24 fail again, and -j33 succeed again...) Unfortunately, flameeyes, the only one I know who really gets into such things, was fresh out of the hospital at the time, and I think it was beyond the maintainer's abilities, so WORKSFORME was about the best that could be done. I've long since changed and changed again my makeopts, and don't remember the pkg now, tho I could probably find it in my old bug mail if I needed to. So I gotta admit you have a point, with that one. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: mtime preservation
Ulrich Mueller posted on Wed, 25 Nov 2009 23:59:45 +0100 as excerpted: Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing that could be easily dismissed or worked around. Both issues are fixed with Portage since a long time. Thanks. I don't know of any example where non-preservation of nanosecond timestamps would cause problems. That's really what I was hoping for, a bug where the nanosecond times thing made a difference. But Ciaran does have a point about such things being potentially closed as WORKSFORME or the like, as I've seen that in other cases, see my just earlier post. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] mtime preservation
Brian Harring wrote: This discussion in generall is daft. No package can rely on nanonsecond resolution for installation because the most common FS out there (ext3) does *second* level resolution only. As such, I can pretty much gurantee there is *zero* packages out there that require nanosecond resolution for installation. Your guarantee is filesystem-specific. However, if we can establish that all known packages with timestamp preservation requirements do their timestamp comparisons with 1-second granularity, then we'll have a much safer (filesystem-independent) assumption. -- Thanks, Zac
[gentoo-dev] Re: mtime preservation
Brian Harring posted on Wed, 25 Nov 2009 17:14:27 -0800 as excerpted: On Wed, Nov 25, 2009 at 04:49:17PM -0800, Zac Medico wrote: Ciaran McCreesh wrote: On Wed, 25 Nov 2009 23:59:45 +0100 Ulrich Mueller u...@gentoo.org wrote: Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing that could be easily dismissed or worked around. Both issues are fixed with Portage since a long time. Yes, those are examples of packages relying upon something that is undefined behaviour, and that behaves differently depending upon the Portage version you use. I don't know of any example where non-preservation of nanosecond timestamps would cause problems. Not non-preservation. Partial and inconsistent corruption. Wouldn't loss of precision be a more accurate description? Of the known packages which require timestamp preservation, do any of them use sub-second precision in their timestamp comparisons? This discussion in generall is daft. No package can rely on nanonsecond resolution for installation because the most common FS out there (ext3) does *second* level resolution only. As such, I can pretty much gurantee there is *zero* packages out there that require nanosecond resolution for installation. One of the reasons I was asking for a bug number, was because there's two possible failure modes, and I was hoping reading them would clue me (or at least those who are making the decisions here) in on which one actually occurs. Your posts suggest a mode where subsecond times are always truncated. In such a case, there should be no issue. But it's also possible that times are compared as-is. If that's the case, then there should be no issue on second-resolution filesystems (such as ext3, in your example), because the filesystem would effectively truncate those, reducing to case-one, above. But on higher resolution filesystems, there might very well be issues, due to the subtle double resolution issue Ciaran pointed out, especially when compared against renames where the mtimes were accurately preserved. Now it could well be that it's possible to turn case two into case one by adding a simple option to the the command line or the like. Or maybe not. Again, that's why I wanted specific examples, so people could see if that were tried or not, and I would have to write all this up and possibly look like a moron if I'm getting it wrong, somehow. But since the examples don't seem to be forthcoming, and in view of the IMO legitimate point about such bugs very possibly being closed as WORKSFORME, I guess I post this and hope someone can either explain that I have it all wrong, or say definitively that such command-line time truncation option workarounds are generally practical, or not, thus potentially affecting the direction of the debate. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: mtime preservation
Zac Medico posted on Wed, 25 Nov 2009 21:26:59 -0800 as excerpted: Brian Harring wrote: This discussion in generall is daft. No package can rely on nanonsecond resolution for installation because the most common FS out there (ext3) does *second* level resolution only. As such, I can pretty much gurantee there is *zero* packages out there that require nanosecond resolution for installation. Your guarantee is filesystem-specific. However, if we can establish that all known packages with timestamp preservation requirements do their timestamp comparisons with 1-second granularity, then we'll have a much safer (filesystem-independent) assumption. Thanks. That's basically what I'm wondering also, well, if they do it with one second granularity, or if it's possible to make them do it that with with a simple command-line sed, adding an option, but here it's stated in different (fewer) words. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] mtime preservation
On Wed, Nov 25, 2009 at 09:26:59PM -0800, Zac Medico wrote: Brian Harring wrote: This discussion in generall is daft. No package can rely on nanonsecond resolution for installation because the most common FS out there (ext3) does *second* level resolution only. As such, I can pretty much gurantee there is *zero* packages out there that require nanosecond resolution for installation. Your guarantee is filesystem-specific. However, if we can establish that all known packages with timestamp preservation requirements do their timestamp comparisons with 1-second granularity, then we'll have a much safer (filesystem-independent) assumption. I've no complaints with mandating that ebuilds can rely on second level resolution- it's a valid gurantee as far as I'm concerned. Further any fs that can't offer it involves the user doing something seriously wonky, thus their problem if the ebuild horks. If/when the major filesystems out there all do NS level resolution, and are in common deployment, I'd have no arguement extending the spec to mandating NS level resolution. I've serious problems w/ mandating NS resolution in PMS prior to that however. ~harring pgpaRgCDR4if6.pgp Description: PGP signature
Re: [gentoo-dev] mtime preservation
On Thu, 26 Nov 2009, Ciaran McCreesh wrote: Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing that could be easily dismissed or worked around. Both issues are fixed with Portage since a long time. Yes, those are examples of packages relying upon something that is undefined behaviour, That was the reason why I filed bug 264130, in the first place. ;-) and that behaves differently depending upon the Portage version you use. Fact is that all versions of Portage in the tree get it right for both packages, but Paludis fails for them. objfile, whose mtime should be a bit later than srctime's, gets installed with its mtime corrupted to be slightly less than it should be, and less than srctime's, because it is installed the slow way. The major usage cases where mtimes play a role (*.py vs *.pyc, *.lisp vs *.fasl, and *.el vs *.elc) install source and object files in the same directory. So why would the PM use different methods for installing them? A program checks that objfile's mtime is greater than or equal to srcfile's mtime. That check will fail, and reinstalls will randomly fix and break it depending upon which way the corruption goes. Again, you don't come up with a concrete example where such breakage would occur. I'm starting to get tired of this whole discussion. Maybe we should just follow Calchan's suggestion and do nothing. Portage works, after all. Ulrich
