[gentoo-dev] Re: mtime preservation
David Leverton posted on Thu, 26 Nov 2009 12:35:53 + as excerpted: If we're not going to insist on preserving nanoseconds as far as possible, then package managers should be required to explcitly clear the nanoseconds part. While I'm not sure what it's going to do to install-times and thus whether it's practical, this /does/ at least in theory make quite some sense to me. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] Re: mtime preservation
Duncan wrote: David Leverton posted on Thu, 26 Nov 2009 12:35:53 + as excerpted: If we're not going to insist on preserving nanoseconds as far as possible, then package managers should be required to explcitly clear the nanoseconds part. While I'm not sure what it's going to do to install-times and thus whether it's practical, this /does/ at least in theory make quite some sense to me. This seems like a pretty reasonable (and easy) thing to do. All that we have to do is round the mtime value before it's passed into the os.utime() call. -- Thanks, Zac
[gentoo-dev] Re: mtime preservation
A quick note to tell you that I have tried to look for examples of breakage due to how mtime preservation is currently implemented in portage. Unfortunately I didn't find anything, maybe because I'm not knowledgeable enough or because I can't afford to spend any more time on this. If any of you can provide an example then please do so. In case nobody could show any sign of such breakage I would have to add to the list of two propositions to vote on: 3- Do nothing. We can always point to code that shows the implementation is suboptimal but we have to understand that some of us who are less proactive at fixing issues may want to procrastinate until breakage actually happens. Denis.
[gentoo-dev] Re: mtime preservation
Ciaran McCreesh posted on Wed, 25 Nov 2009 21:27:18 + as excerpted: Portage uses two ways of merging a file: os.rename() and the manual way. os.rename() correctly preserves mtimes. Python's os.utime call, which is what Portage uses to preserve mtimes for files that it installs the hard way, and which is not a wrapper for the Unix utime call, uses an IEEE 754 double to handle timestamps. But a double only gives sixteen accurate decimal digits, and you've got ten to the left of the decimal point, leaving only six reliably preserved, with the remaining three digits being corrupted. Thus, packages can end up being installed with some files having accurately-preserved mtimes, and some having corrupted mtimes, which will lead to mtime comparisons giving incorrect results. That's a great explanation (thanks, I now know the details to the degree I'd be interested), but what was asked for was examples of breakage, aka actual bugs. I seem to remember either you or someone else mentioning that there had been a couple issues already, and I'll take the claim at face value, but actual bug numbers would be nice. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] Re: mtime preservation
On Wed, 25 Nov 2009 21:52:00 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: That's a great explanation (thanks, I now know the details to the degree I'd be interested), but what was asked for was examples of breakage, aka actual bugs. Why? You can easily look and see that it's broken. Examples will merely be dismissed as one-off cases that can be worked around, or as relying upon a string of coincidences that will obviously never really happen, right up until they do, at which point they'll be dismissed with a WORKSFORME. What you have is a proof that it's broken, which is far better than an example. -- Ciaran McCreesh signature.asc Description: PGP signature
[gentoo-dev] Re: mtime preservation
Ciaran McCreesh posted on Wed, 25 Nov 2009 22:13:27 + as excerpted: Examples will merely be dismissed as one-off cases that can be worked around, or as relying upon a string of coincidences that will obviously never really happen, right up until they do, at which point they'll be dismissed with a WORKSFORME. What you have is a proof that it's broken, which is far better than an example. Actually, that dismissed with WORKSFORME strikes a chord, here. There was a very strange parallel make bug that I filed that was closed with that. I'd have really liked to see someone with some skill tackle it, as that was the only one I've ever seen that had striped fail and working zones, and I've have loved to see some logic as to why... (If -j10 failed, -j3 and -j15 might succeed, -l24 fail again, and -j33 succeed again...) Unfortunately, flameeyes, the only one I know who really gets into such things, was fresh out of the hospital at the time, and I think it was beyond the maintainer's abilities, so WORKSFORME was about the best that could be done. I've long since changed and changed again my makeopts, and don't remember the pkg now, tho I could probably find it in my old bug mail if I needed to. So I gotta admit you have a point, with that one. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: mtime preservation
Ulrich Mueller posted on Wed, 25 Nov 2009 23:59:45 +0100 as excerpted: Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing that could be easily dismissed or worked around. Both issues are fixed with Portage since a long time. Thanks. I don't know of any example where non-preservation of nanosecond timestamps would cause problems. That's really what I was hoping for, a bug where the nanosecond times thing made a difference. But Ciaran does have a point about such things being potentially closed as WORKSFORME or the like, as I've seen that in other cases, see my just earlier post. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: mtime preservation
Brian Harring posted on Wed, 25 Nov 2009 17:14:27 -0800 as excerpted: On Wed, Nov 25, 2009 at 04:49:17PM -0800, Zac Medico wrote: Ciaran McCreesh wrote: On Wed, 25 Nov 2009 23:59:45 +0100 Ulrich Mueller u...@gentoo.org wrote: Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing that could be easily dismissed or worked around. Both issues are fixed with Portage since a long time. Yes, those are examples of packages relying upon something that is undefined behaviour, and that behaves differently depending upon the Portage version you use. I don't know of any example where non-preservation of nanosecond timestamps would cause problems. Not non-preservation. Partial and inconsistent corruption. Wouldn't loss of precision be a more accurate description? Of the known packages which require timestamp preservation, do any of them use sub-second precision in their timestamp comparisons? This discussion in generall is daft. No package can rely on nanonsecond resolution for installation because the most common FS out there (ext3) does *second* level resolution only. As such, I can pretty much gurantee there is *zero* packages out there that require nanosecond resolution for installation. One of the reasons I was asking for a bug number, was because there's two possible failure modes, and I was hoping reading them would clue me (or at least those who are making the decisions here) in on which one actually occurs. Your posts suggest a mode where subsecond times are always truncated. In such a case, there should be no issue. But it's also possible that times are compared as-is. If that's the case, then there should be no issue on second-resolution filesystems (such as ext3, in your example), because the filesystem would effectively truncate those, reducing to case-one, above. But on higher resolution filesystems, there might very well be issues, due to the subtle double resolution issue Ciaran pointed out, especially when compared against renames where the mtimes were accurately preserved. Now it could well be that it's possible to turn case two into case one by adding a simple option to the the command line or the like. Or maybe not. Again, that's why I wanted specific examples, so people could see if that were tried or not, and I would have to write all this up and possibly look like a moron if I'm getting it wrong, somehow. But since the examples don't seem to be forthcoming, and in view of the IMO legitimate point about such bugs very possibly being closed as WORKSFORME, I guess I post this and hope someone can either explain that I have it all wrong, or say definitively that such command-line time truncation option workarounds are generally practical, or not, thus potentially affecting the direction of the debate. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: mtime preservation
Zac Medico posted on Wed, 25 Nov 2009 21:26:59 -0800 as excerpted: Brian Harring wrote: This discussion in generall is daft. No package can rely on nanonsecond resolution for installation because the most common FS out there (ext3) does *second* level resolution only. As such, I can pretty much gurantee there is *zero* packages out there that require nanosecond resolution for installation. Your guarantee is filesystem-specific. However, if we can establish that all known packages with timestamp preservation requirements do their timestamp comparisons with 1-second granularity, then we'll have a much safer (filesystem-independent) assumption. Thanks. That's basically what I'm wondering also, well, if they do it with one second granularity, or if it's possible to make them do it that with with a simple command-line sed, adding an option, but here it's stated in different (fewer) words. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: mtime preservation
Brian Harring posted on Mon, 23 Nov 2009 15:19:00 -0800 as excerpted: Alternatively, we could simply make portage spawn the mv binary whenever rename fails (it fails when the source and destination are on different devices). Although it's relatively slow, it should solve the problem. Yeah... no. Slowing down the main manager for a thereotical edge case doesn't seem particularly useful to me ;) I say go for second resolution now, basically using the #1 proposal, and apply it to all eapis for the reasons zac outlined, but with a fixed effective date say 90 or 180 days out from the council resolution to give PMs time to come into compliance. To try for nanosecond resolution now is simply letting the complex perfect be the enemy of the simple good, when what we have at present is the no-good. Then make a donsmtimes like the #2 proposal, that individual ebuilds can call for specific paths if it's really necessary. It's more work for the ebuilder, but that's in relative measure to the slowdown it'll cause on two out of three PMs including our core PM, so a bit of discouragement from using it could be considered a good thing, tho it would be there, if required. If at some point in the future ns resolution becomes a much bigger issue and many ebuilds are using the donsmtimes call, there's nothing stopping us from looking at further tightening up the requirements for the general case, probably as part of an eapi, then. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
