Re: [gentoo-dev] Preserving mtimes for EAPI3
On Tue, 31 Mar 2009, Michael Haubenwallner wrote:
On Mon, 2009-03-30 at 19:14 +0200, Ulrich Mueller wrote:
1. Record two timestamps:
before calling pkg_setup, timestamp1;
after src_install has completed, timestamp2.
2. After src_install and before merging (the exact time would be
implementation dependent), scan ${D} for files having
mtime timestamp1 or mtime timestamp2.
Update their mtimes to timestamp1 or timestamp2, respectively.
3. Otherwise (i.e. for files with timestamp1 = mtime = timestamp2),
preserve modification times when merging ${D} to ${ROOT}.
Or, even shorter: The basic idea is to preserve the mtimes from ${D}
when merging. Except for old mtimes which will be advanced to the
time when the build process began. (And mtimes from the future will
also be fixed, but this case should normally not occur.)
But from discussions in #-dev I conclude that several people would be
happier if just mtimes from ${D} were preserved. In fact, this is the
current Portage and Pkgcore behaviour.
One could think about making step 1 and 2 of above procedure optional
(which was what I had originally in mind, see [1]). Then there would
be no need to change anything in Portage and Pkgcore. Or Portage could
control it via FEATURES, giving the user the choice.
Maybe this could be done somehow like this, with 'merge time'
calculation for each file based on above steps 1.-3.:
* When a to-be-merged file does not exist before, set mtime to
'merge time'.
* When a to-be-merged file does exist already, and its content
does not change, take the mtime from the already existing file.
* When a to-be-merged file does exist already, and its content
does change, set mtime to 'merge time'.
Unfortunately, this will not work. Looking at Lisp again, it may lead
to a fasl file older than the corresponding lisp (imagine that there
could be only a comment change in the source, and the compiled file
would be identical). Also setting mtime to merge time will break
some packages (see bug 83877 [2] for an example).
Maybe this should be done for header files only, or with some black- or
whitelisting mechanisms, or for files which have mtime'build time'?
Too complicated, for my taste.
Ulrich
[1] https://bugs.gentoo.org/show_bug.cgi?id=264130#c12
[2] https://bugs.gentoo.org/show_bug.cgi?id=83877
Re: [gentoo-dev] Preserving mtimes for EAPI3
On Tuesday 31 March 2009 00:59:57 Ulrich Mueller wrote: cut [1] For find -newermt we will need =findutils-4.3.3 which shouldn't be a problem because 4.3.4 went stable in May 2007. No, BSD find does not support it and we don't want to use findutils. Neither busybox find supports it. Make it POSIX[1] please. [1] http://www.opengroup.org/onlinepubs/009695399/utilities/find.html -- Timothy `Drizzt` Redaelli FreeSBIE Developer, Gentoo Developer, GUFI Staff There are two major products that come out of Berkeley: LSD and UNIX. We don't believe this to be a coincidence. -- Jeremy S. Anderson signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Preserving mtimes for EAPI3
On Mon, 2009-03-30 at 19:14 +0200, Ulrich Mueller wrote:
I'll try to summarise the current state of discussion in
https://bugs.gentoo.org/264130. The goal is to satisfy two
(apparently contradictory) requirements:
a) Some packages need a preserved ordering of file modification
times, therwise things will break at run time (see above).
b) Files in the installed system should not have mtimes that are
older than the build time of the package.
Now, is it possible to satisfy both? I think that the following
procedure would accomplish it:
1. Record two timestamps:
before calling pkg_setup, timestamp1;
after src_install has completed, timestamp2.
2. After src_install and before merging (the exact time would be
implementation dependent), scan ${D} for files having
mtime timestamp1 or mtime timestamp2.
Update their mtimes to timestamp1 or timestamp2, respectively.
3. Otherwise (i.e. for files with timestamp1 = mtime = timestamp2),
preserve modification times when merging ${D} to ${ROOT}.
This way, any files generated during the build process (*.pyc, *.fasl,
*.elc) would end up with timestamps newer than their corresponding
source files (*.py, *.lisp, *.el).
Please keep this user-situation in mind, which complicates things:
When developing local applications outside of portage, they often have
autogenerated makefile-dependencies on host-os headerfiles.
Now when a host package gets remerged, and the headerfiles don't change,
all the local applications recompile everything for nothing...
OTOH, when the headerfile changes, it should have mtime updated to
'merge time', because local applications _should_ recompile then.
And using 'build time' is of less use with binary packages, it should be
'merge time' instead.
Maybe this could be done somehow like this, with 'merge time'
calculation for each file based on above steps 1.-3.:
* When a to-be-merged file does not exist before, set mtime to
'merge time'.
* When a to-be-merged file does exist already, and its content
does not change, take the mtime from the already existing file.
* When a to-be-merged file does exist already, and its content
does change, set mtime to 'merge time'.
Maybe this should be done for header files only, or with some black- or
whitelisting mechanisms, or for files which have mtime'build time'?
/haubi/
--
Michael Haubenwallner
Gentoo on a different level
Re: [gentoo-dev] Preserving mtimes for EAPI3
On Mon, 30 Mar 2009 15:40:14 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: No, an EAPI bump is necessary. Older (post-EAPI) Portage versions do something different, so any ebuild relying upon particular behaviour is already broken. For an example of this, see http://bugs.gentoo.org/264308
Re: [gentoo-dev] Preserving mtimes for EAPI3
On Mon, 30 Mar 2009, Peter Alfredsen wrote: No, an EAPI bump is necessary. Older (post-EAPI) Portage versions do something different, so any ebuild relying upon particular behaviour is already broken. For an example of this, see http://bugs.gentoo.org/264308 I would say that is a secondary problem, caused by the package manager not preserving mtimes, and the eclass trying to work around it. Ulrich
Re: [gentoo-dev] Preserving mtimes for EAPI3
On Mon, 30 Mar 2009, Marijn Schouten (hkBst) wrote:
On behalf of the Lisp project (which includes the Emacs subproject)
I'd like to propose that preservation of mtimes be included as a
requirement of EAPI3.
[...]
Background: Dynamic languages such as Common Lisp and Elisp, but
also python (and ruby?) compile source files to some form which
loads and executes faster; in Lisp-speak these are called fasl's
(for FASt Load), for python these are the pyc files. Need for
recompilation is determined by comparing the mtimes of the original
source and the fasl. Both source and fasl are usually installed.
If the mtimes are modified such that the fasl is not newer than
the original source anymore then implementations will attempt to
recompile these sources and will try to write the output alongside
the sources. This will cause a sandbox violation if it happens in
an ebuild or fail due to permissions when done as a user.
I'll try to summarise the current state of discussion in
https://bugs.gentoo.org/264130. The goal is to satisfy two
(apparently contradictory) requirements:
a) Some packages need a preserved ordering of file modification
times, therwise things will break at run time (see above).
b) Files in the installed system should not have mtimes that are
older than the build time of the package.
Currently, Portage and Pkgcore preserve file modification times when
merging, which doesn't always fulfil b). Paludis updates mtimes, which
breaks a).
Now, is it possible to satisfy both? I think that the following
procedure would accomplish it:
1. Record two timestamps:
before calling pkg_setup, timestamp1;
after src_install has completed, timestamp2.
2. After src_install and before merging (the exact time would be
implementation dependent), scan ${D} for files having
mtime timestamp1 or mtime timestamp2.
Update their mtimes to timestamp1 or timestamp2, respectively.
3. Otherwise (i.e. for files with timestamp1 = mtime = timestamp2),
preserve modification times when merging ${D} to ${ROOT}.
This way, any files generated during the build process (*.pyc, *.fasl,
*.elc) would end up with timestamps newer than their corresponding
source files (*.py, *.lisp, *.el).
Problems remain for packages with pre-compiled files, or for packages
requiring exact mtimes (like ghdl). But I think this affects only very
few packages, and it could be fixed on the ebuild level. For example,
a Lisp package with both *.lisp and precompiled *.fasl could touch its
fasl files at the end of src_install.
I am aware of the fact that we are late for EAPI 3 (partly because I
didn't expect that the change would require an EAPI bump). Question to
the council: is it still possible to include this? Considering that
there is a lot of breakage, as well as strange workarounds related to
the current inconsistent behaviour of package managers.
Ulrich
Re: [gentoo-dev] Preserving mtimes for EAPI3
Am Montag, den 30.03.2009, 18:05 +0200 schrieb Peter Alfredsen: On Mon, 30 Mar 2009 15:40:14 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: No, an EAPI bump is necessary. Older (post-EAPI) Portage versions do something different, so any ebuild relying upon particular behaviour is already broken. For an example of this, see http://bugs.gentoo.org/264308 I currently fail to see how mtime preservation is related to this since .py[co] files are generated in pkg_postinst() and therefore not installed by the PM. signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Re: [gentoo-dev] Preserving mtimes for EAPI3
Ulrich Mueller wrote: I am aware of the fact that we are late for EAPI 3 (partly because I didn't expect that the change would require an EAPI bump). Question to the council: is it still possible to include this? Considering that there is a lot of breakage, as well as strange workarounds related to the current inconsistent behaviour of package managers. For most features the block is the need for Portage to implement the feature. If I read the thread correctly, Portage already implements what is wanted here so it's just a matter of agreeing on the specification. I don't see any reason not to have something in EAPI 3 if it's specified and implemented in the same time frame as the main driving features of EAPI 3. Regards, Petteri signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Preserving mtimes for EAPI3
On Mon, 30 Mar 2009, Petteri Räty wrote:
For most features the block is the need for Portage to implement the
feature. If I read the thread correctly, Portage already implements
what is wanted here so it's just a matter of agreeing on the
specification.
Not completely. Portage preserves modification times already when
merging, but if we make updating of old timestamps mandatory (as
Ciaran has suggested), then this part is still missing in Portage.
But as far as I can see, something along the lines of the following
two commands [1] should be all that is needed:
find ${D} -type f \( -newermt @${stamp1} -o -print0 \) \
| ${XARGS} -0 touch -c -d @${stamp1}
find ${D} -type f -newermt @${stamp2} -print0 \
| ${XARGS} -0 touch -c -d @${stamp2}
Variables stamp1 and stamp2 would be assigned from $(date -u +%s)
before pkg_setup and after src_install, respectively.
The second find command is sort of redundant, since it shouldn't
happen that ${D} contains files with timestamps from the future.
Maybe it's better to emit a warning in this case.
Ulrich
[1] For find -newermt we will need =findutils-4.3.3 which shouldn't
be a problem because 4.3.4 went stable in May 2007.
Re: [gentoo-dev] preserving mtimes
On Wed, 07 May 2008 00:44:28 +0200 Marijn Schouten (hkBst) [EMAIL PROTECTED] wrote: and I think that would be the correct thing to do, but either way I'd like PMS to specify what should happen wrt to mtimes, so that I can rely on that. PMS makes no guarantee as to what happens with mtimes, which means you can't rely upon things happening one way or the other. This is deliberate -- preserving mtimes leads to all kinds of weirdness on packages that are generated from a raw tar file rather than from a build system. Current work-around is tarring up and untarring to preserve mtimes. That's not really any good either. The proper solution would be to fix whatever it is that's mtime-sensitive. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] preserving mtimes
On Wed, May 07, 2008 at 01:52:19AM +0100, Ciaran McCreesh wrote: On Wed, 07 May 2008 00:44:28 +0200 Marijn Schouten (hkBst) [EMAIL PROTECTED] wrote: and I think that would be the correct thing to do, but either way I'd like PMS to specify what should happen wrt to mtimes, so that I can rely on that. PMS makes no guarantee as to what happens with mtimes, which means you can't rely upon things happening one way or the other. This is deliberate -- preserving mtimes leads to all kinds of weirdness on packages that are generated from a raw tar file rather than from a build system. I'd be curious what weirdness you're referring to, since pkgcore has preserved mtime from the get go. Yet to see a single issue from it, although *I* could think of a few instances where it would be problematic- that said, those pkgs already do postinst resetting of mtime to work around portage/paludis resetting mtime, so those issues doesn't appear. Current work-around is tarring up and untarring to preserve mtimes. That's not really any good either. The proper solution would be to fix whatever it is that's mtime-sensitive. That's not exactly much of a solution; simplest example, that results in python.eclass:python_mod_compile, invoked during postinst to reset the cached bytecode mtimes (essentially). Aside from this being uncontrolled/untracked access to the live fs, this slows down merges due to redundant work. Finally, it also trashes the chksums that the manager records upon merging to the fs- so an mtime/chksum based unmerger can/will orphan those files. Frankly, the mtime issue keeps rearing its head and needs killing- it's been an issue for near 4 years even, back in the OSX days we had to rewrite .a TOC's since the linker was mtime aware. See no reason to preserve this misfeature. Can't comment on paludis, but it shouldn't be an issue for portage to make the leap from what I've seen source wise. ~harring pgpmpRrDwQqsf.pgp Description: PGP signature
Re: [gentoo-dev] preserving mtimes
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Marijn Schouten (hkBst) wrote:
Hi list,
files installed by portage to ${ROOT} do not have the same time stamps
as the same files
in ${D}.
Any timestamp difference here is not due to portage itself (since
portage-2.1.3). The timestamp change is usually due to the default
values of these variables which are defined in ebuild.sh:
export INSOPTIONS=-m0644
export EXEOPTIONS=-m0755
export LIBOPTIONS=-m0644
export DIROPTIONS=-m0755
It's currently possible for ebuilds to call the insopts, diropts,
exeopts, and libopts functions to modify these variables. If they
add the -p option, then timestamps will be preserved. I suppose we
can add -p to the default options if that's what everybody wants.
Zac
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.9 (GNU/Linux)
iEYEARECAAYFAkghFakACgkQ/ejvha5XGaOLSQCeNOXhp5BY7pIeB/dfQ0lQGkEM
7doAoL9y/VH24DAQ9xDnmV4BlwB2Q5rt
=fW6M
-END PGP SIGNATURE-
--
gentoo-dev@lists.gentoo.org mailing list
