[Distutils] Re: Timestamp of installed files
On 4 August 2018 at 23:29, Paul Moore wrote: > On Sat, 4 Aug 2018 at 13:31, Jeroen Demeyer wrote: >> For the record: my post wasn't about *building* a wheel, but about >> *installing* a wheel. > > But given that the current behaviour of pip is to copy the timestamps > held in the wheel[1], where those timestamps came from is relevant to > the discussion, surely? Not really. As a simple case where getting file timestamps to change is important, consider this scenario: 1. A dev is hacking on an extension that depends on NumPy while they have version 1.15 installed. 2. They realise they're supposed to be supporting NumPy 1.14+, so they downgrade to the older version 3. They kick off a rebuild of their extension, since they want to recompile against the NumPy 1.14 headers This only works if step 2 changes the timestamp on all the NumPy headers to be *newer* than the previously installed NumPy 1.15 headers. Regardless of how the wheels were generated, it's a pretty safe bet the timestamps in the NumPy 1.15 wheel are going to be newer than the ones in the NumPy 1.14 wheel, so getting the latter to be newer pretty much requires ignoring the wheel, and using the installation time instead. It isn't a disaster if it doesn't (since C/C++ folks are used to needing to forcibly rebuild the world after upgrading or downgrading dependencies), but it's still a nice developer quality of life enhancement if "downgrade + incremental rebuild" just works most of the time. None of this affects reproducible builds, as the whole point of SOURCE_DATE_EPOCH is to allow everything that affects the final output artifacts to *ignore* the modification times on any checked out, generated or unpacked files. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/YNR3ZVJ7X7WFZI667GBFNDG2WUNI6DM2/
[Distutils] Re: Timestamp of installed files
On Sat, 4 Aug 2018 at 13:31, Jeroen Demeyer wrote: > > On 2018-08-04 14:02, Thomas Kluyver wrote: > > On Sat, Aug 4, 2018, at 9:34 AM, Paul Moore wrote: > >> Whether timestamps are > >> preserved by the wheel building process depends on the build system - > >> so the question boils down to "does setup.py bdist_wheel preserve > >> timestamps?" in the case of the setuptools backend - which is really a > >> question for the wheel project. In the more general case, you'd have > >> to ask the same question of flit, and any other backends you cared > >> about. > > > > IIRC, Flit will preserve the timestamps of the files when you build a wheel > > For the record: my post wasn't about *building* a wheel, but about > *installing* a wheel. But given that the current behaviour of pip is to copy the timestamps held in the wheel[1], where those timestamps came from is relevant to the discussion, surely? Paul [1] Or is it? The wheel is expanded into a temporary directory, and then copied from there to the final location. It's only step 2 that has been confirmed to copy timestamps currently. I don't think anyone has checked what the expansion step does. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7POOVHXPW54JGMPO2UOIJT5ANFLXIIZM/
[Distutils] Re: Timestamp of installed files
On 2018-08-04 14:02, Thomas Kluyver wrote: On Sat, Aug 4, 2018, at 9:34 AM, Paul Moore wrote: Whether timestamps are preserved by the wheel building process depends on the build system - so the question boils down to "does setup.py bdist_wheel preserve timestamps?" in the case of the setuptools backend - which is really a question for the wheel project. In the more general case, you'd have to ask the same question of flit, and any other backends you cared about. IIRC, Flit will preserve the timestamps of the files when you build a wheel For the record: my post wasn't about *building* a wheel, but about *installing* a wheel. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SXN2SFT4CTHXR6GZ4WDEJRMHJVJPNTLX/
[Distutils] Re: Timestamp of installed files
On Sat, Aug 4, 2018, at 9:34 AM, Paul Moore wrote: > Whether timestamps are > preserved by the wheel building process depends on the build system - > so the question boils down to "does setup.py bdist_wheel preserve > timestamps?" in the case of the setuptools backend - which is really a > question for the wheel project. In the more general case, you'd have > to ask the same question of flit, and any other backends you cared > about. IIRC, Flit will preserve the timestamps of the files when you build a wheel, unless you use SOURCE_DATE_EPOCH to override them for a reproducible build. I think this is the norm for adding files to a zipfile from a regular filesystem. Thomas -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/ZKBALPPRUCMPXZJHZX7YHCGMFR7XTDDQ/
[Distutils] Re: Timestamp of installed files
On Sat, 4 Aug 2018 at 12:25, Jeroen Demeyer wrote: > > On 2018-08-04 13:16, Paul Moore wrote: > > Can you give a > > specific example of an end to end process where the packaging > > toolchain's current behaviour gives demonstrably the wrong result? > > Yes I can. I will work out a detailed proposal about timestamps in the > build/install process in general. But please give me some time to write > it up. No rush. As I say, I don't have a strong opinion either way, so it's not really me that you need to get agreement from anyway. Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2TEL7LSSXPTEOEC5ZESY3NL7I5WPDJBZ/
[Distutils] Re: Timestamp of installed files
The fact that installs from wheels don't preserve timestamps is a very good argument that it's OK to NOT preserve timestamps in general. Any package which would rely on preserving timestamps would already be broken when installing using a wheel. By the way, initially I thought that this was a distutils issue, but it's clear now that it's really a pip issue. Jeroen. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/75I4HHOCPNGI7HZK2LAZ46IHST3JNTGI/
[Distutils] Re: Timestamp of installed files
On Sat, 4 Aug 2018 at 10:05, Jeroen Demeyer wrote: > > On 2018-08-04 10:34, Paul Moore wrote: > > Jeroen seemed to say he agreed with this, but > > I'm not sure I see how that matches his stated requirement for > > installs to not preserve timestamps... > > The way how pip currently works (I assume that this stays the case) is > that it uses a temporary build directory for everything, for example > /tmp/pip-build-uvfWuR > > So concretely for a wheel install: > > (A) Extract the wheel in build directory > > (B) Generate .pyc files in build directory > > (C) Copy files from build directory to final installation directory in > site-packages. > > The way I understand it, https://github.com/pypa/pip/issues/5648 is > about (A) while my proposal is about (C). So I think that timestamps > should be preserved in (A) but not for (C). Ah, OK. At the moment I believe we preserve in (C) - see https://github.com/pypa/pip/blob/master/src/pip/_internal/wheel.py#L304. The original code here just used shutil.move, but there were changes over time to restrict it to only copying certain metadata (specifically timestamps). But it's always preserved timestamps back to the original implementation in 2012. I still have no opinion on what's "right" here. Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IJRLGMZQZTX2DL2NVECWDVSP2WCM5D6T/
[Distutils] Re: Timestamp of installed files
On 2018-08-04 10:34, Paul Moore wrote: Jeroen seemed to say he agreed with this, but I'm not sure I see how that matches his stated requirement for installs to not preserve timestamps... The way how pip currently works (I assume that this stays the case) is that it uses a temporary build directory for everything, for example /tmp/pip-build-uvfWuR So concretely for a wheel install: (A) Extract the wheel in build directory (B) Generate .pyc files in build directory (C) Copy files from build directory to final installation directory in site-packages. The way I understand it, https://github.com/pypa/pip/issues/5648 is about (A) while my proposal is about (C). So I think that timestamps should be preserved in (A) but not for (C). Jeroen. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/45B2RHBSXNID7GEY2XTSKKSTRAFF43NH/
[Distutils] Re: Timestamp of installed files
On Sat, 4 Aug 2018 at 09:18, Chris Jerdonek wrote: > > On Sat, Aug 4, 2018 at 1:08 AM, Paul Moore wrote: > > On Sat, 4 Aug 2018 at 08:35, Jeroen Demeyer wrote: > > > >> So both are different issues, and I agree with both: during the source > >> extraction and build process, you want to preserve timestamps as much as > >> possible. But for the installation, you do NOT want to preserve timestamps. > > > > If this is about the distutils install command, it's worth noting that > > pip is moving to a situation where we'll never use the > > distutils/setuptools "install" command, bur rather we'll build wheels > > and install from wheel. The new PEP 517 installation code will use > > that route exclusively (there's not even a provision in PEP 517 for > > direct installs from source). So the behaviour of the "setup.py > > install" command won't affect pip installs at all in the longer term. > > (How long, I don't know - it depends on how soon we feel we can switch > > fully to PEP 517, and it's a bit premature to decide on that, as the > > PEP 517 code isn't even released yet!) > > How is the new pip code path slated to behave with respect to Jeroen's > question / preference? Is it already specified or does it fall out of > the behavior of existing components, or is it still TBD? For files extracted from wheels, this is the subject of https://github.com/pypa/pip/issues/5648, as noted earlier in the thread by Jeroen (and the commenters on that issue seem to favour preserving timestamps - Jeroen seemed to say he agreed with this, but I'm not sure I see how that matches his stated requirement for installs to not preserve timestamps...). Whether timestamps are preserved by the wheel building process depends on the build system - so the question boils down to "does setup.py bdist_wheel preserve timestamps?" in the case of the setuptools backend - which is really a question for the wheel project. In the more general case, you'd have to ask the same question of flit, and any other backends you cared about. So: * wheel -> installed: Currently under discussion, with (as far as I can see) conflicting preferences. From the comments in issue 5648 it seems like pip currently alters timestamps, but the commenters there think it would be a "rather uncontroversial patch" (https://github.com/pypa/pip/issues/5648#issuecomment-408439568) to change that * source -> wheel: the backend's responsibility Personally, I have no opinion on the matter. Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/KY66KDCRR72PRK6RFMAK5O6EGVPX4BK4/
[Distutils] Re: Timestamp of installed files
On Sat, Aug 4, 2018 at 1:08 AM, Paul Moore wrote: > On Sat, 4 Aug 2018 at 08:35, Jeroen Demeyer wrote: > >> So both are different issues, and I agree with both: during the source >> extraction and build process, you want to preserve timestamps as much as >> possible. But for the installation, you do NOT want to preserve timestamps. > > If this is about the distutils install command, it's worth noting that > pip is moving to a situation where we'll never use the > distutils/setuptools "install" command, bur rather we'll build wheels > and install from wheel. The new PEP 517 installation code will use > that route exclusively (there's not even a provision in PEP 517 for > direct installs from source). So the behaviour of the "setup.py > install" command won't affect pip installs at all in the longer term. > (How long, I don't know - it depends on how soon we feel we can switch > fully to PEP 517, and it's a bit premature to decide on that, as the > PEP 517 code isn't even released yet!) How is the new pip code path slated to behave with respect to Jeroen's question / preference? Is it already specified or does it fall out of the behavior of existing components, or is it still TBD? --Chris -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/WSCIBRFPXNZRMJJQOYQP6UE5RZXH7BOW/
[Distutils] Re: Timestamp of installed files
On 2018-08-04 00:02, Chris Jerdonek wrote:
I'm not sure how relevant it is, but this issue was recently filed on
pip's issue tracker ("Reproducible installs"):
https://github.com/pypa/pip/issues/5648
This seems to be about preserving timestamps when extracting wheel files.
There is also an older (closed) pip issue that might be relevant
("Preserving timestamps on copy"):
https://github.com/pypa/pip/issues/3201
This is a different issue about preserving timestamps from the source
directory to the temporary build directory.
So both are different issues, and I agree with both: during the source
extraction and build process, you want to preserve timestamps as much as
possible. But for the installation, you do NOT want to preserve timestamps.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/RDXKUK5UQLW33OMCQPK3JPQYXE6ZRZIC/
[Distutils] Re: Timestamp of installed files
I'm not sure how relevant it is, but this issue was recently filed on
pip's issue tracker ("Reproducible installs"):
https://github.com/pypa/pip/issues/5648
Is there any overlap?
There is also an older (closed) pip issue that might be relevant
("Preserving timestamps on copy"):
https://github.com/pypa/pip/issues/3201
--Chris
On Fri, Aug 3, 2018 at 2:14 PM, Jeroen Demeyer wrote:
> Hello,
>
> I would like to draw attention to https://bugs.python.org/issue32773
>
> Currently, distutils (and everything that uses it, such as setuptools and
> pip) installs files with the same timestamp as in the sources (I'm only
> discussing copied files, not generated/compiled files). In other words, it
> preserves the timestamp when it copies the file from source to installation
> directory. There is a comment in the distutils sources
>
> # XXX copy_file by default preserves atime and mtime. IMHO this is
> # the right thing to do, but perhaps it should be an option -- in
> # particular, a site administrator might want installed files to
> # reflect the time of installation rather than the last
> # modification time before the installed release.
>
> It just says "IMHO, this is the right thing to do" without any
> justification.
>
> Now IMHO, preserving timestamps is NOT the right thing to do: the timestamp
> of installed files should be the time of installation. Autotools (which is
> probably the most-used installer for open source/free software projects)
> does that.
>
> The reason why preserving timestamps is bad is because it breaks dependency
> checking: when you install something, you typically want to rebuild other
> things depending on it. For .py files, this isn't relevant (nothing depends
> on them), but it is relevant for .h files and certain Cython files.
>
> So I suggested in bpo-32773 to fix this. The main open question is: to what
> extent could this break backwards compatibility? Personally, I cannot
> imagine a case where it is important that timestamps are preserved.
>
> If it's decided that this should become optional, it should be done on a
> per-project basis (and not by the user installing the package). For example,
> I would want all my projects to not preserve timestamps. Suggestions on how
> to do this (a new argument to setup()?) are welcome.
>
>
> Jeroen
>
>
> PS: if you're interested in the history, this goes back to
>
> commit 13ae1c8ff81befcfd0b0ece98ef471cd504642d8
> Author: Greg Ward
> Date: Mon Mar 22 14:55:25 1999 +
>
> First checkin of real Distutils command modules.
>
> So yes, I want to change a 19-year old "feature".
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/4FHEGHZYWCDRWVPGYLAU5VUS5BAX73MO/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/NY36MB4HBXKURKQKT7ARRI6AP4KKQ445/
