[Distutils] Re: Timestamp of installed files
On Sat, Aug 4, 2018, at 9:34 AM, Paul Moore wrote: > Whether timestamps are > preserved by the wheel building process depends on the build system - > so the question boils down to "does setup.py bdist_wheel preserve > timestamps?" in the case of the setuptools backend - which is really a > question for the wheel project. In the more general case, you'd have > to ask the same question of flit, and any other backends you cared > about. IIRC, Flit will preserve the timestamps of the files when you build a wheel, unless you use SOURCE_DATE_EPOCH to override them for a reproducible build. I think this is the norm for adding files to a zipfile from a regular filesystem. Thomas -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/ZKBALPPRUCMPXZJHZX7YHCGMFR7XTDDQ/
[Distutils] Re: Timestamp of installed files
On 2018-08-04 14:02, Thomas Kluyver wrote: On Sat, Aug 4, 2018, at 9:34 AM, Paul Moore wrote: Whether timestamps are preserved by the wheel building process depends on the build system - so the question boils down to "does setup.py bdist_wheel preserve timestamps?" in the case of the setuptools backend - which is really a question for the wheel project. In the more general case, you'd have to ask the same question of flit, and any other backends you cared about. IIRC, Flit will preserve the timestamps of the files when you build a wheel For the record: my post wasn't about *building* a wheel, but about *installing* a wheel. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SXN2SFT4CTHXR6GZ4WDEJRMHJVJPNTLX/
[Distutils] Re: Packaging Advice for EFF's Certbot
[Nick wrote:] >>> However, there *are* folks that have been working on allowing >>> applications to be defined primarily as Python projects, and then have >>> the creation of wrapper native installers be a pushbutton exercise, >>> rather than requiring careful human handholding. [Nathaniel wrote:] >> But it sounds like they also want to be able to install/remove/upgrade >> *parts* of the Python project, for their plugin support. [Brad wrote:] > That’s correct. We currently have 18 official plugins for Certbot with plans > to add more and a few dozen third-party plugins. >> Do any of these tools allow that? > This is a good question. If we went with something like dh-virtualenv or > packaged virtualenvs with fpm, would we be able to have separate packages for > our plugins that installed things in the same virtualenv? I haven’t looked > into this yet, but I wouldn’t expect this to work. As long as the base package puts the virtualenv in a pre-defined location, then other packages can add to it (at least in the world of RPM - I'm assuming similar things would be possible with Debian control files). The trick is that the base package then becomes a build dependency in addition to being a runtime dependency, as you'll need it to set up the correct filesystem layout in the build root. I don't think there are any predefined helpers in fpm for that kind of model though, so you'd likely need some additional automation around setting up auto-generation of certbot-plugin packages (whereas certbot itself could likely use the existing virtualenv helpers). However, the other main option is to handle this in a way similar to what Mozilla does for Firefox plugins: use the system package manager to keep the base application up to date, but use application provided tooling to keep the plugins up to date. If you did that, then the plugins wouldn't get native system packages at all - they'd all remain solely as Python packages, and then the system level certbot package would install a privileged cron job to handle venv updates. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Q7PLR5DU46EEI4MOJWFG4B66OXSVKWXU/
[Distutils] Re: Timestamp of installed files
On Sat, 4 Aug 2018 at 13:31, Jeroen Demeyer wrote: > > On 2018-08-04 14:02, Thomas Kluyver wrote: > > On Sat, Aug 4, 2018, at 9:34 AM, Paul Moore wrote: > >> Whether timestamps are > >> preserved by the wheel building process depends on the build system - > >> so the question boils down to "does setup.py bdist_wheel preserve > >> timestamps?" in the case of the setuptools backend - which is really a > >> question for the wheel project. In the more general case, you'd have > >> to ask the same question of flit, and any other backends you cared > >> about. > > > > IIRC, Flit will preserve the timestamps of the files when you build a wheel > > For the record: my post wasn't about *building* a wheel, but about > *installing* a wheel. But given that the current behaviour of pip is to copy the timestamps held in the wheel[1], where those timestamps came from is relevant to the discussion, surely? Paul [1] Or is it? The wheel is expanded into a temporary directory, and then copied from there to the final location. It's only step 2 that has been confirmed to copy timestamps currently. I don't think anyone has checked what the expansion step does. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7POOVHXPW54JGMPO2UOIJT5ANFLXIIZM/
[Distutils] Re: packaging guide and private packages index
On 31 July 2018 at 08:13, Paul Moore wrote: > On 30 July 2018 at 23:01, David Cournapeau wrote: >> Hi, >> >> I see that there is almost no mention of private packages index in the >> packaging guide, and no recommendation on what to use. >> >> Currently googling for private packages mostly return obsolete (and not very >> practical) recommendations based on dependency links. >> >> In 2018, what would be the recommended practices for teams that develop >> private packages that depend on each other ? > > That's a very good point, and I think it would be a really useful > addition to the packaging guide. I believe the most commonly > recommended approach is to use a local,devpi instance as a private > index, but in terms of development workflow around that basis, I don't > really know that there's much in the way of a consensus. (I should say > that I don't work in an environment like this myself, so my comments > are based purely on what I've heard, not on personal experience). Ah, I thought we already had something on this: https://packaging.python.org/guides/index-mirrors-and-caches/ That emphasises the devpi caching proxy use case though, and doesn't point out that devpi is actually a full-fledged private Python package repository manager that's still lightweight enough to run locally as a caching proxy: https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html So even with that page, there's likely value in adding a second guide called something like "Running a private index server" that covers the options from "Just use a simple static file server", through to a Python specific package management engine like devpi, and on to open source multi-format repository management systems like Pulp and the various commercial options (I don't think we should be overly averse to mentioning those - we're mentioning commercial deployment platforms in the packaging overview, and honestly, there are a lot of folks out there where one of the professional lessons they're still learning is "your time as a developer is expensive, so you typically don't want to build what you can already buy"). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/5KSCJ44O43JB4SKIVO5FIQYL2NOIM5TS/
[Distutils] Re: Timestamp of installed files
On Sat, 4 Aug 2018 at 12:25, Jeroen Demeyer wrote: > > On 2018-08-04 13:16, Paul Moore wrote: > > Can you give a > > specific example of an end to end process where the packaging > > toolchain's current behaviour gives demonstrably the wrong result? > > Yes I can. I will work out a detailed proposal about timestamps in the > build/install process in general. But please give me some time to write > it up. No rush. As I say, I don't have a strong opinion either way, so it's not really me that you need to get agreement from anyway. Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2TEL7LSSXPTEOEC5ZESY3NL7I5WPDJBZ/
[Distutils] Re: Timestamp of installed files
On 4 August 2018 at 23:29, Paul Moore wrote: > On Sat, 4 Aug 2018 at 13:31, Jeroen Demeyer wrote: >> For the record: my post wasn't about *building* a wheel, but about >> *installing* a wheel. > > But given that the current behaviour of pip is to copy the timestamps > held in the wheel[1], where those timestamps came from is relevant to > the discussion, surely? Not really. As a simple case where getting file timestamps to change is important, consider this scenario: 1. A dev is hacking on an extension that depends on NumPy while they have version 1.15 installed. 2. They realise they're supposed to be supporting NumPy 1.14+, so they downgrade to the older version 3. They kick off a rebuild of their extension, since they want to recompile against the NumPy 1.14 headers This only works if step 2 changes the timestamp on all the NumPy headers to be *newer* than the previously installed NumPy 1.15 headers. Regardless of how the wheels were generated, it's a pretty safe bet the timestamps in the NumPy 1.15 wheel are going to be newer than the ones in the NumPy 1.14 wheel, so getting the latter to be newer pretty much requires ignoring the wheel, and using the installation time instead. It isn't a disaster if it doesn't (since C/C++ folks are used to needing to forcibly rebuild the world after upgrading or downgrading dependencies), but it's still a nice developer quality of life enhancement if "downgrade + incremental rebuild" just works most of the time. None of this affects reproducible builds, as the whole point of SOURCE_DATE_EPOCH is to allow everything that affects the final output artifacts to *ignore* the modification times on any checked out, generated or unpacked files. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/YNR3ZVJ7X7WFZI667GBFNDG2WUNI6DM2/
[Distutils] Re: Timestamp of installed files
On Sat, 4 Aug 2018 at 10:05, Jeroen Demeyer wrote: > > On 2018-08-04 10:34, Paul Moore wrote: > > Jeroen seemed to say he agreed with this, but > > I'm not sure I see how that matches his stated requirement for > > installs to not preserve timestamps... > > The way how pip currently works (I assume that this stays the case) is > that it uses a temporary build directory for everything, for example > /tmp/pip-build-uvfWuR > > So concretely for a wheel install: > > (A) Extract the wheel in build directory > > (B) Generate .pyc files in build directory > > (C) Copy files from build directory to final installation directory in > site-packages. > > The way I understand it, https://github.com/pypa/pip/issues/5648 is > about (A) while my proposal is about (C). So I think that timestamps > should be preserved in (A) but not for (C). Ah, OK. At the moment I believe we preserve in (C) - see https://github.com/pypa/pip/blob/master/src/pip/_internal/wheel.py#L304. The original code here just used shutil.move, but there were changes over time to restrict it to only copying certain metadata (specifically timestamps). But it's always preserved timestamps back to the original implementation in 2012. I still have no opinion on what's "right" here. Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IJRLGMZQZTX2DL2NVECWDVSP2WCM5D6T/
[Distutils] Re: Timestamp of installed files
The fact that installs from wheels don't preserve timestamps is a very good argument that it's OK to NOT preserve timestamps in general. Any package which would rely on preserving timestamps would already be broken when installing using a wheel. By the way, initially I thought that this was a distutils issue, but it's clear now that it's really a pip issue. Jeroen. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/75I4HHOCPNGI7HZK2LAZ46IHST3JNTGI/
[Distutils] Re: Timestamp of installed files
On 2018-08-04 00:02, Chris Jerdonek wrote:
I'm not sure how relevant it is, but this issue was recently filed on
pip's issue tracker ("Reproducible installs"):
https://github.com/pypa/pip/issues/5648
This seems to be about preserving timestamps when extracting wheel files.
There is also an older (closed) pip issue that might be relevant
("Preserving timestamps on copy"):
https://github.com/pypa/pip/issues/3201
This is a different issue about preserving timestamps from the source
directory to the temporary build directory.
So both are different issues, and I agree with both: during the source
extraction and build process, you want to preserve timestamps as much as
possible. But for the installation, you do NOT want to preserve timestamps.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/RDXKUK5UQLW33OMCQPK3JPQYXE6ZRZIC/
[Distutils] Re: Timestamp of installed files
On Sat, 4 Aug 2018 at 09:18, Chris Jerdonek wrote: > > On Sat, Aug 4, 2018 at 1:08 AM, Paul Moore wrote: > > On Sat, 4 Aug 2018 at 08:35, Jeroen Demeyer wrote: > > > >> So both are different issues, and I agree with both: during the source > >> extraction and build process, you want to preserve timestamps as much as > >> possible. But for the installation, you do NOT want to preserve timestamps. > > > > If this is about the distutils install command, it's worth noting that > > pip is moving to a situation where we'll never use the > > distutils/setuptools "install" command, bur rather we'll build wheels > > and install from wheel. The new PEP 517 installation code will use > > that route exclusively (there's not even a provision in PEP 517 for > > direct installs from source). So the behaviour of the "setup.py > > install" command won't affect pip installs at all in the longer term. > > (How long, I don't know - it depends on how soon we feel we can switch > > fully to PEP 517, and it's a bit premature to decide on that, as the > > PEP 517 code isn't even released yet!) > > How is the new pip code path slated to behave with respect to Jeroen's > question / preference? Is it already specified or does it fall out of > the behavior of existing components, or is it still TBD? For files extracted from wheels, this is the subject of https://github.com/pypa/pip/issues/5648, as noted earlier in the thread by Jeroen (and the commenters on that issue seem to favour preserving timestamps - Jeroen seemed to say he agreed with this, but I'm not sure I see how that matches his stated requirement for installs to not preserve timestamps...). Whether timestamps are preserved by the wheel building process depends on the build system - so the question boils down to "does setup.py bdist_wheel preserve timestamps?" in the case of the setuptools backend - which is really a question for the wheel project. In the more general case, you'd have to ask the same question of flit, and any other backends you cared about. So: * wheel -> installed: Currently under discussion, with (as far as I can see) conflicting preferences. From the comments in issue 5648 it seems like pip currently alters timestamps, but the commenters there think it would be a "rather uncontroversial patch" (https://github.com/pypa/pip/issues/5648#issuecomment-408439568) to change that * source -> wheel: the backend's responsibility Personally, I have no opinion on the matter. Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/KY66KDCRR72PRK6RFMAK5O6EGVPX4BK4/
[Distutils] Re: Timestamp of installed files
On Sat, Aug 4, 2018 at 1:08 AM, Paul Moore wrote: > On Sat, 4 Aug 2018 at 08:35, Jeroen Demeyer wrote: > >> So both are different issues, and I agree with both: during the source >> extraction and build process, you want to preserve timestamps as much as >> possible. But for the installation, you do NOT want to preserve timestamps. > > If this is about the distutils install command, it's worth noting that > pip is moving to a situation where we'll never use the > distutils/setuptools "install" command, bur rather we'll build wheels > and install from wheel. The new PEP 517 installation code will use > that route exclusively (there's not even a provision in PEP 517 for > direct installs from source). So the behaviour of the "setup.py > install" command won't affect pip installs at all in the longer term. > (How long, I don't know - it depends on how soon we feel we can switch > fully to PEP 517, and it's a bit premature to decide on that, as the > PEP 517 code isn't even released yet!) How is the new pip code path slated to behave with respect to Jeroen's question / preference? Is it already specified or does it fall out of the behavior of existing components, or is it still TBD? --Chris -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/WSCIBRFPXNZRMJJQOYQP6UE5RZXH7BOW/
[Distutils] Re: Timestamp of installed files
On 2018-08-04 10:34, Paul Moore wrote: Jeroen seemed to say he agreed with this, but I'm not sure I see how that matches his stated requirement for installs to not preserve timestamps... The way how pip currently works (I assume that this stays the case) is that it uses a temporary build directory for everything, for example /tmp/pip-build-uvfWuR So concretely for a wheel install: (A) Extract the wheel in build directory (B) Generate .pyc files in build directory (C) Copy files from build directory to final installation directory in site-packages. The way I understand it, https://github.com/pypa/pip/issues/5648 is about (A) while my proposal is about (C). So I think that timestamps should be preserved in (A) but not for (C). Jeroen. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/45B2RHBSXNID7GEY2XTSKKSTRAFF43NH/
