[Distutils] Re: Depending on external library
On 2019-01-22 13:04, Robert T. McGibbon wrote: It's necessary to bundle the external library with the wheel. On linux, this is what the manylinux tag is all about (see PEP 513 & 571) and the auditwheel tool (https://github.com/pypa/auditwheel) will bundle the library into the wheel. On OS X and Windows, I'm less familiar with the toolchain but the same principle applies. Thanks for your reply. Unfortunately as you say, it only applies to Linux. So how do projects in practice deal with this in a cross-platform way? Does every package need to reinvent the "wheel"? -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/ZKIXSIHGQVREKRKD6WR6NVMSHBBM5YID/
[Distutils] Depending on external library
Hello, I have a Python project that depends on an external C library (which is unrelated to Python and NOT part of the OS). For an sdist, this is easy: my setup.py assumes that the library is pre-installed somewhere on the system where setuptools can find it. However, is there a standard solution for packaging such a project as wheel? Ideally, the project should "just work" when doing pip install on it, which means that the external library should somehow be bundled in the wheel. I know that for example numpy does that, but it also has a very complicated build system (containing a fork of distutils!). Does anybody know any pointers for this? Or if you think that this is a minefield which is not really supported, feel free to say so. Thanks, Jeroen. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/6RZVKM5YQVAAOR5ENDS34HH5VFXHZ22G/
[Distutils] Re: disable building wheel for a package
On 2018-09-14 16:39, Paul Moore wrote: My understanding (and I'm not an expert here, so hopefully someone else will confirm or correct) is that yes, the data directory is installed to "the installation root", which is $VIRTUAL_ENV for a virtualenv, and "something else" for non-virtualenvs (I think it's / on Unix and sys.prefix on Windows, no idea what happens for user installs). More precisely: it's always sys.prefix for non-user installs (which is "/usr" not "/" on my system). For user installs, it's site.USER_BASE which is "/home/jdemeyer/.local" for me. I think that this works great, I see nothing to fix here (except being more explicit that absolute paths don't work). Jeroen. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/PRYNKSX2ZSP3NVFRLSBXC7ETQ4PSOEXZ/
[Distutils] Re: disable building wheel for a package
On 2018-09-14 12:55, Alex Grönholm wrote: I'm curious: what data does it attempt to install and where? Have you created a ticket for this somewhere? The OP mentioned absolute paths. However, it really sounds like a bad idea to hard-code an absolute installation path. Let's consider it a feature that wheel doesn't support that. See https://github.com/pypa/wheel/issues/92 -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/DSYIJDTAHDA7O25YMWPJULPTYI6UN542/
[Distutils] Re: Proposal for timestamps of installed files
On 2018-08-06 19:49, Chris Jerdonek wrote: Does this proposal mean that .pyc files would need to be regenerated after installing from a wheel (because the timestamp inside the .pyc file would no longer match the .py file from which it came)? No, I specifically mentioned that the timestamps of .py files would be preserved. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/4LMQ3ONLBMKU72TRRRT4YYCN2JFAJ4YP/
[Distutils] Proposal for timestamps of installed files
Proposal: change timestamps of files installed from a wheel to the time of installation, except for .py files. **Existing situation** Let's consider two ways of installing a Python package: installation from a source tarball and installation from a wheel, in both cases using "pip install". The installation roughly works as follows: (A) Create a temporary directory for pip to work in, with a filename like /tmp/pip-6N_2Kx-build (B) Extract the archive (sources or wheel) somewhere in the temporary directory (C) Build the package in the temporary directory (in both cases, this involves creating .pyc files for the .py files) (D) Copy the installation from the temporary directory to the final installation directory (typically somewhere in site-packages) (E) Delete the temporary directory Both (B) and (D) involve a copy operation of some sort, so the question is: what to do with timestamps? In both cases, (D) currently preserves the timestamp of the files. For (B), timestamps are preserved when extracting a tarball but not when extracting a wheel. **Proposal** Change step (D) for wheel installs to NOT preserve timestamps. It might be safer to make sure that all installed files have the same timestamp, so this would mean: give all installed files the timestamp of a fixed time, namely the time when step (D) was started in the installation process. Important exception: for .py files, do preserve the timestamp. I think that (B) should preserve timestamps. However, that is a different issue which should be fixed independently. **The problem** For a source installation, the timestamps are preserved in both (B) and (D), so the timestamp of the installed files can be much older than the time of installation. This is a problem for dependency checking: suppose that I am the developer of a package MYPKG which has a dependency on the C API of a package OTHERPKG. The C API of OTHERPKG is defined by .h files which are installed by OTHERPKG. The package MYPKG correctly declares this dependency using the "depends" keyword of the Extension objects. Now, when I install a different version of OTHERPKG, I want that distutils notices that the dependency has been changed and automatically rebuilds the Extensions with that dependency. This requires that the timestamp of the installed .h files is more recent than the timestamp of the compiled extension. In other words, the timestamp of installed files should be the time of installation. A similar situation happens for a Cython API exposed by installed .pxd (and possibly also .h) files. **Comparison with automake** Automake (which is probably the most widely used build system for open source/free software) installs files with timestamp the time of installation. So, my proposal is doing what automake already does. **The strange case of .pyc files** As noted in https://github.com/pypa/pip/issues/5648 it is important that the timestamp of .py files are preserved after the .pyc file is created. This is because Python checks the exact timestamp of the .py file to determine whether it is up to date. This explains the exception for .py files in the proposal. **Backwards compatibility** In other words: won't this break packages which rely on timestamps to be preserved? When installing a wheel, timestamps from the wheel are already not preserved in step (B). So any package which would rely on preserving timestamps would already be broken when installed as wheel. So this proposal can only break stuff which depends on timestamps from step (C), which happens to be the case for .py files. In my proposal, I'm only suggesting to make changes for wheel installs. I do think that the change makes sense for installs from source, but that will be irrelevant after PEP 517 is implemented anyway. Jeroen. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SOBBI2YA7TVEUI3EVXJX2Z4EDJOU4WYY/
[Distutils] Re: Timestamp of installed files
On 2018-08-04 14:02, Thomas Kluyver wrote: On Sat, Aug 4, 2018, at 9:34 AM, Paul Moore wrote: Whether timestamps are preserved by the wheel building process depends on the build system - so the question boils down to "does setup.py bdist_wheel preserve timestamps?" in the case of the setuptools backend - which is really a question for the wheel project. In the more general case, you'd have to ask the same question of flit, and any other backends you cared about. IIRC, Flit will preserve the timestamps of the files when you build a wheel For the record: my post wasn't about *building* a wheel, but about *installing* a wheel. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SXN2SFT4CTHXR6GZ4WDEJRMHJVJPNTLX/
[Distutils] Re: Timestamp of installed files
The fact that installs from wheels don't preserve timestamps is a very good argument that it's OK to NOT preserve timestamps in general. Any package which would rely on preserving timestamps would already be broken when installing using a wheel. By the way, initially I thought that this was a distutils issue, but it's clear now that it's really a pip issue. Jeroen. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/75I4HHOCPNGI7HZK2LAZ46IHST3JNTGI/
[Distutils] Re: Timestamp of installed files
On 2018-08-04 10:34, Paul Moore wrote: Jeroen seemed to say he agreed with this, but I'm not sure I see how that matches his stated requirement for installs to not preserve timestamps... The way how pip currently works (I assume that this stays the case) is that it uses a temporary build directory for everything, for example /tmp/pip-build-uvfWuR So concretely for a wheel install: (A) Extract the wheel in build directory (B) Generate .pyc files in build directory (C) Copy files from build directory to final installation directory in site-packages. The way I understand it, https://github.com/pypa/pip/issues/5648 is about (A) while my proposal is about (C). So I think that timestamps should be preserved in (A) but not for (C). Jeroen. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/45B2RHBSXNID7GEY2XTSKKSTRAFF43NH/
[Distutils] Re: Timestamp of installed files
On 2018-08-04 00:02, Chris Jerdonek wrote:
I'm not sure how relevant it is, but this issue was recently filed on
pip's issue tracker ("Reproducible installs"):
https://github.com/pypa/pip/issues/5648
This seems to be about preserving timestamps when extracting wheel files.
There is also an older (closed) pip issue that might be relevant
("Preserving timestamps on copy"):
https://github.com/pypa/pip/issues/3201
This is a different issue about preserving timestamps from the source
directory to the temporary build directory.
So both are different issues, and I agree with both: during the source
extraction and build process, you want to preserve timestamps as much as
possible. But for the installation, you do NOT want to preserve timestamps.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/RDXKUK5UQLW33OMCQPK3JPQYXE6ZRZIC/
[Distutils] Timestamp of installed files
Hello, I would like to draw attention to https://bugs.python.org/issue32773 Currently, distutils (and everything that uses it, such as setuptools and pip) installs files with the same timestamp as in the sources (I'm only discussing copied files, not generated/compiled files). In other words, it preserves the timestamp when it copies the file from source to installation directory. There is a comment in the distutils sources # XXX copy_file by default preserves atime and mtime. IMHO this is # the right thing to do, but perhaps it should be an option -- in # particular, a site administrator might want installed files to # reflect the time of installation rather than the last # modification time before the installed release. It just says "IMHO, this is the right thing to do" without any justification. Now IMHO, preserving timestamps is NOT the right thing to do: the timestamp of installed files should be the time of installation. Autotools (which is probably the most-used installer for open source/free software projects) does that. The reason why preserving timestamps is bad is because it breaks dependency checking: when you install something, you typically want to rebuild other things depending on it. For .py files, this isn't relevant (nothing depends on them), but it is relevant for .h files and certain Cython files. So I suggested in bpo-32773 to fix this. The main open question is: to what extent could this break backwards compatibility? Personally, I cannot imagine a case where it is important that timestamps are preserved. If it's decided that this should become optional, it should be done on a per-project basis (and not by the user installing the package). For example, I would want all my projects to not preserve timestamps. Suggestions on how to do this (a new argument to setup()?) are welcome. Jeroen PS: if you're interested in the history, this goes back to commit 13ae1c8ff81befcfd0b0ece98ef471cd504642d8 Author: Greg Ward Date: Mon Mar 22 14:55:25 1999 + First checkin of real Distutils command modules. So yes, I want to change a 19-year old "feature". -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/4FHEGHZYWCDRWVPGYLAU5VUS5BAX73MO/
