[Distutils] Re: Making setup.py run external command to build
On Sun, 28 Mar 2021 18:13:26 +0100 Julian Smith wrote: > On Sat, 27 Mar 2021 00:05:35 + > Julian Smith wrote: > > I eventually managed to local installs from sdist to work with pip-18.1 > by modifying my pyproject.toml, changing: > > [build-system] > requires = [] > > to: > [build-system] > requires = ["wheel"] > > [Using "setup" also works.] > > This seems to make a difference even though my setup.py and pipcl.py > files don't use setuptools. > > I also had to make my setup.py script handle the 'dist_info' command. > setuptools.setup()'s help text doesn't mention this so it's not > immediately clear what it is supposed to do, but it turns out to be > similar to egg_info except for putting files in > /.dist-info/ instead of /.egg-info/. > > > Unfortunately although i can now install from a local sdist with old > and new pip, the 'requires = ["wheel"]' seems to break things when > testing on test.pypi.org - i get: > > requests.exceptions.HTTPError: 404 Client Error: Not Found for url: > https://test.pypi.org/simple/wheel/ > > pip-18.1 then gives up. pip-21.0.1 carries on, running setup.py > directly to do the install of my package. > > I guess i could try uploading to pypi.org, and see whether it can give > the dummy "wheel" package, but i was hoping to be able to get things > working reliably on test.pypi.org first. In the end i decided not to try to support installation with old pip-18.1. So i've now uploaded the project to pypi.org. It's a set of Python bindings for the PDF library MuPDF, and can be installed with: pip install mupdf The install takes a few minutes to compile original source and run SWIG etc. More information is at: https://twiki.ghostscript.com/do/view/Main/MuPDFWrap If anyone is interested, here's the MuPDF commit that adds support for Python packaging, adding a Python module pipcl.py that helps to provide generic PEP-517 support and basic setup.py command-line usage: https://git.ghostscript.com/?p=mupdf.git;a=commitdiff;h=153c67ddf5 Thanks, - Julian -- http://op59.net -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/GQZQNXMXIY4WTSDTHRSIBBFKEACVFTQH/
[Distutils] Re: Making setup.py run external command to build
On Sat, 27 Mar 2021 00:05:35 +
Julian Smith wrote:
> On Fri, 26 Mar 2021 23:19:30 +
> "Thomas Kluyver" wrote:
>
> > On Fri, 26 Mar 2021, at 23:04, Julian Smith wrote:
> > > I can't tell from pip-18.1's diagnostics what exactly is going wrong.
> > >
> > > Given that my setup.py implements the normal distutils-style argv
> > > handling, i was expecting things to work, but pip-18.1 appears to fail
> > > before it even tries to run setup.py, so the only thing i can think of
> > > is that the PKG-INFO file could be causing a problem?
> >
> > Looking at your package, I think the issue is that you have the files
> > (setup.py etc.) in the top level of the tarball, but pip is expecting a
> > - folder in the tarball which contains all the files.
> >
> > The actual format of sdists is something that's not yet properly
> > documented, as far as I know, but you can look at essentially any package
> > to see what I mean in this case. E.g. six:
> >
> > https://files.pythonhosted.org/packages/6b/34/415834bfdafca3c5f451532e8a8d9ba89a21c9743a0c59fbd0205c7f9426/six-1.15.0.tar.gz
> >
>
> Thanks for this. I had high hopes that this would solve things, but
> unfortunately i'm seeing the same behaviour as before.
>
> I've uploaded my new sdist, optest-1.0.20210326.2328.tar.gz, so one can
> attempt to install it with:
>
> pip install -i https://test.pypi.org/simple optest
>
>
> The six-1.15.0.tar.gz file you mention does install with pip-18.1, in
> an interesting way - it fails at first before eventually succeeding by
> running 'setup.py install':
>
> Processing ./six-1.15.0.tar.gz
> Building wheels for collected packages: six
> Running setup.py bdist_wheel for six ... error
> Complete output from command
> /home/jules/artifex-remote/pylocal/bin/python3 -u -c "import
> setuptools,
>
> tokenize;__file__='/tmp/pip-req-build-naagtvzk/setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel
> -d /tmp/pip-wheel-ouokji77 --python-tag cp37: usage: -c
> [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help
> [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help
>
> error: invalid command 'bdist_wheel'
>
>
> Failed building wheel for six
> Running setup.py clean for six
> Failed to build six
> Installing collected packages: six
> Running setup.py install for six ... done
> Successfully installed six-1.15.0
>
>
> This is how i thought my sdist would install, so i guess i could try to
> move my sdist to be similar to six-1.15.0.tar.gz, and things should
> eventually start to work.
I eventually managed to local installs from sdist to work with pip-18.1
by modifying my pyproject.toml, changing:
[build-system]
requires = []
to:
[build-system]
requires = ["wheel"]
[Using "setup" also works.]
This seems to make a difference even though my setup.py and pipcl.py
files don't use setuptools.
I also had to make my setup.py script handle the 'dist_info' command.
setuptools.setup()'s help text doesn't mention this so it's not
immediately clear what it is supposed to do, but it turns out to be
similar to egg_info except for putting files in
/.dist-info/ instead of /.egg-info/.
Unfortunately although i can now install from a local sdist with old
and new pip, the 'requires = ["wheel"]' seems to break things when
testing on test.pypi.org - i get:
requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
https://test.pypi.org/simple/wheel/
pip-18.1 then gives up. pip-21.0.1 carries on, running setup.py
directly to do the install of my package.
I guess i could try uploading to pypi.org, and see whether it can give
the dummy "wheel" package, but i was hoping to be able to get things
working reliably on test.pypi.org first.
Thanks,
- Julian
--
http://op59.net
--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/T2E5QLTMVEYOICV3CJSHTTISPUOWY62A/
[Distutils] Re: Making setup.py run external command to build
On Fri, 26 Mar 2021 23:19:30 +
"Thomas Kluyver" wrote:
> On Fri, 26 Mar 2021, at 23:04, Julian Smith wrote:
> > I can't tell from pip-18.1's diagnostics what exactly is going wrong.
> >
> > Given that my setup.py implements the normal distutils-style argv
> > handling, i was expecting things to work, but pip-18.1 appears to fail
> > before it even tries to run setup.py, so the only thing i can think of
> > is that the PKG-INFO file could be causing a problem?
>
> Looking at your package, I think the issue is that you have the files
> (setup.py etc.) in the top level of the tarball, but pip is expecting a
> - folder in the tarball which contains all the files.
>
> The actual format of sdists is something that's not yet properly documented,
> as far as I know, but you can look at essentially any package to see what I
> mean in this case. E.g. six:
>
> https://files.pythonhosted.org/packages/6b/34/415834bfdafca3c5f451532e8a8d9ba89a21c9743a0c59fbd0205c7f9426/six-1.15.0.tar.gz
Thanks for this. I had high hopes that this would solve things, but
unfortunately i'm seeing the same behaviour as before.
I've uploaded my new sdist, optest-1.0.20210326.2328.tar.gz, so one can
attempt to install it with:
pip install -i https://test.pypi.org/simple optest
The six-1.15.0.tar.gz file you mention does install with pip-18.1, in
an interesting way - it fails at first before eventually succeeding by
running 'setup.py install':
Processing ./six-1.15.0.tar.gz
Building wheels for collected packages: six
Running setup.py bdist_wheel for six ... error
Complete output from command
/home/jules/artifex-remote/pylocal/bin/python3 -u -c "import
setuptools,
tokenize;__file__='/tmp/pip-req-build-naagtvzk/setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n',
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel
-d /tmp/pip-wheel-ouokji77 --python-tag cp37: usage: -c
[global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help
[cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help
error: invalid command 'bdist_wheel'
Failed building wheel for six
Running setup.py clean for six
Failed to build six
Installing collected packages: six
Running setup.py install for six ... done
Successfully installed six-1.15.0
This is how i thought my sdist would install, so i guess i could try to
move my sdist to be similar to six-1.15.0.tar.gz, and things should
eventually start to work.
Thanks,
- Jules
--
http://op59.net
--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/T27FPCHGTF6F6XUI2ZAKSNI76XRTENJU/
[Distutils] Re: Making setup.py run external command to build
On Fri, 26 Mar 2021, at 23:04, Julian Smith wrote: > I can't tell from pip-18.1's diagnostics what exactly is going wrong. > > Given that my setup.py implements the normal distutils-style argv > handling, i was expecting things to work, but pip-18.1 appears to fail > before it even tries to run setup.py, so the only thing i can think of > is that the PKG-INFO file could be causing a problem? Looking at your package, I think the issue is that you have the files (setup.py etc.) in the top level of the tarball, but pip is expecting a - folder in the tarball which contains all the files. The actual format of sdists is something that's not yet properly documented, as far as I know, but you can look at essentially any package to see what I mean in this case. E.g. six: https://files.pythonhosted.org/packages/6b/34/415834bfdafca3c5f451532e8a8d9ba89a21c9743a0c59fbd0205c7f9426/six-1.15.0.tar.gz Thomas -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/W6SDV7GN5AQ3IRCHMUGVHOSFS5WUNUOV/
[Distutils] Re: Making setup.py run external command to build
On Thu, 25 Mar 2021 20:01:12 + "Thomas Kluyver" wrote: > On Thu, 25 Mar 2021, at 19:07, Julian Smith wrote: > > I was a little surprised to find out that one can't use pip to create > > an sdist, but i'm a bit late to this party and it looks like there's > > been plenty of discussion about this in the past. > > There is now the (quite new) 'build' tool which can be used to make > both sdists and wheels by working with a PEP 517 backend: > https://pypi.org/project/build/ Thanks for this, i tried it - see below. > > > I've come across a problem where an old pip-18.1 fails to install from > > an sdist or wheel, with error: > > I'm surprised it fails on the wheels - how exactly are you using pip? Ah, sorry, i was confused, installing from a wheel works. It's just installing from a local or remote sdist that fails. For example with pip-18.1: pip -vvv install -i https://test.pypi.org/simple optest We get: Created temporary directory: /tmp/pip-ephem-wheel-cache-n_a2wzyf Created temporary directory: /tmp/pip-req-tracker-6oichqha Created requirements tracker '/tmp/pip-req-tracker-6oichqha' Created temporary directory: /tmp/pip-install-kep8x9wu Looking in indexes: https://test.pypi.org/simple Collecting optest 1 location(s) to search for versions of optest: * https://test.pypi.org/simple/optest/ Getting page https://test.pypi.org/simple/optest/ Looking up "https://test.pypi.org/simple/optest/"; in the cache Current age based on date: 8 Freshness lifetime from max-age: 600 Freshness lifetime from request max-age: 0 Starting new HTTPS connection (1): test.pypi.org:443 https://test.pypi.org:443 "GET /simple/optest/ HTTP/1.1" 304 0 Analyzing links from page https://test.pypi.org/simple/optest/ Found link https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0fc22562ae63e75de095054400dda7a94dc8/optest-1.0.20210326.2051.tar.gz#sha256=f0a961404d6eba2f59cf75c9bc6e61064e5f191aac4c63f1a7780e77d2156b5d (from https://test.pypi.org/simple/optest/), version: 1.0.20210326.2051 Using version 1.0.20210326.2051 (newest of versions: 1.0.20210326.2051) Created temporary directory: /tmp/pip-unpack-v76_dlvl Looking up "https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0fc22562ae63e75de095054400dda7a94dc8/optest-1.0.20210326.2051.tar.gz"; in the cache Current age based on date: 7 Freshness lifetime from max-age: 0 Starting new HTTPS connection (1): test-files.pythonhosted.org:443 https://test-files.pythonhosted.org:443 "GET /packages/20/53/b75897caba2b7249802439ab0fc22562ae63e75de095054400dda7a94dc8/optest-1.0.20210326.2051.tar.gz HTTP/1.1" 200 6951 Downloading https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0fc22562ae63e75de095054400dda7a94dc8/optest-1.0.20210326.2051.tar.gz Downloading from URL https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0fc22562ae63e75de095054400dda7a94dc8/optest-1.0.20210326.2051.tar.gz#sha256=f0a961404d6eba2f59cf75c9bc6e61064e5f191aac4c63f1a7780e77d2156b5d (from https://test.pypi.org/simple/optest/) Updating cache with response from "https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0fc22562ae63e75de095054400dda7a94dc8/optest-1.0.20210326.2051.tar.gz"; Caching due to etag Added optest from https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0fc22562ae63e75de095054400dda7a94dc8/optest-1.0.20210326.2051.tar.gz#sha256=f0a961404d6eba2f59cf75c9bc6e61064e5f191aac4c63f1a7780e77d2156b5d to build tracker '/tmp/pip-req-tracker-6oichqha' Created temporary directory: /tmp/pip-build-env-yx5wdf16 Installing build dependencies: started Running command /home/jules/pylocal/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yx5wdf16 --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://test.pypi.org/simple -- Created temporary directory: /tmp/pip-ephem-wheel-cache-3hs3xx6m Re-using requirements tracker '/tmp/pip-req-tracker-6oichqha' Created temporary directory: /tmp/pip-install-xif4sjdq Cleaning up... Cleaned build tracker '/tmp/pip-req-tracker-6oichqha' ERROR: You must give at least one requirement to install (see "pip help install") Exception information: Traceback (most recent call last): File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main status = self.run(options, args) File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 313, in run self.name, wheel_cache File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 252, in populate_requirement_set '(see "pip help
[Distutils] Re: Making setup.py run external command to build
On 2021-03-25 20:01:12 + (+), Thomas Kluyver wrote: > I'm surprised it fails on the wheels - how exactly are you using > pip? > > If you're relying on PEP 517, then pip 18.1 is too old to install > it from an sdist - PEP 517 support was added in 19.0 > (https://pip.pypa.io/en/stable/news/#id448 ). But that shouldn't > be a problem for installing from a wheel that has already been > made. Unless you're making ABI3 wheels, which require pip 20.0 or later to find them (and then they'll fall back to pulling the sdist which they can't build because of no pyproject.toml support before 19.0). -- Jeremy Stanley signature.asc Description: PGP signature -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/Y4DS6NLB4GMWSZUQBUJPFSH7ED326DFP/
[Distutils] Re: Making setup.py run external command to build
On 2021-03-25 19:07:50 + (+), Julian Smith wrote: [...] > I've come across a problem where an old pip-18.1 fails to install from > an sdist or wheel, with error: > > ERROR: You must give at least one requirement to install (see "pip help > install") > > pip-20.2.1 and pip-21.0.1 work fine, including installing from an sdist > on test.py.pi. > > Should i be worried about this? I'd like my package to install on > anyone's machine, regardless of their pip version. But maybe users are > expected to always run a recent pip version? After all it's easy enough > to upgrade it. According to https://pip.pypa.io/en/stable/news/#id448 support for pyproject.toml files was first added in pip 19.0 (just over two years ago). If you have users who rely on distribution-packaged pip then this may not be old enough for them to have it yet. For example the current Debian stable release provides pip 18.1, and the next-most-recent Ubuntu LTS (18.04) provides 9.0.1. -- Jeremy Stanley signature.asc Description: PGP signature -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/FEXQGFTNDL5NKTR2UZDV77BVULMFIBU7/
[Distutils] Re: Making setup.py run external command to build
On Thu, 25 Mar 2021, at 19:07, Julian Smith wrote: > I was a little surprised to find out that one can't use pip to create > an sdist, but i'm a bit late to this party and it looks like there's > been plenty of discussion about this in the past. There is now the (quite new) 'build' tool which can be used to make both sdists and wheels by working with a PEP 517 backend: https://pypi.org/project/build/ > I've come across a problem where an old pip-18.1 fails to install from > an sdist or wheel, with error: I'm surprised it fails on the wheels - how exactly are you using pip? If you're relying on PEP 517, then pip 18.1 is too old to install it from an sdist - PEP 517 support was added in 19.0 (https://pip.pypa.io/en/stable/news/#id448 ). But that shouldn't be a problem for installing from a wheel that has already been made. 18.1 is over 2 years outdated, so I think it's reasonable to ask people to upgrade, but you may well still encounter people who have older versions, mostly where they've got it through a Linux distro package manager. Thomas -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/YTRTTRMK56KK4NHB2HGKXM77WXWSVN7W/
[Distutils] Re: Making setup.py run external command to build
On Tue, 23 Mar 2021 12:34:47 + "Thomas Kluyver" wrote: > On Tue, 23 Mar 2021, at 11:13, Julian Smith wrote: > > So as far as i can tell, there are two levels of abstraction at which > > on can implement customised Python packaging (the setuptools.setup()'s > > callbacks or the setup.py command line), but neither one seems to be > > documented or standardised. > > That's right. PEP 517 (https://www.python.org/dev/peps/pep-0517/ ) is our > effort to have a standardised, documented interface to replace running > setup.py. setup.py is now run by setuptools as part of a PEP 517 backend. Many thanks for this, and the other replies from Bernat Gabor and Tzu-ping Chung . I had already come across PEP-517 actually, but hadn't appreciated its importance. I think i've managed to get my packaging to work using PEP-517 callbacks. I was a little surprised to find out that one can't use pip to create an sdist, but i'm a bit late to this party and it looks like there's been plenty of discussion about this in the past. I've ended up with a pyproject.toml that points to a setup.py file which functions both as a PEP-517 module (providing functions build_wheel() and (unused) build_sdist()), and also as a conventional setup.py script for things like './setup.py sdist'. I've come across a problem where an old pip-18.1 fails to install from an sdist or wheel, with error: ERROR: You must give at least one requirement to install (see "pip help install") pip-20.2.1 and pip-21.0.1 work fine, including installing from an sdist on test.py.pi. Should i be worried about this? I'd like my package to install on anyone's machine, regardless of their pip version. But maybe users are expected to always run a recent pip version? After all it's easy enough to upgrade it. Thanks, - Jules -- http://op59.net -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/TSMVUAACJTCCKPWXLHA2P3LQ6GV2UR6R/
[Distutils] Re: Making setup.py run external command to build
> On 23 Mar 2021, at 19:13, Julian Smith wrote: > > Approach 1 is to pass callbacks to distutils.core.setup() or > setuptools.setup(). However there doesn't appear to be documentation in > either of these modules about what such callbacks should do or how/when > they are called. The only way to figure things out seems to be to look > at the distutils or setuptools source, which i'm finding pretty opaque. [snipped] > As far as i can tell, callbacks are only given information about the > original command line arguments rather than more abstract information > such as where to put output files, so distutils and setuptools don't > seem to be giving any added value here. [snipped] > Given how fundamental the pip tool is to Python packaging, i was hoping > that the command-line arguments that pip passes to setup.py would be > standardised and documented, but this doesn't seem to be the case. What you get from pip is standardised and documented in PEP 517. The part that’s not so is what happens *after* setup tools receives those things, which is entirely setuptools internals. [snipped] > So as far as i can tell, there are two levels of abstraction at which > on can implement customised Python packaging (the setuptools.setup()'s > callbacks or the setup.py command line), but neither one seems to be > documented or standardised. > > Is that right? Or am i missing something fundamental here? You’re basically correct, there are two abstractions in play. The first (between pip and setuptools) is documented. The second (between setuptools and the external command) is not, because that’s setuptools internals and setuptools doesn’t really expect people to mess with them. You still can if you want to, of course, but you’re on your own. So my advice would be to ditch setup tools entirely and work directly with the PEP 517 interface. You don’t need to start from scratch though; there are several PEP 517 implementations other than setuptools, and some of them already implemented some mechanisms to call external build commands, such as enscons[1] (via SCons) and PDM[2] (via setuptools). Some other projects also have already expressed interests supporting this, such as Flit[3], and I think they’d be more than happy to discuss this if you want to work this with them. [1]: https://github.com/dholth/enscons [2]: https://github.com/frostming/pdm [3]: https://github.com/takluyver/flit -- Tzu-ping Chung (@uranusjr) [email protected] https://uranusjr.com -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BDGNJTUU7A36XZIRKE64LA73GJ6LWXFT/
[Distutils] Re: Making setup.py run external command to build
The only interface for the backend (setuptools) talking with the frontend
(pip) is https://www.python.org/dev/peps/pep-0517/#build-backend-interface.
The PEP itself is the documentation. Lack of documentation on how to fully
automate backend parts it's up to the backend. distutils is deprecated and
will be removed from the python standard library in two years. No
development planned on it. If you want distutils/setuptools like
functionality reach out to the setuptools issue tracker (same for
documentation on how to do things - note though this is an open-source
community project, so the answer well might be there's no such at the
moment, but you're free to contribute it).
Bernat,
On Tue, Mar 23, 2021 at 11:31 AM Julian Smith wrote:
> I've recently returned to this issue after a while away.
>
> I've ended up with a custom setup.py that runs the external build
> system as required, and allows pip install/sdist/bdist_wheel to work.
>
>
> However as a result of all this. i have some general questions.
>
> Basically i'm confused about the lack of information that is available
> on how to write a setup.py for packaging a project.
>
> If a project is pure python, or it uses compiled code and it's possible
> to specify how to build it using the swig/compile/link facilities
> provided by distutils or setuptools, then it's easy enough to use
> distutils or setuptools.
>
> However if, like me, one needs to use an external build system, it's
> very difficult to figure out what to do.
>
> There seem to be two approaches:
>
> Approach 1 is to pass callbacks to distutils.core.setup() or
> setuptools.setup(). However there doesn't appear to be documentation in
> either of these modules about what such callbacks should do or how/when
> they are called. The only way to figure things out seems to be to look
> at the distutils or setuptools source, which i'm finding pretty opaque.
>
> As far as i can tell, callbacks are only given information about the
> original command line arguments rather than more abstract information
> such as where to put output files, so distutils and setuptools don't
> seem to be giving any added value here.
>
> Approach 2 is to give up on distutils and setuptools, and instead write
> setup.py from scratch, directly handling the command-line arguments
> from pip.
>
> Given how fundamental the pip tool is to Python packaging, i was hoping
> that the command-line arguments that pip passes to setup.py would be
> standardised and documented, but this doesn't seem to be the case.
>
> One can gather some information by running pip commands and have
> setup.py print out sys.argv, but this is hardly a robust way of doing
> things. For example there's no guarantee that future versions of pip
> won't start using different command-line arguments that setuptools
> already knows about.
>
>
> So as far as i can tell, there are two levels of abstraction at which
> on can implement customised Python packaging (the setuptools.setup()'s
> callbacks or the setup.py command line), but neither one seems to be
> documented or standardised.
>
> Is that right? Or am i missing something fundamental here?
>
>
> Many thanks for any help.
>
> - Jules
>
>
> On Mon, 14 Dec 2020 13:29:28 -0500
> Daniel Holth wrote:
>
> > You would have to have a high tolerance for learning SCons. I'm aware
> that
> > this is not for everyone. Then you could write a SConstruct with
> dependent
> > tasks in a normal build system way. e.g.
> >
> > target = env.Command("a task", ...)
> > platlib = env.Whl("platlib", target, root=".")
> > whl = env.WhlFile(platlib)
> >
> > and have each step re-build only when its sources change.
> >
> > Here is a stackoverflow answer about custom distutils commands.
> > https://stackoverflow.com/questions/1710839/custom-distutils-commands
> >
>
>
>
> --
> http://op59.net
> --
> Distutils-SIG mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/77Z7LLYBBW5T4DKCFUTDEG5PM4N5DSIW/
>
--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/NCUTRTAHKYOVJDNURNWI7NA7DTWJY6WK/
[Distutils] Re: Making setup.py run external command to build
On Tue, 23 Mar 2021, at 11:13, Julian Smith wrote: > So as far as i can tell, there are two levels of abstraction at which > on can implement customised Python packaging (the setuptools.setup()'s > callbacks or the setup.py command line), but neither one seems to be > documented or standardised. That's right. PEP 517 (https://www.python.org/dev/peps/pep-0517/ ) is our effort to have a standardised, documented interface to replace running setup.py. setup.py is now run by setuptools as part of a PEP 517 backend. Thomas -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/4OGHTCTCPM7WDFVPPFOUUSU7B6V3YSKR/
[Distutils] Re: Making setup.py run external command to build
I've recently returned to this issue after a while away.
I've ended up with a custom setup.py that runs the external build
system as required, and allows pip install/sdist/bdist_wheel to work.
However as a result of all this. i have some general questions.
Basically i'm confused about the lack of information that is available
on how to write a setup.py for packaging a project.
If a project is pure python, or it uses compiled code and it's possible
to specify how to build it using the swig/compile/link facilities
provided by distutils or setuptools, then it's easy enough to use
distutils or setuptools.
However if, like me, one needs to use an external build system, it's
very difficult to figure out what to do.
There seem to be two approaches:
Approach 1 is to pass callbacks to distutils.core.setup() or
setuptools.setup(). However there doesn't appear to be documentation in
either of these modules about what such callbacks should do or how/when
they are called. The only way to figure things out seems to be to look
at the distutils or setuptools source, which i'm finding pretty opaque.
As far as i can tell, callbacks are only given information about the
original command line arguments rather than more abstract information
such as where to put output files, so distutils and setuptools don't
seem to be giving any added value here.
Approach 2 is to give up on distutils and setuptools, and instead write
setup.py from scratch, directly handling the command-line arguments
from pip.
Given how fundamental the pip tool is to Python packaging, i was hoping
that the command-line arguments that pip passes to setup.py would be
standardised and documented, but this doesn't seem to be the case.
One can gather some information by running pip commands and have
setup.py print out sys.argv, but this is hardly a robust way of doing
things. For example there's no guarantee that future versions of pip
won't start using different command-line arguments that setuptools
already knows about.
So as far as i can tell, there are two levels of abstraction at which
on can implement customised Python packaging (the setuptools.setup()'s
callbacks or the setup.py command line), but neither one seems to be
documented or standardised.
Is that right? Or am i missing something fundamental here?
Many thanks for any help.
- Jules
On Mon, 14 Dec 2020 13:29:28 -0500
Daniel Holth wrote:
> You would have to have a high tolerance for learning SCons. I'm aware that
> this is not for everyone. Then you could write a SConstruct with dependent
> tasks in a normal build system way. e.g.
>
> target = env.Command("a task", ...)
> platlib = env.Whl("platlib", target, root=".")
> whl = env.WhlFile(platlib)
>
> and have each step re-build only when its sources change.
>
> Here is a stackoverflow answer about custom distutils commands.
> https://stackoverflow.com/questions/1710839/custom-distutils-commands
>
--
http://op59.net
--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/77Z7LLYBBW5T4DKCFUTDEG5PM4N5DSIW/
[Distutils] Re: Making setup.py run external command to build
I've had some luck with this sort of thing: https://stromberg.dnsalias.org/svn/binary-tree-mod/trunk/setup.py It just os.system's make, and then assumes everything is built thenceforth. On Fri, Dec 11, 2020 at 7:10 AM Julian Smith wrote: > Hello > > I have a project with a fairly involved build process where we generate > C++ via a python-clang-based code analyser, generates Python bindings > for the resulting C++ API with SWIG, and finally compile and link to > create various .so's and .py files. > > I'd like to package things up with a setup.py script, but from what i > have read, it looks like distutils expects to compile, link and run > swig itself, using distutils.core.setup()'s ext_modules arg? > > Is there a way to write setup.py so that it instead runs an external > command (e.g. 'cd foo && make all') when it is asked to build? > > Thanks for any help, > > - Jules > > -- > http://op59.net > > -- > Distutils-SIG mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/AJDJU5DYMDBPGZ75P35M5647O5VH22FL/ > -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/JP6SAGJU5CGZCUQROPLH3DZJDZIG6IQH/
[Distutils] Re: Making setup.py run external command to build
You would have to have a high tolerance for learning SCons. I'm aware that
this is not for everyone. Then you could write a SConstruct with dependent
tasks in a normal build system way. e.g.
target = env.Command("a task", ...)
platlib = env.Whl("platlib", target, root=".")
whl = env.WhlFile(platlib)
and have each step re-build only when its sources change.
Here is a stackoverflow answer about custom distutils commands.
https://stackoverflow.com/questions/1710839/custom-distutils-commands
--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/QMFJZ4O6XFUE3FLRSH5J4BBJT436F43L/
[Distutils] Re: Making setup.py run external command to build
On Mon, 14 Dec 2020 10:25:41 -0500 Daniel Holth wrote: > enscons is an alternative to distutils. > Yes, i understand this. But enscons is for building python packages with scons. My project does not use scons, so i don't understand how it can help here. Unless... are you are suggesting that i write a new alternative to distutils, based on enscons, which builds by running a external commands ? Thanks, - Jules -- http://op59.net -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/ERCRH6KB3OKBBKM35ZHK5ELJVV3JICB2/
[Distutils] Re: Making setup.py run external command to build
enscons is an alternative to distutils. -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/EJQKJHKKWD77YMPTSZFJBYUFJNKTQFZX/
[Distutils] Re: Making setup.py run external command to build
On Fri, 11 Dec 2020 10:32:13 -0500 Daniel Holth wrote: > I've been working on ensconced which lets you use a SConstruct to > build. It is easier than customizing distutils. > > https://github.com/dholth/pysdl2-cffi/blob/master/SConstruct is the > most code-generation-y project. Its build might be a little out of > date and it has an independent setup.py building the project with > setuptools. > > https://github.com/dholth/bioopenssl doesn't have C extensions but is > an example of a stub setup.py (only necessary for the develop > command) and pyproject.toml (which pip can use without a setup.py via > PEP 517). Thanks for this, but i'm struggling to get from what those two projects are doing, to learning how to get pip/distutils to run a 'make' command? Thanks, - Jules > > On Fri, Dec 11, 2020 at 10:09 AM Julian Smith wrote: > > > Hello > > > > I have a project with a fairly involved build process where we > > generate C++ via a python-clang-based code analyser, generates > > Python bindings for the resulting C++ API with SWIG, and finally > > compile and link to create various .so's and .py files. > > > > I'd like to package things up with a setup.py script, but from what > > i have read, it looks like distutils expects to compile, link and > > run swig itself, using distutils.core.setup()'s ext_modules arg? > > > > Is there a way to write setup.py so that it instead runs an external > > command (e.g. 'cd foo && make all') when it is asked to build? > > > > Thanks for any help, > > > > - Jules > > > > -- > > http://op59.net > > > > -- > > Distutils-SIG mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > > Message archived at > > https://mail.python.org/archives/list/[email protected]/message/AJDJU5DYMDBPGZ75P35M5647O5VH22FL/ > > > -- http://op59.net -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/HJ4XJ2BFI3K4MPN5XINM6Z5RJFTLINXG/
[Distutils] Re: Making setup.py run external command to build
I've been working on ensconced which lets you use a SConstruct to build. It is easier than customizing distutils. https://github.com/dholth/pysdl2-cffi/blob/master/SConstruct is the most code-generation-y project. Its build might be a little out of date and it has an independent setup.py building the project with setuptools. https://github.com/dholth/bioopenssl doesn't have C extensions but is an example of a stub setup.py (only necessary for the develop command) and pyproject.toml (which pip can use without a setup.py via PEP 517). On Fri, Dec 11, 2020 at 10:09 AM Julian Smith wrote: > Hello > > I have a project with a fairly involved build process where we generate > C++ via a python-clang-based code analyser, generates Python bindings > for the resulting C++ API with SWIG, and finally compile and link to > create various .so's and .py files. > > I'd like to package things up with a setup.py script, but from what i > have read, it looks like distutils expects to compile, link and run > swig itself, using distutils.core.setup()'s ext_modules arg? > > Is there a way to write setup.py so that it instead runs an external > command (e.g. 'cd foo && make all') when it is asked to build? > > Thanks for any help, > > - Jules > > -- > http://op59.net > > -- > Distutils-SIG mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/AJDJU5DYMDBPGZ75P35M5647O5VH22FL/ > -- Distutils-SIG mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/FBC7LEB3WUK2XTOKYXKQMO3J7WA47SNE/
