[Distutils] Re: Making setup.py run external command to build
> On 23 Mar 2021, at 19:13, Julian Smith wrote: > > Approach 1 is to pass callbacks to distutils.core.setup() or > setuptools.setup(). However there doesn't appear to be documentation in > either of these modules about what such callbacks should do or how/when > they are called. The only way to figure things out seems to be to look > at the distutils or setuptools source, which i'm finding pretty opaque. [snipped] > As far as i can tell, callbacks are only given information about the > original command line arguments rather than more abstract information > such as where to put output files, so distutils and setuptools don't > seem to be giving any added value here. [snipped] > Given how fundamental the pip tool is to Python packaging, i was hoping > that the command-line arguments that pip passes to setup.py would be > standardised and documented, but this doesn't seem to be the case. What you get from pip is standardised and documented in PEP 517. The part that’s not so is what happens *after* setup tools receives those things, which is entirely setuptools internals. [snipped] > So as far as i can tell, there are two levels of abstraction at which > on can implement customised Python packaging (the setuptools.setup()'s > callbacks or the setup.py command line), but neither one seems to be > documented or standardised. > > Is that right? Or am i missing something fundamental here? You’re basically correct, there are two abstractions in play. The first (between pip and setuptools) is documented. The second (between setuptools and the external command) is not, because that’s setuptools internals and setuptools doesn’t really expect people to mess with them. You still can if you want to, of course, but you’re on your own. So my advice would be to ditch setup tools entirely and work directly with the PEP 517 interface. You don’t need to start from scratch though; there are several PEP 517 implementations other than setuptools, and some of them already implemented some mechanisms to call external build commands, such as enscons[1] (via SCons) and PDM[2] (via setuptools). Some other projects also have already expressed interests supporting this, such as Flit[3], and I think they’d be more than happy to discuss this if you want to work this with them. [1]: https://github.com/dholth/enscons [2]: https://github.com/frostming/pdm [3]: https://github.com/takluyver/flit -- Tzu-ping Chung (@uranusjr) uranu...@gmail.com https://uranusjr.com -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/BDGNJTUU7A36XZIRKE64LA73GJ6LWXFT/
[Distutils] Re: Making setup.py run external command to build
The only interface for the backend (setuptools) talking with the frontend
(pip) is https://www.python.org/dev/peps/pep-0517/#build-backend-interface.
The PEP itself is the documentation. Lack of documentation on how to fully
automate backend parts it's up to the backend. distutils is deprecated and
will be removed from the python standard library in two years. No
development planned on it. If you want distutils/setuptools like
functionality reach out to the setuptools issue tracker (same for
documentation on how to do things - note though this is an open-source
community project, so the answer well might be there's no such at the
moment, but you're free to contribute it).
Bernat,
On Tue, Mar 23, 2021 at 11:31 AM Julian Smith wrote:
> I've recently returned to this issue after a while away.
>
> I've ended up with a custom setup.py that runs the external build
> system as required, and allows pip install/sdist/bdist_wheel to work.
>
>
> However as a result of all this. i have some general questions.
>
> Basically i'm confused about the lack of information that is available
> on how to write a setup.py for packaging a project.
>
> If a project is pure python, or it uses compiled code and it's possible
> to specify how to build it using the swig/compile/link facilities
> provided by distutils or setuptools, then it's easy enough to use
> distutils or setuptools.
>
> However if, like me, one needs to use an external build system, it's
> very difficult to figure out what to do.
>
> There seem to be two approaches:
>
> Approach 1 is to pass callbacks to distutils.core.setup() or
> setuptools.setup(). However there doesn't appear to be documentation in
> either of these modules about what such callbacks should do or how/when
> they are called. The only way to figure things out seems to be to look
> at the distutils or setuptools source, which i'm finding pretty opaque.
>
> As far as i can tell, callbacks are only given information about the
> original command line arguments rather than more abstract information
> such as where to put output files, so distutils and setuptools don't
> seem to be giving any added value here.
>
> Approach 2 is to give up on distutils and setuptools, and instead write
> setup.py from scratch, directly handling the command-line arguments
> from pip.
>
> Given how fundamental the pip tool is to Python packaging, i was hoping
> that the command-line arguments that pip passes to setup.py would be
> standardised and documented, but this doesn't seem to be the case.
>
> One can gather some information by running pip commands and have
> setup.py print out sys.argv, but this is hardly a robust way of doing
> things. For example there's no guarantee that future versions of pip
> won't start using different command-line arguments that setuptools
> already knows about.
>
>
> So as far as i can tell, there are two levels of abstraction at which
> on can implement customised Python packaging (the setuptools.setup()'s
> callbacks or the setup.py command line), but neither one seems to be
> documented or standardised.
>
> Is that right? Or am i missing something fundamental here?
>
>
> Many thanks for any help.
>
> - Jules
>
>
> On Mon, 14 Dec 2020 13:29:28 -0500
> Daniel Holth wrote:
>
> > You would have to have a high tolerance for learning SCons. I'm aware
> that
> > this is not for everyone. Then you could write a SConstruct with
> dependent
> > tasks in a normal build system way. e.g.
> >
> > target = env.Command("a task", ...)
> > platlib = env.Whl("platlib", target, root=".")
> > whl = env.WhlFile(platlib)
> >
> > and have each step re-build only when its sources change.
> >
> > Here is a stackoverflow answer about custom distutils commands.
> > https://stackoverflow.com/questions/1710839/custom-distutils-commands
> >
>
>
>
> --
> http://op59.net
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/77Z7LLYBBW5T4DKCFUTDEG5PM4N5DSIW/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/NCUTRTAHKYOVJDNURNWI7NA7DTWJY6WK/
[Distutils] Re: New packaging security funding & NYU
This is super cool. Congratulations! On Sat, Mar 20, 2021 at 9:00 PM Justin Cappos wrote: > I'm happy to be a part of this and would love to help the PSF get more > support in the future. I'm looking forward to making a positive difference > for Python Packaging through this effort! > > Justin > > On Sat, Mar 20, 2021 at 11:30 AM Sumana Harihareswara > wrote: > >> Good news! >> >> New York University -- specifically Professor Justin Cappos -- and I >> have successfully asked the US National Science Foundation for a grant >> to improve Python packaging security. The NSF is awarding NYU $800,000 >> over two years -- from mid-2021 to mid-2023 -- to further improve the >> pip dependency resolver and to integrate The Update Framework further >> into the packaging toolchain. >> >> >> https://nsf.gov/awardsearch/showAward?AWD_ID=2054692=false >> >> For what we're planning to do, what this means in the short term, an >> explanation of why NYU and the NSF are involved, and thank-yous, please >> see https://discuss.python.org/t/new-packaging-security-funding-nyu/7792 >> . >> >> -- >> Sumana Harihareswara >> Changeset Consulting >> https://changeset.nyc >> -- >> Distutils-SIG mailing list -- distutils-sig@python.org >> To unsubscribe send an email to distutils-sig-le...@python.org >> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ >> Message archived at >> https://mail.python.org/archives/list/distutils-sig@python.org/message/MUH254XTCE5EUL5YJV7ZD6HSUYNFXUD6/ >> > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/3DYZUUJCXQKZFFUTQP4OPQHS7U3V7K7B/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/JIACBC6L6WBVGQMNJLUBGQDI6IYIQ4TZ/
[Distutils] Re: Making setup.py run external command to build
On Tue, 23 Mar 2021, at 11:13, Julian Smith wrote: > So as far as i can tell, there are two levels of abstraction at which > on can implement customised Python packaging (the setuptools.setup()'s > callbacks or the setup.py command line), but neither one seems to be > documented or standardised. That's right. PEP 517 (https://www.python.org/dev/peps/pep-0517/ ) is our effort to have a standardised, documented interface to replace running setup.py. setup.py is now run by setuptools as part of a PEP 517 backend. Thomas -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/4OGHTCTCPM7WDFVPPFOUUSU7B6V3YSKR/
[Distutils] Re: Making setup.py run external command to build
I've recently returned to this issue after a while away.
I've ended up with a custom setup.py that runs the external build
system as required, and allows pip install/sdist/bdist_wheel to work.
However as a result of all this. i have some general questions.
Basically i'm confused about the lack of information that is available
on how to write a setup.py for packaging a project.
If a project is pure python, or it uses compiled code and it's possible
to specify how to build it using the swig/compile/link facilities
provided by distutils or setuptools, then it's easy enough to use
distutils or setuptools.
However if, like me, one needs to use an external build system, it's
very difficult to figure out what to do.
There seem to be two approaches:
Approach 1 is to pass callbacks to distutils.core.setup() or
setuptools.setup(). However there doesn't appear to be documentation in
either of these modules about what such callbacks should do or how/when
they are called. The only way to figure things out seems to be to look
at the distutils or setuptools source, which i'm finding pretty opaque.
As far as i can tell, callbacks are only given information about the
original command line arguments rather than more abstract information
such as where to put output files, so distutils and setuptools don't
seem to be giving any added value here.
Approach 2 is to give up on distutils and setuptools, and instead write
setup.py from scratch, directly handling the command-line arguments
from pip.
Given how fundamental the pip tool is to Python packaging, i was hoping
that the command-line arguments that pip passes to setup.py would be
standardised and documented, but this doesn't seem to be the case.
One can gather some information by running pip commands and have
setup.py print out sys.argv, but this is hardly a robust way of doing
things. For example there's no guarantee that future versions of pip
won't start using different command-line arguments that setuptools
already knows about.
So as far as i can tell, there are two levels of abstraction at which
on can implement customised Python packaging (the setuptools.setup()'s
callbacks or the setup.py command line), but neither one seems to be
documented or standardised.
Is that right? Or am i missing something fundamental here?
Many thanks for any help.
- Jules
On Mon, 14 Dec 2020 13:29:28 -0500
Daniel Holth wrote:
> You would have to have a high tolerance for learning SCons. I'm aware that
> this is not for everyone. Then you could write a SConstruct with dependent
> tasks in a normal build system way. e.g.
>
> target = env.Command("a task", ...)
> platlib = env.Whl("platlib", target, root=".")
> whl = env.WhlFile(platlib)
>
> and have each step re-build only when its sources change.
>
> Here is a stackoverflow answer about custom distutils commands.
> https://stackoverflow.com/questions/1710839/custom-distutils-commands
>
--
http://op59.net
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/77Z7LLYBBW5T4DKCFUTDEG5PM4N5DSIW/
