On maandag 10 september 2018 12:47:16 CEST Paul Moore wrote:
> On Mon, 10 Sep 2018 at 10:13, Melvyn Sopacua <m.r.sopa...@gmail.com> wrote:
> > First post, so short introduction:
> Hi, and welcome.
>
> > I'm mostly working on Django projects, currently for 3yourminD in Berlin.
> > Working with python / Django for 6 going on 7 years. Also experienced in
> > PHP, some Js and build systems like docker.
> > We're currently migrating everything to docker and pip / requirements.txt
> > isn't cutting it anymore and I've tasked myself with looking for or
> > building an alternative tool.
> >
> > This weekend I did quite a bit of research into the packaging tools and
> > came accross PEP 518. What it doesn't address is that there now is a file
> > resvered that declares very little. It does not declare and define
> > sections to serve as a deployment file for a project and apparently it is
> > unclear to tool builders what the one section is does define is supposed
> > to be used for.
>
> It is true, PEP 518 doesn't define a lot by itself. But as it notes,
> "Tables not specified in this PEP are reserved for future use by other
> PEPs" - in other words, it's possible to add further details, but they
> should be backed by a formal specification, so that tools have a
> common understanding of what to do with the data.
Great. Time to read "how to write a pep" and get started.
> > Case in point: poetry doesn't add itself to the build-system table when
> > writing the py-project.toml file. Which, as I read it, it should. Ideally,
> > every tool.${name} should be in the build system?
>
> Well, poetry isn't (as I understand it) a build system. It doesn't
> define *how* to build a wheel from project sources, rather it
> orchestrates the process of building an application from a set of
> projects (as I understand it from a brief read of the project page).
> That puts it in the same area as pipenv, I guess? That process
> (building *applications*) isn't something that's been really looked at
> yet in terms of standardisation, so yes, it'd still very much fair
> game for something to be discussed and agreed.
>
> One reason this hasn't yet been looked at, as I understand things, is
> that there are a lot of very different scenarios to consider - web
> applications (which I guess are your area of interest), standalone
> command line utilities like Mercurial or the Leo editor, orchestration
> tools like Ansible, background OS service processes, etc. And it's
> unlikely that a "one size fits all" approach would work, so there's a
> whole question of how any standard would establish its scope and
> boundaries. One note of caution - the attempts I've seen in the past
> to define "build processes" tend to promote a certain amount of
> confusion by assuming all projects are similar, and then getting
> bogged down when people with very different requirements come along
> with different assumptions and mistaken expectations.
>
..
> You say "a project (not a library)". I presume by that you mean
> something like the application building process I described above?
Ok, so there's the first confusion. My use of project is fostered first and
foremost by django's "startproject", but also by the use of project in
bitbucket, github and gitlab: a collection of libraries working towards a
common goal, that is usually unique in composition.
That + build-system put me on the path to believe that it's similar to a
Pipfile. And maybe there is a good need to introduce more then one file, with
shared traits or even building upon each other. So for now, I'll use your
definition of project further on.
> Yes, there's not been any standardisation effort in this area yet, so
> the situation is as you describe. It's not intentional as such, simply
> that no-one has really stepped up to propose anything yet. And maybe
> because there aren't that many formal tools in this area - a lot of
> what goes on appears to be simply workflow and local standards. But
> certainly it's something that might be worth exploring.
>
> > It would help if the PEP had some explanation of the choices made to
> > reserve a file that partially sets out to replace (all?) other files, but
> > does not do anything to accomplish that.
>
> PEP 518 defines a means of "Specifying Minimum Build System
> Requirements for Python Projects". It defines a new, flexible, file
> format to hold that information, and defers further definition of the
> file contents to future PEPs, but it was never (to my knowledge) a
> goal of that PEP to look at "all other files" and fit them into the
> pyproject.toml format.
There is a section "sticking with setup.cfg", that at least implies setup.cfg
can be replaced, but everything I can put in there, has no reserved spot in
pyproject.toml.
> It did reserve a namespace "tool.XXX" for
> projects that choose to consider themselves as "build tools" - and a
> number of projects (including poetry, I believe) have made use of
> that. (And as you can see, this opens up some confusion over what
> constitutes a "build tool", in contrast to my fairly strict definition
> above).
Indeed it does. Especially since pip is less used to build the packages of a
given project and more to install it on to the system - at least in my
experience. Even in tutorials you'll see "python setup.py build" advocated.
> PEP 517 extended the pyproject.toml format to add data for projects to
> define which build system they want to be used when they are built.
>
> There's nothing yet for "building a project (not a library)". You
> could start with getting agreement on some terminology explaining what
> "a project (not a library)" is, and should be called :-) Would it
> cover pip? What about Mercurial? Ansible? A Django site? A blogging
> application backed by Django (or Flask, or whatever)? Maybe we need
> multiple terms...
Yes, that would sure help any standardization going forward here.
>
> > I realize that PEP-517 defines the API and leaves
> > the implementation up to the tool, but that doesn't necessarily mean the
> > py- project file should not name / reserve tables that the PEP 517 API
> > should use.
> It does - tools.flit and tools.setuptools, and similar, for other build
> tools.
>
> Yes, there are places where further standardisation would be good (for
> example, config_options has no defined semantics, which means tools
> like pip can do little beyond pass unmodified key/value pairs to the
> backend), but I don't think that's what you're talking about here. I
> don't think PEP 517 is relevant to your proposal, though (unless you
> expect a tool like poetry to add a PEP 517 backend?)
There's a lot of similarity, but in the end I don't think a web application
with all it's sensitive information in configuration files, would beneift from
being packaged as a wheel. So I think the first distinction one should make is
between "installable" and "deployable": installed on a system to be ready to
use, or deployed to continously running systems, constantly in use and
upgraded with minimal downtime. Maybe, deployable is out of scope for python
packaging even.
That said, even if it's out of scope, it would help to have a standardization
of what a project needs and provides (to resolve dependencies), what version
it has (to resolve updating needs) and how it can be tested.
Further more, I think that ironically, Pypi.org would suffer the most from
this tool.* namespace, since it'll need to implement all the "metadata"
sections, "version" sections and custom-stuff sections from all tools to
provide all the useful information it does now.
So while it may not be relevant for my specific usecase, I still believe that
PEP517's input data should be standardized inside py-project without the tool
namespace.
>
> > I'm considering writing a tool that focuses solely on the deployment and
> > dependency maintenance of a project and would like to not "vendor lock-in"
> > my users, with even those dependency lists, because I have to use the
> > tool namespace and can fill it every way I see fit and so will others.
> >
> > So ideally, we would reserve dependency tables:
> > [dependencies.$name]
> >
> > Reserved names: standard, dev, test
> >
> > Tools can add more names which they can then interpret themselves:
> > [dependencies.stage]
> > django-anonymizer = "^0.5"
> >
> > And test tables and probably one or two more that come to mind. Is there
> > any interest from the team to improve the file format and lock down such
> > tables? I'd love to do the initial proposal if that helps move things
> > forward.
> For those tables to *not* lock people into your tool, you'd need other
> tools to agree to use the same names
No, I'm advocating that those sections be formalized (through new PEPs), so
that tools SHALL use those sections for those bits of the PEP 517 api and we
don't need 27 (random) different parses to obtain a dependency list for a
given package.
--
Melvyn Sopacua
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/VCTD5KTT3UINYD4OIVHJYW6MRFX6YQG2/
