Re: [Distutils] Mirroring PyPI JSON Locally

[Nick Coghlan]

On 22 August 2017 at 01:46, Wes Turner  wrote:
> ## Justify JSONLD
> - This is a graph. If we use an existing spec for graphs as JSON (ie
> JSONLD), we win:
>   - all of the tools that already exist for working with said graphs in that
> format
>   - easy indexability (as RDF quads)
>   - compatibility with compatible specs like ld-signatures

No, that's the argument I've already said doesn't work, since it
doesn't address the readability problem Paul mentioned: yes, JSON-LD
*can* represent aribtrary graphs, but to *read* a JSON-LD data
structure as a human, you need to know how *JSON-LD* represents
graphs.

I don't consider that an acceptable limitation: the raw metadata needs
to be readable by someone that's *only* familiar with the specifics of
dependency management in Python, and couldn't care less about the
representation of graphs as a general concept.

> ## Implement JSONLD
> - [ ] decide which URI(s) a project on {pypi,} is identified by
>- some projects should not have an implicit pypi.org URI prefix
> - [ ] write a new JSONLD view for pypi and warehouse
> - [ ] write a JSONLD metadata spec for eggs and wheels

None of which are dependent on JSON-LD being the raw format for the
metadata - this can instead be done as a postprocessing step that
accepts any of the existing metadata formats as input. Defining such a
transformation is going to be critical for your goals of making the
JSON-LD representation useful anyway, as even if we defined a new
metadata format tomorrow, that would still mean there were more than
700 thousand existing releases on pypi.org that didn't natively
provide their metadata in that format.

The added bonus of doing things that way is that it means that you
*don't* need anyone else's agreement or consensus to start design work
- you can do an initial proof of concept using a domain you control,
similar to the way Donald started out by building the PyPI that he
wished existed as an independent service before we thanked him for his
efforts by lumbering him with the spectacularly difficult task of
figuring out how to upgrade or replace pypi.python.org itself :)

> ## Support metadata retrieval without exec'ing setup.py
>
> - develop a declarative format for expressing {sys.platform[...],}-dependent
> dependency edges

This is already part of PEP 508:
https://www.python.org/dev/peps/pep-0508/#environment-markers

This is why given a wheel file, you can *already* extract declarative
dependency metadata, using the METADATA file + PEP 508.

Given just an sdist, you can also do something similar by looking at
PKG-INFO, but that's less reliable (since that file may not even be
present, and even if it is, the sdist -> wheel build step may still
inject additional dependencies).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Bootstrapping setuptools

[xoviat]

This statement comes from something that Donald said:

> The unvendoring means that setuptools and the project code are now
competing over who gets to define what an acceptable version is for these
libraries to be installed with.

As if this isn't going to be true for any other build system, and
setuptools won't at all be in any way special in a PEP 517 world. So you
are always going to have complaints about "well build system X required a
newer version but my project requires an older version" when the real
problem is that people are requiring an older version/ projects do not have
a stable API.

2017-08-21 16:27 GMT-05:00 Jim Fulton :

>
>
> On Mon, Aug 21, 2017 at 5:17 PM, xoviat  wrote:
>
>> Of course, to be frank, the principle failure with this plan is that
>> third-party tools (buildout, os packagers) will not be compliant with PEP
>> 517 even after it is adopted, and will then complain about having to update
>> their build systems.
>>
>
> I don't understand this statement. If buildout needs to be compliant, it
> will be (eventually).  I can imagine a future where everything is
> distributed with wheels and buildout wouldn't need to build anything.
>
> Jim
>
>
>>
>> 2017-08-21 16:05 GMT-05:00 xoviat :
>>
>>> Previously, the attempt to move setuptools off of vendored dependencies
>>> failed because it was not done correctly: install_requires was set to the
>>> vendored dependencies but not setup_requires, which would have been
>>> required to correctly specify dependencies. However, setup_requires would
>>> have introduced circular dependencies that are impossible to correctly
>>> resolve, so that would have simply shifted an impossible problem onto pip.
>>>
>>> The principle issue then, is that setuptools requires setuptools to
>>> build itself. And although, this problem was previously not worth solving
>>> because of 'setup.py install', it is now not a difficult problem to solve
>>> if we rethink slightly what should be included in the setuptools
>>> respository:
>>> - Rather than re-generating egg_info on each setup.py invocation, we can
>>> cache dist-info in the setuptools respository.
>>> - We can implement a simple entry_points script that updates dist-info
>>> based on the contents of setup.py, which is only runnable of course after
>>> setuptools is installed
>>> - We can implement the reference build backend for setuptools: simply
>>> copy the files and dist-info into a new compliant zip archive
>>>
>>> Viola! Setuptools is no longer required to build setuptools, and the
>>> installation process is fully compliant with Python PEPs.
>>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>
> --
> Jim Fulton
> http://jimfulton.info
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Bootstrapping setuptools

[Jim Fulton]

On Mon, Aug 21, 2017 at 5:17 PM, xoviat  wrote:

> Of course, to be frank, the principle failure with this plan is that
> third-party tools (buildout, os packagers) will not be compliant with PEP
> 517 even after it is adopted, and will then complain about having to update
> their build systems.
>

I don't understand this statement. If buildout needs to be compliant, it
will be (eventually).  I can imagine a future where everything is
distributed with wheels and buildout wouldn't need to build anything.

Jim


>
> 2017-08-21 16:05 GMT-05:00 xoviat :
>
>> Previously, the attempt to move setuptools off of vendored dependencies
>> failed because it was not done correctly: install_requires was set to the
>> vendored dependencies but not setup_requires, which would have been
>> required to correctly specify dependencies. However, setup_requires would
>> have introduced circular dependencies that are impossible to correctly
>> resolve, so that would have simply shifted an impossible problem onto pip.
>>
>> The principle issue then, is that setuptools requires setuptools to build
>> itself. And although, this problem was previously not worth solving because
>> of 'setup.py install', it is now not a difficult problem to solve if we
>> rethink slightly what should be included in the setuptools respository:
>> - Rather than re-generating egg_info on each setup.py invocation, we can
>> cache dist-info in the setuptools respository.
>> - We can implement a simple entry_points script that updates dist-info
>> based on the contents of setup.py, which is only runnable of course after
>> setuptools is installed
>> - We can implement the reference build backend for setuptools: simply
>> copy the files and dist-info into a new compliant zip archive
>>
>> Viola! Setuptools is no longer required to build setuptools, and the
>> installation process is fully compliant with Python PEPs.
>>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Bootstrapping setuptools

[xoviat]

Of course, to be frank, the principle failure with this plan is that
third-party tools (buildout, os packagers) will not be compliant with PEP
517 even after it is adopted, and will then complain about having to update
their build systems.

2017-08-21 16:05 GMT-05:00 xoviat :

> Previously, the attempt to move setuptools off of vendored dependencies
> failed because it was not done correctly: install_requires was set to the
> vendored dependencies but not setup_requires, which would have been
> required to correctly specify dependencies. However, setup_requires would
> have introduced circular dependencies that are impossible to correctly
> resolve, so that would have simply shifted an impossible problem onto pip.
>
> The principle issue then, is that setuptools requires setuptools to build
> itself. And although, this problem was previously not worth solving because
> of 'setup.py install', it is now not a difficult problem to solve if we
> rethink slightly what should be included in the setuptools respository:
> - Rather than re-generating egg_info on each setup.py invocation, we can
> cache dist-info in the setuptools respository.
> - We can implement a simple entry_points script that updates dist-info
> based on the contents of setup.py, which is only runnable of course after
> setuptools is installed
> - We can implement the reference build backend for setuptools: simply copy
> the files and dist-info into a new compliant zip archive
>
> Viola! Setuptools is no longer required to build setuptools, and the
> installation process is fully compliant with Python PEPs.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP 517: Bootstrapping setuptools

[xoviat]

Previously, the attempt to move setuptools off of vendored dependencies
failed because it was not done correctly: install_requires was set to the
vendored dependencies but not setup_requires, which would have been
required to correctly specify dependencies. However, setup_requires would
have introduced circular dependencies that are impossible to correctly
resolve, so that would have simply shifted an impossible problem onto pip.

The principle issue then, is that setuptools requires setuptools to build
itself. And although, this problem was previously not worth solving because
of 'setup.py install', it is now not a difficult problem to solve if we
rethink slightly what should be included in the setuptools respository:
- Rather than re-generating egg_info on each setup.py invocation, we can
cache dist-info in the setuptools respository.
- We can implement a simple entry_points script that updates dist-info
based on the contents of setup.py, which is only runnable of course after
setuptools is installed
- We can implement the reference build backend for setuptools: simply copy
the files and dist-info into a new compliant zip archive

Viola! Setuptools is no longer required to build setuptools, and the
installation process is fully compliant with Python PEPs.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Mirroring PyPI JSON Locally

[Wes Turner]

On Monday, August 21, 2017, Nick Coghlan  wrote:

> On 21 August 2017 at 19:38, Paul Moore >
> wrote:
> > On 21 August 2017 at 09:54, Nick Coghlan  > wrote:
> >> While I'm still generally negative on the idea of native reliance on
> >> JSON-LD, I'll note one thing that has changed since I last looked at
> >> it: I now see some potential concrete practical benefits to adopting
> >> it, rather than purely theoretical ones. In particular,
> >> https://github.com/scienceai/jsonld-vis now exists, and there wasn't
> >> anything like that around at the time of previous discussions.
> >
> > Personally, I fairly often write adhoc scripts that use the JSON API,
> > and as it stands it's very convenient for that. From what I can see of
> > JSON-LD (which basically equates to "it adds some extra metadata keys
> > that don't change the data content but do change the list of keys and
> > maybe the nesting levels") it would be somewhat inconvenient for my
> > scripts, and add no extra capability that I would ever use.
>
> Right, and this is still my main concern with the idea as well: I'd
> never be OK with a JSON-LD-only API, because it adds too much
> irrelevant cognitive overhead for the vast majority of Python
> packaging specific use cases. (I would see it as being akin to Python
> itself deciding to require type annotations, rather than merely
> allowing them).
>
> However, where I'm starting to see a potential niche for it is as an
> opt-in capability, whereby we explicitly define how our metadata can
> be translated *to* JSON-LD, for folks that want to apply general
> purpose tools that know how to manipulate arbitrary JSON-LD data (like
> the graph visualiser I linked earlier).
>
> That way, everybody wins - folks that have never heard of schema.org
> or linked data in general won't need to learn any concepts that are
> completely irrelevant to them, while folks that are aware of those
> things and the related tools will be free to use them without first
> having to figure out their own mapping from the Python specific
> metadata formats to a JSON-LD compatible format.
>
> That approach then doesn't even need to wait for PEP 426: it could be
> done using the wheel METADATA file as a basis instead.
>
> It will probably still be up to Wes to actually define that
> transformation though - I don't think anybody else is anywhere near
> keen enough to make use of the available JSON-LD tooling to spend any
> time working on enabling it :)


So,

## Justify JSONLD
- This is a graph. If we use an existing spec for graphs as JSON (ie
JSONLD), we win:
  - all of the tools that already exist for working with said graphs in
that format
  - easy indexability (as RDF quads)
  - compatibility with compatible specs like ld-signatures

## Implement JSONLD
- [ ] decide which URI(s) a project on {pypi,} is identified by
   - some projects should not have an implicit pypi.org URI prefix
- [ ] write a new JSONLD view for pypi and warehouse
- [ ] write a JSONLD metadata spec for eggs and wheels

## Support metadata retrieval without exec'ing setup.py

- develop a declarative format for expressing
{sys.platform[...],}-dependent dependency edges


Signed,
Wes T.

P.P.S. This is just a hard week for me,
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Mirroring PyPI JSON Locally

[Nick Coghlan]

On 21 August 2017 at 19:38, Paul Moore  wrote:
> On 21 August 2017 at 09:54, Nick Coghlan  wrote:
>> While I'm still generally negative on the idea of native reliance on
>> JSON-LD, I'll note one thing that has changed since I last looked at
>> it: I now see some potential concrete practical benefits to adopting
>> it, rather than purely theoretical ones. In particular,
>> https://github.com/scienceai/jsonld-vis now exists, and there wasn't
>> anything like that around at the time of previous discussions.
>
> Personally, I fairly often write adhoc scripts that use the JSON API,
> and as it stands it's very convenient for that. From what I can see of
> JSON-LD (which basically equates to "it adds some extra metadata keys
> that don't change the data content but do change the list of keys and
> maybe the nesting levels") it would be somewhat inconvenient for my
> scripts, and add no extra capability that I would ever use.

Right, and this is still my main concern with the idea as well: I'd
never be OK with a JSON-LD-only API, because it adds too much
irrelevant cognitive overhead for the vast majority of Python
packaging specific use cases. (I would see it as being akin to Python
itself deciding to require type annotations, rather than merely
allowing them).

However, where I'm starting to see a potential niche for it is as an
opt-in capability, whereby we explicitly define how our metadata can
be translated *to* JSON-LD, for folks that want to apply general
purpose tools that know how to manipulate arbitrary JSON-LD data (like
the graph visualiser I linked earlier).

That way, everybody wins - folks that have never heard of schema.org
or linked data in general won't need to learn any concepts that are
completely irrelevant to them, while folks that are aware of those
things and the related tools will be free to use them without first
having to figure out their own mapping from the Python specific
metadata formats to a JSON-LD compatible format.

That approach then doesn't even need to wait for PEP 426: it could be
done using the wheel METADATA file as a basis instead.

It will probably still be up to Wes to actually define that
transformation though - I don't think anybody else is anywhere near
keen enough to make use of the available JSON-LD tooling to spend any
time working on enabling it :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] [issue166] Install path changes when data_files is present

[Salvo Tomaselli]

New submission from Salvo Tomaselli:

I have a python thing that I'm installing in a custom path, using

setup.py install --root=/tmp/turi13 --install-purelib=custom/path

However, if the setup.py file contains a data_files field, then, unless i also 
specify --install-data, the --install-purelib parameter gets ignored.

I think an error should be generated and the entire thing should fail instead. 
Same for all of those --install-* parameters.

--
messages: 791
nosy: ltworf
priority: bug
status: unread
title: Install path changes when data_files is present

___
Setuptools tracker 

___
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Mirroring PyPI JSON Locally

[Paul Moore]

On 21 August 2017 at 09:54, Nick Coghlan  wrote:
> While I'm still generally negative on the idea of native reliance on
> JSON-LD, I'll note one thing that has changed since I last looked at
> it: I now see some potential concrete practical benefits to adopting
> it, rather than purely theoretical ones. In particular,
> https://github.com/scienceai/jsonld-vis now exists, and there wasn't
> anything like that around at the time of previous discussions.

Personally, I fairly often write adhoc scripts that use the JSON API,
and as it stands it's very convenient for that. From what I can see of
JSON-LD (which basically equates to "it adds some extra metadata keys
that don't change the data content but do change the list of keys and
maybe the nesting levels") it would be somewhat inconvenient for my
scripts, and add no extra capability that I would ever use.

Before we consider anything like JSON-LD, I think we need a much
clearer picture of who uses the JSON API. If it's production-type
applications, then maybe it would be useful, but if it's mostly advoc
scripts (as I suspect) it's additional complexity for little or no
benefit.

But this remains off-topic for now, so that's all I'll say.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Transfering modules

[Nick Coghlan]

On 21 August 2017 at 02:16, Ian Hartley  wrote:
> Ronald,
>
> I was sure that was the case re: python.org installation.  It's strange that
> it installed in two different locations,  I really have no idea how that
> might have come about.  FYI I've included a pic of the file pathways when
> you look at the interpreter photo.  I'd first installed python using
> terminal btw.

Investigating a but further, I think that's actually the source of the
discrepancy: /usr/local/Cellar is where homebrew puts things, while
the python.org installers put things into /Library/Frameworks/.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Mirroring PyPI JSON Locally

[Nick Coghlan]

On 21 August 2017 at 00:51, Wes Turner  wrote:
> IIUC, the task is still to:
> Download transitive portions of a [linked data] graph as JSON[LD] (optimally
> without iteratively downloading and decompressing package archives in order
> to retrieve their platform-dependent dependency edge metadata from a
> setup.py that is executed with filesystem privileges).

While I'm still generally negative on the idea of native reliance on
JSON-LD, I'll note one thing that has changed since I last looked at
it: I now see some potential concrete practical benefits to adopting
it, rather than purely theoretical ones. In particular,
https://github.com/scienceai/jsonld-vis now exists, and there wasn't
anything like that around at the time of previous discussions.

However, that's still only of potential interest for PEP 426, which in
turn still isn't needed for any of our practical near term objectives
(not even the "dependencies without downloads" one - if we were to
prioritise that, we'd likely go for something closer to the way client
side dependency resolution already works, such as extracting the
METADATA file from uploaded wheels and making it available for
download in addition to the full wheel archives).

So for this thread, Paul's right: a PEP-503-style document describing
the current PyPI JSON API would likely be a reasonable thing to write,
as it would allow for more complete emulations of the current
production PyPI service, including checking that Warehouse replicates
that part of the API correctly.

Anything more than that is still on the "No, not until some point
after Legacy PyPI has been shut down" list.

Cheers,
Nick.

P.S. Some of the tools mentioned at
http://www.seoskeptic.com/structured-data-markup-validation-testing-tools/
may also prove useful if we go down the JSON-LD path. However, that is
the only reason I'd ever support us going down that path: useful
functionality that we get for free by virtue of adopting an
established convention. I've wrangled volunteer contributors to open
source projects for long enough now to know that "because it's the
right thing to do" simply doesn't cut it as a motivational tool -
there's need to be some kind of actual benefit to the folks doing the
work :)

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig