Re: [Distutils] Getting dependecies of package from PyPiJSON

[Marius Gedminas]

On Fri, Jul 21, 2017 at 11:39:01AM +0200, Krzysiek Płachno wrote:
> Guys! Thanks a lot for all your responses and willingness to help!
> 
> Yesterday I noticed this `requires_dist` in response for Requests package. But
> actually it was the only one that had this field populated from many packages
> that I asked API for. So this cannot be reliable way of getting dependecies.
> 
> So if not using pip, the only way to get dependencies is from built packages:
> .whl and .egg.
> So in .egg it'd be: EGG_INFO/requires.txt and in .whl : 
> .dist-info/metadata.json, right?

I've had some luck parsing *.egg-info/requires.txt files in sdists.
This is not 100% reliable (a setup.py may dynamically-compute the
install_requires list during install time depending on sys.version_info
or other factors), but it was good enough for my purposes.

https://github.com/mgedmin/ztk-py3-status/blob/master/get_deps.py

Regards,
Marius Gedminas
-- 
Hi. I'm the "I love you" .signature virus. You have been infected.
Please panic immediately.


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Nick Coghlan]

On 21 July 2017 at 19:39, Krzysiek Płachno  wrote:
> Guys! Thanks a lot for all your responses and willingness to help!
>
> Yesterday I noticed this `requires_dist` in response for Requests package.
> But actually it was the only one that had this field populated from many
> packages that I asked API for. So this cannot be reliable way of getting
> dependecies.
>
> So if not using pip, the only way to get dependencies is from built
> packages: .whl and .egg.
> So in .egg it'd be: EGG_INFO/requires.txt and in .whl :
> .dist-info/metadata.json, right?

For wheel archives, the JSON format technically isn't stable yet (and
wheel generators aren't required to include it), so the file you want
is {distribution}-{version}.dist-info/METADATA, and specifically the
"Requires-Dist" entries

The relevant specifications:

METADATA file: 
https://www.python.org/dev/peps/pep-0345/#requires-dist-multiple-use
Requires-Dist values: https://www.python.org/dev/peps/pep-0508/
Wheel format: https://www.python.org/dev/peps/pep-0427/
Draft JSON metadata format: https://www.python.org/dev/peps/pep-0426/

There's no interoperability spec for the egg format, but the
setuptools docs have pretty good documentation for it:
https://setuptools.readthedocs.io/en/latest/formats.html#dependency-metadata

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Krzysiek Płachno]

Guys! Thanks a lot for all your responses and willingness to help!

Yesterday I noticed this `requires_dist` in response for Requests package.
But actually it was the only one that had this field populated from many
packages that I asked API for. So this cannot be reliable way of getting
dependecies.

So if not using pip, the only way to get dependencies is from built
packages: .whl and .egg.
So in .egg it'd be: *EGG_INFO/requires.txt* and in .whl :
*.dist-info/metadata.json*, right?

Thanks a lot!
Best,
Krzysztof



2017-07-21 3:56 GMT+02:00 Jeremy Stanley :

> On 2017-07-20 21:09:28 -0400 (-0400), John Thorvald Wodder II wrote:
> > [Sending to the list this time]
> >
> > On 2017 Jul 20, at 12:41, Jeremy Stanley  wrote:
> > > So while it might be possible to add some sort of feature
> > > to inspect wheels at upload and then store the specific dependencies
> > > declared therein and report those back via an API method, I expect
> > > coverage across packages in general would be fairly low today.
> >
> > PyPI (both Legacy and Warehouse) actually does do this already;
> > see the `requires_dist` field in, e.g.,
> > . However, this only seems to
> > work if the maintainer uploads the wheel before uploading the
> > sdist (unless the sdist is a .zip instead of a .tar.gz, then it
> > can be uploaded first?  I'm not sure).
>
> Indeed, I'd never noticed that. And the projects I work on upload
> wheels before sdists so seem to have everything from our
> install_requires reflected there (including extras and environment
> markers... even the versioned pages work). Very neat, and glad to
> learn it already exists.
>
> I wonder though how it deals with projects that build multiple
> wheels for different platforms with different install_requires. It
> looks like that's a top-level field in the info dict so can't
> reasonably be differentiated. Takes the first one uploaded I guess
> and ignores the subsequent ones?
>
> Anyway, this looks like it probably fulfils Krzysiek's need for
> XWiki. Thanks for pointing it out! I very well may try to leverage
> this for a few things myself.
> --
> Jeremy Stanley
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Jeremy Stanley]

On 2017-07-20 21:09:28 -0400 (-0400), John Thorvald Wodder II wrote:
> [Sending to the list this time]
> 
> On 2017 Jul 20, at 12:41, Jeremy Stanley  wrote:
> > So while it might be possible to add some sort of feature
> > to inspect wheels at upload and then store the specific dependencies
> > declared therein and report those back via an API method, I expect
> > coverage across packages in general would be fairly low today.
> 
> PyPI (both Legacy and Warehouse) actually does do this already;
> see the `requires_dist` field in, e.g.,
> . However, this only seems to
> work if the maintainer uploads the wheel before uploading the
> sdist (unless the sdist is a .zip instead of a .tar.gz, then it
> can be uploaded first?  I'm not sure).

Indeed, I'd never noticed that. And the projects I work on upload
wheels before sdists so seem to have everything from our
install_requires reflected there (including extras and environment
markers... even the versioned pages work). Very neat, and glad to
learn it already exists.

I wonder though how it deals with projects that build multiple
wheels for different platforms with different install_requires. It
looks like that's a top-level field in the info dict so can't
reasonably be differentiated. Takes the first one uploaded I guess
and ignores the subsequent ones?

Anyway, this looks like it probably fulfils Krzysiek's need for
XWiki. Thanks for pointing it out! I very well may try to leverage
this for a few things myself.
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[John Thorvald Wodder II]

[Sending to the list this time]

On 2017 Jul 20, at 12:41, Jeremy Stanley  wrote:
> So while it might be possible to add some sort of feature
> to inspect wheels at upload and then store the specific dependencies
> declared therein and report those back via an API method, I expect
> coverage across packages in general would be fairly low today.

PyPI (both Legacy and Warehouse) actually does do this already; see the 
`requires_dist` field in, e.g., . However, 
this only seems to work if the maintainer uploads the wheel before uploading 
the sdist (unless the sdist is a .zip instead of a .tar.gz, then it can be 
uploaded first?  I'm not sure).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Jeremy Stanley]

On 2017-07-20 11:11:38 -0300 (-0300), Leonardo Rochael Almeida wrote:
[...]
> A small clarification: Packages can define their own dependencies
> dynamically only at *build* time, not at *installation* time.
> 
> The difference is subtle (considering many packages (the ones with only
> sdist on PyPI) are built at the same time they're installed), but important:
> 
> In practice it means that if you have a wheel (or an egg), you can
> determine the dependencies without installing, just by looking at the
> metadata inside the package.

Agreed, I elided those details not knowing the extent of familiarity
of the question's author with nuances of Python packaging, but
you're right I should take more care to not overload the term
"installation" without clarifying the circumstances.

As many projects on PyPI lack wheels (and most non-pure-Python
projects lack wheels for at least some platforms), sdists are for
better or worse the most prevalent and portable "packaging" format
on PyPI. So while it might be possible to add some sort of feature
to inspect wheels at upload and then store the specific dependencies
declared therein and report those back via an API method, I expect
coverage across packages in general would be fairly low today. It
might be sufficient if sticking with some popular subset of packages
though.
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Jannis Gebauer]

Unfortunately, yes there is no public API for a Python packages’ dependencies 
available.

> On 20. Jul 2017, at 15:43, Krzysiek Płachno  wrote:
> 
> None of them sends back required dependencies for package. 
> Does it mean - this information is not obtainable via JSON Api?
> 
> Best,
> Krzysztof
> 
> 2017-07-20 15:04 GMT+02:00 Jannis Gebauer  >:
> Hi Krzysztof,
> 
> Two options, e.g. for the “graphene” package:
> 
> - https://pypi.python.org/pypi/graphene/json 
> 
> - https://pypi.org/pypi/graphene/json 
> 
> Best,
> 
> Jannis
> 
>> On 20. Jul 2017, at 14:55, Krzysiek Płachno > > wrote:
>> 
>> Hi!
>> 
>> I'm contributing to XWiki open source project http://www.xwiki.org/ 
>> . XWiki platform is written in Java and apart from 
>> many wonderful features allows for scripting in python on its pages. 
>> 
>> My current task is to enable installing packages form PyPi repository so 
>> that they can be later used in scripts. The installment is done in custom 
>> way making python packages available for Java classloader.
>> 
>> To make downloaded package working it's needed to install also dependencies. 
>> Is it possible to get dependencies information directly from PyPiJSON API? 
>> (e.g. by adding some request parameter or header in GET request) 
>> Or is it possible to this data in any other way (apart from downloading 
>> package)?
>> 
>> Thanks for help in advance!
>> 
>> Best,
>> Krzysztof
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org 
>> 
>> https://mail.python.org/mailman/listinfo/distutils-sig 
>> 
> 
> 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Leonardo Rochael Almeida]

Hi Jeremy,

On 20 July 2017 at 10:22, Jeremy Stanley  wrote:

> On 2017-07-20 14:55:53 +0200 (+0200), Krzysiek Płachno wrote:
> [...]
> >
> > Or is it possible to this data in any other way (apart from
> > downloading package)?
>
> Unfortunately, no, not with the current state of the Python package
> ecosystem. Packages are allowed to define their own dependencies
> dynamically and conditionally at the time of installation so there's
> no good way for PyPI to know what dependencies each package has.
>
> [...]


A small clarification: Packages can define their own dependencies
dynamically only at *build* time, not at *installation* time.

The difference is subtle (considering many packages (the ones with only
sdist on PyPI) are built at the same time they're installed), but important:

In practice it means that if you have a wheel (or an egg), you can
determine the dependencies without installing, just by looking at the
metadata inside the package.

Cheers,

Leo
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Nick Coghlan]

On 20 July 2017 at 23:43, Krzysiek Płachno  wrote:
> None of them sends back required dependencies for package.
> Does it mean - this information is not obtainable via JSON Api?

Unfortunately not, as dependencies currently aren't generally
available in a declarative form, they're obtained by executing the
setup.py file contained in the source archive :(

As a result, the approach most folks currently take is to let
installation tools like `pip` worry about the problem, as they
recursively download the required dependencies as part of the
installation process. The `pip download` command is provided
specifically for the use case where the goal is just to obtain the
required package set, without necessarily installing them:
https://pip.pypa.io/en/stable/reference/pip_download/

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Jannis Gebauer]

Hi Krzysztof,

Two options, e.g. for the “graphene” package:

- https://pypi.python.org/pypi/graphene/json 

- https://pypi.org/pypi/graphene/json 

Best,

Jannis

> On 20. Jul 2017, at 14:55, Krzysiek Płachno  wrote:
> 
> Hi!
> 
> I'm contributing to XWiki open source project http://www.xwiki.org/ 
> . XWiki platform is written in Java and apart from 
> many wonderful features allows for scripting in python on its pages. 
> 
> My current task is to enable installing packages form PyPi repository so that 
> they can be later used in scripts. The installment is done in custom way 
> making python packages available for Java classloader.
> 
> To make downloaded package working it's needed to install also dependencies. 
> Is it possible to get dependencies information directly from PyPiJSON API? 
> (e.g. by adding some request parameter or header in GET request) 
> Or is it possible to this data in any other way (apart from downloading 
> package)?
> 
> Thanks for help in advance!
> 
> Best,
> Krzysztof
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Jeremy Stanley]

On 2017-07-20 14:55:53 +0200 (+0200), Krzysiek Płachno wrote:
[...]
> To make downloaded package working it's needed to install also
> dependencies. Is it possible to get dependencies information
> directly from PyPiJSON API? (e.g. by adding some request parameter
> or header in GET request)
> 
> Or is it possible to this data in any other way (apart from
> downloading package)?

Unfortunately, no, not with the current state of the Python package
ecosystem. Packages are allowed to define their own dependencies
dynamically and conditionally at the time of installation so there's
no good way for PyPI to know what dependencies each package has.

PEP 518 aims to solve this eventually:

https://www.python.org/dev/peps/pep-0518/

-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Krzysiek Płachno]

None of them sends back required dependencies for package.
Does it mean - this information is not obtainable via JSON Api?

Best,
Krzysztof

2017-07-20 15:04 GMT+02:00 Jannis Gebauer :

> Hi Krzysztof,
>
> Two options, e.g. for the “graphene” package:
>
> - https://pypi.python.org/pypi/graphene/json
> - https://pypi.org/pypi/graphene/json
>
> Best,
>
> Jannis
>
> On 20. Jul 2017, at 14:55, Krzysiek Płachno 
> wrote:
>
> Hi!
>
> I'm contributing to XWiki open source project http://www.xwiki.org/.
> XWiki platform is written in Java and apart from many wonderful features
> allows for scripting in python on its pages.
>
> My current task is to enable installing packages form PyPi repository so
> that they can be later used in scripts. The installment is done in custom
> way making python packages available for Java classloader.
>
> To make downloaded package working it's needed to install also
> dependencies. Is it possible to get dependencies information directly from
> PyPiJSON API? (e.g. by adding some request parameter or header in GET
> request)
> Or is it possible to this data in any other way (apart from downloading
> package)?
>
> Thanks for help in advance!
>
> Best,
> Krzysztof
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Getting dependecies of package from PyPiJSON

[Krzysiek Płachno]

Hi!

I'm contributing to XWiki open source project http://www.xwiki.org/. XWiki
platform is written in Java and apart from many wonderful features allows
for scripting in python on its pages.

My current task is to enable installing packages form PyPi repository so
that they can be later used in scripts. The installment is done in custom
way making python packages available for Java classloader.

To make downloaded package working it's needed to install also
dependencies. Is it possible to get dependencies information directly from
PyPiJSON API? (e.g. by adding some request parameter or header in GET
request)
Or is it possible to this data in any other way (apart from downloading
package)?

Thanks for help in advance!

Best,
Krzysztof
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig