Re: [Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Saturday, December 10, 2016, Wes Turner  wrote:

> Here are some standardized (conda) package versions: https://github.com/
> jupyter/docker-stacks/blob/master/scipy-notebook/Dockerfile
>
>
IDK how they choose packages - what "criteria for inclusion" - for the
kaggle/docker-python Dockerfile:
https://github.com/Kaggle/docker-python/blob/master/Dockerfile

(Ubuntu, Conda, *, TPOT)


> On Thursday, December 8, 2016, Wes Turner  > wrote:
>
>>
>>
>> On Thursday, December 8, 2016, Nick Coghlan  wrote:
>>
>>> Putting the conclusion first, I do see value in better publicising
>>> "Recommended libraries" based on some automated criteria like:
>>>
>>> - recommended in the standard library documentation
>>> - available via 1 or more cross-platform commercial Python redistributors
>>> - available via 1 or more Linux distro vendors
>>> - available via 1 or more web service development platforms
>>>
>>>
>> So these would be attributes tracked by a project maintainer and verified
>> by the known-good-set maintainer? Or?
>>
>> (Again, here I reach for JSONLD. "count n" is only so useful; *which*
>> {[re]distros, platforms, heartfelt testimonials from incredible experts}
>> URLs )
>>
>> - test coverage
>> - seclist contact info AND procedures
>> - more than one super admin maintainer
>> - what other criteria should/could/can we use to vet open source
>> libraries?
>>
>>
>>> That would be a potentially valuable service for folks new to the
>>> world of open source that are feeling somewhat overwhelmed by the
>>> sheer number of alternatives now available to them.
>>>
>>> However, I also think that would better fit in with the aims of an
>>> open source component tracking community like libraries.io than it
>>> does a publisher-centric community like distutils-sig.
>>
>>
>> IDK if libraries are really in scope for stackshare. The feature
>> upcoming/down voting is pretty cool.
>>
>> https://stackshare.io/python
>>
>>
>>>
>>> The further comments below are just a bit more background on why I
>>> feel the integration testing aspect of the suggestion isn't likely to
>>> be particularly beneficial :)
>>
>>
>> A catch-all for testing bits from application-specific integration test
>> suites could be useful (and would likely require at least docker-compose,
>> dox, kompose for working with actual data stores)
>>
>>
>>>
>>> On 9 December 2016 at 01:10, Barry Warsaw  wrote:
>>> > Still, there may be value in inter-Python package compatibility tests,
>>> but
>>> > it'll take serious engineering effort (i.e. $ and time), ongoing
>>> maintenance,
>>> > ongoing effort to fix problems, and tooling to gate installability of
>>> failing
>>> > packages (with overrides for downstreams which don't care or already
>>> expend
>>> > such effort).
>>>
>>> I think this is really the main issue, as both desktop and server
>>> environments are moving towards the integrated platform + isolated
>>> applications approach popularised by mobile devices.
>>>
>>> That means we end up with two very different variants of automated
>>> integration testing:
>>>
>>> - the application focused kind offered by the likes of requires.io and
>>> pyup.io (i.e. monitor for dependency updates, submit PRs to trigger
>>> app level CI)
>>> - the platform focused kind employed by distro vendors (testing all
>>> the platform components work together, including the app isolation
>>> features)
>>>
>>> The first kind makes sense if you're building something that runs *on*
>>> platforms (Docker containers, Snappy or FlatPak apps, web services,
>>> mobile apps, etc).
>>>
>>> The second kind inevitably ends up intertwined with the component
>>> review and release engineering systems of the particular platform, so
>>> it becomes really hard to collaborate cross-platform outside the
>>> context of specific projects like OpenStack that provide clear
>>> definitions for "What components do we collectively depend on that we
>>> need to test together?" and "What does 'working' mean in the context
>>> of this project?".
>>>
>>> Accordingly, for an initiative like this to be successful, it would
>>> need to put some thought up front into the questions of:
>>>
>>> 1. Who are the intended beneficiaries of the proposal?
>>> 2. What problem does it address that will prompt them to contribute
>>> time and/or money to solving it?
>>> 3. What do we expect people to be able to *stop doing* if the project
>>> proves successful?
>>>
>>> For platform providers, a generic "stdlib++" project wouldn't really
>>> reduce the amount of integration testing we'd need to do ourselves (we
>>> already don't test arbitrary combinations of dependencies, just the
>>> ones we provide at any given point in time).
>>>
>>> For application and service developers, the approach of pinning
>>> dependencies to specific versions and treating updates like any other
>>> source code 

Re: [Distutils] Maintaining a curated set of Python packages

[Nick Coghlan]

On 16 December 2016 at 14:29, Glyph Lefkowitz 
wrote:

> On Dec 15, 2016, at 8:18 PM, Nick Coghlan  wrote:
>
> At the beginning of your story you mentioned the GUI client - *that* is
> the missing piece ;).  I've been saying for years that we need a Python.app
> that lets you easily bootstrap all this stuff: walk you through installing
> C dev tools if your packages need them, present a GUI search interface to
> finding packages, present a normal "file->open" dialog for selecting a
> location for a new virtualenv, automatically pop open a terminal, launching
> a Jupyter notebook whose kernel is pointed at said environment...
>

It isn't really, as we started looking at this for IDLE, and the entire
current UX is just fundamentally beginner hostile:

- virtual environments are hard
- requirements files are hard
- knowing what packages are trustworthy and worth your time is hard
- limiting students to a set of "known safe" packages is hard
- components that assume command line use are hard

They're especially hard if the only way to distribute a fix is to release
an entire new edition of CPython rather than having IDLE talk to a
(preferably configurable) backend cloud service for updated instructions.

So there's a reason so many learning and even full development environments
are moving online - they let the service provider deal with all the hassles
of providing an appropriately configured environment, while the students
can focus on learning how to code, and the developers can focus on defining
their application logic.

However, the reason I brought up the Curse and Firefox GUI examples was to
emphasise the problems they hide from the default rich client experience:

- their default focus is on managing one environment per device
- they both may require environment restarts for changes to take effect
- they both reference an at least somewhat moderated back end (by Curse in
the Curse client case, by Mozilla in the Firefox case)
- they both incorporate popularity metrics and addon ratings into the
client experience

Mobile app store clients also share those four characteristics (where
"number of installations" and "star ratings" are critically important to
search rankings, but gaming the latter is mitigated by hiding the "Write a
review" feature if you haven't actually installed the app anywhere)


> User-curated package sets strikes me as the _lowest_ priority feature out
> of all of those, if we are ordering by priority to deliver a good user
> experience.  I know "steam curators" have been brought up before - but
> we're talking about adding curators (one of my least favorite features of
> Steam, for what it's worth) before we've added "install game" ;-).
>

In many educational contexts, adding "install game" without support for
institutional curators of some kind is a complete non-starter (even if
those curators are a collaborative community like a Linux distribution,
there's still more accountability than software publishing sites like PyPI
tend to provide).


> Users might even figure out this sort of stuff for themselves if they are
> given a discoverable API for things like search and installation of
> packages.
>

That sounds a bit like agreement that we're still missing some of the
backend pieces needed to make a beginner-friendly client really viable :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Thursday, December 15, 2016, Nick Coghlan  wrote:

> On 16 December 2016 at 05:50, Paul Moore  > wrote:
> > On 15 December 2016 at 19:13, Wes Turner  > wrote:
> >>> Just to add my POV, I also find your posts unhelpful, Wes. There's not
> >>> enough information for me to evaluate what you say, and you offer no
> >>> actual solutions to what's being discussed.
> >>
> >>
> >> I could quote myself suggesting solutions in this thread, if you like?
> >
> > You offer lots of pointers to information. But that's different.
>
> Exactly. There are *lots* of information processing standards out
> there, and lots of things we *could* provide natively that simply
> aren't worth the hassle since folks that care can provide them as
> "after market addons" for the audiences that considers them relevant.
>
> For example, a few things that can matter to different audiences are:
>
> - SPDX (Software Package Data Exchange) identifiers for licenses
> - CPE (Common Product Enumeration) and SWID (Software Identification)
> tags for published software
> - DOI (Digital Object Identifier) tags for citation purposes
> - Common Criteria certification for software supply chains


These are called properties with RDFS.

It takes very little effort to add additional properties. If the
unqualified attribute is not listed in a JSONLD @context, it can still be
added by specifying a URI


>
> I don't push for these upstream in distutils-sig not because I don't
> think they're important in general, but because I *don't think they're
> a priority for distutils-sig*. If you're teaching Python to school
> students, or teaching engineers and scientists how to better analyse
> their own data, or building a web service for yourself or your
> employer, these kinds of things simply don't matter.


#31 lists a number of advantages.
OTOMH, CVE security reports could be linked to the project/package URI (and
thus displayed along with the project detail page)


>
> The end users that care about them are well-positioned to tackle them
> on their own (or pay other organisations to do it for them), and
> because they span arbitrary publishing communities anyway, it doesn't
> really matter all that much if any given publishing community
> participates directly in the process (the only real beneficiaries are
> the intermediaries that actively blur the distinctions between the
> cooperative communities and the recalcitrant ones).


Linked Data minimizes


>
> > Anyway, let's just agree to differ - I can skip your mails if they
> > aren't helpful to me, and you don't need to bother about the fact that
> > you're not getting your points across to me.
>
> I consider it fairly important that we have a reasonably common
> understanding of the target userbase for direct consumption of PyPI
> data, and what we expect to be supplied as third party services. It's
> also important that we have a shared understanding of how to
> constructively frame proposals for change.


When I can afford the time, I'll again take a look at fixing the metadata
specification once and for all by (1) defining an @context for the existing
metadata, and (2) producing an additional pydist.jsonld TODO metadata
document (because the releases are currently keyed by version), and (3)
adding the model attribute and view to Warehouse.


>
> For the former, the Semantic Web, and folks that care about Semantic
> Web concepts like "Linked Data" in the abstract sense are not part of
> our primary audience. We don't go out of our way to make their lives
> difficult, but "it makes semantic analysis easier" also isn't a
> compelling rationale for change.


Unfortunately, you types are not well-versed in the problems that Linked
Data solves: it's all your data in your schema in your database; and URIs
are far less useful than RAM-local references (pointers).

See: BP-LD


>
> For the latter, some variants of constructive proposals look like:
>
> - "this kind of user has this kind of problem and this proposed
> solution will help mitigate it this way (and, by the way, here's an
> existing standard we can use)"
> - "this feature exists in , it's really
> valuable to users for , how about we offer it by
> default?"
> - "I wrote  for myself, and I think it would also help others
> for , can you help me make it more widely known and
> available?"


One could stuff additional metadata in # comments of a requirements.txt,
but that would be an ad-hoc parsing scheme with a SPOF tool dependency.


> They don't look like "Here's a bunch of technologies and organisations
> that exist on the internet that may in some way potentially be
> relevant to the management of a software distribution network", and
> nor does it look like "This data modeling standard exists, so we
> should use it, even though it doesn't actually simplify our lives or
> our users' lives in any way, and in fact makes them more complicated".


Those 

Re: [Distutils] Maintaining a curated set of Python packages

[Glyph Lefkowitz]

> On Dec 15, 2016, at 8:33 PM, Donald Stufft  wrote:
> 
> 
>> On Dec 15, 2016, at 11:29 PM, Glyph Lefkowitz > > wrote:
>> 
>> User-curated package sets strikes me as the _lowest_ priority feature out of 
>> all of those
> 
> 
> 
> I don’t think anyone in the PyPA is planning on working on this currently. It 
> was a possible idea that was spawned from this thread. However the nature of 
> volunteer OSS is that volunteer time is not fungible and if someone feels 
> particularly enthused about this idea they are free to pursue it.

I did very consciously choose my words there: "strikes me as", not "is" ;-)

-glyph

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Donald Stufft]

> On Dec 15, 2016, at 11:29 PM, Glyph Lefkowitz  wrote:
> 
> User-curated package sets strikes me as the _lowest_ priority feature out of 
> all of those



I don’t think anyone in the PyPA is planning on working on this currently. It 
was a possible idea that was spawned from this thread. However the nature of 
volunteer OSS is that volunteer time is not fungible and if someone feels 
particularly enthused about this idea they are free to pursue it.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Glyph Lefkowitz]

> On Dec 15, 2016, at 8:18 PM, Nick Coghlan  wrote:
> 
> On 16 December 2016 at 07:14, Glyph Lefkowitz  > wrote:
>> On Dec 15, 2016, at 6:39 AM, Donald Stufft > > wrote:
>> Theoretically we could allow people to not just select packages, but also 
>> package specifiers for their “curated package set”, so instead of saying 
>> “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we 
>> really wanted to get slick we could even provide a requirements.txt file 
>> format, and have people able to install the entire set by doing something 
>> like:
>> 
>> $ pip install -r 
>> https://pypi.org/sets/dstufft/my-cool-set/requirements.txt 
>> 
> Can't people already do this by publishing a package that just depends on 
> their whole 'package set'?
> 
> Technically, sure, but it adds a lot of overhead. The closest equivalent 
> right now would be maintaining a source control repo somewhere with various 
> requirements files in it.
> 
> However, at an ecosystem level, that doesn't have the same user experience 
> impact. The idea of building this into PyPI itself would be to *reshape the 
> learning curve of how people learn about dependency management as they're 
> introduced to Python*.
> 
> Going back to the CurseGaming example, I actually use the free version of 
> their client to manage the Warcraft addons on my gaming PC. The basic usage 
> model is really simple and (not coincidentally) very similar to the way the 
> Add-on manager works in Firefox and other GUI apps with integrated plugin 
> managers:
> 
> - you have an "Installed" tab for the addons you have installed
> - when you start the client, it checks for updates for all your installed 
> addons and the out of date ones gain an "Update" button
> - there's a separate tab where you can search all the available addons and 
> install new ones
> 
> I've never used any of Curse's other clients (like the Minecraft or Kerbal 
> Space Program ones), but I assume they operate in a similar way.
> 
> The paid tier of the Curse Client, and the account sync feature of Firefox, 
> then offer the ability to synchronize your installed addons across machines. 
> (There are also a lot of similarities between this model and the way mobile 
> app stores work)
> 
> A comparable UX for Python/PyPI/pip would focus less on the 
> library-and-application development cases (where the presence of source 
> control is assumed), and more on the ad hoc scripting and learning-to-program 
> use cases, where you're typically more interested in "--user" installations 
> and the question of which parts of the Python ecosystem are just an import 
> away than you are in reproducability and maintainability.
> 
> The ecosystem level learning curve then becomes:
> 
> - did you know you can back up your list of user installed packages to PyPI?
> - did you know you can use PyPI to sync your user installs between systems?
> - did you know you can use PyPI to categorise your user installs and share 
> them with others?
> - OK, now it's time to start learning about version control, virtual 
> environments and automated testing
> 
> It wouldn't necessarily make sense to bake this *directly* into Warehouse, 
> and the Mozilla folks responsible for Firefox Sync could no doubt offer real 
> word guidance on the infrastructure and support needed to operate a service 
> like that at scale, but the core concept of allowing package management to be 
> introduced independently of both version control and virtual environments 
> sounds potentially valuable to me.

Yeah, I think that this focus on curating packages on PyPI is reminiscent about 
the old yarn about looking for lost keys under the streetlight because it's 
dark everywhere else.  We're all familiar with web services and data formats, 
so we want to somehow have a data format or a web service be the answer to this 
problem.  But I don't believe that's where the problem is.

("this problem" being "let's make it easy and fun to a) bootstrap a common 
Python experimentation environment across multiple machines and b) _know that 
you have to do that_")

At the beginning of your story you mentioned the GUI client - that is the 
missing piece ;).  I've been saying for years that we need a Python.app that 
lets you easily bootstrap all this stuff: walk you through installing C dev 
tools if your packages need them, present a GUI search interface to finding 
packages, present a normal "file->open" dialog for selecting a location for a 
new virtualenv, automatically pop open a terminal, launching a Jupyter notebook 
whose kernel is pointed at said environment...

User-curated package sets strikes me as the _lowest_ priority feature out of 
all of those, if we are ordering by priority to deliver a good user experience. 
 I know "steam curators" have been 

Re: [Distutils] Maintaining a curated set of Python packages

[Nick Coghlan]

On 16 December 2016 at 07:14, Glyph Lefkowitz 
wrote:

> On Dec 15, 2016, at 6:39 AM, Donald Stufft  wrote:
> Theoretically we could allow people to not just select packages, but also
> package specifiers for their “curated package set”, so instead of saying
> “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we
> really wanted to get slick we could even provide a requirements.txt file
> format, and have people able to install the entire set by doing something
> like:
>
> $ pip install -r https://pypi.org/sets/dstufft/
> my-cool-set/requirements.txt
>
>
> Can't people already do this by publishing a package that just depends on
> their whole 'package set'?
>

Technically, sure, but it adds a lot of overhead. The closest equivalent
right now would be maintaining a source control repo somewhere with various
requirements files in it.

However, at an ecosystem level, that doesn't have the same user experience
impact. The idea of building this into PyPI itself would be to *reshape the
learning curve of how people learn about dependency management as they're
introduced to Python*.

Going back to the CurseGaming example, I actually use the free version of
their client to manage the Warcraft addons on my gaming PC. The basic usage
model is really simple and (not coincidentally) very similar to the way the
Add-on manager works in Firefox and other GUI apps with integrated plugin
managers:

- you have an "Installed" tab for the addons you have installed
- when you start the client, it checks for updates for all your installed
addons and the out of date ones gain an "Update" button
- there's a separate tab where you can search all the available addons and
install new ones

I've never used any of Curse's other clients (like the Minecraft or Kerbal
Space Program ones), but I assume they operate in a similar way.

The paid tier of the Curse Client, and the account sync feature of Firefox,
then offer the ability to synchronize your installed addons across
machines. (There are also a lot of similarities between this model and the
way mobile app stores work)

A comparable UX for Python/PyPI/pip would focus less on the
library-and-application development cases (where the presence of source
control is assumed), and more on the ad hoc scripting and
learning-to-program use cases, where you're typically more interested in
"--user" installations and the question of which parts of the Python
ecosystem are just an import away than you are in reproducability and
maintainability.

The ecosystem level learning curve then becomes:

- did you know you can back up your list of user installed packages to PyPI?
- did you know you can use PyPI to sync your user installs between systems?
- did you know you can use PyPI to categorise your user installs and share
them with others?
- OK, now it's time to start learning about version control, virtual
environments and automated testing

It wouldn't necessarily make sense to bake this *directly* into Warehouse,
and the Mozilla folks responsible for Firefox Sync could no doubt offer
real word guidance on the infrastructure and support needed to operate a
service like that at scale, but the core concept of allowing package
management to be introduced independently of both version control and
virtual environments sounds potentially valuable to me.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Nick Coghlan]

On 16 December 2016 at 05:50, Paul Moore  wrote:
> On 15 December 2016 at 19:13, Wes Turner  wrote:
>>> Just to add my POV, I also find your posts unhelpful, Wes. There's not
>>> enough information for me to evaluate what you say, and you offer no
>>> actual solutions to what's being discussed.
>>
>>
>> I could quote myself suggesting solutions in this thread, if you like?
>
> You offer lots of pointers to information. But that's different.

Exactly. There are *lots* of information processing standards out
there, and lots of things we *could* provide natively that simply
aren't worth the hassle since folks that care can provide them as
"after market addons" for the audiences that considers them relevant.

For example, a few things that can matter to different audiences are:

- SPDX (Software Package Data Exchange) identifiers for licenses
- CPE (Common Product Enumeration) and SWID (Software Identification)
tags for published software
- DOI (Digital Object Identifier) tags for citation purposes
- Common Criteria certification for software supply chains

I don't push for these upstream in distutils-sig not because I don't
think they're important in general, but because I *don't think they're
a priority for distutils-sig*. If you're teaching Python to school
students, or teaching engineers and scientists how to better analyse
their own data, or building a web service for yourself or your
employer, these kinds of things simply don't matter.

The end users that care about them are well-positioned to tackle them
on their own (or pay other organisations to do it for them), and
because they span arbitrary publishing communities anyway, it doesn't
really matter all that much if any given publishing community
participates directly in the process (the only real beneficiaries are
the intermediaries that actively blur the distinctions between the
cooperative communities and the recalcitrant ones).

> Anyway, let's just agree to differ - I can skip your mails if they
> aren't helpful to me, and you don't need to bother about the fact that
> you're not getting your points across to me.

I consider it fairly important that we have a reasonably common
understanding of the target userbase for direct consumption of PyPI
data, and what we expect to be supplied as third party services. It's
also important that we have a shared understanding of how to
constructively frame proposals for change.

For the former, the Semantic Web, and folks that care about Semantic
Web concepts like "Linked Data" in the abstract sense are not part of
our primary audience. We don't go out of our way to make their lives
difficult, but "it makes semantic analysis easier" also isn't a
compelling rationale for change.

For the latter, some variants of constructive proposals look like:

- "this kind of user has this kind of problem and this proposed
solution will help mitigate it this way (and, by the way, here's an
existing standard we can use)"
- "this feature exists in , it's really
valuable to users for , how about we offer it by
default?"
- "I wrote  for myself, and I think it would also help others
for , can you help me make it more widely known and
available?"

They don't look like "Here's a bunch of technologies and organisations
that exist on the internet that may in some way potentially be
relevant to the management of a software distribution network", and
nor does it look like "This data modeling standard exists, so we
should use it, even though it doesn't actually simplify our lives or
our users' lives in any way, and in fact makes them more complicated".

> Who knows, one day I
> might find the time to look into JSON-LD, at which point I may or may
> not understand why you think it's such a useful tool for solving all
> these problems (in spite of the fact that no-one else seems to think
> the same...)

I *have* looked at JSON-LD (based primarily on Wes's original
suggestions), both from the perspective of the Python packaging
ecosystem specifically, as well as my day job working on software
supply chain management.

My verdict was that for managing a dependency graph implementation, it
ends up in the category of technologies that qualify as "interesting,
but not helpful". In many ways, it's the urllib2 of data linking -
just as urllib2 gives you a URL handling framework which you can
configure to handle HTTP rather than just providing a HTTP-specific
interface the way requests does [1], JSON-LD gives you a data linking
framework, which you can then use to define links between your data,
rather than just linking the data directly in a domain-appropriate
fashion. Using a framework for the sake of using a framework rather
than out of a genuine engineering need doesn't tend to lead to good
software systems.

Wes seems to think that my perspective on this is born out of
ignorance, so repeatedly bringing it up may make me change my point of
view. However, our problems haven't changed, and the nature 

Re: [Distutils] Maintaining a curated set of Python packages

[Glyph Lefkowitz]

> On Dec 15, 2016, at 6:39 AM, Donald Stufft  wrote:
> 
> 
>> On Dec 15, 2016, at 9:35 AM, Steve Dower > > wrote:
>> 
>> The "curated package sets" on PyPI idea sounds a bit like Steam's curator 
>> lists, which I like to think of as Twitter for game reviews. You can follow 
>> a curator to see their comments on particular games, and the most popular 
>> curators have their comments appear on the actual listings too.
>> 
>> Might be interesting to see how something like that worked for PyPI, though 
>> the initial investment is pretty high. (It doesn't solve the coherent bundle 
>> problem either, just the discovery of good libraries problem.)
>> 
> 
> Theoretically we could allow people to not just select packages, but also 
> package specifiers for their “curated package set”, so instead of saying 
> “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we 
> really wanted to get slick we could even provide a requirements.txt file 
> format, and have people able to install the entire set by doing something 
> like:
> 
> $ pip install -r 
> https://pypi.org/sets/dstufft/my-cool-set/requirements.txt 
> 

Can't people already do this by publishing a package that just depends on their 
whole 'package set'?

-glyph

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Paul Moore]

On 15 December 2016 at 19:13, Wes Turner  wrote:
>> Just to add my POV, I also find your posts unhelpful, Wes. There's not
>> enough information for me to evaluate what you say, and you offer no
>> actual solutions to what's being discussed.
>
>
> I could quote myself suggesting solutions in this thread, if you like?

You offer lots of pointers to information. But that's different.

Anyway, let's just agree to differ - I can skip your mails if they
aren't helpful to me, and you don't need to bother about the fact that
you're not getting your points across to me. Who knows, one day I
might find the time to look into JSON-LD, at which point I may or may
not understand why you think it's such a useful tool for solving all
these problems (in spite of the fact that no-one else seems to think
the same...)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Thursday, December 15, 2016, Paul Moore  wrote:

> On 15 December 2016 at 15:58, Nick Coghlan  > wrote:
> > On 16 December 2016 at 01:38, Wes Turner  > wrote:
> >> On Thursday, December 15, 2016, Nick Coghlan  > wrote:
> >>> This answer hasn't changed the last dozen times you've brought up
> >>> JSON-LD. It isn't *going* to change. So please stop bringing it up.
> >>
> >>
> >> No, the problem is the same; and solving it (joining user-specific
> package
> >> metadata with central repository metadata on a common URI) with web
> >> standards is the best approach.
> >
> > Then do what Donald did with crate.io: go develop your own PyPI
> > competitor that uses your new and better approach to prove your point.
> >
> > As things stand, you're just generating noise and link spam on the
> > list, and asking politely for you to stop it doesn't appear to be
> > working
>
> Just to add my POV, I also find your posts unhelpful, Wes. There's not
> enough information for me to evaluate what you say, and you offer no
> actual solutions to what's being discussed.


I could quote myself suggesting solutions in this thread, if you like?


>
> As Nick says, if you can demonstrate what you're suggesting via a
> prototype implementation, that might help. But endless posts of links
> to standards documents that I have neither the time nor the
> inclination to read are not useful.


I'm suggesting that the solution here is to create version-controlled
collections of resources with metadata.

With schema.org types, those are Collection s of CreativeWork s. (Or, e.g.
SoftwareApplicationCollection s of SoftwareApplication s)

With Django, one would create a ListView generic view for the given
resource type; and then define e.g. an api.py for DRF (Django REST
Framework), and an index.py for Haystack (Elastic search).

A Pyramid implementation would be similar; with additional tests and code
just like Donald.

[Links omitted because you can now copy/paste keywords into the search
engines yourself; because you're unable to scroll past citations to find
the content you're looking for.]

DWBP and BP-LD would be useful reading for Warehouse API developers seeking
to add additional functionality to support the aforementioned use cases
with maximally-compatible linked open data.

Again,
Have a good day.


> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Thursday, December 15, 2016, Nick Coghlan  wrote:

> On 16 December 2016 at 01:38, Wes Turner  > wrote:
> > On Thursday, December 15, 2016, Nick Coghlan  > wrote:
> >> This answer hasn't changed the last dozen times you've brought up
> >> JSON-LD. It isn't *going* to change. So please stop bringing it up.
> >
> >
> > No, the problem is the same; and solving it (joining user-specific
> package
> > metadata with central repository metadata on a common URI) with web
> > standards is the best approach.
>
> Then do what Donald did with crate.io: go develop your own PyPI
> competitor that uses your new and better approach to prove your point.
>
> As things stand, you're just generating noise and link spam on the
> list, and asking politely for you to stop it doesn't appear to be
> working


I've intentionally included the URLs which support my position; none of
which do I have any commercial interest in. This is not link spam, this is
me explaining your problem to you in order to save time and money.

There are your:
- existing solutions that work today
- types (web standard types)
- use cases
- server load engineering challenges (web apps must cache)

Have a good day.


> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com    |   Brisbane,
> Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Paul Moore]

On 15 December 2016 at 15:58, Nick Coghlan  wrote:
> On 16 December 2016 at 01:38, Wes Turner  wrote:
>> On Thursday, December 15, 2016, Nick Coghlan  wrote:
>>> This answer hasn't changed the last dozen times you've brought up
>>> JSON-LD. It isn't *going* to change. So please stop bringing it up.
>>
>>
>> No, the problem is the same; and solving it (joining user-specific package
>> metadata with central repository metadata on a common URI) with web
>> standards is the best approach.
>
> Then do what Donald did with crate.io: go develop your own PyPI
> competitor that uses your new and better approach to prove your point.
>
> As things stand, you're just generating noise and link spam on the
> list, and asking politely for you to stop it doesn't appear to be
> working

Just to add my POV, I also find your posts unhelpful, Wes. There's not
enough information for me to evaluate what you say, and you offer no
actual solutions to what's being discussed.

As Nick says, if you can demonstrate what you're suggesting via a
prototype implementation, that might help. But endless posts of links
to standards documents that I have neither the time nor the
inclination to read are not useful.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Nick Coghlan]

On 16 December 2016 at 01:38, Wes Turner  wrote:
> On Thursday, December 15, 2016, Nick Coghlan  wrote:
>> This answer hasn't changed the last dozen times you've brought up
>> JSON-LD. It isn't *going* to change. So please stop bringing it up.
>
>
> No, the problem is the same; and solving it (joining user-specific package
> metadata with central repository metadata on a common URI) with web
> standards is the best approach.

Then do what Donald did with crate.io: go develop your own PyPI
competitor that uses your new and better approach to prove your point.

As things stand, you're just generating noise and link spam on the
list, and asking politely for you to stop it doesn't appear to be
working

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Thursday, December 15, 2016, Wes Turner  wrote:

>
>
> On Thursday, December 15, 2016, Nick Coghlan  > wrote:
>
>> On 16 December 2016 at 00:39, Donald Stufft  wrote:
>> > Theoretically we could allow people to not just select packages, but
>> also
>> > package specifiers for their “curated package set”, so instead of saying
>> > “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we
>> > really wanted to get slick we could even provide a requirements.txt file
>> > format, and have people able to install the entire set by doing
>> something
>> > like:
>> >
>> > $ pip install -r
>> > https://pypi.org/sets/dstufft/my-cool-set/requirements.txt
>>
>> CurseGaming provide addon managers for a variety of game addons
>> (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is
>> one of the ways they make it useful to have an account on the site
>> even if you don't publish any addons of your own. Even if you don't
>> make them public, you can still use them to sync your addon sets
>> between different machines.
>>
>> In the context of Python, where I can see this kind of thing being
>> potentially useful is for folks to manage package sets that aren't
>> necessarily coupled to any specific project, but match the way they
>> *personally* work.
>>
>> - "These are the packages I like to have installed to --user"
>> - "These are the packages I use to start a CLI app"
>> - "These are the packages I use to start a web app"
>> - etc...
>
>
> Does a requirements.txt in a {git,} repo solve for this already?
>

Could you just generate a README.rst for a long_description from
requirements.txt (or requirements.in, or pipfile),
store that in a {git,} repository,
and use the existing pypi release versioning and upload machinery?

Or would it be more useful to surface the other graph edges on project
pages?


- These additional queries would be less burdensome as AJAX requests to
JSON REST API views. Project pages could continue to load without waiting
for these additional cached edge bundles (and interactionStatistic(s)) to
load:
  - "This SoftwarePackage is in the following n SoftwarePackageCollections,
with the following comments and reviews"
  - "This SoftwarePackage has received n UserLikes (through Warehouse)"



> A Collection contains (hasPart) CreativeWorks
>
> - https://schema.org/Collection
> - https://schema.org/hasPart
>
> RDFa and JSONLD representations do parse as ordered lists.
>
> SoftwarePackageCollection
> SoftwareApplicationCollection
>
>
>> It also provides a way for people to vote on projects that's a little
>> more meaningful than stars - projects that appear in a lot of personal
>> stack definitions are likely to be generally valuable (the closest
>> equivalent to that today is mining code repositories like GitHub for
>> requirements.txt files and seeing what people are using that way).
>
>
> https://schema.org/InteractionCounter > https://schema.org/UserLikes
>
> D: CreativeWork
> - https://schema.org/interactionCount is now
> - https://schema.org/interactionStatistic
>
> (These are write-heavy features: they would change the database load of
> Warehouse)
>
>
>>
>> So yeah, if folks interested in this were to add it to Warehouse (and
>> hence pypi.org), I think it would definitely be a valuable enhancement
>> to the overall ecosystem. "What needs to be implemented in order to be
>> able to shut down the legacy service at pypi.python.org?" is the
>> *PSF's* focus, but that doesn't mean it needs to be everyone's focus.
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Thursday, December 15, 2016, Nick Coghlan  wrote:

> On 16 December 2016 at 00:57, Wes Turner  > wrote:
> > This would be a graph. JSONLD?
> > #PEP426JSONLD:
> > - https://www.google.com/search?q=pep426jsonld
> > - https://github.com/pypa/interoperability-peps/issues/31
> >
> > With JSONLD, we could merge SoftwarePackage metadata with
> > SoftwarePackageCollection metadata (just throwing some types out there).
>
> Wes, JSON-LD is a metasystem used for descriptive analytics across
> mixed datasets, which *isn't a problem we have*. We have full
> authority over the data formats we care about, and the user needs that
> matter to distutils-sig are:
>
> - publishers of Python packages
> - consumers of Python packages
> - maintainers of the toolchain
>
> It would *absolutely* make sense for Semantic Web folks to get
> involved in the libraries.io project (either directly, or by building
> a separate service backed by the libraries.io data set) and seek to
> produce a global set of semantically linked data that spans not only
> dependencies within language ecosystems, but also dependencies between
> them. It *doesn't* make sense for every single language ecosystem to
> come up with its own unique spin on how to incorporate software
> packages into semantic web models, nor does it make sense to try to
> warp the Python packaging user experience to better meet the needs of
> taxonomists of knowledge.
>
> This answer hasn't changed the last dozen times you've brought up
> JSON-LD. It isn't *going* to change. So please stop bringing it up.


No, the problem is the same; and solving it (joining user-specific package
metadata with central repository metadata on a common URI) with web
standards is the best approach.


>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com    |   Brisbane,
> Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Thursday, December 15, 2016, Nick Coghlan  wrote:

> On 16 December 2016 at 00:39, Donald Stufft  > wrote:
> > Theoretically we could allow people to not just select packages, but also
> > package specifiers for their “curated package set”, so instead of saying
> > “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we
> > really wanted to get slick we could even provide a requirements.txt file
> > format, and have people able to install the entire set by doing something
> > like:
> >
> > $ pip install -r
> > https://pypi.org/sets/dstufft/my-cool-set/requirements.txt
>
> CurseGaming provide addon managers for a variety of game addons
> (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is
> one of the ways they make it useful to have an account on the site
> even if you don't publish any addons of your own. Even if you don't
> make them public, you can still use them to sync your addon sets
> between different machines.
>
> In the context of Python, where I can see this kind of thing being
> potentially useful is for folks to manage package sets that aren't
> necessarily coupled to any specific project, but match the way they
> *personally* work.
>
> - "These are the packages I like to have installed to --user"
> - "These are the packages I use to start a CLI app"
> - "These are the packages I use to start a web app"
> - etc...


Does a requirements.txt in a {git,} repo solve for this already?

A Collection contains (hasPart) CreativeWorks

- https://schema.org/Collection
- https://schema.org/hasPart

RDFa and JSONLD representations do parse as ordered lists.

SoftwarePackageCollection
SoftwareApplicationCollection


> It also provides a way for people to vote on projects that's a little
> more meaningful than stars - projects that appear in a lot of personal
> stack definitions are likely to be generally valuable (the closest
> equivalent to that today is mining code repositories like GitHub for
> requirements.txt files and seeing what people are using that way).


https://schema.org/InteractionCounter > https://schema.org/UserLikes

D: CreativeWork
- https://schema.org/interactionCount is now
- https://schema.org/interactionStatistic

(These are write-heavy features: they would change the database load of
Warehouse)


>
> So yeah, if folks interested in this were to add it to Warehouse (and
> hence pypi.org), I think it would definitely be a valuable enhancement
> to the overall ecosystem. "What needs to be implemented in order to be
> able to shut down the legacy service at pypi.python.org?" is the
> *PSF's* focus, but that doesn't mean it needs to be everyone's focus.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com    |   Brisbane,
> Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org 
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Nick Coghlan]

On 16 December 2016 at 00:57, Wes Turner  wrote:
> This would be a graph. JSONLD?
> #PEP426JSONLD:
> - https://www.google.com/search?q=pep426jsonld
> - https://github.com/pypa/interoperability-peps/issues/31
>
> With JSONLD, we could merge SoftwarePackage metadata with
> SoftwarePackageCollection metadata (just throwing some types out there).

Wes, JSON-LD is a metasystem used for descriptive analytics across
mixed datasets, which *isn't a problem we have*. We have full
authority over the data formats we care about, and the user needs that
matter to distutils-sig are:

- publishers of Python packages
- consumers of Python packages
- maintainers of the toolchain

It would *absolutely* make sense for Semantic Web folks to get
involved in the libraries.io project (either directly, or by building
a separate service backed by the libraries.io data set) and seek to
produce a global set of semantically linked data that spans not only
dependencies within language ecosystems, but also dependencies between
them. It *doesn't* make sense for every single language ecosystem to
come up with its own unique spin on how to incorporate software
packages into semantic web models, nor does it make sense to try to
warp the Python packaging user experience to better meet the needs of
taxonomists of knowledge.

This answer hasn't changed the last dozen times you've brought up
JSON-LD. It isn't *going* to change. So please stop bringing it up.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Freddy Rietdijk]

> Theoretically we could allow people to not just select packages, but also
package specifiers for their “curated package set”, so instead of saying
“requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we
really wanted to get slick we could even provide a requirements.txt file
format, and have people able to install the entire set by doing something
like:
   $ pip install -r https://pypi.org/sets/dstufft/
my-cool-set/requirements.txt

What you want to be able to do is install a set of (abstract) packages
based on a curated set of (concrete) packages. Doing that you just need a
file with your concrete dependencies, and now we're back at pipfile and
https://github.com/pypa/pipfile/issues/10#issuecomment-262229620



On Thu, Dec 15, 2016 at 4:10 PM, Nick Coghlan  wrote:

> On 16 December 2016 at 00:39, Donald Stufft  wrote:
> > Theoretically we could allow people to not just select packages, but also
> > package specifiers for their “curated package set”, so instead of saying
> > “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we
> > really wanted to get slick we could even provide a requirements.txt file
> > format, and have people able to install the entire set by doing something
> > like:
> >
> > $ pip install -r
> > https://pypi.org/sets/dstufft/my-cool-set/requirements.txt
>
> CurseGaming provide addon managers for a variety of game addons
> (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is
> one of the ways they make it useful to have an account on the site
> even if you don't publish any addons of your own. Even if you don't
> make them public, you can still use them to sync your addon sets
> between different machines.
>
> In the context of Python, where I can see this kind of thing being
> potentially useful is for folks to manage package sets that aren't
> necessarily coupled to any specific project, but match the way they
> *personally* work.
>
> - "These are the packages I like to have installed to --user"
> - "These are the packages I use to start a CLI app"
> - "These are the packages I use to start a web app"
> - etc...
>
> It also provides a way for people to vote on projects that's a little
> more meaningful than stars - projects that appear in a lot of personal
> stack definitions are likely to be generally valuable (the closest
> equivalent to that today is mining code repositories like GitHub for
> requirements.txt files and seeing what people are using that way).
>
> So yeah, if folks interested in this were to add it to Warehouse (and
> hence pypi.org), I think it would definitely be a valuable enhancement
> to the overall ecosystem. "What needs to be implemented in order to be
> able to shut down the legacy service at pypi.python.org?" is the
> *PSF's* focus, but that doesn't mean it needs to be everyone's focus.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Thursday, December 15, 2016, Wes Turner  wrote:

>
>
> On Thursday, December 15, 2016, Donald Stufft  > wrote:
>
>>
>> On Dec 15, 2016, at 9:35 AM, Steve Dower  wrote:
>>
>> The "curated package sets" on PyPI idea sounds a bit like Steam's curator
>> lists, which I like to think of as Twitter for game reviews. You can follow
>> a curator to see their comments on particular games, and the most popular
>> curators have their comments appear on the actual listings too.
>>
>> Might be interesting to see how something like that worked for PyPI,
>> though the initial investment is pretty high. (It doesn't solve the
>> coherent bundle problem either, just the discovery of good libraries
>> problem.)
>>
>>
>> Theoretically we could allow people to not just select packages, but also
>> package specifiers for their “curated package set”, so instead of saying
>> “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we
>> really wanted to get slick we could even provide a requirements.txt file
>> format, and have people able to install the entire set by doing something
>> like:
>>
>> $ pip install -r https://pypi.org/sets/dstufft/
>> my-cool-set/requirements.txt
>>
>
> With version control?
>
> $ pip install -r https://pypi.org/sets/dstufft/my-cool-set/abcd123/
> requirements.txt
> 
>
> $ pip install -r https://pypi.org/sets/dstufft/my-cool-set/v0.0.1/
> requirements.txt
> 
>
> This would be a graph. JSONLD?
> #PEP426JSONLD:
> - https://www.google.com/search?q=pep426jsonld
> - https://github.com/pypa/interoperability-peps/issues/31
>
> With JSONLD, we could merge SoftwarePackage metadata with
> SoftwarePackageCollection metadata (just throwing some types out there).
>
> A http://schema.org/SoftwareApplication is a http://schema.org/
> CreativeWork .
>
> http://schema.org/softwareVersion
>
>
- https://pypi.org/project/ as the canonical project
http://schema.org/url

- There's almost certainly a transform of TOML to JSONLD (" TOMLLD ")
- There is a standardized transform of JSONLD to RDF
- YAMLLD is a stricter subset of YAML (because just OrderedDicts


>>
>>
>> —
>> Donald Stufft
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Nick Coghlan]

On 16 December 2016 at 00:39, Donald Stufft  wrote:
> Theoretically we could allow people to not just select packages, but also
> package specifiers for their “curated package set”, so instead of saying
> “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we
> really wanted to get slick we could even provide a requirements.txt file
> format, and have people able to install the entire set by doing something
> like:
>
> $ pip install -r
> https://pypi.org/sets/dstufft/my-cool-set/requirements.txt

CurseGaming provide addon managers for a variety of game addons
(Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is
one of the ways they make it useful to have an account on the site
even if you don't publish any addons of your own. Even if you don't
make them public, you can still use them to sync your addon sets
between different machines.

In the context of Python, where I can see this kind of thing being
potentially useful is for folks to manage package sets that aren't
necessarily coupled to any specific project, but match the way they
*personally* work.

- "These are the packages I like to have installed to --user"
- "These are the packages I use to start a CLI app"
- "These are the packages I use to start a web app"
- etc...

It also provides a way for people to vote on projects that's a little
more meaningful than stars - projects that appear in a lot of personal
stack definitions are likely to be generally valuable (the closest
equivalent to that today is mining code repositories like GitHub for
requirements.txt files and seeing what people are using that way).

So yeah, if folks interested in this were to add it to Warehouse (and
hence pypi.org), I think it would definitely be a valuable enhancement
to the overall ecosystem. "What needs to be implemented in order to be
able to shut down the legacy service at pypi.python.org?" is the
*PSF's* focus, but that doesn't mean it needs to be everyone's focus.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Thursday, December 15, 2016, Donald Stufft  wrote:

>
> On Dec 15, 2016, at 9:35 AM, Steve Dower  > wrote:
>
> The "curated package sets" on PyPI idea sounds a bit like Steam's curator
> lists, which I like to think of as Twitter for game reviews. You can follow
> a curator to see their comments on particular games, and the most popular
> curators have their comments appear on the actual listings too.
>
> Might be interesting to see how something like that worked for PyPI,
> though the initial investment is pretty high. (It doesn't solve the
> coherent bundle problem either, just the discovery of good libraries
> problem.)
>
>
> Theoretically we could allow people to not just select packages, but also
> package specifiers for their “curated package set”, so instead of saying
> “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we
> really wanted to get slick we could even provide a requirements.txt file
> format, and have people able to install the entire set by doing something
> like:
>
> $ pip install -r https://pypi.org/sets/dstufft/
> my-cool-set/requirements.txt
>

With version control?

$ pip install -r https://pypi.org/sets/dstufft/
my-cool-set/abcd123/requirements.txt


$ pip install -r https://pypi.org/sets/dstufft/
my-cool-set/v0.0.1/requirements.txt


This would be a graph. JSONLD?
#PEP426JSONLD:
- https://www.google.com/search?q=pep426jsonld
- https://github.com/pypa/interoperability-peps/issues/31

With JSONLD, we could merge SoftwarePackage metadata with
SoftwarePackageCollection metadata (just throwing some types out there).

A http://schema.org/SoftwareApplication is a http://schema.org/CreativeWork
.

http://schema.org/softwareVersion


>
>
> —
> Donald Stufft
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Steve Dower]

The "curated package sets" on PyPI idea sounds a bit like Steam's curator 
lists, which I like to think of as Twitter for game reviews. You can follow a 
curator to see their comments on particular games, and the most popular 
curators have their comments appear on the actual listings too.

Might be interesting to see how something like that worked for PyPI, though the 
initial investment is pretty high. (It doesn't solve the coherent bundle 
problem either, just the discovery of good libraries problem.)

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" 
Sent: ‎12/‎15/‎2016 4:21
To: "Freddy Rietdijk" 
Cc: "DistUtils mailing list" ; "Barry Warsaw" 

Subject: Re: [Distutils] Maintaining a curated set of Python packages



On Dec 15, 2016, at 7:13 AM, Freddy Rietdijk  wrote:


> Putting the conclusion first, I do see value in better publicising

> "Recommended libraries" based on some automated criteria like:


Yes, we should recommend third-party libraries in a trusted place like the 
documentation of CPython. The amount of packages that are available can be 
overwhelming. Yet, defining a set of packages that are recommended, and perhaps 
working together, is still far from defining an exact set of packages that are 
known to work together, something which I proposed here.




We could theoretically bake this into PyPI itself, though I’m not sure if that 
makes sense.


We could also probably bake something like “curated package sets” into PyPI 
where individual users (or groups of users) can create their own view of PyPI 
for people to use, while still relying on all of the infrastructure of PyPI. 
Although I’m not sure that makes any sense either.


—
Donald Stufft___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Donald Stufft]

> On Dec 15, 2016, at 9:35 AM, Steve Dower  wrote:
> 
> The "curated package sets" on PyPI idea sounds a bit like Steam's curator 
> lists, which I like to think of as Twitter for game reviews. You can follow a 
> curator to see their comments on particular games, and the most popular 
> curators have their comments appear on the actual listings too.
> 
> Might be interesting to see how something like that worked for PyPI, though 
> the initial investment is pretty high. (It doesn't solve the coherent bundle 
> problem either, just the discovery of good libraries problem.)
> 

Theoretically we could allow people to not just select packages, but also 
package specifiers for their “curated package set”, so instead of saying 
“requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really 
wanted to get slick we could even provide a requirements.txt file format, and 
have people able to install the entire set by doing something like:

$ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Donald Stufft]

> On Dec 15, 2016, at 7:13 AM, Freddy Rietdijk  wrote:
> 
> > Putting the conclusion first, I do see value in better publicising
> > "Recommended libraries" based on some automated criteria like:
> 
> Yes, we should recommend third-party libraries in a trusted place like the 
> documentation of CPython. The amount of packages that are available can be 
> overwhelming. Yet, defining a set of packages that are recommended, and 
> perhaps working together, is still far from defining an exact set of packages 
> that are known to work together, something which I proposed here.



We could theoretically bake this into PyPI itself, though I’m not sure if that 
makes sense.

We could also probably bake something like “curated package sets” into PyPI 
where individual users (or groups of users) can create their own view of PyPI 
for people to use, while still relying on all of the infrastructure of PyPI. 
Although I’m not sure that makes any sense either.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Freddy Rietdijk]

It's interesting to read about how other distributions upgrade their
package sets. In Nixpkgs most packages are updated manually. Some
frameworks/languages provide their dependencies declarative, in which case
it becomes 'straightforward' to include whole package sets, like in the
case of Haskell. Some expressions need to be overridden manually because
e.g. they require certain system libraries. It's manual work, but not that
much. This is what I would like to see for Python as well.

The Python packages we still update manually although we have tools to
automate most of it. The reason for not using those tools is because a) it
means evaluating or building parts of the packages to get the dependencies
and b) too often upstream pins versions of dependencies which turns out to
be entirely unnecessary, and would therefore prevent an upgrade. Even so,
we have over 1500 Python packages per interpreter version that according to
our CI seem to work together. We do only build on 3 architectures (i386,
amd64 and darwin/osx). Compatibility with the latter is sometimes an issue
because its guessing what Apple has changed when releasing a new version.

> I'm not sure how useful it would be higher up the food chain, since those
contexts will be
> different enough to cause both false positives and false negatives.  And
it
> does often take quite a bit of focused engineering effort to monitor
packages
> which don't promote (something we want to automate),

In my experience the manual work that typically needs to be done is a)
making available Python dependencies, b) unpinning versions when
unnecessary and report upstream, and c) making sure the package finds the
system libraries. Issues with packages that cannot be upgraded because of a
version of a system dependency I haven't yet encountered. In my proposal a)
and b) would be fixed by the curated package set.

> https://github.com/nvie/pip-tools :
> - requirements.in -> pip-compile -> requirements.txt (~pipfile.lock)

Yep, there are tools, which get the job done when developing and using a
set of packages. Now, you want to deploy your app. You can't use your
requirements.txt on a Linux distribution because they have a curated set of
packages which is typically different from your set (although maybe they do
provide tools to package the versions you need). But, you can choose to use
a second package manager, pip or conda. System dependencies? That's
something conda can somewhat take of. Problem solved you would say, except
now you have multiple package managers that you need.

> Practically, a developer would want a subset of the given known-good-set
(and then additional packages), so:
>
> - fork/copy requirements--MM-REV--.txt
> - #comment out unused deps
> - add '-r addl-requirements.txt'

See the link I shared earlier on how this is already done with Haskell and
stack.yaml and how it could be used with `pipfile`
https://github.com/pypa/pipfile/issues/10#issuecomment-262229620

> Putting the conclusion first, I do see value in better publicising
> "Recommended libraries" based on some automated criteria like:

Yes, we should recommend third-party libraries in a trusted place like the
documentation of CPython. The amount of packages that are available can be
overwhelming. Yet, defining a set of packages that are recommended, and
perhaps working together, is still far from defining an exact set of
packages that are known to work together, something which I proposed here.

> As pointed out by others, there are external groups doing "curating".
conda-forge is one such project, so I'll comment from that perspective

I haven't used conda in a long time, and conda-forge didn't exist back
then. I see versions are pinned, but versions of dependencies sometimes as
well. If I choose to install *all* the packages available via conda-forge,
will I get a fixed package set, or will the SAT-solver try to find a
working set (and possibly fail at it)? I hope it is the former, since if it
is the latter then it is not curated in how I meant it.


On Thu, Dec 15, 2016 at 6:22 AM, Nick Coghlan  wrote:

> On 15 December 2016 at 03:41, Chris Barker  wrote:
> [Barry wrote]
> >> Ubuntu has an elaborate automated system for testing some dimension of
> >> compatibility issues between packages, not just Python packages.  Debian
> >> has
> >> the same system but isn't gated on the results.
> >
> > This brings up the larger issue -- PyPi is inherently different than
> these
> > efforts -- PyPi has always been about each package author maintaining
> their
> > own package -- Linux distros and conda-forge, and ??? all have a small
> set
> > of core contributions that do the package maintenance.
>
> Fedora at least has just shy of 1900 people in the "packager" group,
> so I don't know that "small" is the right word in absolute terms :)
>
> However, relatively speaking, even a packager group that size is still
> an order of magnitude smaller than the 30k+