[Python-ideas] Re: Add an export keyword to better manage __all__

[Oscar Benjamin]

On Mon, 12 Apr 2021 at 21:27, Brendan Barnwell  wrote:
>
> On 2021-04-12 03:13, Stéfane Fermigier wrote:
> > The public API of a library is the one which is documented as
> > such.
> >
> > Right, except that in practice:
> >
> > 1) Many useful libraries are not documented or properly documented.
>
> Then they're buggy.  I'm not convinced by the argument that "in
> practice" people do things they shouldn't do and that therefore we
> should encourage them to do more of that.

A library can be useful and buggy at the same time or it can be useful
but have some parts that are buggy and less useful.

Ideally if some code is (potentially) useful but buggy then someone
would come along and make it less buggy. If one of the deficiencies of
the code to be improved is that it does not have a clear distinction
between public and internal API then that task can be made much more
difficult.

> > 2) People don't read the docs (at least not always, and/or not in details).
>
> Insofar as someone relies on behavior other than that given in the
> docs, they are being foolish.  Again, I'm not convinced by the argument
> that "in practice" people do foolish things and that therefore we should
> encourage them to do more of that.

Maybe the docs were not so clear or maybe the library just has a lot
of users or maybe some combination of the two. Either way it's much
better for everyone involved if the code can be improved or extended
without breaking things for people who are already using it.
Standardising on simple practice that helps to make that happen is no
bad thing.


Oscar
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HYMDSAFXYAOKY62DSRBUI5QJ4IZDEGVF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add an export keyword to better manage __all__

[Guido van Rossum]

Wow, super helpful response.

On Mon, Apr 12, 2021 at 1:26 PM Brendan Barnwell 
wrote:

> On 2021-04-12 03:13, Stéfane Fermigier wrote:
> > The public API of a library is the one which is documented as
> > such.
> >
> >
> > Right, except that in practice:
> >
> > 1) Many useful libraries are not documented or properly documented.
>
> Then they're buggy.  I'm not convinced by the argument that "in
> practice" people do things they shouldn't do and that therefore we
> should encourage them to do more of that.
>
> > 2) People don't read the docs (at least not always, and/or not in
> details).
>
> Insofar as someone relies on behavior other than that given in the
> docs, they are being foolish.  Again, I'm not convinced by the argument
> that "in practice" people do foolish things and that therefore we should
> encourage them to do more of that.
>
> --
> Brendan Barnwell
> "Do not follow where the path may lead.  Go, instead, where there is no
> path, and leave a trail."
> --author unknown
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/DWI2MI3FRPRNOC7AS6CC3YBWLOOVZYZA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/F57J672KTTK6FRA7ITRUVTRYBF2A3P57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add an export keyword to better manage __all__

[Brendan Barnwell]

On 2021-04-12 03:13, Stéfane Fermigier wrote:

The public API of a library is the one which is documented as
such.


Right, except that in practice:

1) Many useful libraries are not documented or properly documented.


	Then they're buggy.  I'm not convinced by the argument that "in 
practice" people do things they shouldn't do and that therefore we 
should encourage them to do more of that.



2) People don't read the docs (at least not always, and/or not in details).


	Insofar as someone relies on behavior other than that given in the 
docs, they are being foolish.  Again, I'm not convinced by the argument 
that "in practice" people do foolish things and that therefore we should 
encourage them to do more of that.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DWI2MI3FRPRNOC7AS6CC3YBWLOOVZYZA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Revive: PEP 396 -- Module Version Numbers ?

[Paul Moore]

On Mon, 12 Apr 2021 at 19:52, Christopher Barker  wrote:
>
> Over the years, I've seen __version__ used very broadly but not *quite* in 
> all packages. I've always known it was a convention, not a requirement. But 
> it turns out it's not even a "official" convention.
>
> But it really would be nice if there was a consistent way that I could count 
> on to get the version of a package at run time.
>
> Turns out this was suggested in PEP 396 -- and deferred almost 13 years ago!
>
> https://www.python.org/dev/peps/pep-0396/
>
> In the status note, it says:
>
> """
> Further exploration of the concepts covered in this PEP has been deferred for 
> lack of a current champion interested in promoting the goals of the PEP and 
> collecting and incorporating feedback, and with sufficient available time to 
> do so effectively.
> """
>
> Well, I may be willing to be that champion, if a core dev is willing to 
> sponsor, and I see some interest from this group.
>
> And, well, after 13 years, we've seen __version__ be very broadly, though 
> certainly not universally used.
>
> Honestly, I haven't looked to see to what extent setuptools supports it, but 
> will, of course, do so if folks think this is worth pursuing.
>
> Or maybe this is a settled issue in setuptools, and we just need to change 
> the status of the PEP.
>
> For my part, I FAR prefer the version info to be embedded in the code of the 
> package in a simple way, rather than hiding among the setuptools/egg/pip 
> metadata, and requiring setuptools.pkg_resources to get a version at runtime.
>
> I note that PEP8 uses __version__ as an example of a "module level dunder" -- 
> but only suggests where it should be put, not how it be used :-)
>
> Of course, there will be a need to update the PEP to match current practice 
> (and setuptools and pip)

Having a __version__ attribute is fairly common these days, but
definitely not universal even now. So the PEP still has a place, IMO.
But a lot has changed in the last 13 years. Python packaging is built
very much on packages having versions these days, so the
*distribution* version (as covered in
https://packaging.python.org/specifications/core-metadata/) is
essential. And with importlib.metadata, that version is introspectable
at runtime. But that's different from a *package* version as exposed
via __version__. And I suspect there's still some debate over whether
we need the two, so I wouldn't assume the PEP is self-evidently a good
thing.

There's a bunch of straightforward updating that needs to be done (all
of the PEPs that this one references have since been superseded, and
in the case off package metadata, it's no longer standardised via a
PEP but at https://packaging.python.org/specifications/core-metadata/).

Also, you should look at
https://packaging.python.org/guides/single-sourcing-package-version/.
The whole question of how to derive the distribution version from the
package version is quite complex, with a number of common solutions,
and also a number of people who will argue that you *shouldn't*
single-source, but should just copy the value into the two places. The
section of the PEP that covers this needs rewriting, and possibly even
omitting, as there is no "standard" answer.

As another data point, flit *requires* that the package has a
__version__ attribute.

FWIW, I'm personally ambivalent on the subject - having a __version__
feels like it's probably a good thing, but it's very rare that I
actually care in practice as a package consumer, so as a package
creator it feels a bit like boilerplate that I add "for the sake of
it".

I'd suggest keeping the scope of the PEP fairly limited - opinions
vary in this area, and are held fairly strongly, so you'd stand more
chance of getting something accepted if you keep things focused on the
basics.

> Or should this be brought up on The Distutils-SIG list?

(That's the packaging category on Discourse these days).

Honestly, I don't really know. It *could* be a packaging
interoperability standard, but the rules it includes about stdlib
modules push it into core python territory. And packaging standards
tend to be more about distribution-level metadata than package-level.
I suspect the best thing to do would be to check with the SC on their
view, and if they want to toss it in my direction, I'm happy to make
the decision.

Sorry, but I'd rather not be a sponsor for this, as I'm pretty busy
with other things at the moment. But I hope this helps.

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/W6RDBPPM77QQRNJ3YI3MFTXSMSCRD6PW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Revive: PEP 396 -- Module Version Numbers ?

[Thomas Grainger]

I prefer to use importlib.metadata.version("dist-name")

On Mon, 12 Apr 2021, 19:51 Christopher Barker,  wrote:

> Over the years, I've seen __version__ used very broadly but not *quite* in
> all packages. I've always known it was a convention, not a requirement. But
> it turns out it's not even a "official" convention.
>
> But it really would be nice if there was a consistent way that I could
> count on to get the version of a package at run time.
>
> Turns out this was suggested in PEP 396 -- and deferred almost 13 years
> ago!
>
> https://www.python.org/dev/peps/pep-0396/
>
> In the status note, it says:
>
> """
> Further exploration of the concepts covered in this PEP has been deferred
> for lack of a current champion interested in promoting the goals of the PEP
> and collecting and incorporating feedback, and with sufficient available
> time to do so effectively.
> """
>
> Well, I may be willing to be that champion, if a core dev is willing to
> sponsor, and I see some interest from this group.
>
> And, well, after 13 years, we've seen __version__ be very broadly, though
> certainly not universally used.
>
> Honestly, I haven't looked to see to what extent setuptools supports it,
> but will, of course, do so if folks think this is worth pursuing.
>
> Or maybe this is a settled issue in setuptools, and we just need to change
> the status of the PEP.
>
> For my part, I FAR prefer the version info to be embedded in the code of
> the package in a simple way, rather than hiding among the
> setuptools/egg/pip metadata, and requiring setuptools.pkg_resources to get
> a version at runtime.
>
> I note that PEP8 uses __version__ as an example of a "module level dunder"
> -- but only suggests where it should be put, not how it be used :-)
>
> Of course, there will be a need to update the PEP to match current
> practice (and setuptools and pip)
>
> Or should this be brought up on The Distutils-SIG list?
>
> -CHB
>
> ,--
> Christopher Barker, PhD (Chris)
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/VP4JWZTXYXGUFI76UR6AIKTSCUK3ZM53/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3XJBGT3QNGDBVS5LUS4B5FLLXZECBCVC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add an export keyword to better manage __all__

[Neil Girdhar]

This is a great discussion.  However, there's one issue that seems to
be getting lost.  Consider this library structure:

test/lemon
test/lemon/__init__.py
test/module.py

where test/module.py is:

__all__ = ['lemon']
def lemon():
pass

and test/__init__.py is:

from .module import *
from .lemon import *

and test/lemon/__init__.py is empty.

What do you think test.lemon is?  It's not the function because it's
hidden by the eponymous package.  Also, test.module is exported
whether you wanted to export it or not.

This is the problem with any mechanism like __all__ or export.  I
think the main motivation behind the Jax team tucking everything in
_src is that it hides the package names and module names from the
public interface.

I think I'm going to start only using __all__ for collecting private
symbols into packages under a _src package.  For the external
interface, I'm going to explicitly import from dummy modules.  I'd
love to know if something better exists, but I don't think the export
keyword works because of the above problem.

Best,

Neil

On Mon, Apr 12, 2021 at 7:39 AM Joseph Martinot-Lagarde
 wrote:
>
> M.-A. Lemburg wrote:
> > The public API of a library is the one which is documented as
> > such. That's really all there is to it. Documentation is written
> > explicitly by the author of a package and languages provides
> > a lot more nuances than using some programmatic mechanism.
> I use __all__ with sphinx to automatically build my documentation, so __all__ 
> effectively becomes part of the documentation. I don't expect people to use 
> import *. Maybe for big projects there is a need to write all the 
> documentation by hand, but for small libraries using __all__ and 
> automatically build the documentation (along other pages like FAQ, HOWTO and 
> so on) is very powerful and makes a good documentation while assurint it's up 
> to date with the code.
> There are definitely use cases for __all__ outside of import *.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/ASZOJHLO2HDSQDDA2KQPTELUZOVNIL3C/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the Google 
> Groups "python-ideas" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/python-ideas/c2JTmIJnX5c/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> python-ideas+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/python-ideas/161822751700.5428.12873752338560477293%40mail.python.org.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TNGY6TTTQBX5QS62C24E7ZG6GEH3P7A6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Revive: PEP 396 -- Module Version Numbers ?

[Christopher Barker]

Over the years, I've seen __version__ used very broadly but not *quite* in
all packages. I've always known it was a convention, not a requirement. But
it turns out it's not even a "official" convention.

But it really would be nice if there was a consistent way that I could
count on to get the version of a package at run time.

Turns out this was suggested in PEP 396 -- and deferred almost 13 years ago!

https://www.python.org/dev/peps/pep-0396/

In the status note, it says:

"""
Further exploration of the concepts covered in this PEP has been deferred
for lack of a current champion interested in promoting the goals of the PEP
and collecting and incorporating feedback, and with sufficient available
time to do so effectively.
"""

Well, I may be willing to be that champion, if a core dev is willing to
sponsor, and I see some interest from this group.

And, well, after 13 years, we've seen __version__ be very broadly, though
certainly not universally used.

Honestly, I haven't looked to see to what extent setuptools supports it,
but will, of course, do so if folks think this is worth pursuing.

Or maybe this is a settled issue in setuptools, and we just need to change
the status of the PEP.

For my part, I FAR prefer the version info to be embedded in the code of
the package in a simple way, rather than hiding among the
setuptools/egg/pip metadata, and requiring setuptools.pkg_resources to get
a version at runtime.

I note that PEP8 uses __version__ as an example of a "module level dunder"
-- but only suggests where it should be put, not how it be used :-)

Of course, there will be a need to update the PEP to match current practice
(and setuptools and pip)

Or should this be brought up on The Distutils-SIG list?

-CHB

,--
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VP4JWZTXYXGUFI76UR6AIKTSCUK3ZM53/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add an export keyword to better manage __all__

[Joseph Martinot-Lagarde]

M.-A. Lemburg wrote:
> The public API of a library is the one which is documented as
> such. That's really all there is to it. Documentation is written
> explicitly by the author of a package and languages provides
> a lot more nuances than using some programmatic mechanism.
I use __all__ with sphinx to automatically build my documentation, so __all__ 
effectively becomes part of the documentation. I don't expect people to use 
import *. Maybe for big projects there is a need to write all the documentation 
by hand, but for small libraries using __all__ and automatically build the 
documentation (along other pages like FAQ, HOWTO and so on) is very powerful 
and makes a good documentation while assurint it's up to date with the code.
There are definitely use cases for __all__ outside of import *.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ASZOJHLO2HDSQDDA2KQPTELUZOVNIL3C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add an export keyword to better manage __all__

[Paul Moore]

On Mon, 12 Apr 2021 at 11:41, M.-A. Lemburg  wrote:

> > Right, except that in practice:
> >
> > 1) Many useful libraries are not documented or properly documented.
>
> In those cases, I'd argue that such libraries then do not really care
> for a specific public API either :-)
>
> > 2) People don't read the docs (at least not always, and/or not in details).
>
> Uhm, how can you write code against a package without knowing how it
> is used ?

Is the problem here that we're trying to apply a technical solution to
what is, in practice, a people problem? I don't think I've ever seen
(in any language) a system of declaring names as
public/private/whatever that substituted well for writing (and
reading!) good documentation... At best, hiding stuff makes people
work a bit harder to write bad code :-)

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NU23H5AOOCBGXJTEHLDXBXLYY6XOI6T6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add an export keyword to better manage __all__

[M.-A. Lemburg]

On 12.04.2021 12:13, Stéfane Fermigier wrote:
> 
> 
> On Mon, Apr 12, 2021 at 11:39 AM M.-A. Lemburg  > wrote:
> 
> I don't think that trying to do this programmatically via some
> kind of language construct is a good approach.
> 
> 
> It's not the only approach, but I argue that it's useful. Two use cases:
> 
> - IDE's that will nudge the developer towards importing the "right" objects.
> - Linters and runtime checkers that will issue warnings / errors and minimise
> the amount of human code auditing or reviewing.
  
When writing code against an API you typically read the documentation.
Without knowing which APIs you need or how they are used, a list of
public APIs won't really get you to running code :-)

> The public API of a library is the one which is documented as
> such. 
> 
> 
> Right, except that in practice:
> 
> 1) Many useful libraries are not documented or properly documented. 

In those cases, I'd argue that such libraries then do not really care
for a specific public API either :-)

> 2) People don't read the docs (at least not always, and/or not in details).
  
Uhm, how can you write code against a package without knowing how it
is used ?

> That's really all there is to it. Documentation is written
> explicitly by the author of a package and languages provides
> a lot more nuances than using some programmatic mechanism.
> 
> E.g. you may have situations where you have more than one level of
> documented APIs, e.g. a public stable one and an experimental
> one which is likely to change over time, or a high level one and
> a low-level one, where more care has to be applied in using it.
> 
> 
> Yes. This can be formalized by having more than one namespace for the Public 
> API.
  
I just listed some variants. If you want to capture all nuances,
you'd end up creating a mini-language specifically for defining
the public APIs... then why not just write this down in the
documentation rather than inventing a new DSL ?

> Or you have APIs which are deprecated but still public and you
> know that the APIs will go away in a future version, so you should
> probably not start using them in new code and plan for a refactoring
> of existing code.
> 
> 
> There are already conventions and tools for that, e.g. the @deprecated 
> decorator
> (https://pypi.org/project/Deprecated/).
  
Sure, but again: this is just another aspect to consider.

There can be plenty more, e.g. say some APIs are only available on
specific platforms, or may only be used when certain external
libraries are present or configurations are used.

Or let's say that a specific API only works on CPython, but not in
PyPy, or Jython.

And the platform your IDE is running on may not have those
requirements enabled or available.

Should those "public API" attributes then include all variants or
just the ones which do work on your platform ?

(playing devil's advocate here)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Apr 12 2021)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ADO573OG7IZBW4ROOCMDSMFD4EUHIT2J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add an export keyword to better manage __all__

[Stéfane Fermigier]

On Mon, Apr 12, 2021 at 11:39 AM M.-A. Lemburg  wrote:

> I don't think that trying to do this programmatically via some
> kind of language construct is a good approach.
>

It's not the only approach, but I argue that it's useful. Two use cases:

- IDE's that will nudge the developer towards importing the "right" objects.
- Linters and runtime checkers that will issue warnings / errors and
minimise the amount of human code auditing or reviewing.


> The public API of a library is the one which is documented as
> such.


Right, except that in practice:

1) Many useful libraries are not documented or properly documented.

2) People don't read the docs (at least not always, and/or not in details).


> That's really all there is to it. Documentation is written
> explicitly by the author of a package and languages provides
> a lot more nuances than using some programmatic mechanism.
>
> E.g. you may have situations where you have more than one level of
> documented APIs, e.g. a public stable one and an experimental
> one which is likely to change over time, or a high level one and
> a low-level one, where more care has to be applied in using it.
>

Yes. This can be formalized by having more than one namespace for the
Public API.


> Or you have APIs which are deprecated but still public and you
> know that the APIs will go away in a future version, so you should
> probably not start using them in new code and plan for a refactoring
> of existing code.
>

There are already conventions and tools for that, e.g. the @deprecated
decorator (https://pypi.org/project/Deprecated/).

  S.

-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ &
http://pydata.fr/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/YEELZYXKUGRQ5LWB6Y27BUXZFMSXOWTZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add an export keyword to better manage __all__

[M.-A. Lemburg]

I don't think that trying to do this programmatically via some
kind of language construct is a good approach.

The public API of a library is the one which is documented as
such. That's really all there is to it. Documentation is written
explicitly by the author of a package and languages provides
a lot more nuances than using some programmatic mechanism.

E.g. you may have situations where you have more than one level of
documented APIs, e.g. a public stable one and an experimental
one which is likely to change over time, or a high level one and
a low-level one, where more care has to be applied in using it.

Or you have APIs which are deprecated but still public and you
know that the APIs will go away in a future version, so you should
probably not start using them in new code and plan for a refactoring
of existing code.

Superimposing the extra meaning of __all__ = "public API", which has
been done at times, is misleading, esp. when using
"from module import *", for which __all__ was added, is frowned upon
unless needed in special use cases.

You'd really only define __all__ if you expect that "from module import *"
will be used, which you typically don't :-) ...

"Package authors may also decide not to support it, if they don’t
see a use for importing * from their package."
https://docs.python.org/3/tutorial/modules.html?highlight=__all__#importing-from-a-package

Aside: __all__ and namespace package don't work well together either -
they don't have __init__.py initializers.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Apr 12 2021)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HU4YO6HHB63EG4ZLPWLHO3W2W4GTSHO4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add an export keyword to better manage __all__

[Stéfane Fermigier]

On Sun, Apr 11, 2021 at 2:07 PM Oscar Benjamin 
wrote:

> On Sun, 11 Apr 2021 at 10:25, Stéfane Fermigier  wrote:
>
> > 2) What's a "public API" ? A particular library can have an internal API
> (with regular public functions and methods, etc,) but only willingly expose
> a subset of this API to its clients. __all__ doesn't help much here, as the
> tools I know of don't have this distinction in mind.
>
> I don't think that this distinction is so important in practice.

What matters is the boundary between different codebases. Within a codebase
> a team can agree whatever conventions they want because any time
> something is changed it can be changed everywhere at once in a single
> commit. What matters is communicating from project A to all downstream
> developers and users what can be expected to remain compatible across
> different versions of project A.
>

We agree that the most important API is the Public API, not the internal
API (except for very large scale libraries where this can become important).

But Python mostly supports (via
https://docs.python.org/3/tutorial/classes.html#tut-private or the _-prefix
naming convention) the notion of internal APIs, not of Public API.

> 3) A library author can decide that the public API of its library is the
> part exposed by its top-level namespace, e.g:
> >
> > from flask import Flask, resquest, ...
> >
> > where Flask, request, etc. are defined in subpackages of flask.
>
> This approach really doesn't scale.


It scales as long as it's expected that when you import one object from the
library, you will use most of the library.

In some cases (for instance: SQLAlchemy where you can want to use the
Engine but not the ORM) the library author(s) have already thought about
the issue and made several top-level namespaces.

See for instance:

https://github.com/sqlalchemy/sqlalchemy/blob/master/lib/sqlalchemy/__init__.py
https://github.com/sqlalchemy/sqlalchemy/blob/master/lib/sqlalchemy/orm/__init__.py
etc.

For one thing imports in Python
> have a run-time performance cost and if everything is imported at
> top-level like this then it implies that the top level __init__.py has
> imported every module in the codebase. Also organising around
> sub-packages and modules is much nicer for organising documentation,
> module level docstrings etc.
>
> > I guess this issue also comes from the lack of a proper way to define
> (either as a language-level construct or as a convention among developers
> and tools authors) a proper notion of a "public API", and, once again, I
> don't believe __all__ helps much with this issue.
> >
> > => For these use cases, a simple solution could be devised, that doesn't
> involve language-level changes but  needs a wide consensus among both
> library authors and tools authors:
> >
> > 1) Mark public API namespaces package with a special marker (for
> instance: __public_api__ = True).
> >
> > 2) Statical and runtime tools could be easily devised that raise a
> warning when:
> >
> > a) Such a marker is present in one or more modules of the package.
> >
> > b) and: one imports another module of the same package from another
> package.
> >
> > This is just a rough idea. Additional use cases could easily be added by
> adding other marker types.
> >
> > An alternative could be to use decorators (eg. @public like mentioned in
> another message), as long as we don't confuse "public = part of the public
> API of the library" with "public = not private to this particular module".
>
> I don't think __all__ helps either but for different reasons.


I think we agree for the same reason actually.


> Python makes everything implicitly public. What is actually needed is a way
> to clearly mark the internal code (which is most of the code in a
> large project) as being obviously private.
>

We agree that there needs to be a way to mark what is part of the public
API vs. what is the (library-)private implementation.

Being explicit about what is public vs. being explicit what is private are
two complementary approaches (as long as we agree that public vs. private
is a two-state choice, in this context).


> That's why I prefer the Jax approach of putting the implementation in
> a jax._src package.


That's another approach, which reminds me of some Java projects I have
worked on in the past.

First time I've seen it in a Python project, though, vs. the other approach
which I have found in several projects I work with on a daily basis.

  S.

-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ &
http://pydata.fr/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org