Re: [Python-ideas] Idea: Importing from arbitrary filenames
On Mon, Apr 16, 2018 at 10:23 AM, Brett Cannon wrote: > > > On Mon, 16 Apr 2018 at 09:58 Eric Fahlgren wrote: > >> The documentation is pretty opaque or non-existent on other aspects of >> importlib use, too. >> > > Well, we are diving into the dark corners of import here. (Details can be > found in the language reference: https://docs.python.org/3/ > reference/import.html) > Thanks, Brett, I'll read through that and see where I get. Those corners /are/ pretty dark. The backstory is that I'm doing the final port from Py2 to Py3 (it's been a long time coming, mostly years of waiting for extension modules to get ported, notably wxPython and VTK). In Py2, all warnings were enabled and disallowed, so big surprise on first run, hundreds of lines of the aforementioned one and "ImportWarning: __package__ != __spec__.parent". We have manually defined "__package__" all over the place, for reasons lost in the fog of time, which I believe to be the culprit for the latter warning. Eric ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
On Mon, 16 Apr 2018 at 09:58 Eric Fahlgren wrote: > The documentation is pretty opaque or non-existent on other aspects of > importlib use, too. > Well, we are diving into the dark corners of import here. (Details can be found in the language reference: https://docs.python.org/3/reference/import.html). > If I enable warnings, I see this (and many more like it). I've read PEP > 302 a couple times, read the code in importlib that detects the warning and > searched down several rabbit holes, only to come up empty... > > T:\Python36\lib\importlib\_bootstrap.py:219: ImportWarning: can't resolve > package from __spec__ or __package__, falling back on __name__ and __path__ > > My thoughts when I see it: "Ok. So what does that mean? > It means that the mechanisms import typically uses to calculate the importing module's name in order to resolve relative imports wasn't where it should be, and so we fell back to the Python 2 way of doing it. > Is it bad? > Eh, it isn't ideal. ;) > It must be bad, otherwise I wouldn't get a warning. How do I reconcile > __spec__ and __package__? > You should be setting __spec__.parent, but we will fall back to __package__ if that doesn't exist (and raise a different warning). :) > Which one is missing and/or incorrect?" > Both are missing. :) -Brett > > > On Mon, Apr 16, 2018 at 9:36 AM, Paul Moore wrote: > >> On 16 April 2018 at 17:22, Nick Coghlan wrote: >> > If we're not covering explicit __path__ manipulation anywhere, we >> > should definitely mention that possibility. >> > https://docs.python.org/3/library/pkgutil.html#pkgutil.extend_path >> > does talk about it, but only in the context of scanning sys.path for >> > matching names, not in the context of building a package from an >> > arbitrary set of directory names. >> >> It's quite possible that we're not. >> >> > I'm not sure where we could put an explanation of some of the broader >> > implications of that fact, though - while __path__ manipulation is >> > usually fairly safe, we're always a little hesitant about encouraging >> > too many dynamic modifications to the import system state, since it >> > can sometimes have odd side effects based on whether imports happen >> > before or after that state is adjusted.. >> >> One of the problems with PEP 302 was that there was no really good >> place in the documentation to put all the information that was present >> (certainly not in the version of the docs that was around when we >> wrote it). So a lot of the important details remained buried in PEP >> 302. Since then, a lot of the details ended up in the docs, mostly in >> the importlib sections, but I don't recall ever seeing anything about >> __path__ (and particularly not the nice summary you gave, "packages >> are ultimately just modules with a >> __path__ attribute that works like sys.path". >> >> Paul >> ___ >> Python-ideas mailing list >> Python-ideas@python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
The documentation is pretty opaque or non-existent on other aspects of importlib use, too. If I enable warnings, I see this (and many more like it). I've read PEP 302 a couple times, read the code in importlib that detects the warning and searched down several rabbit holes, only to come up empty... T:\Python36\lib\importlib\_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ My thoughts when I see it: "Ok. So what does that mean? Is it bad? It must be bad, otherwise I wouldn't get a warning. How do I reconcile __spec__ and __package__? Which one is missing and/or incorrect?" On Mon, Apr 16, 2018 at 9:36 AM, Paul Moore wrote: > On 16 April 2018 at 17:22, Nick Coghlan wrote: > > If we're not covering explicit __path__ manipulation anywhere, we > > should definitely mention that possibility. > > https://docs.python.org/3/library/pkgutil.html#pkgutil.extend_path > > does talk about it, but only in the context of scanning sys.path for > > matching names, not in the context of building a package from an > > arbitrary set of directory names. > > It's quite possible that we're not. > > > I'm not sure where we could put an explanation of some of the broader > > implications of that fact, though - while __path__ manipulation is > > usually fairly safe, we're always a little hesitant about encouraging > > too many dynamic modifications to the import system state, since it > > can sometimes have odd side effects based on whether imports happen > > before or after that state is adjusted.. > > One of the problems with PEP 302 was that there was no really good > place in the documentation to put all the information that was present > (certainly not in the version of the docs that was around when we > wrote it). So a lot of the important details remained buried in PEP > 302. Since then, a lot of the details ended up in the docs, mostly in > the importlib sections, but I don't recall ever seeing anything about > __path__ (and particularly not the nice summary you gave, "packages > are ultimately just modules with a > __path__ attribute that works like sys.path". > > Paul > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
On 16 April 2018 at 17:22, Nick Coghlan wrote: > If we're not covering explicit __path__ manipulation anywhere, we > should definitely mention that possibility. > https://docs.python.org/3/library/pkgutil.html#pkgutil.extend_path > does talk about it, but only in the context of scanning sys.path for > matching names, not in the context of building a package from an > arbitrary set of directory names. It's quite possible that we're not. > I'm not sure where we could put an explanation of some of the broader > implications of that fact, though - while __path__ manipulation is > usually fairly safe, we're always a little hesitant about encouraging > too many dynamic modifications to the import system state, since it > can sometimes have odd side effects based on whether imports happen > before or after that state is adjusted.. One of the problems with PEP 302 was that there was no really good place in the documentation to put all the information that was present (certainly not in the version of the docs that was around when we wrote it). So a lot of the important details remained buried in PEP 302. Since then, a lot of the details ended up in the docs, mostly in the importlib sections, but I don't recall ever seeing anything about __path__ (and particularly not the nice summary you gave, "packages are ultimately just modules with a __path__ attribute that works like sys.path". Paul ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
On 16 April 2018 at 03:45, Steve Barnes wrote: > On 15/04/2018 08:12, Nick Coghlan wrote: >> The discoverability of these kinds of techniques could definitely >> stand to be improved, but the benefit of adopting them is that they >> work on all currently supported versions of Python (even >> importlib.import_module exists in Python 2.7 as a convenience wrapper >> around __import__), rather than needing to wait for new language level >> syntax for them. > > As you say not too discoverable at the moment - I have just reread > PEP328 & https://docs.python.org/3/library/importlib.html but did not > find any mention of these mechanisms or even that setting an external > __path__ variable existed as a possibility. Yeah, the fact that "packages are ultimately just modules with a __path__ attribute that works like sys.path" tends to get obscured by the close association between package hierarchies and file system layouts in the default filesystem importer. The docs for that are all the way back in PEP 302: https://www.python.org/dev/peps/pep-0302/#packages-and-the-role-of-path > Maybe a documentation enhancement proposal would be in order? If we're not covering explicit __path__ manipulation anywhere, we should definitely mention that possibility. https://docs.python.org/3/library/pkgutil.html#pkgutil.extend_path does talk about it, but only in the context of scanning sys.path for matching names, not in the context of building a package from an arbitrary set of directory names. I'm not sure where we could put an explanation of some of the broader implications of that fact, though - while __path__ manipulation is usually fairly safe, we're always a little hesitant about encouraging too many dynamic modifications to the import system state, since it can sometimes have odd side effects based on whether imports happen before or after that state is adjusted.. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
On 15/04/2018 08:12, Nick Coghlan wrote:
> On 14 April 2018 at 19:22, Steve Barnes wrote:
>> I generally love the current import system for "just working" regardless
>> of platform, installation details, etc., but what I would like to see is
>> a clear import local, (as opposed to import from wherever you can find
>> something to satisfy mechanism). This is the one thing that I miss from
>> C/C++ where #include is system includes and #include "x" search
>> differing include paths, (if used well).
>
> For the latter purpose, we prefer that folks use either explicit
> relative imports (if they want to search the current package
> specifically), or else direct manipulation of package.__path__.
>
> That is, if you do:
>
> from . import custom_imports # Definitely from your own project
> custom_imports.__path__[:] = (some_directory, some_other_directory)
>
> then:
>
> from .custom_imports import name
>
> will search those directories for packages & modules to import, while
> still cleanly mapping to a well-defined location in the module
> namespace for the process as a whole (and hence being able to use all
> the same caches as other imports, without causing name conflicts or
> other problems).
>
> If you want to do this dynamically relative to the current module,
> then it's possible to do:
>
> global __path__
> __path__[:] = (some_directory, some_other_directory)
> custom_mod = importlib.import_module(".name", package=__name__)
>
> The discoverability of these kinds of techniques could definitely
> stand to be improved, but the benefit of adopting them is that they
> work on all currently supported versions of Python (even
> importlib.import_module exists in Python 2.7 as a convenience wrapper
> around __import__), rather than needing to wait for new language level
> syntax for them.
>
> Cheers,
> Nick.
>
Thanks Nick,
As you say not too discoverable at the moment - I have just reread
PEP328 & https://docs.python.org/3/library/importlib.html but did not
find any mention of these mechanisms or even that setting an external
__path__ variable existed as a possibility.
Maybe a documentation enhancement proposal would be in order?
--
Steve (Gadget) Barnes
Any opinions in this message are my personal opinions and do not reflect
those of my employer.
---
This email has been checked for viruses by AVG.
http://www.avg.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
On 15 April 2018 at 17:12, Nick Coghlan wrote:
> If you want to do this dynamically relative to the current module,
> then it's possible to do:
>
> global __path__
> __path__[:] = (some_directory, some_other_directory)
> custom_mod = importlib.import_module(".name", package=__name__)
Copy and paste error there: to make this work in non-package modules,
drop the "[:]" from the __path__ assignment.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
On 14 April 2018 at 19:22, Steve Barnes wrote:
> I generally love the current import system for "just working" regardless
> of platform, installation details, etc., but what I would like to see is
> a clear import local, (as opposed to import from wherever you can find
> something to satisfy mechanism). This is the one thing that I miss from
> C/C++ where #include is system includes and #include "x" search
> differing include paths, (if used well).
For the latter purpose, we prefer that folks use either explicit
relative imports (if they want to search the current package
specifically), or else direct manipulation of package.__path__.
That is, if you do:
from . import custom_imports # Definitely from your own project
custom_imports.__path__[:] = (some_directory, some_other_directory)
then:
from .custom_imports import name
will search those directories for packages & modules to import, while
still cleanly mapping to a well-defined location in the module
namespace for the process as a whole (and hence being able to use all
the same caches as other imports, without causing name conflicts or
other problems).
If you want to do this dynamically relative to the current module,
then it's possible to do:
global __path__
__path__[:] = (some_directory, some_other_directory)
custom_mod = importlib.import_module(".name", package=__name__)
The discoverability of these kinds of techniques could definitely
stand to be improved, but the benefit of adopting them is that they
work on all currently supported versions of Python (even
importlib.import_module exists in Python 2.7 as a convenience wrapper
around __import__), rather than needing to wait for new language level
syntax for them.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
On 4/13/2018 11:28 PM, Ken Hilton wrote:
Hi all,
First of all, please excuse me if I'm presenting this idea in the wrong
way or at the wrong time - I'm new to this mailing list and haven't seen
anyone propose a new idea on it yet, so I don't know the customs.
I have an idea for importing files with arbitrary names. Currently, the
"official"
Alex Martelli intentionally put that in quotes.
way to import arbitrary files is to use the "imp" module, as
shown by this answer: https://stackoverflow.com/a/3137914/6605349
Read the first comment -- the above is deprecated. There was always the
__import__(name) function, but importlib.import_module is recommended now.
However, this method takes two function calls and is not as
(aesthetically pleasing? is that the word?) as a simple "import" statement.
Only one is needed for most purposes. importlib has separate find and
load functions, which are used by 'import', and which are available to
those who need them.
Second, importing with a string literal specifying the path to a file:
import '/home/pi/anti-gravity.py' *as antigravity*
antigravity = import_module('/home/pi/anti-gravity.py')
--
Terry Jan Reedy
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
On 14/04/2018 06:27, Nick Coghlan wrote: > On 14 April 2018 at 13:28, Ken Hilton wrote: >> Hi all, >> >> First of all, please excuse me if I'm presenting this idea in the wrong way >> or at the wrong time - I'm new to this mailing list and haven't seen anyone >> propose a new idea on it yet, so I don't know the customs. >> >> I have an idea for importing files with arbitrary names. Currently, the >> "official" way to import arbitrary files is to use the "imp" module, as >> shown by this answer: https://stackoverflow.com/a/3137914/6605349 >> However, this method takes two function calls and is not as (aesthetically >> pleasing? is that the word?) as a simple "import" statement. > > Modules aren't required to be stored on the filesystem, so we have no > plans to offer this. > > `runpy.run_path()` exists to let folks run arbitrary Python files and > collect the resulting namespace, while if folks really want to > implement pseudo-imports based on filenames we expose the necessary > building blocks in importlib > (https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly) > > The fact that run_path() has a nice straightforward invocation model, > and the import emulation recipe doesn't is intended as a hint :) > > Cheers, > Nick. > I generally love the current import system for "just working" regardless of platform, installation details, etc., but what I would like to see is a clear import local, (as opposed to import from wherever you can find something to satisfy mechanism). This is the one thing that I miss from C/C++ where #include is system includes and #include "x" search differing include paths, (if used well). -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
On 14 April 2018 at 13:28, Ken Hilton wrote: > Hi all, > > First of all, please excuse me if I'm presenting this idea in the wrong way > or at the wrong time - I'm new to this mailing list and haven't seen anyone > propose a new idea on it yet, so I don't know the customs. > > I have an idea for importing files with arbitrary names. Currently, the > "official" way to import arbitrary files is to use the "imp" module, as > shown by this answer: https://stackoverflow.com/a/3137914/6605349 > However, this method takes two function calls and is not as (aesthetically > pleasing? is that the word?) as a simple "import" statement. Modules aren't required to be stored on the filesystem, so we have no plans to offer this. `runpy.run_path()` exists to let folks run arbitrary Python files and collect the resulting namespace, while if folks really want to implement pseudo-imports based on filenames we expose the necessary building blocks in importlib (https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly) The fact that run_path() has a nice straightforward invocation model, and the import emulation recipe doesn't is intended as a hint :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Idea: Importing from arbitrary filenames
I'm fairly certain similar changes have been discussed in the past. Someone
else can probably find / link / rehash the reasons why imports deliberately
use dot notation instead of path?
I can think of a few:
1) Portability. dotted imports looked up from sys.path are platform-portable
# $HOME/\this\/__init__.py
sys.path.append(
os.path.expanduser(
os.path.join('~', 'this',)))
2) Flexibility. importlib Loaders (3.1+) are abstract; they only know what
to do with dotted paths. They can import from the filesystem, zip files,
... git repos,
https://docs.python.org/3/library/importlib.html
https://docs.python.org/3/library/importlib.html#importlib.abc.SourceLoader
https://pypi.org/search/?q=importlib
https://docs.python.org/3/library/imp.html#imp.find_module
https://docs.python.org/3/library/imp.html#imp.load_module
sys.path (`python -m site`) can be configured with:
- $PYTHONSTARTUP='~/.pythonrc.py'
- modules.py and dirs/__init__.py in site-packages/
- .pth files in site-packages/
- idempotent sys.path config at the top of a .py source file
- sys.USER_SITE in sys.path
- ~/.local/lib/python*X.Y*/site-packages
- ~/Library/Python/*X.Y*/lib/python/site-packages
- *%APPDATA%*\Python\Python*XY*\site-packages
- https://docs.python.org/3/library/site.html
- https://docs.python.org/3/using/cmdline.html#envvar-PYTHONSTARTUP
- https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUSERBASE
- https://docs.python.org/3/library/site.html#site.USER_SITE
Is there a good write-up of how, where, and in what order sys.path is
configured, by default, in Python?
TLDR dotted names are preferable for sharing code with people who don't
have the same paths, os.pathsep, or os.platform_info. (Stevedore and
Jupyter Notebook take different approaches to handling plugins, if that's
your use case?)
Though I could just be arguing for the status quo; there are probably good
reasons to consider changing EVERYTHING
On Friday, April 13, 2018, Ken Hilton wrote:
> Hi all,
>
> First of all, please excuse me if I'm presenting this idea in the wrong
> way or at the wrong time - I'm new to this mailing list and haven't seen
> anyone propose a new idea on it yet, so I don't know the customs.
>
> I have an idea for importing files with arbitrary names. Currently, the
> "official" way to import arbitrary files is to use the "imp" module, as
> shown by this answer: https://stackoverflow.com/a/3137914/6605349
> However, this method takes two function calls and is not as (aesthetically
> pleasing? is that the word?) as a simple "import" statement.
>
> Therefore, my idea is to allow the "import" statement to accept one of
> three targets.
> First, the normal "import":
>
> import antigravity
>
> which simply imports from sys.path.
>
> Second, importing with a string literal specifying the path to a file:
>
> import '/home/pi/anti-gravity.py' *as antigravity*
>
> Note the "as antigravity" in this statement - this is to avoid ambiguities
> when choosing the global name to bind to. Should "import
> '/home/pi/anti-gravity.py'" import to the name "/home/pi/anti-gravity.py",
> "anti-gravity.py", "anti-gravity", or "anti_gravity"? None of those are
> really ideal. Therefore, when the import target is a string literal, the
> statement must include "as NAME".
>
> Third, importing with an expression providing a value castable to a
> string, specifying the path to a file:
>
> def file_in_home(filename):
> return '/home/pi/' + filename
> import *$*file_in_home('anti-gravity.py') *as antigravity*
>
> Once again, for the same reasons, import statements like this must include
> "as NAME" to avoid ambiguities. Notice that the expression is preceded by a
> dollar sign ($) to indicate that what follows is an expression rather than
> a name - imagine a scenario like this:
>
> antigravity_file = '/home/pi/anti-gravity.py'
> import antigravity_file as antigravity
>
> Should it look for a sys.path module with the name "antigravity_file" or
> should it use the value of the variable "antigravity_file"? Looking for the
> sys.path module first before trying a variable's value would waste
> processing time and potentially be unexpected behavior. Trying a variable's
> value first before looking for a sys.path module would be even less
> expected behavior. Therefore, a dollar sign must come before expression
> imports to indicate that the import target is an expression.
> Side note: the dollar sign was chosen because it mimics other languages'
> conventions of preceding variable names with dollar signs, but any
> arbitrary character not present at the start of an expression would work.
> One more thing about expression imports: if the final returned value of
> the expression is not a string, I believe the statement should raise a
> TypeError (the same way that __repr__ or __str__ raise TypeError if they
> return a non-string). Why? If the statement attempted to cast the return
> value to a string, and the return value's __str_
