[issue25294] Absolute imports fail in some cases where relative imports would work

[Nick Coghlan]

Nick Coghlan added the comment:

We seem to be talking past each other, so let's take a step back and ensure we 
have a common understanding of the absolute/relative terminology:

"import x.y.z" is an absolute import
"from x.y import z" is still an absolute import.
"from . import z" is an explicit relative import of a child or sibling module
"from .y import z" is also an explicit relative import

The relevant change in behaviour is between the "import x.y.z" form and the 
"from x.y import z" form due to the change in the way the related name binding 
(and name lookup in 3.5+) works, not between absolute and relative imports.

The relevant terminology to distinguish between "from ... import ..." vs 
"import ..." is just "from import" vs "non-from import", and there are 
definitely cases where from imports will work, but non-from imports will fail. 
That's not a style issue, it's a use-whichever-one-works for your code issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Nick Coghlan]

Nick Coghlan added the comment:

(Oops, it seems Brett already clarified the terminology. My apologies for the 
noise).

Modularity and module design is hard. It's one of the hardest problems in 
software engineering, which is why folks invent entire new vocabularies to 
attempt to describe the problem space effectively: http://connascence.io/

The simplest way to avoid these kinds of import related problems as a beginner 
in Python is to just put all your code in one module, and only factor it out 
into multiple modules when the single file becomes difficult to maintain.

Unfortunately, Java's "one public class per file" restriction, it's lack of 
standalone module level functions, and it's popularity as a university level 
teaching level has given a lot of people a lot of bad ideas as to what good 
multi-level modularity looks like, as Java throws out the notion of using 
modules to group closely related public classes and standalone functions into a 
single file, and instead makes you jump almost directly from classes to 
packages.

So perhaps that's the currently unwritten rule that would be worth documenting? 
That Python has 3 levels of namespacing (classes, modules, packages), and that 
flat namespaces are preferred in the standard library. If the namespace is 
broken up internally into multiple files for maintainability reasons, we prefer 
to still present a flat *public* API, as unittest, asyncio and 
concurrent.futures do.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Brett Cannon]

Brett Cannon added the comment:

You have to realize, Patrick, that the ability for `from ... import ...` to 
work in some situations that `import ...` won't is an age-old problem -- they 
are different at the bytecode level as well as how __import__ handles them -- 
and starting in Python 3.5 it got tweaked to potentially make the differences 
more pronounced by making a certain situation involving circular imports not 
trip users up as much where it was technically feasible to make the change 
without backwards-compatibility issues. It was a "practicality over purity" 
decision.

You also said that "PEP 8 prescribes a way of doing something that often won't 
work" which I disagree with. I think you're reading "absolute import" as 
synonymous with `import ...` which is not what the term means. In actuality, 
absolute imports means no leading dot in the name, e.g. `from .x import y` is a 
relative import while `from z.x import y` is an absolute one (as is `import 
z.x.y`). If you are using the term "absolute import" in the correct way then I 
still don't see how PEP 8 is suggesting any practice "that won't often work".

Circular imports are just plain hard to deal with. Due to the long-standing 
design of import being nothing more than syntactic sugar around exec() we don't 
get to know statically that a circular import is going to happen, so we can't 
error out early to tell the user that they *may* be in trouble. And not 
everyone gets themselves in a position where a circular import dependencies is 
a problem since it only causes issues when module-level code depends on each 
other in a circular way (which does include import statements which is where 
people typically trip themselves up when they get in this situation). I realize 
you're trying to do the right thing here and get the docs clarified to help 
newcomers out, but please realize it's just a difficult subject to explain 
succinctly, especially since a vast majority of people don't get themselves 
into a position where they have to deal with a circular import.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Patrick Maupin]

Patrick Maupin added the comment:

You are correct that I have conflated two issues, but they are not orthogonal 
-- if you choose to use relative imports, you will never encounter this issue, 
because your imports will all be of the 'from ... import' form.

(And, as you point out, the fact that you don't have this issue with absolute 
"from ... import" statements is due to some special-casing in the import logic 
that doesn't help the other kind of import statement.)

But PEP 8 denigrates relative imports, and then goes on to describe the use of 
the "import x.y.z; x.y.z.foo" form as a way to avoid name clashes.

So PEP 8 promotes absolute imports, and then it presents using "import" instead 
of "from ... import" as a solution to some problems.  It never mentions that 
the 'as' clause could also solve those problems, and also never mentions that 
"import x.y.z" can actually cause problems in some cases.

And the importlib documentation is also a bit sparse on where things can fail.

We're in agreement that it will be difficult to document properly, and maybe I 
overstated my case, but my opinion remains that the current documentation 
promotes practices that _will_ sometimes get people in trouble.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Patrick Maupin]

Patrick Maupin added the comment:

concurrent/futures/__init__.py may be a better example than 2to3 for this 
issue.  It's relatively new code, it's part of the standard library, it's 
fairly small and self-contained, and it doesn't follow the promulgated standard.

If it's bad code, it should be modified.  If it's not bad code, then the docs 
shouldn't denigrate the coding style (especially not to the extent of requiring 
absolute imports in standard library code), because a lot of newbies take the 
docs to heart and spend a lot of time and energy beating up themselves and 
others about conformance.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Patrick Maupin]

Patrick Maupin added the comment:

I don't think anything is wrong with that code.  But PEP 8 prescribes a way of 
doing something that often won't work (which is unusual for PEP 8), with no 
discussion of this fact.

> I think the key thing to take away from this whole discussion is "don't have 
> circular imports" is the key practice to follow.

If this is a "key practice" then why the heck is the recommended way to do 
things the one that is guaranteed to break it?

I have empirical evidence that it is surprising to some users that the 
semantics of "from .z import foo" and "from x.y.z import foo" are not identical 
-- in other words, some of them have a hard time classifying the second as a 
circular import but not the first.  And the documentation is silent on this.

> And if someone wants to propose a patch to update some documentation

I'll try to get to this in the next couple of weeks.

Thanks,
Pat

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Brett Cannon]

Brett Cannon added the comment:

I don't quite follow what you think is wrong with 
https://hg.python.org/cpython/file/tip/Lib/concurrent/futures/__init__.py . It 
looks totally fine to me.

And I should mention that you shouldn't follow PEP 8 like it's in stone and the 
only way to format code. The PEP is a set of guidelines only and not rules 
(this is a long-standing position of python-dev on PEP 8). For instance, I 
don't agree with the absolute import recommendation and do not follow it in my 
own code nor in importlib (it makes vendoring impossible without modifying the 
import statements).

I think the key thing to take away from this whole discussion is "don't have 
circular imports" is the key practice to follow. And if someone wants to 
propose a patch to update some documentation to point out that `from ... import 
...` may work in some situations where `import ...; ... = ...` doesn't then I 
will personally review such a patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Nick Coghlan]

Nick Coghlan added the comment:

Issue 992389 is the previous incarnation of this bug report, while issue 17636 
made the change so that from imports will resolve in some situations where this 
error will occur.

That fact that "from x.y.b import foo" may now resolve in cases where "import 
x.y.b; foo = x.y.b.foo" will fail should indeed be covered in the documentation.

If PEP 8 were to be updated at all, it should just say "Don't use circular 
imports. If you can't refactor your design to move the common components out to 
a shared helper module, then move everything into the same module - the 
irremovable circular dependency indicates the code is too tightly coupled to 
live in different modules".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Patrick Maupin]

Patrick Maupin added the comment:

The PEP 8 recommendation to "use absolute imports" is completely, totally, 
unambiguously meaningless absent the expectation that packages refer to parts 
of themselves.  And it works, too!  (For a single level of package.)

As soon as packages are nested, this recommendation falls over, with the most 
innocuous code:

x/__init__.py: import x.y
x/y/__init__.py: import x.y.z; x.y.z
x/y/z/__init__.py: 

The ability to nest packages is an attractive nuisance from a programmer's 
perspective.  He's neatly organized his code, and now he finds that there are 
two ways to make it work:  (1) Use the disparaged relative imports; or (2) 
flatten his package to a single level, because importing X.Z from within X.Y 
will work fine.

IMO, the language that Nick proposes for PEP 8 will either (a) not be 
understood at all by the frustrated junior programmer -- sure, the import 
system views it as a circular import, but he's not seeing it that way; or (b) 
be understood to expose a huge wart on the side of Python:  Even though Z is 
only used by Y/__init__, and doesn't itself use anything else in Y, it cannot 
live alongside Y/__init__. Instead, unless Y is a top level module or the 
programmer uses denigrated relative imports, he will now have to move it to a 
different place, so that from Y he can then "import X.Y_HELPER.Z".

Another PEP 8 prescription is that "Standard library code should avoid complex 
package layouts and always use absolute imports."  Here's a serious offer -- 
I'll give $200 to whoever gets the patch accepted that makes lib2to3 conformant 
without breaking it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Nick Coghlan]

Nick Coghlan added the comment:

If that's the concern, then the relevant guidance is to avoid running code
at package import time (which many new developers will now do by default
with __init__.py becoming optional).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Patrick Maupin]

Patrick Maupin added the comment:

I'm a big fan of stitching things together at the top myself -- maybe that's 
partly an unconscious reaction to this very issue.

But I'm not sanguine about how easy it is to express this practice in the docs.

This issue arose in the context of me answering a question on Stack Overflow.  
My initial response was "well, duh, obviously relative imports are more 
Pythonic here because that's the obvious way to do it (that works)."

But then, of course, PEP 8 disagrees.

For that actual question on Stack Overflow, you would have to carefully define 
the scope of "executing" so that it was fully understood to include 
"subclassing a class defined in a peer module or submodule" -- because that's 
what was breaking.

Just as most people don't think of imports that can be placed in a DAG as 
circular, most people also don't think of subclassing as "executing."

But the Python interpreter sometimes vehemently disagrees in both cases.  I'm 
not surprised at its behavior, but I'm also not surprised that some people who 
haven't thought deeply about the behavior find it surprising.

I'm not fully convinced that this even can be documented in a way that renders 
observed behavior unsurprising in the general case, but I am convinced that 
doing so would require a lot of care.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[Berker Peksag]

Changes by Berker Peksag :


--
nosy: +brett.cannon, eric.snow, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25294] Absolute imports fail in some cases where relative imports would work

[J Richard Snape]

Changes by J Richard Snape :


--
nosy: +J Richard Snape

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com