[issue26646] Allow built-in module in package

[Brett Cannon]

Brett Cannon added the comment:

Referring to the other issue was more about tying the two issues together than 
necessarily expecting the other patch to work.

As for supporting packages as well as submodules, I'm not sure. It seems a 
little odd not to support the idea, but then again if something is built-in 
then there really isn't a need to support the concept of a directory containing 
more modules. But then again since there is a difference in terms of how a 
module vs package looks it should probably be supported.

And if you do try and support it, there might be nothing more needed than to 
have the module be named pkg.__init__ and have some special handling to strip 
out the '__init__' part of the name and to set `__path__ = []` in importlib. 
Then importlib can simply try for `name` and `name.__init__` and then use that 
to figure out if the module should be considered a package or not.

This whole issue is unfortunately one of those things where it's uncommon 
enough to have to think about all edge cases and the overall impact on other 
users if the suggested changes are made and to not have a clear-cut answer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26646] Allow built-in module in package

[Daniel Shaulov]

Daniel Shaulov added the comment:

Hi Brett,

I don't think that the patch from that issue is relevant anymore.
I did take the test case that was proposed in that issue and I am attaching a 
fixed version here.

I did realize from the discussion that my patch probably doesn't work on 
Windows (I think the change itself is fine - It's just won't have the test 
module), I will try to get a working Windows environment and make the 
appropriate changes tomorrow.

Also, the other issue was also asking for built-in packages and not just 
built-in submodules. I already have a note about that in my original message.
Can we move forward as-is or do you want me to add support for built-in 
packages as well?

--
Added file: http://bugs.python.org/file42350/test_builtin_submodule.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26646] Allow built-in module in package

[Brett Cannon]

Brett Cannon added the comment:

http://bugs.python.org/issue1644818 is also related to this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26646] Allow built-in module in package

[Daniel Shaulov]

Changes by Daniel Shaulov :


Added file: http://bugs.python.org/file42298/importlib_h.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26646] Allow built-in module in package

[Daniel Shaulov]

New submission from Daniel Shaulov:

Hi,

I was trying to build a python interpreter that has the cpp part of msgpack as 
a built-in module. I thought that I could just copy the msgpack folder to the 
Modules folder and add this 2 lines to Modules/Setup.local:

msgpack._packer msgpack/_packer.cpp
msgpack._unpacker msgpack/_unpacker.cpp

I had a few obstacles, the attached patch fixes them all.

The first - makesetup has a list of regexes to match and it has *.* after going 
through all known extensions to throw a "bad word" error.
I removed the check. All those things will now be assumed to be a module with a 
package.
Now the actual init function name is PyInit__packer and doesn't have msgpack in 
it, so I also added code in makesetup to use the full name as the name and only 
the module name for the PyInit function.


The second is that in Lib/importlib/_bootsrap.py in BuiltinImporter.find_spec 
there is a specific case to ignore modules that are part of a package.
Is there a reason to forbid it? I removed that check.
There were also unit tests that checked this behavior which I deleted.
I added tests that check module in package instead of them.
Changing _bootsrap.py also changes Python/importlib.h (it is the frozen 
importlib), I added a separate patch with that change, to not clutter the main 
patch.

The third - the __name__ didn't have the package prefix.
Digging around I found a comment in PyModule_Create2 that says that the shared 
library loader stores the full name _Py_PackageContext before loading the 
module, so I did the same in _imp_create_builtin that is done in 
_PyImport_LoadDynamicModuleWithSpec to set _Py_PackageContext and the __name__ 
was fixed too.

(If anyone tries to do this with msgpack and wants to see that it works - you 
also need to copy the msgpack directory to the Lib directory, and in 
__init__.py, where it catches the import error and goes to the fallback, just 
reraise the exception instead of letting it go to the fallback)

Do note that this does not allow for built-in packages, only build-it module in 
package. If we want to allow built-in packages, we will need to decide on some 
syntax to distinguish it in the Setup files and some way to distinguish them in 
PyImport_Inittab (for example - an asterix before the name of the package?)

Thanks, Daniel.

--
components: Build, Interpreter Core
files: builtin_package.patch
keywords: patch
messages: 262495
nosy: Daniel Shaulov, brett.cannon, eric.snow, ncoghlan
priority: normal
severity: normal
status: open
title: Allow built-in module in package
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file42297/builtin_package.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com