[issue45653] Freeze the encodings module.

[Christian Heimes]

Christian Heimes  added the comment:

Eric, I have a simple reproducer for the issue:

This works:

$ LC_ALL=en_US.utf-8 TESTPATH=$(pwd)/Lib:$(pwd)/build/lib.linux-x86_64-3.11 
./Programs/_testembed test_init_setpath_config

This fails because it cannot load ISO-8859-1 / latin-1 codec

$ LC_ALL=en_US.latin1 TESTPATH=$(pwd)/Lib:$(pwd)/build/lib.linux-x86_64-3.11 
./Programs/_testembed test_init_setpath_config
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'conf_program_name'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  is in build tree = 0
  stdlib dir = ''
  sys._base_executable = 'conf_executable'
  sys.base_prefix = ''
  sys.base_exec_prefix = ''
  sys.platlibdir = 'lib'
  sys.executable = 'conf_executable'
  sys.prefix = ''
  sys.exec_prefix = ''
  sys.path = [
'/home/heimes/dev/python/cpython/Lib',
'/home/heimes/dev/python/cpython/build/lib.linux-x86_64-3.11',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the 
filesystem encoding
Python runtime state: core initialized
LookupError: unknown encoding: ISO-8859-1

Current thread 0x7f9c42be6740 (most recent call first):
  



With this patch I'm seeing that encodings.__path__ is not absolute and that 
__spec__ has an empty submodule_search_locations.

--- a/Lib/encodings/__init__.py
+++ b/Lib/encodings/__init__.py
@@ -98,9 +98,12 @@ def search_function(encoding):
 # module with side-effects that is not in the 'encodings' package.
 mod = __import__('encodings.' + modname, fromlist=_import_tail,
  level=0)
-except ImportError:
+except ImportError as e:
 # ImportError may occur because 'encodings.(modname)' does not 
exist,
 # or because it imports a name that does not exist (see mbcs and 
oem)
+sys.stderr.write(f"exception: {e}\n")
+sys.stderr.write(f"encodings.__path__: {__path__}\n")
+sys.stderr.write(f"encodings.__spec__: {__spec__}\n")
 pass
 else:
 break


$ LC_ALL=en_US.latin1 TESTPATH=$(pwd)/Lib:$(pwd)/build/lib.linux-x86_64-3.11 
./Programs/_testembed test_init_setpath_config
exception: No module named 'encodings.latin_1'
encodings.__path__: ['encodings']
encodings.__spec__: ModuleSpec(name='encodings', loader=, origin='frozen', 
submodule_search_locations=[])
exception: No module named 'encodings.iso_8859_1'
encodings.__path__: ['encodings']
encodings.__spec__: ModuleSpec(name='encodings', loader=, origin='frozen', 
submodule_search_locations=[])


It should have this search location:

>>> import encodings
>>> encodings.__spec__
ModuleSpec(name='encodings', loader=, 
origin='frozen', 
submodule_search_locations=['/home/heimes/dev/python/cpython/Lib/encodings'])

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Christian Heimes]

Change by Christian Heimes :


--
nosy: +christian.heimes
nosy_count: 5.0 -> 6.0
pull_requests: +28255
pull_request: https://github.com/python/cpython/pull/30030

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Guido van Rossum]

Guido van Rossum  added the comment:


New changeset 02b5ac6091ada0c2df99c4e1eae37ddccbcd91f0 by Kumar Aditya in 
branch 'main':
bpo-45653: fix test_embed on windows (GH-29814)
https://github.com/python/cpython/commit/02b5ac6091ada0c2df99c4e1eae37ddccbcd91f0


--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Kumar Aditya]

Change by Kumar Aditya :


--
pull_requests: +28047
pull_request: https://github.com/python/cpython/pull/29814

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Kumar Aditya]

Change by Kumar Aditya :


--
nosy: +kumaraditya303
nosy_count: 3.0 -> 4.0
pull_requests: +28024
pull_request: https://github.com/python/cpython/pull/29788

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Filipe Laíns]

Change by Filipe Laíns :


--
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Filipe Laíns]

Filipe Laíns  added the comment:

I have already opened up the PR, but I can change if desired.

--
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

On 30.10.2021 17:54, Filipe Laíns wrote:
> 
> I just tested partially freezing the package, and it seems to working fine :)
FWIW: I think it's best not bother and simply freeze the whole thing.

It's mostly char mappings which compress well and there's a benefit
in sharing these using mmap (which the OS does for you with static
C data).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Filipe Laíns]

Change by Filipe Laíns :


--
keywords: +patch
pull_requests: +27599
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/29331

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Filipe Laíns]

Filipe Laíns  added the comment:

I just tested partially freezing the package, and it seems to working fine :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Eric Snow]

Eric Snow  added the comment:

On Thu, Oct 28, 2021 at 12:15 PM Marc-Andre Lemburg
 wrote:
> encodings is a package. I think you first have to check whether mixing
> frozen and non-frozen submodules are even supported. I've never tried
> having only part of a package frozen.

It works as long as __path__ is set properly, which it is now.  FWIW,
I tested freezing only some of the submodules a while back and it
worked fine.  That was using a different branch that I never merged
but it should be fine with the different change that got merged.  Of
course, we'd need to verify that if we went that route.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

encodings is a package. I think you first have to check whether mixing
frozen and non-frozen submodules are even supported. I've never tried
having only part of a package frozen.

Freezing the whole package certainly works.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45653] Freeze the encodings module.

[Eric Snow]

New submission from Eric Snow :

Currently we freeze all the modules imported during runtime initialization, 
except for the encodings module.  It has a lot of submodules and this results 
in a lot of extra noise in builds.  We hadn't frozen it yet because we were 
still ironing out changes related to frozen modules and the extra noise was a 
pain.  We also waited because we weren't sure if we should freeze all the 
submodules or just the most likely ones to be used during startup.  In the case 
of the latter, we were also blocked on having __path__ set on the package.

At this point there are no blockers.  So we should freeze the encodings modules 
with either all submodules or the most commonly used subset.

--
components: Build
messages: 405211
nosy: FFY00, eric.snow
priority: normal
severity: normal
stage: needs patch
status: open
title: Freeze the encodings module.
type: behavior
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com