[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:

I opened a thread on python-dev about this issue:
"Configure Python initialization (PyConfig) in Python"
https://mail.python.org/archives/list/python-...@python.org/thread/HQNFTXOCDD5ROIQTDXPVMA74LMCDZUKH/#X45X2K4PICTDJQYK3YPRPR22IGT2CDXB

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 14d81dcaf827f6b66bda45e8f5689d07d7d5735c by Serhiy Storchaka in 
branch 'master':
bpo-42260: Improve error handling in _PyConfig_FromDict (GH-23488)
https://github.com/python/cpython/commit/14d81dcaf827f6b66bda45e8f5689d07d7d5735c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +22376
pull_request: https://github.com/python/cpython/pull/23488

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Please don't use PyDict_GetItemString(), it will be deprecated. You can use 
_PyDict_GetItemStringWithError().

Also always check the raised exception type before overwriting the exception, 
so you will not swallow MemoryError or other unexpected error.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[Nick Coghlan]

Change by Nick Coghlan :


--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ef75a625cdf8377d687a04948b4db9bc1917bf19 by Victor Stinner in 
branch 'master':
bpo-42260: Initialize time and warnings earlier at startup (GH-23249)
https://github.com/python/cpython/commit/ef75a625cdf8377d687a04948b4db9bc1917bf19


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22146
pull_request: https://github.com/python/cpython/pull/23249

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[Chris Meyer]

Chris Meyer  added the comment:

>> I would like to request that this ability to dynamically load Python DLLs 
>> remains even with any new initialization mechanism.

> I don't plan to remove any feature :-)

I am glad to hear that. I'm somewhat nervous about it nevertheless. In 
particular, the implementation of Py_DECREF changed from 3.7 to 3.8 to 3.9. 3.7 
worked entirely in a header; but 3.8 had a quirky definition of _Py_Dealloc 
which used _Py_Dealloc_inline but was defined out of order (used before 
defined). This was somewhat addressed in 
https://github.com/python/cpython/pull/18361/files; however 3.9 now has another 
mechanism that defines _Py_Dealloc in Objects/object.c. This isn't a major 
problem because it has the same implementation as before, but changes like this 
have the potential to make the launcher binary be version specific. Again, not 
a deal breaker, but it still makes me nervous.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[Chris Meyer]

Chris Meyer  added the comment:

> How do you configure sys.path currently? Do you parse a configuration file? 
> Do you use a registry key on Windows?

We have several launch scenarios - but for the currently most common one, which 
is to launch using a separate, existing Python environment, we call 
Py_SetPythonHome and Py_SetPath with the home directory of the environment. 
Then, presumably, the more complete path gets set in either Py_Initialize or 
when we call PyImport_ImportModule(“sys”). I might have tracked the details 
down once, but I don't recall them. By the time our Python code starts running, 
sys.path is reasonably populated.

However, in another scenario, we launch with an embedded Python environment, 
essentially a virtual environment. In that case, we have a config file to 
explicitly add lib, DLLs, and site packages. But something goes wrong [cannot 
find/load the unicode DLL IIRC] unless we call site.addsitedir for each 
directory already in sys.path near the start of our Python portion of code. My 
notes point to two issues to explain this: https://bugs.python.org/issue22213 
and https://bugs.python.org/issue35706.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[Chris Meyer]

Chris Meyer  added the comment:

Responding to your request for feedback on Python-Dev:

We embed Python dynamically by finding the libPython DLL, loading it, and 
looking up the required symbols. We make appropriate define's so that the 
Python headers (and NumPy headers) point to our functions which in turn point 
to the looked up symbols.

Our launcher works on Linux, macOS, and Windows and works with many 
environments including standard Python and conda and brew. It also supports 
virtual environments in most cases. Also, a single executable [per platform] is 
able to work with Python versions 3.7 - 3.9 (3.6 was recently dropped, but only 
for external reasons).

So my comment is not directly addressing the usefulness of configuring Python 
initialization - but I would like to request that this ability to dynamically 
load Python DLLs remains even with any new initialization mechanism.

As another note, the main issues we run into are configuring the Python path to 
properly find packages and DLLs. A goal of ours is to be able to provide the 
base application as a drag-and-drop style installer with its own full embedded 
Python distribution (but still loaded dynamically) and then be able to supply 
additional plug-in packages (Python packages) by drag and drop. This is 
somewhat similar to conda packaging but without support for command line tools.

--
nosy: +cmeyer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:

> I would like to request that this ability to dynamically load Python DLLs 
> remains even with any new initialization mechanism.

I don't plan to remove any feature :-)

> As another note, the main issues we run into are configuring the Python path 
> to properly find packages and DLLs.

Do you mean sys.path? If yes, that's one of the goal of this issue. Allow you 
to write your own Python code to configure sys.path, rather than having to 
write C code, before the first (external) import.

How do you configure sys.path currently? Do you parse a configuration file? Do 
you use a registry key on Windows?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:

The main drawback of rewriting Modules/getpath.c as Lib/_getpath.py (and 
removing getpath.c) is that PyConfig_Read() could no longer compute the Python 
Path Configuration. It would return an "empty" path configuration.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ace3f9a0ce7b9fe8ae757fdd614f1e7a171f92b0 by Victor Stinner in 
branch 'master':
bpo-42260: Fix _PyConfig_Read() if compute_path_config=0 (GH-23220)
https://github.com/python/cpython/commit/ace3f9a0ce7b9fe8ae757fdd614f1e7a171f92b0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22118
pull_request: https://github.com/python/cpython/pull/23220

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:

> If we remove Modules/getpath.c, it will no longer be possible to 
> automatically computes the path configuration when one of the following 
> getter function will be called: (...)

It is not really an incompatible change according to the documentation:

"Note: The following functions should not be called before Py_Initialize(): 
Py_EncodeLocale(), Py_GetPath(), Py_GetPrefix(), Py_GetExecPrefix(), 
Py_GetProgramFullPath(), Py_GetPythonHome(), Py_GetProgramName() and 
PyEval_InitThreads().".

https://docs.python.org/dev/c-api/init.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:

If we remove Modules/getpath.c, it will no longer be possible to automatically 
computes the path configuration when one of the following getter function will 
be called:

* Py_GetPath()
* Py_GetPrefix()
* Py_GetExecPrefix()
* Py_GetProgramFullPath()
* Py_GetPythonHome()
* Py_GetProgramName()

It means that these functions would not return NULL if called before Python is 
initialiazed, but return the expected string once Python is initialized.

Moreover, Py_SetPath() would no longer automatically computes the "program full 
path" (sys.executable).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9e1b828265e6bfb58f1e0299bd78d8ff6347a2ba by Victor Stinner in 
branch 'master':
bpo-42260: Compute the path config in the main init (GH-23211)
https://github.com/python/cpython/commit/9e1b828265e6bfb58f1e0299bd78d8ff6347a2ba


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22109
pull_request: https://github.com/python/cpython/pull/23211

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22081
pull_request: https://github.com/python/cpython/pull/23169

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:


New changeset dc42af8fd16b10127ce1fc93c13bc1bfd2674aa2 by Victor Stinner in 
branch 'master':
bpo-42260: PyConfig_Read() only parses argv once (GH-23168)
https://github.com/python/cpython/commit/dc42af8fd16b10127ce1fc93c13bc1bfd2674aa2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22080
pull_request: https://github.com/python/cpython/pull/23168

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:


New changeset f3cb81431574453aac3b6dcadb3120331e6a8f1c by Victor Stinner in 
branch 'master':
bpo-42260: Add _PyConfig_FromDict() (GH-23167)
https://github.com/python/cpython/commit/f3cb81431574453aac3b6dcadb3120331e6a8f1c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22079
pull_request: https://github.com/python/cpython/pull/23167

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 048a35659aa8074afe7d7d054e7cea1f8ee6d366 by Victor Stinner in 
branch 'master':
bpo-42260: Add _PyInterpreterState_SetConfig() (GH-23158)
https://github.com/python/cpython/commit/048a35659aa8074afe7d7d054e7cea1f8ee6d366


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22069
pull_request: https://github.com/python/cpython/pull/23158

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:


New changeset af1d64d9f7a7cf673279725fdbaf4adcca51d41f by Victor Stinner in 
branch 'master':
bpo-42260: Main init modify sys.flags in-place (GH-23150)
https://github.com/python/cpython/commit/af1d64d9f7a7cf673279725fdbaf4adcca51d41f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

STINNER Victor  added the comment:


New changeset cfb41e80c1ac5940ec6f2246c9ab4a3d16ef757e by Victor Stinner in 
branch 'master':
bpo-42260: Reorganize PyConfig (GH-23149)
https://github.com/python/cpython/commit/cfb41e80c1ac5940ec6f2246c9ab4a3d16ef757e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22062
pull_request: https://github.com/python/cpython/pull/23150

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +22061
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/23149

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

New submission from STINNER Victor :

This issue is a follow-up of the PEP 567 which introduced the PyConfig C API 
and is related to PEP 432 which wants to rewrite Modules/getpath.c in Python.

I would like to add a new PyInterpreterState_SetConfig() function to be able to 
reconfigure a Python interpreter in C. One example is to write a custom 
sys.path, to implement of virtual environment (common request for embedded 
Python), etc. Currently, it's really complex to tune the Python configuration.

The use case is to tune Python for embedded Python. First, I would like to add 
new functions to the C API for that:

* PyInterpreterState_GetConfigCopy()
* PyInterpreterState_SetConfig()

The second step will to be expose these two functions in Python (I'm not sure 
where for now), and gives the ablity to tune the Python configuration in pure 
Python.

The site module already does that for sys.path, but it is running "too late" in 
the Python initialization. Here the idea is to configure Python before it does 
access any file on disk, after the "core" initialization and before the "main" 
initialization.

One concrete example would be to reimplement Modules/getpath.c in Python, 
convert it to a frozen module, and run it at Python startup to populate 
sys.path. It would allow to move some of the site code into this module to run 
it earlier.

Pseudo-code in C:
-
void init_core(void)
{
  // "Core" initialization
  PyConfig config;
  PyConfig_InitPython();
  PyConfig._init_main = 0
  Py_InitializeFromc();
  PyConfig_Clear();
}

void tune_config(void)
{
  PyConfig config;
  PyConfig_InitPython();

  // Get a copy of the current configuration
  PyInterpreterState_GetConfigCopy();  // <== NEW API!

  // ... put your code to tune config ...

  // dummy example, current not possible in Python
  config.bytes_warnings = 1;

  // Reconfigure Python with the updated configuration
  PyInterpreterState_SetConfig();  // <=== NEW API!
  PyConfig_Clear();
}
  
int main()
{
  init_core();
  tune_config(); // <=== THE USE CASE!
  _Py_InitializeMain();
  return Py_RunMain();
}
-

In this example, tune_config() is implemented in C. But later, it will be 
possible to convert the configuration to a Python dict and run Python code to 
tune the configuration.

The PEP 587 added a "Multi-Phase Initialization Private Provisional API":

* PyConfig._init_main = 0
* _Py_InitializeMain()

https://docs.python.org/dev/c-api/init_config.html#multi-phase-initialization-private-provisional-api

--
components: C API
messages: 380327
nosy: vstinner
priority: normal
severity: normal
status: open
title: [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com