[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

STINNER Victor added the comment:

About _PY_ONCEVAR_INIT() vs _Py_IDENTIFIER.

Using _Py_IDENTIFIER works well if you have an API accepting directly a 
_Py_IDENTIFIER*. If you call functions requesting a PyObject*, you need to call 
_PyUnicode_FromId() and test for failure. If you start by calling 
_PyUnicode_FromId() when the object is not initialized yet, above you have to 
use var.object, whereas previously Serhiy and Nick weren't confortable with 
this specific case.

I prefer to use _PY_ONCEVAR_INIT() to keep a regular PyObject* variable and 
makes the code simpler.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

STINNER Victor added the comment:

3rd round of the API:

static PyObject *assertion_error = NULL;
...
if (_PY_ONCEVAR_INIT(assertion_error,
 PyUnicode_InternFromString("AssertionError"))) {
return 0;
}
...

(Not the best example, _Py_IDENTIFIER() would be more appropriate here ;-))

--

I just added a second commit to remove the next field from _Py_Identifier and 
reuse _PyOnceVar_Set() in _PyUnicode_FromId().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I think we can get rid of _Py_ONCEVAR(). Just keep current declarations. 
_PY_ONCEVAR_INIT() can work even with non-static global variables. It could 
work even with non-PyObject pointers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

STINNER Victor added the comment:

About the pull request: to be clear, I know that some modified local variables 
should use _Py_IDENTIFIER() rather than _Py_ONCEVAR(), but I chose to use 
_Py_ONCEVAR() just to give examples of the API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

STINNER Victor added the comment:

> * do we ever declare non-statics this way? (OTOH, if we do, this could be an 
> opportunity to hide them behind read-only accessor functions)
> * do we ever declare more specific types than PyObject * this way? (OTOH, if 
> that's the uncommon case, requiring a cast to the more specific type probably 
> wouldn't hurt)

Well, with my 2nd API, I'm not sure that the macro to declare a variable is 
very simple:

+/* Declare a static PyObject* variable which is only initialized once.
+   _PyOnceVar_Fini() will clear the variable at Python finalization. */
+#define _Py_ONCEVAR(var) \
+static PyObject* var = NULL

Technically, the variable doesn't have to be static. But do we want to use this 
API for global variables initialized once? It would increase the memory usage, 
whereas currently we have specialized code like _PyUnicode_Fini() which clears 
its unicode_empty singleton. I don't want to touch this code. At least, not yet.

If you want to support other types than PyObject*, _PY_ONCEVAR_INIT() macro can 
cast the first argument to PyObject*.

I would prefer to start with something simpler, and discuss case by case for 
other variables.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[Nick Coghlan]

Nick Coghlan added the comment:

Passing var_decl was based on not knowing the answer to two questions:

* do we ever declare non-statics this way? (OTOH, if we do, this could be an 
opportunity to hide them behind read-only accessor functions)
* do we ever declare more specific types than PyObject * this way? (OTOH, if 
that's the uncommon case, requiring a cast to the more specific type probably 
wouldn't hurt)

As far as stack usage goes, all _Py_OnceVar instances should be in the data 
segment when using the linked list approach, so they shouldn't impact stack 
usage, and doing so retains the benefit of avoiding dynamic memory allocation 
just to track the static variable declarations. Since the macro is 
dereferencing the address of a static variable in the initialiser of another 
static variable, that shouldn't require any runtime stack space either. OTOH, I 
haven't actually ever tried compiling a macro like that, so it's entirely 
possible compilers won't like it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The new API rather combines pthread_once() and pthread_cleanup_push().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> "use gc_next/gc_refs for managing a linked list"

Forgot about this idea. Not all objects have the GC header.

But if they have it they could be removed from the GC list and added in the 
separate list since in any case the garbage collector shouldn't destroy objects 
referenced by static variables.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

STINNER Victor added the comment:

I wrote a new different API: https://github.com/python/cpython/pull/780

New C API for variables only initialized once to be able to clear
them at exit:

New macro _Py_ONCEVAR(var) to declare a variable
New macro _PY_ONCEVAR_INIT(var, expr) to initialize a variable
once
New function _PyOnceVar_Set() to explicitly set a variable once
to initialize it
New _PyOnceVar_Fini() function clearing all variables (initialized
once) at exit

Variables keep their PyObject* type, but the API hides an internal C array 
tracking all Python objects to Py_DECREF them at exit in _PyOnceVar_Fini().

I used Nick's naming scheme since it seems like pthread has an existing API 
which is similar, but different (my API doesn't require an initialization 
callback).

I really prefer the second API using PyObject*.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +684

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

My apologies Victor. It seems that the browser on my netbook shown only the 
part of changes and I seen only the change in array_array___reduce_ex__().

I like Nick's idea for hiding the indirection, but passing var_decl doesn't 
look having much sense to me. It should always be "static PyObject *".

I don't have a preference for the name, but the purpose of a new API is not 
just to ensure that a piece of initialization code is executed at most once, 
but that the object will be destroyed and the reference will be cleared after 
finalizing the interpreter. Seems this is different from the purpose of 
pthread_once().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

STINNER Victor added the comment:

Serhiy: "use gc_next/gc_refs for managing a linked list"

I'm not sure that I understand your idea. These fields are already used 
internally by the garbage collector. If I modify one of these fields, it would 
corrupt the GC, no?

Serhiy: "You can use a dynamic array of PyObject** instead of a linked list for 
collecting references to "static variables""

Ah yes, I like the idea of using a single array to track all variables. An 
array reduces memory fragmentation and is simple to maintain, since the API 
only needs two operations: list.append(obj) and list.clear(). The API already 
requires to check for errors, so another memory allocation failure wouldn't be 
suprising.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

STINNER Victor added the comment:

#define _Py_ONCE_VAR(var_decl, var) \
  var_decl var = NULL;
  static _Py_OnceVar var ## _once_meta = {.next = NULL, .obj_ref = 
(PyObject **) }

Yeah, I had a similar idea, but I fear that it will increase the usage of the C 
stack memory.

See the issue #28858: my old _PyObject_CallArg1() macro allocated an implicit 
array on the stack and increased the stack usage, whereas the purpose of the 
macro was to *reduce* the stack usage (just the opposite!). So I removed the 
macro.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

STINNER Victor added the comment:

Serhiy: "The patch contains an example of using _Py_STATICVAR(). But it doesn't 
use _PY_STATICVAR_INIT()."

I'm not sure that I understand your comment. All modified code use 
_Py_STATICVAR() to declare the declare. All modified code use 
_PY_STATICVAR_INIT() to initialize the variable, except of 
array_array___reduce_ex__() which uses explicitly _PyStaticVar_Set() beause its 
initialization code is more complex.

_PyStaticVar_Set() is the low-level function, I would prefer to avoid it since 
it fails with an assertion error if it's called twice. _PY_STATICVAR_INIT() 
helper should be perfer since it makes the code shorter, but it's similar to 
simple expressions like PyUnicode_FromString("\n").

I tried to include examples of usage of both APIs to give you an idea of the 
API.

I chose to *not* patch the whole Python code base, because I would like to 
first get a review of a the API. It seems like alternatives have been proposed 
;-)


> I like the idea in general, but I want to see more examples.

Which kind of other examples do you need?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[Nick Coghlan]

Nick Coghlan added the comment:

I like this idea in principle, and suspect it may be helpful in the 
implementation of the new thread-specific-storage API proposed in PEP 539 (also 
see http://bugs.python.org/issue25658 ).

How would you feel about calling it _Py_ONCE_VAR? The use case here is quite 
similar to the concepts behind the pthread_once API (just with the lifecycle 
tied to Py_Initialize/Py_Finalize rather than the C level process), and it's 
the "initialise-on-first-use" behaviour that's significant here, moreso than 
the fact that the typical storage target is a static variable.

As far as the linked list goes, the nice aspect of Victor's approach is that it 
doesn't need to do any additional runtime memory allocations - all the storage 
involved is still statically allocated in the modules that initialise the 
values, there are just some extra  pointer assignments to link everything 
together (with the GIL protecting against race conditions).

However, the user visible ".obj" indirection could still be avoided at the cost 
of an additional pointer per entry:

typedef struct _Py_OnceVar {
struct _Py_OnceVar *next;
PyObject **obj_ref;
} _Py_OnceVar;

#define _Py_ONCE_VAR(var_decl, var) \
  var_decl var = NULL;
  static _Py_OnceVar var ## _once_meta = {.next = NULL, .obj_ref = 
(PyObject **) }

Intended declaration:

_Py_ONCE_VAR(static PyObject *, array_reconstructor);

_Py_ONCE_VAR_INIT would similarly be adjusted to assign the supplied value to 
"var", while also setting "var_once_meta.next" to hook the value into the 
linked list.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The patch contains an example of using _Py_STATICVAR(). But it doesn't use 
_PY_STATICVAR_INIT(). _PY_STATICVAR_INIT() can be used if extract the code of 
getting _array_reconstructor into separate function.

I like the idea in general, but I want to see more examples. Ideally -- the 
patch should use a new API for *all* static PyObject* variables. This will help 
to design good API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

STINNER Victor added the comment:

The purpose of the issue is to fix memory leaks like issue #21387 (this one is 
not the best example since it seems like the leak doesn't come from a static 
variable.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The problem with existing static variables are that they are not properly 
cleared. When the Python interpreter is finalized and reinitialized they can 
contain invalid references. This patch fixes this issue.

> * It requires to write "var.obj" instead of just "var" to access the Python 
> object

You can use a dynamic array of PyObject** instead of a linked list for 
collecting references to "static variables" or use gc_next/gc_refs for managing 
a linked list.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +677

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +eric.snow, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

New submission from STINNER Victor:

When I read Serhiy Storshaka's idea in issue #29878, it recalled me an old idea 
of writing a generalization of the _Py_IDENTIFIER() API.

_Py_IDENTIFIER is an API to initialize a static string variable once. The 
_Py_Identifier structure has a "next" field to create a single-linked chained 
list. It allows to clear all variables at exit in 
_PyUnicode_ClearStaticStrings().

I propose a similar API but for any PyObject* object, to be able to clear all 
static variables at exit. It should help to release all memory in Py_Finalize() 
and have a safer Python finalization.

See attached pull request for the API itself.

"Static variables" in C are variables with a limited scope: a single C file or 
a single function.

It seems like the API can remove some lines of code. Example of patch:

@@ -1452,14 +1450,14 @@ compiler_mod(struct compiler *c, mod_ty mod)
 {
 PyCodeObject *co;
 int addNone = 1;
-static PyObject *module;
-if (!module) {
-module = PyUnicode_InternFromString("");
-if (!module)
-return NULL;
+_Py_STATICVAR(module);
+
+if (_PY_STATICVAR_INIT(, PyUnicode_InternFromString(""))) {
+return 0;
 }
+
 /* Use 0 for firstlineno initially, will fixup in assemble(). */
-if (!compiler_enter_scope(c, module, COMPILER_SCOPE_MODULE, mod, 0))
+if (!compiler_enter_scope(c, module.obj, COMPILER_SCOPE_MODULE, mod, 0))
 return NULL;
 switch (mod->kind) {
 case Module_kind:

--

Drawbacks of the API:

* It adds one pointer per static variables, so increase the memory footprint of 
8 bytes per variable
* It requires to write "var.obj" instead of just "var" to access the Python 
object


The API doesn't try to remove duplicated objects. I consider that it's not an 
issue since functions like PyLong_FromLong() and 
PyUnicode_InternFromString("c string") already do it for us. Some functions 
create mutable variables like PyImport_Import() which creates an empty list.

--

Note: Eric Snow proposed a solution "solving multi-core Python":

* https://mail.python.org/pipermail/python-ideas/2015-June/034177.html
* http://ericsnowcurrently.blogspot.fr/2016/09/solving-mutli-core-python.html

I'm not sure if this API would help or not to implement such idea, but Eric's 
project is experimental and wasn't taken in account when designing the API.

--
components: Interpreter Core
messages: 28
nosy: haypo
priority: normal
severity: normal
status: open
title: Add a new private API for "static C variables" (_PyStaticVar) to clear 
them at exit
type: resource usage
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com