[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[STINNER Victor]

STINNER Victor  added the comment:

> The problem is resolved if call PyGC_Collect() after PyDict_DelItemString(). 
> Is it expected to call PyGC_Collect() here?

Yeah sadly, to handle such reference cycles, you have to trigger an explicit 
garbage collection.

It doesn't sound like a bug to me.

Python 3.4 made this way better with PEP 442.

Anyway, that's an old issue with no activity since 2016. I close it.

--
nosy: +vstinner
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Changes by Jack Liu :


--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Nick Coghlan]

Nick Coghlan added the comment:

To be entirely clear about what's going on, the reference cycle seen in the 
example arises for *any* module level function, even if it's completely empty:

>>> def f():
... pass
... 
>>> f.__globals__["f"] is f
True

The existence of that cycle will then keep other module globals alive until the 
next garbage collection run.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Nick Coghlan]

Nick Coghlan added the comment:

The most likely relevant difference here is that Python 3.4+ no longer forcibly 
break cycles through the module globals when the module is deallocated: 
https://docs.python.org/dev/whatsnew/3.4.html#whatsnew-pep-442

Due to the implicit cycles created between function definitions and their 
global namespace via the __globals__ attribute on the function, this means that 
embedding applications will need to explicitly run a GC collection cycle after 
deleting a module in order to fully finalise it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

@serhiy.storchaka, The reference counts before PyDict_DelItemString are same on 
Python 3.3, 3.5 and 3.6.
Py_REFCNT(py_module)1
Py_REFCNT(py_dict)4

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Please output Py_REFCNT(py_module) and Py_REFCNT(py_dict) before deleting the 
module from sys.modules. Is there a difference between 3.5 and 3.6?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

Looks to me, there is NO reference cycle on the Simple object in the python 
test code. Why needs to call PyGC_Collect() here?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Josh Rosenberg]

Josh Rosenberg added the comment:

The fact that it's resolved by PyGC_Collect indicates there is a reference 
cycle somewhere. PyGC_Collect is just looking for cyclic garbage and breaking 
the cycles so it can be cleaned; it would happen eventually unless GC was 
explicitly disabled or the process exited before the next implicit GC 
invocation, so it means this bug is really about timing (and possibly cycles 
we'd prefer to avoid), not reference leaks.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

The problem is resolved if call PyGC_Collect() after PyDict_DelItemString(). Is 
it expected to call PyGC_Collect() here?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

I know there is a workaround to set the global variables to None at last in 
Python scripts. But my app just provide a framework for my customers to run 
python scripts. That means the workaround requires my customers to update their 
python scripts. That may make them unhappy :(. First, we need to confirm if 
it's a bug of Python 3.5. If it's a bug of Python 3.5, is there a workaround in 
my code C++ side to call Python C APIs to resolve the memory leak issue? If 
it's not a bug of Python 3.5, is there any mistake in my C++ code?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

I wrote the test code as below. I also attached the files in attachment.
Python
=

class Simple:
 def __init__( self ):
 print('Simple__init__')
 def __del__( self ):
 print('Simple__del__')

simple = None

def run():
global simple
simple = Simple()

if __name__ == '__main__':
run()
==

C++
=

#include "stdafx.h"
#include 
#include 
#include 
#include 
#include 
#include 

using namespace std;

namespace {
wstring readfile(const wchar_t *filename)
{
wifstream wifs;

wifs.open(filename);
//wifs.imbue(locale(wifs.getloc(), new codecvt_utf8()));
wstring wstr((std::istreambuf_iterator(wifs)),
std::istreambuf_iterator());
return wstr;
}

string wstrtostr(const wstring& ws)
{
std::wstring_convert converter;
return converter.to_bytes(ws);
}
}

int main()
{
cout << "Input py file full path:" << endl;
wstring filePath;
wcin >> filePath;
//filePath = L"L:\\Dev\\PyTest\\SimpleTest.py";

string moduleName = "__main__";

string script = wstrtostr(readfile(filePath.c_str()));
if (script.empty())
{
cout << "Invalid python file path" << endl;
return 1;
}

string sfileName = wstrtostr(filePath);

// Initialize the Python Interpreter
Py_Initialize();

PyObject *py_module = PyImport_AddModule(moduleName.c_str());
PyObject *py_dict = PyModule_GetDict(py_module);

PyObject* res = nullptr;
auto arena = PyArena_New();
if (arena)
{
auto mod = PyParser_ASTFromString(script.c_str(), 
sfileName.c_str(), Py_file_input, nullptr, arena);
if (mod)
{
auto co = PyAST_Compile(mod, sfileName.c_str(), 
nullptr, arena);
if (co)
{
res = PyEval_EvalCode((PyObject*)(co), py_dict, 
py_dict);
Py_DECREF(co);
}
}
PyArena_Free(arena);
}
if (res)
{
Py_DECREF(res);
}

// Delete the module from sys.modules
PyObject* modules = PyImport_GetModuleDict();
cout << "PyDict_DelItemString" << endl;
PyDict_DelItemString(modules, moduleName.c_str());

// May run many scripts here

// Finish the Python Interpreter
cout << "Py_Finalize" << endl;
Py_Finalize();

return 0;
}
===

The expected output in console should be:
Simple__init__
PyDict_DelItemString
Simple__del__
Py_Finalize

I tested with Python 3.2.5, 3.3.5, 3.5.1 and 3.6.0 beta.
It worked as expected with Python 3.2.5 and 3.3.5, but did not work Python 
3.5.1 and 3.6.0 beta.

On Python 3.5.1 and 3.6.0 beta, the output is:
Simple__init__
PyDict_DelItemString
Py_Finalize
Simple__del__
That means the Simple object is not released at PyDict_DelItemString, it's 
released at Py_Finalize.

So it it a regression bug since Python 3.5? I wish there is a solution to 
resolve memory leak issue.

--
Added file: http://bugs.python.org/file44748/PyTest.zip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

@eric.snow, Thank you for the replay. You understood right.

I run this module as __main__ module, so there is no other modules to reference 
this module. And as I debugged, the ref count of this module became 0 after 
calling PyDict_DelItemString, but global variable in this module was not 
released with Python 3.5.1. Is that memory leak? As I said, it worked on python 
3.3. Is it a regression in python 3.5.1? Any workaround to resolve this 
problem? It's an urgency issue to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Eric Snow]

Eric Snow added the comment:

To make sure I'm understanding:

* you are using PyDict_DelItemString() on sys.modules
* a module-level variable in the module is not getting cleaned up when the 
module is deleted from sys.modules
* this worked in Python 3.3 but not in 3.5

It may help to have a more complete test case, perhaps uploaded to this issue 
with the multiple files zipped up.

Also, does 3.2 have the same behavior as 3.3 or 3.5?  What about 3.6 (currently 
in beta)?

Note that deleting the module from sys.modules only reduces the refcount by 
one.  Other objects may still hold a reference to the module or any of its 
variables.  So nothing in the module would be cleaned up until the refcount 
hits zero.  For example, if the module was imported in another module and that 
second module still has a variable bound to the imported module (or the 
not-destroyed variable) then you would not see your printed message.

The fact that the behavior is different between 3.3 and 3.5 is concerning 
though.  I'd expect 3.3 to behave like 3.5 is.  It could be that a change in 
Lib/importlib (or Python/import.c) since 3.3 is leaking module references, 
though it's unlikely.

--
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Xiang Zhang]

Changes by Xiang Zhang :


--
nosy: +brett.cannon, eric.snow, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

I have a app loading python35.dll. Use python API PyImport_AddModule to run a 
py file. And use PyDict_DelItemString to delete the module. There is a global 
vailable in the py file. The global variable is not destroyed when calling 
PyDict_DelItemString to delete the module. The global variable is destroyed 
when calling Py_Finalize. It's too late. That cause the memory leak. Because 
the Py_Initialize is called at the app startup, the Py_Finalize is called at 
the app shutdown. 

But it is ok with python33.dll, the global variable can be destroyed when 
calling PyDict_DelItemString to delete the module.

How to resolve the problem? Is there a workaround? I need to use python35.dll 
and wish the global variable in a module can be released automatically when 
call PyDict_DelItemString to delete the module.

Here is the python test code:

class Simple:  
 def __init__( self ):  
 print('Simple__init__')
 def __del__( self ):  
 print('Simple__del__') 

simple = Simple()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Changes by Jack Liu :


--
components: +Extension Modules -Library (Lib)
title: Python 3.5.1 C API, the global available available is not destroyed when 
delete the module -> Python 3.5.1 C API, the global variable is not destroyed 
when delete the module

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com