[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[Nikhil Patel]

Nikhil Patel added the comment:

I'd like to respectfully disagree with the suggestions/conclusions mentioned 
here. I'll explain my reasoning and conclusions below.

First the short version - I'm fairly confident the leak actually has to do 
with: 
https://github.com/python/cpython/blob/master/Lib/test/test_tools/test_unparse.py#L282


my approach:

It was initially implied that we can begin to pinpoint the leak from within the 
test_files() method in test_tools.test_unparse.DirectoryTestCase.

I ran the tests using the command:
python_d -m test -R 3:3 test_tools.test_unparse -m test_files

I had test_files() print the files selected in line 282 (and limited sample 
size to 1).

Result:
(note: i shortened printed pathnames manually after the fact, as it printed the 
absolute paths)

python_d -m test -R 3:3 test_tools.test_unparse -m test_files
Run tests sequentially
0:00:00 [1/1] test_tools.test_unparse
Testing cpython\Lib\test\test_spwd.py
beginning 6 repetitions
123456
Testing cpython\Lib\copy.py
.Testing cpython\Lib\test\test_isinstance.py
.Testing cpython\Lib\test\test_bytes.py
.Testing cpython\Lib\test\test_csv.py
.Testing cpython\Lib\test\test_dynamic.py
.Testing cpython\Lib\test\test_xml_etree.py
.
test_tools.test_unparse leaked [3, 0, 13] references, sum=16
test_tools.test_unparse leaked [6, 2, 4] memory blocks, sum=12
test_tools.test_unparse failed

1 test failed:
test_tools.test_unparse

Total duration: 4 sec
Tests result: FAILURE

Then I made a slight modification: I replaced random.sample(names,1) on line 
282 with a list containing the absolute paths to each of those files it tested 
in the run I described above. That means each of those files would be loaded, 
compiled and their tests run 6 times.

Result:

python_d -m test -R 3:3 test_tools.test_unparse -m test_files
Run tests sequentially
0:00:00 [1/1] test_tools.test_unparse
beginning 6 repetitions
123456
..
1 test OK.

Total duration: 24 sec
Tests result: SUCCESS


Conclusion:
If the same source files are being read, compiled, and their tests run - with 
different results - then I do not believe the leak can be associated with the 
self.checkRoundTrip() method or the compile() method


I don't have an in-depth explanation why this is the case. But removing the 
call to random.sample() fixes the issue, I am reasonably sure of it. 

I presume to say this because I commented out lines 280-282 and instead I made 
it instead iterate through subsets of size 50 using slicing (so names[:50] etc) 
in line 284. I went through the entire set of files, 50 at a time. All ran 
successfully.

--
nosy: +npatel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[STINNER Victor]

STINNER Victor added the comment:

It seems like compile() only leaks when using PyCF_ONLY_AST:

compile(source, "filename", "exec", ast.PyCF_ONLY_AST)

It's maybe a leak in the PyAST_mod2obj() function when creating dictionaries of 
created AST objects. But in ast_dealloc() of Python/Python-ast.c, I see that a 
"DECREF" (Py_CLEAR) for the dict. Maybe it's something else.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[STINNER Victor]

STINNER Victor added the comment:

(Oops, I modified the file while uploading here. I reuploaded the correct 
bug.py.)

--
Added file: http://bugs.python.org/file46930/bug.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[STINNER Victor]

Changes by STINNER Victor :


Removed file: http://bugs.python.org/file46929/bug.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[STINNER Victor]

STINNER Victor added the comment:

I isolated the leak to attached bug.py script. Output:
---
haypo@selma$ ./python bug.py 
[ Top 10 differences ]
bug.py:40: size=6728 B (+6728 B), count=80 (+80), average=84 B
bug.py:46: size=2464 B (+2464 B), count=28 (+28), average=88 B
bug.py:43: size=2464 B (+2464 B), count=28 (+28), average=88 B
---

It seems like compiling a Python source to AST causes the leak. The Python 
source uses the following bytecodes:

 25 170 LOAD_DEREF   0 (trace)
172 LOAD_METHOD  0 (append)
174 LOAD_CONST   0 (None)
176 CALL_METHOD  1
178 POP_TOP

AST of the code:

Expr(value=Call(func=Attribute(value=Name(id='trace', ctx=Load()), 
attr='append', ctx=Load()), args=[NameConstant(value=None)], keywords=[]))

--
Added file: http://bugs.python.org/file46929/bug.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[Louie Lu]

Louie Lu added the comment:

Briefly speaking, I think this is not a *easy* issue. There is two things need 
to concern in this issue. 

First, it that the test case is *randomize*, lineno 282 randomly choice 10 file 
to test. So you will need to stabilize this first.

Second, the minimize range is inside `self.check_roundtrip`'s `compile`, the 
main reason that this issue can not be *easy issue* is at here, test_unparse 
read the test file and put it into compile, so generally, you can't find the 
bug inside the compile function, is the result after compile it. So the main 
ref leak is in other test files, not this one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[Milan Oberkirch]

Milan Oberkirch added the comment:

I tried to pin this leak down and got to the point where I suspect the compile 
call:

def check_roundtrip(self, code1, filename="internal"):
ast1 = compile(code1, filename, "exec", ast.PyCF_ONLY_AST)

the leaks occur with only that line in the function and do not occur without it.

To me it looks like a direct result of some leak somewhere in the tested code?

Garrett: I'm happy for you to take over (please leave a comment if you do)!

--
nosy: +zvyn

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[STINNER Victor]

STINNER Victor added the comment:

> Haypo: How do you know that this is leaked at test_files() of 
> test_tools.test_unparse.DirectoryTestCase? The result of your command didn't 
> reveal that is leaked at which test case or function.

Using the script I wrote which is attached to issue29512.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[Louie Lu]

Louie Lu added the comment:

Haypo: How do you know that this is leaked at test_files() of 
test_tools.test_unparse.DirectoryTestCase? The result of your command didn't 
reveal that is leaked at which test case or function.

--
nosy: +louielu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30542] [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase leaks references

[STINNER Victor]

STINNER Victor added the comment:

I tagged this issue as easy. I consider that it's a good exercice for new 
contributors. Core developers: please let new contributors try to fix it since 
this issue is not critical, try to explain how to fix it rather than writing 
the fix ;-)

--
keywords: +easy (C)
title: test_files() of test_tools.test_unparse.DirectoryTestCase leaks 
references -> [EASY] test_files() of test_tools.test_unparse.DirectoryTestCase 
leaks references

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com