[issue33565] strange tracemalloc results

Alexander Mohr added the comment: this is how my friends talk here, see: https://english.stackexchange.com/questions/11816/is-guy-gender-neutral -- ___ Python tracker ___

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: of course, sorry, I meant in a gender neutral way -- ___ Python tracker ___ ___ Python-bugs-list m

[issue33565] strange tracemalloc results

Mariatta Wijaya added the comment: Alexander, in the future please use gender-neutral words such as "folks" and "y'all" instead of "guys". Thanks. -- nosy: +Mariatta ___ Python tracker __

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: ok fair enough, however a caveat is that it's not necessarily end of function as I was able to expand the function by a few more lines and the callstack stayed the same, however I'm guessing the interpreter was smart enough to realize the calls below the try/

[issue33565] strange tracemalloc results

INADA Naoki added the comment: > The callstack I initially mentioned still has no explanation and we now know > is not correct. I don't think so. The callstack is Python's callstack where caused leak in C. (Python doesn't call malloc directly) In this case, local variables are freed "at end

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: actually, thinking about this more on my way to work, this should NOT be closed. The callstack I initially mentioned still has no explanation and we now know is not correct. It should either have listed something related to warnings, or nothing at all () or

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: Ok I've verified that the patch does indeed fix the leak detected. Thank you very much INADA for knowing that there was a leak in the warnings module, I would have never guessed, especially given the tracemalloc stack given. Had it showed a callstack where

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: I'll try with that fix and see if the leak is fixed, is the reasoning that if the warning happens after the try/except scope and that's why the callstack has it? -- ___ Python tracker

[issue33565] strange tracemalloc results

STINNER Victor added the comment: > Hmm, GH-4587 is merged in 3.6 branch in last November, but 3.6.5 doesn't > include it? We decided to not fix the memory leak in Python 3.6: https://github.com/python/cpython/pull/4587#issuecomment-347659163 """ This change makes warning.warn() 1.4x slower o

[issue33565] strange tracemalloc results

INADA Naoki added the comment: I was wrong. It is not merged into 3.6 branch. The pull request is just closed. Now nothing strange in tracemalloc. -- ___ Python tracker ___ _

[issue33565] strange tracemalloc results

Change by Antoine Pitrou : -- nosy: -pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyth

[issue33565] strange tracemalloc results

INADA Naoki added the comment: Hmm, GH-4587 is merged in 3.6 branch in last November, but 3.6.5 doesn't include it? -- ___ Python tracker ___ _

[issue33565] strange tracemalloc results

INADA Naoki added the comment: I confirmed the leak on 3.6.5, and I can't reproduce it on 3.7.0b4. -- ___ Python tracker ___ ___ Pyt

[issue33565] strange tracemalloc results

STINNER Victor added the comment: > Thanks, I can reproduce. Are you testing the master branch? I fixed a memory leak in the _warnings module for ignored ResourceWarning warnings. -- ___ Python tracker

[issue33565] strange tracemalloc results

INADA Naoki added the comment: Alexander Mohr: Thanks, I can reproduce. I found ResourceWarning is happened when head_object. re may be used for filtering warning. You can use `-W always` to see the warnings. I don't know why the leak happened yet. -- __

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: INADA Naoki: Unfortunately you'll need to use credentials from a free AWS account: https://aws.amazon.com/free/. Then create a credentials file in ~/.aws/credentials: https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html -- ___

[issue33565] strange tracemalloc results

INADA Naoki added the comment: I can't reproduce it: File "test1.py", line 19, in do_get_obj response = s3_client.get_object(Bucket='archpi.dabase.com', Key='style.css') File "/Users/inada-n/tmp/boto-leak/venv/lib/python3.6/site-packages/botocore/client.py", line 314, in _api_call

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: yes, memory does go up. If you click the botocore bug link you'll see a graph of memory usage over time. -- ___ Python tracker ___ _

[issue33565] strange tracemalloc results

STINNER Victor added the comment: > this means 21 blocks were not released, and in this case leaked because > nothing should be held onto after the first iteration (creating the initial > connector in the connection pool. In the head object case that's going to be > a new connector per itera

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: that's not going to affect http://pytracemalloc.readthedocs.io/api.html#get_traced_memory. There is no filter for that :) as to your sum that's exactly what my original callstack lists: 21 memory blocks: 4.7 KiB this means 21 blocks were not released, and i

[issue33565] strange tracemalloc results

STINNER Victor added the comment: Oh. Usually, I strip traces allocated by tracemalloc using filters: http://pytracemalloc.readthedocs.io/examples.html#pretty-top snapshot = snapshot.filter_traces(( tracemalloc.Filter(False, ""), tracemalloc.Filter(False, ""), )) -

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: I believe your method is flawed, when enabling tracemalloc it's going to be using memory as well to store the traces. I still believe you need to use the method I mentioned and further even if we don't take into account the total memory the traces I mentione

[issue33565] strange tracemalloc results

STINNER Victor added the comment: A memory leak is when each iteration adds a fixed number of bytes of the memory: I'm talking about tracemalloc.get_traced_memory()[0] value. For example, if you compare the total traced memory between your iteration 30 and iteration 20, you get a value 10N. I

[issue33565] strange tracemalloc results

Change by INADA Naoki : -- nosy: +inada.naoki ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: here's another problem, if I change that function signature to: def test(s3_client): try: method = getattr(s3_client, sys.argv[1]) method(Bucket='archpi.dabase.com', Key='doesnotexist') except botocore.exceptions.ClientError as e:

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: actually it does show in 3.5.2, it doesn't show when using get_object, but it does when using head_object, and they both throw the same exception, so weird. -- ___ Python tracker

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: so based on my last comment I just realized we could easily see if something was amiss by comparing results from python 3.5.2 to 3.6.5 and low and behold the callstack in question does not appear in the tracemalloc results from 3.5.2. --

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: I realize it doesn't track leaks, it's a tool to help find leaks when used correctly :) This example should be similar to using the compare snapshots mechanism as I start tracking from a stable point (several iterations in after a gc), and then compare to an

[issue33565] strange tracemalloc results

STINNER Victor added the comment: tracemalloc doesn't track memory leaks. It tracks memory allocations. I'm not sure that you are using tracemalloc properly. I suggest you to enable tracemalloc as soon as possible, take snapshot frequently (eg. at each iteration) and write them on disk, and l

[issue33565] strange tracemalloc results

STINNER Victor added the comment: Once you found and fixed your memory leak, any documentation enhancement is welcome :-) -- ___ Python tracker ___ ___

[issue33565] strange tracemalloc results

Alexander Mohr added the comment: here's a version that tries to do something similar but does not reproduce the issue -- Added file: https://bugs.python.org/file47602/tracemalloc_test2.py ___ Python tracker ___

[issue33565] strange tracemalloc results

Antoine Pitrou added the comment: Can you post a reproducer that doesn't involve S3? -- nosy: +pitrou ___ Python tracker ___ ___ Pyt

[issue33565] strange tracemalloc results

Andrew Svetlov added the comment: Victor, could you take a look? -- nosy: +asvetlov, vstinner ___ Python tracker ___ ___ Python-bugs

[issue33565] strange tracemalloc results

Change by Alexander Mohr : -- type: -> behavior ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.

[issue33565] strange tracemalloc results

New submission from Alexander Mohr : while investigating https://github.com/boto/botocore/issues/1464 I used tracemalloc (like I've done before in 3.5.2) to try to figure out where the leak was. To my surprise tracemalloc listed stacks that didn't make any sense. Strangest example is the top