Re: Why I fail so bad to check for memory leak with this code?
On 21/07/2022 23:39, Marco Sulla wrote: I've done this other simple test: #!/usr/bin/env python3 import tracemalloc import gc import pickle tracemalloc.start() snapshot1 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) for i in range(1000): pickle.dumps(iter([])) gc.collect() snapshot2 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) top_stats = snapshot2.compare_to(snapshot1, 'lineno') tracemalloc.stop() for stat in top_stats: print(stat) The result is: /home/marco/sources/test.py:14: size=3339 B (+3339 B), count=63 (+63), average=53 B /home/marco/sources/test.py:9: size=464 B (+464 B), count=1 (+1), average=464 B /home/marco/sources/test.py:10: size=456 B (+456 B), count=1 (+1), average=456 B /home/marco/sources/test.py:13: size=28 B (+28 B), count=1 (+1), average=28 B It seems that, after 10 million loops, only 63 have a leak, with only ~3 KB. It seems to me that we can't call it a leak, no? Probably pickle needs a lot more cycles to be sure there's actually a real leakage. If it was a leak, then the amount of memory used or the counts would increase with increasing iterations. If that's not happening, if the memory used and the counts stay the roughly the same, then it's probably not a leak, unless it's a leak of something that happens only once, such as creating a cache or buffer on first use. -- https://mail.python.org/mailman/listinfo/python-list
Re: Why I fail so bad to check for memory leak with this code?
I've done this other simple test: #!/usr/bin/env python3 import tracemalloc import gc import pickle tracemalloc.start() snapshot1 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) for i in range(1000): pickle.dumps(iter([])) gc.collect() snapshot2 = tracemalloc.take_snapshot().filter_traces( (tracemalloc.Filter(True, __file__), ) ) top_stats = snapshot2.compare_to(snapshot1, 'lineno') tracemalloc.stop() for stat in top_stats: print(stat) The result is: /home/marco/sources/test.py:14: size=3339 B (+3339 B), count=63 (+63), average=53 B /home/marco/sources/test.py:9: size=464 B (+464 B), count=1 (+1), average=464 B /home/marco/sources/test.py:10: size=456 B (+456 B), count=1 (+1), average=456 B /home/marco/sources/test.py:13: size=28 B (+28 B), count=1 (+1), average=28 B It seems that, after 10 million loops, only 63 have a leak, with only ~3 KB. It seems to me that we can't call it a leak, no? Probably pickle needs a lot more cycles to be sure there's actually a real leakage. -- https://mail.python.org/mailman/listinfo/python-list
RE: Resolving Weekday Schedules to Dates
Do you know of a library that resolves schedules like every Wednesday
at 3:00pm to absolute time, that is return the datetime of the next
occurrence?
It may be as simple, as this to add seven days, assuming a slip of a second
is not important or even a hour when things change.
Enddate = Begindate + timedelta(days=7)
This assumes you used code like this before:
from datetime import datetime
from datetime import timedelta
Begindate = datetime.strptime("2020-10-11 12:34")
Of course if you want to include times, you adjust to your needs. And, of
course, the above can be implemented as an iterator that keeps producing as
long as you need.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Why I fail so bad to check for memory leak with this code?
This naif code shows no leak:
import resource
import pickle
c = 0
while True:
pickle.dumps(iter([]))
if (c % 1) == 0:
max_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print(f"iteration: {c}, max rss: {max_rss} kb")
c += 1
--
https://mail.python.org/mailman/listinfo/python-list
Re: Why I fail so bad to check for memory leak with this code?
On Thu, 21 Jul 2022 at 22:28, MRAB wrote: > > It's something to do with pickling iterators because it still occurs > when I reduce func_76 to: > > @trace > def func_76(): > pickle.dumps(iter([])) It's too strange. I found a bunch of true memory leaks with this decorator. It seems to be reliable. It's correct with pickle and with iter, but not when pickling iters. -- https://mail.python.org/mailman/listinfo/python-list
Re: Why I fail so bad to check for memory leak with this code?
On 21/07/2022 20:47, Marco Sulla wrote:
I tried to check for memory leaks in a bunch of functions of mine using a
simple decorator. It works, but it fails with this code, returning a random
count_diff at every run. Why?
import tracemalloc
import gc
import functools
from uuid import uuid4
import pickle
def getUuid():
return str(uuid4())
def trace(func):
@functools.wraps(func)
def inner():
tracemalloc.start()
snapshot1 = tracemalloc.take_snapshot().filter_traces(
(tracemalloc.Filter(True, __file__), )
)
for i in range(100):
func()
gc.collect()
snapshot2 = tracemalloc.take_snapshot().filter_traces(
(tracemalloc.Filter(True, __file__), )
)
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
tracemalloc.stop()
for stat in top_stats:
if stat.count_diff > 3:
raise ValueError(f"count_diff: {stat.count_diff}")
return inner
dict_1 = {getUuid(): i for i in range(1000)}
@trace
def func_76():
pickle.dumps(iter(dict_1))
func_76()
It's something to do with pickling iterators because it still occurs
when I reduce func_76 to:
@trace
def func_76():
pickle.dumps(iter([]))
--
https://mail.python.org/mailman/listinfo/python-list
Re: Resolving Weekday Schedules to Dates
On Thursday, July 21, 2022 at 8:19:34 PM UTC+1, rambius wrote: > Hello, > > Do you know of a library that resolves schedules like every Wednesday > at 3:00pm to absolute time, that is return the datetime of the next > occurrence? > > Regards > rambius > > P.S. > > -- > Tangra Mega Rock: http://www.radiotangra.com Not exactly, and it's not a library as such; but Andy Valencia's 'mypim' http://sources.vsta.org:7100/mypim/index includes a calendar facility, so you can add repeating dates and query 'what's coming up in the next 30 days' etc. I haven't looked at this very closely, but it looks like a version of this might scratch an itch of my own. you might also look at 'rrule': https://dateutil.readthedocs.io/en/stable/rrule.html and also look at iCalendar support in general. I would be interested to learn if you come across anything to fit your needs. Regards, J^n -- https://mail.python.org/mailman/listinfo/python-list
Why I fail so bad to check for memory leak with this code?
I tried to check for memory leaks in a bunch of functions of mine using a
simple decorator. It works, but it fails with this code, returning a random
count_diff at every run. Why?
import tracemalloc
import gc
import functools
from uuid import uuid4
import pickle
def getUuid():
return str(uuid4())
def trace(func):
@functools.wraps(func)
def inner():
tracemalloc.start()
snapshot1 = tracemalloc.take_snapshot().filter_traces(
(tracemalloc.Filter(True, __file__), )
)
for i in range(100):
func()
gc.collect()
snapshot2 = tracemalloc.take_snapshot().filter_traces(
(tracemalloc.Filter(True, __file__), )
)
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
tracemalloc.stop()
for stat in top_stats:
if stat.count_diff > 3:
raise ValueError(f"count_diff: {stat.count_diff}")
return inner
dict_1 = {getUuid(): i for i in range(1000)}
@trace
def func_76():
pickle.dumps(iter(dict_1))
func_76()
--
https://mail.python.org/mailman/listinfo/python-list
Re: Resolving Weekday Schedules to Dates
> Do you know of a library that resolves schedules like every Wednesday > at 3:00pm to absolute time, that is return the datetime of the next > occurrence? Take a look at the `rrule` module in the `dateutil` package: https://dateutil.readthedocs.io/en/stable/rrule.html Skip -- https://mail.python.org/mailman/listinfo/python-list
Resolving Weekday Schedules to Dates
Hello, Do you know of a library that resolves schedules like every Wednesday at 3:00pm to absolute time, that is return the datetime of the next occurrence? Regards rambius P.S. -- Tangra Mega Rock: http://www.radiotangra.com -- https://mail.python.org/mailman/listinfo/python-list
Re: exec() an locals() puzzle
Thanks. That cdef-locals concept is consistent with the following example:
def f():
i = 1
def g(): print('i' in globals(), 'i' in locals())
def h(): print('i' in globals(), 'i' in locals()); i
g()
h()
f()
False False
False True
It is a mystery, which may be why the documentation for globals() and
locals() is 2-line long.
Le mer. 20 juill. 2022, à 19 h 31, Martin Di Paola <
martinp.dipa...@gmail.com> a écrit :
> I did a few tests
>
> # test 1
> def f():
> i = 1
> print(locals())
> exec('y = i; print(y); print(locals())')
> print(locals())
> a = eval('y')
> print(locals())
> u = a
> print(u)
> f()
>
> {'i': 1}
> 1
> {'i': 1, 'y': 1}
> {'i': 1, 'y': 1}
> {'i': 1, 'y': 1, 'a': 1}
> 1
>
> # test 2
> def f():
> i = 1
> print(locals())
> exec('y = i; print(y); print(locals())')
> print(locals())
> a = eval('y')
> print(locals())
> y = a
> print(y)
> f()
> {'i': 1}
> 1
> {'i': 1, 'y': 1}
> {'i': 1}
> Traceback (most recent call last):
> NameError: name 'y' is not defined
>
>
> So test 1 and 2 are the same except that the variable 'y' is not
> present/present in the f's code.
>
> When it is not present, exec() modifies the f's locals and adds an 'y'
> to it but when the variable 'y' is present in the code (even if not
> present in the locals()), exec() does not add any 'y' (and the next
> eval() then fails)
>
> The interesting part is that if the 'y' variable is in the f's code
> *and* it is defined in the f's locals, no error occur but once again the
> exec() does not modify f's locals:
>
> # test 3
> def f():
> i = 1
> y = 42
> print(locals())
> exec('y = i; print(y); print(locals())')
> print(locals())
> a = eval('y')
> print(locals())
> y = a
> print(y)
> f()
> {'i': 1, 'y': 42}
> 1
> {'i': 1, 'y': 1}
> {'i': 1, 'y': 42}
> {'i': 1, 'y': 42, 'a': 42}
> 42
>
> Why does this happen? No idea.
>
> I may be related with this:
>
> # test 4
> def f():
> i = 1
> print(locals())
> exec('y = i; print(y); print(locals())')
> print(locals())
> print(y)
> f()
> Traceback (most recent call last):
> NameError: name 'y' is not defined
>
> Despite exec() adds the 'y' variable to f's locals, the variable is not
> accessible/visible from f's code.
>
> So, a few observations (by no means this is how the vm works):
>
> 1) each function has a set of variables defined by the code (let's call
> this "code-defined locals" or "cdef-locals").
> 2) each function also has a set of "runtime locals" accessible from
> locals().
> 3) exec() can add variables to locals() (runtime) set but it cannot add
> any to cdef-locals.
> 4) locals() may be a superset of cdef-locals (but entries in cdef-locals
> which value is still undefined are not shown in locals())
> 5) due rule 4, exec() cannot add a variable to locals() if it is already
> present in the in cdef-locals.
> 6) when eval() runs, it uses locals() set for lookup
>
> Perhaps rule 5 is to prevent exec() to modify any arbitrary variable of
> the caller...
>
> Anyways, nice food for our brains.
>
> On Wed, Jul 20, 2022 at 04:56:02PM +, george trojan wrote:
> >I wish I could understand the following behaviour:
> >
> >1. This works as I expect it to work:
> >
> >def f():
> >i = 1
> >print(locals())
> >exec('y = i; print(y); print(locals())')
> >print(locals())
> >exec('y *= 2')
> >print('ok:', eval('y'))
> >f()
> >
> >{'i': 1}
> >1
> >{'i': 1, 'y': 1}
> >{'i': 1, 'y': 1}
> >ok: 2
> >
> >2. I can access the value of y with eval() too:
> >
> >def f():
> >i = 1
> >print(locals())
> >exec('y = i; print(y); print(locals())')
> >print(locals())
> >u = eval('y')
> >print(u)
> >f()
> >
> >{'i': 1}
> >1
> >{'i': 1, 'y': 1}
> >{'i': 1, 'y': 1}
> >1
> >
> >3. When I change variable name u -> y, somehow locals() in the body of
> >the function loses an entry:
> >
> >def f():
> >i = 1
> >print(locals())
> >exec('y = i; print(y); print(locals())')
> >print(locals())
> >y = eval('y')
> >print(y)
> >f()
> >
> >{'i': 1}
> >1
> >{'i': 1, 'y': 1}
> >{'i': 1}
> >
>
> >---NameError
> >Traceback (most recent call last)
> >Input In [1], in () 7 print(y) 8 # y
> >= eval('y') 9 #print('ok:', eval('y'))---> 10 f()
> >
> >Input In [1], in f() 4 exec('y = i; print(y); print(locals())')
> > 5 print(locals())> 6 y = eval('y') 7 print(y)
> >
> >File :1, in
> >NameError: name 'y' is not defined1.
> >
> >Another thing: within the first exec(), the print order seems
> >reversed. What is going on?
> >
> >BTW, I am using python 3.8.13.
> >
> >George
> >--
> >https://mail.python.org/mailman/listinfo/python-list
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: list indices must be integers or slices, not str
On 2022-07-20 4:45 PM, Chris Angelico wrote:
On Wed, 20 Jul 2022 at 23:50, Peter Otten <__pete...@web.de> wrote:
I found
https://peps.python.org/pep-3101/
"""
PEP 3101 – Advanced String Formatting
...
An example of the ‘getitem’ syntax:
"My name is {0[name]}".format(dict(name='Fred'))
It should be noted that the use of ‘getitem’ within a format string is
much more limited than its conventional usage. In the above example, the
string ‘name’ really is the literal string ‘name’, not a variable named
‘name’. The rules for parsing an item key are very simple. If it starts
with a digit, then it is treated as a number, otherwise it is used as a
string.
Cool. I think this is a good justification for a docs patch, since
that really should be mentioned somewhere other than a historical
document.
ChrisA
I have submitted the following -
https://github.com/python/cpython/issues/95088
Frank
--
https://mail.python.org/mailman/listinfo/python-list
