[Python-Dev] Hold multiple tracebacks per-object in tracemalloc

2019-11-19 Thread Sümer Cip 
Hi everyone,

First of all, I would like to thank for such a useful tool for debugging
memory issues. I am pretty proud as a Python lover that we have such a
tool: tracemalloc:)

I would like to ask if something is feasible/practical regarding
tracemalloc:

AFAIU, tracemalloc holds a single traceback per-object:

Example:
class A:

def __init__(self):
self._d = []

def alloc(self):
self._d += [1] * 512

def alloc_more(self):
self._d += [1] * 512


tracemalloc.start(25)
a = A()
a.alloc()
a.alloc_more()
stats = tracemalloc.take_snapshot().statistics('traceback')
for t in stats:
print(t)
print(t.traceback.format())

"""
Output:
...
5.py:38: size=9264 B, count=1, average=9264 B
['  File "5.py", line 38', 'a.alloc_more()', '  File "5.py", line 32',
'self._d += [1] * 512']
"""

Now, allocation of alloc and alloc_more functions seems to be merged into a
single traceback. Is it possible we add (maybe a constant number of
tracebacks) per object? This might be pretty useful to find the responsible
leak in situations where same object is modified in many places?

I have been playing with this module only lately on tracemalloc, so please
correct if a major piece is missing in my example or my understanding is
wrong in some way.

Thanks as always!
-- 
Sümer Cip
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/GLTOV6CV5AGM465CFSRSZHIVU7R4436G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

2019-11-19 Thread Victor Stinner 
Hi,

Le mar. 19 nov. 2019 à 01:42, Pablo Galindo Salgado
 a écrit :
> What do people feel about exposing Tools/parser/unparse.py in the standard 
> library?

I like the idea. I compared Lib/os.py to the unparse output and now I
have many questions :-)

(*) unparse loses all formatting and comments. The output is not
really "pretty". If you want to put unparse in the stdlib, maybe we
should design an API to be able to plug an external formatter to get
"PEP 8"-like coding style or Black coding style for example.

(*) unparse adds many useless parenthesis. The algorithm seems naive.
For example, it adds "()" to "class _AddedDllDirectory():". It also
adds parenthesis around yield, like "(yield (top, dirs, nondirs))",
whereas the AST node was at "top level": it isn't a sub-expression.
Maybe this algortihm should be made smarter.

(*) the main() function is only used for testing. I would prefer to
move this code to Lib/test/test_tools/test_unparse.py and simply
remove it. For example, to avoid "import os" in unparse.

(*) newlines in docstring are rendered as two characters: "\\" + "n"
(escaped newline: \n), rather than a newline character. I would expect
a newline, it's more common that \n... But it may "break" inline
doctests rendering... Maybe it should be an option (render newlines as
newline character or escape them?), or maybe even let the user choose
how to render them using a callback (that's related to the "pluggable
formatter" question).

(*) Do you plan to provide any warranty about the output stability?
Like Black formatter which tries to avoid changing output between
minor versions. Or should we expect to get a different output per
Python version? If we provide a warranty, new tests should be written.
Currently, test_unparse only compares AST, not the text output.

(*) Indentation is hardcoded to 4 spaces. What if I want 2 spaces or
something different? Should it become an option?

(*) Float infinity is replaces with 1e309. Again, maybe someone wants
to render this differently? It sounds like an arbitrary choice (which
"works" as expected).

(*) Do we talk about 'string' vs "string" quotes? :-)


Maybe it would help to have a written down PEP to answer to these
questions and maybe specify the module API.


> We could add the public interface to the ast.py module or a new one if people 
> feel strongly about it.

I would prefer to keep a separated module, like "import ast.unparse"
or "import unparse".

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/EU7FGOUGEUYBOA74U4OHZH2236VR3NTZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Hold multiple tracebacks per-object in tracemalloc

2019-11-19 Thread Victor Stinner 
Hi,

Le mar. 19 nov. 2019 à 18:33, Sümer Cip  a écrit :
> First of all, I would like to thank for such a useful tool for debugging 
> memory issues. I am pretty proud as a Python lover that we have such a tool: 
> tracemalloc:)

As the module author: you're welcome ;-)


> AFAIU, tracemalloc holds a single traceback per-object:

"tracemalloc" name means that a "trace" is stored in mapping
associated to an allocated memory block. A trace is just: (size,
traceback): memory block in bytes, and the Python traceback where the
memory block has been allocated.

tracemalloc is not aware of Python objects at all. For example, it is
unable to associate an object __dict__ to the object. It works at
malloc() and free() level, not at the _PyObject_GC_Malloc() level.

One advantage is that memory which is not tracked by the GC is tracked
by tracemalloc. Another is that the implementation is "simple" (from
my point of view :-)).

One drawback is that tracemalloc traces are harder to analyze.


> """
> Output:
> ...
> 5.py:38: size=9264 B, count=1, average=9264 B
> ['  File "5.py", line 38', 'a.alloc_more()', '  File "5.py", line 32', '  
>   self._d += [1] * 512']
> """

When the internal storage of a Python list grows, the old memory block
is released and a new memory block is allocated. From the tracemalloc
point of view, the old trace is removed, a new trace is created. There
is no relationship between the two traces, even if realloc() has been
used. In tracemalloc, realloc() traces work as two isolated
operations: free() + malloc().


> Now, allocation of alloc and alloc_more functions seems to be merged into a 
> single traceback. Is it possible we add (maybe a constant number of 
> tracebacks) per object? This might be pretty useful to find the responsible 
> leak in situations where same object is modified in many places?

You might try to take snapshots frequently and write an heuristic to
link traces which look similar... good luck with that :-)

--

There are other tools to track Python memory usage which may help you
to provide a view closer to what you are looking for.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/4DTIRXTKUB5XBBNWNGASGD2ZMOS2WDVJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

2019-11-19 Thread Pablo Galindo Salgado 
> (*) unparse loses all formatting and comments. The output is not
really "pretty". If you want to put unparse in the stdlib, maybe we
should design an API to be able to plug an external formatter to get
"PEP 8"-like coding style or Black coding style for example.

The ast does not have comment or formatting information so those will
be lost as soon as you have an ast object. Formatting code with a formatter
after generating it should be straightforward enough that I am not sure
that we
should expose an API to hook that into the unparser. Especially because it
will likely
imply still an adaptor over the autoformatted API.

> unparse adds many useless parenthesis. The algorithm seems naive.
For example, it adds "()" to "class _AddedDllDirectory():". It also
adds parenthesis around yield, like "(yield (top, dirs, nondirs))",
whereas the AST node was at "top level": it isn't a sub-expression.
Maybe this algortihm should be made smarter.

I agree as this hurts visibility a bit, we can solve this issue separately.

> the main() function is only used for testing. I would prefer to
move this code to Lib/test/test_tools/test_unparse.py and simply
remove it. For example, to avoid "import os" in unparse.

Totally agreed.

> Do you plan to provide any warranty about the output stability?
Like Black formatter which tries to avoid changing output between
minor versions. Or should we expect to get a different output per
Python version? If we provide a warranty, new tests should be written.
Currently, test_unparse only compares AST, not the text output.

I would say that it provides the same stability as the ast module does. In
particular, I think the contract is that ast.parse(ast.unparse(the_ast))
should
produce the same ast as the_ast. Any other guaranteed over the generated
code can
bite us very soon as is very limiting. Maybe we can think about stabilizing
it
if many users ask for it.

>  newlines in docstring are rendered as two characters: "\\" + "n"
(escaped newline: \n), rather than a newline character. I would expect
a newline, it's more common that \n... But it may "break" inline
doctests rendering... Maybe it should be an option (render newlines as
newline character or escape them?), or maybe even let the user choose
how to render them using a callback (that's related to the "pluggable
formatter" question).

Hu, i still prefer that the unparse() function does something very
straighfoward
and anything else can be done via post-processing (or maybe inheriting from
the
Unparser class if we would like to go that route). This is based on my
opinion that
providing guarantees too soon over the generated code other than the ast
generated
by parsing is the same can limit us very soon.

> Indentation is hardcoded to 4 spaces. What if I want 2 spaces or
something different? Should it become an option?

No in my opinion (at least initially).

>  Float infinity is replaced with 1e309. Again, maybe someone wants
to render this differently? It sounds like an arbitrary choice (which
"works" as expected).

That is not true. float('inf') is rendered as float('inf´):

>>> Unparser(ast.parse("float('inf')"))
float('inf')


> Do we talk about 'string' vs "string" quotes?

Sounds like a painful discussion. But I would say that it does not matter
for the time being (the resulting code is not something that should be
maintained).
And if you want to stabilize you can use an autoformatted.

> Maybe it would help to have a written down PEP to answer to these
questions and maybe specify the module API.

I think a PEP may be too much but I can write if people think it makes
sense. Being
the API as simple as 'ast.unparse' I think is unnecessary as my view is
that we should
not guarantee anything other than roundtrip over the generated source
initially.

> I would prefer to keep a separated module, like "import ast.unparse"
or "import unparse".

Why? I think ast.unparse is a natural fit. It will likely be only one
function exposed.


On Tue, 19 Nov 2019 at 20:57, Victor Stinner  wrote:

> Hi,
>
> Le mar. 19 nov. 2019 à 01:42, Pablo Galindo Salgado
>  a écrit :
> > What do people feel about exposing Tools/parser/unparse.py in the
> standard library?
>
> I like the idea. I compared Lib/os.py to the unparse output and now I
> have many questions :-)
>
> (*) unparse loses all formatting and comments. The output is not
> really "pretty". If you want to put unparse in the stdlib, maybe we
> should design an API to be able to plug an external formatter to get
> "PEP 8"-like coding style or Black coding style for example.
>
> (*) unparse adds many useless parenthesis. The algorithm seems naive.
> For example, it adds "()" to "class _AddedDllDirectory():". It also
> adds parenthesis around yield, like "(yield (top, dirs, nondirs))",
> whereas the AST node was at "top level": it isn't a sub-expression.
> Maybe this algortihm should be made smarter.
>
> (*) the main() function is only used for testing. I would prefer to
> move this code to Lib/te

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

2019-11-19 Thread Nathaniel Smith 
On Mon, Nov 18, 2019 at 4:41 PM Pablo Galindo Salgado
 wrote:
>
> Hi,
>
> What do people feel about exposing Tools/parser/unparse.py in the standard 
> library? Here is my initial rationale:
>
> * The tool already needs to be maintained and updated as is tested as part of 
> the test suite.
> * I have used the tool almost all the time I needed to deal with AST 
> transformations.
> * The public interface will have a very low surface API, keeping maintaining 
> it (the public interface) a very small burden IMHO.
>
> We could add the public interface to the ast.py module or a new one if people 
> feel strongly about it.

How does it compare to Berker's popular and well-maintained PyPI
package for this? https://github.com/berkerpeksag/astor

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/EDOBFEJDKANKWCAYEVLWTTXSCM3OIMXE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

2019-11-19 Thread Batuhan Taskaya 
This is a really great idea, after pretty ast.dump it will be great
addition for people who works with AST code transformations.

I think with the new end_lineno and end_col_offset information output can
be little bit close the original.

I want to volunteer the work if there is an open opportunity after
consensus about adding this.

On Tue, Nov 19, 2019, 3:45 AM Pablo Galindo Salgado 
wrote:

> Hi,
>
> What do people feel about exposing Tools/parser/unparse.py in the standard
> library? Here is my initial rationale:
>
> * The tool already needs to be maintained and updated as is tested as part
> of the test suite.
> * I have used the tool almost all the time I needed to deal with AST
> transformations.
> * The public interface will have a very low surface API, keeping
> maintaining it (the public interface) a very small burden IMHO.
>
> We could add the public interface to the ast.py module or a new one if
> people feel strongly about it.
>
> Does anyone feel strongly against this or have any objection that I am not
> contemplating?
>
> Regards from rainy London,
> Pablo Galindo Salgado
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/QG5GKAJ6LXT5A64OPRIUGVYYJQGFAQ5R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

2019-11-19 Thread Victor Stinner 
Le mar. 19 nov. 2019 à 22:12, Pablo Galindo Salgado
 a écrit :
> >  Float infinity is replaced with 1e309. Again, maybe someone wants
> to render this differently? It sounds like an arbitrary choice (which
> "works" as expected).
>
> That is not true. float('inf') is rendered as float('inf´):
>
> >>> Unparser(ast.parse("float('inf')"))
> float('inf')

I was thinking at:

>>> Unparser(ast.parse("1e999"))
1e309

Maybe just move the constant as a class attribute, so it can be
overriden in a subclass?

> > I would prefer to keep a separated module, like "import ast.unparse"
> or "import unparse".
>
> Why? I think ast.unparse is a natural fit. It will likely be only one 
> function exposed.

It's mostly to minimize the number of imports on "import ast". unparse
requires extra imports like tokenize which has also tons of
dependencies.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/YOQEOSIJIVNWIHX42HZ7SZXSEDLTV6VC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

2019-11-19 Thread Ivan Levkivskyi 
Just wanted to add my +1 to this idea. Moving it to ast module will add
little maintenance costs, but will make it easier to use.

--
Ivan



On Tue, 19 Nov 2019 at 00:46, Pablo Galindo Salgado 
wrote:

> Hi,
>
> What do people feel about exposing Tools/parser/unparse.py in the standard
> library? Here is my initial rationale:
>
> * The tool already needs to be maintained and updated as is tested as part
> of the test suite.
> * I have used the tool almost all the time I needed to deal with AST
> transformations.
> * The public interface will have a very low surface API, keeping
> maintaining it (the public interface) a very small burden IMHO.
>
> We could add the public interface to the ast.py module or a new one if
> people feel strongly about it.
>
> Does anyone feel strongly against this or have any objection that I am not
> contemplating?
>
> Regards from rainy London,
> Pablo Galindo Salgado
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/GKFY5NBBD5VL5EODKMH7D2TFNB7RWZEJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.9.0a1 available for testing

2019-11-19 Thread Łukasz Langa 
Go get it here: https://www.python.org/downloads/release/python-390a1/ 

This is an early developer preview of Python 3.9

Python 3.9 is still in development. This releasee, 3.9.0a1 is the first of six 
planned alpha releases. Alpha releases are intended to make it easier to test 
the current state of new features and bug fixes and to test the release 
process. During the alpha phase, features may be added up until the start of 
the beta phase (2020-05-18) and, if necessary, may be modified or deleted up 
until the release candidate phase (2020-08-10). Please keep in mind that this 
is a preview release and its use is not recommended for production environments.

Major new features of the 3.9 series, compared to 3.8

Many new features for Python 3.9 are still being planned and written. Among the 
new major new features and changes so far:

PEP 602 , Python adopts a stable 
annual release cadence
BPO 38379 , garbage collection does not 
block on resurrected objects;
BPO 38692 , os.pidfd_open added that allows 
process management without races and signals;
A number of standard library modules (audioop, ast, grp, _hashlib, pwd, 
_posixsubprocess, random, select, struct, termios, zlib) are now using the 
stable ABI defined by PEP 384 .
(Hey, fellow core developer, if a feature you find important is missing from 
this list, let Łukasz know .)
The next pre-release of Python 3.9 will be 3.9.0a2, currently scheduled for 
2019-12-16.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/G2A2PXSKK7IZOLLIDC455F33SWRZTX6A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

2019-11-19 Thread Guido van Rossum 
On Tue, Nov 19, 2019 at 10:24 PM Nathaniel Smith  wrote:

> On Mon, Nov 18, 2019 at 4:41 PM Pablo Galindo Salgado
>  wrote:
> >
> > Hi,
> >
> > What do people feel about exposing Tools/parser/unparse.py in the
> standard library? Here is my initial rationale:
> >
> > * The tool already needs to be maintained and updated as is tested as
> part of the test suite.
> > * I have used the tool almost all the time I needed to deal with AST
> transformations.
> > * The public interface will have a very low surface API, keeping
> maintaining it (the public interface) a very small burden IMHO.
> >
> > We could add the public interface to the ast.py module or a new one if
> people feel strongly about it.
>
> How does it compare to Berker's popular and well-maintained PyPI
> package for this? https://github.com/berkerpeksag/astor
>

Does that even have unparse() functionality? From the README it seems to
focus on a nicer ast.dump(), which is quite different (in behavior and how
it's used) from unparse().

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/3Z34GG2B2REAR7JHIBZJCKDGL7656NBB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

2019-11-19 Thread Batuhan Taskaya 
Yes, there is a to_source function which allows AST to source conversation.

On Wed, Nov 20, 2019, 9:02 AM Guido van Rossum  wrote:

> On Tue, Nov 19, 2019 at 10:24 PM Nathaniel Smith  wrote:
>
>> On Mon, Nov 18, 2019 at 4:41 PM Pablo Galindo Salgado
>>  wrote:
>> >
>> > Hi,
>> >
>> > What do people feel about exposing Tools/parser/unparse.py in the
>> standard library? Here is my initial rationale:
>> >
>> > * The tool already needs to be maintained and updated as is tested as
>> part of the test suite.
>> > * I have used the tool almost all the time I needed to deal with AST
>> transformations.
>> > * The public interface will have a very low surface API, keeping
>> maintaining it (the public interface) a very small burden IMHO.
>> >
>> > We could add the public interface to the ast.py module or a new one if
>> people feel strongly about it.
>>
>> How does it compare to Berker's popular and well-maintained PyPI
>> package for this? https://github.com/berkerpeksag/astor
>>
>
> Does that even have unparse() functionality? From the README it seems to
> focus on a nicer ast.dump(), which is quite different (in behavior and how
> it's used) from unparse().
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/3Z34GG2B2REAR7JHIBZJCKDGL7656NBB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/EA5ZN4IONDV5AIXSAGT4YRINY64AYIW6/
Code of Conduct: http://python.org/psf/codeofconduct/