[issue46072] Unify handling of stats in the CPython VM

[Brandt Bucher]

Brandt Bucher  added the comment:


New changeset 580cd9ab2992b7df6f4815020b5841e14a5a6977 by Brandt Bucher in 
branch 'main':
bpo-46072: Add detailed failure stats for BINARY_OP (GH-31289)
https://github.com/python/cpython/commit/580cd9ab2992b7df6f4815020b5841e14a5a6977


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29481
pull_request: https://github.com/python/cpython/pull/31324

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Brandt Bucher]

Change by Brandt Bucher :


--
nosy: +brandtbucher
nosy_count: 2.0 -> 3.0
pull_requests: +29449
pull_request: https://github.com/python/cpython/pull/31289

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29425
pull_request: https://github.com/python/cpython/pull/31259

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29422
pull_request: https://github.com/python/cpython/pull/31254

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29420
pull_request: https://github.com/python/cpython/pull/31251

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29419
pull_request: https://github.com/python/cpython/pull/31250

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29404
pull_request: https://github.com/python/cpython/pull/31234

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:


New changeset f71a69aa9209cf67cc1060051b147d6afa379bba by Mark Shannon in 
branch 'main':
bpo-46072: Output stats as markdown with collapsible sections. (GH-31228)
https://github.com/python/cpython/commit/f71a69aa9209cf67cc1060051b147d6afa379bba


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29398
pull_request: https://github.com/python/cpython/pull/31228

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29381
pull_request: https://github.com/python/cpython/pull/31211

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 9c979d7afd839abbb080028bdfeb73727e5cf633 by Mark Shannon in 
branch 'main':
bpo-46072: Merge dxpairs into py_stats. (GH-31197)
https://github.com/python/cpython/commit/9c979d7afd839abbb080028bdfeb73727e5cf633


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29368
pull_request: https://github.com/python/cpython/pull/31197

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 062460e8fd54e53c9a1a6f175ef49c9d730851b8 by Mark Shannon in 
branch 'main':
bpo-46072: Improve LOAD_METHOD stats (GH-31104)
https://github.com/python/cpython/commit/062460e8fd54e53c9a1a6f175ef49c9d730851b8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29291
pull_request: https://github.com/python/cpython/pull/31108

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29288
pull_request: https://github.com/python/cpython/pull/31105

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29287
pull_request: https://github.com/python/cpython/pull/31104

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29263
pull_request: https://github.com/python/cpython/pull/31079

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 187930f74c44e460ba09c60ba5d9bb4fac543d8f by Mark Shannon in 
branch 'main':
bpo-46072: Add some frame stats. (GH-31060)
https://github.com/python/cpython/commit/187930f74c44e460ba09c60ba5d9bb4fac543d8f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29242
pull_request: https://github.com/python/cpython/pull/31060

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 48be46ec1f3f8010570165daa1da4bf9961f3a83 by Mark Shannon in 
branch 'main':
bpo-46072: Add some object layout and allocation stats (GH-31051)
https://github.com/python/cpython/commit/48be46ec1f3f8010570165daa1da4bf9961f3a83


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29233
pull_request: https://github.com/python/cpython/pull/31051

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 90ab138bbdc63763ad825ed6d4821367c09c4015 by Mark Shannon in 
branch 'main':
bpo-46072: Add simple stats for Python calls. (GH-30989)
https://github.com/python/cpython/commit/90ab138bbdc63763ad825ed6d4821367c09c4015


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +29168
pull_request: https://github.com/python/cpython/pull/30989

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:


New changeset efd6236d36b292c2c43540132c87cf8425e8d627 by Mark Shannon in 
branch 'main':
bpo-46072:  Add top level stats struct (GH-30169)
https://github.com/python/cpython/commit/efd6236d36b292c2c43540132c87cf8425e8d627


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +28386
pull_request: https://github.com/python/cpython/pull/30169

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +28363
pull_request: https://github.com/python/cpython/pull/30145

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:

The --enable-stats option is for CPython development and shouldn't be turned on 
for a release version, so I'm not really concerned about people attacking their 
own machines.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Christian Heimes]

Christian Heimes  added the comment:

I just noticed that you are using hard-coded paths with /tmp for the pystats 
directory. That's problematic and opens the possibility of a symlink race 
attack.

Could please add exclusive create to _Py_PrintSpecializationStats()? The will 
prevent symlink attacks. fopen() mode "x" is not generally available in all 
libcs. You have to combine open() and fdopen():


int flags = O_WRONLY | O_CREAT | O_EXCL;
#ifdef O_NOFOLLOW
flags |= O_NOFOLLOW;
#endif
#ifdef O_CLOEXEC
flags |= O_CLOEXEC;
#endif

int fd = open(path, flags);
if (fd >= 0) {
FILE *fout = fdopen(fd, "w");
}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 4506bbede1644e985991884964b43afa7ee6f609 by Mark Shannon in 
branch 'main':
bpo-46072: Document --enable-stats option. (GH-30139)
https://github.com/python/cpython/commit/4506bbede1644e985991884964b43afa7ee6f609


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +28357
pull_request: https://github.com/python/cpython/pull/30139

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Christian Heimes]

Christian Heimes  added the comment:

Could you please add the new option to Doc/using/configure.rst ?

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 342b93f9f28746abb7b221a61d5a9b26ccbb395a by Mark Shannon in 
branch 'main':
bpo-46072: Add --with-pystats configure option to simplify gathering of VM 
stats (GH-30116)
https://github.com/python/cpython/commit/342b93f9f28746abb7b221a61d5a9b26ccbb395a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

Change by Mark Shannon :


--
keywords: +patch
pull_requests: +28338
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/30116

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46072] Unify handling of stats in the CPython VM

[Mark Shannon]

New submission from Mark Shannon :

By "stats" I mean the internal numbers gathered by the VM for performance and 
debugging. This has nothing to do with any statistics module.

Currently various parts of the VM gather stats: the GC, dicts, the bytecode 
interpreter, type lookup cache, etc.

These stats have various compile time flags to turn them on and off. They have 
differing ways to display the stats, and no unified way to gather stats across 
different runs.

For the specialization stats we dump stats, which we can parse to collate stats 
across runs.

We should:
1. Add a --with-pystats config flag (like with-pydebug) to turn on stat 
gathering at build time.
2. Convert the other stats to the format used by the specialization stats, so 
all stats can be parsed and collated.

--
messages: 408543
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: Unify handling of stats in the CPython VM

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com