[issue46788] regrtest fails to start on missing performance counter names

[Eryk Sun]

Change by Eryk Sun :


Removed file: https://bugs.python.org/file50695/loadtracker.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Eryk Sun]

Change by Eryk Sun :


--
Removed message: https://bugs.python.org/msg415781

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Steve Dower]

Steve Dower  added the comment:

Backports have been merged

--
resolution:  -> fixed
stage: backport needed -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Eryk Sun]

Eryk Sun  added the comment:

I implemented a ctypes prototype that replaces the registry-based 
implementation with the API calls PdhOpenQueryW(), PdhAddEnglishCounterW(), 
PdhCollectQueryData(), PdhGetRawCounterValue(), and PdhCloseQuery(). I'm 
attaching the script, but here's the class itself without the ctypes 
definitions:

class WindowsLoadTracker():
"""
This class asynchronously reads the system "Processor Queue Length"
counter to calculate the system load on Windows.  A raw thread is
used to avoid interfering with tests of the threading module.
"""

def __init__(self):
self._values = []
self._load = None
self._hrunning = kernel32.CreateEventW(None, True, False, None)
self._hstopped = kernel32.CreateEventW(None, True, False, None)
self._hquery = wintypes.HANDLE()
self._hcounter = wintypes.HANDLE()
pdh.PdhOpenQueryW(None, None, ctypes.byref(self._hquery))
pdh.PdhAddEnglishCounterW(self._hquery,
  r"\System\Processor Queue Length",
  None,
  ctypes.byref(self._hcounter))
pdh.PdhCollectQueryData(self._hquery)
_thread.start_new_thread(self._update_load, (), {})

def _update_load(self,
 # Localize module access to prevent shutdown errors.
 WaitForSingleObject=_winapi.WaitForSingleObject,
 SetEvent=kernel32.SetEvent):
# run until signaled to stop
while WaitForSingleObject(self._hrunning, 1000):
self._calculate_load()
# notify stopped
SetEvent(self._hstopped)

def _calculate_load(self,
# Lcalize module access to prevent shutdown errors.
PdhCollectQueryData=pdh.PdhCollectQueryData,
PdhGetRawCounterValue=pdh.PdhGetRawCounterValue):
counter_type = wintypes.DWORD()
raw = PDH_RAW_COUNTER()
PdhCollectQueryData(self._hquery)
PdhGetRawCounterValue(self._hcounter,
  ctypes.byref(counter_type),
  ctypes.byref(raw))
if raw.CStatus < 0:
return
processor_queue_length = raw.FirstValue

# Use an exponentially weighted moving average, imitating the
# load calculation on Unix systems.
# 
https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation
# 
https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average
if self._load is not None:
self._load = (self._load * LOAD_FACTOR_1
+ processor_queue_length  * (1.0 - 
LOAD_FACTOR_1))
elif len(self._values) < NVALUE:
self._values.append(processor_queue_length)
else:
self._load = sum(self._values) / len(self._values)

def getloadavg(self):
return self._load

def close(self,
  # Localize module access to prevent shutdown errors.
  WaitForSingleObject=_winapi.WaitForSingleObject,
  INFINITE=_winapi.INFINITE,
  SetEvent=kernel32.SetEvent,
  CloseHandle=_winapi.CloseHandle,
  PdhCloseQuery=pdh.PdhCloseQuery):
if self._hrunning is None:
return
# Tell the update thread to quit.
SetEvent(self._hrunning)
# Wait for the update thread to stop before cleanup.
WaitForSingleObject(self._hstopped, INFINITE)
CloseHandle(self._hrunning)
CloseHandle(self._hstopped)
PdhCloseQuery(self._hquery)
self._hrunning = self._hstopped = None

def __del__(self):
self.close()
return

--
Added file: https://bugs.python.org/file50695/loadtracker.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Jeremy Kloth]

Jeremy Kloth  added the comment:

Backports state that they are ready...  I'm just a little uneasy as I've never 
used cherry_picker before.  3.10 went smooth, but 3.9 required manual merging.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Steve Dower]

Steve Dower  added the comment:

Jeremy - backports need to be done manually. Can I leave that with you and just 
ping me on here when you're ready for a merge?

--
stage:  -> backport needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Eryk Sun]

Eryk Sun  added the comment:

I was just wondering whether it's worth implementing it using the API. To be 
clear, I wasn't implying to hold off on applying Jeremy's PR. The existing code 
is causing him problems, and he has a working solution.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Steve Dower]

Steve Dower  added the comment:

"So close" being some months ago ;)

I need to retrigger address sanitiser so that it passes before merging, but 
after that, I'll merge (and hopefully the backport is clean).

Copying Eryk Sun's comment here for posterity:

This would be less fragile and more useful in general if support were added to 
_winapi for PdhOpenQueryW(), PdhAddEnglishCounterW(), PdhCollectQueryData(), 
PdhCollectQueryDataEx(), PdhGetFormattedCounterValue(), and PdhCloseQuery(). 
The English counter name is "\System\Processor Queue Length" -- no need to 
worry about localized names, hard-coded values, or hard-coded struct sizes and 
offsets.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Jeremy Kloth]

Jeremy Kloth  added the comment:

With 3.8 so close to security only, I would doubt it is worth it anymore.  I've 
just run the PR against HEAD and it still works as is, so should be good to go.

--
versions:  -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Steve Dower]

Steve Dower  added the comment:

You're closer to this stuff than I am, so if you say it works for your 
scenarios and it doesn't break the existing ones, I'll merge it.

Do we need to backport to 3.8 at this point?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Jeremy Kloth]

Jeremy Kloth  added the comment:

OK, I know it has been a busy month for Python, but this issue is really 
hampering my bug fixing efforts.  It makes the complete regrtest useless for 
me.  I am required to run each affected test directly so it is possible to miss 
side-effects of changes made.

The original PR is now 9 months old.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Jeremy Kloth]

Change by Jeremy Kloth :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46788] regrtest fails to start on missing performance counter names

[Jeremy Kloth]

New submission from Jeremy Kloth :

When attempting to run the test harness, I receive the following:

Traceback (most recent call last):
  File "", line 198, in _run_module_as_main
  File "", line 88, in _run_code
  File "C:\Public\Devel\cpython\main\Lib\test\__main__.py", line 2, in 
main()
^^
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\main.py", line 736, 
in main
Regrtest().main(tests=tests, **kwargs)
^^
  File "C:\Public\Devel\cpython\main\Lib\contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
^
  File "C:\Public\Devel\cpython\main\Lib\contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
^
  File "C:\Public\Devel\cpython\main\Lib\test\support\os_helper.py", line 396, 
in temp_dir
yield path
^^
  File "C:\Public\Devel\cpython\main\Lib\contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
^
  File "C:\Public\Devel\cpython\main\Lib\test\support\os_helper.py", line 427, 
in change_cwd
yield os.getcwd()
^
  File "C:\Public\Devel\cpython\main\Lib\test\support\os_helper.py", line 449, 
in temp_cwd
yield cwd_dir
^
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\main.py", line 658, 
in main
self._main(tests, kwargs)
^
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\main.py", line 704, 
in _main
self.win_load_tracker = WindowsLoadTracker()

  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\win_utils.py", line 
41, in __init__
self.start()

  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\win_utils.py", line 
70, in start
counter_name = self._get_counter_name()
   
  File "C:\Public\Devel\cpython\main\Lib\test\libregrtest\win_utils.py", line 
90, in _get_counter_name
system = counters_dict['2']
 ~^
KeyError: '2'

This is due to my machine missing the localized names for the performance 
counters.  Other performance monitoring tools operate just fine.

While I have been working around this issue for some time, it has become 
difficult to seperate the workarounds from actually changes in the test harness.

The PR (https://github.com/python/cpython/pull/26578) from 
https://bugs.python.org/issue44336 also solves this issue by accessing the 
counters directly instead of relying on their localized names.

--
components: Tests, Windows
messages: 413493
nosy: jkloth, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: regrtest fails to start on missing performance counter names
versions: Python 3.10, Python 3.11, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com