Thanks for the feedback. After doing some research, it does seem to be a very Windows-specific issue. A general set_clock() API is probably better then.
I will be looking into doing patches for set_clock() and a new time() method on Windows (without period management, of course). One quick question though: how do you feel about ctypes? I see that's it's occasionally used in stdlib, but given that you went out of your way to create the _overlapped module for asyncio, would it be acceptable to use ctypes for timeGetTime? Mathias Den søndag den 18. maj 2014 04.25.52 UTC+2 skrev Guido van Rossum: > > On Sat, May 17, 2014 at 6:36 PM, Mathias Kærlev <[email protected]<javascript:> > > wrote: > >> I wouldn't want asyncio to default to calling timeBeginPeriod either, as >> it would hurt battery life and potentially affect other applications, which >> is not an ideal default behavior. >> However, it would be great if there was an API to request a certain time >> resolution from the eventloop. >> > > This looks like it would be Windows specific functionality though. I'd > prefer to do it somewhat differently. > > >> This would be useful to e.g. my LoopingCall class, where I would be able >> to find the highest call frequency and feed that to the eventloop. >> Generally, if the application *knows* that it will need a high-resolution >> timer, it would be able to request one. >> > > Only on Windows, AFAIK. > > >> I've changed the previous code to add new set_clock and >> set_clock_resolution methods: >> https://github.com/matpow2/cuwo/blob/py3/cuwo/win32.py >> >> To show what my original example would look like with the new methods: >> http://bpaste.net/show/pcrIZAV3R7QaefJGMtRw/ >> >> If setting the clock resolution was to be added to asyncio, it would >> obviously be a no-op on platforms where there are no timer resolution >> issues. >> In any case, I would definitely consider adding a set_clock method (so if >> someone wants to use e.g. QPC, they can). >> > > I propose to leave the resolution management to the application's > Windows-specific code. You can write code that sets the resolution to what > you want and then pass both it and your timer code to the event loop. > > One problem with using timeGetTime is that we don't know the initial >> system-wide timer resolution. >> We can be conservative and guess that it has not been changed from >> startup, in which case we would get a resolution of 15.6ms. >> However, a lot of applications (Chrome, GUI applications, Flash, games, >> etc.) change the global timer resolution to 1ms, so in reality, it may >> already be a lot more precise than we expect. >> There is also the possibility of using a period which is higher than the >> current system-wide resolution, in which case we would also be getting a >> more precise timer. >> From what I can see, there is no way to query the current resolution >> without using NtQueryTimerResolution, which is an undocumented MS function. >> > > All this just adds to my desire to make changing the system resolution the > app's problem, and just provide set_clock(time_func, clock_resolution) as a > loop method. (BTW, the resolution is only used in one place, and even that > place is dubious -- it's mostly because Victor doesn't like to busy-wait if > the sleep time is shorter than requested. Perhaps we can live without it > after all.) > > >> However, we can still look at timeGetTime deltas in the time() method, >> and if we find a delta that is lower than the current clock resolution, we >> use that as our resolution. >> > > Again, I think that ought to be made the app's problem. > > >> The last alternative would be to set _clock_resolution to the highest >> possible resolution, i.e. 1ms, even if the actual resolution may be lower. >> This way, we can ensure the scheduler never tries to fire a call which is >> in the future. This is the approach I have taken in my code. >> With some testing, it appears that my system in fact uses a timer >> resolution of 1 ms most of the time (since a lot of applications set it by >> default), so to me, this seems like the best approach. >> > > There are very few side effects of setting _clock_resolution lower than > the actual clock resolution. > > >> At the very least, timeGetTime could be used as the default time() method >> on Windows. >> Unfortunately, timeBeginPeriod does not affect GetTickCount (i.e. >> time.monotonic), as GetTickCount will try to emulate the 15.6ms resolution >> even if the system-wide resolution has been changed. >> > > Hm. If you want to argue over the implementation of time.monotonic(), I'd > prefer it if you did that on python-dev, as it is not specific to asyncio. > But it's another reason why I want the period management out of asyncio. :-) > > >> Hopefully my input has been valuable to some extent! >> > > I suggest that you file a tracker issue and propose a patch just to change > the default time() function on Windows. And perhaps a separate one to lobby > for a set_clock() API. > > >> It would be great if asyncio becomes the go-to eventloop for all Python >> things. >> > > Yes, that's my hope too! > > >> This is probably why I'm very keen on having a good default >> implementation. For all platforms. ;-) >> >> Cheers, >> Mathias >> >> Den lørdag den 17. maj 2014 23.29.30 UTC+2 skrev Guido van Rossum: >>> >>> Sorry, saw this after sending my reply. >>> >>> Looking at your code and the MS docs for timeBeginPeriod() I can see why >>> we didn't choose this as our default asyncio clock or as the implementation >>> of time.monotonic() -- the docs warn that the requested period can affect >>> the performance of the entire system. Your implementation also selfishly >>> sets exactly the period that you are interested in. >>> >>> But it seems that using timeBeginPeriod() is pretty independent from >>> timeGetTime() -- perhaps it even affects time.monotonic()? If not, perhaps >>> we can leave calling timeBeginPeriod() up to the app, but still override >>> the time() method? We would then have to decide whether we want to just >>> override time() and _clock_resolution in the Windows-specific Loop classes >>> (_WindowsSelectorEventLoop and ProactorEventLoop) or whether we want to >>> define an API to change the time implementation more generally. >>> >>> >>> On Sat, May 17, 2014 at 6:54 AM, Mathias Kærlev <[email protected]>wrote: >>> >>>> I experimented with offsetting the timer using the timeout passed to >>>> the selector, and that did seem to fix it for my case. >>>> However, I found it difficult to see if that would be a general >>>> solution to the problem, especially when the selector would actually >>>> return >>>> something on each iteration (in which case we couldn't use the timeout). >>>> >>>> I ended up just subclassing the eventloops from windows_events.py: >>>> https://github.com/matpow2/cuwo/blob/py3/cuwo/win32.py >>>> >>>> It uses timeGetTime and timeGetPeriod/timeEndPeriod with a timer >>>> resolution of 10ms, which should be enough for the majority of cases. >>>> According to the documentation, timeGetTime should work exactly like >>>> GetTickCount, except for the fact that you can change the timer resolution. >>>> >>>> I still consider this to be a hack though. It would be nice if >>>> something was done about this eventually. >>>> >>>> Thanks again, >>>> Mathias >>>> >>>> Den lørdag den 17. maj 2014 11.35.08 UTC+2 skrev Mathias Kærlev: >>>> >>>>> That seems like a cop-out to me. >>>>> >>>>> I get that GetTickCount64 does not have high precision, but you could >>>>> consider using a timeGetTime/GetTickCount64 hybrid (like in GLib) for >>>>> asyncio. >>>>> Otherwise, with these scheduling issues, the default eventloop >>>>> implementation is practically unusable for games on Windows, which I >>>>> think >>>>> is a shame. >>>>> Obviously, I agree that a precise, stable (i.e. not QPC) monotonic >>>>> clock would be the only proper timer solution, but ultimately, it should >>>>> not come at the expense of usability. >>>>> timeGetTime/timeBeginPeriod appears to be both high-res, stable and >>>>> precise, so I was wondering what the rationale is for not using these >>>>> APIs? >>>>> >>>>> In asyncio, would an API to modify the timer implementation be >>>>> applicable? >>>>> Just changing loop.time will not work, as you would also have to >>>>> change loop._clock_resolution, so now we're in the business of >>>>> monkey-patching eventloop and implementation-specific details. >>>>> Generally, having to add hacks for Windows is not very attractive when >>>>> e.g. Twisted works out of the box. >>>>> >>>>> About the original example I posted, it seems like the stability >>>>> problems only occur if the 100 FPS loop is added. >>>>> Perhaps the issue could be rectified if the eventloop was more clever >>>>> with its scheduling? >>>>> libuv also seems to use GetQueuedCompletionStatus timeouts to adjust >>>>> the timer, so I think there may be room for improvement with the timer in >>>>> the Windows eventloop. >>>>> I will do some more testing, and see if I can come up with a solution >>>>> without changing the timer implementation. >>>>> >>>>> Thanks, >>>>> Mathias >>>>> >>>>> Den lørdag den 17. maj 2014 09.52.54 UTC+2 skrev Victor Stinner: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I had very very long discussion about timer granularity and choose >>>>>> the "right" clock in Python and in asyncio. See for example the PEP 418 >>>>>> to >>>>>> learn why time.monotonic() has such very low precision. >>>>>> >>>>>> I don't want to change the default clock of asyncio or change >>>>>> time.monotonic(). As Guido wrote, it's easy to change the clock of an >>>>>> event >>>>>> loop. Try loop.time = myclock. >>>>>> >>>>>> Victor >>>>>> >>>>>> Le samedi 17 mai 2014, Mathias Kærlev <[email protected]> a écrit : >>>>>> >>>>>>> So after some quick tests, it's apparent that switching out >>>>>>> time.monotonic() with either time.clock() or time.time() completely >>>>>>> fixes >>>>>>> the issue (so the framerate is always between 49~50). >>>>>>> For my application, I can probably subclass the eventloop and switch >>>>>>> out the timer implementation, but ideally, this should be fixed >>>>>>> upstream >>>>>>> somehow. >>>>>>> I never had this issue with Twisted, even though I think Twisted >>>>>>> uses time.time() with some heuristics to detect time drift and similar >>>>>>> issues. >>>>>>> Personally, I've always found a combination of timeGetTime() and >>>>>>> timeBeginPeriod(1) to yield a very stable timer on Windows (especially >>>>>>> for >>>>>>> games), but perhaps you have greater insight into the issue. >>>>>>> >>>>>>> Thanks for the help! Is there any chance this will be fixed in an >>>>>>> upcoming asyncio release? >>>>>>> >>>>>>> Mathias >>>>>>> >>>>>>> Den lørdag den 17. maj 2014 06.31.18 UTC+2 skrev Guido van Rossum: >>>>>>>> >>>>>>>> This could well be timer granularity on Windows; this has been a >>>>>>>> big problem for the unittests too. Could you try with a different >>>>>>>> timer? It >>>>>>>> should be easy to change the timer function. >>>>>>>> >>>>>>>> On Friday, May 16, 2014, Mathias Kærlev <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi everyone >>>>>>>>> >>>>>>>>> I'm currently porting some code from Python 2.7 with Twisted to >>>>>>>>> Python 3.4 with asyncio. So far, it's been a pleasant experience. >>>>>>>>> However, my application in question is a game server which needs >>>>>>>>> to run a 50FPS update loop to simulate the game world. >>>>>>>>> When running multiple update loops, I start to get some very >>>>>>>>> unstable behavior, where the actual delay between a call_later call >>>>>>>>> and the >>>>>>>>> time it fires wobbles between the specified delay and half the delay. >>>>>>>>> >>>>>>>>> Please see the following example: http://bpaste.net/sho >>>>>>>>> w/Cxn0jxqufnvNk7qL8MnJ/ >>>>>>>>> In the example, I run two update loops: one on 50FPS, and one on >>>>>>>>> 100FPS. I only monitor the update loop running at 50 FPS. >>>>>>>>> Yet, on Windows 7, I get the following output: http://bpaste.net/ >>>>>>>>> show/XKp4LtKpEsXWxacZjQlc/ >>>>>>>>> The framerate is very wobbly, going between ~100 FPS and ~50 FPS >>>>>>>>> (the 100 FPS LoopingCall is not being logged, of course). >>>>>>>>> >>>>>>>>> Is this a bug in my LoopingCall class, or is it a problem with the >>>>>>>>> eventloop timer granularity? >>>>>>>>> >>>>>>>>> This is using the asyncio release found in stdlib on Python 3.4, >>>>>>>>> by the way. >>>>>>>>> >>>>>>>>> Thanks for the help! >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> --Guido van Rossum (on iPad) >>>>>>>> >>>>>>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> > > > -- > --Guido van Rossum (python.org/~guido) >
