Sounds good. Regarding ctypes: I'm not crazy about it -- since asyncio doesn't work on Windows without _overlapped anyway, I'd rather see timeGetTime() added there.
--Guido On Sat, May 17, 2014 at 9:04 PM, Mathias Kærlev <[email protected]> wrote: > Thanks for the feedback. > > After doing some research, it does seem to be a very Windows-specific > issue. > A general set_clock() API is probably better then. > > I will be looking into doing patches for set_clock() and a new time() > method on Windows (without period management, of course). > One quick question though: how do you feel about ctypes? I see that's it's > occasionally used in stdlib, but given that you went out of your way to > create the _overlapped module for asyncio, would it be acceptable to use > ctypes for timeGetTime? > > Mathias > > Den søndag den 18. maj 2014 04.25.52 UTC+2 skrev Guido van Rossum: > >> On Sat, May 17, 2014 at 6:36 PM, Mathias Kærlev <[email protected]> wrote: >> >>> I wouldn't want asyncio to default to calling timeBeginPeriod either, as >>> it would hurt battery life and potentially affect other applications, which >>> is not an ideal default behavior. >>> However, it would be great if there was an API to request a certain time >>> resolution from the eventloop. >>> >> >> This looks like it would be Windows specific functionality though. I'd >> prefer to do it somewhat differently. >> >> >>> This would be useful to e.g. my LoopingCall class, where I would be able >>> to find the highest call frequency and feed that to the eventloop. >>> Generally, if the application *knows* that it will need a >>> high-resolution timer, it would be able to request one. >>> >> >> Only on Windows, AFAIK. >> >> >>> I've changed the previous code to add new set_clock and >>> set_clock_resolution methods: >>> https://github.com/matpow2/cuwo/blob/py3/cuwo/win32.py >>> >>> To show what my original example would look like with the new methods: >>> http://bpaste.net/show/pcrIZAV3R7QaefJGMtRw/ >>> >>> If setting the clock resolution was to be added to asyncio, it would >>> obviously be a no-op on platforms where there are no timer resolution >>> issues. >>> In any case, I would definitely consider adding a set_clock method (so >>> if someone wants to use e.g. QPC, they can). >>> >> >> I propose to leave the resolution management to the application's >> Windows-specific code. You can write code that sets the resolution to what >> you want and then pass both it and your timer code to the event loop. >> >> One problem with using timeGetTime is that we don't know the initial >>> system-wide timer resolution. >>> We can be conservative and guess that it has not been changed from >>> startup, in which case we would get a resolution of 15.6ms. >>> However, a lot of applications (Chrome, GUI applications, Flash, games, >>> etc.) change the global timer resolution to 1ms, so in reality, it may >>> already be a lot more precise than we expect. >>> There is also the possibility of using a period which is higher than the >>> current system-wide resolution, in which case we would also be getting a >>> more precise timer. >>> From what I can see, there is no way to query the current resolution >>> without using NtQueryTimerResolution, which is an undocumented MS function. >>> >> >> All this just adds to my desire to make changing the system resolution >> the app's problem, and just provide set_clock(time_func, clock_resolution) >> as a loop method. (BTW, the resolution is only used in one place, and even >> that place is dubious -- it's mostly because Victor doesn't like to >> busy-wait if the sleep time is shorter than requested. Perhaps we can live >> without it after all.) >> >> >>> However, we can still look at timeGetTime deltas in the time() method, >>> and if we find a delta that is lower than the current clock resolution, we >>> use that as our resolution. >>> >> >> Again, I think that ought to be made the app's problem. >> >> >>> The last alternative would be to set _clock_resolution to the highest >>> possible resolution, i.e. 1ms, even if the actual resolution may be lower. >>> This way, we can ensure the scheduler never tries to fire a call which is >>> in the future. This is the approach I have taken in my code. >>> With some testing, it appears that my system in fact uses a timer >>> resolution of 1 ms most of the time (since a lot of applications set it by >>> default), so to me, this seems like the best approach. >>> >> >> There are very few side effects of setting _clock_resolution lower than >> the actual clock resolution. >> >> >>> At the very least, timeGetTime could be used as the default time() >>> method on Windows. >>> Unfortunately, timeBeginPeriod does not affect GetTickCount (i.e. >>> time.monotonic), as GetTickCount will try to emulate the 15.6ms resolution >>> even if the system-wide resolution has been changed. >>> >> >> Hm. If you want to argue over the implementation of time.monotonic(), I'd >> prefer it if you did that on python-dev, as it is not specific to asyncio. >> But it's another reason why I want the period management out of asyncio. :-) >> >> >>> Hopefully my input has been valuable to some extent! >>> >> >> I suggest that you file a tracker issue and propose a patch just to >> change the default time() function on Windows. And perhaps a separate one >> to lobby for a set_clock() API. >> >> >>> It would be great if asyncio becomes the go-to eventloop for all Python >>> things. >>> >> >> Yes, that's my hope too! >> >> >>> This is probably why I'm very keen on having a good default >>> implementation. For all platforms. ;-) >>> >>> Cheers, >>> Mathias >>> >>> Den lørdag den 17. maj 2014 23.29.30 UTC+2 skrev Guido van Rossum: >>>> >>>> Sorry, saw this after sending my reply. >>>> >>>> Looking at your code and the MS docs for timeBeginPeriod() I can see >>>> why we didn't choose this as our default asyncio clock or as the >>>> implementation of time.monotonic() -- the docs warn that the requested >>>> period can affect the performance of the entire system. Your implementation >>>> also selfishly sets exactly the period that you are interested in. >>>> >>>> But it seems that using timeBeginPeriod() is pretty independent from >>>> timeGetTime() -- perhaps it even affects time.monotonic()? If not, perhaps >>>> we can leave calling timeBeginPeriod() up to the app, but still override >>>> the time() method? We would then have to decide whether we want to just >>>> override time() and _clock_resolution in the Windows-specific Loop classes >>>> (_WindowsSelectorEventLoop and ProactorEventLoop) or whether we want to >>>> define an API to change the time implementation more generally. >>>> >>>> >>>> On Sat, May 17, 2014 at 6:54 AM, Mathias Kærlev <[email protected]>wrote: >>>> >>>>> I experimented with offsetting the timer using the timeout passed to >>>>> the selector, and that did seem to fix it for my case. >>>>> However, I found it difficult to see if that would be a general >>>>> solution to the problem, especially when the selector would actually >>>>> return >>>>> something on each iteration (in which case we couldn't use the timeout). >>>>> >>>>> I ended up just subclassing the eventloops from windows_events.py: >>>>> https://github.com/matpow2/cuwo/blob/py3/cuwo/win32.py >>>>> >>>>> It uses timeGetTime and timeGetPeriod/timeEndPeriod with a timer >>>>> resolution of 10ms, which should be enough for the majority of cases. >>>>> According to the documentation, timeGetTime should work exactly like >>>>> GetTickCount, except for the fact that you can change the timer >>>>> resolution. >>>>> >>>>> I still consider this to be a hack though. It would be nice if >>>>> something was done about this eventually. >>>>> >>>>> Thanks again, >>>>> Mathias >>>>> >>>>> Den lørdag den 17. maj 2014 11.35.08 UTC+2 skrev Mathias Kærlev: >>>>> >>>>>> That seems like a cop-out to me. >>>>>> >>>>>> I get that GetTickCount64 does not have high precision, but you could >>>>>> consider using a timeGetTime/GetTickCount64 hybrid (like in GLib) for >>>>>> asyncio. >>>>>> Otherwise, with these scheduling issues, the default eventloop >>>>>> implementation is practically unusable for games on Windows, which I >>>>>> think >>>>>> is a shame. >>>>>> Obviously, I agree that a precise, stable (i.e. not QPC) monotonic >>>>>> clock would be the only proper timer solution, but ultimately, it should >>>>>> not come at the expense of usability. >>>>>> timeGetTime/timeBeginPeriod appears to be both high-res, stable and >>>>>> precise, so I was wondering what the rationale is for not using these >>>>>> APIs? >>>>>> >>>>>> In asyncio, would an API to modify the timer implementation be >>>>>> applicable? >>>>>> Just changing loop.time will not work, as you would also have to >>>>>> change loop._clock_resolution, so now we're in the business of >>>>>> monkey-patching eventloop and implementation-specific details. >>>>>> Generally, having to add hacks for Windows is not very attractive >>>>>> when e.g. Twisted works out of the box. >>>>>> >>>>>> About the original example I posted, it seems like the stability >>>>>> problems only occur if the 100 FPS loop is added. >>>>>> Perhaps the issue could be rectified if the eventloop was more clever >>>>>> with its scheduling? >>>>>> libuv also seems to use GetQueuedCompletionStatus timeouts to adjust >>>>>> the timer, so I think there may be room for improvement with the timer in >>>>>> the Windows eventloop. >>>>>> I will do some more testing, and see if I can come up with a solution >>>>>> without changing the timer implementation. >>>>>> >>>>>> Thanks, >>>>>> Mathias >>>>>> >>>>>> Den lørdag den 17. maj 2014 09.52.54 UTC+2 skrev Victor Stinner: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I had very very long discussion about timer granularity and choose >>>>>>> the "right" clock in Python and in asyncio. See for example the PEP 418 >>>>>>> to >>>>>>> learn why time.monotonic() has such very low precision. >>>>>>> >>>>>>> I don't want to change the default clock of asyncio or change >>>>>>> time.monotonic(). As Guido wrote, it's easy to change the clock of an >>>>>>> event >>>>>>> loop. Try loop.time = myclock. >>>>>>> >>>>>>> Victor >>>>>>> >>>>>>> Le samedi 17 mai 2014, Mathias Kærlev <[email protected]> a écrit : >>>>>>> >>>>>>>> So after some quick tests, it's apparent that switching out >>>>>>>> time.monotonic() with either time.clock() or time.time() completely >>>>>>>> fixes >>>>>>>> the issue (so the framerate is always between 49~50). >>>>>>>> For my application, I can probably subclass the eventloop and >>>>>>>> switch out the timer implementation, but ideally, this should be fixed >>>>>>>> upstream somehow. >>>>>>>> I never had this issue with Twisted, even though I think Twisted >>>>>>>> uses time.time() with some heuristics to detect time drift and similar >>>>>>>> issues. >>>>>>>> Personally, I've always found a combination of timeGetTime() and >>>>>>>> timeBeginPeriod(1) to yield a very stable timer on Windows (especially >>>>>>>> for >>>>>>>> games), but perhaps you have greater insight into the issue. >>>>>>>> >>>>>>>> Thanks for the help! Is there any chance this will be fixed in an >>>>>>>> upcoming asyncio release? >>>>>>>> >>>>>>>> Mathias >>>>>>>> >>>>>>>> Den lørdag den 17. maj 2014 06.31.18 UTC+2 skrev Guido van Rossum: >>>>>>>>> >>>>>>>>> This could well be timer granularity on Windows; this has been a >>>>>>>>> big problem for the unittests too. Could you try with a different >>>>>>>>> timer? It >>>>>>>>> should be easy to change the timer function. >>>>>>>>> >>>>>>>>> On Friday, May 16, 2014, Mathias Kærlev <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi everyone >>>>>>>>>> >>>>>>>>>> I'm currently porting some code from Python 2.7 with Twisted to >>>>>>>>>> Python 3.4 with asyncio. So far, it's been a pleasant experience. >>>>>>>>>> However, my application in question is a game server which needs >>>>>>>>>> to run a 50FPS update loop to simulate the game world. >>>>>>>>>> When running multiple update loops, I start to get some very >>>>>>>>>> unstable behavior, where the actual delay between a call_later call >>>>>>>>>> and the >>>>>>>>>> time it fires wobbles between the specified delay and half the delay. >>>>>>>>>> >>>>>>>>>> Please see the following example: http://bpaste.net/sho >>>>>>>>>> w/Cxn0jxqufnvNk7qL8MnJ/ >>>>>>>>>> In the example, I run two update loops: one on 50FPS, and one on >>>>>>>>>> 100FPS. I only monitor the update loop running at 50 FPS. >>>>>>>>>> Yet, on Windows 7, I get the following output: http://bpaste.net/ >>>>>>>>>> show/XKp4LtKpEsXWxacZjQlc/ >>>>>>>>>> The framerate is very wobbly, going between ~100 FPS and ~50 FPS >>>>>>>>>> (the 100 FPS LoopingCall is not being logged, of course). >>>>>>>>>> >>>>>>>>>> Is this a bug in my LoopingCall class, or is it a problem with >>>>>>>>>> the eventloop timer granularity? >>>>>>>>>> >>>>>>>>>> This is using the asyncio release found in stdlib on Python 3.4, >>>>>>>>>> by the way. >>>>>>>>>> >>>>>>>>>> Thanks for the help! >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> --Guido van Rossum (on iPad) >>>>>>>>> >>>>>>>> >>>> >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >>>> >>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido)
