[issue34819] Executor.map and as_completed timeouts are able to deviate

[miss-islington]

miss-islington  added the comment:


New changeset 2b01121fd4200f1c27873422f7f72d02eec08630 by Miss Islington (bot) 
in branch '3.7':
bpo-34819: Use a monotonic clock to compute timeouts in concurrent.futures 
(GH-9599)
https://github.com/python/cpython/commit/2b01121fd4200f1c27873422f7f72d02eec08630


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[Antoine Pitrou]

Change by Antoine Pitrou :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[orlnub123]

orlnub123  added the comment:

Happy to help! I'm surprised it got merged so quickly, amazing response times 
all-around.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[miss-islington]

miss-islington  added the comment:


New changeset 3a4aa6ac55e04c42757443d5b5854b6d893e0461 by Miss Islington (bot) 
in branch '3.6':
bpo-34819: Use a monotonic clock to compute timeouts in concurrent.futures 
(GH-9599)
https://github.com/python/cpython/commit/3a4aa6ac55e04c42757443d5b5854b6d893e0461


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[STINNER Victor]

STINNER Victor  added the comment:

Oh, I missed the Lib/concurrent/futures/ directory when I tried to patch the 
whole stdlib to use monotonic clocks for timeouts :-( Thanks for the fix!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[miss-islington]

Change by miss-islington :


--
pull_requests: +9000

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[miss-islington]

Change by miss-islington :


--
pull_requests: +9001

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset a94ee12c26aa8dd7dce01373779df8055aff765b by Antoine Pitrou 
(orlnub123) in branch 'master':
bpo-34819: Use a monotonic clock to compute timeouts in concurrent.futures 
(GH-9599)
https://github.com/python/cpython/commit/a94ee12c26aa8dd7dce01373779df8055aff765b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Sorry, please ignore the comment reference to 
https://bugs.python.org/issue29733#msg289116 . I misunderstood that it was 
about concurrent.futures using time.time but it turns out it was about the 
program using time.time as verifying the PR against the issue still has the 
timeout error.

Thanks

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Thanks for the report and PR. I can find a very similar issue where time.time 
was changed to time.monotonic across multiprocessing module by Victor for the 
same reason : issue34054. I don't know why it wasn't changed in 
concurrent.futures . Seems this was reported here : 
https://bugs.python.org/issue29733#msg289116. I am adding Victor and Antoine 
who might have thoughts on this. Feel free to remove yourself if it's not 
relevant.

Thanks

--
nosy: +pitrou, vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[orlnub123]

Change by orlnub123 :


--
keywords: +patch
pull_requests: +8996
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34819] Executor.map and as_completed timeouts are able to deviate

[orlnub123]

New submission from orlnub123 :

The map and as_completed functions can randomly timeout earlier or later due to 
NTP stepping the clock or something changing the time and/or date.

Here's some code to reproduce the issue for map:

import concurrent.futures
import time

with concurrent.futures.ThreadPoolExecutor() as executor:
list(executor.map(time.sleep, [29, 30], timeout=3000))

And similar code to reproduce it for as_completed:

import concurrent.futures
import time

with concurrent.futures.ThreadPoolExecutor() as executor:
future1 = executor.submit(time.sleep, 29)
future2 = executor.submit(time.sleep, 30)
list(concurrent.futures.as_completed([future1, future2], timeout=3000))

It doesn't error normally, as it shouldn't, but if you move your clock an hour 
ahead within 30 seconds of running it you get this:

Traceback (most recent call last):
  File "", line 2, in 
  File "C:\Python\Python37\lib\concurrent\futures\_base.py", line 588, in 
result_iterator
yield fs.pop().result(end_time - time.time())
  File "C:\Python\Python37\lib\concurrent\futures\_base.py", line 434, in result
raise TimeoutError()
concurrent.futures._base.TimeoutError

Or this if you used the as_completed example:

Traceback (most recent call last):
  File "", line 4, in 
  File "C:\Python\Python37\lib\concurrent\futures\_base.py", line 238, in 
as_completed
len(pending), total_futures))
concurrent.futures._base.TimeoutError: 1 (of 2) futures unfinished

The error stems from map and as_completed using time.time to calculate the 
duration for the timeout. The problem with time.time is that it's adjustable 
(i.e. it can skip around) which makes it unideal for comparing relative times. 
A solution would be to replace the time.time calls with either time.monotonic 
(preferable?) or time.perf_counter.

--
components: Library (Lib)
messages: 326535
nosy: orlnub123
priority: normal
severity: normal
status: open
title: Executor.map and as_completed timeouts are able to deviate
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com