[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Changes by Alexandre Conrad alexandre.con...@gmail.com: -- nosy: +aconrad ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Tim Peters added the comment: It is really bad that roundtripping current microsecond datetimes doesn't work. About half of all microsecond-resolution datetimes fail to roundtrip correctly now. While the limited precision of a C double guarantees roundtripping of microsecond datetimes far enough in the future will necessarily fail, that point is about 200 years from now. Rather than argue endlessly about rounding, it's possible instead to make the tiniest possible change to the timestamp _produced_ at the start. Here's code explaining it: ts = d.timestamp() # Will microseconds roundtrip correctly? For times far # enough in the future, there aren't enough bits in a C # double for that to always work. But for years through # about 2241, there are enough bits. How does it fail # before then? Very few microsecond datetimes are exactly # representable as a binary float. About half the time, the # closest representable binary float is a tiny bit less than # the decimal value, and that causes truncating 1e6 times # the fraction to be 1 less than the original microsecond # value. if int((ts - int(ts)) * 1e6) != d.microsecond: # Roundtripping fails. Add 1 ulp to the timestamp (the # tiniest possible change) and see whether that repairs # it. It's enough of a change until doubles just plain # run out of enough bits. mant, exp = math.frexp(ts) ulp = math.ldexp(0.5, exp - 52) ts2 = ts + ulp if int((ts2 - int(ts2)) * 1e6) == d.microsecond: ts = ts2 else: # The date is so late in time that a C double's 53 # bits of precision aren't sufficient to represent # microseconds faithfully. Leave the original # timestamp alone. pass # Now ts exactly reproduces the original datetime, # if that's at all possible. This assumes timestamps are = 0, and that C doubles have 53 bits of precision. Note that because a change of 1 ulp is the smallest possible change for a C double, this cannot make closest-possible unequal datetimes produce out-of-order after-adjustment timestamps. And, yes, this sucks ;-) But it's far better than having half of timestamps fail to convert back for the next two centuries. Alas, it does nothing to get the intended datetime from a microsecond-resolution timestamp produced _outside_ of Python. That requires rounding timestamps on input - which would be a better approach. Whatever theoretical problems may exist with rounding, the change to use truncation here is causing real problems now. Practicality beats purity. -- nosy: +tim.peters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Changes by Ethan Furman et...@stoneleaf.us: -- nosy: -ethan.furman ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
STINNER Victor added the comment: My rationale is more general than datetime. But problems araise when different API use different rounding methods. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
STINNER Victor added the comment: Le vendredi 3 juillet 2015, Alexander Belopolsky rep...@bugs.python.org a écrit : UNIX doesn't like timestamps in the future I don't think this is a serious consideration. The problematic scenario would be obtaining high-resolution timestamp (from say time.time()), converting it to datetime and passing it back to OS as a possibly 0.5µs higher value. Given that timestamp - datetime - timestamp roundtrip by itself takes over 1µs, it is very unlikely that by the time rounded value hits the OS it is still in the future. In many cases the resolution is 1 second. For example, a filesystem with a resolution of 1second. Or an API only supporting a resolution of 1 second. With a resoltuion of 1 second, timestamps in the future are likely (50%). Sorry I don't remember all detail of timestamp rounding and all issues that I saw. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Alexander Belopolsky added the comment: I'll let others fight this battle. In my view, introducing floating point timestamp method for datetime objects was a mistake. See issue #2736. Specifically, I would like to invite Velko Ivanov to rethink his rant at msg124197. If anyone followed his advise and started using timestamp method to JSON-serialize datetimes around 3.3, have undoubtedly being bitten by the present bug (but may not know it yet.) For those who need robust code, I will continue recommending (dt - EPOCH)/timedelta(seconds=1) expression over the timestamp method and for JSON serialization (dt - EPOCH) // datetime.resolution to convert to integers and EPOCH + n * datetime.resolution to convert back. -- nosy: +vivanov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Changes by Alexander Belopolsky alexander.belopol...@gmail.com: -- nosy: -belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Larry Hastings added the comment: I'm not going to hold up beta 3 while you guys argue about how to round up or down the number of angels that can dance on the head of a pin. -- priority: release blocker - deferred blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
R. David Murray added the comment: Because this seems to be a regression, I'm marking this as a release blocker. The RM can decide is isn't, of course. -- nosy: +larry priority: normal - release blocker versions: +Python 3.5, Python 3.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
STINNER Victor added the comment:
I'm concerned by this example:
dt = datetime(2015, 2, 24, 22, 34, 28, 274000)
dt - datetime.fromtimestamp(dt.timestamp())
datetime.timedelta(0, 0, 1)
I don't know yet if it should be fixed or not.
If we modify .fromtimestamp(), should we use the same rounding method in
datetime constructor? And in datetime.now()/.utcnow()?
I would prefer to keep ROUND_DOWN for .now() and .utcnow() to avoid timestamps
in the future. I care less for other methods.
What do you think of this plan?
---
Hum, I don't remember the whole story line of rounding timestamps in Python.
Some raw data.
Include/pytime.h of Python 3.5+ has:
typedef enum {
/* Round towards minus infinity (-inf).
For example, used to read a clock. */
_PyTime_ROUND_FLOOR=0,
/* Round towards infinity (+inf).
For example, used for timeout to wait at least N seconds. */
_PyTime_ROUND_CEILING
} _PyTime_round_t;
Include/pytime.h of Python 3.4 had:
typedef enum {
/* Round towards zero. */
_PyTime_ROUND_DOWN=0,
/* Round away from zero. */
_PyTime_ROUND_UP
} _PyTime_round_t;
Include/pytime.h of Python 3.3 and older didn't have rounding.
C files using pytime.h rounding in Python 3.4 (grep -l _PyTime_ROUND */*.c):
Modules/_datetimemodule.c
Modules/posixmodule.c
Modules/selectmodule.c
Modules/signalmodule.c
Modules/_testcapimodule.c
Modules/timemodule.c
Python/pytime.c
It is used by 3 mores C files in Python 3.5:
Modules/socketmodule.c
Modules/_ssl.c
Modules/_threadmodule.c
NEAREST was never implemented in pytime.h.
If I recall correctly, there were inconsitencies between the Python and the C
implementation of the datetime module. At least in Python 3.5, both
implementations should be consistent (even if some people would prefer a
different rounding method).
The private pytime API was rewritten in Python 3.5 to get nanosecond
resolution. This API is only used by the datetime module to get the current
time.
My rationale for ROUND_DOWN was to follow how UNIX rounds timestmaps. As
Alexander wrote, UNIX doesn't like timestamps in the future, so rounding
towards minus infinity avoids such issue. Rounding issues become more common on
file timestamps with filesystems supporting microsecond resolution or event
nanosecond resolution.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23517
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Alexander Belopolsky added the comment: Victor I don't know yet if it should be fixed or not. It is my understanding that datetime - timestamp - datetime round-tripping was exact in 3.3 for datetimes not too far in the future (as of 2015), but now it breaks for datetime(2015, 2, 24, 22, 34, 28, 274000). This is clearly a regression and should be fixed. UNIX doesn't like timestamps in the future I don't think this is a serious consideration. The problematic scenario would be obtaining high-resolution timestamp (from say time.time()), converting it to datetime and passing it back to OS as a possibly 0.5µs higher value. Given that timestamp - datetime - timestamp roundtrip by itself takes over 1µs, it is very unlikely that by the time rounded value hits the OS it is still in the future. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Larry Hastings added the comment: Yes, by all means, fix for 3.4, 3.5, and 3.6. If possible I'd appreciate you getting the fix checked in to 3.5 within the next 48 hours, as I'm tagging the next beta release of 3.5 around then, and it'd be nice if this fix went out in that release. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Timothy Cardenas added the comment: We are seeing this behavior influencing other libraries in python 3.4. This should never fail if timestamp and fromtimestamp are implemented correctly: from datetime import datetime t = datetime.utcnow().timestamp() t2 = datetime.utcfromtimestamp(t) assert t == t2, 'Moving from timestamp and back should always work' -- nosy: +trcarden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
R. David Murray added the comment: Most likely this was a rounding fix (ie: not a bug), but hopefully Alexander will know for sure. -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Changes by Alexander Belopolsky alexander.belopol...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Alexander Belopolsky added the comment: Victor's motivation for the change was (msg154811): I chose this rounding method because it is the method used by int(float) and int(time.time()) is a common in programs (more than round(time.time()). Rounding towards zero avoids also producing timestamps in the future. I recall the earlier discussions of rounding in the datetime module and Mark's explanation that rounding up is fine as long as ordering is preserved. i.e. for x y round(x) = round(y). There are cases when producing times in the future are problematic, for example UNIX make really dislikes when file timestamps are in the future, but if this was the main motivation - rounding towards -infinity would be more appropriate. In any case, as long as we have the following in the datetime module documentation, I think this behavior is a bug: On the POSIX compliant platforms, utcfromtimestamp(timestamp) is equivalent to the following expression: datetime(1970, 1, 1) + timedelta(seconds=timestamp) timestamp = 1424817268.274 datetime.utcfromtimestamp(timestamp) == datetime(1970, 1, 1) + timedelta(seconds=timestamp) False -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Alexander Belopolsky added the comment:
Let me dig up the history, but this does not look like correct rounding to me:
datetime.utcfromtimestamp(1424817268.274)
datetime.datetime(2015, 2, 24, 22, 34, 28, 273999)
decimal.Decimal(1424817268.274)
Decimal('1424817268.273294281005859375')
--
nosy: +mark.dickinson
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23517
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Alexander Belopolsky added the comment: It looks like it was an intentional change. See #14180 (changeset 75590:1e9cc1a03365). I am not sure what the motivation was. Note that this change made utcfromtimestamp(t) different from datetime(1970,1,1) + timedelta(seconds=t). -- keywords: +3.2regression ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Alexander Belopolsky added the comment: I noticed that the rounding mode of datetime is currently wrong. What do you mean by currently? What versions of python have it wrong? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
STINNER Victor added the comment:
I started a large change set to support nanoseconds in the C pytime API: see
the issue #22117. While working on this change, I noticed that the rounding
mode of datetime is currently wrong. Extract of a private patch:
typedef enum {
/* Round towards zero. */
_PyTime_ROUND_DOWN=0,
/* Round away from zero.
For example, used for timeout to wait at least N seconds. */
_PyTime_ROUND_UP=1,
/* Round towards minus infinity (-inf).
For example, used for the system clock with UNIX epoch (time_t). */
_PyTime_ROUND_FLOOR=2
} _PyTime_round_t;
I changed Modules/_datetimemodule.c to use _PyTime_ROUND_FLOOR, instead of
_PyTime_ROUND_DOWN.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23517
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Changes by Alexander Belopolsky alexander.belopol...@gmail.com: -- keywords: +3.3regression -3.2regression ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Alexander Belopolsky added the comment: For example, in my local patch, I'm using ROUND_FLOOR for: - datetime.date.fromtimestamp() - datetime.datetime.fromtimestamp() These should use ROUND_HALF_EVEN - datetime.datetime.now() - datetime.datetime.utcnow() These should not involve floating point arithmetics, but when converting from nanoseconds to microseconds, you should round to nearest 1000 ns with 500 ns ties resolved to even number of microseconds. - os.utime() This takes nanoseconds as an optional argument. Passing floats in times should probably be deprecated. In any case, here you would be rounding floats to nanoseconds and what you do with 0.5 nanoseconds is less important because in most cases they are not even representable as floats. - time.clock_settime() Is this a new method? I don't see it in 3.5.0a1. - time.gmtime() This should be fixed time.gmtime(1.9).tm_sec 1 is really bad and time.gmtime(-1.9)[:6] (1969, 12, 31, 23, 59, 59) is probably even worse. - time.localtime() - time.ctime() Same story as in time.gmtime. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
STINNER Victor added the comment: What do you mean by currently? What versions of python have it wrong? I search for ROUND in Modules/_datetimemodule.c: in the Python development branch (default), I found _PyTime_ROUND_DOWN (Round towards zero). Since a bug was reported, I understand that it's not the good rounding method? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Alexander Belopolsky added the comment: I don't understand nearest. Sorry for using loose terms. I was hoping the in the context of going back, it would be clear. I believe the correct mode is ROUND_HALF_EVEN. This is the mode used by the builtin round() function: round(0.5) 0 round(1.5) 2 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Alexander Belopolsky added the comment: Victor, Would you consider going back to round to nearest? Mark and I put in a lot of effort to get the rounding in the datetime module right. (See for example, #8860.) Sub-microsecond timesources are still rare and users who work with such should avoid FP timestamps in any case. On the other hand, double precision timestamps are adequate for microsecond resolution now and in the next few decades. Timestamps like OP's (sec=1424817268, us=274000) should not change when converted to double and back. IMO, the following behavior is a bug. dt = datetime(2015, 2, 24, 22, 34, 28, 274000) datetime.utcfromtimestamp(dt.timestamp()) datetime.datetime(2015, 2, 25, 3, 34, 28, 273999) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
STINNER Victor added the comment: Would you consider going back to round to nearest? I don't understand nearest. I prefer to use names of decimal rounding modes: https://docs.python.org/dev/library/decimal.html#rounding-modes In my local patch, I'm using ROUND_FLOOR in _decimal: Round towards -Infinity. Mark and I put in a lot of effort to get the rounding in the datetime module right. (See for example, #8860.) I'm unable right now to say which rounding mode should be used in the decimal module. But it's important to use the same rounding mode for all similar operations. For example, time.time() and datetime.datetime.now() should have the same rounding method (bad example, time.time() returns a float, which doesn't round the result). For example, in my local patch, I'm using ROUND_FLOOR for: - datetime.date.fromtimestamp() - datetime.datetime.fromtimestamp() - datetime.datetime.now() - datetime.datetime.utcnow() - os.utime() - time.clock_settime() - time.gmtime() - time.localtime() - time.ctime() Note: the Python implementation of datetime uses time.localtime() and time.gmtime() for fromtimestamp(), so these functions should also have the same rounding method. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
New submission from Tommaso Barbugli: Hi, I am porting a library from python 2.7 to 3.4 and I noticed that the behaviour of datetime.utcfromtimestamp is not consistent between the two versions. For example on python 2.7.5 datetime.utcfromtimestamp(1424817268.274) returns a datetime with 274000 microseconds the same code in python 3.4 returns a datetime with 273999 microseconds. -- components: Library (Lib) messages: 236552 nosy: tbarbugli priority: normal severity: normal status: open title: datetime.utcfromtimestamp parses timestamps incorrectly type: behavior versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23517] datetime.utcfromtimestamp parses timestamps incorrectly
Ethan Furman added the comment: This seems to have changed in 3.3 (versions up to 3.2 return 274000). -- nosy: +ethan.furman ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23517 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
