Re: [weewx-development] Re: [weewx-user] 4 beta test report
Thomas Keffer writes: > OK, so we've isolated the issue to a problem with how NetBSD implements > time.mktime(). I don't think it's a problem in the implementation, just not matching the weewx code's beyond-standards expectation. As I understand it, an error return for times that do not exist in the local timezone is the standard approach for mktime, and guessing what it might mean, while obviously useful, is an extension. (If it is a NetBSD bug, I'll file a ticket and see about fixing it; we are as a group pretty hard-core about standards compliance.) I just looked at the sources, and there is a default-off option NO_ERROR_IN_DST_GAP. Reading the logs, it seems people think the error behavior in the gap is correct. But even if is a bug, it would be nice to avoid it. > I suppose we could trap the exception in intervalgen() and, instead, return > the simple arithmetic result of adding increment. Something like: > > delta = datetime.timedelta(seconds=interval) > last_stamp1 = 0 > while dt1 < stop_dt: > dt2 = min(dt1 + delta, stop_dt) > stamp1 = int(time.mktime(dt1.timetuple())) > try: > stamp2 = int(time.mktime(dt2.timetuple())) > except OverflowError: > stamp2 = stamp1 + interval > if stamp2 > stamp1 > last_stamp1: > yield TimeSpan(stamp1, stamp2) > last_stamp1 = stamp1 > dt1 = dt2 We could, and I think that would work fine. But, taking a step back, I think the code is ending up running in a situation where it isn't helpful, and in the case mktime doens't raise an exception, returning not what's expected. First, I get it that with dt1 being some date and , and interval 24h, we want the posix time_t values corresponding to local midnight, even if a day is 23h or 25h. And simlarly when interval is 3h, 6h, or 12h. (I have seen the graphs that result in plotting programs that aren't aware of this, and they are difficult to read and ugly!) But when interval is 1h, I think we want to use and display the local tz naming of times that are actually 1h apart. So for the day in question, that's 0100 0300 0400 and we want those to appear equi-spaced on the graph. With the try/except above, that I think will happen on systems where mktime rejects 0200. With systems where mktime of 0200 returns the time_t for 0300 (which is 3600 after the time_t for 0100), I think it will result in 0100 0300 0300 0400 So I wonder about another if block that checks for interval == 3600 (or maybe <= 3600?) and just does straightforward time_t operations. For these intervals, a one hour change will not cause misrounding. I suppose there is the issue of DST shifts that are less than 1h, but my impression is that DST is always 1h or 2h, even if TZ offsets are :30 or :15. -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/rmi8sk8m9ae.fsf%40s1.lexort.com.
Re: [weewx-development] Re: [weewx-user] 4 beta test report
OK, so we've isolated the issue to a problem with how NetBSD implements
time.mktime().
I suppose we could trap the exception in intervalgen() and, instead, return
the simple arithmetic result of adding increment. Something like:
delta = datetime.timedelta(seconds=interval)
last_stamp1 = 0
while dt1 < stop_dt:
dt2 = min(dt1 + delta, stop_dt)
stamp1 = int(time.mktime(dt1.timetuple()))
try:
stamp2 = int(time.mktime(dt2.timetuple()))
except OverflowError:
stamp2 = stamp1 + interval
if stamp2 > stamp1 > last_stamp1:
yield TimeSpan(stamp1, stamp2)
last_stamp1 = stamp1
dt1 = dt2
On Mon, Mar 9, 2020 at 5:35 PM Greg Troxel wrote:
> I get
>
> Starting datetime is 2020-03-08 01:00:00
> Represented as a time tuple, this is time.struct_time(tm_year=2020,
> tm_mon=3, tm_mday=8, tm_hour=1, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=68,
> tm_isdst=-1)
> After adding an hour, the resultant datetime is 2020-03-08 02:00:00
> Represented as a timetuple this is time.struct_time(tm_year=2020,
> tm_mon=3, tm_mday=8, tm_hour=2, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=68,
> tm_isdst=-1)
> Traceback (most recent call last):
> File "./test.py", line 22, in
> ts = time.mktime(tt1)
> OverflowError: mktime argument out of range
>
>
> I wrote a test program for mktime. With the isdst/gmtoff values from
> localtime, it works. With this, I can provoke EINVAL, which I think is
> per the spec.
>
>
> #include
> #include
> #include
>
> int
> main()
> {
> struct tm *t;
> time_t start = 1583647200;/* 20200308T0100 EST */
> time_t plus1;
>
> t = localtime();
>
> printf("isdst %d tm_gmtoff %ld\n", t->tm_isdst, t->tm_gmtoff);
> t->tm_isdst = -1;
> t->tm_gmtoff = 0;
> printf("isdst %d tm_gmtoff %ld\n", t->tm_isdst, t->tm_gmtoff);
>
> t->tm_hour += 1;
> printf("tm_hour %d\n", t->tm_hour);
>
> plus1 = mktime(t);
>
> printf("plus1 %jd errno %d\n", (intmax_t) plus1, errno);
>
> printf("result: %s\n", plus1 == 1583650800 ? "ok" : "bad");
>
>
> return 0;
> }
>
> isdst 0 tm_gmtoff -18000
> isdst -1 tm_gmtoff 0
> tm_hour 2
> plus1 -1 errno 22
> result: bad
>
>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-development/CAPq0zEB8H%2BNw_u99vN3D00pFn4HWkp1t3B3eChauPA5v8ba_-w%40mail.gmail.com.
[weewx-development] Re: [weewx-user] 4 beta test report
Greg Troxel writes: (Earlier, on weewx-user, I posted that I was seeing a backtrace from v4beta (recent master). I'm following up here now that I've read the code and have something perhaps useful to say.) > Traceback (most recent call last): > File "/usr/weewx/bin/weewx/reportengine.py", line 197, in run > obj.start() > File "/usr/weewx/bin/weewx/reportengine.py", line 280, in start > self.run() > File "/usr/weewx/bin/weewx/imagegenerator.py", line 41, in run > self.genImages(self.gen_ts) > File "/usr/weewx/bin/weewx/imagegenerator.py", line 180, in genImages > aggregate_interval=aggregate_interval) > File "/usr/weewx/bin/weewx/xtypes.py", line 86, in get_series > aggregate_interval) > File "/usr/weewx/bin/weewx/xtypes.py", line 138, in get_series > for stamp in weeutil.weeutil.intervalgen(startstamp, stopstamp, > aggregate_interval): > File "/usr/weewx/bin/weeutil/weeutil.py", line 339, in intervalgen > stamp2 = int(time.mktime(dt2.timetuple())) > OverflowError: mktime argument out of range I am using python 2.7, NetBSD 8 on earmv7hf-el. weewx 3.x has run for over 2 years with essentially no problems. Ihave added printfs to weeutil.py, in intervalgen. The crash happens during the loop in else block. The last dt2 printout is dt2 2020-03-08 02:00:00 which is a time that does not exist (here in EST5EDT). So apparently the python time doing the addition in the datetime object is not going back to timeval each time. I am using the older skin that was standard before the Seasons skin became normal. Is anyone else seeing this? You can tell you are having this problem two ways: traceback in log during html report generation graphs on week/month/year page are stale I looked at python datetime docs: https://docs.python.org/3/library/datetime.html https://docs.python.org/2/library/datetime.html There is a notion of "aware" vs "naive" datetime objects, and as I read it the code is creating naive objects, which don't have an associated timezone. So adding 3600s to 2020-03-08T01:00 leads to 02:00, even though I think that time doesn't exist and the right answer is 03:00. time.mktime() says it will raise an error if the time cannot be represented: https://docs.python.org/2/library/time.html I think to fix, we need a try: block that just does mktime on dt2 and discards the result, with an catch block that does continue, so we increment dt2 again. Or, to pass the local tz in the constructor, perhaps. I suppose it is possible that the underlying mktime library function on NetBSD is stricter than on Other Systems. The man page says: The function returns the specified calendar time; if the calendar time cannot be represented, it returns (time_t)-1. This can happen either because the resulting conversion would not fit in a time_t variable, or because the time specified happens to be in the daylight savings gap and tm_isdst was set to -1. Other mktime() implementations do not return an error in the second case and return the appropriate time offset after the daylight savings gap. There is code to mimick this behavior, but it is not enabled by default. and also makes a claim of conforming to C89. I have previously observed test failures in bup (a backup program using git data storage, written in python), and reproduced problems in git. I didn't ever chase that down, but I am now suspecting the same underlying cause, namely invocations of mktime which are incorrect according to standards, but happen to work anyway on Linux. (That is of course guess, not a substantiated conclusion.) -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/rmipndlv950.fsf%40s1.lexort.com.
