FYI, it does appear to be that the database locking caused this issue. Increasing the timeout fixed the issue for me.
On Wednesday, July 29, 2020 at 8:53:34 PM UTC-4 [email protected] wrote: > Weewx 4.1.1, Python 3, using weewx-sdr on Raspberry PI 3 w/ sqlite > database, 1min archive times, outdoor thermometer is an Acurite 5n1. > > We've had quite a bit of rain recently, so it's given me several > opportunities to notice that the Acurite app rainfall numbers vary greatly > with Weewx. Given that they are using the same data gathered over the air, > I tried to dig into what might be causing it. > > I too today's rainfall and gathered info out of the weewx logs to > determine how much rain we got - that jives with what is in the Acurite > App. Here is the first and last (stopped measuring at 5:47) entries. My > math tells me we have had .66 in between midnight and 5:47. > > Jul 29 00:00:34 localhost weewxd: sdr: MainThread: packet={u'windDir': > 22.5, u'windSpeed': 0.0, u'rain_total': 13.8, 'usUnits': 1, 'dateTime': > 1595995230} > ... > Jul 29 17:47:06 localhost weewx[8995] DEBUG user.sdr: packet={'windDir': > 247.5, 'windSpeed': 5.139, 'rain_total': 14.46, 'dateTime': 1596059222, > 'usUnits': 1} > > > I dumped the rainfall records out of the sqlite db to see what it > gathered. Out of 890 rows for this time period, 49 of them are empty (for > instance, the record at 5:38). So I add up all the non-empty numbers for > the day and get .449999 in- which matches what Weewx shows me for the day. > > 2020-07-29 00:00:00|0.0 > 2020-07-29 00:01:00|0.0 > 2020-07-29 00:02:00|0.0 > 2020-07-29 00:03:00|0.0 > 2020-07-29 00:04:00|0.0 > 2020-07-29 00:05:00|0.00999999999999979 > 2020-07-29 00:06:00|0.0 > 2020-07-29 00:07:00|0.0 > 2020-07-29 00:08:00|0.0 > 2020-07-29 00:09:00|0.0 > ... > 2020-07-29 17:38:00| > 2020-07-29 17:39:00|0.0 > 2020-07-29 17:40:00|0.0 > 2020-07-29 17:41:00|0.0 > 2020-07-29 17:42:00|0.0 > 2020-07-29 17:43:00|0.0 > 2020-07-29 17:44:00|0.0 > 2020-07-29 17:45:00|0.0100000000000016 > 2020-07-29 17:46:00|0.0 > 2020-07-29 17:47:00|0.0 > 2020-07-29 17:48:00|0.0 > 2020-07-29 17:49:00|0.0 > 2020-07-29 17:50:00|0.0 > > # cat jul29-sqlite-rain | awk -F'|' '{print $2}' | grep 0. | paste -sd+ - > | bc > .44999999999999951 > > > I'm thinking the blank entries might be due to a database lock error > (mid-day, I turned on debug logging and found the error below), but I'm > still not seeing how 49 out of 890 empty records could skew the numbers > this much, but I guess it could? If all the numbers in the logs add up to > the right amount, does this mean weewx might have seen it, but didn't get > that record written? I suppose if it was pouring (like it has been), this > might cause some meaningful records to get lost. Any thoughts from users > who might have seen this? I did adjust the sqlite write timeout and it > hasn't crapped out on me, so perhaps the next time I get rain I will know, > but wondering if anyone else had seen this much variance between weewx and > other apps. Don't get me wrong - I love weewx because I own my data, but I > need the data to be accurate, so hopefully it is the database lock issue. I > will report back. > > Jul 26 02:14:19 localhost weewxd: bmp280a: found pressure value of > 29.9625272895 mbar > Jul 26 02:14:19 localhost weewxd: PoolService: found entry Temperature -> > extraTemp1 with value of 85.7 using offset of 0.1 to get 85.8 > Jul 26 02:14:19 localhost weewx[6062] INFO weewx.manager: Added record > 2020-07-26 02:14:00 EDT (1595744040) to database 'weewx.sdb' > Jul 26 02:14:19 localhost weewx[6062] INFO weewx.manager: Added record > 2020-07-26 02:14:00 EDT (1595744040) to daily summary in 'weewx.sdb' > Jul 26 02:14:24 localhost weewx[6062] INFO weewx.engine: Main loop exiting > . Shutting engine down. > Jul 26 02:14:24 localhost weewx[6062] INFO weewx.engine: Shutting down > StdReport thread > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: Caught > unrecoverable exception in generator > 'user.belchertown.HighchartsJsonGenerator' > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** database is locked > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** Traceback (most recent call last): > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** File "/usr/share/weewx/weewx/reportengine.py", line 197, in run > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** obj.start() > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** File "/usr/share/weewx/weewx/reportengine.py", line 280, in start > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** self.run() > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** File "/usr/share/weewx/user/belchertown.py", line 1158, in run > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** start_ts = archive.firstGoodStamp() > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** File "/usr/share/weewx/weewx/manager.py", line 243, in > firstGoodStamp > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** _row = self.getSql("SELECT MIN(dateTime) FROM %s" % self. > table_name) > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** File "/usr/share/weewx/weewx/manager.py", line 426, in getSql > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** _cursor.execute(sql, sqlargs) > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** File "/usr/share/weewx/weedb/sqlite.py", line 43, in guarded_fn > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** raise weedb.OperationalError(e) > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** OperationalError: database is locked > Jul 26 02:14:31 localhost weewx[6062] ERROR weewx.reportengine: > **** Generator terminated > Jul 26 02:14:31 localhost weewxd: sdr: MainThread: shutdown process > rtl_433 -M utc -F json > Jul 26 02:14:31 localhost weewxd: sdr: MainThread: waiting for stdout- > thread > Jul 26 02:14:31 localhost weewxd: sdr: MainThread: waiting for stderr- > thread > Jul 26 02:14:41 localhost weewxd: sdr: MainThread: timed out waiting for > stderr-thread > Jul 26 02:14:41 localhost weewxd: sdr: MainThread: close stdout > Jul 26 02:14:41 localhost weewxd: sdr: MainThread: close stderr > > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c5d3a300-908c-4031-8981-068a2f84afa9n%40googlegroups.com.
