The backup of weewx_wbx1 normally takes 10-11 seconds. This morning (with the deathlock error) it took 16 seconds. I have added option --skip-lock-tables
Op zaterdag 1 mei 2021 om 12:03:41 UTC-3 schreef [email protected]: > OK, what I suspected. > > We could add a retry to the transaction. However, the database backup is > likely to take a lot longer than the weewx write, so even with a couple of > retries, it may still deadlock, unless very lengthy delays are added. This > would be undesirable because the weewx writes are in the main thread. So, > the program will block, waiting for your database dump to finish, which in > some cases could be several minutes. > > I'm not an expert on this, but how about instructing mysqldump not to lock > the database by using option --lock-table=false? Or, is there a way to get > mysqldump to do a retry? It's likely to be more successful than having > weewx do the retry. > > -tk > > > On Sat, May 1, 2021 at 7:26 AM Luc Heijst <[email protected]> wrote: > >> The weewx instances run on 11 different raspberry pi's. weewx_wbx1 and >> weewx_cmon(33) run on the same PI. >> They all run weewx v.4.4.0 and Python 3.7.3. >> All weewx instances write to remote MySQL databases on my Synology NAS. >> >> See below the backup on my NAS.and some more of my syslog file >> >> ============================ >> #! /bin/sh >> # Dayly backup of main weewx tables >> DATETIME=$(date +"%Y-%m-%d %H:%M:%S") >> echo "$DATETIME start" >> BACKUP="/usr/local/mariadb10/bin/mysqldump -uroot -pXXXXXXXX" >> OUTDIR=/volume1/data-h5/_mysql >> ... >> DATETIME=$(date +"%Y-%m-%d %H:%M:%S") >> echo "$DATETIME cmon33" >> $BACKUP cmon33 > $OUTDIR/cmon33.sql >> DATETIME=$(date +"%Y-%m-%d %H:%M:%S") >> ... >> DATETIME=$(date +"%Y-%m-%d %H:%M:%S") >> echo "$DATETIME weewx_wbx1" >> $BACKUP weewx_wbx1 > $OUTDIR/weewx_wbx1.sql >> DATETIME=$(date +"%Y-%m-%d %H:%M:%S") >> echo "$DATETIME finished" >> >> ============================ >> >> May 1 03:15:14 pi33 wbx1[495] DEBUG user.meteostick: meteostick: >> data['rxCheckPercent']: 94 >> May 1 03:15:14 pi33 wbx1[495] DEBUG user.meteostick: meteostick: RF >> summary: rf_sensitivity=73 (values in dB) >> May 1 03:15:14 pi33 wbx1[495] DEBUG user.meteostick: meteostick: >> Station max min avg last count [missed] [good] >> May 1 03:15:14 pi33 wbx1[495] DEBUG user.meteostick: meteostick: iss >> -49 -52 -50 -51 104 8 93 >> May 1 03:15:14 pi33 wbx1[495] DEBUG user.meteostick: meteostick: >> anemometer -58 -59 -58 -59 107 7 94 >> May 1 03:15:14 pi33 wbx1[495] DEBUG user.meteostick: meteostick: >> leaf_soil -43 -46 -43 -46 94 6 94 >> May 1 03:15:15 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:15:15 -03 (1619849715) to database 'cmon33' >> May 1 03:15:26 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:15:15 -03 (1619849715) to daily summary in 'cmon33' >> May 1 03:15:32 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:15:00 -03 (1619849700) to database 'wx_cmon33' >> May 1 03:15:36 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:15:00 -03 (1619849700) to daily summary in 'wx_cmon33' >> May 1 03:15:36 pi33 cmon[544] DEBUG weewx.reportengine: Running reports >> for latest time in the database. >> May 1 03:15:36 pi33 cmon[544] DEBUG weewx.reportengine: Running report >> 'cmon33' >> May 1 03:15:36 pi33 cmon[544] DEBUG weewx.reportengine: Found >> configuration file /home/weewx/skins/cmon33/skin.conf for report 'cmon33' >> May 1 03:15:36 pi33 cmon[544] DEBUG weewx.cheetahgenerator: Using search >> list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', >> 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', >> 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras'] >> May 1 03:15:36 pi33 cmon[544] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:15:42 pi33 cmon[544] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:15:47 pi33 wbx1[495] INFO weewx.manager: Added record >> 2021-05-01 03:15:00 -03 (1619849700) to database 'weewx_wbx1' >> May 1 03:15:51 pi33 wbx1[495] INFO weewx.manager: Added record >> 2021-05-01 03:15:00 -03 (1619849700) to daily summary in 'weewx_wbx1' >> May 1 03:15:51 pi33 wbx1[495] DEBUG weewx.reportengine: Running reports >> for latest time in the database. >> May 1 03:15:51 pi33 wbx1[495] DEBUG weewx.reportengine: Running report >> 'SeasonsReport' >> May 1 03:15:51 pi33 wbx1[495] DEBUG weewx.reportengine: Found >> configuration file /home/weewx/skins/Seasons/skin.conf for report >> 'SeasonsReport' >> May 1 03:15:52 pi33 wbx1[495] DEBUG user.weather365: restx: Weather365: >> Data uploaded to https://channel1.weather365.net/stations/index.php is: >> ... >> May 1 03:15:52 pi33 wbx1[495] DEBUG weewx.cheetahgenerator: Using search >> list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', >> 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', >> 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras'] >> May 1 03:15:52 pi33 wbx1[495] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:16:07 pi33 cmon[544] DEBUG weewx.reportengine: Report 'FTP' not >> enabled. Skipping. >> May 1 03:16:07 pi33 cmon[544] DEBUG weewx.reportengine: Report 'RSYNC' >> not enabled. Skipping. >> May 1 03:16:16 pi33 wbx1[495] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:16:18 pi33 wbx1[495] DEBUG weewx.reportengine: Running report >> 'exfoliation' >> May 1 03:16:18 pi33 wbx1[495] DEBUG weewx.reportengine: Found >> configuration file /home/weewx/skins/exfoliation/skin.conf for report >> 'exfoliation' >> May 1 03:16:18 pi33 wbx1[495] DEBUG weewx.cheetahgenerator: Using search >> list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', >> 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', >> 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras'] >> May 1 03:16:18 pi33 wbx1[495] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:16:33 pi33 wbx1[495] DEBUG weewx.cheetahgenerator: Skip >> 'forecast.html': last_mod=1619846747.117392 age=3046.250206708908 stale=3570 >> May 1 03:16:34 pi33 wbx1[495] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:16:36 pi33 wbx1[495] DEBUG weewx.reportengine: Running report >> 'FTP' >> May 1 03:16:36 pi33 wbx1[495] DEBUG weewx.reportengine: Found >> configuration file /home/weewx/skins/Ftp/skin.conf for report 'FTP' >> May 1 03:16:36 pi33 wbx1[495] DEBUG weeutil.ftpupload: Attempting >> connection to ftp.lucdesign.nl >> May 1 03:16:38 pi33 wbx1[495] DEBUG weeutil.ftpupload: Connected to >> ftp.lucdesign.nl >> May 1 03:16:38 pi33 wbx1[495] DEBUG weewx.reportengine: Report 'RSYNC' >> not enabled. Skipping. >> May 1 03:17:01 pi33 CRON[5754]: (root) CMD ( cd / && run-parts >> --report /etc/cron.hourly) >> May 1 03:20:15 pi33 wbx1[495] DEBUG user.meteostick: meteostick: >> data['rxCheckPercent']: 91 >> May 1 03:20:15 pi33 wbx1[495] DEBUG user.meteostick: meteostick: RF >> summary: rf_sensitivity=73 (values in dB) >> May 1 03:20:15 pi33 wbx1[495] DEBUG user.meteostick: meteostick: >> Station max min avg last count [missed] [good] >> May 1 03:20:15 pi33 wbx1[495] DEBUG user.meteostick: meteostick: iss >> -49 -51 -50 -51 104 8 93 >> May 1 03:20:15 pi33 wbx1[495] DEBUG user.meteostick: meteostick: >> anemometer -58 -60 -58 -59 104 9 92 >> May 1 03:20:15 pi33 wbx1[495] DEBUG user.meteostick: meteostick: >> leaf_soil -43 -44 -43 -44 87 12 88 >> May 1 03:20:17 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:20:17 -03 (1619850017) to database 'cmon33' >> May 1 03:20:24 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:20:17 -03 (1619850017) to daily summary in 'cmon33' >> May 1 03:20:29 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:20:00 -03 (1619850000) to database 'wx_cmon33' >> May 1 03:20:31 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:20:00 -03 (1619850000) to daily summary in 'wx_cmon33' >> May 1 03:20:31 pi33 cmon[544] DEBUG weewx.reportengine: Running reports >> for latest time in the database. >> May 1 03:20:31 pi33 cmon[544] DEBUG weewx.reportengine: Running report >> 'cmon33' >> May 1 03:20:31 pi33 cmon[544] DEBUG weewx.reportengine: Found >> configuration file /home/weewx/skins/cmon33/skin.conf for report 'cmon33' >> May 1 03:20:31 pi33 cmon[544] DEBUG weewx.cheetahgenerator: Using search >> list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', >> 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', >> 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras'] >> May 1 03:20:32 pi33 cmon[544] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:20:38 pi33 cmon[544] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:20:43 pi33 wbx1[495] INFO weewx.engine: Main loop exiting. >> Shutting engine down. >> May 1 03:20:43 pi33 wbx1[495] INFO weewx.engine: Shutting down StdReport >> thread >> May 1 03:20:43 pi33 wbx1[495] DEBUG weewx.engine: StdReport thread has >> been terminated >> May 1 03:20:43 pi33 wbx1[495] DEBUG weewx.restx: Shut down Weather365 >> thread. >> May 1 03:20:43 pi33 wbx1[495] DEBUG weewx.restx: Shut down MQTT thread. >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: Caught unrecoverable >> exception: >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** (1213, >> 'Deadlock found when trying to get lock; try restarting transaction') >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** Traceback >> (most recent call last): >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/engine.py", line 214, in run >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> self.dispatchEvent(weewx.Event(weewx.CHECK_LOOP, packet=packet)) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/engine.py", line 245, in dispatchEvent >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> callback(event) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/engine.py", line 617, in check_loop >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** raise >> BreakLoop >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> weewx.engine.BreakLoop >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** During >> handling of the above exception, another exception occurred: >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** Traceback >> (most recent call last): >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weedb/mysql.py", line 52, in guarded_fn >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** return >> fn(*args, **kwargs) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weedb/mysql.py", line 262, in execute >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> self.cursor.execute(mysql_string, tuple(sql_tuple)) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 250, in execute >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> self.errorhandler(self, exc, value) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 50, in >> defaulterrorhandler >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** raise >> errorvalue >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 247, in execute >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** res = >> self._query(query) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 411, in _query >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** rowcount >> = self._do_query(q) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 374, in _do_query >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> db.query(q) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 292, in query >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> _mysql.connection.query(self, query) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> _mysql_exceptions.OperationalError: (1213, 'Deadlock found when trying to >> get lock; try restarting transaction') >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** During >> handling of the above exception, another exception occurred: >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** Traceback >> (most recent call last): >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewxd", line 157, in main >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> engine.run() >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/engine.py", line 221, in run >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> self.dispatchEvent(weewx.Event(weewx.POST_LOOP)) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/engine.py", line 245, in dispatchEvent >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> callback(event) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/engine.py", line 627, in post_loop >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> self._software_catchup() >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/engine.py", line 697, in _software_catchup >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> origin='software')) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/engine.py", line 245, in dispatchEvent >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> callback(event) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/engine.py", line 658, in new_archive_record >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> dbmanager.addRecord(event.record, accumulator=self.old_accumulator) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/manager.py", line 275, in addRecord >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> self._addSingleRecord(record, cursor) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/manager.py", line 902, in _addSingleRecord >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> super(DaySummaryManager, self)._addSingleRecord(record, cursor) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weewx/manager.py", line 325, in _addSingleRecord >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> cursor.execute(sql_insert_stmt, value_list) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >> "/home/weewx/bin/weedb/mysql.py", line 61, in guarded_fn >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** raise >> klass(e) >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >> weedb.DatabaseError: (1213, 'Deadlock found when trying to get lock; try >> restarting transaction') >> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** Exiting. >> May 1 03:21:00 pi33 cmon[544] DEBUG weewx.reportengine: Report 'FTP' not >> enabled. Skipping. >> May 1 03:21:00 pi33 cmon[544] DEBUG weewx.reportengine: Report 'RSYNC' >> not enabled. Skipping. >> May 1 03:25:22 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:25:23 -03 (1619850323) to database 'cmon33' >> May 1 03:25:24 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:25:23 -03 (1619850323) to daily summary in 'cmon33' >> May 1 03:25:27 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:25:00 -03 (1619850300) to database 'wx_cmon33' >> May 1 03:25:28 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:25:00 -03 (1619850300) to daily summary in 'wx_cmon33' >> May 1 03:25:28 pi33 cmon[544] DEBUG weewx.reportengine: Running reports >> for latest time in the database. >> May 1 03:25:28 pi33 cmon[544] DEBUG weewx.reportengine: Running report >> 'cmon33' >> May 1 03:25:28 pi33 cmon[544] DEBUG weewx.reportengine: Found >> configuration file /home/weewx/skins/cmon33/skin.conf for report 'cmon33' >> May 1 03:25:28 pi33 cmon[544] DEBUG weewx.cheetahgenerator: Using search >> list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', >> 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', >> 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras'] >> May 1 03:25:28 pi33 cmon[544] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:25:35 pi33 cmon[544] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:25:49 pi33 cmon[544] DEBUG weewx.reportengine: Report 'FTP' not >> enabled. Skipping. >> May 1 03:25:49 pi33 cmon[544] DEBUG weewx.reportengine: Report 'RSYNC' >> not enabled. Skipping. >> May 1 03:30:19 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:30:19 -03 (1619850619) to database 'cmon33' >> May 1 03:30:21 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:30:19 -03 (1619850619) to daily summary in 'cmon33' >> May 1 03:30:27 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:30:00 -03 (1619850600) to database 'wx_cmon33' >> May 1 03:30:29 pi33 cmon[544] INFO weewx.manager: Added record >> 2021-05-01 03:30:00 -03 (1619850600) to daily summary in 'wx_cmon33' >> May 1 03:30:29 pi33 cmon[544] DEBUG weewx.reportengine: Running reports >> for latest time in the database. >> May 1 03:30:29 pi33 cmon[544] DEBUG weewx.reportengine: Running report >> 'cmon33' >> May 1 03:30:29 pi33 cmon[544] DEBUG weewx.reportengine: Found >> configuration file /home/weewx/skins/cmon33/skin.conf for report 'cmon33' >> May 1 03:30:29 pi33 cmon[544] DEBUG weewx.cheetahgenerator: Using search >> list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', >> 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', >> 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras'] >> May 1 03:30:29 pi33 cmon[544] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:30:40 pi33 cmon[544] DEBUG weewx.manager: Daily summary version >> is 4.0 >> May 1 03:30:58 pi33 cmon[544] DEBUG weewx.reportengine: Report 'FTP' not >> enabled. Skipping. >> May 1 03:30:58 pi33 cmon[544] DEBUG weewx.reportengine: Report 'RSYNC' >> not enabled. Skipping. >> >> >> Op zaterdag 1 mei 2021 om 10:54:15 UTC-3 schreef [email protected]: >> >>> Trying to understand your setup >>> >>> 1. You have many instances of weewx running simultaneously. >>> 2. They all get backed up every day. >>> 3. One of them, weewx_wbx1, crashes about once a month during the backup. >>> 4. What version of WeeWX? >>> >>> I don't attach any significance to the three different nested >>> exceptions: that's how the stack would normally get unwound from a database >>> exception. However, I don't understand the last one: dbmanager.addRecord(). >>> By itself, this does not lock the database. Is there more to the log? >>> Something below where you cut it off? >>> >>> Depending on the answer to that, we may have to retry the transaction. >>> Right now, there is only one attempt. >>> >>> -tk >>> >>> >>> >>> On Sat, May 1, 2021 at 4:58 AM Luc Heijst <[email protected]> wrote: >>> >>>> Once a day my weewx MySQL databases are backupped, see log of the >>>> backup below. >>>> The weewx programs are not stopped during these backups. >>>> One weewx program (weewx_wbx1) crashes once in a wile (once per month) >>>> with a number of deadlock found messages. >>>> >>>> It looks like weewx gets two more nested deadlock errors before a retry >>>> could be invoked. >>>> >>>> Any solutions? >>>> >>>> Backup log: >>>> ========================================= >>>> Taak: mysql_backup >>>> Starttijd: Sat, 01 May 2021 03:00:02 GMT >>>> Stoptijd: Sat, 01 May 2021 03:21:38 GMT >>>> Huidige status: 0 (Normaal) >>>> Standaard uitvoer/fout: >>>> 2021-05-01 03:00:03 start >>>> 2021-05-01 03:00:03 cmon11 >>>> 2021-05-01 03:00:10 cmon21 >>>> 2021-05-01 03:00:27 cmon31 >>>> 2021-05-01 03:02:31 cmon32 >>>> 2021-05-01 03:02:37 cmon33 >>>> 2021-05-01 03:02:48 cmon34 >>>> 2021-05-01 03:03:05 cmon35 >>>> 2021-05-01 03:03:22 cmon36 >>>> 2021-05-01 03:03:49 cmon37 >>>> 2021-05-01 03:04:09 cmon41 >>>> 2021-05-01 03:04:12 cmon42 >>>> 2021-05-01 03:04:15 lightning >>>> 2021-05-01 03:04:15 mbemdevices >>>> 2021-05-01 03:05:51 weewx_klim >>>> 2021-05-01 03:07:45 weewx_mbem >>>> 2021-05-01 03:14:26 weewx_mstk >>>> 2021-05-01 03:17:18 weewx_rtld >>>> 2021-05-01 03:18:03 weewx_tfrc >>>> 2021-05-01 03:18:26 weewx_vpro >>>> 2021-05-01 03:20:06 weewx_wamo >>>> 2021-05-01 03:20:40 weewx_wamodevices >>>> 2021-05-01 03:21:21 weewx_wbx1 >>>> 2021-05-01 03:21:37 finished >>>> >>>> weewx error log: >>>> ========================================= >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: Caught unrecoverable >>>> exception: >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** (1213, >>>> 'Deadlock found when trying to get lock; try restarting transaction') >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** Traceback >>>> (most recent call last): >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewx/engine.py", line 214, in run >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> self.dispatchEvent(weewx.Event(weewx.CHECK_LOOP, packet=packet)) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewx/engine.py", line 245, in dispatchEvent >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> callback(event) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewx/engine.py", line 617, in check_loop >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** raise >>>> BreakLoop >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> weewx.engine.BreakLoop >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** During >>>> handling of the above exception, another exception occurred: >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** Traceback >>>> (most recent call last): >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weedb/mysql.py", line 52, in guarded_fn >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** return >>>> fn(*args, **kwargs) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weedb/mysql.py", line 262, in execute >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> self.cursor.execute(mysql_string, tuple(sql_tuple)) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 250, in execute >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> self.errorhandler(self, exc, value) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 50, in >>>> defaulterrorhandler >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** raise >>>> errorvalue >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 247, in execute >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** res = >>>> self._query(query) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 411, in _query >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> rowcount = self._do_query(q) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 374, in _do_query >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> db.query(q) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 292, in query >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> _mysql.connection.query(self, query) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> _mysql_exceptions.OperationalError: (1213, 'Deadlock found when trying to >>>> get lock; try restarting transaction') >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** During >>>> handling of the above exception, another exception occurred: >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** Traceback >>>> (most recent call last): >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewxd", line 157, in main >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> engine.run() >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewx/engine.py", line 221, in run >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> self.dispatchEvent(weewx.Event(weewx.POST_LOOP)) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewx/engine.py", line 245, in dispatchEvent >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> callback(event) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewx/engine.py", line 627, in post_loop >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> self._software_catchup() >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewx/engine.py", line 697, in _software_catchup >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> origin='software')) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewx/engine.py", line 245, in dispatchEvent >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> callback(event) >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** File >>>> "/home/weewx/bin/weewx/engine.py", line 658, in new_archive_record >>>> May 1 03:20:43 pi33 wbx1[495] CRITICAL __main__: **** >>>> dbmanager.addRecord(event.record, accumulator=self.old_accumulator) >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/09845fe4-5e21-4bc4-a835-54392dbf9589n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-user/09845fe4-5e21-4bc4-a835-54392dbf9589n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/17e9742e-94d0-48d7-b964-aa17f9da21fen%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-user/17e9742e-94d0-48d7-b964-aa17f9da21fen%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/20858265-e38f-40b8-b42a-427718d8f9d5n%40googlegroups.com.
