If the backup takes long enough, it could interfere with writing a record to the database. Eventually, the write will time out, causing weewxd to restart from the top. It won't crash weewxd (that is, cause it to exit), nor corrupt the database, but the record would be lost.
That's the advantage of the incremental backup approach. The backup process never holds a lock on the database for very long. Just a second or two. BTW, the backup API <https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup> is now included in Python starting with V3.7. If you have a modern version of Python, you don't have to write a C program to access the API. I'm hoping to inspire someone to write a simple script that would run once a day using the backup API. -tk On Sun, Jan 10, 2021 at 7:13 AM David Levine <[email protected]> wrote: > I understand database consistency in a transactional database and I'm > wondering about the risk of copying weewx.sdb without stopping weewx first. > Would you possibly lose an in-flight transaction or might the entire sdb be > inconsistent and unusable? An in-flight copy would seem to be similar to > losing power to the device vs a graceful shutdown. > > On Sunday, January 10, 2021 at 9:46:55 AM UTC-5 [email protected] wrote: > >> I have a script that stops weewx, makes a local copy of the db restarts >> weewx and then copies the backup to google drive. >> >> The copy takes less than 2 minutes so I don't lose data. >> >> On Sun, Jan 10, 2021, 5:39 AM Tom Keffer <[email protected]> wrote: >> >>> Your approach will certainly work, but requires stopping weewxd for what >>> could potentially be a long period of time, so you might miss a weather >>> event. >>> >>> Another approach is to use the sqlite3 ".backup" command. Replace your >>> tar command with >>> >>> tar czf $dest/$archive_file $backup_files2 $backup_files3 $backup_files4 >>> sqlite3 $backup_files1 ".backup $dest/$backup_files1.backup" >>> >>> This avoids stopping weewxd, because the sqlite3 utility will take care >>> of any necessary locking. However, it has the disadvantage that if sqlite3 >>> holds on to the lock for too long, a database write will get delayed and, >>> ultimately, could time out, causing weewxd to restart. >>> >>> Finally, the most sophisticated approach is to incrementally back up the >>> database. Take a look at this page on backing up a running database >>> <https://sqlite.org/backup.html>. It copies a number of database pages, >>> then releases the lock, sleeps for a bit to allow other processes to gain >>> access to the database, then goes on to the next set of pages. This allows >>> the database to be backed up without stopping weewxd, and without the >>> potential hazard of a database timeout. >>> >>> Something to think about... >>> >>> -tk >>> >>> >>> >>> On Sun, Jan 10, 2021 at 4:01 AM Jan Stelling <[email protected]> >>> wrote: >>> >>>> For some time, I was looking for an easy and independent (from weewx) >>>> way to automatically backup my weewx data, as I do not want to lose data if >>>> the Micro SD breaks down. >>>> Recently, I found this small repo >>>> <https://github.com/PapyLeCelte/Weewx-Backup> on github which only >>>> contains a backup script file. It mounts a USB drive, stops weewx, creates >>>> an archived backup of the most relevant user files and folders on the USB >>>> drive, unmounts the drive and restarts weewx. >>>> >>>> This was almost perfect for me, but I had to introduce some changes to >>>> make it suitable for my environment. I forked it >>>> <https://github.com/jnstllng/Weewx-Backup> to make it available for >>>> others. It now does the following: >>>> >>>> 1. Stop weewx >>>> 2. Create an archived backup on a mounted network drive (under >>>> /home/pi/Shares/Temp) >>>> 3. Start weewx >>>> >>>> I tested it manually and running via crontab on my RasPi 3B. >>>> >>>> Maybe this is useful for some of you... >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/2671c065-719a-4435-9657-06841af8fed7n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/weewx-user/2671c065-719a-4435-9657-06841af8fed7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/CAPq0zEDK%3DkX-zkzewA%2B%2BDXzOkBPMLBUPHaVgJ69qYbaYWNUGzw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/weewx-user/CAPq0zEDK%3DkX-zkzewA%2B%2BDXzOkBPMLBUPHaVgJ69qYbaYWNUGzw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/fa03941e-6ea2-41fb-bf62-998ea49651ddn%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/fa03941e-6ea2-41fb-bf62-998ea49651ddn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEBn3MeXjYuJ5mXHYccT1Bs4zG6%2BKancwC1dnpTzThDJiw%40mail.gmail.com.
