Trying it. Seems very slow compared to a file copy. Doing it one page at a time is really slow. Like 100x slower than a file copy. I don't have any sense as to the page size of this database. I'm probably going to end up doing 100 pages at a time.
The example here https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup is trivial to implement. Cut and paste and change the filenames/paths trivial. I'm going to run the backup continuously in a loop on my dev system for a while today and if it runs without problems, I'll update my backup script to use it rather than a file copy. On Sunday, January 10, 2021 at 7:22:43 AM UTC-8 [email protected] wrote: > If the backup takes long enough, it could interfere with writing a record > to the database. Eventually, the write will time out, causing weewxd to > restart from the top. It won't crash weewxd (that is, cause it to exit), > nor corrupt the database, but the record would be lost. > > That's the advantage of the incremental backup approach. The backup > process never holds a lock on the database for very long. Just a second or > two. > > BTW, the backup API > <https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup> is > now included in Python starting with V3.7. If you have a modern version of > Python, you don't have to write a C program to access the API. > > I'm hoping to inspire someone to write a simple script that would run once > a day using the backup API. > > -tk > > > > On Sun, Jan 10, 2021 at 7:13 AM David Levine <[email protected]> wrote: > >> I understand database consistency in a transactional database and I'm >> wondering about the risk of copying weewx.sdb without stopping weewx first. >> Would you possibly lose an in-flight transaction or might the entire sdb be >> inconsistent and unusable? An in-flight copy would seem to be similar to >> losing power to the device vs a graceful shutdown. >> >> On Sunday, January 10, 2021 at 9:46:55 AM UTC-5 [email protected] >> wrote: >> >>> I have a script that stops weewx, makes a local copy of the db restarts >>> weewx and then copies the backup to google drive. >>> >>> The copy takes less than 2 minutes so I don't lose data. >>> >>> On Sun, Jan 10, 2021, 5:39 AM Tom Keffer <[email protected]> wrote: >>> >>>> Your approach will certainly work, but requires stopping weewxd for >>>> what could potentially be a long period of time, so you might miss a >>>> weather event. >>>> >>>> Another approach is to use the sqlite3 ".backup" command. Replace your >>>> tar command with >>>> >>>> tar czf $dest/$archive_file $backup_files2 $backup_files3 $backup_files4 >>>> sqlite3 $backup_files1 ".backup $dest/$backup_files1.backup" >>>> >>>> This avoids stopping weewxd, because the sqlite3 utility will take care >>>> of any necessary locking. However, it has the disadvantage that if sqlite3 >>>> holds on to the lock for too long, a database write will get delayed and, >>>> ultimately, could time out, causing weewxd to restart. >>>> >>>> Finally, the most sophisticated approach is to incrementally back up >>>> the database. Take a look at this page on backing up a running database >>>> <https://sqlite.org/backup.html>. It copies a number of database >>>> pages, then releases the lock, sleeps for a bit to allow other processes >>>> to >>>> gain access to the database, then goes on to the next set of pages. This >>>> allows the database to be backed up without stopping weewxd, and without >>>> the potential hazard of a database timeout. >>>> >>>> Something to think about... >>>> >>>> -tk >>>> >>>> >>>> >>>> On Sun, Jan 10, 2021 at 4:01 AM Jan Stelling <[email protected]> >>>> wrote: >>>> >>>>> For some time, I was looking for an easy and independent (from weewx) >>>>> way to automatically backup my weewx data, as I do not want to lose data >>>>> if >>>>> the Micro SD breaks down. >>>>> Recently, I found this small repo >>>>> <https://github.com/PapyLeCelte/Weewx-Backup> on github which only >>>>> contains a backup script file. It mounts a USB drive, stops weewx, >>>>> creates >>>>> an archived backup of the most relevant user files and folders on the USB >>>>> drive, unmounts the drive and restarts weewx. >>>>> >>>>> This was almost perfect for me, but I had to introduce some changes to >>>>> make it suitable for my environment. I forked it >>>>> <https://github.com/jnstllng/Weewx-Backup> to make it available for >>>>> others. It now does the following: >>>>> >>>>> 1. Stop weewx >>>>> 2. Create an archived backup on a mounted network drive (under >>>>> /home/pi/Shares/Temp) >>>>> 3. Start weewx >>>>> >>>>> I tested it manually and running via crontab on my RasPi 3B. >>>>> >>>>> Maybe this is useful for some of you... >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "weewx-user" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/weewx-user/2671c065-719a-4435-9657-06841af8fed7n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/weewx-user/2671c065-719a-4435-9657-06841af8fed7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/CAPq0zEDK%3DkX-zkzewA%2B%2BDXzOkBPMLBUPHaVgJ69qYbaYWNUGzw%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-user/CAPq0zEDK%3DkX-zkzewA%2B%2BDXzOkBPMLBUPHaVgJ69qYbaYWNUGzw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/fa03941e-6ea2-41fb-bf62-998ea49651ddn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-user/fa03941e-6ea2-41fb-bf62-998ea49651ddn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/bd898be4-67e6-4888-acf1-7e73d4339238n%40googlegroups.com.
