Ok, I am getting the gist of your concern. A possible approach here is if you are running MariaDB on your WeeWX server, you install MariaDBs client tools on the target server where you store your backups and run their Maria Backup solution to do incremental/differential backups. Here is an article to review : Incremental Backup and Restore with Mariabackup - MariaDB Knowledge Base <https://mariadb.com/kb/en/incremental-backup-and-restore-with-mariabackup/>
This configuration gives you the ability to backup the instance over the wire and keep incremental backups until you perform a full backup again. If you do this approach, make sure you run periodic full backups between incremental backups as it can be tedious to restore a database with a large number of incremental backups to apply. Personally, I just run mysqldump on each of my databases each day, compress the export, and copy it over to my NAS. Not the most efficient process, but it is simple for me and has been helpful for those situations I needed to restore a backup. DDJ On Thu, Jul 28, 2022 at 10:35 AM Guido Cioni <[email protected]> wrote: > Hey Doug, > thanks for chiming in. > As a matter of fact I'm not really worried about the data to be > transmitted incrementally which, as you say, is not large by any mean. > > What I'm trying to avoid is having to transfer the whole database dump > every time that a backup is performed, because that can become demanding > when the archive grows for many years: I have another station running a > similar DB and after more than 10 years the full DB has become about 400 MB > in size... > > On Thursday, July 28, 2022 at 4:31:55 PM UTC+2 [email protected] > wrote: > >> Guido: >> >> I was going to recommend if this is that important to you to configure a >> database cluster. MariaDB has Galera Clustering (What is MariaDB Galera >> Cluster? - MariaDB Knowledge Base >> <https://mariadb.com/kb/en/what-is-mariadb-galera-cluster/>) which will >> perform database replication and distribution for you. You need at least 3 >> nodes/servers to have a proper cluster. >> >> Keep in mind that WeeWX only writes to the database during the archive >> operation, so if you archive every minute, you are only writing 1440 >> records to the archive table (and another 1440+ records to the summary >> tables per measurement). that is literally kilobytes of data per day. With >> today's modern networks, performance is not an issue. >> >> You can spend the time to build an incremental backup solution, but I >> think that is a lot of work for a solution that writes so little amount of >> data each day. >> >> Just my $0.02 >> >> DDJ >> >> On Thu, Jul 28, 2022 at 10:27 AM Guido Cioni <[email protected]> wrote: >> >>> From what I understand weewx uses sqlite because of the default, and >>> with sqlite the only possibility is to dump the whole database every time ( >>> https://github.com/glennmckechnie/weewx-sqlbackup/wiki/SQLBackup-README), >>> which is obviously time consuming. >>> >>> With Pandas you can get around that by doing queries and requesting only >>> the data that you need to transfer every time, and the data gets saved into >>> the memory allocated by Python so the db is left untouched. Of course one >>> does not necessarily have to use Pandas, it's just one of the framework >>> that I'm familiar with. >>> >>> Plus, it seems that none of the extensions in the wiki allows for saving >>> data directly into a DB but only for making a physical copy of the DB >>> (again, from my understanding). >>> >>> On Thursday, July 28, 2022 at 4:23:04 PM UTC+2 [email protected] wrote: >>> >>>> Can you get there with doing mysql replication at the database level ? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-development/391c83da-fb02-447e-bcee-fed1b42bbbccn%40googlegroups.com >>> <https://groups.google.com/d/msgid/weewx-development/391c83da-fb02-447e-bcee-fed1b42bbbccn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-development/b45cf968-ee04-4256-bd06-b70269d1eb4bn%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-development/b45cf968-ee04-4256-bd06-b70269d1eb4bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CACC0i0yyJoCaqBzGJh6DR4nWNfAeMnQ%2Bm-8ekL6cD7caMhdZKg%40mail.gmail.com.
