Hi,
On 2026/02/09 20:07, Gordan Bobic via discuss wrote:
On Mon, 9 Feb 2026 at 19:35, Xavier Figari via discuss
<[email protected]> wrote:
According to the MariaDB operator documentation, the snapshot is taken
as follows:
- Temporarily pause the MariaDB writes by executing a `FLUSH TABLES WITH
READ LOCK` command in one of the secondary Pods.
- Create a VolumeSnapshot resource of the data PVC mounted by the
MariaDB primary Pod.
- Wait until the VolumeSnapshot is provisioned by the storage system.
When timing out, the operator will delete the VolumeSnapshot resource
and retry the operation.
- Issue a `UNLOCK TABLE` statement.
If your snapshots are atomic (and they should be on any
snapshot-capable solution, or the snapshots are useless anyway), you
shouldn't need to do that.
Our experience has been that you're right. We still do flush tables
with read lock prior to making snapshots (using LVM, the thin variant on
newer systems, and desperately wishing we could convert our "thick"
volumes). We then use rsync to clone the snapshots off-host. We still
flush tables because we have encountered cases where somehow the backup
was unusable without it - I *suspect* it relates to a specific host
where disk IO was of such a nature (I really dislike virtual machines)
that we had to tune innodb to be less resilient in order to cope with
load. And in the case of galera it's also (as per my understanding)
less important to have all the relevant ACID safety settings enabled
since if a node crashes it'll simply SST from another again (although,
where we can we leave it on so that upon restart we can utilize an IST
rather where possible as this is much faster to restore).
InnoDB is crash-safe, so just taking a snapshot will still be
restorable, you'll just have to wait for redo log replay at startup,
which is not a big deal if the redo log is sanely sized.
Indeed. Assuming all settings are correctly set.
Regarding restoring galera - these snapshots will bring up a single node
(you will need to force it as it will be marked crashed, so you need to
set safe_to_bootstrap: 1 in grastate.dat upon restore).
Thereafter you SST the other nodes from that first one (which will use
mariabackup in our configurations). We don't use mariabackup for normal
day to day backups due to the space requirements, finding that volume
based snapshots are lower impact (on thin volumes at least), and
consumes less space (and more reliable, we found that under some
conditions mariabackup would not restore properly, at least observed
during galera SSTs where it took multiple tries, this was fixed by
tweaking some settings, increasing ring buffers and a few other innodb
specific settings - don't recall the details right at the moment). I
believe some of this was recently addressed in mariadb itself (at least,
I saw stuff on the ML I think is/could be related).
Kind regards,
Jaco
(https://github.com/mariadb-operator/mariadb-operator/blob/main/docs/physical_backup.md#volumesnapshots)
I'm asking myself a few questions on this strategy :
- Does it work with Galera clusters ? (In principle, yes; the example
given in the documentation uses Galera).
- Does it handle Galera desynchronization correctly, as explained in
https://mariadb.com/docs/galera-cluster/galera-management/general-operations/backing-up-a-mariadb-galera-cluster
?
- How do you perform a clean restore when using Galera ?
Restore to a single node, bring it up in bootstrap mode, SST to the other two.
So far, until MariaDB 12.3 is out, the binlogs are not restored with
this strategy. I'll have to backup them in another way.
Not sure how that is related, it depends on how/where your binlogs are
stored, but bear in mind that by default when binlogs are in use,
binlogs are used for recovery because there is an optimisation that
was brought in a while back that omits one of the flushes when binlogs
are enabled.
On ZFS, I normally put binlogs somewhere like $datadir/.bin/ so the
whole subtree can be atomically snapshotted and backed up.
They should be saved in the VolumeSnapshots, however, starting with
MariaDB 12.3 if I understand correctly : the
binlog_storage_engine=innodb parameter allows binlogs to be stored in
the InnoDB engine, instead of separate binary files (mysql-bin.*).
Binlogs will become an integral part of InnoDB data files, so they will
be included in the snapshot (tell me if I'm wrong).
Snapshots are file system based. If you are on LVM, if your binlogs
are in your $datadir they will be included.
On ZFS you might have something like:
pool/mysql -> /var/lib/mysql
pool/mysql/.bin -> /var/lib/mysql/.bin
And you can take an atomic recursive snapshot of pool/mysql to get the
snapshot of both pool/mysql and pool/mysql/.bin at the same point in
time, i.e. mutually consistent.
If you arrange things that way there is no reason why you would need
to treat your binlogs in a special, separate way.
Instead of using the MariaDB operator to perform the snapshot, could the
volume snapshot be achieved with a tool like Kasten? Kasten probably
won't do "FLUSH TABLES WITH READ LOCK" before the snapshot, in
particular, but I guess we could create pre-exec and post-exec hooks.
No idea what that is, but as long as it takes a point-in-time
consistent snapshot that then remains static or the duration of the
backup, it should work just fine.
_______________________________________________
discuss mailing list -- [email protected]
To unsubscribe send an email to [email protected]
