Re: btrfs issue with mariadb incremental backup
Hi Chris, Thank you very much for your suggestion. I didn't remember which steps that I made a mistake and made the mysql had Received UUID. I have done the following and it work as it should be right now. [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708070830 rw_201708070830 Create a snapshot of 'mbroken_201708070830' in './rw_201708070830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 568 top level 5 path mbroken_201708080830 ID 319 gen 569 top level 5 path mbroken_201708090830 ID 320 gen 570 top level 5 path mbroken_201708100830 ID 321 gen 571 top level 5 path mbroken_201708110830 ID 322 gen 572 top level 5 path mbroken_201708120830 ID 323 gen 573 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708080830 rw_201708080830 Create a snapshot of 'mbroken_201708080830' in './rw_201708080830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708090830 rw_201708090830 Create a snapshot of 'mbroken_201708090830' in './rw_201708090830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708100830 rw_201708100830 Create a snapshot of 'mbroken_201708100830' in './rw_201708100830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708110830 rw_201708110830 Create a snapshot of 'mbroken_201708110830' in './rw_201708110830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708120830 rw_201708120830 Create a snapshot of 'mbroken_201708120830' in './rw_201708120830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708130830 rw_201708130830 Create a snapshot of 'mbroken_201708130830' in './rw_201708130830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 577 top level 5 path mbroken_201708080830 ID 319 gen 578 top level 5 path mbroken_201708090830 ID 320 gen 579 top level 5 path mbroken_201708100830 ID 321 gen 580 top level 5 path mbroken_201708110830 ID 322 gen 581 top level 5 path mbroken_201708120830 ID 323 gen 582 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 ID 349 gen 577 top level 5 path rw_201708080830 ID 350 gen 578 top level 5 path rw_201708090830 ID 351 gen 579 top level 5 path rw_201708100830 ID 352 gen 580 top level 5 path rw_201708110830 ID 353 gen 581 top level 5 path rw_201708120830 ID 354 gen 582 top level 5 path rw_201708130830 [root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep "3ad0334a-4063-654c-add6-b1cbcdeaa639" ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708070830 ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708080830 ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708090830 ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708100830 ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708110830 ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708120830 ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708130830 [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708070830 mysql_201708070830 Create a readonly snapshot of 'rw_201708070830' in './mysql_201708070830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708080830 mysql_201708080830 Create a readonly snapshot of 'rw_201708080830' in './mysql_201708080830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708090830 mysql_201708090830 Create a readonly snapshot of 'rw_201708090830' in './mysql_201708090830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708100830 mysql_201708100830 Create a readonly snapshot of 'rw_201708100830' in './mysql_201708100830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708110830 mysql_201708110830 Create a readonly snapshot of 'rw_201708110830' in './mysql_201708110830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708120830 mysql_201708120830 Create a readonly snapshot of 'rw_201708120830' in './mysql_201708120830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708130830 mysql_201708130830 Create a readonly snapshot of 'rw_201708130830' in './mysql_201708130830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 577 top level 5 path mbroken_201708080830 ID 319 gen 578 top level 5 path mbroken_201708090830 ID 320 gen 579 top level 5 path mbroken_201708100830 ID 32
Re: btrfs issue with mariadb incremental backup
Hi "A L", Thank you very much for your suggestion. I've got it. It work properly right now. what I have done are following. mv all current mysql and snapshot mysql to mbroken and mbroken_yymmddhhmm btrfs subvolume snapshot mbroken to mysql (to make it rw subvolume without Recieve UUID) btrfs subvolume snapshot mbroken_yymmddhhmm to rw_yymmddhhmm (to make it rw subvolume without Recieve UUID) btrfs subvolume snap -r rw_yymmddhhmm to mysql_yymmddhhmm ( to make ro snapshot) after this just send full for the first ro snapshot (mysql_201708070830) and send incremental until the end (mysql_201708130830). [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708070830 rw_201708070830 Create a snapshot of 'mbroken_201708070830' in './rw_201708070830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 568 top level 5 path mbroken_201708080830 ID 319 gen 569 top level 5 path mbroken_201708090830 ID 320 gen 570 top level 5 path mbroken_201708100830 ID 321 gen 571 top level 5 path mbroken_201708110830 ID 322 gen 572 top level 5 path mbroken_201708120830 ID 323 gen 573 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708080830 rw_201708080830 Create a snapshot of 'mbroken_201708080830' in './rw_201708080830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708090830 rw_201708090830 Create a snapshot of 'mbroken_201708090830' in './rw_201708090830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708100830 rw_201708100830 Create a snapshot of 'mbroken_201708100830' in './rw_201708100830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708110830 rw_201708110830 Create a snapshot of 'mbroken_201708110830' in './rw_201708110830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708120830 rw_201708120830 Create a snapshot of 'mbroken_201708120830' in './rw_201708120830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708130830 rw_201708130830 Create a snapshot of 'mbroken_201708130830' in './rw_201708130830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 577 top level 5 path mbroken_201708080830 ID 319 gen 578 top level 5 path mbroken_201708090830 ID 320 gen 579 top level 5 path mbroken_201708100830 ID 321 gen 580 top level 5 path mbroken_201708110830 ID 322 gen 581 top level 5 path mbroken_201708120830 ID 323 gen 582 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 ID 349 gen 577 top level 5 path rw_201708080830 ID 350 gen 578 top level 5 path rw_201708090830 ID 351 gen 579 top level 5 path rw_201708100830 ID 352 gen 580 top level 5 path rw_201708110830 ID 353 gen 581 top level 5 path rw_201708120830 ID 354 gen 582 top level 5 path rw_201708130830 [root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep "3ad0334a-4063-654c-add6-b1cbcdeaa639" ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708070830 ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708080830 ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708090830 ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708100830 ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708110830 ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708120830 ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708130830 [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708070830 mysql_201708070830 Create a readonly snapshot of 'rw_201708070830' in './mysql_201708070830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708080830 mysql_201708080830 Create a readonly snapshot of 'rw_201708080830' in './mysql_201708080830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708090830 mysql_201708090830 Create a readonly snapshot of 'rw_201708090830' in './mysql_201708090830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708100830 mysql_201708100830 Create a readonly snapshot of 'rw_201708100830' in './mysql_201708100830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708110830 mysql_201708110830 Create a readonly snapshot of 'rw_201708110830' in './mysql_201708110830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708120830 mysql_201708120830 Create a readonly snapshot of 'rw_201708120830' in './mysql_201708120830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_
Re: btrfs issue with mariadb incremental backup
From: siranee...@tpc.co.th -- Sent: 2017-08-13 - 16:00
> Hi "A L",
>
> As your suggestion. Does it mean I have to make all my current subvolumes to
> clear
> "Receive UUID" and send them with full snapshots?
You need to clear all subvolumes with 'Received UUID' in order to use
send-receive safely. once you have done that you can start over with new
incremental sends (after the initial full send).
I suggest to remove any existing snapshots of those volumes to avoid confusion.
>>> mysql
>>> mysql_201708060830
>>> mysql_201708070830
>>> mysql_201708080830
>>> mysql_201708090830
>>> mysql_201708100830
>>> mysql_201708110830
>>> mysql_201708120830
>>> mysql_201708130830
>
> The next day that I take snapshot with the script will be
> mysql_201708140830
>
> which will be sent with incremental from mysql_201708130830 to
> mysql_201708140830
>
> Is this correct?
yes, but your current snapshots can't be used because of the 'Received UUID'.
>
> I am not understand with the url that you gave me which said
btrbk is a backup tool for btrfs. It detects the problem of the 'Received
UUID'. The FAQ explains how to remove the 'Received UUID' from the source
volumes and snapshots.
It is safer to remove all existing snapshots then trying to fix them and
risking the integrity of the data.
>
> "I'm getting an error: Aborted: "Received UUID" is set
>
> You probably restored a backup with send-receive, and made it read/write
> using btrfs
> property set. This is bad, as all snapshots and backups will inherit this
> identical
> "Received UUID", which results in all these subvolumes will be treated as
> "containing same data".
>
> To fix this, create a "proper" snapshot:
>
> - This is as your suggestion for the subvolume "mysql"
>
> # cd /mnt/btr_pool
> # mv mysubvolume mysubvolume.broken
> # btrfs subvolume snapshot mysubvolume.broken mysubvolume
>
> Now, mysubvolume should have an empty "Received UUID". Note that in order to
> have a
> clean environment, you also need to fix all subvolumes (snapshots as well as
> backups) that you created with the broken subvolume.
>
> Check if there are more broken subvolumes:
>
> # btrfs subvolume show mysubvolume.broken
> # btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above>
> # btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from
> above>
>
> - This guide seem that I have to clear <"Received UUID" > only the subvolume
> "mysql"
> and the others ("mysql_201708070830" should using btrfs subvolume snapshot -r
> instead of btrfs subvolume snapshot. Is this correct?
>
> Now clean all subvolume listed (same as above, but using btrfs subvolume
> snapshot -r
> now). Then delete all the broken subvolumes:
>
> # btrfs subvolume delete *.broken
>
> Finally, you should have a clean environment, and btrbk will not complain any
> more.
>
>
> Best Regards,
> Siranee Jaraswachirakul.
>
>>
>>
>> On 8/13/2017 12:52 PM, siranee...@tpc.co.th wrote:
>>> Hi "A L",
>>>
>>> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql
>>> /var/lib/mariadb/mysql
>>> Name: mysql
>>> UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44
>>> Parent UUID:-
>>> Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639
>>> Creation time: 2017-06-21 13:27:41 +0700
>>> Subvolume ID: 257
>>> Generation: 539
>>> Gen at creation:9
>>> Parent ID: 5
>>> Top level ID: 5
>>> Flags: -
>>> Snapshot(s):
>>> mysql_201708060830
>>> mysql_201708070830
>>> mysql_201708080830
>>> mysql_201708090830
>>> mysql_201708100830
>>> mysql_201708110830
>>> mysql_201708120830
>>> mysql_201708130830
>>>
>>> yes I think it has Received UUID because I restored the source from snapshot
>>> mysql_201708040830 for prove that the local snapshot was work.
>>>
>>> How to clear the Received UUID ?
>>> What to do next?
>> You need to make a read-write snapshot of /var/lib/mariadb/mysql and
>> then remove the old subvolume and all its snapshots.
>>
>> Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md
>>
>> # cd /mnt/btr_pool
>> # mv mysubvolume mysubvolume.broken
>> # btrfs subvolume snapshot mysubvolume.broken mysubvolume
>>
>> You can do the same with each of your snaps
Re: btrfs issue with mariadb incremental backup
On Sun, Aug 13, 2017 at 4:52 AM, wrote: > Hi "A L", > > [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql > /var/lib/mariadb/mysql > Name: mysql > UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 > Parent UUID:- > Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 > Creation time: 2017-06-21 13:27:41 +0700 > Subvolume ID: 257 > Generation: 539 > Gen at creation:9 > Parent ID: 5 > Top level ID: 5 > Flags: - > Snapshot(s): > mysql_201708060830 > mysql_201708070830 > mysql_201708080830 > mysql_201708090830 > mysql_201708100830 > mysql_201708110830 > mysql_201708120830 > mysql_201708130830 > > yes I think it has Received UUID because I restored the source from snapshot > mysql_201708040830 for prove that the local snapshot was work. > > How to clear the Received UUID ? > What to do next? I'm using btrfs-progs 4.12, and I just did a btrfs send/receive from file system A to B; and then on B I took a rw snapshot of the ro snapshot. The ro snapshot has Received UUID, but the rw snapshot made from it does not have Received UUID. So now I'm confused how you have a rw snapshot with Received UUID set. Did you ever use 'btrfs property set' to change a ro snapshot to rw? That's the only way I can think it's possible. The only other thing is maybe the behavior has changed since btrfs-progs 4.4. It is possible this behavior has changed since btrfs-progs 4.4. The changelogs shows there are improvements in send/receive in particular 4.8.3 and 4.8.4, but also others. I have no idea which are related. But anyway, this is one of the reasons why the expert users on this almost always say use something newer because it's just too hard to remember, even with changelogs, what's fixed and where. My suggestion is to investigate moving to kernel 4.9 series; and it should be safe to move to btrfs-progs 4.12 now. Any new features in 4.12 that can't be supported with an older kernel should fail gracefully. No matter what, you must have separate backups. You can't depend exclusively on one backup, no matter the method. It's fine to use send/receive for backup1. But backup2 should be a conventional backup, e.g. XFS with rsync. Basically this is a game of risk and the more independent backups you have, the more failure cases (especially user error) you can recover from, at the expense of some complexity. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
On Sun, Aug 13, 2017 at 4:49 AM, wrote: > [root@backuplogC7 ~]# btrfs send /var/lib/mariadb/mysql_201708090830 | ssh > 192.168.45.166 btrfs receive /var/lib/mariadb > At subvol /var/lib/mariadb/mysql_201708090830 > At subvol mysql_201708090830 > [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ > root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/ > sending incremental file list > ./ > > sent 3773 bytes received 19 bytes 842.67 bytes/sec > total size is 718361496 speedup is 189441.32 (DRY RUN) OK so the full send is sane. That suggests the file systems on both sides are OK, and that the problem is related strictly to incremental send/receive. It suggests that in a critical way either an origin parent or its copy on the destination are not identical. Why will be hard to figure out, and I'm not even sure it's worth it. Right now you basically have a work around: a known good full send, you can start doing incremental send/receive again using this full send/receive snapshot as the parent (with -p). Where confusion can happen is if you have to restore a snapshot, I think you must commit to a whole new canonical tree with the reverse send/receive being a full one, NOT incremental. I'm asserting this, I do not know if it is true, I don't know if the tools should prevent reverse incremental send/receive, but the very idea of reversing an incremental send receive to me just seems logically problematic. Any time you reverse the send/receive direction it must be full. I think. I did once get into trouble with this myself, having to do a restore from backup, to a new file system. I then took a rw snapshot of it, and made that the canonical live modify subvolume. I then ro snapshotted it, and tried to do an incremental send *back* to backup and it failed with an error I don't remember but I think it was later send/receive versions than 4.4 where there's some extract sanity checking to prevent this. And I was able to work around it with -c (clone) option. Anyway, I got sufficiently confused myself after all of that, that I pretty much gave up on such a strategy, blew away all the ro snapshots on both sides, and started from scratch with a new full send, and then resumed the incrementals unidirectionally. And anytime they are reversed I assume it must be a full send. Anyway, I think it's perilous to reverse directions while also trying to do incremental send/receive without a lot of testing to thoroughly understand what's going on. And I basically got to fuck this, not worth the complexity. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
Hi "A L",
As your suggestion. Does it mean I have to make all my current subvolumes to
clear
"Receive UUID" and send them with full snapshots?
>> mysql
>> mysql_201708060830
>> mysql_201708070830
>> mysql_201708080830
>> mysql_201708090830
>> mysql_201708100830
>> mysql_201708110830
>> mysql_201708120830
>> mysql_201708130830
The next day that I take snapshot with the script will be
mysql_201708140830
which will be sent with incremental from mysql_201708130830 to
mysql_201708140830
Is this correct?
I am not understand with the url that you gave me which said
"I'm getting an error: Aborted: "Received UUID" is set
You probably restored a backup with send-receive, and made it read/write using
btrfs
property set. This is bad, as all snapshots and backups will inherit this
identical
"Received UUID", which results in all these subvolumes will be treated as
"containing same data".
To fix this, create a "proper" snapshot:
- This is as your suggestion for the subvolume "mysql"
# cd /mnt/btr_pool
# mv mysubvolume mysubvolume.broken
# btrfs subvolume snapshot mysubvolume.broken mysubvolume
Now, mysubvolume should have an empty "Received UUID". Note that in order to
have a
clean environment, you also need to fix all subvolumes (snapshots as well as
backups) that you created with the broken subvolume.
Check if there are more broken subvolumes:
# btrfs subvolume show mysubvolume.broken
# btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above>
# btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above>
- This guide seem that I have to clear <"Received UUID" > only the subvolume
"mysql"
and the others ("mysql_201708070830" should using btrfs subvolume snapshot -r
instead of btrfs subvolume snapshot. Is this correct?
Now clean all subvolume listed (same as above, but using btrfs subvolume
snapshot -r
now). Then delete all the broken subvolumes:
# btrfs subvolume delete *.broken
Finally, you should have a clean environment, and btrbk will not complain any
more.
Best Regards,
Siranee Jaraswachirakul.
>
>
> On 8/13/2017 12:52 PM, siranee...@tpc.co.th wrote:
>> Hi "A L",
>>
>> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql
>> /var/lib/mariadb/mysql
>> Name: mysql
>> UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44
>> Parent UUID:-
>> Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639
>> Creation time: 2017-06-21 13:27:41 +0700
>> Subvolume ID: 257
>> Generation: 539
>> Gen at creation:9
>> Parent ID: 5
>> Top level ID: 5
>> Flags: -
>> Snapshot(s):
>> mysql_201708060830
>> mysql_201708070830
>> mysql_201708080830
>> mysql_201708090830
>> mysql_201708100830
>> mysql_201708110830
>> mysql_201708120830
>> mysql_201708130830
>>
>> yes I think it has Received UUID because I restored the source from snapshot
>> mysql_201708040830 for prove that the local snapshot was work.
>>
>> How to clear the Received UUID ?
>> What to do next?
> You need to make a read-write snapshot of /var/lib/mariadb/mysql and
> then remove the old subvolume and all its snapshots.
>
> Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md
>
> # cd /mnt/btr_pool
> # mv mysubvolume mysubvolume.broken
> # btrfs subvolume snapshot mysubvolume.broken mysubvolume
>
> You can do the same with each of your snapshots too, and send them as
> full snapshots (without -p).
>
> ~A
>
>> Best Regards,
>> Siranee Jaraswachirakul.
>>
>>> Have you checked that there is no Received UUID on the source subvolume?
>>>
>>> # btrfs sub show volume/mysql/
>>> volume/mysql
>>> Name: mysql
>>> UUID: 8a94524e-a956-c14b-bb8d-d453627f27d5
>>> Parent UUID: -
>>> Received UUID: -
>>> Creation time: 2017-04-17 11:46:20 +0200
>>> Subvolume ID: 1469
>>> Generation: 122934
>>> Gen at creation: 78671
>>> Parent ID: 5
>>> Top level ID: 5
>>> Flags: -
>>> Snapshot(s):
>>>
>>> There is no Received UUID here. If it has, then btrfs send-receive will
>>> have problems, sin
Re: btrfs issue with mariadb incremental backup
On 8/13/2017 12:52 PM, siranee...@tpc.co.th wrote: Hi "A L", [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql /var/lib/mariadb/mysql Name: mysql UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 Parent UUID:- Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 Creation time: 2017-06-21 13:27:41 +0700 Subvolume ID: 257 Generation: 539 Gen at creation:9 Parent ID: 5 Top level ID: 5 Flags: - Snapshot(s): mysql_201708060830 mysql_201708070830 mysql_201708080830 mysql_201708090830 mysql_201708100830 mysql_201708110830 mysql_201708120830 mysql_201708130830 yes I think it has Received UUID because I restored the source from snapshot mysql_201708040830 for prove that the local snapshot was work. How to clear the Received UUID ? What to do next? You need to make a read-write snapshot of /var/lib/mariadb/mysql and then remove the old subvolume and all its snapshots. Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md # cd /mnt/btr_pool # mv mysubvolume mysubvolume.broken # btrfs subvolume snapshot mysubvolume.broken mysubvolume You can do the same with each of your snapshots too, and send them as full snapshots (without -p). ~A Best Regards, Siranee Jaraswachirakul. Have you checked that there is no Received UUID on the source subvolume? # btrfs sub show volume/mysql/ volume/mysql Name: mysql UUID: 8a94524e-a956-c14b-bb8d-d453627f27d5 Parent UUID: - Received UUID: - Creation time: 2017-04-17 11:46:20 +0200 Subvolume ID: 1469 Generation: 122934 Gen at creation: 78671 Parent ID: 5 Top level ID: 5 Flags: - Snapshot(s): There is no Received UUID here. If it has, then btrfs send-receive will have problems, since all snapshots of the source subvolume will have the same Received UUID and it can't tell the differences between the snapshots. On 8/13/2017 5:40 AM, siranee...@tpc.co.th wrote: Hi Chris, I started as your suggestion again. The diff occured since snapshot mysql_201708090830 manually send. What should I do next? - delete all the bad/mismatching snapshots only on the destination computer. [root@joytest ~]# date Sun Aug 13 10:27:23 ICT 2017 [root@joytest ~]# cd /var/lib/mariadb [root@joytest mariadb]# btrfs sub list . ID 313 gen 220 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 318 gen 205 top level 5 path mysql_201708090830 ID 320 gen 211 top level 5 path mysql_201708100830 ID 322 gen 219 top level 5 path mysql_201708110830 ID 323 gen 219 top level 5 path mysql_201708120830 ID 324 gen 224 top level 5 path mysql_201708130830 ID 325 gen 225 top level 5 path mysql [root@joytest mariadb]# btrfs sub del mysql_201708130830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' [root@joytest mariadb]# btrfs sub del mysql_201708120830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' [root@joytest mariadb]# btrfs sub del mysql_201708110830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' [root@joytest mariadb]# btrfs sub del mysql_201708100830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' [root@joytest mariadb]# btrfs sub del mysql_201708090830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' [root@joytest mariadb]# btrfs sub sync . [root@joytest mariadb]# systemctl status mariadb â— mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min ago Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE) Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS) Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS) Main PID: 19870 (code=exited, status=0/SUCCESS) Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mariadb/mysql Aug 13 09:07:00 joyte
Re: btrfs issue with mariadb incremental backup
Hi "A L", [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql /var/lib/mariadb/mysql Name: mysql UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 Parent UUID:- Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 Creation time: 2017-06-21 13:27:41 +0700 Subvolume ID: 257 Generation: 539 Gen at creation:9 Parent ID: 5 Top level ID: 5 Flags: - Snapshot(s): mysql_201708060830 mysql_201708070830 mysql_201708080830 mysql_201708090830 mysql_201708100830 mysql_201708110830 mysql_201708120830 mysql_201708130830 yes I think it has Received UUID because I restored the source from snapshot mysql_201708040830 for prove that the local snapshot was work. How to clear the Received UUID ? What to do next? Best Regards, Siranee Jaraswachirakul. > Have you checked that there is no Received UUID on the source subvolume? > > # btrfs sub show volume/mysql/ > volume/mysql > Name: mysql > UUID: 8a94524e-a956-c14b-bb8d-d453627f27d5 > Parent UUID: - > Received UUID: - > Creation time: 2017-04-17 11:46:20 +0200 > Subvolume ID: 1469 > Generation: 122934 > Gen at creation: 78671 > Parent ID: 5 > Top level ID: 5 > Flags: - > Snapshot(s): > > There is no Received UUID here. If it has, then btrfs send-receive will > have problems, since all snapshots of the source subvolume will have the > same Received UUID and it can't tell the differences between the snapshots. > > On 8/13/2017 5:40 AM, siranee...@tpc.co.th wrote: >> Hi Chris, >> >> I started as your suggestion again. The diff occured since snapshot >> mysql_201708090830 manually send. What should I do next? >> >> - delete all the bad/mismatching snapshots only on the destination computer. >> [root@joytest ~]# date >> Sun Aug 13 10:27:23 ICT 2017 >> [root@joytest ~]# cd /var/lib/mariadb >> [root@joytest mariadb]# btrfs sub list . >> ID 313 gen 220 top level 5 path mysql_201708070830 >> ID 316 gen 199 top level 5 path mysql_201708080830 >> ID 318 gen 205 top level 5 path mysql_201708090830 >> ID 320 gen 211 top level 5 path mysql_201708100830 >> ID 322 gen 219 top level 5 path mysql_201708110830 >> ID 323 gen 219 top level 5 path mysql_201708120830 >> ID 324 gen 224 top level 5 path mysql_201708130830 >> ID 325 gen 225 top level 5 path mysql >> [root@joytest mariadb]# btrfs sub del mysql_201708130830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' >> [root@joytest mariadb]# btrfs sub del mysql_201708120830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' >> [root@joytest mariadb]# btrfs sub del mysql_201708110830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' >> [root@joytest mariadb]# btrfs sub del mysql_201708100830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' >> [root@joytest mariadb]# btrfs sub del mysql_201708090830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' >> [root@joytest mariadb]# btrfs sub sync . >> [root@joytest mariadb]# systemctl status mariadb >> â mariadb.service - MariaDB database server >> Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor >> preset: >> disabled) >> Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h >> 24min >> ago >>Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID >> (code=exited, status=1/FAILURE) >>Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, >> status=0/SUCCESS) >>Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n >> (code=exited, >> status=0/SUCCESS) >> Main PID: 19870 (code=exited, status=0/SUCCESS) >> >> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... >> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe >> Logging to >> '/var/log/mariadb/mariadb.log'. >> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe >> Starting >> mysqld daemon with databases from /var/lib/mariadb/mysql >> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, >> code=exited status=1 >> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. >> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed >> state. >> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed.
Re: btrfs issue with mariadb incremental backup
Hi Chris, Try deleting mysql_201708090830/ snapshot on the destination. And resend but this time do a full send of that snapshot, don't use -p. I wonder if a full send, rather than incremental makes a difference. Follow it up with the rsync command to compare origin and destination. Yes, It's different. [root@backuplogC7 ~]# btrfs send /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708090830 At subvol mysql_201708090830 [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 842.67 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) Best Regards, Siranee Jaraswachirakul > On Sat, Aug 12, 2017 at 9:40 PM, wrote: > >> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ >> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ >> sending incremental file list >> ./ >> ib_logfile1 >> ibdata1 >> >> sent 3779 bytes received 25 bytes 507.20 bytes/sec >> total size is 718361496 speedup is 188843.72 (DRY RUN) > > > OK so I don't think this can be a sync related problem. That snapshot > has been committed to disk days ago. There's definitely something > wrong with the incremental send/receive, but it's unclear whether this > is a kernel bug (send side) or btrfs-progs (receive side), or if > there's any chance of file system corruption/confusion happening with > either of the two subvolumes on the origin or the subvolume (parent) > on the destination. > > So that means you're really in the weeds on what to do next. > > Try deleting mysql_201708090830/ snapshot on the destination. And > resend but this time do a full send of that snapshot, don't use -p. I > wonder if a full send, rather than incremental makes a difference. > Follow it up with the rsync command to compare origin and destination. > > > > -- > Chris Murphy > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
Have you checked that there is no Received UUID on the source subvolume? # btrfs sub show volume/mysql/ volume/mysql Name: mysql UUID: 8a94524e-a956-c14b-bb8d-d453627f27d5 Parent UUID: - Received UUID: - Creation time: 2017-04-17 11:46:20 +0200 Subvolume ID: 1469 Generation: 122934 Gen at creation: 78671 Parent ID: 5 Top level ID: 5 Flags: - Snapshot(s): There is no Received UUID here. If it has, then btrfs send-receive will have problems, since all snapshots of the source subvolume will have the same Received UUID and it can't tell the differences between the snapshots. On 8/13/2017 5:40 AM, siranee...@tpc.co.th wrote: Hi Chris, I started as your suggestion again. The diff occured since snapshot mysql_201708090830 manually send. What should I do next? - delete all the bad/mismatching snapshots only on the destination computer. [root@joytest ~]# date Sun Aug 13 10:27:23 ICT 2017 [root@joytest ~]# cd /var/lib/mariadb [root@joytest mariadb]# btrfs sub list . ID 313 gen 220 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 318 gen 205 top level 5 path mysql_201708090830 ID 320 gen 211 top level 5 path mysql_201708100830 ID 322 gen 219 top level 5 path mysql_201708110830 ID 323 gen 219 top level 5 path mysql_201708120830 ID 324 gen 224 top level 5 path mysql_201708130830 ID 325 gen 225 top level 5 path mysql [root@joytest mariadb]# btrfs sub del mysql_201708130830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' [root@joytest mariadb]# btrfs sub del mysql_201708120830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' [root@joytest mariadb]# btrfs sub del mysql_201708110830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' [root@joytest mariadb]# btrfs sub del mysql_201708100830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' [root@joytest mariadb]# btrfs sub del mysql_201708090830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' [root@joytest mariadb]# btrfs sub sync . [root@joytest mariadb]# systemctl status mariadb â— mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min ago Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE) Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS) Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS) Main PID: 19870 (code=exited, status=0/SUCCESS) Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mariadb/mysql Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, code=exited status=1 Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state. Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed. [root@joytest mariadb]# btrfs sub list . ID 313 gen 220 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 325 gen 225 top level 5 path mysql [root@joytest mariadb]# - The most recent good snapshot pair, which rsync shows origin and destination match, is mysql_201708080830 so you can keep that one on both sides. [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb ID 257 gen 538 top level 5 path mysql ID 316 gen 498 top level 5 path mysql_201708060830 ID 317 gen 503 top level 5 path mysql_201708070830 ID 318 gen 507 top level 5 path mysql_201708080830 ID 319 gen 514 top level 5 path mysql_201708090830 ID 320 gen 524 top level 5 path mysql_201708100830 ID 321 gen 529 top level 5 path mysql_201708110830 ID 322 gen 533 top level 5 path mysql_201708120830 ID 323 gen 538 top level 5 path mysql_201708130830 [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ root@192.168.45.166://var/lib/mariadb/mysql_201708070830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 842.67 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ root@192.168.45.166://var/lib/mariadb/mysql_201708080830/ sending incremental file list ./ sent 3769 bytes received 19 bytes 841.78 bytes/sec total size is 718361496 speedup is 189641.37 (DRY RUN) [root@backuplogC7 ~]# date Sun Aug 13 10:34:05 ICT 2017
Re: btrfs issue with mariadb incremental backup
On Sat, Aug 12, 2017 at 9:40 PM, wrote: > [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ > root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ > sending incremental file list > ./ > ib_logfile1 > ibdata1 > > sent 3779 bytes received 25 bytes 507.20 bytes/sec > total size is 718361496 speedup is 188843.72 (DRY RUN) OK so I don't think this can be a sync related problem. That snapshot has been committed to disk days ago. There's definitely something wrong with the incremental send/receive, but it's unclear whether this is a kernel bug (send side) or btrfs-progs (receive side), or if there's any chance of file system corruption/confusion happening with either of the two subvolumes on the origin or the subvolume (parent) on the destination. So that means you're really in the weeds on what to do next. Try deleting mysql_201708090830/ snapshot on the destination. And resend but this time do a full send of that snapshot, don't use -p. I wonder if a full send, rather than incremental makes a difference. Follow it up with the rsync command to compare origin and destination. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
Hi Chris, I started as your suggestion again. The diff occured since snapshot mysql_201708090830 manually send. What should I do next? - delete all the bad/mismatching snapshots only on the destination computer. [root@joytest ~]# date Sun Aug 13 10:27:23 ICT 2017 [root@joytest ~]# cd /var/lib/mariadb [root@joytest mariadb]# btrfs sub list . ID 313 gen 220 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 318 gen 205 top level 5 path mysql_201708090830 ID 320 gen 211 top level 5 path mysql_201708100830 ID 322 gen 219 top level 5 path mysql_201708110830 ID 323 gen 219 top level 5 path mysql_201708120830 ID 324 gen 224 top level 5 path mysql_201708130830 ID 325 gen 225 top level 5 path mysql [root@joytest mariadb]# btrfs sub del mysql_201708130830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' [root@joytest mariadb]# btrfs sub del mysql_201708120830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' [root@joytest mariadb]# btrfs sub del mysql_201708110830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' [root@joytest mariadb]# btrfs sub del mysql_201708100830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' [root@joytest mariadb]# btrfs sub del mysql_201708090830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' [root@joytest mariadb]# btrfs sub sync . [root@joytest mariadb]# systemctl status mariadb â mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min ago Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE) Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS) Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS) Main PID: 19870 (code=exited, status=0/SUCCESS) Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mariadb/mysql Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, code=exited status=1 Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state. Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed. [root@joytest mariadb]# btrfs sub list . ID 313 gen 220 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 325 gen 225 top level 5 path mysql [root@joytest mariadb]# - The most recent good snapshot pair, which rsync shows origin and destination match, is mysql_201708080830 so you can keep that one on both sides. [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb ID 257 gen 538 top level 5 path mysql ID 316 gen 498 top level 5 path mysql_201708060830 ID 317 gen 503 top level 5 path mysql_201708070830 ID 318 gen 507 top level 5 path mysql_201708080830 ID 319 gen 514 top level 5 path mysql_201708090830 ID 320 gen 524 top level 5 path mysql_201708100830 ID 321 gen 529 top level 5 path mysql_201708110830 ID 322 gen 533 top level 5 path mysql_201708120830 ID 323 gen 538 top level 5 path mysql_201708130830 [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ root@192.168.45.166://var/lib/mariadb/mysql_201708070830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 842.67 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ root@192.168.45.166://var/lib/mariadb/mysql_201708080830/ sending incremental file list ./ sent 3769 bytes received 19 bytes 841.78 bytes/sec total size is 718361496 speedup is 189641.37 (DRY RUN) [root@backuplogC7 ~]# date Sun Aug 13 10:34:05 ICT 2017 [root@backuplogC7 ~]# - manually do incremental send/receive, starting with mysql_201708090830/, to make the destination current again with the origin. [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708090830 At snapshot mysql_201708090830 [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ sending incremental file list ./ ib_logfile1 ibdata1 sent 3779 bytes received 25 bytes 507.20 bytes/sec total size is 718361496 speedup is 188843.72 (DRY RUN) [root@backuplogC7 ~]# Best Regards, Siranee Jaraswachirakul. > On Sat, Aug 12, 2017 at 8:20 PM, wrote: > >> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 >> ro
Re: btrfs issue with mariadb incremental backup
On Sat, Aug 12, 2017 at 8:20 PM, wrote: > [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 > root@192.168.45.166://var/lib/mariadb/mysql_201708090830 You need trailing / for the first directory with -a option. rsync -a dir dir is not the same command as rsync -a dir/ dir It's confusing but your command is trying to create mysql_201708090830 directory on the source, in the mysql_201708090830 on the destination. That is why everything mismatches. To make it mean "contents of" you need trailing slash on at least the origin. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
Hi Chris, I did as you suggest and the result was bad then I decided to start over with snapshot mysql_201708070830 and manually send incremental the result rsync always said "a log diff" but the dest can start mariadb until snapshot "mysql_201708100830" it couldn't start mariadb. The following are the result. - delete all the bad/mismatching snapshots only on the destination computer. tpcorp@virtualtrust3:~$ lxc exec joytest -- bash [root@joytest ~]# btrfs sub list /var/lib/mariadb ID 298 gen 177 top level 5 path mysql_201708070830 ID 301 gen 148 top level 5 path mysql_201708080830 ID 303 gen 156 top level 5 path mysql_201708090830 ID 305 gen 162 top level 5 path mysql_201708100830 ID 309 gen 175 top level 5 path mysql_201708110830 ID 310 gen 176 top level 5 path mysql ID 311 gen 180 top level 5 path mysql_201708120830 [root@joytest ~]# cd /var/lib/mariadb [root@joytest mariadb]# btrfs sub list . ID 298 gen 177 top level 5 path mysql_201708070830 ID 301 gen 148 top level 5 path mysql_201708080830 ID 303 gen 156 top level 5 path mysql_201708090830 ID 305 gen 162 top level 5 path mysql_201708100830 ID 309 gen 175 top level 5 path mysql_201708110830 ID 310 gen 176 top level 5 path mysql ID 311 gen 180 top level 5 path mysql_201708120830 [root@joytest mariadb]# btrfs sub del mysql_201708090830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' [root@joytest mariadb]# btrfs sub del mysql_201708100830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' [root@joytest mariadb]# btrfs sub del mysql_201708110830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' [root@joytest mariadb]# btrfs sub del mysql_201708120830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' [root@joytest mariadb]# btrfs sub sync . - The most recent good snapshot pair, which rsync shows origin and destination match, is mysql_201708080830 so you can keep that one on both sides. [root@joytest mariadb]# btrfs sub list . ID 298 gen 177 top level 5 path mysql_201708070830 ID 301 gen 148 top level 5 path mysql_201708080830 ID 310 gen 176 top level 5 path mysql - manually do incremental send/receive, starting with mysql_201708090830/, to make the destination current again with the origin. tpcorp@virtualtrust3:~$ lxc exec backuplogC7 -- bash [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb ID 257 gen 537 top level 5 path mysql ID 316 gen 498 top level 5 path mysql_201708060830 ID 317 gen 503 top level 5 path mysql_201708070830 ID 318 gen 507 top level 5 path mysql_201708080830 ID 319 gen 514 top level 5 path mysql_201708090830 ID 320 gen 524 top level 5 path mysql_201708100830 ID 321 gen 529 top level 5 path mysql_201708110830 ID 322 gen 533 top level 5 path mysql_201708120830 [root@backuplogC7 ~]# more /var/log/btrfs_send.log Start Send 201708040905 btrfs send -p /var/lib/mariadb/mysql_201708030830 /var/lib/mariadb/mysql_201708040830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708040830 Stop Send 201708040905 Start Send 201708050905 btrfs send -p /var/lib/mariadb/mysql_201708040830 /var/lib/mariadb/mysql_201708050830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708050830 Stop Send 201708050905 Start Send 201708060905 btrfs send -p /var/lib/mariadb/mysql_201708050830 /var/lib/mariadb/mysql_201708060830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708060830 Stop Send 201708060905 Start Send 201708070905 btrfs send -p /var/lib/mariadb/mysql_201708060830 /var/lib/mariadb/mysql_201708070830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708070830 Stop Send 201708070905 Start Send 201708080905 btrfs send -p /var/lib/mariadb/mysql_201708070830 /var/lib/mariadb/mysql_201708080830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708080830 Stop Send 201708080905 Start Send 201708090905 btrfs send -p /var/lib/mariadb/mysql_201708080830 /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708090830 Stop Send 201708090905 Start Send 201708100905 btrfs send -p /var/lib/mariadb/mysql_201708090830 /var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708100830 Stop Send 201708100905 Start Send 201708110905 btrfs send -p /var/lib/mariadb/mysql_201708100830 /var/lib/mariadb/mysql_201708110830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708110830 Stop Send 201708110905 Start Send 201708120905 btrfs send -p /var/lib/mariadb/mysql_201708110830 /var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708120830 Stop Send 201708120905 [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708090830 At snapshot mysql_201708090830
Re: btrfs issue with mariadb incremental backup
On Sat, Aug 12, 2017 at 4:49 PM, Hugo Mills wrote: > On Sat, Aug 12, 2017 at 03:34:01PM -0600, Chris Murphy wrote: >> On Fri, Aug 11, 2017 at 11:08 PM, wrote: >> >> >> > The backup script has the btrfs sync command since Aug 3 >> >> >> From your script: >> > system btrfs sub snap -r $basepath $snappath >> > system btrfs sub sync $basepath >> >> From the man page: sync [subvolid...] >>Wait until given subvolume(s) are completely removed from the >>filesystem after deletion. >> >> >> This 'subvolume sync' command, per the man page, is only about >> subvolume deletion. I suggest replacing it with a regular sync >> command. >> >> I think the problem is that the script does things so fast that the >> snapshot is not always consistent on disk before btrfs send starts. >> It's just a guess though. If I'm right, this means the rsync mismaches >> mean the destination snapshots are bad. Here's what I would do: > >I don't see how that can happen. Snapshots are atomic -- they're > either there or not there. It's not a matter even of copying the > metadata part of the subvol. It's literally just adding a pointer to > point at an existing FS tree. Do snapshots come with sync and flush to disk? Or does it just set a checkpoint and the extents that are as yet uncommitted prior to the snapshot aren't for sure on disk yet until the next commit time or sync? Nevertheless the wiki explicitly says to do sync after taking a snapshot in the context of send/receive. https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Initial_Bootstrapping And people on the list have had this "stale NFS file handle" error . I have no idea if it's still a problem with current kernels, or if it applies to 4.4 kernel. Anyway, the OP does appear to be having a real problem. rync is very clearly showing a given snapshot with incremental send/receive differ by *a lot* on source and destination machines. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
On Sat, Aug 12, 2017 at 4:41 PM, Janos Toth F. wrote: > On Sat, Aug 12, 2017 at 11:34 PM, Chris Murphy > wrote: >> On Fri, Aug 11, 2017 at 11:08 PM, wrote: >> >> I think the problem is that the script does things so fast that the >> snapshot is not always consistent on disk before btrfs send starts. >> It's just a guess though. If I'm right, this means the rsync mismaches >> mean the destination snapshots are bad. > > Hmm. I was wondering about this exact issue the other day when I > fiddled with my own backup script. > - Should I issue a sync between creating a snapshot and starting to > rsync (or send/receive) the content of that fresh snapshot to an > external backup storage? A sync won't hurt, but I think we need btrfs send to include sync if this is still required as stated by the Wiki. And if the problem stated in the Wiki has been fixed, then it'd be useful to know when, but a git log search doesn't reveal anything related to NFS and Btrfs. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
On Sat, Aug 12, 2017 at 03:34:01PM -0600, Chris Murphy wrote: > On Fri, Aug 11, 2017 at 11:08 PM, wrote: > > > > The backup script has the btrfs sync command since Aug 3 > > > From your script: > > system btrfs sub snap -r $basepath $snappath > > system btrfs sub sync $basepath > > From the man page: sync [subvolid...] >Wait until given subvolume(s) are completely removed from the >filesystem after deletion. > > > This 'subvolume sync' command, per the man page, is only about > subvolume deletion. I suggest replacing it with a regular sync > command. > > I think the problem is that the script does things so fast that the > snapshot is not always consistent on disk before btrfs send starts. > It's just a guess though. If I'm right, this means the rsync mismaches > mean the destination snapshots are bad. Here's what I would do: I don't see how that can happen. Snapshots are atomic -- they're either there or not there. It's not a matter even of copying the metadata part of the subvol. It's literally just adding a pointer to point at an existing FS tree. Hugo. -- Hugo Mills | If it's December 1941 in Casablanca, what time is it hugo@... carfax.org.uk | in New York? http://carfax.org.uk/ | PGP: E2AB1DE4 | Rick Blaine, Casablanca signature.asc Description: Digital signature
Re: btrfs issue with mariadb incremental backup
On Sat, Aug 12, 2017 at 11:34 PM, Chris Murphy wrote: > On Fri, Aug 11, 2017 at 11:08 PM, wrote: > > I think the problem is that the script does things so fast that the > snapshot is not always consistent on disk before btrfs send starts. > It's just a guess though. If I'm right, this means the rsync mismaches > mean the destination snapshots are bad. Hmm. I was wondering about this exact issue the other day when I fiddled with my own backup script. - Should I issue a sync between creating a snapshot and starting to rsync (or send/receive) the content of that fresh snapshot to an external backup storage? I dismissed the thought because I figured rsync should see the proper state regardless if some data is still waiting in the system write-back cache. Now I am confused again. Is it only for send/receive? (I don't use that feature yet but thought about switching to it from rsync.) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
On Fri, Aug 11, 2017 at 11:08 PM, wrote: > The backup script has the btrfs sync command since Aug 3 >From your script: > system btrfs sub snap -r $basepath $snappath > system btrfs sub sync $basepath >From the man page: sync [subvolid...] Wait until given subvolume(s) are completely removed from the filesystem after deletion. This 'subvolume sync' command, per the man page, is only about subvolume deletion. I suggest replacing it with a regular sync command. I think the problem is that the script does things so fast that the snapshot is not always consistent on disk before btrfs send starts. It's just a guess though. If I'm right, this means the rsync mismaches mean the destination snapshots are bad. Here's what I would do: - delete all the bad/mismatching snapshots only on the destination computer. - he most recent good snapshot pair, which rsync shows origin and destination match, is mysql_201708080830 so you can keep that one on both sides. - manually do incremental send/receive, starting with mysql_201708090830/, to make the destination current again with the origin. - confirm with rsync that the snapshot pairs on origin and destination are the same - now resume using the modified script, which will do snapshot -> sync -> send. OPTIONAL, you could add to your script an rsync -avnc to double check that the incremental send receive is working. This is admittedly inefficient because it checks the *entire* contents of the snapshots on both sides, it's not just checking the incremental data. But if it doesn't take too long, it will help restore trust in send/receive, and confirm that a regular sync is needed in between snapshot and send. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
Hi Chris,
Sorry for the misunderstanding and inappropriate attached the file (I thought
that
too long text in the mail message so I used the attached file).
The backup script has the btrfs sync command since Aug 3
I sent the following rsync command as you said (sorry for the misunderstanding
with
the rsync command that I'd never used before)
The snapshot has been different since snapshot mysql_201708090830 but the
mariadb
can recovery and start until mysql_201708110830 seems too much differences that
can
not start.
[root@backuplogC7 ~]# more /root/script/backup/backupsnap.sh
#Backup
#
user=$1
password=$2
basepath=$3
datet=$(date +%Y%m%d%H%M)
snappath=${basepath}_${datet}
echo "Locking databases ${datet}"
mysql -u$user -p$password << EOF
FLUSH TABLES WITH READ LOCK;
system btrfs sub snap -r $basepath $snappath
system btrfs sub sync $basepath
UNLOCK TABLES;
quit
EOF
echo "Databases unlocked ${datet}"
[root@backuplogC7 ~]# ls -l /root/script/backup/backupsnap.sh
-rwxr--r-- 1 root root 333 Aug 3 09:11 /root/script/backup/backupsnap.sh
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708070830/
sending incremental file list
./
sent 3773 bytes received 19 bytes 1083.43 bytes/sec
total size is 718361496 speedup is 189441.32 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708080830/
sending incremental file list
./
sent 3769 bytes received 19 bytes 841.78 bytes/sec
total size is 718361496 speedup is 189641.37 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/
sending incremental file list
./
ib_logfile1
ibdata1
sent 3779 bytes received 25 bytes 1086.86 bytes/sec
total size is 718361496 speedup is 188843.72 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708100830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708100830/
sending incremental file list
./
ib_logfile1
ibdata1
sent 3779 bytes received 25 bytes 1086.86 bytes/sec
total size is 718361496 speedup is 188843.72 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708110830/
root@192.168.45.166:/var/lib/mariadb/mysql_20170811830/
sending incremental file list
created directory /var/lib/mariadb/mysql_20170811830
./
aria_log.0001
aria_log_control
ib_logfile0
ib_logfile1
ibdata1
mysql-bin.01
mysql-bin.index
backups_db/
backups_db/db.opt
backups_db/ints.frm
backups_db/tpc_backup_detail.frm
backups_db/tpc_backup_header.frm
backups_db/tpc_backup_history.frm
backups_db/tpc_backup_schedule_level_master.frm
backups_db/tpc_backup_schedule_master.frm
backups_db/tpc_backup_schedule_master_extra.frm
backups_db/tpc_backup_schedule_old.frm
backups_db/tpc_backup_server.frm
backups_db/tpc_backup_step.frm
backups_db/tpc_backup_user.frm
backups_db/v_backup_header_status.frm
backups_db/v_calendar.frm
backups_db/v_dup_backup.frm
mysql/
mysql/columns_priv.MYD
mysql/columns_priv.MYI
mysql/columns_priv.frm
mysql/db.MYD
mysql/db.MYI
mysql/db.frm
mysql/event.MYD
mysql/event.MYI
mysql/event.frm
mysql/func.MYD
mysql/func.MYI
mysql/func.frm
mysql/general_log.CSM
mysql/general_log.CSV
mysql/general_log.frm
mysql/help_category.MYD
mysql/help_category.MYI
mysql/help_category.frm
mysql/help_keyword.MYD
mysql/help_keyword.MYI
mysql/help_keyword.frm
mysql/help_relation.MYD
mysql/help_relation.MYI
mysql/help_relation.frm
mysql/help_topic.MYD
mysql/help_topic.MYI
mysql/help_topic.frm
mysql/host.MYD
mysql/host.MYI
mysql/host.frm
mysql/ndb_binlog_index.MYD
mysql/ndb_binlog_index.MYI
mysql/ndb_binlog_index.frm
mysql/plugin.MYD
mysql/plugin.MYI
mysql/plugin.frm
mysql/proc.MYD
mysql/proc.MYI
mysql/proc.frm
mysql/procs_priv.MYD
mysql/procs_priv.MYI
mysql/procs_priv.frm
mysql/proxies_priv.MYD
mysql/proxies_priv.MYI
mysql/proxies_priv.frm
mysql/servers.MYD
mysql/servers.MYI
mysql/servers.frm
mysql/slow_log.CSM
mysql/slow_log.CSV
mysql/slow_log.frm
mysql/tables_priv.MYD
mysql/tables_priv.MYI
mysql/tables_priv.frm
mysql/time_zone.MYD
mysql/time_zone.MYI
mysql/time_zone.frm
mysql/time_zone_leap_second.MYD
mysql/time_zone_leap_second.MYI
mysql/time_zone_leap_second.frm
mysql/time_zone_name.MYD
mysql/time_zone_name.MYI
mysql/time_zone_name.frm
mysql/time_zone_transition.MYD
mysql/time_zone_transition.MYI
mysql/time_zone_transition.frm
mysql/time_zone_transition_type.MYD
mysql/time_zone_transition_type.MYI
mysql/time_zone_transition_type.frm
mysql/user.MYD
mysql/user.MYI
mysql/user.frm
performance_schema/
performance_schema/cond_instances.frm
performance_schema/db.opt
performance_schema/events_waits_current.frm
performance_schema/events_waits_history.frm
performance_schema/events_waits_history_long.frm
performance_schema/events_waits_summary_by_instance.frm
performance_schema/events_waits_summary_by_thread_by_event_name.frm
performance_schema/events_waits_summary_global_by_event_name.frm
performance_schem
Re: btrfs issue with mariadb incremental backup
On Fri, Aug 11, 2017 at 8:38 PM, wrote: > Hi Chris, > > I explain what I have done in the attached file. That is way too long. And please don't attach files directed at me, that's not appropriate. Just message the list normally, so it can be searched by others down the road. There is a problem with your rsync command, you really don't understand what I was asking. The whole point of rsync is to compare the origin and destination subvolumes to see if they are different from each other. You ran the command on a single subvolume on a single machine which is pointless. On machine A: $ rsync -avnc /mnt/snapshot6/ chris@192.168.1.116:/mnt/snapshot6/ chris@192.168.1.116's password: sending incremental file list sent 125 bytes received 12 bytes 39.14 bytes/sec total size is 30,686 speedup is 223.99 (DRY RUN) [chris@f26h ~]$ Because no files are listed, all are confirmed to be identical. If files are listed, there's a difference. > Please suggest me what I should do next? Check this. I wonder if you're running into this obscure bug where maybe your read only snapshot is not yet synced before it's being sent. Hence it *is* changing on origin machine, but not because of other snapshots being deleted. I have no idea if your scripting handles errors like the bogus stale NFS handle error, maybe it's possible it happens but you're not seeing it? Anyway, you might try just adding a single sync right after taking the snapshot, change nothing else and see if the problem reproduces itself. https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Initial_Bootstrapping I'm pretty sure this is fixed in newer kernels, I haven't run into it in a long time myself. But I don't know when it was fixed. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
Hi Chris, I explain what I have done in the attached file. Please suggest me what I should do next? I plan to reproduce the data mannually for the daily inserted data to check the detail step by step to know exactly which steps make the btrfs send /receive result diff on the Second box (Machine Y). I will start with the first btrfs snapshot that I already had (mysql_201707210830). Do you think I should do others? Best Regards, Siranee Jaraswachirakul. > On Fri, Aug 11, 2017 at 12:00 AM, wrote: >> Sorry Chris, >> >> I forgot to send the diff from rsync -anc result >> >> the source container A start data as data on snapshot mysql_201708040830 >> >> [root@backuplogC7 tmp]# ls -l /var/lib/mariadb >> total 0 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql >> drwxrwxr-x+ 1 mysql mysql 260 Jul 12 08:29 mysql_201708040830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708050830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708060830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708070830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708080830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708090830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708100830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708110830 >> >> the destcontainer B start data as data on snapshot mysql_201708070830 >> >> [root@joytest tmp]# ls -l /var/lib/mariadb >> total 0 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 11 10:24 mysql >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708070830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708080830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708090830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708100830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708110830 >> >> >> >> tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt >> dest_mysql_201708110830.txt >> 1c1 >> < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708110830 >> --- >>> drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708110830 >> 5c5 >> < -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1 >> --- >>> -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1 >> > > I don't really understand this. I don't see the actual rsync -anc > command, what I see is a diff of two text files whose contents I also > don't understand. So I have to guess that ib_logfile1 is a file inside > of a snapshot on machine A, that was btrfs send -p / receive to > machine B and is now different for some reason? > > If the problem is that ib_logfile1 is wrong only on machine B after > Btrfs send receive, that suggests it might be a network problem. The > Btrfs send receive stream only checksums btrfs metadata (the internal > commands in the stream). The data is not checksummed so it is possible > an uncaught network error can inject silent data corruption which > Btrfs will not catch - it's just the normal TCP/IP network > checksumming happening. > > Anyway, I'm still confused whether the problem is a change only during > send/receive, or if there's a change happening on a machine in > isolation just when you delete other snapshots. > > > -- > Chris Murphy > Hi Chris, I would like to describe what I have done. Exactly I have 3 boxes of ubuntu 16.04 LTS in mycompany environment and the other on hosting provider. Overall steps First -> started first box in company environment and send btrfs offline (batch) to receive with scp to remote box on hosting provider. After incorrect result of incremental I had to send the new base to start over on remote box around 3 times I changed to the Second step. Second -> Setup the second box in company environment on the different network segment of the first box and still send btrfs offline (batch) to receive with scp from first box to second box Yes the result still the same as the First even different date but the incremental unusable vary (not only on the second incremental send) Third -> Change from test with 2 boxes to be only one box with 2 containers and use online send and receive btrfs as you see the current result of btrfs sub volumes. The result still incorrect some day which I had to start send the new base btrfs snapshot whenever it started to be incorrect. I suspect the send/recieve machanism then I uses the snapshot on mysql_201708040830 to be the start base on the source btrfs in container A (backuplogC7) (As I said the local without send/receive it work) until now. Since mysql_201708040830 the incremental still work until mysql_201708110830 it start to incorrect. 1. I started with the first box (Machine X called "virtualtrust3") tpcorp@virtualtrust3:~$ uname -a Linux virtualtrust3 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux tpcorp@virtualtrust3:~$ cat /etc/*release* DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS" NA
Re: btrfs issue with mariadb incremental backup
On Fri, Aug 11, 2017 at 12:00 AM, wrote: > Sorry Chris, > > I forgot to send the diff from rsync -anc result > > the source container A start data as data on snapshot mysql_201708040830 > > [root@backuplogC7 tmp]# ls -l /var/lib/mariadb > total 0 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql > drwxrwxr-x+ 1 mysql mysql 260 Jul 12 08:29 mysql_201708040830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708050830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708060830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708070830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708080830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708090830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708100830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708110830 > > the destcontainer B start data as data on snapshot mysql_201708070830 > > [root@joytest tmp]# ls -l /var/lib/mariadb > total 0 > drwxrwxr-x+ 1 mysql mysql 260 Aug 11 10:24 mysql > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708070830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708080830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708090830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708100830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708110830 > > > > tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt > dest_mysql_201708110830.txt > 1c1 > < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708110830 > --- >> drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708110830 > 5c5 > < -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1 > --- >> -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1 > I don't really understand this. I don't see the actual rsync -anc command, what I see is a diff of two text files whose contents I also don't understand. So I have to guess that ib_logfile1 is a file inside of a snapshot on machine A, that was btrfs send -p / receive to machine B and is now different for some reason? If the problem is that ib_logfile1 is wrong only on machine B after Btrfs send receive, that suggests it might be a network problem. The Btrfs send receive stream only checksums btrfs metadata (the internal commands in the stream). The data is not checksummed so it is possible an uncaught network error can inject silent data corruption which Btrfs will not catch - it's just the normal TCP/IP network checksumming happening. Anyway, I'm still confused whether the problem is a change only during send/receive, or if there's a change happening on a machine in isolation just when you delete other snapshots. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
On Thu, Aug 10, 2017 at 10:40 PM, wrote:
> Hi Chris,
> The kernel version that I test is "4.4.0-89-generic" as I tested on ubuntu
> lxd If I
> want to change the kernel version I have to upgrade the host box.
I can't parse what kernel that really is compared to upstream, which
is at 4.4.80.
> As you suggest the rsync to compare the subvolumes. I found the point.
> the subvolumes are different only after I start to del old subvolumes on
> machine A
> the steps are
You're saying a read only snapshot's contents are changing after
deleting other (child) snapshots?
If you have a subvolume "subvolume" and you make read only snapshots
of it over time "snapshot1" "snapshot2" "snapshot3" "snapshot4"
"snapshot5" "snapshot6" on both machine A and machine B.
And rsync -anc machineA/snapshot6 machineB/snapshot6 they are identical.
And then you delete "snapshot1" "snapshot2" "snapshot3" "snapshot4" on
machine A.
And rsync -anc machineA/snapshot6 machineB/snapshot6 they are no
longer identical?
I've never heard of this before.
>
> 30 08 * * * root /root/script/backup/backupsnap.sh root password
> /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log
> 05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb
> 192.168.45.166
> /var/lib/mariadb >> /var/log/btrfs_send.log
> 30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh
> /var/lib/mariadb 7 >>
> /var/log/btrfs_del.log
I have no idea what this means or why it's relevant.
>
>
> The following script maintain snapshot to currently only 7 snapshots.
> [root@backuplogC7 ~]# cat /root/script/delete_btrfs_sub_snap_volume.sh
> basepath=$1
> keepcount=$2
> havecount=`btrfs sub list -s ${basepath}|cut -d' ' -f14|wc -l`
> delcount=$[$keepcount-$havecount];
> datet=$(date +%Y%m%d%H%M)
> echo "Start Delete ${datet}"
> if [ $delcount -lt 0 ]; then
> # list only snapshot subvolume
> for i in `btrfs sub list -s ${basepath}|cut -d' ' -f14|head ${delcount}`
> do
> echo "btrfs sub delete ${basepath}/$i"
> btrfs sub delete ${basepath}/$i
> btrfs sub sync ${basepath}
Seems sane.
> Does it mean my delete script is not the properly way of the btrfs purge old
> snapshot on source?
I don't think so. But I still don't understand the problem, and what
exactly has changed and when, and it might be my confusion. But if you
have an ro snapshot that's correct at one moment, but then changes
when some other (child) snapshots are deleted, that's pretty
remarkable so I have to assume I'm confused.
--
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
Sorry Chris,
I forgot to send the diff from rsync -anc result
the source container A start data as data on snapshot mysql_201708040830
[root@backuplogC7 tmp]# ls -l /var/lib/mariadb
total 0
drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql
drwxrwxr-x+ 1 mysql mysql 260 Jul 12 08:29 mysql_201708040830
drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708050830
drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708060830
drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708070830
drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708080830
drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708090830
drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708100830
drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708110830
the destcontainer B start data as data on snapshot mysql_201708070830
[root@joytest tmp]# ls -l /var/lib/mariadb
total 0
drwxrwxr-x+ 1 mysql mysql 260 Aug 11 10:24 mysql
drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708070830
drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708080830
drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708090830
drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708100830
drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708110830
tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt
dest_mysql_201708110830.txt
1c1
< drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708110830
---
> drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708110830
5c5
< -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1
---
> -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1
Best Regards,
Siranee Jaraswachirakul.
> Hi Chris,
> The kernel version that I test is "4.4.0-89-generic" as I tested on ubuntu
> lxd If I
> want to change the kernel version I have to upgrade the host box.
>
> As you suggest the rsync to compare the subvolumes. I found the point.
> the subvolumes are different only after I start to del old subvolumes on
> machine A
> the steps are
>
> 30 08 * * * root /root/script/backup/backupsnap.sh root password
> /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log
> 05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb
> 192.168.45.166
> /var/lib/mariadb >> /var/log/btrfs_send.log
> 30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh
> /var/lib/mariadb 7 >>
> /var/log/btrfs_del.log
>
>
> The following script maintain snapshot to currently only 7 snapshots.
> [root@backuplogC7 ~]# cat /root/script/delete_btrfs_sub_snap_volume.sh
> basepath=$1
> keepcount=$2
> havecount=`btrfs sub list -s ${basepath}|cut -d' ' -f14|wc -l`
> delcount=$[$keepcount-$havecount];
> datet=$(date +%Y%m%d%H%M)
> echo "Start Delete ${datet}"
> if [ $delcount -lt 0 ]; then
> # list only snapshot subvolume
> for i in `btrfs sub list -s ${basepath}|cut -d' ' -f14|head ${delcount}`
> do
> echo "btrfs sub delete ${basepath}/$i"
> btrfs sub delete ${basepath}/$i
> btrfs sub sync ${basepath}
> done
> else
> echo "$delcount -gt 0 nothing to delete"
> fi
> echo "Stop Delete ${datet}"
>
> Does it mean my delete script is not the properly way of the btrfs purge old
> snapshot on source?
>
> Best Regards,
>
> Siranee Jarwachirakul.
>
>> On Wed, Aug 9, 2017 at 12:36 AM, wrote:
>>
>>> 488 btrfs sub snap mysql_201707230830 mysql
>>> 489 systemctl start mariadb
>>> 490 btrfs sub list .
>>> 491 cat /var/log/mariadb/mariadb.log
>>
>> OK so mysql_201707230830 once on machine B is inconsistent somehow. So
>> the questions I have are:
>>
>> Is mysql_201707230830 on machine A really identical to
>> mysql_201707230830 on machine B? You can do an rsync -anc (double
>> check those options) which should independently check whether those
>> two subvolumes are in fact identical. The -n is a no op, which doesn't
>> really matter much because as read only subvolumes any attempt to sync
>> will just result in noisy messages. The -c causes rsync to do its own
>> checksum verification on both sides.
>>
>> If the subvolumes are different, we need to find out why.
>>
>> If the subvolumes are the same, then I wonder if you can reproduce the
>> mariadb complaint on machine A merely by making a rw snapshot of
>> mysql_201707230830 and trying to start it. If so, then it's not a send
>> receive problem, it sounds like the snapshot itself is inconsistent,
>> maybe mariadb hasn't actually completely closed out the database at
>> the time the read only snapshot was taken? I'm not sure.
>>
>> If the subvolumes are different, I'm going to recommend updating at
>> least the btrfs-progs because 4.4 is kinda old at this point. The
>> kernel code is what's mainly responsible for the send stream, and the
>> user space code is mainly responsible for receiving. And I don't off
>> hand know or want to look up all the send receive changes between 4.4
>> and 4.12 to speculate on whether this is has already been fixed.
>>
>> What's the kernel version?
>>
>> --
>> Chris Murphy
>>
>
>
--
To unsub
Re: btrfs issue with mariadb incremental backup
Hi Chris,
The kernel version that I test is "4.4.0-89-generic" as I tested on ubuntu lxd
If I
want to change the kernel version I have to upgrade the host box.
As you suggest the rsync to compare the subvolumes. I found the point.
the subvolumes are different only after I start to del old subvolumes on
machine A
the steps are
30 08 * * * root /root/script/backup/backupsnap.sh root password
/var/lib/mariadb/mysql >> /var/log/btrfs_snap.log
05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166
/var/lib/mariadb >> /var/log/btrfs_send.log
30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb
7 >>
/var/log/btrfs_del.log
The following script maintain snapshot to currently only 7 snapshots.
[root@backuplogC7 ~]# cat /root/script/delete_btrfs_sub_snap_volume.sh
basepath=$1
keepcount=$2
havecount=`btrfs sub list -s ${basepath}|cut -d' ' -f14|wc -l`
delcount=$[$keepcount-$havecount];
datet=$(date +%Y%m%d%H%M)
echo "Start Delete ${datet}"
if [ $delcount -lt 0 ]; then
# list only snapshot subvolume
for i in `btrfs sub list -s ${basepath}|cut -d' ' -f14|head ${delcount}`
do
echo "btrfs sub delete ${basepath}/$i"
btrfs sub delete ${basepath}/$i
btrfs sub sync ${basepath}
done
else
echo "$delcount -gt 0 nothing to delete"
fi
echo "Stop Delete ${datet}"
Does it mean my delete script is not the properly way of the btrfs purge old
snapshot on source?
Best Regards,
Siranee Jarwachirakul.
> On Wed, Aug 9, 2017 at 12:36 AM, wrote:
>
>> 488 btrfs sub snap mysql_201707230830 mysql
>> 489 systemctl start mariadb
>> 490 btrfs sub list .
>> 491 cat /var/log/mariadb/mariadb.log
>
> OK so mysql_201707230830 once on machine B is inconsistent somehow. So
> the questions I have are:
>
> Is mysql_201707230830 on machine A really identical to
> mysql_201707230830 on machine B? You can do an rsync -anc (double
> check those options) which should independently check whether those
> two subvolumes are in fact identical. The -n is a no op, which doesn't
> really matter much because as read only subvolumes any attempt to sync
> will just result in noisy messages. The -c causes rsync to do its own
> checksum verification on both sides.
>
> If the subvolumes are different, we need to find out why.
>
> If the subvolumes are the same, then I wonder if you can reproduce the
> mariadb complaint on machine A merely by making a rw snapshot of
> mysql_201707230830 and trying to start it. If so, then it's not a send
> receive problem, it sounds like the snapshot itself is inconsistent,
> maybe mariadb hasn't actually completely closed out the database at
> the time the read only snapshot was taken? I'm not sure.
>
> If the subvolumes are different, I'm going to recommend updating at
> least the btrfs-progs because 4.4 is kinda old at this point. The
> kernel code is what's mainly responsible for the send stream, and the
> user space code is mainly responsible for receiving. And I don't off
> hand know or want to look up all the send receive changes between 4.4
> and 4.12 to speculate on whether this is has already been fixed.
>
> What's the kernel version?
>
> --
> Chris Murphy
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
I'll test and response the result to you soon. I have another question about the combination within Host lxd ubuntu and lxd container centos Host Machine X (16.04.3 LTS with lxd 2.0.10 and btrfs-progs v4.4 kernel 4.4 ) set 1 lvm raw partitions and format to btrfs filesystem (on host) Partition A mount to /opt/partA Container A (CentOS Linux release 7.3.1611 (Core) ) is in Host Machine X mount /opt/partA to /var/lib/mariadb in container A Container A making local readonly snapshot everyday The Options are (online "btrfs send ||ssh btrfs receive" or batch "btrfs send > file.btrfs" and scp from local to remote then "btrfs receive -f file.btrfs") 1. use btrfs send from Host Machine X to Host Machine Y 2. use btrfs send from Container A to Host Machine Y 3. use btrfs send from Container A to Container B 4. use btrfs send incremental from Host Machine X to Host Machine Y 5. use btrfs send incremental from Container A to Host Machine Y 6. use btrfs send incremental from Container A to Container B Host Machine Y (16.04.3 LTS with lxd 2.0.10 and btrfs-progs v4.4 kernel 4.4 ) set 1 lvm raw partitions and format to btrfs filesystem (on host) Partition B mount to /opt/partB Container B (CentOS Linux release 7.3.1611 (Core) ) is in Host Machine Y mount /opt/partB to /var/lib/mariadb in container B The Options are (online "btrfs send ||ssh btrfs receive" or batch "btrfs send > file.btrfs" and scp from local to remote then "btrfs receive -f file.btrfs") 1. use btrfs receive from Host Machine X to Host Machine Y 2. use btrfs receive from Container A to Host Machine Y 3. use btrfs receive from Container A to Container B 4. use btrfs receive incremental from Host Machine X to Host Machine Y 5. use btrfs receive incremental from Container A to Host Machine Y 6. use btrfs receive incremental from Container A to Container B Could you please suggest me what is the combination that should work properly with btrfs send and receive? I plan to setup 2 Site (Production Site and DR site) each site has 1 box (Host Machine X and Y) and want to test btrfs to send incremental. Best Regards, Siranee Jarwachirakul. > On Wed, Aug 9, 2017 at 12:36 AM, wrote: > >> 488 btrfs sub snap mysql_201707230830 mysql >> 489 systemctl start mariadb >> 490 btrfs sub list . >> 491 cat /var/log/mariadb/mariadb.log > > OK so mysql_201707230830 once on machine B is inconsistent somehow. So > the questions I have are: > > Is mysql_201707230830 on machine A really identical to > mysql_201707230830 on machine B? You can do an rsync -anc (double > check those options) which should independently check whether those > two subvolumes are in fact identical. The -n is a no op, which doesn't > really matter much because as read only subvolumes any attempt to sync > will just result in noisy messages. The -c causes rsync to do its own > checksum verification on both sides. > > If the subvolumes are different, we need to find out why. > > If the subvolumes are the same, then I wonder if you can reproduce the > mariadb complaint on machine A merely by making a rw snapshot of > mysql_201707230830 and trying to start it. If so, then it's not a send > receive problem, it sounds like the snapshot itself is inconsistent, > maybe mariadb hasn't actually completely closed out the database at > the time the read only snapshot was taken? I'm not sure. > > If the subvolumes are different, I'm going to recommend updating at > least the btrfs-progs because 4.4 is kinda old at this point. The > kernel code is what's mainly responsible for the send stream, and the > user space code is mainly responsible for receiving. And I don't off > hand know or want to look up all the send receive changes between 4.4 > and 4.12 to speculate on whether this is has already been fixed. > > What's the kernel version? > > -- > Chris Murphy > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
On Wed, Aug 9, 2017 at 12:36 AM, wrote: > 488 btrfs sub snap mysql_201707230830 mysql > 489 systemctl start mariadb > 490 btrfs sub list . > 491 cat /var/log/mariadb/mariadb.log OK so mysql_201707230830 once on machine B is inconsistent somehow. So the questions I have are: Is mysql_201707230830 on machine A really identical to mysql_201707230830 on machine B? You can do an rsync -anc (double check those options) which should independently check whether those two subvolumes are in fact identical. The -n is a no op, which doesn't really matter much because as read only subvolumes any attempt to sync will just result in noisy messages. The -c causes rsync to do its own checksum verification on both sides. If the subvolumes are different, we need to find out why. If the subvolumes are the same, then I wonder if you can reproduce the mariadb complaint on machine A merely by making a rw snapshot of mysql_201707230830 and trying to start it. If so, then it's not a send receive problem, it sounds like the snapshot itself is inconsistent, maybe mariadb hasn't actually completely closed out the database at the time the read only snapshot was taken? I'm not sure. If the subvolumes are different, I'm going to recommend updating at least the btrfs-progs because 4.4 is kinda old at this point. The kernel code is what's mainly responsible for the send stream, and the user space code is mainly responsible for receiving. And I don't off hand know or want to look up all the send receive changes between 4.4 and 4.12 to speculate on whether this is has already been fixed. What's the kernel version? -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
Hi Chris,
Sorry for the incompleted command that I used in the post.
This is the real command from Machine A
backup step :
in crontab
30 08 * * * root /root/script/backup/backupsnap.sh root password
/var/lib/mariadb/mysql
[root@backuplogC7 ~]# cat /root/script/backup/backupsnap.sh
#Backup
#
user=$1
password=$2
basepath=$3
datet=$(date +%Y%m%d%H%M)
snappath=${basepath}_${datet}
echo "Locking databases"
mysql -u$user -p$password << EOF
FLUSH TABLES WITH READ LOCK;
system btrfs sub snap -r $basepath $snappath
UNLOCK TABLES;
quit
EOF
echo "Databases unlocked"
The send incremental step :
487 history|grep send
488 btrfs send /var/lib/mariadb/mysql_201707210830 | ssh 192.168.10.29 btrfs
receive /var/lib/mariadb
489 btrfs send -p /var/lib/mariadb/mysql_201707210830
/var/lib/mariadb/mysql_201707220830 |ssh 192.168.10.29 btrfs receive
/var/lib/mariadb
490 btrfs send -p /var/lib/mariadb/mysql_201707220830
/var/lib/mariadb/mysql_201707230830 |pv| ssh 192.168.10.29 btrfs receive
/var/lib/mariadb
This is the real command from Machine B
463 systemctl stop mariadb
464 cd /var/lib/mariadb
465 ls -l
466 btrfs sub del mysql*
467 btrfs sub sync .
468 ip addr
469 btrfs sub list
470 btrfs sub list .
471 btrfs sub snap mysql_201707210830 mysql
472 systemctl start mariadb
473 mysql -uroot -ppassword
474 btrfs sub list
475 btrfs sub list .
476 mysql -uroot -ppassword
477 systemctl stop mariadb
478 btrfs sub list .
479 btrfs sub del mysql
480 btrfs sub sync .
481 btrfs sub snap mysql_201707220830 mysql
482 systemctl start mariadb
483 mysql -uroot -ppassword
484 systemctl stop mariadb
485 btrfs sub del mysql
486 btrfs sub sync .
487 btrfs sub list .
488 btrfs sub snap mysql_201707230830 mysql
489 systemctl start mariadb
490 btrfs sub list .
491 cat /var/log/mariadb/mariadb.log
PS: No any error in btrfs command or in /var/log/messages.
Best Regards
Siranee Jaraswachirakul.
> On Tue, Aug 8, 2017 at 10:32 PM, wrote:
>> Hi btrfs support team,
>>
>> My name is siranee jaraswachirakul. I tested btrfs incremental send and
>> receive
>> and
>> I found something incorrect.
>>
>> I posted detail in the url
>> http://www.linuxquestions.org/questions/showthread.php?p=5746238#post5746238
>>
>> Could you please help me to find the reason?
>
> The command "btrfs sub snap mysql_201707210830 mysql" is unexpected
> for two reasons:
>
> 1. This is a rw snapshot. It needs to be ro using -r flag to make it
> ro, and only ro snapshots can be used with btrfs send receive.
> 2. The naming convention seems reversed. The subvolume you're
> snapshotting should be first, and the resulting snapshot is second,
> and it is the second one that you'll send. So I think really what
> you'll end up wanting is:
>
> Machine A
> btrfs sub snap -r mysql mysql_20170721
> btrfs send mysql_20170721
>
> Machine B
> btrfs sub snap mysql_20170721 mysql
>
> That will make a rw snapshot from the ro snapshot.
>
> The next problem with the post is that you say "incremental" but I do
> not see the exact send receive command you're using. An incremental
> send requires -p option.
>
>
>
> Machine A
> btrfs sub snap -r mysql mysql_20170721
> btrfs send mysql_20170721
>
> Machine B
> btrfs sub snap mysql_20170721 mysql
> Test mysql
> btrfs sub del mysql
>
> Machine A (next day)
>
> Machine A
> btrfs sub snap -r mysql mysql_20170722
> btrfs send -p mysql_20170721 mysql_20170722
>
> Machine B
> btrfs sub snap mysql_20170722 mysql
> Test mysql
> btrfs sub del mysql
>
> And so on. You must have the -p snapshot on both file systems for the
> incremental send/receive to work.
>
>
>
> --
> Chris Murphy
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs issue with mariadb incremental backup
On Tue, Aug 8, 2017 at 10:32 PM, wrote: > Hi btrfs support team, > > My name is siranee jaraswachirakul. I tested btrfs incremental send and > receive and > I found something incorrect. > > I posted detail in the url > http://www.linuxquestions.org/questions/showthread.php?p=5746238#post5746238 > > Could you please help me to find the reason? The command "btrfs sub snap mysql_201707210830 mysql" is unexpected for two reasons: 1. This is a rw snapshot. It needs to be ro using -r flag to make it ro, and only ro snapshots can be used with btrfs send receive. 2. The naming convention seems reversed. The subvolume you're snapshotting should be first, and the resulting snapshot is second, and it is the second one that you'll send. So I think really what you'll end up wanting is: Machine A btrfs sub snap -r mysql mysql_20170721 btrfs send mysql_20170721 Machine B btrfs sub snap mysql_20170721 mysql That will make a rw snapshot from the ro snapshot. The next problem with the post is that you say "incremental" but I do not see the exact send receive command you're using. An incremental send requires -p option. Machine A btrfs sub snap -r mysql mysql_20170721 btrfs send mysql_20170721 Machine B btrfs sub snap mysql_20170721 mysql Test mysql btrfs sub del mysql Machine A (next day) Machine A btrfs sub snap -r mysql mysql_20170722 btrfs send -p mysql_20170721 mysql_20170722 Machine B btrfs sub snap mysql_20170722 mysql Test mysql btrfs sub del mysql And so on. You must have the -p snapshot on both file systems for the incremental send/receive to work. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
