Re: How to get back a deleted sub-volume.

[Tomasz Kusmierz]

Chris,

the "btrfs-show-super -fa" gives me nothing useful to work with.

the "btrfs-find-root -a " is actually something that I was
already using (see original post), but the list of roots given had a
rather LARGE hole of 200 generations that is located between right
after I've had everything removed and 1 month before the whole
situation.

On 12 December 2016 at 04:14, Chris Murphy  wrote:
> Tomasz - try using 'btrfs-find-root -a ' I totally forgot about
> this option. It goes through the extent tree and might have a chance
> of finding additional generations that aren't otherwise being found.
> You can then plug those tree roots into 'btrfs restore -t '
> and do it with the -D and -v options so it's a verbose dry run, and
> see if the file listing it spits out is at all useful - if it has any
> of the data you're looking for.
>
>
> Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to get back a deleted sub-volume.

[Chris Murphy]

Tomasz - try using 'btrfs-find-root -a ' I totally forgot about
this option. It goes through the extent tree and might have a chance
of finding additional generations that aren't otherwise being found.
You can then plug those tree roots into 'btrfs restore -t '
and do it with the -D and -v options so it's a verbose dry run, and
see if the file listing it spits out is at all useful - if it has any
of the data you're looking for.


Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to get back a deleted sub-volume.

[Chris Murphy]

On Sun, Dec 11, 2016 at 5:56 PM, Tomasz Kusmierz  wrote:
> Chris, for all the time you helped so far I have to really appologize
> I've led you a stray ... so, reson the subvolumes were deleted is
> nothing to do with btrfs it self, I'm using "Rockstor" to ease
> managment tasks. This tool / environment / distribution treats a
> singular btrfs FS as a "pool" ( something in line of zfs :/ ) and when
> one removes a pool from the system it will actually go and delete
> subvolumes from FS before unmounting it and removing reference of it
> from it's DB (yes a bit shiet I know). so I'm not blaming anybody here
> for disapearing subvolumes, it's just me commig back to belive in man
> kind to get kicked in the gonads by mankind stupidity.
>
> ALSO by importing the fs to their "solution" is just actually mounting
> and walking the tree of subvolumes to to create all the references in
> local DB (for rockstor of course, still nothing to do with btrfs
> functionality). To be able to ïmport" I've had to remove before
> mentioned snpshots becouse imports script was timing out.
>
> So for a single subvolume (called physically "share") I was left with
> no snapshots (removed by me to make import not time out) and then this
> subvolume was removed when I was trying to remove a fs (pool) from a
> running system.
>
> I've polled both disks (2 disk fs raid1) and I'm trying to rescue as
> much data as I can.
>
> The question is, why suddenly when I removed the snapshots and
> (someone else removed) the subvolume, there is such a great gap in
> generations of FS (over 200 generations missing) and the most recent
> generation that actually can me touched by btrfs restore is over a
> month old.
>
> How to over come that ?

Well it depends on how long it was from the time of the snapshots
being deleted to the time the file system was unmounted. If it wasn't
that long it might be possible to find a root from 'btrfs insp-in
dump-s -fa ' (or btrfs-show-super with older progs) to see if you
can use any of the backup roots. Once so much time goes by, the no
longer used metadata for generations has their roots deleted and all
of the blocks used for both metadata and data are subject to being
overwritten. So my expectation is that there's too much time between
delete and umount, so there's nothing in the current file system that
points to the old generations.

It might be the metadata and data still exist. It's not entirely a
shot in the dark to find it.

First, try to find the oldest chunk tree you can with btrfs-show-super
-fa (btrfs insp dump-s -fa) and look in the backup roots list for the
chunk tree:

backup_roots[4]:
backup 0:
backup_tree_root:21037056gen: 7level: 0
backup_chunk_root:147456gen: 6level: 0
backup_extent_root:21053440gen: 7level: 0
backup_fs_root:4194304gen: 4level: 0
backup_dev_root:20987904gen: 6level: 0
backup_csum_root:21069824gen: 7level: 0
backup_total_bytes:268435456000
backup_bytes_used:393216
backup_num_devices:1

backup 1:
backup_tree_root:21086208gen: 8level: 0
backup_chunk_root:147456gen: 6level: 0
backup_extent_root:21102592gen: 8level: 0
backup_fs_root:4194304gen: 4level: 0
backup_dev_root:20987904gen: 6level: 0
backup_csum_root:21118976gen: 8level: 0
backup_total_bytes:268435456000
backup_bytes_used:393216
backup_num_devices:1

backup 2:
backup_tree_root:21069824gen: 9level: 0
backup_chunk_root:147456gen: 6level: 0
backup_extent_root:21053440gen: 9level: 0
backup_fs_root:21004288gen: 9level: 0
backup_dev_root:21135360gen: 9level: 0
backup_csum_root:21037056gen: 9level: 0
backup_total_bytes:268435456000
backup_bytes_used:487424
backup_num_devices:1

backup 3:
backup_tree_root:21151744gen: 10level: 0
backup_chunk_root:147456gen: 6level: 0
backup_extent_root:21168128gen: 10level: 0
backup_fs_root:21004288gen: 9level: 0
backup_dev_root:21135360gen: 9level: 0
backup_csum_root:21184512gen: 10level: 0
backup_total_bytes:268435456000
backup_bytes_used:487424
backup_num_devices:1

IN this case, all of the chunk roots are the same generation - so it's
no help really.

Second, list the chunk tree using either -t 3 for the current one, or
you can plug in the bytenr for an older backup chunk root if
available.

[root@f25h ~]# btrfs-debug-tree -t 3 /dev/mapper/vg-test2
btrfs-progs v4.8.5
chunk tree
leaf 147456 items 5 free space 15740 generation 

Re: How to get back a deleted sub-volume.

[Tomasz Kusmierz]

Chris, for all the time you helped so far I have to really appologize
I've led you a stray ... so, reson the subvolumes were deleted is
nothing to do with btrfs it self, I'm using "Rockstor" to ease
managment tasks. This tool / environment / distribution treats a
singular btrfs FS as a "pool" ( something in line of zfs :/ ) and when
one removes a pool from the system it will actually go and delete
subvolumes from FS before unmounting it and removing reference of it
from it's DB (yes a bit shiet I know). so I'm not blaming anybody here
for disapearing subvolumes, it's just me commig back to belive in man
kind to get kicked in the gonads by mankind stupidity.

ALSO by importing the fs to their "solution" is just actually mounting
and walking the tree of subvolumes to to create all the references in
local DB (for rockstor of course, still nothing to do with btrfs
functionality). To be able to ïmport" I've had to remove before
mentioned snpshots becouse imports script was timing out.

So for a single subvolume (called physically "share") I was left with
no snapshots (removed by me to make import not time out) and then this
subvolume was removed when I was trying to remove a fs (pool) from a
running system.

I've polled both disks (2 disk fs raid1) and I'm trying to rescue as
much data as I can.

The question is, why suddenly when I removed the snapshots and
(someone else removed) the subvolume, there is such a great gap in
generations of FS (over 200 generations missing) and the most recent
generation that actually can me touched by btrfs restore is over a
month old.

How to over come that ?



On 11 December 2016 at 19:00, Chris Murphy  wrote:
> On Sun, Dec 11, 2016 at 10:40 AM, Tomasz Kusmierz
>  wrote:
>> Hi,
>>
>> So, I've found my self in a pickle after following this steps:
>> 1. trying to migrate an array to different system, it became apparent
>> that importing array there was not possible to import it because I've
>> had a very large amount of snapshots (every 15 minutes during office
>> hours amounting to few K) so I've had to remove snapshots for main
>> data storage.
>
> True, there is no recursive incremental send.
>
>> 2. while playing with live array, it become apparent that some bright
>> spark implemented a "delete all sub-volumes while removing array from
>> system" ... needles to say that this behaviour is unexpected to say al
>> least ... and I wanted to punch somebody in face.
>
> The technical part of this is vague. I'm guessing you used 'btrfs
> device remove' butt it works no differently than lvremove - when a
> device is removed from an array, it wipes the signature from that
> device.  You probably can restore that signature and use that device
> again, depending on what the profile is for metadata and data, it may
> be usable stand alone.
>
> Proposing assault is probably not the best way to ask for advice
> though. Just a guess.
>
>
>
>
>>
>> Since then I was trying to rescue as much data as I can, luckily I
>> managed to get a lot of data from snapshots for "other than share"
>> volumes (because those were not deleted :/) but the most important
>> volume "share" prove difficult. This subvolume comes out with a lot of
>> errors on readout with "btrfs restore /dev/sda /mnt2/temp2/ -x -m -S
>> -s -i -t".
>>
>> Also for some reason I can't use a lot of root blocks that I find with
>> btrfs-find-root ..
>>
>> To put some detail here:
>> btrfs-find-root -a /dev/sda
>> Superblock thinks the generation is 184540
>> Superblock thinks the level is 1
>> Well block 919363862528(gen: 184540 level: 1) seems good, and it
>> matches superblock
>> Well block 919356325888(gen: 184539 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919343529984(gen: 184538 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920041308160(gen: 184537 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919941955584(gen: 184536 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919670538240(gen: 184535 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920045371392(gen: 184532 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920070209536(gen: 184531 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920117510144(gen: 184530 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1 <<< here
>> stuff is gone
>> Well block 920139055104(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920139022336(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920138989568(gen: 184511 

Re: How to get back a deleted sub-volume.

[Chris Murphy]

On Sun, Dec 11, 2016 at 10:40 AM, Tomasz Kusmierz
 wrote:
> Hi,
>
> So, I've found my self in a pickle after following this steps:
> 1. trying to migrate an array to different system, it became apparent
> that importing array there was not possible to import it because I've
> had a very large amount of snapshots (every 15 minutes during office
> hours amounting to few K) so I've had to remove snapshots for main
> data storage.

True, there is no recursive incremental send.

> 2. while playing with live array, it become apparent that some bright
> spark implemented a "delete all sub-volumes while removing array from
> system" ... needles to say that this behaviour is unexpected to say al
> least ... and I wanted to punch somebody in face.

The technical part of this is vague. I'm guessing you used 'btrfs
device remove' butt it works no differently than lvremove - when a
device is removed from an array, it wipes the signature from that
device.  You probably can restore that signature and use that device
again, depending on what the profile is for metadata and data, it may
be usable stand alone.

Proposing assault is probably not the best way to ask for advice
though. Just a guess.




>
> Since then I was trying to rescue as much data as I can, luckily I
> managed to get a lot of data from snapshots for "other than share"
> volumes (because those were not deleted :/) but the most important
> volume "share" prove difficult. This subvolume comes out with a lot of
> errors on readout with "btrfs restore /dev/sda /mnt2/temp2/ -x -m -S
> -s -i -t".
>
> Also for some reason I can't use a lot of root blocks that I find with
> btrfs-find-root ..
>
> To put some detail here:
> btrfs-find-root -a /dev/sda
> Superblock thinks the generation is 184540
> Superblock thinks the level is 1
> Well block 919363862528(gen: 184540 level: 1) seems good, and it
> matches superblock
> Well block 919356325888(gen: 184539 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919343529984(gen: 184538 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920041308160(gen: 184537 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919941955584(gen: 184536 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919670538240(gen: 184535 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920045371392(gen: 184532 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920070209536(gen: 184531 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920117510144(gen: 184530 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1 <<< here
> stuff is gone
> Well block 920139055104(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920139022336(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920138989568(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920138973184(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920137596928(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920137531392(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920137515008(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135991296(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135958528(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135925760(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135827456(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135811072(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920133697536(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920133664768(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 92017088(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920133206016(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920132976640(gen: 184511 level: 0) seems good, but
> 

How to get back a deleted sub-volume.

[Tomasz Kusmierz]

Hi,

So, I've found my self in a pickle after following this steps:
1. trying to migrate an array to different system, it became apparent
that importing array there was not possible to import it because I've
had a very large amount of snapshots (every 15 minutes during office
hours amounting to few K) so I've had to remove snapshots for main
data storage.
2. while playing with live array, it become apparent that some bright
spark implemented a "delete all sub-volumes while removing array from
system" ... needles to say that this behaviour is unexpected to say al
least ... and I wanted to punch somebody in face.
3. the backup off-site server that was making backups every 30 minutes
was located in CEO house and his wife decide that it's not necessary
to have it connected

(laughs can start roughly here)

So I've got array with all the data there (theoretically COW, right ?)
with additional of plethora of snapshots (important data was snapped
every 15 minutes during a office hours to capture all the changes,
other sub-volumes were snapshoted daily)

This occurred roughly on 4-12-2016.

Since then I was trying to rescue as much data as I can, luckily I
managed to get a lot of data from snapshots for "other than share"
volumes (because those were not deleted :/) but the most important
volume "share" prove difficult. This subvolume comes out with a lot of
errors on readout with "btrfs restore /dev/sda /mnt2/temp2/ -x -m -S
-s -i -t".

Also for some reason I can't use a lot of root blocks that I find with
btrfs-find-root ..

To put some detail here:
btrfs-find-root -a /dev/sda
Superblock thinks the generation is 184540
Superblock thinks the level is 1
Well block 919363862528(gen: 184540 level: 1) seems good, and it
matches superblock
Well block 919356325888(gen: 184539 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919343529984(gen: 184538 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920041308160(gen: 184537 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919941955584(gen: 184536 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919670538240(gen: 184535 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920045371392(gen: 184532 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920070209536(gen: 184531 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920117510144(gen: 184530 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1 <<< here
stuff is gone
Well block 920139055104(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920139022336(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920138989568(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920138973184(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920137596928(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920137531392(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920137515008(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135991296(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135958528(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135925760(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135827456(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135811072(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920133697536(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920133664768(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 92017088(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920133206016(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920132976640(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920132878336(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920132845568(gen: 184511 level: 0) seems good, but
generation/level doesn't match,