Re: [ceph-users] Recover Data from Deleted RBD Volume

[Georgios Dimitrakakis]

As a closure I would like to thank all people who contributed with 
their knowledge in my problem although the final decision was not to try 
any sort of recovery since the effort required would have been 
tremendous with unambiguous results (to say at least).


Jason, Ilya, Brad, David, George, Burkhard thank you very much for your 
contribution


Kind regards,

G.


On Wed, Aug 10, 2016 at 10:55 AM, Ilya Dryomov  
wrote:

I think Jason meant to write "rbd_id." here.



Whoops -- thanks for the typo correction.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Ilya Dryomov]

On Mon, Aug 8, 2016 at 11:47 PM, Jason Dillaman  wrote:
> On Mon, Aug 8, 2016 at 5:39 PM, Jason Dillaman  wrote:
>> Unfortunately, for v2 RBD images, this image name to image id mapping
>> is stored in the LevelDB database within the OSDs and I don't know,
>> offhand, how to attempt to recover deleted values from there.
>
> Actually, to correct myself, the "rbd_id." object just

I think Jason meant to write "rbd_id." here.

> writes the image id in binary to the file. So if you can recover that
> file and retrieve its contents, you can again determine the block name
> prefix in the form of "rbd_data..".

So if you had a v2 image named "myimage" in a pool named "rbd"

$ ceph osd map rbd rbd_id.myimage
osdmap e11 pool 'rbd' (0) object 'rbd_id.myimage' -> pg 0.f8d9dc15
(0.5) -> up ([0,2,1], p0) acting ([0,2,1], p0)

you'd need to search for a file whose name starts with
"rbd\uid.myimage" under /current/0.5_head/.
The 0.5 is the short PG id from the ceph osd map command above (the
object doesn't have to exist for it to work).  The "\u" is literally
a "\" followed by a "u" - ceph's FileStore uses underscores as
separators so underscores in object names get translated to "\u" in the
corresponding file names.  The actual file name is going to be
something along the lines of "rbd\uid.myimage__head_F8D9DC15__0":

$ xxd "./0.5_head/rbd\uid.myimage__head_F8D9DC15__0"
: 0c00  3130 3130 3734 6230 6463 3531  101074b0dc51

That's the prefix for the image.  myimage actually exists here, so
I can verify it with:

$ rbd info rbd/myimage | grep block_name_prefix
block_name_prefix: rbd_data.101074b0dc51

With the prefix at hand, you'd need to search all /current/
directories for files whose names start with "rbd\udata.101074b0dc51",
doing the equivalent of:

$ find . -regex ".*/rbd\\\udata.101074b0dc51.*"
./0.4_head/rbd\udata.101074b0dc51.0003__head_64B130D4__0
./0.0_head/rbd\udata.101074b0dc51.__head_7A694010__0
./0.3_head/rbd\udata.101074b0dc51.0004__head_85FCAA2B__0
./0.1_head/rbd\udata.101074b0dc51.0002__head_660B5009__0
./0.6_head/rbd\udata.101074b0dc51.0001__head_33B916C6__0
...


There is a rbd-recover-tool tool in the ceph source tree, which can
reconstruct rbd images from a FileStore structure outlined in this
thread.  I'm not sure if we document it or even build it (probably not,
and it won't be of much use to you anyway since the files are gone),
but you can peruse the code for the exact object name regexes.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Jason Dillaman]

The image's associated metadata is removed from the directory once the
image is removed.  Also, the default librbd log level will not log an
image's internal id.  Therefore, unfortunately, the only way to
proceed is how I previously described.

On Wed, Aug 10, 2016 at 2:48 AM, Brad Hubbard  wrote:
>
>
> On Wed, Aug 10, 2016 at 3:16 PM, Georgios Dimitrakakis  
> wrote:
>>
>> Hello!
>>
>> Brad,
>>
>> is that possible from the default logging or verbose one is needed??
>>
>> I 've managed to get the UUID of the deleted volume from OpenStack but don't
>> really know how to get the offsets and OSD maps since "rbd info" doesn't
>> provide any information for that volume.
>
> Did you grep for the UUID (might be safer to grep for the first 8 chars or
> so since I'm not 100% sure of the format) in the logs?
>
> There is also a RADOS object called the rbd directory that contains some
> mapping information for rbd images but I don't know if this is erased when an
> image is deleted, nor how to look at it but someone more adept at RBD may be
> able to make suggestions how to confirm this?
>
> HTH,
> Brad
>
>>
>> Is it possible to somehow get them from leveldb?
>>
>> Best,
>>
>> G.
>>
>>
>>> On Tue, Aug 9, 2016 at 7:39 AM, George Mihaiescu
>>>  wrote:

 Look in the cinder db, the volumes table to find the Uuid of the deleted
 volume.
>>>
>>>
>>> You could also look through the logs at the time of the delete and I
>>> suspect you should
>>> be able to see how the rbd image was prefixed/named at the time of
>>> the delete.
>>>
>>> HTH,
>>> Brad
>>>

 If you go through yours OSDs and look for the directories for PG index
 20, you might find some fragments from the deleted volume, but it's a long
 shot...

> On Aug 8, 2016, at 4:39 PM, Georgios Dimitrakakis 
> wrote:
>
> Dear David (and all),
>
> the data are considered very critical therefore all this attempt to
> recover them.
>
> Although the cluster hasn't been fully stopped all users actions have. I
> mean services are running but users are not able to read/write/delete.
>
> The deleted image was the exact same size of the example (500GB) but it
> wasn't the only one deleted today. Our user was trying to do a "massive"
> cleanup by deleting 11 volumes and unfortunately one of them was very
> important.
>
> Let's assume that I "dd" all the drives what further actions should I do
> to recover the files? Could you please elaborate a bit more on the phrase
> "If you've never deleted any other rbd images and assuming you can recover
> data with names, you may be able to find the rbd objects"??
>
> Do you mean that if I know the file names I can go through and check for
> them? How?
> Do I have to know *all* file names or by searching for a few of them I
> can find all data that exist?
>
> Thanks a lot for taking the time to answer my questions!
>
> All the best,
>
> G.
>
>> I dont think theres a way of getting the prefix from the cluster at
>> this point.
>>
>> If the deleted image was a similar size to the example youve given,
>> you will likely have had objects on every OSD. If this data is
>> absolutely critical you need to stop your cluster immediately or make
>> copies of all the drives with something like dd. If youve never
>> deleted any other rbd images and assuming you can recover data with
>> names, you may be able to find the rbd objects.
>>
>> On Mon, Aug 8, 2016 at 7:28 PM, Georgios Dimitrakakis  wrote:
>>
> Hi,
>
> On 08.08.2016 10:50, Georgios Dimitrakakis wrote:
>
>>> Hi,
>>>
 On 08.08.2016 09:58, Georgios Dimitrakakis wrote:

 Dear all,

 I would like your help with an emergency issue but first
 let me describe our environment.

 Our environment consists of 2OSD nodes with 10x 2TB HDDs
 each and 3MON nodes (2 of them are the OSD nodes as well)
 all with ceph version 0.80.9
 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)

 This environment provides RBD volumes to an OpenStack
 Icehouse installation.

 Although not a state of the art environment is working
 well and within our expectations.

 The issue now is that one of our users accidentally
 deleted one of the volumes without keeping its data first!

 Is there any way (since the data are considered critical
 and very important) to recover them from CEPH?
>>>
>>>
>>> Short answer: no
>>>
>>> Long answer: no, but
>>>
>>> Consider the way Ceph 

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Brad Hubbard]

On Wed, Aug 10, 2016 at 3:16 PM, Georgios Dimitrakakis  
wrote:
>
> Hello!
>
> Brad,
>
> is that possible from the default logging or verbose one is needed??
>
> I 've managed to get the UUID of the deleted volume from OpenStack but don't
> really know how to get the offsets and OSD maps since "rbd info" doesn't
> provide any information for that volume.

Did you grep for the UUID (might be safer to grep for the first 8 chars or
so since I'm not 100% sure of the format) in the logs?

There is also a RADOS object called the rbd directory that contains some
mapping information for rbd images but I don't know if this is erased when an
image is deleted, nor how to look at it but someone more adept at RBD may be
able to make suggestions how to confirm this?

HTH,
Brad

>
> Is it possible to somehow get them from leveldb?
>
> Best,
>
> G.
>
>
>> On Tue, Aug 9, 2016 at 7:39 AM, George Mihaiescu
>>  wrote:
>>>
>>> Look in the cinder db, the volumes table to find the Uuid of the deleted
>>> volume.
>>
>>
>> You could also look through the logs at the time of the delete and I
>> suspect you should
>> be able to see how the rbd image was prefixed/named at the time of
>> the delete.
>>
>> HTH,
>> Brad
>>
>>>
>>> If you go through yours OSDs and look for the directories for PG index
>>> 20, you might find some fragments from the deleted volume, but it's a long
>>> shot...
>>>
 On Aug 8, 2016, at 4:39 PM, Georgios Dimitrakakis 
 wrote:

 Dear David (and all),

 the data are considered very critical therefore all this attempt to
 recover them.

 Although the cluster hasn't been fully stopped all users actions have. I
 mean services are running but users are not able to read/write/delete.

 The deleted image was the exact same size of the example (500GB) but it
 wasn't the only one deleted today. Our user was trying to do a "massive"
 cleanup by deleting 11 volumes and unfortunately one of them was very
 important.

 Let's assume that I "dd" all the drives what further actions should I do
 to recover the files? Could you please elaborate a bit more on the phrase
 "If you've never deleted any other rbd images and assuming you can recover
 data with names, you may be able to find the rbd objects"??

 Do you mean that if I know the file names I can go through and check for
 them? How?
 Do I have to know *all* file names or by searching for a few of them I
 can find all data that exist?

 Thanks a lot for taking the time to answer my questions!

 All the best,

 G.

> I dont think theres a way of getting the prefix from the cluster at
> this point.
>
> If the deleted image was a similar size to the example youve given,
> you will likely have had objects on every OSD. If this data is
> absolutely critical you need to stop your cluster immediately or make
> copies of all the drives with something like dd. If youve never
> deleted any other rbd images and assuming you can recover data with
> names, you may be able to find the rbd objects.
>
> On Mon, Aug 8, 2016 at 7:28 PM, Georgios Dimitrakakis  wrote:
>
 Hi,

 On 08.08.2016 10:50, Georgios Dimitrakakis wrote:

>> Hi,
>>
>>> On 08.08.2016 09:58, Georgios Dimitrakakis wrote:
>>>
>>> Dear all,
>>>
>>> I would like your help with an emergency issue but first
>>> let me describe our environment.
>>>
>>> Our environment consists of 2OSD nodes with 10x 2TB HDDs
>>> each and 3MON nodes (2 of them are the OSD nodes as well)
>>> all with ceph version 0.80.9
>>> (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)
>>>
>>> This environment provides RBD volumes to an OpenStack
>>> Icehouse installation.
>>>
>>> Although not a state of the art environment is working
>>> well and within our expectations.
>>>
>>> The issue now is that one of our users accidentally
>>> deleted one of the volumes without keeping its data first!
>>>
>>> Is there any way (since the data are considered critical
>>> and very important) to recover them from CEPH?
>>
>>
>> Short answer: no
>>
>> Long answer: no, but
>>
>> Consider the way Ceph stores data... each RBD is striped
>> into chunks
>> (RADOS objects with 4MB size by default); the chunks are
>> distributed
>> among the OSDs with the configured number of replicates
>> (probably two
>> in your case since you use 2 OSD hosts). RBD uses thin
>> provisioning,
>> so chunks are allocated upon first write access.
>> If an RBD is deleted all of its chunks are deleted on the

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Georgios Dimitrakakis]

Hello!

Brad,

is that possible from the default logging or verbose one is needed??

I 've managed to get the UUID of the deleted volume from OpenStack but 
don't really know how to get the offsets and OSD maps since "rbd info" 
doesn't provide any information for that volume.


Is it possible to somehow get them from leveldb?

Best,

G.


On Tue, Aug 9, 2016 at 7:39 AM, George Mihaiescu
 wrote:
Look in the cinder db, the volumes table to find the Uuid of the 
deleted volume.


You could also look through the logs at the time of the delete and I
suspect you should
be able to see how the rbd image was prefixed/named at the time of
the delete.

HTH,
Brad



If you go through yours OSDs and look for the directories for PG 
index 20, you might find some fragments from the deleted volume, but 
it's a long shot...


On Aug 8, 2016, at 4:39 PM, Georgios Dimitrakakis 
 wrote:


Dear David (and all),

the data are considered very critical therefore all this attempt to 
recover them.


Although the cluster hasn't been fully stopped all users actions 
have. I mean services are running but users are not able to 
read/write/delete.


The deleted image was the exact same size of the example (500GB) 
but it wasn't the only one deleted today. Our user was trying to do a 
"massive" cleanup by deleting 11 volumes and unfortunately one of 
them was very important.


Let's assume that I "dd" all the drives what further actions should 
I do to recover the files? Could you please elaborate a bit more on 
the phrase "If you've never deleted any other rbd images and assuming 
you can recover data with names, you may be able to find the rbd 
objects"??


Do you mean that if I know the file names I can go through and 
check for them? How?
Do I have to know *all* file names or by searching for a few of 
them I can find all data that exist?


Thanks a lot for taking the time to answer my questions!

All the best,

G.

I dont think theres a way of getting the prefix from the cluster 
at

this point.

If the deleted image was a similar size to the example youve 
given,

you will likely have had objects on every OSD. If this data is
absolutely critical you need to stop your cluster immediately or 
make

copies of all the drives with something like dd. If youve never
deleted any other rbd images and assuming you can recover data 
with

names, you may be able to find the rbd objects.

On Mon, Aug 8, 2016 at 7:28 PM, Georgios Dimitrakakis  wrote:


Hi,

On 08.08.2016 10:50, Georgios Dimitrakakis wrote:


Hi,


On 08.08.2016 09:58, Georgios Dimitrakakis wrote:

Dear all,

I would like your help with an emergency issue but first
let me describe our environment.

Our environment consists of 2OSD nodes with 10x 2TB HDDs
each and 3MON nodes (2 of them are the OSD nodes as well)
all with ceph version 0.80.9
(b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)

This environment provides RBD volumes to an OpenStack
Icehouse installation.

Although not a state of the art environment is working
well and within our expectations.

The issue now is that one of our users accidentally
deleted one of the volumes without keeping its data first!

Is there any way (since the data are considered critical
and very important) to recover them from CEPH?


Short answer: no

Long answer: no, but

Consider the way Ceph stores data... each RBD is striped
into chunks
(RADOS objects with 4MB size by default); the chunks are
distributed
among the OSDs with the configured number of replicates
(probably two
in your case since you use 2 OSD hosts). RBD uses thin
provisioning,
so chunks are allocated upon first write access.
If an RBD is deleted all of its chunks are deleted on the
corresponding OSDs. If you want to recover a deleted RBD,
you need to
recover all individual chunks. Whether this is possible
depends on
your filesystem and whether the space of a former chunk is
already
assigned to other RADOS objects. The RADOS object names are
composed
of the RBD name and the offset position of the chunk, so if
an
undelete mechanism exists for the OSDs filesystem, you have
to be
able to recover file by their filename, otherwise you might
end up
mixing the content of various deleted RBDs. Due to the thin
provisioning there might be some chunks missing (e.g. never
allocated
before).

Given the fact that
- you probably use XFS on the OSDs since it is the
preferred
filesystem for OSDs (there is RDR-XFS, but Ive never had to
use it)
- you would need to stop the complete ceph cluster
(recovery tools do
not work on mounted filesystems)
- your cluster has been in use after the RBD was deleted
and thus
parts of its former space might already have been
overwritten
(replication might help you here, since there are two OSDs
to try)
- XFS undelete does not work well on fragmented files (and
OSDs tend
to introduce fragmentation...)

the answer is no, since it might not be feasible and the
chance of
success are way too low.

If you want to spend time 

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Brad Hubbard]

On Tue, Aug 9, 2016 at 7:39 AM, George Mihaiescu  wrote:
> Look in the cinder db, the volumes table to find the Uuid of the deleted 
> volume.

You could also look through the logs at the time of the delete and I
suspect you should
be able to see how the rbd image was prefixed/named at the time of the delete.

HTH,
Brad

>
> If you go through yours OSDs and look for the directories for PG index 20, 
> you might find some fragments from the deleted volume, but it's a long shot...
>
>> On Aug 8, 2016, at 4:39 PM, Georgios Dimitrakakis  
>> wrote:
>>
>> Dear David (and all),
>>
>> the data are considered very critical therefore all this attempt to recover 
>> them.
>>
>> Although the cluster hasn't been fully stopped all users actions have. I 
>> mean services are running but users are not able to read/write/delete.
>>
>> The deleted image was the exact same size of the example (500GB) but it 
>> wasn't the only one deleted today. Our user was trying to do a "massive" 
>> cleanup by deleting 11 volumes and unfortunately one of them was very 
>> important.
>>
>> Let's assume that I "dd" all the drives what further actions should I do to 
>> recover the files? Could you please elaborate a bit more on the phrase "If 
>> you've never deleted any other rbd images and assuming you can recover data 
>> with names, you may be able to find the rbd objects"??
>>
>> Do you mean that if I know the file names I can go through and check for 
>> them? How?
>> Do I have to know *all* file names or by searching for a few of them I can 
>> find all data that exist?
>>
>> Thanks a lot for taking the time to answer my questions!
>>
>> All the best,
>>
>> G.
>>
>>> I dont think theres a way of getting the prefix from the cluster at
>>> this point.
>>>
>>> If the deleted image was a similar size to the example youve given,
>>> you will likely have had objects on every OSD. If this data is
>>> absolutely critical you need to stop your cluster immediately or make
>>> copies of all the drives with something like dd. If youve never
>>> deleted any other rbd images and assuming you can recover data with
>>> names, you may be able to find the rbd objects.
>>>
>>> On Mon, Aug 8, 2016 at 7:28 PM, Georgios Dimitrakakis  wrote:
>>>
>> Hi,
>>
>> On 08.08.2016 10:50, Georgios Dimitrakakis wrote:
>>
 Hi,

> On 08.08.2016 09:58, Georgios Dimitrakakis wrote:
>
> Dear all,
>
> I would like your help with an emergency issue but first
> let me describe our environment.
>
> Our environment consists of 2OSD nodes with 10x 2TB HDDs
> each and 3MON nodes (2 of them are the OSD nodes as well)
> all with ceph version 0.80.9
> (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)
>
> This environment provides RBD volumes to an OpenStack
> Icehouse installation.
>
> Although not a state of the art environment is working
> well and within our expectations.
>
> The issue now is that one of our users accidentally
> deleted one of the volumes without keeping its data first!
>
> Is there any way (since the data are considered critical
> and very important) to recover them from CEPH?

 Short answer: no

 Long answer: no, but

 Consider the way Ceph stores data... each RBD is striped
 into chunks
 (RADOS objects with 4MB size by default); the chunks are
 distributed
 among the OSDs with the configured number of replicates
 (probably two
 in your case since you use 2 OSD hosts). RBD uses thin
 provisioning,
 so chunks are allocated upon first write access.
 If an RBD is deleted all of its chunks are deleted on the
 corresponding OSDs. If you want to recover a deleted RBD,
 you need to
 recover all individual chunks. Whether this is possible
 depends on
 your filesystem and whether the space of a former chunk is
 already
 assigned to other RADOS objects. The RADOS object names are
 composed
 of the RBD name and the offset position of the chunk, so if
 an
 undelete mechanism exists for the OSDs filesystem, you have
 to be
 able to recover file by their filename, otherwise you might
 end up
 mixing the content of various deleted RBDs. Due to the thin
 provisioning there might be some chunks missing (e.g. never
 allocated
 before).

 Given the fact that
 - you probably use XFS on the OSDs since it is the
 preferred
 filesystem for OSDs (there is RDR-XFS, but Ive never had to
 use it)
 - you would need to stop the complete ceph cluster
 (recovery tools do
 not work on mounted filesystems)

Re: [ceph-users] Recover Data from Deleted RBD Volume

[David]

On Mon, Aug 8, 2016 at 9:39 PM, Georgios Dimitrakakis 
wrote:

> Dear David (and all),
>
> the data are considered very critical therefore all this attempt to
> recover them.
>
> Although the cluster hasn't been fully stopped all users actions have. I
> mean services are running but users are not able to read/write/delete.
>
> The deleted image was the exact same size of the example (500GB) but it
> wasn't the only one deleted today. Our user was trying to do a "massive"
> cleanup by deleting 11 volumes and unfortunately one of them was very
> important.
>
> Let's assume that I "dd" all the drives what further actions should I do
> to recover the files? Could you please elaborate a bit more on the phrase
> "If you've never deleted any other rbd images and assuming you can recover
> data with names, you may be able to find the rbd objects"??
>

Sorry that last comment was a bit confusing, I was suggesting at this stage
you just need to concentrate on recovering everything you can and then try
and find the data you need.

The dd is to make a backup of the partition so you can work on it safely.
Ideally you would make a 2nd copy of the dd'd partition and work on that.
Then you need to find tools to attempt the recovery which is going to be
slow and painful and not guaranteed to be successful.



>
> Do you mean that if I know the file names I can go through and check for
> them? How?
> Do I have to know *all* file names or by searching for a few of them I can
> find all data that exist?
>
> Thanks a lot for taking the time to answer my questions!
>
> All the best,
>
> G.
>
> I dont think theres a way of getting the prefix from the cluster at
>> this point.
>>
>> If the deleted image was a similar size to the example youve given,
>> you will likely have had objects on every OSD. If this data is
>> absolutely critical you need to stop your cluster immediately or make
>> copies of all the drives with something like dd. If youve never
>> deleted any other rbd images and assuming you can recover data with
>> names, you may be able to find the rbd objects.
>>
>> On Mon, Aug 8, 2016 at 7:28 PM, Georgios Dimitrakakis  wrote:
>>
>> Hi,
>
> On 08.08.2016 10:50, Georgios Dimitrakakis wrote:
>
> Hi,
>>>
>>> On 08.08.2016 09:58, Georgios Dimitrakakis wrote:
>>>
>>> Dear all,

 I would like your help with an emergency issue but first
 let me describe our environment.

 Our environment consists of 2OSD nodes with 10x 2TB HDDs
 each and 3MON nodes (2 of them are the OSD nodes as well)
 all with ceph version 0.80.9
 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)

 This environment provides RBD volumes to an OpenStack
 Icehouse installation.

 Although not a state of the art environment is working
 well and within our expectations.

 The issue now is that one of our users accidentally
 deleted one of the volumes without keeping its data first!

 Is there any way (since the data are considered critical
 and very important) to recover them from CEPH?

>>>
>>> Short answer: no
>>>
>>> Long answer: no, but
>>>
>>> Consider the way Ceph stores data... each RBD is striped
>>> into chunks
>>> (RADOS objects with 4MB size by default); the chunks are
>>> distributed
>>> among the OSDs with the configured number of replicates
>>> (probably two
>>> in your case since you use 2 OSD hosts). RBD uses thin
>>> provisioning,
>>> so chunks are allocated upon first write access.
>>> If an RBD is deleted all of its chunks are deleted on the
>>> corresponding OSDs. If you want to recover a deleted RBD,
>>> you need to
>>> recover all individual chunks. Whether this is possible
>>> depends on
>>> your filesystem and whether the space of a former chunk is
>>> already
>>> assigned to other RADOS objects. The RADOS object names are
>>> composed
>>> of the RBD name and the offset position of the chunk, so if
>>> an
>>> undelete mechanism exists for the OSDs filesystem, you have
>>> to be
>>> able to recover file by their filename, otherwise you might
>>> end up
>>> mixing the content of various deleted RBDs. Due to the thin
>>> provisioning there might be some chunks missing (e.g. never
>>> allocated
>>> before).
>>>
>>> Given the fact that
>>> - you probably use XFS on the OSDs since it is the
>>> preferred
>>> filesystem for OSDs (there is RDR-XFS, but Ive never had to
>>>
>>> use it)
>>> - you would need to stop the complete ceph cluster
>>> (recovery tools do
>>> not work on mounted filesystems)
>>> - your cluster has been in use after the RBD was deleted
>>> and thus
>>> parts of its former space might already have been
>>> overwritten
>>> 

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Jason Dillaman]

On Mon, Aug 8, 2016 at 5:39 PM, Jason Dillaman  wrote:
> Unfortunately, for v2 RBD images, this image name to image id mapping
> is stored in the LevelDB database within the OSDs and I don't know,
> offhand, how to attempt to recover deleted values from there.

Actually, to correct myself, the "rbd_id." object just
writes the image id in binary to the file. So if you can recover that
file and retrieve its contents, you can again determine the block name
prefix in the form of "rbd_data..".

-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Jason Dillaman]

All RBD images use a backing RADOS object to facilitate mapping
between the external image name and the internal image id.  For v1
images this object would be named ".rbd" and for v2 images
 this object would be named "rbd_id.". You would need to
find this deleted object first in order to start figuring out the
internal image id.

For example, if image "test" was a v1 RBD image in the pool rbd, you would run:
# ceph osd map rbd test.rbd
osdmap e9 pool 'rbd' (0) object 'rbd_id.test' -> pg 0.9a2f7478 (0.0)
-> up ([0,2,1], p0) acting ([0,2,1], p0)

In this example, the object would be placed in PG 0.0 on OSD 0, 2, and
1. Since this object still exists for me, I can locate it on the OSD
disk:
# find /var/lib/ceph/osd/ceph-0/current/0.0_head/ -name "test.rbd*"
/var/lib/ceph/osd/ceph-0/current/0.0_head/test.rbd__head_9A2F7478__0

In your case, you would need to search for a deleted file substring
matching your image name within the appropriate PG directory on the
associated OSDs.

If you can recover the v1 RBD header (e.g. test.rbd), you can dump its
contents and extract the block name prefix. In my example, if I
hexdump my v1 header object, I see:
# hexdump -C 
/var/lib/ceph/osd/ceph-0/current/0.0_head/test.rbd__head_9A2F7478__0
  3c 3c 3c 20 52 61 64 6f  73 20 42 6c 6f 63 6b 20  |<<< Rados Block |
0010  44 65 76 69 63 65 20 49  6d 61 67 65 20 3e 3e 3e  |Device Image >>>|
0020  0a 00 00 00 00 00 00 00  72 62 2e 30 2e 31 30 31  |rb.0.101|
0030  30 2e 37 34 62 30 64 63  35 31 00 00 00 00 00 00  |0.74b0dc51..|
0040  52 42 44 00 30 30 31 2e  30 30 35 00 16 00 00 00  |RBD.001.005.|
0050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ||
*
0070

All the data blocks associated with this image will be named with a
"rb.0.1010.74b0dc51." prefix followed by a 12 character, zero-padded,
hexadecimal number representing the object offset within the image.
For example, assuming the default 4MB object size,
"rb.0.1010.74b0dc51.00cf" would represent the offset 188MB
through 192MB within the image.

For each one of these object names, you would need to run the "ceph
osd map" command to determine where these data blocks would have lived
and then attempt to undelete them.

Unfortunately, for v2 RBD images, this image name to image id mapping
is stored in the LevelDB database within the OSDs and I don't know,
offhand, how to attempt to recover deleted values from there.


On Mon, Aug 8, 2016 at 4:39 PM, Georgios Dimitrakakis
 wrote:
> Dear David (and all),
>
> the data are considered very critical therefore all this attempt to recover
> them.
>
> Although the cluster hasn't been fully stopped all users actions have. I
> mean services are running but users are not able to read/write/delete.
>
> The deleted image was the exact same size of the example (500GB) but it
> wasn't the only one deleted today. Our user was trying to do a "massive"
> cleanup by deleting 11 volumes and unfortunately one of them was very
> important.
>
> Let's assume that I "dd" all the drives what further actions should I do to
> recover the files? Could you please elaborate a bit more on the phrase "If
> you've never deleted any other rbd images and assuming you can recover data
> with names, you may be able to find the rbd objects"??
>
> Do you mean that if I know the file names I can go through and check for
> them? How?
> Do I have to know *all* file names or by searching for a few of them I can
> find all data that exist?
>
> Thanks a lot for taking the time to answer my questions!
>
> All the best,
>
> G.
>
>> I dont think theres a way of getting the prefix from the cluster at
>> this point.
>>
>> If the deleted image was a similar size to the example youve given,
>> you will likely have had objects on every OSD. If this data is
>> absolutely critical you need to stop your cluster immediately or make
>> copies of all the drives with something like dd. If youve never
>> deleted any other rbd images and assuming you can recover data with
>> names, you may be able to find the rbd objects.
>>
>> On Mon, Aug 8, 2016 at 7:28 PM, Georgios Dimitrakakis  wrote:
>>
> Hi,
>
> On 08.08.2016 10:50, Georgios Dimitrakakis wrote:
>
>>> Hi,
>>>
>>> On 08.08.2016 09:58, Georgios Dimitrakakis wrote:
>>>
 Dear all,

 I would like your help with an emergency issue but first
 let me describe our environment.

 Our environment consists of 2OSD nodes with 10x 2TB HDDs
 each and 3MON nodes (2 of them are the OSD nodes as well)
 all with ceph version 0.80.9
 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)

 This environment provides RBD volumes to an OpenStack
 Icehouse installation.

 Although not a state of the art environment is working
 well and within our expectations.

 The issue now is that one of our users 

Re: [ceph-users] Recover Data from Deleted RBD Volume

[George Mihaiescu]

Look in the cinder db, the volumes table to find the Uuid of the deleted 
volume. 

If you go through yours OSDs and look for the directories for PG index 20, you 
might find some fragments from the deleted volume, but it's a long shot...

> On Aug 8, 2016, at 4:39 PM, Georgios Dimitrakakis  
> wrote:
> 
> Dear David (and all),
> 
> the data are considered very critical therefore all this attempt to recover 
> them.
> 
> Although the cluster hasn't been fully stopped all users actions have. I mean 
> services are running but users are not able to read/write/delete.
> 
> The deleted image was the exact same size of the example (500GB) but it 
> wasn't the only one deleted today. Our user was trying to do a "massive" 
> cleanup by deleting 11 volumes and unfortunately one of them was very 
> important.
> 
> Let's assume that I "dd" all the drives what further actions should I do to 
> recover the files? Could you please elaborate a bit more on the phrase "If 
> you've never deleted any other rbd images and assuming you can recover data 
> with names, you may be able to find the rbd objects"??
> 
> Do you mean that if I know the file names I can go through and check for 
> them? How?
> Do I have to know *all* file names or by searching for a few of them I can 
> find all data that exist?
> 
> Thanks a lot for taking the time to answer my questions!
> 
> All the best,
> 
> G.
> 
>> I dont think theres a way of getting the prefix from the cluster at
>> this point.
>> 
>> If the deleted image was a similar size to the example youve given,
>> you will likely have had objects on every OSD. If this data is
>> absolutely critical you need to stop your cluster immediately or make
>> copies of all the drives with something like dd. If youve never
>> deleted any other rbd images and assuming you can recover data with
>> names, you may be able to find the rbd objects.
>> 
>> On Mon, Aug 8, 2016 at 7:28 PM, Georgios Dimitrakakis  wrote:
>> 
> Hi,
> 
> On 08.08.2016 10:50, Georgios Dimitrakakis wrote:
> 
>>> Hi,
>>> 
 On 08.08.2016 09:58, Georgios Dimitrakakis wrote:
 
 Dear all,
 
 I would like your help with an emergency issue but first
 let me describe our environment.
 
 Our environment consists of 2OSD nodes with 10x 2TB HDDs
 each and 3MON nodes (2 of them are the OSD nodes as well)
 all with ceph version 0.80.9
 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)
 
 This environment provides RBD volumes to an OpenStack
 Icehouse installation.
 
 Although not a state of the art environment is working
 well and within our expectations.
 
 The issue now is that one of our users accidentally
 deleted one of the volumes without keeping its data first!
 
 Is there any way (since the data are considered critical
 and very important) to recover them from CEPH?
>>> 
>>> Short answer: no
>>> 
>>> Long answer: no, but
>>> 
>>> Consider the way Ceph stores data... each RBD is striped
>>> into chunks
>>> (RADOS objects with 4MB size by default); the chunks are
>>> distributed
>>> among the OSDs with the configured number of replicates
>>> (probably two
>>> in your case since you use 2 OSD hosts). RBD uses thin
>>> provisioning,
>>> so chunks are allocated upon first write access.
>>> If an RBD is deleted all of its chunks are deleted on the
>>> corresponding OSDs. If you want to recover a deleted RBD,
>>> you need to
>>> recover all individual chunks. Whether this is possible
>>> depends on
>>> your filesystem and whether the space of a former chunk is
>>> already
>>> assigned to other RADOS objects. The RADOS object names are
>>> composed
>>> of the RBD name and the offset position of the chunk, so if
>>> an
>>> undelete mechanism exists for the OSDs filesystem, you have
>>> to be
>>> able to recover file by their filename, otherwise you might
>>> end up
>>> mixing the content of various deleted RBDs. Due to the thin
>>> provisioning there might be some chunks missing (e.g. never
>>> allocated
>>> before).
>>> 
>>> Given the fact that
>>> - you probably use XFS on the OSDs since it is the
>>> preferred
>>> filesystem for OSDs (there is RDR-XFS, but Ive never had to
>>> use it)
>>> - you would need to stop the complete ceph cluster
>>> (recovery tools do
>>> not work on mounted filesystems)
>>> - your cluster has been in use after the RBD was deleted
>>> and thus
>>> parts of its former space might already have been
>>> overwritten
>>> (replication might help you here, since there are two OSDs
>>> to try)
>>> - XFS undelete does not work well on fragmented files (and
>>> OSDs tend
>>> to introduce 

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Georgios Dimitrakakis]

Dear David (and all),

the data are considered very critical therefore all this attempt to 
recover them.


Although the cluster hasn't been fully stopped all users actions have. 
I mean services are running but users are not able to read/write/delete.


The deleted image was the exact same size of the example (500GB) but it 
wasn't the only one deleted today. Our user was trying to do a "massive" 
cleanup by deleting 11 volumes and unfortunately one of them was very 
important.


Let's assume that I "dd" all the drives what further actions should I 
do to recover the files? Could you please elaborate a bit more on the 
phrase "If you've never deleted any other rbd images and assuming you 
can recover data with names, you may be able to find the rbd objects"??


Do you mean that if I know the file names I can go through and check 
for them? How?
Do I have to know *all* file names or by searching for a few of them I 
can find all data that exist?


Thanks a lot for taking the time to answer my questions!

All the best,

G.


I dont think theres a way of getting the prefix from the cluster at
this point.

If the deleted image was a similar size to the example youve given,
you will likely have had objects on every OSD. If this data is
absolutely critical you need to stop your cluster immediately or make
copies of all the drives with something like dd. If youve never
deleted any other rbd images and assuming you can recover data with
names, you may be able to find the rbd objects.

On Mon, Aug 8, 2016 at 7:28 PM, Georgios Dimitrakakis  wrote:


Hi,

On 08.08.2016 10:50, Georgios Dimitrakakis wrote:


Hi,

On 08.08.2016 09:58, Georgios Dimitrakakis wrote:


Dear all,

I would like your help with an emergency issue but first
let me describe our environment.

Our environment consists of 2OSD nodes with 10x 2TB HDDs
each and 3MON nodes (2 of them are the OSD nodes as well)
all with ceph version 0.80.9
(b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)

This environment provides RBD volumes to an OpenStack
Icehouse installation.

Although not a state of the art environment is working
well and within our expectations.

The issue now is that one of our users accidentally
deleted one of the volumes without keeping its data first!

Is there any way (since the data are considered critical
and very important) to recover them from CEPH?


Short answer: no

Long answer: no, but

Consider the way Ceph stores data... each RBD is striped
into chunks
(RADOS objects with 4MB size by default); the chunks are
distributed
among the OSDs with the configured number of replicates
(probably two
in your case since you use 2 OSD hosts). RBD uses thin
provisioning,
so chunks are allocated upon first write access.
If an RBD is deleted all of its chunks are deleted on the
corresponding OSDs. If you want to recover a deleted RBD,
you need to
recover all individual chunks. Whether this is possible
depends on
your filesystem and whether the space of a former chunk is
already
assigned to other RADOS objects. The RADOS object names are
composed
of the RBD name and the offset position of the chunk, so if
an
undelete mechanism exists for the OSDs filesystem, you have
to be
able to recover file by their filename, otherwise you might
end up
mixing the content of various deleted RBDs. Due to the thin
provisioning there might be some chunks missing (e.g. never
allocated
before).

Given the fact that
- you probably use XFS on the OSDs since it is the
preferred
filesystem for OSDs (there is RDR-XFS, but Ive never had to
use it)
- you would need to stop the complete ceph cluster
(recovery tools do
not work on mounted filesystems)
- your cluster has been in use after the RBD was deleted
and thus
parts of its former space might already have been
overwritten
(replication might help you here, since there are two OSDs
to try)
- XFS undelete does not work well on fragmented files (and
OSDs tend
to introduce fragmentation...)

the answer is no, since it might not be feasible and the
chance of
success are way too low.

If you want to spend time on it I would propose the stop
the ceph
cluster as soon as possible, create copies of all involved
OSDs, start
the cluster again and attempt the recovery on the copies.

Regards,
Burkhard


Hi! Thanks for the info...I understand that this is a very
difficult and probably not feasible task but in case I need to
try a recovery what other info should I need? Can I somehow
find out on which OSDs the specific data were stored and
minimize my search there?
Any ideas on how should I proceed?

First of all you need to know the exact object names for the
RADOS
objects. As mentioned before, the name is composed of the RBD
name and
an offset.

In case of OpenStack, there are three different patterns for
RBD names:

, e.g. 50f2a0bd-15b1-4dbb-8d1f-fc43ce535f13
for glance images,
, e.g. 9aec1f45-9053-461e-b176-c65c25a48794_disk for nova
images
, e.g. volume-0ca52f58-7e75-4b21-8b0f-39cbcd431c42 for
cinder volumes

(not considering 

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Georgios Dimitrakakis]

Hi,


On 08.08.2016 10:50, Georgios Dimitrakakis wrote:

Hi,


On 08.08.2016 09:58, Georgios Dimitrakakis wrote:

Dear all,

I would like your help with an emergency issue but first let me 
describe our environment.


Our environment consists of 2OSD nodes with 10x 2TB HDDs each and 
3MON nodes (2 of them are the OSD nodes as well) all with ceph 
version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)


This environment provides RBD volumes to an OpenStack Icehouse 
installation.


Although not a state of the art environment is working well and 
within our expectations.


The issue now is that one of our users accidentally deleted one 
of the volumes without keeping its data first!


Is there any way (since the data are considered critical and very 
important) to recover them from CEPH?


Short answer: no

Long answer: no, but

Consider the way Ceph stores data... each RBD is striped into 
chunks
(RADOS objects with 4MB size by default); the chunks are 
distributed
among the OSDs with the configured number of replicates (probably 
two
in your case since you use 2 OSD hosts). RBD uses thin 
provisioning,

so chunks are allocated upon first write access.
If an RBD is deleted all of its chunks are deleted on the
corresponding OSDs. If you want to recover a deleted RBD, you need 
to

recover all individual chunks. Whether this is possible depends on
your filesystem and whether the space of a former chunk is already
assigned to other RADOS objects. The RADOS object names are 
composed

of the RBD name and the offset position of the chunk, so if an
undelete mechanism exists for the OSDs' filesystem, you have to be
able to recover file by their filename, otherwise you might end up
mixing the content of various deleted RBDs. Due to the thin
provisioning there might be some chunks missing (e.g. never 
allocated

before).

Given the fact that
- you probably use XFS on the OSDs since it is the preferred
filesystem for OSDs (there is RDR-XFS, but I've never had to use 
it)
- you would need to stop the complete ceph cluster (recovery tools 
do

not work on mounted filesystems)
- your cluster has been in use after the RBD was deleted and thus
parts of its former space might already have been overwritten
(replication might help you here, since there are two OSDs to try)
- XFS undelete does not work well on fragmented files (and OSDs 
tend

to introduce fragmentation...)

the answer is no, since it might not be feasible and the chance of
success are way too low.

If you want to spend time on it I would propose the stop the ceph
cluster as soon as possible, create copies of all involved OSDs, 
start

the cluster again and attempt the recovery on the copies.

Regards,
Burkhard


Hi! Thanks for the info...I understand that this is a very 
difficult and probably not feasible task but in case I need to try a 
recovery what other info should I need? Can I somehow find out on 
which OSDs the specific data were stored and minimize my search 
there?

Any ideas on how should I proceed?

First of all you need to know the exact object names for the RADOS
objects. As mentioned before, the name is composed of the RBD name 
and

an offset.

In case of OpenStack, there are three different patterns for RBD 
names:


, e.g. 50f2a0bd-15b1-4dbb-8d1f-fc43ce535f13
for glance images,
, e.g. 9aec1f45-9053-461e-b176-c65c25a48794_disk for nova 
images

, e.g. volume-0ca52f58-7e75-4b21-8b0f-39cbcd431c42 for
cinder volumes

(not considering snapshots etc, which might use different patterns)

The RBD chunks are created using a certain prefix (using examples
from our openstack setup):

# rbd -p os-images info 8fa3d9eb-91ed-4c60-9550-a62f34aed014
rbd image '8fa3d9eb-91ed-4c60-9550-a62f34aed014':
size 446 MB in 56 objects
order 23 (8192 kB objects)
block_name_prefix: rbd_data.30e57d54dea573
format: 2
features: layering, striping
flags:
stripe unit: 8192 kB
stripe count: 1

# rados -p os-images ls | grep rbd_data.30e57d54dea573
rbd_data.30e57d54dea573.0015
rbd_data.30e57d54dea573.0008
rbd_data.30e57d54dea573.000a
rbd_data.30e57d54dea573.002d
rbd_data.30e57d54dea573.0032

I don't know how whether the prefix is derived from some other
information, but the recover the RBD you definitely need it.

_If_ you are able to recover the prefix, you can use 'ceph osd map'
to find the OSDs for each chunk:

# ceph osd map os-images rbd_data.30e57d54dea573.001a
osdmap e418590 pool 'os-images' (38) object
'rbd_data.30e57d54dea573.001a' -> pg 38.d5d81d65 (38.65)
-> up ([45,17,108], p45) acting ([45,17,108], p45)

With 20 OSDs in your case you will likely have to process all of 
them

if the RBD has a size of several GBs.

Regards,
Burkhard



Is it possible to get the prefix if the RBD has been deleted
already?? Is this info somewhere stored? Can I retrieve it with
another way besides "rbd info"? Because when I try to get it using 
the

"rbd info" 

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Georgios Dimitrakakis]

Hi,


On 08.08.2016 10:50, Georgios Dimitrakakis wrote:

Hi,


On 08.08.2016 09:58, Georgios Dimitrakakis wrote:

Dear all,

I would like your help with an emergency issue but first let me 
describe our environment.


Our environment consists of 2OSD nodes with 10x 2TB HDDs each and 
3MON nodes (2 of them are the OSD nodes as well) all with ceph 
version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)


This environment provides RBD volumes to an OpenStack Icehouse 
installation.


Although not a state of the art environment is working well and 
within our expectations.


The issue now is that one of our users accidentally deleted one of 
the volumes without keeping its data first!


Is there any way (since the data are considered critical and very 
important) to recover them from CEPH?


Short answer: no

Long answer: no, but

Consider the way Ceph stores data... each RBD is striped into 
chunks
(RADOS objects with 4MB size by default); the chunks are 
distributed
among the OSDs with the configured number of replicates (probably 
two
in your case since you use 2 OSD hosts). RBD uses thin 
provisioning,

so chunks are allocated upon first write access.
If an RBD is deleted all of its chunks are deleted on the
corresponding OSDs. If you want to recover a deleted RBD, you need 
to

recover all individual chunks. Whether this is possible depends on
your filesystem and whether the space of a former chunk is already
assigned to other RADOS objects. The RADOS object names are 
composed

of the RBD name and the offset position of the chunk, so if an
undelete mechanism exists for the OSDs' filesystem, you have to be
able to recover file by their filename, otherwise you might end up
mixing the content of various deleted RBDs. Due to the thin
provisioning there might be some chunks missing (e.g. never 
allocated

before).

Given the fact that
- you probably use XFS on the OSDs since it is the preferred
filesystem for OSDs (there is RDR-XFS, but I've never had to use 
it)
- you would need to stop the complete ceph cluster (recovery tools 
do

not work on mounted filesystems)
- your cluster has been in use after the RBD was deleted and thus
parts of its former space might already have been overwritten
(replication might help you here, since there are two OSDs to try)
- XFS undelete does not work well on fragmented files (and OSDs 
tend

to introduce fragmentation...)

the answer is no, since it might not be feasible and the chance of
success are way too low.

If you want to spend time on it I would propose the stop the ceph
cluster as soon as possible, create copies of all involved OSDs, 
start

the cluster again and attempt the recovery on the copies.

Regards,
Burkhard


Hi! Thanks for the info...I understand that this is a very difficult 
and probably not feasible task but in case I need to try a recovery 
what other info should I need? Can I somehow find out on which OSDs 
the specific data were stored and minimize my search there?

Any ideas on how should I proceed?

First of all you need to know the exact object names for the RADOS
objects. As mentioned before, the name is composed of the RBD name 
and

an offset.

In case of OpenStack, there are three different patterns for RBD 
names:


, e.g. 50f2a0bd-15b1-4dbb-8d1f-fc43ce535f13
for glance images,
, e.g. 9aec1f45-9053-461e-b176-c65c25a48794_disk for nova 
images

, e.g. volume-0ca52f58-7e75-4b21-8b0f-39cbcd431c42 for
cinder volumes

(not considering snapshots etc, which might use different patterns)

The RBD chunks are created using a certain prefix (using examples
from our openstack setup):

# rbd -p os-images info 8fa3d9eb-91ed-4c60-9550-a62f34aed014
rbd image '8fa3d9eb-91ed-4c60-9550-a62f34aed014':
size 446 MB in 56 objects
order 23 (8192 kB objects)
block_name_prefix: rbd_data.30e57d54dea573
format: 2
features: layering, striping
flags:
stripe unit: 8192 kB
stripe count: 1

# rados -p os-images ls | grep rbd_data.30e57d54dea573
rbd_data.30e57d54dea573.0015
rbd_data.30e57d54dea573.0008
rbd_data.30e57d54dea573.000a
rbd_data.30e57d54dea573.002d
rbd_data.30e57d54dea573.0032

I don't know how whether the prefix is derived from some other
information, but the recover the RBD you definitely need it.

_If_ you are able to recover the prefix, you can use 'ceph osd map'
to find the OSDs for each chunk:

# ceph osd map os-images rbd_data.30e57d54dea573.001a
osdmap e418590 pool 'os-images' (38) object
'rbd_data.30e57d54dea573.001a' -> pg 38.d5d81d65 (38.65)
-> up ([45,17,108], p45) acting ([45,17,108], p45)

With 20 OSDs in your case you will likely have to process all of them
if the RBD has a size of several GBs.

Regards,
Burkhard



Is it possible to get the prefix if the RBD has been deleted already?? 
Is this info somewhere stored? Can I retrieve it with another way 
besides "rbd info"? Because when I try to get it using the "rbd info" 

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Burkhard Linke]

Hi,


On 08.08.2016 10:50, Georgios Dimitrakakis wrote:

Hi,


On 08.08.2016 09:58, Georgios Dimitrakakis wrote:

Dear all,

I would like your help with an emergency issue but first let me 
describe our environment.


Our environment consists of 2OSD nodes with 10x 2TB HDDs each and 
3MON nodes (2 of them are the OSD nodes as well) all with ceph 
version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)


This environment provides RBD volumes to an OpenStack Icehouse 
installation.


Although not a state of the art environment is working well and 
within our expectations.


The issue now is that one of our users accidentally deleted one of 
the volumes without keeping its data first!


Is there any way (since the data are considered critical and very 
important) to recover them from CEPH?


Short answer: no

Long answer: no, but

Consider the way Ceph stores data... each RBD is striped into chunks
(RADOS objects with 4MB size by default); the chunks are distributed
among the OSDs with the configured number of replicates (probably two
in your case since you use 2 OSD hosts). RBD uses thin provisioning,
so chunks are allocated upon first write access.
If an RBD is deleted all of its chunks are deleted on the
corresponding OSDs. If you want to recover a deleted RBD, you need to
recover all individual chunks. Whether this is possible depends on
your filesystem and whether the space of a former chunk is already
assigned to other RADOS objects. The RADOS object names are composed
of the RBD name and the offset position of the chunk, so if an
undelete mechanism exists for the OSDs' filesystem, you have to be
able to recover file by their filename, otherwise you might end up
mixing the content of various deleted RBDs. Due to the thin
provisioning there might be some chunks missing (e.g. never allocated
before).

Given the fact that
- you probably use XFS on the OSDs since it is the preferred
filesystem for OSDs (there is RDR-XFS, but I've never had to use it)
- you would need to stop the complete ceph cluster (recovery tools do
not work on mounted filesystems)
- your cluster has been in use after the RBD was deleted and thus
parts of its former space might already have been overwritten
(replication might help you here, since there are two OSDs to try)
- XFS undelete does not work well on fragmented files (and OSDs tend
to introduce fragmentation...)

the answer is no, since it might not be feasible and the chance of
success are way too low.

If you want to spend time on it I would propose the stop the ceph
cluster as soon as possible, create copies of all involved OSDs, start
the cluster again and attempt the recovery on the copies.

Regards,
Burkhard


Hi! Thanks for the info...I understand that this is a very difficult 
and probably not feasible task but in case I need to try a recovery 
what other info should I need? Can I somehow find out on which OSDs 
the specific data were stored and minimize my search there?

Any ideas on how should I proceed?
First of all you need to know the exact object names for the RADOS 
objects. As mentioned before, the name is composed of the RBD name and 
an offset.


In case of OpenStack, there are three different patterns for RBD names:

, e.g. 50f2a0bd-15b1-4dbb-8d1f-fc43ce535f13  
for glance images,

, e.g. 9aec1f45-9053-461e-b176-c65c25a48794_disk for nova images
, e.g. volume-0ca52f58-7e75-4b21-8b0f-39cbcd431c42 for 
cinder volumes


(not considering snapshots etc, which might use different patterns)

The RBD chunks are created using a certain prefix (using examples from 
our openstack setup):


# rbd -p os-images info 8fa3d9eb-91ed-4c60-9550-a62f34aed014
rbd image '8fa3d9eb-91ed-4c60-9550-a62f34aed014':
size 446 MB in 56 objects
order 23 (8192 kB objects)
block_name_prefix: rbd_data.30e57d54dea573
format: 2
features: layering, striping
flags:
stripe unit: 8192 kB
stripe count: 1

# rados -p os-images ls | grep rbd_data.30e57d54dea573
rbd_data.30e57d54dea573.0015
rbd_data.30e57d54dea573.0008
rbd_data.30e57d54dea573.000a
rbd_data.30e57d54dea573.002d
rbd_data.30e57d54dea573.0032

I don't know how whether the prefix is derived from some other 
information, but the recover the RBD you definitely need it.


_If_ you are able to recover the prefix, you can use 'ceph osd map' to 
find the OSDs for each chunk:


# ceph osd map os-images rbd_data.30e57d54dea573.001a
osdmap e418590 pool 'os-images' (38) object 
'rbd_data.30e57d54dea573.001a' -> pg 38.d5d81d65 (38.65) -> 
up ([45,17,108], p45) acting ([45,17,108], p45)


With 20 OSDs in your case you will likely have to process all of them if 
the RBD has a size of several GBs.


Regards,
Burkhard
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recover Data from Deleted RBD Volume

[David]

That will be down to the pool the rbd was in, the crush rule for that pool
will dictate which osd's store objects. In a standard config that rbd will
likely have objects on every osd in your cluster.

On 8 Aug 2016 9:51 a.m., "Georgios Dimitrakakis" 
wrote:

> Hi,
>>
>>
>> On 08.08.2016 09:58, Georgios Dimitrakakis wrote:
>>
>>> Dear all,
>>>
>>> I would like your help with an emergency issue but first let me describe
>>> our environment.
>>>
>>> Our environment consists of 2OSD nodes with 10x 2TB HDDs each and 3MON
>>> nodes (2 of them are the OSD nodes as well) all with ceph version 0.80.9
>>> (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)
>>>
>>> This environment provides RBD volumes to an OpenStack Icehouse
>>> installation.
>>>
>>> Although not a state of the art environment is working well and within
>>> our expectations.
>>>
>>> The issue now is that one of our users accidentally deleted one of the
>>> volumes without keeping its data first!
>>>
>>> Is there any way (since the data are considered critical and very
>>> important) to recover them from CEPH?
>>>
>>
>> Short answer: no
>>
>> Long answer: no, but
>>
>> Consider the way Ceph stores data... each RBD is striped into chunks
>> (RADOS objects with 4MB size by default); the chunks are distributed
>> among the OSDs with the configured number of replicates (probably two
>> in your case since you use 2 OSD hosts). RBD uses thin provisioning,
>> so chunks are allocated upon first write access.
>> If an RBD is deleted all of its chunks are deleted on the
>> corresponding OSDs. If you want to recover a deleted RBD, you need to
>> recover all individual chunks. Whether this is possible depends on
>> your filesystem and whether the space of a former chunk is already
>> assigned to other RADOS objects. The RADOS object names are composed
>> of the RBD name and the offset position of the chunk, so if an
>> undelete mechanism exists for the OSDs' filesystem, you have to be
>> able to recover file by their filename, otherwise you might end up
>> mixing the content of various deleted RBDs. Due to the thin
>> provisioning there might be some chunks missing (e.g. never allocated
>> before).
>>
>> Given the fact that
>> - you probably use XFS on the OSDs since it is the preferred
>> filesystem for OSDs (there is RDR-XFS, but I've never had to use it)
>> - you would need to stop the complete ceph cluster (recovery tools do
>> not work on mounted filesystems)
>> - your cluster has been in use after the RBD was deleted and thus
>> parts of its former space might already have been overwritten
>> (replication might help you here, since there are two OSDs to try)
>> - XFS undelete does not work well on fragmented files (and OSDs tend
>> to introduce fragmentation...)
>>
>> the answer is no, since it might not be feasible and the chance of
>> success are way too low.
>>
>> If you want to spend time on it I would propose the stop the ceph
>> cluster as soon as possible, create copies of all involved OSDs, start
>> the cluster again and attempt the recovery on the copies.
>>
>> Regards,
>> Burkhard
>>
>
> Hi! Thanks for the info...I understand that this is a very difficult and
> probably not feasible task but in case I need to try a recovery what other
> info should I need? Can I somehow find out on which OSDs the specific data
> were stored and minimize my search there?
> Any ideas on how should I proceed?
>
>
> Best,
>
> G.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Georgios Dimitrakakis]

Hi,


On 08.08.2016 09:58, Georgios Dimitrakakis wrote:

Dear all,

I would like your help with an emergency issue but first let me 
describe our environment.


Our environment consists of 2OSD nodes with 10x 2TB HDDs each and 
3MON nodes (2 of them are the OSD nodes as well) all with ceph version 
0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)


This environment provides RBD volumes to an OpenStack Icehouse 
installation.


Although not a state of the art environment is working well and 
within our expectations.


The issue now is that one of our users accidentally deleted one of 
the volumes without keeping its data first!


Is there any way (since the data are considered critical and very 
important) to recover them from CEPH?


Short answer: no

Long answer: no, but

Consider the way Ceph stores data... each RBD is striped into chunks
(RADOS objects with 4MB size by default); the chunks are distributed
among the OSDs with the configured number of replicates (probably two
in your case since you use 2 OSD hosts). RBD uses thin provisioning,
so chunks are allocated upon first write access.
If an RBD is deleted all of its chunks are deleted on the
corresponding OSDs. If you want to recover a deleted RBD, you need to
recover all individual chunks. Whether this is possible depends on
your filesystem and whether the space of a former chunk is already
assigned to other RADOS objects. The RADOS object names are composed
of the RBD name and the offset position of the chunk, so if an
undelete mechanism exists for the OSDs' filesystem, you have to be
able to recover file by their filename, otherwise you might end up
mixing the content of various deleted RBDs. Due to the thin
provisioning there might be some chunks missing (e.g. never allocated
before).

Given the fact that
- you probably use XFS on the OSDs since it is the preferred
filesystem for OSDs (there is RDR-XFS, but I've never had to use it)
- you would need to stop the complete ceph cluster (recovery tools do
not work on mounted filesystems)
- your cluster has been in use after the RBD was deleted and thus
parts of its former space might already have been overwritten
(replication might help you here, since there are two OSDs to try)
- XFS undelete does not work well on fragmented files (and OSDs tend
to introduce fragmentation...)

the answer is no, since it might not be feasible and the chance of
success are way too low.

If you want to spend time on it I would propose the stop the ceph
cluster as soon as possible, create copies of all involved OSDs, 
start

the cluster again and attempt the recovery on the copies.

Regards,
Burkhard


Hi! Thanks for the info...I understand that this is a very difficult 
and probably not feasible task but in case I need to try a recovery what 
other info should I need? Can I somehow find out on which OSDs the 
specific data were stored and minimize my search there?

Any ideas on how should I proceed?


Best,

G.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recover Data from Deleted RBD Volume

[Burkhard Linke]

Hi,


On 08.08.2016 09:58, Georgios Dimitrakakis wrote:

Dear all,

I would like your help with an emergency issue but first let me 
describe our environment.


Our environment consists of 2OSD nodes with 10x 2TB HDDs each and 3MON 
nodes (2 of them are the OSD nodes as well) all with ceph version 
0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)


This environment provides RBD volumes to an OpenStack Icehouse 
installation.


Although not a state of the art environment is working well and within 
our expectations.


The issue now is that one of our users accidentally deleted one of the 
volumes without keeping its data first!


Is there any way (since the data are considered critical and very 
important) to recover them from CEPH?


Short answer: no

Long answer: no, but

Consider the way Ceph stores data... each RBD is striped into chunks 
(RADOS objects with 4MB size by default); the chunks are distributed 
among the OSDs with the configured number of replicates (probably two in 
your case since you use 2 OSD hosts). RBD uses thin provisioning, so 
chunks are allocated upon first write access.
If an RBD is deleted all of its chunks are deleted on the corresponding 
OSDs. If you want to recover a deleted RBD, you need to recover all 
individual chunks. Whether this is possible depends on your filesystem 
and whether the space of a former chunk is already assigned to other 
RADOS objects. The RADOS object names are composed of the RBD name and 
the offset position of the chunk, so if an undelete mechanism exists for 
the OSDs' filesystem, you have to be able to recover file by their 
filename, otherwise you might end up mixing the content of various 
deleted RBDs. Due to the thin provisioning there might be some chunks 
missing (e.g. never allocated before).


Given the fact that
- you probably use XFS on the OSDs since it is the preferred filesystem 
for OSDs (there is RDR-XFS, but I've never had to use it)
- you would need to stop the complete ceph cluster (recovery tools do 
not work on mounted filesystems)
- your cluster has been in use after the RBD was deleted and thus parts 
of its former space might already have been overwritten (replication 
might help you here, since there are two OSDs to try)
- XFS undelete does not work well on fragmented files (and OSDs tend to 
introduce fragmentation...)


the answer is no, since it might not be feasible and the chance of 
success are way too low.


If you want to spend time on it I would propose the stop the ceph 
cluster as soon as possible, create copies of all involved OSDs, start 
the cluster again and attempt the recovery on the copies.


Regards,
Burkhard
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com