Re: Lost LVM data - lost logical volume

[Lisi Reisz]

On Wednesday 28 October 2015 01:03:20 Doug wrote:
> it is now a system which even grandmothers are using.

Hey!!  Ada Lovelace was a grandmother.  

Lisi

Re: Lost LVM data - lost logical volume

[Lisi Reisz]

On Wednesday 28 October 2015 12:30:29 Martin Read wrote:
> On 28/10/15 11:14, Lisi Reisz wrote:
> > On Wednesday 28 October 2015 01:03:20 Doug wrote:
> >> it is now a system which even grandmothers are using.
> >
> > Hey!!  Ada Lovelace was a grandmother.
>
> Not while she was alive; sadly, she died before any of her children had
> children of their own.

Yes, very sad - but I don't believe that, had she lived, she would have turned 
into a clueless, helpless individual who couldn't use a computer without the 
aid of a male.

I didn't cross reference when she died and when her grandchildren were born.  
Thank you!

Lisi

Re: Lost LVM data - lost logical volume

[Martin Read]

On 28/10/15 11:14, Lisi Reisz wrote:

On Wednesday 28 October 2015 01:03:20 Doug wrote:

it is now a system which even grandmothers are using.


Hey!!  Ada Lovelace was a grandmother.


Not while she was alive; sadly, she died before any of her children had 
children of their own.

Re: Lost LVM data - lost logical volume

[John Hasler]

Lisi writes:
> Yes, very sad - but I don't believe that, had she lived, she would
> have turned into a clueless, helpless individual who couldn't use a
> computer without the aid of a male.

Not a male, a young person.  The myth is that everyone over 60 is
clueless when it comes to "technology" and that everyone under 30 is an
expert because they "grew up with it".
-- 
John Hasler 
jhas...@newsguy.com
Elmwood, WI USA

Re: Lost LVM data - lost logical volume

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, Oct 27, 2015 at 09:03:20PM -0400, Doug wrote:
> 
> 
> On 10/28/2015 12:39 AM, David Christensen wrote:
> >On 10/27/2015 06:10 PM, David Christensen wrote:
> >>You are free to study the source code and help the OP solve his problem.
> >
> >Sorry -- that sounds harsh.
> >
> >
> >I say "study the source code" [...]

> I am not a programmer [...]

Who is? I write programms and still the innards of GCC are very
difficult for me to get at.

> What percentage of Debian users does anyone reading this believe to
> be able to read the source code (if they even know where to look to
> find it) and then to modify it to suit their needs?
>
> Surely Linux was a programmer's wonderland when it came out in 1995,
> but it is now a system which even grandmothers are using. I would not
> ever suggest that it become closed source--as I suspect one or two
> distros may be on the way to--but it should be realized that its
> original wonderfulness is lost on the majority of modern-day users.

In one thing I agree: Free Software in itself is useless. You need
the users who know and appreciate the advantages.

In one thing I disagree most decidedly: being fatalist about the
"dumb users". We have to keep up easing access to others, so that
Free actually works, because it presupposes users who can, in a
pinch, help themselves.

Let's go teach grandma (why not grandpa as well?) functional
programming! [1]

- - - - - - - - -
[1] Yeah, oversimplified and that: but still: Free is useless if it
is inaccessible. There are many aspects to accessibility: good docs,
well-written programs and the saturday afternoon thing in which we
exchange cookies for "teach me how to write a bug report" (yes, this
saturday I'll have some cookies, looking forward).

regards
- -- tomás
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlYwfEQACgkQBcgs9XrR2kZ9zACfc9EVxFGYfXmR0MFxiNpm15+V
KSEAnRmv0io+FV7b2peN4C/1Z3r7MdWJ
=6Tps
-END PGP SIGNATURE-

Re: Lost LVM data - lost logical volume

[Lisi Reisz]

On Wednesday 28 October 2015 13:31:31 John Hasler wrote:
> Lisi writes:
> > Yes, very sad - but I don't believe that, had she lived, she would
> > have turned into a clueless, helpless individual who couldn't use a
> > computer without the aid of a male.
>
> Not a male, a young person.  The myth is that everyone over 60 is
> clueless when it comes to "technology" and that everyone under 30 is an
> expert because they "grew up with it".

:-)

Yes, I agree.  But grandmothers get maligned far more often than grandfathers.  
That is, when they are not maligning mothers.

From earlier in this thread:
"it is now a system which even grandmothers are using."

Yes, we are.  A lot of us.  Some of us even tinker with it.

Lisi

Sorry, John for off-list post.

Re: Lost LVM data - lost logical volume

[John Hasler]

Lisi writes:
> But grandmothers get maligned far more often than grandfathers.

They just get mentioned more often.  Grandfathers are mostly ignored.
-- 
John Hasler 
jhas...@newsguy.com
Elmwood, WI USA

Re: Lost LVM data - lost logical volume

[David Christensen]

On 10/28/2015 05:57 PM, Anton Bizzarri wrote:

The efforts so far have been,

- recreated the logical volume with the same drives (as well as the one
that had failed previously and now seemed to be working and is back in the
same place).

- I can see the logical volume in lvm as well as the device in
/dev/mapper/dm0

- I tried repairing it by running fsck and its full of inode failures and
errors.


Please "reply to list".


Looking at your first post, /dev/sd* identifiers are notorious for 
changing whenever a drive, adapter, and/or port changes.  Perhaps the 
drives are being enumerated in a different order now than they were when 
the LV worked.  Get the UUID for every drive.  Correlate those to drive 
makes, models, serial numbers, and ports.  Unplug all the drives.  Reset 
the CMOS via the motherboard jumper.   Reconnect all the drives. 
Reconfigure CMOS settings.  Boot.  Check UUID's again -- if they're the 
same, no luck; if they changed, perhaps.  If that doesn't fix it, tear 
down and rebuild LVM using UUID's for PV's..  If you can script the 
process, that will facilitate checking the rest of the permutations (5! 
= 240 total).




If anyone has any links to more authoritative sources or links or
suggestions let me know. I tried Red Hat LVM list and that proved to return
no results.

Its either a very common problem or its a dead problem.


I STFW for an LVM community the other night, but didn't find one.  My 
guess is that the code stabilized years ago, was rolled in the Linux 
kernel, and people moved on.  You might be able to find LVM people by 
contacting for the Debian "lvm2", etc., package maintainers and/or by 
searching the various Debian mailing lists.  Another place to look would 
be the Linux kernel version control system.



David

Re: Lost LVM data - lost logical volume

[Anton Bizzarri]

On Wed, Oct 28, 2015 at 9:30 PM, David Christensen <
dpchr...@holgerdanske.com> wrote:

>
> Agreed.  But, there's nothing like a business data loss event to convert
> them to the backup religion.  If budget is an issue, there are several
> "free" (as in beer) cloud storage and/or backup services to choose from.
>
>
> David
>
>
>
I will try the Debian lvm2 community. I am trying testdisk[1].

i want to exhaust all options before i decide to start rearranging the
drives and resetting BIOSs' and CMOSs'.

http://www.cgsecurity.org/wiki/TestDisk

Re: Lost LVM data - lost logical volume

[David Christensen]

On 10/28/2015 07:40 PM, Anton Bizzarri wrote:

I will try the Debian lvm2 community.


What is the URL?



 I am trying testdisk[1].

i want to exhaust all options before i decide to start rearranging the
drives and resetting BIOSs' and CMOSs'.

http://www.cgsecurity.org/wiki/TestDisk


The feature list is impressive.  Let us know if it works and what you 
think of the tool.



David

Re: Lost LVM data - lost logical volume

[David Christensen]

On 10/28/2015 05:52 PM, Anton Bizzarri wrote:

Yes, you pretty much nailed it. It's an easily $200 solution but  (some)
clients can't be bothered to spend it.  I am not going to buy them a $200
external drive out of my pocket.


Agreed.  But, there's nothing like a business data loss event to convert 
them to the backup religion.  If budget is an issue, there are several 
"free" (as in beer) cloud storage and/or backup services to choose from.



David

Re: Lost LVM data - lost logical volume

[David Christensen]

On 10/28/2015 07:27 PM, David Christensen wrote:

Check UUID's again


To clarify -- check UUID's against /dev/sd* assignments.


David

Re: Lost LVM data - lost logical volume

[Chris Bannister]

On Tue, Oct 27, 2015 at 09:03:20PM -0400, Doug wrote:
> I am not a programmer. (I am also not a user of Debian, but I do keep
> a watch here to see what might be going on in the other popular
> distros.) So I feel it is reasonable to put this out to the readers of
> this list:
> 
> What percentage of Debian users does anyone reading this believe to be
> able to read the source code (if they even know where to look to find
> it) and then to modify it to suit their needs? Surely Linux was a
> programmer's
> 
> wonderland when it came out in 1995, but it is now a system which even
> grandmothers are using. I would not ever suggest that it become closed
> source--as I suspect one or two distros may be on the way to--but it
> should be realized
> 
> that its original wonderfulness is lost on the majority of modern-day
> users.

Sadly, you maybe right. Remember there are distros, and there are
distros. Consider an analogy, off road vehicles --- there are families
which buy an off road vehicle because it is fashionable but all they
ever do is to drive to and fro from work and pick the kids up.

Then there are the people who actually take their vehicles off road in
the weekends and put them to their proper use for enjoyment.

So consider, for the first case that you compare it to Ubuntu and Linux 
Mint while for the second case you could compare it to Debian etc.

So, keeping that in mind, it's not at all obscure or weird to suggest
looking at the source code. 

-- 
"If you're not careful, the newspapers will have you hating the people
who are being oppressed, and loving the people who are doing the 
oppressing." --- Malcolm X

Re: Lost LVM data - lost logical volume

[Jonathan Dowland]

On Tue, Oct 27, 2015 at 09:03:20PM -0400, Doug wrote:
> What percentage of Debian users does anyone reading this believe to be able
> to read the source code (if they even know where to look to find it) and then
> to modify it to suit their needs? Surely Linux was a programmer's
> wonderland when it came out in 1995, but it is now a system which even
> grandmothers are using. I would not ever suggest that it become closed
> source--as I suspect one or two distros may be on the way to--but it should
> be realized
> that its original wonderfulness is lost on the majority of modern-day users.

Well, you're right that someone's grand parents shouldn't have to read source
code to use a desktop computer. But I wouldn't be expecting them to set up or
recover a striped LVM device either, which is what this thread is about.

David's point about the source code being the canonical home of what is going
on is a good one. And for a forensic operation, which is not something performed
by mere mortals, you want to be damned sure you know what is going on.

Re: Lost LVM data - lost logical volume

[Anton Bizzarri]

On Tue, Oct 27, 2015 at 6:50 AM, Jonathan Dowland  wrote:

>
> Lots of replies on a subject you don't know the answer for ;)
>
>
The efforts so far have been,

- recreated the logical volume with the same drives (as well as the one
that had failed previously and now seemed to be working and is back in the
same place).

- I can see the logical volume in lvm as well as the device in
/dev/mapper/dm0

- I tried repairing it by running fsck and its full of inode failures and
errors.

If anyone has any links to more authoritative sources or links or
suggestions let me know. I tried Red Hat LVM list and that proved to return
no results.

Its either a very common problem or its a dead problem.

Re: Lost LVM data - lost logical volume

[Jonathan Dowland]

On Mon, Oct 26, 2015 at 08:48:17PM -0700, David Christensen wrote:
> I don't know -- a sufficiently skilled person (or group) might be able to do
> it.  I'm not such a person, and don't know of any.  Try STFW for an LVM
> project or community.

Lots of replies on a subject you don't know the answer for ;)

Re: Lost LVM data - lost logical volume

[David Christensen]

On 10/27/2015 03:50 AM, Jonathan Dowland wrote:

Lots of replies on a subject you don't know the answer for


My view is that of an independent consultant who must deal with the 
harsh realities of time and money via contracts.  I responded because 
the OP made it sound like he was in a similar situation.



You are free to study the source code and help the OP solve his problem.


David

Re: Lost LVM data - lost logical volume

[David Christensen]

On 10/27/2015 06:10 PM, David Christensen wrote:

You are free to study the source code and help the OP solve his problem.


Sorry -- that sounds harsh.


I say "study the source code" because the source code is the canonical 
definition of what is actually going on.  If I can understand the source 
code, I can decode the bits and bytes in memory and on disk (e.g. data 
structures) and understand what operations are performed on them (e.g. 
algorithms).  The few times I have done this, I learned details that 
were not in the man pages.  I suspect that solving the OP's problem may 
require such knowledge.



David

Re: Lost LVM data - lost logical volume

[Doug]

On 10/28/2015 12:39 AM, David Christensen wrote:

On 10/27/2015 06:10 PM, David Christensen wrote:

You are free to study the source code and help the OP solve his problem.


Sorry -- that sounds harsh.


I say "study the source code" because the source code is the canonical 
definition of what is actually going on.  If I can understand the source code, I can 
decode the bits and bytes in memory and on disk (e.g. data structures) and understand 
what operations are performed on them (e.g. algorithms).  The few times I have done this, 
I learned details that were not in the man pages.  I suspect that solving the OP's 
problem may require such knowledge.


David




I am not a programmer. (I am also not a user of Debian, but I do keep a watch 
here to see what might be going on in the other popular distros.) So I feel it 
is reasonable to put this out to the readers of this list:

What percentage of Debian users does anyone reading this believe to be able to 
read the source code (if they even know where to look to find it) and then to 
modify it to suit their needs? Surely Linux was a programmer's

wonderland when it came out in 1995, but it is now a system which even 
grandmothers are using. I would not ever suggest that it become closed 
source--as I suspect one or two distros may be on the way to--but it should be 
realized

that its original wonderfulness is lost on the majority of modern-day users.

--doug

Re: Lost LVM data - lost logical volume

[David Christensen]

On 10/26/2015 10:14 AM, Anton Bizzarri wrote:

So just to confirm, there is no way to recover the logical volume.


I don't know -- a sufficiently skilled person (or group) might be able 
to do it.  I'm not such a person, and don't know of any.  Try STFW for 
an LVM project or community.



David

Re: Lost LVM data - lost logical volume

[Anton Bizzarri]

Hi David. I see what you mean.

So just to confirm, there is no way to recover the logical volume.

FWIW.. I am trying the following

I recreated the LV with lvcreate.

Luckily the old LV spanned the 100% the old LVM.

I can now see the LV.

I then ran ext3.fsck to repair /dev/mapper/data


latest errors i get are:
Oct 26 13:07:38 arbor kernel: [349276.552056] EXT3-fs error (device dm-0):
ext3_check_descriptors: Block bitmap for group 384 not in group (block 0)!
Oct 26 13:07:38 arbor kernel: [349276.559433] EXT3-fs (dm-0): error: group
descriptors corrupted

Then I am running ext3.fsck /dev/dm-0

and just waiting for it to finish.

FWIW. There are a multitude of reasons why there is no backup in place. In
this particular case, there is no SLA to cover the regular backup of 3TB of
data. So there is no money available to invest in hardware or services to
backup the data. The agreement is try "best effort" to recover data if it
goes down.






On Sun, Oct 25, 2015 at 6:44 PM, David Christensen <
dpchr...@holgerdanske.com> wrote:

> On 10/23/2015 03:56 AM, Anton Bizzarri wrote:
>
>> On Thu, Oct 22, 2015 at 9:55 PM, David Christensen <
>> dpchr...@holgerdanske.com> wrote:
>>
>>> You *do* have a back up of your data, right?
>>>
>> No. There is no backup. Is the data lost because it was a striped volume.?
>> Its not my server and I never worked on it before. I was not aware it was
>> a
>> stripe.
>>
>
> Please "reply to list".
>
>
> RAID0 and no backups?  The owner already decided to lose his data.
>
>
> If the owner has changed his mind and wants his data back, then I'd take
> two images of every drive in the server, store one set off-site, and have
> the owner send the server to a data recovery consultant or shop. In the
> mean time, now might be an opportune moment to implement a backup system.
>
>
> David
>
>

Re: Lost LVM data - lost logical volume

[Dan Ritter]

On Mon, Oct 26, 2015 at 01:14:59PM -0400, Anton Bizzarri wrote:
> Hi David. I see what you mean.
> 
> So just to confirm, there is no way to recover the logical volume.
> 
> FWIW.. I am trying the following
> 
> I recreated the LV with lvcreate.
> 
> Luckily the old LV spanned the 100% the old LVM.
> 
> I can now see the LV.
> 
> I then ran ext3.fsck to repair /dev/mapper/data
> 
> 
> latest errors i get are:
> Oct 26 13:07:38 arbor kernel: [349276.552056] EXT3-fs error (device dm-0):
> ext3_check_descriptors: Block bitmap for group 384 not in group (block 0)!
> Oct 26 13:07:38 arbor kernel: [349276.559433] EXT3-fs (dm-0): error: group
> descriptors corrupted
> 
> Then I am running ext3.fsck /dev/dm-0
> 
> and just waiting for it to finish.
> 
> FWIW. There are a multitude of reasons why there is no backup in place. In
> this particular case, there is no SLA to cover the regular backup of 3TB of
> data. So there is no money available to invest in hardware or services to
> backup the data. The agreement is try "best effort" to recover data if it
> goes down.
> 

A 3TB external disk on USB is well under $200. 

Was their data worth $200 a year?

No levels of RAID are equivalent to backup. 

-dsr-

Re: Lost LVM data - lost logical volume

[David Christensen]

On 10/23/2015 03:56 AM, Anton Bizzarri wrote:

On Thu, Oct 22, 2015 at 9:55 PM, David Christensen <
dpchr...@holgerdanske.com> wrote:

You *do* have a back up of your data, right?

No. There is no backup. Is the data lost because it was a striped volume.?
Its not my server and I never worked on it before. I was not aware it was a
stripe.


Please "reply to list".


RAID0 and no backups?  The owner already decided to lose his data.


If the owner has changed his mind and wants his data back, then I'd take 
two images of every drive in the server, store one set off-site, and 
have the owner send the server to a data recovery consultant or shop. 
In the mean time, now might be an opportune moment to implement a backup 
system.



David

Lost LVM data - lost logical volume

[Anton Bizzarri]

Hello, we had lost a drive in a LVM on Debian Squeeze. I pulled the drive
and replaced it and then tried to repair it but then when I tried to
recover the LVM it turned out it was a striped volume. (Its not my server!
Never thought they put the data on a striped volume).

So I returned the original drive and I tried to see if I can recover the
data from LVM metadata.

This is Debian wheezy and lvm2 2.02.95-8

The drive that failed was sdc

Below is my logs.

/# pvscan -v
Wiping cache of LVM-capable devices
Wiping internal VG cache
Walking through all physical volumes
  PV /dev/sdb   VG data   lvm2 [465.76 GiB / 465.76 GiB free]
  PV /dev/sdd   VG data   lvm2 [465.76 GiB / 465.76 GiB free]
  PV /dev/sde   VG data   lvm2 [465.76 GiB / 465.76 GiB free]
  PV /dev/sdf   VG data   lvm2 [465.76 GiB / 465.76 GiB free]
  PV /dev/sdc   VG data   lvm2 [465.76 GiB / 465.76 GiB free]
  Total: 5 [2.27 TiB] / in use: 5 [2.27 TiB] / in no VG: 0 [0   ]


:/# vgscan -v
Wiping cache of LVM-capable devices
Wiping internal VG cache
  Reading all physical volumes.  This may take a while...
Finding all volume groups
Finding volume group "data"
  Found volume group "data" using metadata type lvm2


/# lvscan -v
Finding all logical volumes

It is not finding the logical volumes anymore.

I have a file in the archive folder /etc/lvm/archive from the morning
before I ran "vgreduce --removemissing --force data"

Is there any way to restore the metadata? Below is the contents of archive
file.

# Generated by LVM2 version 2.02.95(2) (2012-03-06): Thu Oct 22 09:44:12
2015

contents = "Text Format Volume Group"
version = 1

description = "Created *before* executing 'vgreduce --removemissing --force
data'"

creation_host = "" # Linux  3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u5
x86_64
creation_time = 1445521452  # Thu Oct 22 09:44:12 2015

data {
id = "0UcXlh-1lXG-udrp-T6QJ-fqMt-G5mz-mgpLmN"
seqno = 6
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192  # 4 Megabytes
max_lv = 0
max_pv = 0
metadata_copies = 0

physical_volumes {

pv0 {
id = "mKQGfL-Fs0C-bRIv-rcKT-jbWs-7olM-R2VCCw"
device = "/dev/sdb" # Hint only

status = ["ALLOCATABLE"]
flags = []
dev_size = 976773168# 465.762 Gigabytes
pe_start = 384
pe_count = 119234   # 465.758 Gigabytes
}

pv1 {
id = "KLL51K-SwV9-6VU6-9ulL-DOJy-L9uJ-T5k43a"
device = "unknown device"   # Hint only

status = ["ALLOCATABLE"]
flags = ["MISSING"]
dev_size = 976773168# 465.762 Gigabytes
pe_start = 384
pe_count = 119234   # 465.758 Gigabytes
}

pv2 {
id = "98S7nn-R4sG-OMBc-HR9Y-kHPM-Du8q-ZBNJ3c"
device = "/dev/sdd" # Hint only
dev_size = 976773168# 465.762 Gigabytes
pe_start = 384
pe_count = 119234   # 465.758 Gigabytes
}

pv3 {
id = "d1CHzd-91uQ-7c0X-XA02-0Tlh-bBtv-rusTsj"
device = "/dev/sde" # Hint only

status = ["ALLOCATABLE"]
flags = []
dev_size = 976773168# 465.762 Gigabytes
pe_start = 384
pe_count = 119234   # 465.758 Gigabytes
}

pv4 {
id = "vK0cGK-zz71-rpGC-IcP3-gsKU-EHjx-fY70SB"
device = "/dev/sdf" # Hint only

status = ["ALLOCATABLE"]
flags = []
dev_size = 976773168# 465.762 Gigabytes
pe_start = 384
pe_count = 119234   # 465.758 Gigabytes
}
}

logical_volumes {

home {
id = "DM6HzG-mgeC-Hj4M-MyKk-k8UT-ZjHU-t74qTe"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
segment_count = 5

segment1 {
start_extent = 0
extent_count = 119234   # 465.758 Gigabytes

type = "striped"
stripe_count = 1# linear

stripes = [
"pv0", 0
]
}
  

Re: Lost LVM data - lost logical volume

[David Christensen]

On 10/22/2015 02:41 PM, Anton Bizzarri wrote:

Hello, we had lost a drive in a LVM on Debian Squeeze. I pulled the drive
and replaced it and then tried to repair it but then when I tried to
recover the LVM it turned out it was a striped volume. (Its not my server!
Never thought they put the data on a striped volume).

So I returned the original drive and I tried to see if I can recover the
data from LVM metadata.

This is Debian wheezy and lvm2 2.02.95-8

The drive that failed was sdc


...

It is not finding the logical volumes anymore.

I have a file in the archive folder /etc/lvm/archive from the morning
before I ran "vgreduce --removemissing --force data"

Is there any way to restore the metadata? Below is the contents of archive
file.

...

I want to see if its possible to try to recover the data because all the
drives are actually still in the exact same slots.  The failing drive on
/dev/sdc magically started working again when I put it back in the server.


You *do* have a back up of your data, right?

1.  Make another copy of the back up and take it off-site.

2.  Test all of your hardware:

a.  Power supply -- hardware load tester.

b.  Memory -- many hours; 24 is good.

c.  Drives -- use manufacturer diagnostics.  Understand that 
failures can be caused by data cables, power cables, connectors, HBA's, 
RAID cards, MB ports, etc..  Troubleshooting might involve a live Linux 
image and/ or hardware substitution/ process of elimination.


3.  Optional: Wipe the drives.

4.  Build a new array/ file system.  (As an aside, have you considered ZFS?)

5.  Optional: Backup/ archive your array/ file system meta data.

6.  Restore your data.

7.  Verify that you can back up the data.


David