Package: os-prober
Version: 1.39
Severity: important

An upgrade (os-prober 1.35 -> 1.39) corrupted 3.3 TB of data on our SAN.

I was upgrading the host space1 and the data corruption occurred on space2. An 
install script of os-prober tried mounting as read-only a SAN volume which was 
already mounted on space2. That volume (on sapce2) was in production use so 
EXT3-fs (on space1) concluded that the journal was inconsistent, re-mounted as 
writable and performed a "recovery".

The mount on space2 became unavailable bringing the production host down. 
Re-mounting failed. After rebooting space2 fsck was required on the affected 
partition. It ran for many hours and found a huge number errors. Probably more 
than 10,000 errors. Then I was able to mount the volume and saw that our data 
was turned into gray goo: parts of system prel scripts were replaced by binary 
chunks, databases and web servers would not start. I had 30 containers in 
production. Some actually booted despite major sporadic data corruption in them.

My fellow system administrator from another department on campus said that 
their distribution (CentOS) does not run install scripts. As he worded it - 
Debian ended-up managing your SAN for you.

The reason why I got os-prober was the change in Debian's policy to install all 
recommended packages and os-prober was recommended by Grub. I am not sure why 
the data corruption did not happed when I upgraded to Squeeze a month ago 
(grub-common 1.96+20080724-16 -> 1.98-1).


I'm attaching the aptitiude log and syslog of space1.

The root of the problem is also described in bug #556739 
(http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=556739). The author predicts 
filesystem corruption and data loss back in 2009.

Alex


-- System Information:
Debian Release: squeeze/sid
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32.15+openvz-budarin (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages os-prober depends on:
ii  libc6                         2.11.2-2   Embedded GNU C Library: Shared lib

os-prober recommends no packages.

os-prober suggests no packages.

-- no debconf information



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Reply via email to