subject:"\[gentoo\-user\] Trouble with backup harddisks"

On 05/01 05:32, Peter Humphrey wrote:
> On Friday, 1 May 2020 17:00:56 BST Andrea Conti wrote:
> 
> > GPT is fine too, but for a 1TB disk with a single partition it has 
> > absolutely
> > zero advantage over MBR.
> 
> I can think of one or two people who might demur there.
> 
> -- 
> Regards,
> Peter.
> 
> 
> 
> 

Hi Wolf, hi Andrea, hi Peter,

*Partition type '83':
I changed the type to 83 using fdisk and this worked fine. Now
/dev/sdb1 is recognized again.
Just to confirm this...
I think, I feel better if I repartitioning/reformat both drives,
though.

*GPT/MBR
>From a discussion based on a "GPT or MBR for my system drive" in
conjunction with UEFI it was said, that GPT is more modern and
save.
My question was meant not so much as "MBR or GPT?" 
but more whether there are some variants of GPT (with 
protected MBR for example -- which was completly new to me),
which I should use or avoid.

But: Are rescue systems for USB-stick more UEFI/GPT aware nowadays
or "traditionally" based on MBR/BIOS-boot?

One thing I found is really handy: An USB-stick with an rEfind
installation. As long as your PC supports UEFI (or can switched to it)
rEfind is able to boot "everything" without prior configuration.

Some rescue-system which really shines and with which you have made good
experiences?

Cheers!
Meino

Re: [gentoo-user] Trouble with backup harddisks

On 05/01 09:27, antlists wrote:
> On 01/05/2020 09:03, tu...@posteo.de wrote:
> > Hi Wol,
> > 
> > data copied !:)
> > 
> > I did a
> > 
> >  mdadm --examine /dev/sdb
> 
> Except I pointed you at a utility called lsdrv, not mdadm ... :-)
> 
> Cheers,
> Wol
> 

Hi Wol,

Ouuouud...oh damn! Sorry, Wol...

I donwloaded the lsdrv from github and this is, what it prints

  File "./lsdrv", line 323
os.mkdir('/dev/block', 0755)
  ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use 
an 0o prefix for octal integers
[1]4367 exit 1 ./lsdrv


Seems a little buggy...

Cheers!
Meino

Re: [gentoo-user] Trouble with backup harddisks

2020-05-01 Thread antlists


On 01/05/2020 09:03, tu...@posteo.de wrote:

Hi Wol,

data copied !:)

I did a

 mdadm --examine /dev/sdb


Except I pointed you at a utility called lsdrv, not mdadm ... :-)

Cheers,
Wol

Re: [gentoo-user] Trouble with backup harddisks

2020-05-01 Thread Peter Humphrey

On Friday, 1 May 2020 17:00:56 BST Andrea Conti wrote:

> GPT is fine too, but for a 1TB disk with a single partition it has absolutely
> zero advantage over MBR.

I can think of one or two people who might demur there.

-- 
Regards,
Peter.

Re: [gentoo-user] Trouble with backup harddisks

2020-05-01 Thread Andrea Conti


A very *#BIG THANK YOU#* for all the great help, the research and
the solution. I myself am back in "normal mode" :)


You're welcome!


What is the most reasonable setup here:
GPT without any hybrid magic and ext4 because it is so common?


I would go with MBR and a single ext4 partition. GPT is fine too, but for a 1TB 
disk with a single partition it has absolutely zero advantage over MBR.

andrea

Re: [gentoo-user] Trouble with backup harddisks

2020-05-01 Thread Wynn Wolf Arbor


On 2020-05-01 09:18, tu...@posteo.de wrote:

A very *#BIG THANK YOU#* for all the great help, the research and
the solution. I myself am back in "normal mode" :)


Glad it helped!


One thing remains...
I want to prevent this kind of hassle in the future... ;)


The most important thing to keep in mind here is to _always_ partition 
and format a drive yourself. That way, you know what is on there and how 
it was partitioned.



Perhaps it is a good idea to re-partitions the disk to get rid of
any bogus bit, format the partition and copy back the data then.


Whilst you can definitely just change the partition type from EE to 83, 
now that you can move the data around safely, I'd probably repartition 
the drives fully, yeah.



What is the most reasonable setup here:
GPT without any hybrid magic and ext4 because it is so common?
Any other, possible more robust configuration, which is also
common with rescue tools and -distributions?


For external backup disks like these I'd go with GPT and ext4. Seems to 
be the most robust configuration for that use case.


If by "hybrid magic" you mean the Protective MBR, that is not something 
you can simply disable. The standard includes the Protective MBR, and 
tools adhere to that.


Bottom line: Repartition the disks with GPT, then create the necessary 
file systems. Don't turn on any special knobs, the tool's defaults are 
all decent.


--
Wolf

Re: [gentoo-user] Trouble with backup harddisks

On 05/01 03:59, tu...@posteo.de wrote:
> On 04/30 08:32, antlists wrote:
> > On 30/04/2020 18:04, tu...@posteo.de wrote:
> > > I copied the first 230GB of that disk to an empty partition of my new
> > > system and run "testdisk" on itafter the analysis it came back
> > > with "this partition cannot be recovered" but did not sau. whether the
> > > reason is a partition table, which is broken beyond repair, or simply
> > > due to the incomplete image file...
> > 
> > Just come up with another idea that will hopefully give us some clues...
> > 
> > https://raid.wiki.kernel.org/index.php/Asking_for_help#lsdrv
> > 
> > Ignore the fact that it's the raid wiki - this tool basically delves into
> > the drive and tries to find mbr, gpt, superblock, lvm and raid signatures,
> > everything ...
> > 
> > So run that and see what it tells us ...
> > 
> > Cheers,
> > Wol
> > 
> 
> Hi Wol,
> 
> thank you for the link! I have installed mdadm (its on portage) but I
> have not used it, since there is another soultion for this, which I
> tried first. 
> 
> Andrea posted this wonderful line:
> 
> mount -o ro,offset=512 /dev/sdb /mnt/xxx
> 
> which mounts the filesystem by skipping the partitiontable completly.
> 
> Currently I am rescueing the data. If all is safed, I will tru
> mdadm...
> 
> Thank you very much for your help!
> 
> As soon as there are more information, I will report back.
> Maybe this can also help others with similiar problems.
> 
> Cheers!
> Meino
> 
> 
> 

Hi Wol,

data copied ! :)

I did a 

mdadm --examine /dev/sdb

on one of the damaged partitions and this was print
on the console

host:/root>mdadm --examine /dev/sdb
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   1953458175 sectors at1 (type ee)


Cheers!
Meino

Re: [gentoo-user] Trouble with backup harddisks

On 05/01 08:52, Andrea Conti wrote:
> 
> > does my posting from this morning reached you ?
> > ...I did not received anything back from the mailinglist...
> 
> Nope. Just this night's response to Wol.
> 
> 

(hmmm...ok, two send good news two times is not
that bad in this times, I think... ;)

Hi Andrea, hi Wolf,

this magic one liner does the trick indeed!  I have my data back and
copied it to my new system spread over four separate partitions, which
are currently still empty. No error while copuing and both disk
matched.

A very *#BIG THANK YOU#* for all the great help, the research and
the solution. I myself am back in "normal mode" :)

Wol has suggested to run mdadm on the bad harddisk to see, whether
this tool is able to fix it -- which I will next. Since I have
my data back on the PC internal disk, this is no problem at all.

One thing remains...
I want to prevent this kind of hassle in the future... ;)

Perhaps it is a good idea to re-partitions the disk to get rid of
any bogus bit, format the partition and copy back the data then.

What is the most reasonable setup here:
GPT without any hybrid magic and ext4 because it is so common?
Any other, possible more robust configuration, which is also
common with rescue tools and -distributions?

I have to say it once again: Thank you very much for solving
the hassle puzzle! :)

Cheers!
Meino

Re: [gentoo-user] Trouble with backup harddisks

2020-05-01 Thread Andrea Conti





does my posting from this morning reached you ?
...I did not received anything back from the mailinglist...


Nope. Just this night's response to Wol.

Re: [gentoo-user] Trouble with backup harddisks

On 04/30 10:47, Wynn Wolf Arbor wrote:
> On 2020-04-30 22:21, Andrea Conti wrote:
> > It won't, as long as it recognizes it as a protective MBR. Which is the
> > right thing to do, as a disk with a protective MBR and no valid GPT is
> > inherently broken.
> 
> True. It was more my intention to depict what the system "should" do in
> order to access the file system.
> 
> > ... or just bypass the partition table altogether. The filesystem starts
> > at sector 1, i.e. 1*512B, so:
> > 
> > mount -o ro,offset=512 /dev/sdb /mnt/xxx
> 
> Interesting, thanks. I was initially considering something like this myself,
> but after a cursory check of the manual, I was under the assumption that
> 'offset' was only valid with loop devices and dismissed that solution.
> 
> Turns out if you mount a drive like this, the kernel uses a loop device in
> the background and you can use the 'offset' option with block devices as
> well. I feel the documentation could be improved here.
> 
> -- 
> Wolf
> 

Hi Andrea, hi Wolf

does my posting from this morning reached you ?
...I did not received anything back from the mailinglist...

Cheers!
Meino

Re: [gentoo-user] Trouble with backup harddisks

On 04/30 08:32, antlists wrote:
> On 30/04/2020 18:04, tu...@posteo.de wrote:
> > I copied the first 230GB of that disk to an empty partition of my new
> > system and run "testdisk" on itafter the analysis it came back
> > with "this partition cannot be recovered" but did not sau. whether the
> > reason is a partition table, which is broken beyond repair, or simply
> > due to the incomplete image file...
> 
> Just come up with another idea that will hopefully give us some clues...
> 
> https://raid.wiki.kernel.org/index.php/Asking_for_help#lsdrv
> 
> Ignore the fact that it's the raid wiki - this tool basically delves into
> the drive and tries to find mbr, gpt, superblock, lvm and raid signatures,
> everything ...
> 
> So run that and see what it tells us ...
> 
> Cheers,
> Wol
> 

Hi Wol,

thank you for the link! I have installed mdadm (its on portage) but I
have not used it, since there is another soultion for this, which I
tried first. 

Andrea posted this wonderful line:

mount -o ro,offset=512 /dev/sdb /mnt/xxx

which mounts the filesystem by skipping the partitiontable completly.

Currently I am rescueing the data. If all is safed, I will tru
mdadm...

Thank you very much for your help!

As soon as there are more information, I will report back.
Maybe this can also help others with similiar problems.

Cheers!
Meino

Re: [gentoo-user] Trouble with backup harddisks


On 2020-04-30 22:21, Andrea Conti wrote:
It won't, as long as it recognizes it as a protective MBR. Which is the 
right thing to do, as a disk with a protective MBR and no valid GPT is 
inherently broken.


True. It was more my intention to depict what the system "should" do in 
order to access the file system.


... or just bypass the partition table altogether. The filesystem 
starts at sector 1, i.e. 1*512B, so:


mount -o ro,offset=512 /dev/sdb /mnt/xxx


Interesting, thanks. I was initially considering something like this 
myself, but after a cursory check of the manual, I was under the 
assumption that 'offset' was only valid with loop devices and dismissed 
that solution.


Turns out if you mount a drive like this, the kernel uses a loop device 
in the background and you can use the 'offset' option with block devices 
as well. I feel the documentation could be improved here.


--
Wolf

Re: [gentoo-user] Trouble with backup harddisks

2020-04-30 Thread Andrea Conti

> Since the disk was only ever accessed through an operating system that knew 
> solely about MBR, the GPT data meant nothing to it. It happily wrote data 
> past the MBR headers. Because the protective MBR is positioned before GPT 
> information, the primary GPT header was destroyed and most likely overwritten 
> with the file system. See also [1], the actual file system data probably 
> begins somewhere past LBA 0.

If it was created according to the MBR that was shown previously, it will start 
at the beginning of the protective entry, i.e. at sector 1.

> If the initial assumption is correct, GPT *must not* be restored. Your modern 
> PC sees the GPT partition type and assumes the existence of a GPT. It should, 
> however, access the MBR layout and interpret the partition marked with the 
> GPT ID as a regular partition.

It won't, as long as it recognizes it as a protective MBR. Which is the right 
thing to do, as a disk with a protective MBR and no valid GPT is inherently 
broken.

> 1) Boot an older system that only understands MBR, and mount the disk there. 
> This was suggested earlier but was dismissed because we assumed the sector 
> size had something to do with it. I do not think this is the case anymore - 
> the old system should be able to read it.
> 
> 2) Boot a VM with a kernel that only understands MBR, pass USB through to the 
> virtual machine, mount the disk there.
> 
> 3) Try confirming that there exists file system data past the MBR header.
>
> Maybe something like this:
> 
> # dd if=/dev/sdb of=sdb-data bs=512 skip=1 count=16384
> $ file sdb-data 


... or just bypass the partition table altogether. The filesystem starts at 
sector 1, i.e. 1*512B, so:

mount -o ro,offset=512 /dev/sdb /mnt/xxx

But while this will allow you to access your data, you will still have a broken 
disk until you fix the MBR.


andrea

Re: [gentoo-user] Trouble with backup harddisks


Hi Meino,

On 2020-04-30 21:46, tu...@posteo.de wrote:

I had booted into my old system, attached the disks and both show the
same behaviour: Only the device itself (/dev/sdb) was recognized.


Now that is very curious. Just to make sure, the old system definitely 
does not understand GPT? CONFIG_EFI_PARTITION should be unset.



'file' shows the following output:

file sdb-data
sdb-data: Linux rev 1.0 ext4 filesystem data, 
UUID=2f063705-0d3a-4790-9203-1b4edab7788c (extents) (64bit) (large files) (huge 
files)

Looks better than I have thought...or?


This does indeed look promising (and is what I expected in the best 
case). Now, of course, the problem is figuring out how to get to that 
data.



I will take a deeper look tommorrow...I am too tired to
"fix partition tables manually" this evening!

Read you tommorrow! :)


Hope the rest of your evening will be more relaxing!

--
Wolf

Re: [gentoo-user] Trouble with backup harddisks

Hi Wolf,

thanks for your great input again!

(see below)

On 04/30 09:27, Wynn Wolf Arbor wrote:
> All the following assuming that the disk was originally partitioned as GPT,
> but after that exclusively accessed as an MBR disk.
> 
> > PT fdisk (gdisk) version 1.0.5
> > 
> > Caution: invalid main GPT header, but valid backup; regenerating main header
> > from backup!
> 
> This makes sense since the GPT backup at the very end of the disk would most
> likely still be intact. gdisk identifies it correctly, but assumes (wrongly)
> that the data on the disk is governed by the GPT layout.
> 
> Since the disk was only ever accessed through an operating system that knew
> solely about MBR, the GPT data meant nothing to it. It happily wrote data
> past the MBR headers. Because the protective MBR is positioned before GPT
> information, the primary GPT header was destroyed and most likely
> overwritten with the file system. See also [1], the actual file system data
> probably begins somewhere past LBA 0.
> 
> > Caution! After loading partitions, the CRC doesn't check out!
> > Warning: Invalid CRC on main header data; loaded backup partition table.
> > Warning! Main and backup partition tables differ! Use the 'c' and 'e' 
> > options
> > on the recovery & transformation menu to examine the two tables.
> 
> This is because the backup GPT written when first partitioned does no longer
> match the data present at the very beginning of the disk.
> 
> If the initial assumption is correct, GPT *must not* be restored. Your
> modern PC sees the GPT partition type and assumes the existence of a GPT. It
> should, however, access the MBR layout and interpret the partition marked
> with the GPT ID as a regular partition.
> 
> Now, how to fix this?
> 
> Like Andrea already said earlier:
> 
> > Since the disk is only 1TB, there is no reason to use GPT at all, so
> > your best bet is to use fdisk to make that a standard MBR by changing
> > the partition type from 'ee' to '83'.
> 
> This would *not* repartition or reformat any data, it would simply tell your
> modern operating system to access the protective partition as a regular one.
> 
> It would, however, require writing the new type to disk. What you could do
> to be more safe here is to take a backup of the first 512 bytes with `dd',
> then change the partition ID with `fdisk', and try mounting it.
> 
> If it works, great. If not, you can restore the first 512 bytes of the disk
> with the backup.
> 
> > "fix manually" scares me...especially because I have no place for
> > 1TB of an image file to with which I can experiment ...
> 
> > Any ideas which could ease my burden and to un-scare my
> > "need to fix it manually" ??? ;) ;)
> 
> There's a few alternatives:
> 
> 1) Boot an older system that only understands MBR, and mount the disk there.
> This was suggested earlier but was dismissed because we assumed the sector
> size had something to do with it. I do not think this is the case anymore -
> the old system should be able to read it.
> 
> 2) Boot a VM with a kernel that only understands MBR, pass USB through to
> the virtual machine, mount the disk there.
> 
> 3) Try confirming that there exists file system data past the MBR header.
> 
> Maybe something like this:
> 
> # dd if=/dev/sdb of=sdb-data bs=512 skip=1 count=16384
> $ file sdb-data
> 
> As established, the block size is 512 bytes. We skip the first 512 bytes
> since that is the protective MBR. sdb-data should then contain the first
> 8MiB worth of actual file system data. The `file' utility can tell you what
> kind of data it is.
> 
> [1] 
> https://en.wikipedia.org/wiki/GUID_Partition_Table#/media/File:GUID_Partition_Table_Scheme.svg
> 
> -- 
> Wolf
> 

I had booted into my old system, attached the disks and both show the
same behaviour: Only the device itself (/dev/sdb) was recognized.

'file' shows the following output:

file sdb-data
sdb-data: Linux rev 1.0 ext4 filesystem data, 
UUID=2f063705-0d3a-4790-9203-1b4edab7788c (extents) (64bit) (large files) (huge 
files)

Looks better than I have thought...or?

I will take a deeper look tommorrow...I am too tired to
"fix partition tables manually" this evening!

Read you tommorrow! :)

Cheers!
Meino

Re: [gentoo-user] Trouble with backup harddisks

2020-04-30 Thread antlists


On 30/04/2020 18:04, tu...@posteo.de wrote:

I copied the first 230GB of that disk to an empty partition of my new
system and run "testdisk" on itafter the analysis it came back
with "this partition cannot be recovered" but did not sau. whether the
reason is a partition table, which is broken beyond repair, or simply
due to the incomplete image file...


Just come up with another idea that will hopefully give us some clues...

https://raid.wiki.kernel.org/index.php/Asking_for_help#lsdrv

Ignore the fact that it's the raid wiki - this tool basically delves 
into the drive and tries to find mbr, gpt, superblock, lvm and raid 
signatures, everything ...


So run that and see what it tells us ...

Cheers,
Wol

Re: [gentoo-user] Trouble with backup harddisks

All the following assuming that the disk was originally partitioned as
GPT, but after that exclusively accessed as an MBR disk.

PT fdisk (gdisk) version 1.0.5

Caution: invalid main GPT header, but valid backup; regenerating main header
from backup!

This makes sense since the GPT backup at the very end of the disk would
most likely still be intact. gdisk identifies it correctly, but assumes
(wrongly) that the data on the disk is governed by the GPT layout.

Since the disk was only ever accessed through an operating system that
knew solely about MBR, the GPT data meant nothing to it. It happily
wrote data past the MBR headers. Because the protective MBR is
positioned before GPT information, the primary GPT header was destroyed
and most likely overwritten with the file system. See also [1], the
actual file system data probably begins somewhere past LBA 0.

Caution! After loading partitions, the CRC doesn't check out!
Warning: Invalid CRC on main header data; loaded backup partition table.
Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

This is because the backup GPT written when first partitioned does no
longer match the data present at the very beginning of the disk.

If the initial assumption is correct, GPT *must not* be restored. Your
modern PC sees the GPT partition type and assumes the existence of a
GPT. It should, however, access the MBR layout and interpret the
partition marked with the GPT ID as a regular partition.

Now, how to fix this?

Like Andrea already said earlier:

Since the disk is only 1TB, there is no reason to use GPT at all, so
your best bet is to use fdisk to make that a standard MBR by changing
the partition type from 'ee' to '83'.

This would *not* repartition or reformat any data, it would simply tell
your modern operating system to access the protective partition as a
regular one.

It would, however, require writing the new type to disk. What you could
do to be more safe here is to take a backup of the first 512 bytes with
`dd', then change the partition ID with `fdisk', and try mounting it.

If it works, great. If not, you can restore the first 512 bytes of the
disk with the backup.

"fix manually" scares me...especially because I have no place for
1TB of an image file to with which I can experiment ...

Any ideas which could ease my burden and to un-scare my
"need to fix it manually" ??? ;) ;)

There's a few alternatives:

1) Boot an older system that only understands MBR, and mount the disk
there. This was suggested earlier but was dismissed because we assumed
the sector size had something to do with it. I do not think this is the
case anymore - the old system should be able to read it.

2) Boot a VM with a kernel that only understands MBR, pass USB through
to the virtual machine, mount the disk there.

3) Try confirming that there exists file system data past the MBR header.

Maybe something like this:

# dd if=/dev/sdb of=sdb-data bs=512 skip=1 count=16384
$ file sdb-data

As established, the block size is 512 bytes. We skip the first 512 bytes
since that is the protective MBR. sdb-data should then contain the first
8MiB worth of actual file system data. The `file' utility can tell you
what kind of data it is.

[1]
https://en.wikipedia.org/wiki/GUID_Partition_Table#/media/File:GUID_Partition_Table_Scheme.svg

--
Wolf

Re: [gentoo-user] Trouble with backup harddisks

On 04/30 03:44, Andrea Conti wrote:
> Hi,
> 
> > > CONFIG_PARTITION_ADVANCED=y
> > > CONFIG_MSDOS_PARTITION=y
> > > CONFIG_EFI_PARTITION=y
> 
> That's all you need.
> 
> > This could be the key. Sector sizes have been changing from 512 to 4096
> > over many years. If your kernel has been updated to expect/use 4096 byte
> > sectors, it might not be able to read the disk properly.
> 
> Sector size is only a (fixed) property of a specific block device, not 
> something expected or required by the kernel.
> Sector sizes other than 512B have been around for ages without any problems, 
> even in consumer hardware (e.g. CDs and DVDs have 2KB sectors).
> 
> > Disklabel type: dos
> 
> > Device Boot StartEndSectors   Size Id Type
> > /dev/sdb1   1 1953458175 1953458175 931.5G ee GPT
> 
> That looks... broken.
> fdisk is recognizing the disk as MBR (a GPT disk would have "Disklabel type: 
> gpt"), but the partition table is a protective MBR, which makes no sense in a 
> non-GPT disk.
> 
> My guess is that this disk was at some time partitioned with GPT (possibly it 
> came that way from WD?), but then it was only used in machines with no kernel 
> support for GPT.
> Such a machine will happily treat that as a normal MBR and allow you to 
> access the protective entry as a normal partition, which means you can create 
> a filesystem on it and fill it with data, destroying the GPT structures.
> 
> A GPT-aware kernel on the other hand will recognize that as a protective MBR 
> and it will ignore it --but since the disk does not contain any valid GPT 
> structures, it will not show any partitions.
> 
> Try running "gdisk -l /dev/sdb"; for a valid GPT disk it will say:
> 
> Partition table scan:
>   MBR: protective
>   BSD: not present
>   APM: not present
>   GPT: present
> 
> If that's not the case and you have no GPT, you will have to fix things 
> manually.
> Since the disk is only 1TB, there is no reason to use GPT at all, so your 
> best bet is to use fdisk to make that a standard MBR by changing the 
> partition type from 'ee' to '83'.
> 
> andrea
> 

Hi Andreas,

thank you very much for your analysis! :)

when doing a gdisk -l /dev/sdb I get this Rocky Horror Picture Show:


PT fdisk (gdisk) version 1.0.5

Caution: invalid main GPT header, but valid backup; regenerating main header
from backup!

Caution! After loading partitions, the CRC doesn't check out!
Warning: Invalid CRC on main header data; loaded backup partition table.
Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: ERROR
Backup header: OK
Main partition table: ERROR
Backup partition table: ERROR

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged


Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.

Disk /dev/sdb: 1953458176 sectors, 931.5 GiB
Model: Elements 25A2   
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): BCA95A74-1E0A-4648-9971-20ED4049CAE5
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953458142
Partitions will be aligned on 2048-sector boundaries
Total free space is 1953458109 sectors (931.5 GiB)

Number  Start (sector)End (sector)  Size   Code  Name



"fix manually" scares me...especially because I have no place for
1TB of an image file to with which I can experiment ...

Any ideas which could ease my burden and to un-scare my 
"need to fix it manually" ??? ;) ;)

Cheers!
Meino

Re: [gentoo-user] Trouble with backup harddisks


Hi Meino,

Thanks very much for the info. At this point I'm convinced you're 
running into the problem Andrea described in another reply in this 
thread - best to follow up there :)


--
Wolf

Re: [gentoo-user] Trouble with backup harddisks

Hi Wolf

thank you very much for your analysis ! :)

On 04/30 03:10, Wynn Wolf Arbor wrote:
> Hi,
> 
> On 2020-04-30 13:17, Wols Lists wrote:
> > All I can suggest is to check the kernel and see if it's an option that
> > has been disabled (512-byte sectors, that is).
> 
> As far as I know the kernel still uses 512 bytes internally [1], and I do
> not recall having seen an option that enables or disables support for 512/4K
> sectors.
> 
> That said, the problem may well be stemming from a sector size discrepancy,
> but as I understand it, it would have to do with how and when the partition
> table was created. That is, like described in [2], some USB enclosures seem
> to be a bit overzealous with obscure features, and might take eight disk
> sectors and bundle them together into one 4K sector.
> 
> If the disk was partitioned in the exact same enclosure, and is read from
> the exact same enclosure right now, there shouldn't be any problems. Is this
> the case, Meino?

I have two external disks of that kind, same product, nearly the same
contents, both have a "fixed case" (sorry...no native speaker...the
case and the disk mechanics/electronics is "one thing").
> 
> Also, when did you last access the drives successfully, and with which
> system?

Three weeks ago...about...guessed...with my old system.
Because these were backup disks I *NEVER* touch them with
any partitioning tool whatsoever...

It is complete unknown to me, what could have destroyed the
partitioning...

> On 2020-04-30 11:32, Meino wrote:
> > Disk /dev/sdb: 931.49 GiB, 1000170586112 bytes, 1953458176 sectors
> > Disk model: Elements 25A2   Units: sectors of 1 * 512 = 512 bytes
> > Sector size (logical/physical): 512 bytes / 512 bytes
> > I/O size (minimum/optimal): 512 bytes / 512 bytes
> > 
> > Disklabel type: dos
> > Disk identifier: 0x16f2a91f
> > 
> > Device Boot StartEndSectors   Size Id Type
> > /dev/sdb1   1 1953458175 1953458175 931.5G ee GPT
> 
> Interestingly, this reads *exactly* like the Protective MBR [3] that GPT
> has. That is, the disklabel type is DOS whilst the partition ID is EE.
> There's a single partition that spans the entire drive (and it's also
> seemingly not aligned properly - usually you see Start at 2048).

> As a comparison, here's the output from fdisk for the Protective MBR from
> one of my GPT drives:
> 
> > Disklabel type: dos
> > Disk identifier: 0x
> 
> > Device Boot StartEndSectors Size Id Type
> > /dev/sdc1   1 4294967295 4294967295   2T ee GPT
> 
> I'd assume that the missing disk identifier here is coming from different
> tools writing the protective MBR differently.
> 
> With that said, are you absolutely certain that you did not partition this
> drive with GPT instead of MBR? 

YES! These drives were used as backup and as backup of the backup. I
NEVER touched them with any partitioning tool after I had put a
filesystem and data on it.

> Did you do the partitioning in something like
> fdisk (which asks you specifically what you want), or some other
> application? Did you maybe format this drive on a Windows system?

Windows? no thank you...all my windows here are transparent and
open,.. ;)
Joke aside..
No Windows (tm) here. The disks are used only with my Linux box using
ext4 as filesystem.

> 
> I'm not one to discount entirely strange things happening, but I have never
> before seen a proper MBR laid out like a protective MBR. Indeed it would be
> quite impossible to have systems access data through such a table, since the
> partitions are hidden within that one huge contiguous block.
> 
> Ordinarily I'd point to fdisk not reading the partition table properly, but
> it seems that although your kernel has support for GPT, it doesn't seem to
> see the partitions either (assuming a proper GPT exists at all).

> Do you have some other GPT drives you can access successfully?
With my new PC (my old MBR-based system was 12 years old and needed to
be replaced. My kernel became "GPT aware onlu with my new PC.
Everything before was based on MBR.

My new PC can read GPT disks...my system is based on it.
> 
> I'd say that this requires some more forensic work. Perhaps extracting the
> first few megabytes of the disk and seeing whether there's a proper GPT or
> not. This would of course require manual work.

I copied the first 230GB of that disk to an empty partition of my new
system and run "testdisk" on itafter the analysis it came back
with "this partition cannot be recovered" but did not sau. whether the
reason is a partition table, which is broken beyond repair, or simply
due to the incomplete image file...

> 
> A few more things to try:
> 
> To see what the kernel uses for a particular disk, you can run the
> following: cat /sys/block/sdb/queue/{physical,logical}_block_size
 

host:/home/user>cat /sys/block/sdb/queue/physical_block_size 
512
solfire:/home/user>cat /sys/block/sdb/queue/logical_block_size 
512
host:/home/user>

> fdisk takes

Re: [gentoo-user] Trouble with backup harddisks

2020-04-30 Thread Andrea Conti


Hi,


CONFIG_PARTITION_ADVANCED=y
CONFIG_MSDOS_PARTITION=y
CONFIG_EFI_PARTITION=y


That's all you need.


This could be the key. Sector sizes have been changing from 512 to 4096
over many years. If your kernel has been updated to expect/use 4096 byte
sectors, it might not be able to read the disk properly.


Sector size is only a (fixed) property of a specific block device, not 
something expected or required by the kernel.
Sector sizes other than 512B have been around for ages without any problems, 
even in consumer hardware (e.g. CDs and DVDs have 2KB sectors).


Disklabel type: dos



Device Boot StartEndSectors   Size Id Type
/dev/sdb1   1 1953458175 1953458175 931.5G ee GPT


That looks... broken.
fdisk is recognizing the disk as MBR (a GPT disk would have "Disklabel type: 
gpt"), but the partition table is a protective MBR, which makes no sense in a 
non-GPT disk.

My guess is that this disk was at some time partitioned with GPT (possibly it 
came that way from WD?), but then it was only used in machines with no kernel 
support for GPT.
Such a machine will happily treat that as a normal MBR and allow you to access 
the protective entry as a normal partition, which means you can create a 
filesystem on it and fill it with data, destroying the GPT structures.

A GPT-aware kernel on the other hand will recognize that as a protective MBR 
and it will ignore it --but since the disk does not contain any valid GPT 
structures, it will not show any partitions.

Try running "gdisk -l /dev/sdb"; for a valid GPT disk it will say:

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

If that's not the case and you have no GPT, you will have to fix things 
manually.
Since the disk is only 1TB, there is no reason to use GPT at all, so your best 
bet is to use fdisk to make that a standard MBR by changing the partition type 
from 'ee' to '83'.

andrea

Re: [gentoo-user] Trouble with backup harddisks

Hi,

On 2020-04-30 13:17, Wols Lists wrote:

All I can suggest is to check the kernel and see if it's an option that
has been disabled (512-byte sectors, that is).

As far as I know the kernel still uses 512 bytes internally [1], and I
do not recall having seen an option that enables or disables support for
512/4K sectors.

That said, the problem may well be stemming from a sector size
discrepancy, but as I understand it, it would have to do with how and
when the partition table was created. That is, like described in [2],
some USB enclosures seem to be a bit overzealous with obscure features,
and might take eight disk sectors and bundle them together into one 4K
sector.

If the disk was partitioned in the exact same enclosure, and is read
from the exact same enclosure right now, there shouldn't be any
problems. Is this the case, Meino?

Also, when did you last access the drives successfully, and with which
system?

On 2020-04-30 11:32, Meino wrote:

Disk /dev/sdb: 931.49 GiB, 1000170586112 bytes, 1953458176 sectors
Disk model: Elements 25A2
Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos
Disk identifier: 0x16f2a91f

Device Boot StartEndSectors Size Id Type
/dev/sdb1 1 1953458175 1953458175 931.5G ee GPT

Interestingly, this reads *exactly* like the Protective MBR [3] that GPT
has. That is, the disklabel type is DOS whilst the partition ID is EE.
There's a single partition that spans the entire drive (and it's also
seemingly not aligned properly - usually you see Start at 2048).

As a comparison, here's the output from fdisk for the Protective MBR
from one of my GPT drives:

Disklabel type: dos
Disk identifier: 0x

Device Boot StartEndSectors Size Id Type
/dev/sdc1 1 4294967295 4294967295 2T ee GPT

I'd assume that the missing disk identifier here is coming from
different tools writing the protective MBR differently.

With that said, are you absolutely certain that you did not partition
this drive with GPT instead of MBR? Did you do the partitioning in
something like fdisk (which asks you specifically what you want), or
some other application? Did you maybe format this drive on a Windows
system?

I'm not one to discount entirely strange things happening, but I have
never before seen a proper MBR laid out like a protective MBR. Indeed it
would be quite impossible to have systems access data through such a
table, since the partitions are hidden within that one huge contiguous
block.

Ordinarily I'd point to fdisk not reading the partition table properly,
but it seems that although your kernel has support for GPT, it doesn't
seem to see the partitions either (assuming a proper GPT exists at all).

Do you have some other GPT drives you can access successfully?

I'd say that this requires some more forensic work. Perhaps extracting
the first few megabytes of the disk and seeing whether there's a proper
GPT or not. This would of course require manual work.

A few more things to try:

To see what the kernel uses for a particular disk, you can run the
following: cat /sys/block/sdb/queue/{physical,logical}_block_size

fdisk takes a sector size with -b, --sector-size (should be
non-destructive as long as you don't write anything, but I am not sure).
Also, fdisk has a compatibility mode for dos with -c=dos. Might be worth
a short.

fdisk should support GPT starting with util-linux 2.23, but you can also
try gptfdisk (it's in the tree).

Hope this helps.

[1] https://github.com/torvalds/linux/blob/master/include/linux/types.h#L120
[2]
https://superuser.com/questions/679725/how-to-correct-512-byte-sector-mbr-on-a-4096-byte-sector-disk/679800#679800
[3] https://en.wikipedia.org/wiki/GUID_Partition_Table#PROTECTIVE-MBR

--
Wolf

Re: [gentoo-user] Trouble with backup harddisks

2020-04-30 Thread Wols Lists

On 30/04/20 11:36, tu...@posteo.de wrote:
> But than I have the same problem another way around: I can
> no longer access my new system ... due to the different 
> sector size
> 
> Are there any other ways to fix this problem?

All I can suggest is to check the kernel and see if it's an option that
has been disabled (512-byte sectors, that is).

Cheers,
Wol

Re: [gentoo-user] Trouble with backup harddisks

On 04/30 10:55, Wols Lists wrote:
> On 30/04/20 10:32, tu...@posteo.de wrote:
> > Hi,
> > 
> > recently I switched from the old MBR-scheme to GPT on
> > my new PC.
> > 
> > I have two external USB-harddisk, which were partioned/formatted with
> > a MBR-scheme/MSDOS partition (but were never used to boot from. They are 
> > pure
> > data containers).
> > 
> > When I connect these to my new PC, only the device is shown:
> > /dev/sdb, /dev/sdc.
> > 
> > The kernel is configured (beside other things) as follows:
> > 
> > #
> > # Partition Types
> > #
> > CONFIG_PARTITION_ADVANCED=y
> > # CONFIG_ACORN_PARTITION is not set
> > # CONFIG_AIX_PARTITION is not set
> > # CONFIG_OSF_PARTITION is not set
> > # CONFIG_AMIGA_PARTITION is not set
> > # CONFIG_ATARI_PARTITION is not set
> > # CONFIG_MAC_PARTITION is not set
> > CONFIG_MSDOS_PARTITION=y
> > # CONFIG_BSD_DISKLABEL is not set
> > # CONFIG_MINIX_SUBPARTITION is not set
> > # CONFIG_SOLARIS_X86_PARTITION is not set
> > # CONFIG_UNIXWARE_DISKLABEL is not set
> > # CONFIG_LDM_PARTITION is not set
> > # CONFIG_SGI_PARTITION is not set
> > # CONFIG_ULTRIX_PARTITION is not set
> > # CONFIG_SUN_PARTITION is not set
> > # CONFIG_KARMA_PARTITION is not set
> > CONFIG_EFI_PARTITION=y
> > # CONFIG_SYSV68_PARTITION is not set
> > # CONFIG_CMDLINE_PARTITION is not set
> > # end of Partition Types
> > 
> > CONFIG_BLOCK_COMPAT=y
> > CONFIG_BLK_MQ_PCI=y
> > CONFIG_BLK_MQ_VIRTIO=y
> > CONFIG_BLK_PM=y
> > 
> > dmesg shows this:
> > [14617.672363] usb 2-2: New USB device found, idVendor=1058, 
> > idProduct=25a2, bcdDevice=10.21
> > [14617.672364] usb 2-2: New USB device strings: Mfr=1, Product=2, 
> > SerialNumber=3
> > [14617.672364] usb 2-2: Product: Elements 25A2
> > [14617.672365] usb 2-2: Manufacturer: Western Digital
> > [14617.672366] usb 2-2: SerialNumber: 575844314132383959393934
> > [14617.681660] usb-storage 2-2:1.0: USB Mass Storage device detected
> > [14617.681737] scsi host10: usb-storage 2-2:1.0
> > [14618.725450] scsi 10:0:0:0: Direct-Access WD   Elements 25A2
> > 1021 PQ: 0 ANSI: 6
> > [14618.725594] sd 10:0:0:0: Attached scsi generic sg1 type 0
> > [14618.728090] sd 10:0:0:0: [sdb] Spinning up disk...
> > [14619.748918] ...ready
> > 
> > I tried different USB ports in a desperate hope of success...
> > ...no, same problem.
> > 
> > Interestingly fdisk shows the following:
> > 
> > host> sudo fdisk -l /dev/sdb
> > Disk /dev/sdb: 931.49 GiB, 1000170586112 bytes, 1953458176 sectors
> > Disk model: Elements 25A2   
> > Units: sectors of 1 * 512 = 512 bytes
> > Sector size (logical/physical): 512 bytes / 512 bytes
> > I/O size (minimum/optimal): 512 bytes / 512 bytes
> 
> This could be the key. Sector sizes have been changing from 512 to 4096
> over many years. If your kernel has been updated to expect/use 4096 byte
> sectors, it might not be able to read the disk properly.
> 
> > Disklabel type: dos
> > Disk identifier: 0x16f2a91f
> > 
> > Device Boot StartEndSectors   Size Id Type
> > /dev/sdb1   1 1953458175 1953458175 931.5G ee GPT
> > 
> > The type is shown as GPT...but the drive has a MSDOS partition table.
> > 
> > Reading my (old) internal harddrive with an external USB docking
> > station is possible without any problems, though.
> > 
> > Unfortunatelu I have no space for 1T image of that drive -- otherwise
> > I would have made an image copy and experiment with that.
> > 
> > So better ask, than sorry ;)
> > 
> > Is this fixable or did I lost my backups?
> > 
> Do you have access to an old kernel?
> 
> The other thing is try using gdisk (or that could be fdisk under another
> name :-( But some partitioning schemes can write a GPT with protective
> MBR - if you can find something that will take the MBR and write it as a
> GPT that might help, too.
> 
> Cheers,
> Wol
> 

Hi Wol,

thank you for your posting! :)

I switched to my new PC around 31.3.2020...
so there is no change "over the years" involved here...I think.
Where in the kernel this update is made?
I cannot remember to have configured such a thing manually...

Changing the partitiontable in any way is risky to the data 
I think...I am very unsure to do this.

I have access to an old kernel / system and can boot it.

But than I have the same problem another way around: I can
no longer access my new system ... due to the different 
sector size

Are there any other ways to fix this problem?

Cheers!
Meino

Re: [gentoo-user] Trouble with backup harddisks

2020-04-30 Thread Wols Lists

On 30/04/20 10:32, tu...@posteo.de wrote:
> Hi,
> 
> recently I switched from the old MBR-scheme to GPT on
> my new PC.
> 
> I have two external USB-harddisk, which were partioned/formatted with
> a MBR-scheme/MSDOS partition (but were never used to boot from. They are pure
> data containers).
> 
> When I connect these to my new PC, only the device is shown:
> /dev/sdb, /dev/sdc.
> 
> The kernel is configured (beside other things) as follows:
> 
> #
> # Partition Types
> #
> CONFIG_PARTITION_ADVANCED=y
> # CONFIG_ACORN_PARTITION is not set
> # CONFIG_AIX_PARTITION is not set
> # CONFIG_OSF_PARTITION is not set
> # CONFIG_AMIGA_PARTITION is not set
> # CONFIG_ATARI_PARTITION is not set
> # CONFIG_MAC_PARTITION is not set
> CONFIG_MSDOS_PARTITION=y
> # CONFIG_BSD_DISKLABEL is not set
> # CONFIG_MINIX_SUBPARTITION is not set
> # CONFIG_SOLARIS_X86_PARTITION is not set
> # CONFIG_UNIXWARE_DISKLABEL is not set
> # CONFIG_LDM_PARTITION is not set
> # CONFIG_SGI_PARTITION is not set
> # CONFIG_ULTRIX_PARTITION is not set
> # CONFIG_SUN_PARTITION is not set
> # CONFIG_KARMA_PARTITION is not set
> CONFIG_EFI_PARTITION=y
> # CONFIG_SYSV68_PARTITION is not set
> # CONFIG_CMDLINE_PARTITION is not set
> # end of Partition Types
> 
> CONFIG_BLOCK_COMPAT=y
> CONFIG_BLK_MQ_PCI=y
> CONFIG_BLK_MQ_VIRTIO=y
> CONFIG_BLK_PM=y
> 
> dmesg shows this:
> [14617.672363] usb 2-2: New USB device found, idVendor=1058, idProduct=25a2, 
> bcdDevice=10.21
> [14617.672364] usb 2-2: New USB device strings: Mfr=1, Product=2, 
> SerialNumber=3
> [14617.672364] usb 2-2: Product: Elements 25A2
> [14617.672365] usb 2-2: Manufacturer: Western Digital
> [14617.672366] usb 2-2: SerialNumber: 575844314132383959393934
> [14617.681660] usb-storage 2-2:1.0: USB Mass Storage device detected
> [14617.681737] scsi host10: usb-storage 2-2:1.0
> [14618.725450] scsi 10:0:0:0: Direct-Access WD   Elements 25A2
> 1021 PQ: 0 ANSI: 6
> [14618.725594] sd 10:0:0:0: Attached scsi generic sg1 type 0
> [14618.728090] sd 10:0:0:0: [sdb] Spinning up disk...
> [14619.748918] ...ready
> 
> I tried different USB ports in a desperate hope of success...
> ...no, same problem.
> 
> Interestingly fdisk shows the following:
> 
> host> sudo fdisk -l /dev/sdb
> Disk /dev/sdb: 931.49 GiB, 1000170586112 bytes, 1953458176 sectors
> Disk model: Elements 25A2   
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes

This could be the key. Sector sizes have been changing from 512 to 4096
over many years. If your kernel has been updated to expect/use 4096 byte
sectors, it might not be able to read the disk properly.

> Disklabel type: dos
> Disk identifier: 0x16f2a91f
> 
> Device Boot StartEndSectors   Size Id Type
> /dev/sdb1   1 1953458175 1953458175 931.5G ee GPT
> 
> The type is shown as GPT...but the drive has a MSDOS partition table.
> 
> Reading my (old) internal harddrive with an external USB docking
> station is possible without any problems, though.
> 
> Unfortunatelu I have no space for 1T image of that drive -- otherwise
> I would have made an image copy and experiment with that.
> 
> So better ask, than sorry ;)
> 
> Is this fixable or did I lost my backups?
> 
Do you have access to an old kernel?

The other thing is try using gdisk (or that could be fdisk under another
name :-( But some partitioning schemes can write a GPT with protective
MBR - if you can find something that will take the MBR and write it as a
GPT that might help, too.

Cheers,
Wol

[gentoo-user] Trouble with backup harddisks