Re: [9fans] copying fossil filesystem to a bigger disk
Sorry, I was wrong : I am running plan9/fossil on a 1TB WD Caviar disk: WD1002FBYS, however, this is not what you wanted, sorry. ++pac On Thu, Oct 6, 2011 at 9:35 AM, Peter A. Cejchan tyap...@gmail.com wrote: i use WD Caviar Greenhttp://www.wdc.com/en/products/products.asp?DriveID=773model *WD20EARS (2TB SATA II*) without any problems. I installed from erik's 9atom.iso it is quite silent, but I have no idea how to measure its speed :-(
Re: [9fans] copying fossil filesystem to a bigger disk
fossil/conf /dev/sdE0/fossil fossil.conf fossil/conf -w /dev/sdE0/fossil fossil.conf This was exactly what I needed. Thank you! My migration from old to new drive is now complete.
Re: [9fans] copying fossil filesystem to a bigger disk
you're not the first person to make this mistake, so i should have remembered this problem. sorry. Please don't apologize. You are the one guiding the blind. you need to mount both new and old afresh in /n/ and copy using your destination as /n/new and source as /n/old. using / as your source doesn't work because you have many things other than the file server intended mounted. for example, your first error messages are because #c is mounted on /dev. Aha! Maybe I finally got it. At least no errors: su# con -l /srv/fscons prompt: fsys new config /dev/sdE1/fossil fsys new config /dev/sdE1/fossil prompt: fsys new open -AWVP fsys new open -AWVP q su# mount /srv/boot /n/old main su# mount /srv/boot /n/new new su# disk/mkfs -s /n/old -d /n/new -U -r file system made su# con -l /srv/fscons prompt: fsys new sync fsys new sync new sync: wrote 0 blocks prompt: fsys new halt fsys new halt q Now let's see if it boots.
Re: [9fans] copying fossil filesystem to a bigger disk
snarf-paste error. the command was:
su# disk/mkfs -s /n/old -d /n/new -U -r {echo +}
Re: [9fans] copying fossil filesystem to a bigger disk
Now let's see if it boots. Almost there. I took out the old drive and made the new one sdE0. It started booting, until: fossil(#S/sdE0/fossil)... fsOpen: can't find /dev/sdE1/fossil ... panic Does fossil store the device name somewhere on the disk? (The drive was sdE1 when I formatted it.) How can I change it to sdE0?
Re: [9fans] copying fossil filesystem to a bigger disk
Does fossil store the device name somewhere on the disk? (The drive was sdE1 when I formatted it.) How can I change it to sdE0? The configuration is stored at the 127kB offset of Fossil file system. Read configuration: fossil/conf /dev/sdE0/fossil fossil.conf Write configuration: fossil/conf -w /dev/sdE0/fossil fossil.conf Since June 16, the device parameter is now optional on the line fsys main config. -- David du Colombier
Re: [9fans] copying fossil filesystem to a bigger disk
that's a good question. if you just want to copy everything, i use the shell
idiom
disk/mkfs {echo +}
I ran this in /usr/bootes/ as bootes:
su# disk/mkfs -a -s / {echo +} arch
processing /fd/7
mkfs: /fd/7:1: can't open //dev/consctl: '//dev/consctl' permission denied
mkfs: /fd/7:1: can't open //dev/kprint: '//dev/kprint' device or
object already in use
I guess it wasn't the brightest idea, because the arch file grew much
bigger than all my files put together on the old drive.
Should I mount the new drive to /n/kfs and not use the -a option? I
was a little confused about these references to kfs while my
filesystem is fossil.
Re: [9fans] copying fossil filesystem to a bigger disk
No error, no crash. Just some files/dirs are not copied to new destination. du -s gives different number on linux vs. ext2srv/plan9 native./ Also, du -a | wc -l differ. I was not able to identify any system in which files/ dirs do not appear on ext2srv. First, i thought that dirs with 120 files, or so, are affected annd/or there's some size limit (like 2GB), but this was not the case. I can investigate it deeper, if you are interested. Should I send you a diff of du -a from linux vs. ext2srv? Thank you for your care, regards, Peter. On Fri, Oct 7, 2011 at 3:03 PM, Steve Simon st...@quintile.net wrote: Out of interest - what is the problem with ext2srv with too many files/subdirs? what error does it give, does it crash? -Steve
Re: [9fans] copying fossil filesystem to a bigger disk
su# disk/mkfs -a -s / {echo +} arch
processing /fd/7
mkfs: /fd/7:1: can't open //dev/consctl: '//dev/consctl' permission denied
mkfs: /fd/7:1: can't open //dev/kprint: '//dev/kprint' device or
object already in use
I guess it wasn't the brightest idea, because the arch file grew much
bigger than all my files put together on the old drive.
Should I mount the new drive to /n/kfs and not use the -a option? I
was a little confused about these references to kfs while my
filesystem is fossil.
you're not the first person to make this mistake, so i should
have remembered this problem. sorry.
you need to mount both new and old afresh in /n/ and copy
using your destination as /n/new and source as /n/old. using
/ as your source doesn't work because you have many things other
than the file server intended mounted. for example, your first
error messages are because #c is mounted on /dev.
- erik
Re: [9fans] copying fossil filesystem to a bigger disk
i think disk/mkfs is nearly idea for this, and isn't very dangerous. since your new fossil will start empty, you can't overwrite anything in the old fs. How do I generate the proto file? Do I have to go through an archive? Thank you for your patience.
Re: [9fans] copying fossil filesystem to a bigger disk
On Sun Oct 9 22:34:04 EDT 2011, slash.9f...@gmail.com wrote:
i think disk/mkfs is nearly idea for this, and isn't very dangerous. since
your
new fossil will start empty, you can't overwrite anything in the old fs.
How do I generate the proto file? Do I have to go through an archive?
Thank you for your patience.
that's a good question. if you just want to copy everything, i use the shell
idiom
disk/mkfs {echo +}
which is the same thing as
echo +/tmp/allproto
disk/mkfs /tmp/allproto
- erik
Re: [9fans] copying fossil filesystem to a bigger disk
On Thu, Oct 6, 2011 at 3:35 AM, Peter A. Cejchan tyap...@gmail.com wrote: i use WD Caviar Green model WD20EARS (2TB SATA II) without any problems. I installed from erik's 9atom.iso Did you toggle any jumpers on the drive? I finally gave up and returned it. A new day, a new disk (with 512 byte sectors). I installed mbr, partitioned the disk, prepped the plan9 partition and formatted fossil on the new drive. Here's how my old and new disks look. su# cat /dev/sdE0/ctl # old inquiry SAMSUNG SP2004C model SAMSUNG SP2004C serial S07GJ10Y522190 firmVM100-32 wwn 5f0015522190 flaglba llba smart nop udma6 reg task 50 cmd 4c017 serr 0 ci 0 is 0 sig 101 sstatus 123 cmd hpcp cr fr pod sud st modeauto sataii geometry 390721968 512 part data 0 390721968 part plan9 63 390716865 part 9fat 63 204863 part nvram 204863 204864 part fossil 204864 389668289 part swap 389668289 390716865 su# cat /dev/sdE1/ctl # new inquiry Hitachi HDS5C3020ALA632 model Hitachi HDS5C3020ALA632 serial ML0220FL0220F313JE2D firmML6OA580 wwn 5000cca369cfb243 flaglba llba smart power nop ata8 sct udma6 reg task 50 cmd 4c017 serr 0 ci 0 is 0 sig 101 sstatus 123 cmd hpcp cr fr pod sud st modeauto sataii geometry 3907029168 512 part data 0 3907029168 part plan9 63 3907024065 part 9fat 63 204863 part nvram 204863 204864 part fossil 204864 3905975489 part swap 3905975489 3907024065 su# fossil/conf /dev/sdE0/fossil # old fsys main config /dev/sdE0/fossil fsys main open -V -c 3000 su# fossil/conf /dev/sdE1/fossil # new fsys main config /dev/sdE1/fossil fsys main open -V -c 3000 I would like to copy 9fat, nvram and fossil from the old drive to the new drive. 9fat is no problem. But how about nvram? Can I just use dd? As for fossil, I have not been able to figure out how to do this. replica(1) looks like has the capability to do this, but I don't want to experiment with my live data. Someone who knows how to do this, please advise. Also, do I need to run any additional fossil commands? The goal is to retire the old disk and boot off the new one with the only difference being that there is more free space now.
Re: [9fans] copying fossil filesystem to a bigger disk
i use WD Caviar Green model WD20EARS (2TB SATA II) without any problems. I installed from erik's 9atom.iso Did you toggle any jumpers on the drive? I finally gave up and returned it. No. No jumpers at all are inserted. It worked just out-of-the-box. I had first to switch to Treat SATA as AHCI in BIOS...then I installed from 9atom.iso (not the newest version, however). I still have a huge amount of data on an ext2 formatted HD, because ext2srv had problems with too many files/subdirs. I think I will have to write a tar from linux to (another, unformatted HD) and then untar to fossil... is it an absolutely crazy idea?
Re: [9fans] copying fossil filesystem to a bigger disk
Out of interest - what is the problem with ext2srv with too many files/subdirs? what error does it give, does it crash? -Steve
Re: [9fans] copying fossil filesystem to a bigger disk
I would like to copy 9fat, nvram and fossil from the old drive to the new drive. 9fat is no problem. But how about nvram? Can I just use dd? sure. As for fossil, I have not been able to figure out how to do this. replica(1) looks like has the capability to do this, but I don't want to experiment with my live data. Someone who knows how to do this, please advise. Also, do I need to run any additional fossil commands? i think disk/mkfs is nearly idea for this, and isn't very dangerous. since your new fossil will start empty, you can't overwrite anything in the old fs. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
i use WD Caviar Greenhttp://www.wdc.com/en/products/products.asp?DriveID=773model *WD20EARS (2TB SATA II*) without any problems. I installed from erik's 9atom.iso it is quite silent, but I have no idea how to measure its speed :-( ++pac On Tue, Oct 4, 2011 at 1:07 AM, slash slash.9f...@gmail.com wrote: the way to interpret this information is you may use 512 byte sectors if you really want to suffer terrible performance (usually 1/3 the normal performance for reasonablly random workloads.) That doesn't sound tempting at all. I am still within Amazon's return window. Can anyone recommend a 2 TB SATA drive that works on our favorite operating system out of the box at full speed? If it's quiet and cheap, all the better. let me think a bit about the correct solutions to this. it's clear to me that we just can't assume 512-byte sectors any more. I knew Plan 9 is picky about hardware, but a hard disk? *sigh*
Re: [9fans] copying fossil filesystem to a bigger disk
On Tuesday 04 of October 2011 02:02:31 erik quanstrom wrote: xp won't use it correctly either. in fact, if you're using a standard fdisk layout, chances are things are a little sideways on nearly any os. in any event, if i were buying a 2t hard drive today, i'd get http://www.newegg.com/Product/Product.aspx?Item=N82E16822136365 This RE4-GP is actually `low-power' version -- it spins at 7.200RPM (they tout it as `IntelliPower'). RE4 is the `enterprise' 7.200 version. If you can stand a 1.5TB, I'd recommend Seagate ST31500341AS as fast at 7.200RPM, reliable and 512B sector based. -- dexen deVries [[[↓][→]]] http://xkcd.com/732/
Re: [9fans] copying fossil filesystem to a bigger disk
I said: You could try telling your BIOS to use the disk in ATA (IDE) mode, and see if that gives you 512-byte sector emulation. However I seem to recall posts from Erik advising that some chipsets have bugs in this mode which affect Plan 9. quans...@quanstro.net said: to put a point on it, ata is an abstracted command set. ide is a register set and protocol for delivering ata commands modeled on the isa bus. the disk interface (ide/ahci/something else) doesn't have anything to do with the sector size. the sector size is reported in the ident block, which emulation layers treat as an opaque sack of bits. But if you look at the drivers, disk interface (ide/ahci/something else) has everything to do with the sector size, because: - sdiahci.c uses drive-unit-secsize, which comes from an ata read capacity command in scisonline() - not from the ident block - sdata.c ignores drive-unit-secsize, and uses drive-secsize which is always set to 512 So, if the BIOS can be set to use IDE mode, the sdata driver will be used, and will treat the drive as if it has 512-byte sectors. This may or may not work, depending on what logical sector size actually means. I would have thought it meant that read/write commands are expressed in logical not physical sector terms; but then it seems odd that the WD20EARS reports 4096-byte blocksize in the read capacity command. My WD10EARS behaves differently: term% scuzz /dev/sdC0 block size: 512 capacity 1953525167 512 ok 8
Re: [9fans] copying fossil filesystem to a bigger disk
quans...@quanstro.net: you may use 512 byte sectors if you really want to suffer terrible performance (usually 1/3 the normal performance for reasonablly random workloads.) Why will performance be terrible using 512-byte logical sectors on a 4096-byte physical sector disk? Both fossil and venti use multi sector blocks (the default is 8kb), so as long as partitions are carefully aligned (as I suggested), won't all the actual disk transfers be correctly aligned and sized to take advantage of the large physical sectors?
Re: [9fans] copying fossil filesystem to a bigger disk
quans...@quanstro.net:
word 106 6003 [valid=1]
multiple log/phys? 1
log/phys 8
logical sector size 0 [valid=0]
On my WD10EARS, word 106 of the ATA identify block is 0, indicating
no support for logical sectors.
I've just tried the experiment of creating eight 1GB partitions
aligned at each possible logical sector offset, and found that
alignment doesn't seem to affect the writing speed significantly:
term% awk '/test/ {print $0,align $3 % 8}' /dev/sdC0/ctl
part test0 28249856 30249856 align 0
part test1 30249857 32249857 align 1
part test2 32249858 34249858 align 2
part test3 34249859 36249859 align 3
part test4 36249860 38249860 align 4
part test5 38249861 40249861 align 5
part test6 40249862 42249862 align 6
part test7 42249863 44249863 align 7
term% for (i in /dev/sdC0/test?) { time rc -c 'dd -if /dev/zero -of '$i' -bs 4k
-count 25 [2]/dev/null' }
0.57u 8.43s 20.40r rc -c dd -if /dev/zero -of /dev/sdC0/test0 -bs 4k
-count 25 [2]/dev/null
0.50u 8.65s 20.71r rc -c dd -if /dev/zero -of /dev/sdC0/test1 -bs 4k
-count 25 [2]/dev/null
0.64u 8.44s 20.73r rc -c dd -if /dev/zero -of /dev/sdC0/test2 -bs 4k
-count 25 [2]/dev/null
0.60u 8.86s 20.54r rc -c dd -if /dev/zero -of /dev/sdC0/test3 -bs 4k
-count 25 [2]/dev/null
0.64u 8.70s 20.77r rc -c dd -if /dev/zero -of /dev/sdC0/test4 -bs 4k
-count 25 [2]/dev/null
0.55u 9.00s 20.62r rc -c dd -if /dev/zero -of /dev/sdC0/test5 -bs 4k
-count 25 [2]/dev/null
0.48u 8.51s 20.61r rc -c dd -if /dev/zero -of /dev/sdC0/test6 -bs 4k
-count 25 [2]/dev/null
0.59u 8.16s 20.78r rc -c dd -if /dev/zero -of /dev/sdC0/test7 -bs 4k
-count 25 [2]/dev/null
This all seems to contradict WD's claim (*) that the WD10EARS is an
advanced format drive.
(*) http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-701229.pdf
Re: [9fans] copying fossil filesystem to a bigger disk
perhaps there's an option bit? On 4 October 2011 12:28, Richard Miller 9f...@hamnavoe.com wrote: This all seems to contradict WD's claim (*) that the WD10EARS is an advanced format drive.
Re: [9fans] copying fossil filesystem to a bigger disk
perhaps there's an option bit? If the drive was physically formatted with 4096-byte sectors, I can't see how changing a logical bit could prevent unaligned writes from causing a read-modify-write cycle.
Re: [9fans] copying fossil filesystem to a bigger disk
On Tue Oct 4 07:29:18 EDT 2011, 9f...@hamnavoe.com wrote: quans...@quanstro.net: word 106 6003 [valid=1] multiple log/phys? 1 log/phys 8 logical sector size 0 [valid=0] On my WD10EARS, word 106 of the ATA identify block is 0, indicating no support for logical sectors. the other guy has a WD20EARS which is a completely different drive. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
On Tue Oct 4 08:02:13 EDT 2011, 9f...@hamnavoe.com wrote: perhaps there's an option bit? If the drive was physically formatted with 4096-byte sectors, I can't see how changing a logical bit could prevent unaligned writes from causing a read-modify-write cycle. you aren't up with the level of sleeze involved. what some drive makers do is give you an option to offset lbas by 1. since the standard fat layout tend to give (lba%16) == 15, if the lba used at the interface is 64 (unaligned) the drive will add 1 and get an aligned lba. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
i see that to find all the answers i might have to pay $300 for an individual membership to another organisation; probably the membership will be about as useful as the VGA ones. thieving bastards.
Re: [9fans] copying fossil filesystem to a bigger disk
the other guy has a WD20EARS which is a completely different drive. Sure, but all the EARS series are claimed (in the spec sheet ref'd in my previous message) to be advanced format i.e. 4096-byte physical sectors.
Re: [9fans] copying fossil filesystem to a bigger disk
On Tuesday 04 of October 2011 14:13:40 Charles Forsyth wrote: i see that to find all the answers i might have to pay $300 for an individual membership to another organisation; probably the membership will be about as useful as the VGA ones. thieving bastards. if you'll excuse, Q. What's so great about ISO standardization? A. It is often said that one of the advantages of SGML over some other, proprietary, generic markup scheme is that nobody owns the standard. While this is not strictly true, the ISO's pricing policy certainly has helped to keep the number of people who do own a copy of the Standard at an absolute minimum. from http://www.flightlab.com/~joe/sgml/faq-not.txt -- dexen deVries [[[↓][→]]] http://xkcd.com/732/
Re: [9fans] copying fossil filesystem to a bigger disk
On Tue Oct 4 08:16:00 EDT 2011, 9f...@hamnavoe.com wrote: the other guy has a WD20EARS which is a completely different drive. Sure, but all the EARS series are claimed (in the spec sheet ref'd in my previous message) to be advanced format i.e. 4096-byte physical sectors. sure, that just specifies the physical layout. not what they claim at the interface. drives aren't hardware in the traditional sense anymore. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
On Tue Oct 4 08:09:55 EDT 2011, charles.fors...@gmail.com wrote: no, i meant to select what the drive advertises. it would be a bit disconcerting if flipping a bit had to reformat a drive! well they advertize *two* different sector sizes, a logical size and a physical size. the drive never changes the physical format, but may do read/modify/write cycles in the case of writes that don't cover a full physical sector. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
part test7 42249863 44249863 align 7
term% for (i in /dev/sdC0/test?) { time rc -c 'dd -if /dev/zero -of '$i' -bs
4k -count 25 [2]/dev/null' }
0.57u 8.43s 20.40r rc -c dd -if /dev/zero -of /dev/sdC0/test0 -bs 4k
-count 25 [2]/dev/null
0.50u 8.65s 20.71r rc -c dd -if /dev/zero -of /dev/sdC0/test1 -bs 4k
-count 25 [2]/dev/null
0.64u 8.44s 20.73r rc -c dd -if /dev/zero -of /dev/sdC0/test2 -bs 4k
-count 25 [2]/dev/null
0.60u 8.86s 20.54r rc -c dd -if /dev/zero -of /dev/sdC0/test3 -bs 4k
-count 25 [2]/dev/null
0.64u 8.70s 20.77r rc -c dd -if /dev/zero -of /dev/sdC0/test4 -bs 4k
-count 25 [2]/dev/null
0.55u 9.00s 20.62r rc -c dd -if /dev/zero -of /dev/sdC0/test5 -bs 4k
-count 25 [2]/dev/null
0.48u 8.51s 20.61r rc -c dd -if /dev/zero -of /dev/sdC0/test6 -bs 4k
-count 25 [2]/dev/null
0.59u 8.16s 20.78r rc -c dd -if /dev/zero -of /dev/sdC0/test7 -bs 4k
-count 25 [2]/dev/null
25*4096/20.78 = 49 mb/s. this is less than 1/2 the available
bandwidth giving the drive lots of wiggle room. and since you're
doing sequential i/o the drive can do write combining.
- erik
Re: [9fans] copying fossil filesystem to a bigger disk
i see that to find all the answers i might have to pay $300 for an individual membership to another organisation; google ata.command.set filetype:pdf gives you lots of free draft versions, pick one.
Re: [9fans] copying fossil filesystem to a bigger disk
i had in mind something that would make the drive look as much like an old drive as possible, including lying about simply everything, including the underlying physical sector size. surely that's suitable for the PC heritage. On 4 October 2011 13:17, erik quanstrom quans...@quanstro.net wrote: On Tue Oct 4 08:09:55 EDT 2011, charles.fors...@gmail.com wrote: no, i meant to select what the drive advertises. it would be a bit disconcerting if flipping a bit had to reformat a drive! well they advertize *two* different sector sizes, a logical size and a physical size. the drive never changes the physical format, but may do read/modify/write cycles in the case of writes that don't cover a full physical sector. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
25*4096/20.78 = 49 mb/s. this is less than 1/2 the available bandwidth giving the drive lots of wiggle room. and since you're doing sequential i/o the drive can do write combining. Is there any experiment I can do (not involving a crowbar and a microscope) to find out the real physical sector size? Bigger transfers get more of the bandwidth, but then a smaller proportion of the transfer needs read/modify/write. I could do random addressing but then I would expect seek time to dominate. Not that it matters (my drive works fine), but I'm curious!
Re: [9fans] copying fossil filesystem to a bigger disk
i had in mind something that would make the drive look as much like an old drive as possible, including lying about simply everything, including the underlying physical sector size. surely that's suitable for the PC heritage. we used to care what the c/h/s of a drive was. now we don't. anything with a raid underneath or flash memory is going to lie about the sector/erase block size. the writing is on the wall. logical sector size != physical sector size. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
On Tuesday 04 of October 2011 14:33:27 Richard Miller wrote:
25*4096/20.78 = 49 mb/s. this is less than 1/2 the available
bandwidth giving the drive lots of wiggle room. and since you're
doing sequential i/o the drive can do write combining.
Is there any experiment I can do (not involving a crowbar and a
microscope) to find out the real physical sector size? Bigger
transfers get more of the bandwidth, but then a smaller proportion
of the transfer needs read/modify/write. I could do random addressing
but then I would expect seek time to dominate.
compare write bandwidth for 4096B data chunks at offset modulo 512B:
(n*512B+k), where `n' is random. compare 8 runs, each with const `k' from { 0,
1, ...7 }. sync after every write. write much more than drive cache size at
each run (probably 64MB on modern HDs).
if your drive is 4k, one of the runs will be at exact sector boundary, and 7
others will read-modify-write two sectors every time. thus one run will have
much better performance.
if your drive is 512b, all runs will have same performance.
--
dexen deVries
[[[↓][→]]]
http://xkcd.com/732/
Re: [9fans] copying fossil filesystem to a bigger disk
On Tuesday 04 of October 2011 14:52:49 dexen deVries wrote:
compare write bandwidth for 4096B data chunks at offset modulo 512B:
(n*512B+k), where `n' is random. compare 8 runs, each with const `k' from {
0, 1, ...7 }. sync after every write. write much more than drive cache
size at each run (probably 64MB on modern HDs).
oops, was supposed to be (n+k) * 512B.
if you expect seek times to dominate, perhaps
for (..; ++n; ..) {
write 4096B @ (n*8+k) * 512B; sync;
}
would be better -- that is; walk sequentially but sync in 4096B chunks
possibly across 4k sectors (when `k' mis-aligns).
--
dexen deVries
[[[↓][→]]]
http://xkcd.com/732/
Re: [9fans] copying fossil filesystem to a bigger disk
Is there any experiment I can do (not involving a crowbar and a
microscope) to find out the real physical sector size? Bigger
transfers get more of the bandwidth, but then a smaller proportion
of the transfer needs read/modify/write. I could do random addressing
but then I would expect seek time to dominate.
Not that it matters (my drive works fine), but I'm curious!
i think the attached program ought to work. here are some local
results. no 4k-sector drives. sorry. if this doesn't work, especially for
writing, it's probablly a caching effect. you may need to run till
the cache fills up (64mb or so) and then start timing.
ioszr (s) riops w wiops model
512 12.227 81.78 5.531 180.78 Hitachi HUA721050KLA330
409612.363 80.88 5.649 176.99 Hitachi HUA721050KLA330
512 5.957 167.85 2.636 379.26 SEAGATE ST373455SS (sas)
40965.946 168.15 2.665 375.19 SEAGATE ST373455SS (sas)
512 0.181 5494.54 0.168 5941.86 SSDSA2SH064G1GC INTEL (ssd)
40960.317 3150.64 0.213 4689.26 SSDSA2SH064G1GC INTEL (ssd)
as already known, none of these appear to have any sector size wierdness.
surprising, considering the intel ssd is attached to a openrd!
- erik#include u.h
#include libc.h
enum {
Ios = 1000, /* number of ios / test */
Nanoi = 10, /* 1/nano */
Microi = 100,
};
enum {
Read= 10,
Write = 11,
};
static int otab[4] = {
[Read] OREAD,
[Write] OWRITE,
[Read|Write]ORDWR,
};
static char *rwtab[4] = {
[Read] read,
[Write] write,
[Read|Write]rdwr,
};
void
rio(int fd, uvlong byte0, uvlong bytes, uint ss, int rw)
{
char *buf;
int i;
uvlong maxlba, lba, t, x;
long (*io)(int, void*, long, vlong);
maxlba = (bytes - byte0) / ss;
if(rw == Read)
io = pread;
else if(rw == Write)
io = pwrite;
else{
io = nil; /* compiler noise */
sysfatal(not prepared to mix read/write yet);
}
buf = malloc(ss);
if(buf == nil)
sysfatal(malloc);
srand(nsec());
t = -nsec();
for(i = 0; i Ios; i++){
lba = frand()*maxlba; /* awful. no vlnrand() */
io(fd, buf, ss, byte0 + lba*ss);
}
t += nsec();
free(buf);
print(%s %lld.%03lld\n, rwtab[rw], t/Nanoi, (t%Nanoi)/Microi);
x = (uvlong)Ios*(uvlong)Nanoi*100;
x /= t;
print(%lld.%02lld iops\n, x/100, (x%100));
}
void
usage(void)
{
fprint(2, usage: randomio [-rw] [-o offset] [-s sectorsz] [file
...]\n);
exits(usage);
}
void
main(int argc, char **argv)
{
int i, byte0, fd, rw, ss;
uvlong bytes;
Dir *d;
rw = 0;
ss = 512;
byte0 = 0;
ARGBEGIN{
case 'r':
rw |= Read;
break;
case 'w':
rw |= Write;
break;
case 'o':
byte0 = atoi(EARGF(usage()));
break;
case 's':
ss = atoi(EARGF(usage()));
break;
default:
usage();
}ARGEND;
if(rw == 0)
rw = Read;
for(i = 0; i argc; i++){
d = dirstat(argv[i]);
if(d == nil)
sysfatal(dirstat: %r);
bytes = d-length;
free(d);
fd = open(argv[i], otab[rw]);
if(fd == -1)
sysfatal(open: %r);
if(rw Read)
rio(fd, byte0, bytes, ss, Read);
if(rw Write)
rio(fd, byte0, bytes, ss, Write);
close(fd);
}
exits();
}
Re: [9fans] copying fossil filesystem to a bigger disk
i think the attached program ought to work. Doing i/o to random blocks, aren't you mostly measuring seek time?
Re: [9fans] copying fossil filesystem to a bigger disk
On Tue Oct 4 11:57:27 EDT 2011, 9f...@hamnavoe.com wrote: i think the attached program ought to work. Doing i/o to random blocks, aren't you mostly measuring seek time? the drive will write-combine sequential io, so that's not an option. i've definately seen artifacts when doing completely random io on 4k drives. you can try rewriting it to step by 8k and retry the test. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
you can try rewriting it to step by 8k and retry the test. Bingo, thanks. I was still trying to figure out how to force the drive to sync as Dexen suggested (maybe using scsicmd to issue an ata FLUSH CACHE command?), but stepping by 8k does reveal a strong pattern. Writing 25000 4k records stepping by 8k takes sector offset 1-7 mod 8: 27-30 sec sector offset 0 mod 8: 14-17 sec Conclusion: WD10EARS has 4k byte physical sectors but is pretty good at concealing it.
Re: [9fans] copying fossil filesystem to a bigger disk
you can try rewriting it to step by 8k and retry the test. Bingo, thanks. I was still trying to figure out how to force the drive to sync as Dexen suggested (maybe using scsicmd to issue an ata FLUSH CACHE command?), but stepping by 8k does reveal a strong pattern. you can either use the raw device to send raw write dma ext fua, or you can explicitly turn off the cache. flush cache flushes anything from the cache, so it can be misleading. Writing 25000 4k records stepping by 8k takes sector offset 1-7 mod 8: 27-30 sec sector offset 0 mod 8: 14-17 sec Conclusion: WD10EARS has 4k byte physical sectors but is pretty good at concealing it. it's amazing how fast you can be if you don't care about losing data on power loss. :-) - erik
Re: [9fans] copying fossil filesystem to a bigger disk
that's certainly the linux way, although to be fair, its fsck does a really good job of making a scramble worse. that reminds me that i've still got a big file with a scrambled partition to unscramble to get back some big data i wanted to keep. On 4 October 2011 17:45, erik quanstrom quans...@quanstro.net wrote: it's amazing how fast you can be if you don't care about losing data on power loss. :-)
Re: [9fans] copying fossil filesystem to a bigger disk
On Tuesday 04 October 2011 19:52:10 Charles Forsyth wrote: that's certainly the linux way, although to be fair, its fsck does a really good job of making a scramble worse. for those stuck on linux: http://www.nilfs.org/en/ is quite immune to inconsistencies on dirty shutdown, yet performs well for both read and write (except file removal is slow in some cases). upon mount of dirty filesystem, nilfs2 reverts to last written transaction. aside of the usual 4kB low-level clusters, there's high-level garbage- collection unit of 8MB. any free space on filesystem is at least 8MB large; that provides for decent write performance on highly fragmented and almost full harddrive. this also helps performance on SSD -- little or no erases happen across erase units. oh, and by default it doesn't come with fsck ;-) (available only on a separate branch of utils). -- dexen deVries It's called trolling. It's been done since there were bangs in people's email addresses. thaumaturgy, on HN
Re: [9fans] copying fossil filesystem to a bigger disk
On Tue, 04 Oct 2011 12:45:58 EDT erik quanstrom quans...@quanstro.net wrote: Writing 25000 4k records stepping by 8k takes sector offset 1-7 mod 8: 27-30 sec sector offset 0 mod 8: 14-17 sec Conclusion: WD10EARS has 4k byte physical sectors but is pretty good at concealing it. Weren't there 4k sector disks that mapped the 1st 4k sector to just the mbr? So the next four 512B sectors got mapped to the next 4k sector and so on. As a result your nicely aligned on 4k boundaries IO wouldn't be aligned at all. Are there still such disks one should avoid? it's amazing how fast you can be if you don't care about losing data on power loss. :-) it's amazing how simple you life can be if you don't care about losing data :-) I call it garbage collection! Because if you really cared you'd have already saved your stuff! [Cue George Carlin on stuff]
Re: [9fans] copying fossil filesystem to a bigger disk
On Tue Oct 4 14:25:29 EDT 2011, ba...@bitblocks.com wrote: On Tue, 04 Oct 2011 12:45:58 EDT erik quanstrom quans...@quanstro.net wrote: Writing 25000 4k records stepping by 8k takes sector offset 1-7 mod 8: 27-30 sec sector offset 0 mod 8: 14-17 sec Conclusion: WD10EARS has 4k byte physical sectors but is pretty good at concealing it. Weren't there 4k sector disks that mapped the 1st 4k sector to just the mbr? So the next four 512B sectors got mapped to the next 4k sector and so on. that's not what they do. they offset the lba by one so that the mbr is actually at offset 512 in the first lba, and therefore sector 63 (where most of the action starts, is at (512*63 + 512)/4k offset (512*63 + 512)%4k which is physical sector 8, offset 0. i think the wd tool can change this alignment. i'm sure there's some magic set features commands. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
On Mon Oct 3 19:08:49 EDT 2011, slash.9f...@gmail.com wrote: the way to interpret this information is you may use 512 byte sectors if you really want to suffer terrible performance (usually 1/3 the normal performance for reasonablly random workloads.) That doesn't sound tempting at all. I am still within Amazon's return window. Can anyone recommend a 2 TB SATA drive that works on our favorite operating system out of the box at full speed? If it's quiet and cheap, all the better. let me think a bit about the correct solutions to this. it's clear to me that we just can't assume 512-byte sectors any more. I knew Plan 9 is picky about hardware, but a hard disk? *sigh* relax, it's really just the mbr code and fdisk that don't like a non 512-byte sector size. in fact, as long as you don't boot from it, you can use the disk as-is. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
the way to interpret this information is you may use 512 byte sectors if you really want to suffer terrible performance (usually 1/3 the normal performance for reasonablly random workloads.) That doesn't sound tempting at all. I am still within Amazon's return window. Can anyone recommend a 2 TB SATA drive that works on our favorite operating system out of the box at full speed? If it's quiet and cheap, all the better. let me think a bit about the correct solutions to this. it's clear to me that we just can't assume 512-byte sectors any more. I knew Plan 9 is picky about hardware, but a hard disk? *sigh*
Re: [9fans] copying fossil filesystem to a bigger disk
On Mon Oct 3 19:08:49 EDT 2011, slash.9f...@gmail.com wrote: the way to interpret this information is you may use 512 byte sectors if you really want to suffer terrible performance (usually 1/3 the normal performance for reasonablly random workloads.) That doesn't sound tempting at all. I am still within Amazon's return window. Can anyone recommend a 2 TB SATA drive that works on our favorite operating system out of the box at full speed? If it's quiet and cheap, all the better. let me think a bit about the correct solutions to this. it's clear to me that we just can't assume 512-byte sectors any more. I knew Plan 9 is picky about hardware, but a hard disk? *sigh* xp won't use it correctly either. in fact, if you're using a standard fdisk layout, chances are things are a little sideways on nearly any os. in any event, if i were buying a 2t hard drive today, i'd get http://www.newegg.com/Product/Product.aspx?Item=N82E16822136365 since i'm paranoid about hard drives and only get enterprise ones. if you're not, then (most) anything with 512-byte sectors should be no troubles. the easiest way to check is to do some math. multiply the number of sectors (that's always on the datasheet) by the 512, and see if that equals the claimed capacity. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
if you (slash) could just grab the atazz binary (ftp://ftp.quanstro.net/other/8.atazz) and send me the output of echo 'identify device' | 8.atazz -r [2=] /dev/sdE0 /tmp/somefile that would be great. thanks! Here you go. Again, sdE0 is the old drive, sdE1 is the new one. slash sdE0.out Description: Binary data sdE1.out Description: Binary data
Re: [9fans] copying fossil filesystem to a bigger disk
On Sat Oct 1 06:37:52 EDT 2011, slash.9f...@gmail.com wrote: if you (slash) could just grab the atazz binary (ftp://ftp.quanstro.net/other/8.atazz) and send me the output of echo 'identify device' | 8.atazz -r [2=] /dev/sdE0 /tmp/somefile that would be great. thanks! Here you go. Again, sdE0 is the old drive, sdE1 is the new one. thanks! evidently, [2=] backfired and actually copied stderr to stdoutput. there's no need to resend the ident blocks, since i've fixed it up, but untill this bug is fixed, this echo 'identify device' | 8.atazz -r /dev/sdE0 /tmp/somefile or this echo 'identify device' | 8.atazz -r /dev/sdE0 /tmp/somefile [2]/dev/null would be better - erik
Re: [9fans] copying fossil filesystem to a bigger disk
echo 'identify device' | 8.atazz -r /dev/sdE0 /tmp/somefile would be better Here goes. sdE0.out Description: Binary data sdE1.out Description: Binary data
Re: [9fans] copying fossil filesystem to a bigger disk
since i don't have one of these drives, i wrote a little
program to dig into the ident block.
a small aside. ata disks have two sector sizes. (groan.)
there's the logical sector size and the physical sector size.
by default they are both 512. the ears drive sets the
set logical sector size to 512 and the physical sector
size to 4096 by reporting that there are 8 logicals/physical.
here's the output of this program
3,907,029,168 4096-byte sectors
serial: WD-WCAD-WCAZA995782551.0AB51WDC WD20EARS-00MVWB0
firmware: 51.0AB51WDC WD20EARS-00MVWB0
model: WDC WD20EARS-00MVWB0
pflag: llba smart nop atapi16 sct
note that there are not 3e9 4k sectors there are 3e9/8 4k sectors;
fis has got that wrong. i've never had an 4k sector drive to test
with.
the way to interpret this information is you may use 512
byte sectors if you really want to suffer terrible performance
(usually 1/3 the normal performance for reasonablly random
workloads.)
let me think a bit about the correct solutions to this. it's clear
to me that we just can't assume 512-byte sectors any more.
in the mean time, you can change this
; diff -c /sys/src/libfis/fis.c /tmp/fis.c
/sys/src/libfis/fis.c:320,326 - /tmp/fis.c:320,326
f-lsectsz = sw * 2;
if(i 113)
f-physshift = i 7;
- return f-lsectsz * (1f-physshift);
+ return f-lsectsz /* * (1f-physshift) */;
}
and then recompile your libfis and your kernel, and you
should be good-to-go.
- erik#include u.h
#include libc.h
#include fis.h
typedef struct Sfisx Sfisx;
struct Sfisx {
uvlong sectors;
uintsecsize;
charserial[20];
charfirmware[8];
charmodel[8];
uvlong wwn;
Sfis;
};
int
idss0(Sfis *f, ushort *id)
{
uint sw, i;
if(f-sig16 == 0xeb14)
return 0;
f-lsectsz = 512;
f-physshift = 0;
i = id16(id, 106);
print(word 106 %.4ux [valid=%d]\n, i, i14 == 1);
print( multiple log/phys? %d\n, (i 113) != 0);
if((i 113) != 0)
print( log/phys %d\n, 1(i7));
sw = id32(id, 117);
print( logical sector size %d [valid=%d]\n, sw*2, (i 112) != 0);
return 0;
/*
if(i 14 != 1)
return f-lsectsz;
if((sw = id32(id, 117)) = 256)
f-lsectsz = sw * 2;
if(i 113)
f-physshift = i 7;
return f-lsectsz * (1f-physshift);
*/
}
void
main(void)
{
char buf[1024], buf2[128], *s2;
ushort *id;
int n, i, l;
Sfisx *f;
s2 = az ;
l = strlen(s2);
n = read(0, buf, sizeof buf);
if(n 512)
sysfatal(bad ident: %r);
/* fix my mistake in asking for buffer */
for(i = 0; i = n-l; i++)
if(memcmp(buf+i, s2, l) == 0){
memcpy(buf+i, buf+i+l, n-(i+l));
n -= l;
}
if(n != 512)
sysfatal(bad ident: %d bytes\n, n);
// write(1, buf, 512);
f = malloc(sizeof *f);
if(f == nil)
sysfatal(malloc);
memset(f, 0, sizeof *f);
id = (ushort*)buf;
f-sectors = idfeat(f, id);
f-secsize = idss(f, id);
f-wwn = idwwn(f, id);
idmove(f-serial, id+10, 20);
idmove(f-firmware, id+23, 8);
idmove(f-model, id+27, 40);
print(%,lld %d-byte sectors\n, f-sectors, f-secsize);
print(serial: %s\n, f-serial);
print(firmware: %s\n, f-firmware);
print(model: %s\n, f-model);
pflag(buf2, buf2+sizeof buf2, f);
print(pflag: %s\n, buf2);
idss0(f, id);
exits();
}
Re: [9fans] copying fossil filesystem to a bigger disk
i forgot the interesting part of the program output word 106 6003 [valid=1] multiple log/phys? 1 log/phys 8 logical sector size 0 [valid=0] the logical sector size is by default 512, so the fact the logical sector size is invalid means it's 512. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
inquiry WDC WD20EARS-00MVWB0 ... The new drive is completely blank. It has 4096 byte sector size, and disk/mbr fails. su# disk/mbr -m /386/mbr /dev/sdE1/data mbr: secsize 4096 invalid Interesting. I have a WD10EARS which lies about its sector size - it reports 512 although it's really 4096. On that drive disk/mbr works as normal, but after fdisk and prep I had to readjust some partitions by hand because the cylinder size is 255*63*512, not a multiple of 4096. Plan 9 fdisk aligns things to a cylinder boundary, but the partitions need to be aligned to a multiple of 8 512-byte sectors from the beginning of the disk, or else every write would require a read-modify-write cycle. I suspect disk/mbr isn't the only place in Plan 9 where sector size is assumed to be 512. Has anyone else had experience with these advanced format drives?
Re: [9fans] copying fossil filesystem to a bigger disk
I have a WD10EARS which lies about its sector size Some further information: I'm using the sdata driver rather than sdiahci because my motherboard doesn't support ahci. This may be why I'm seeing the disk as having 512-byte sectors not 4096. You could try telling your BIOS to use the disk in ATA (IDE) mode, and see if that gives you 512-byte sector emulation. However I seem to recall posts from Erik advising that some chipsets have bugs in this mode which affect Plan 9.
Re: [9fans] copying fossil filesystem to a bigger disk
it's impressive that by 2011 we are unable to drive disks in as straightforward a manner as in the 1970s. better still, there are several competing miserable standards (and i'm not even including USB).
Re: [9fans] copying fossil filesystem to a bigger disk
On Fri Sep 30 05:37:57 EDT 2011, fors...@terzarima.net wrote: it's impressive that by 2011 we are unable to drive disks in as straightforward a manner as in the 1970s. better still, there are several competing miserable standards (and i'm not even including USB). this is one case where i don't think the disk driver is at fault. it's the mbr/pbs process claiming it will behave badly with a non-512-byte sector size. - erik
Re: [9fans] copying fossil filesystem to a bigger disk
On Fri Sep 30 04:55:43 EDT 2011, 9f...@hamnavoe.com wrote: I have a WD10EARS which lies about its sector size Some further information: I'm using the sdata driver rather than sdiahci because my motherboard doesn't support ahci. This may be why I'm seeing the disk as having 512-byte sectors not 4096. You could try telling your BIOS to use the disk in ATA (IDE) mode, and see if that gives you 512-byte sector emulation. However I seem to recall posts from Erik advising that some chipsets have bugs in this mode which affect Plan 9. to put a point on it, ata is an abstracted command set. ide is a register set and protocol for delivering ata commands modeled on the isa bus. the disk interface (ide/ahci/something else) doesn't have anything to do with the sector size. the sector size is reported in the ident block, which emulation layers treat as an opaque sack of bits. there are several options in the spec (the 4k sector compatability chapter is ~ 50 pages), so perhaps there's something to be done about it by having the disk driver lie creatively. unfortunately, i don't have one of these disks. the long-term solution is to fix mbr and fdisk to play nicely with 4096-byte sectors. (if you want to do this yourself, you can use the sdloop driver. it will use whatever sector size you give it.) if you (slash) could just grab the atazz binary (ftp://ftp.quanstro.net/other/8.atazz) and send me the output of echo 'identify device' | 8.atazz -r [2=] /dev/sdE0 /tmp/somefile that would be great. thanks! - erik
Re: [9fans] copying fossil filesystem to a bigger disk
On Fri, 30 Sep 2011 10:44:01 +0100 Charles Forsyth fors...@terzarima.net wrote: it's impressive that by 2011 we are unable to drive disks in as straightforward a manner as in the 1970s. better still, there are several competing miserable standards (and i'm not even including USB). Not to be too depressive, but I thought this was the story of computing. I've seen little else, at least as a desktop Linux user since '98 or so.
Re: [9fans] copying fossil filesystem to a bigger disk
Do you use Fossil with or without Venti? There is many way to do it. Without Venti, you could use replica(1) to copy your file system from the old Fossil to the new Fossil. Fossil without Venti. Work has kept me busy but now I finally managed to attach the new drive to the system. Plan9 sees it as sdE1. The old drive is sdE0. (Booting off sdE0. This server does not have a CDROM drive.) old drive: su# cat /dev/sdE0/ctl inquiry SAMSUNG SP2004C model SAMSUNG SP2004C serial S07GJ10Y522190 firmVM100-32 wwn 5f0015522190 flaglba llba smart nop udma6 reg task 50 cmd 4c017 serr 0 ci 0 is 0 sig 101 sstatus 123 cmd hpcp cr fr pod sud st modeauto sataii geometry 390721968 512 part data 0 390721968 part plan9 63 390716865 part 9fat 63 204863 part nvram 204863 204864 part fossil 204864 389668289 part swap 389668289 390716865 new drive: su# cat /dev/sdE1/ctl inquiry WDC WD20EARS-00MVWB0 model WDC WD20EARS-00MVWB0 serial WD-WCAD-WCAZA9957825 firm51.0AB51 wwn 50014ee2b0ed5bde flaglba llba smart nop ata8 sct udma6 reg task 50 cmd 4c017 serr 0 ci 0 is 0 sig 101 sstatus 123 cmd hpcp cr fr pod sud st modeauto sataii geometry 3907029168 4096 part data 0 3907029168 The new drive is completely blank. It has 4096 byte sector size, and disk/mbr fails. su# disk/mbr -m /386/mbr /dev/sdE1/data mbr: secsize 4096 invalid How to get around this? I don't want to install another OS just to get the MBR in...
Re: [9fans] copying fossil filesystem to a bigger disk
I am about to upgrade the disk on my cpu/disk server to a bigger one, and I want to maintain all the data. What is the most elegant way to do this? The new disk is blank. Do you use Fossil with or without Venti? There is many way to do it. Without Venti, you could use replica(1) to copy your file system from the old Fossil to the new Fossil. With Venti, you could use venti/rdarena to export arenas from the old Venti, then import them to the new Venti with venti/wrarena. Finally, format the Fossil partition from the last Fossil score. The new Venti should be already configured and running. You only need read access to the old Venti arena partition. You can do this by booting from another file server or using the Plan 9 CD image. See venti-backup(8). -- David du Colombier
Re: [9fans] copying fossil filesystem to a bigger disk
The advice I got a couple of years ago was to go for 7200RPM SATA3 disks, make sure they are branded as enterprise quality (worth paying for) and buy them from different manufacturers. Then mirror the disks and hopefully one disk will die before the other and thus you get some long term reliabality. I bought two 500Gb disks: WDC WD5002ABYS-01B1B0 SAMSUNG HE502IJ which have been solid and fast (so far). -Steve
Re: [9fans] copying fossil filesystem to a bigger disk
With Venti, you could use venti/rdarena to export arenas from the old Venti, then import them to the new Venti with venti/wrarena. Finally, format the Fossil partition from the last Fossil score. Honestly I would advise against this, I did it but it was really quite painful. When I upgraded my other (work) machine I just copied the old disk to a new partition of the same size and added another partition/arena and told venti about it. I rebuilt the indices and fossil from the arenas and was good to go in a couple of hours. Much easier/quicker. -Steve
