Re: writing to usb very slow
Replying to an old thread, On Feb 11 10:14:51, Aaron Stellman wrote: On Sat, Jan 16, 2010 at 01:44:49AM +0100, T. Tofus von Blisstein wrote: Hello, I have noticed that writing to a usb drive is slow. What does slow mean? It means that compared to other OS's. Which OS's? Yes, it hurts: the penguin. Hello there, I came across a cheap USB flash drive, particularly a SanDisk Cruzer U3: umass0 at uhub1 port 1 configuration 1 interface 0 SanDisk SanDisk Cruzer rev 2.00/2.00 addr 2 umass0: using SCSI over Bulk-Only scsibus3 at umass0: 2 targets, initiator 0 sd1 at scsibus3 targ 1 lun 0: SanDisk, SanDisk Cruzer, 8.02 SCSI0 0/direct removable sd1: 3863MB, 512 bytes/sec, 7913471 sec total Reading and writing from/to a raw disk /dev/rsd1c gives same exact rates as on other OSes: $ sudo dd if=/dev/rsd1c of=/dev/null bs=128k count=1600 1600+0 records in 1600+0 records out 209715200 bytes transferred in 8.822 secs (23771541 bytes/sec) $ sudo dd if=/dev/zero of=/dev/rsd1c bs=128k count=1600 1600+0 records in 1600+0 records out 209715200 bytes transferred in 19.253 secs (10892315 bytes/sec) Which are reasonable numbers. Now let's put a filesystem on it: $ sudo fdisk -i sd1 Do you wish to write new MBR and partition table? [n] y Writing MBR at offset 0. $ sudo disklabel -E sd1 Label editor (enter '?' for help at any prompt) a partition: [a] offset: [63] size: [7903917] FS type: [4.2BSD] w q No label changes. $ sudo newfs sd1a /dev/rsd1a: 3859.3MB in 7903916 sectors of 512 bytes 20 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each super-block backups (for fsck -b #) at: 32, 414688, 829344, 1244000, 1658656, 2073312, 2487968, 2902624, 3317280, 3731936, 4146592, 4561248, 4975904, 5390560, 5805216, 6219872, 6634528, 7049184, 7463840, 7878496, $ sudo mount /dev/sd1a /mnt/test; cd /mnt/test $ sudo dd if=/dev/zero of=./x bs=128k count=1600 1600+0 records in 1600+0 records out 209715200 bytes transferred in 254.224 secs (824923 bytes/sec) (confirmed that pengingwrites is 0 right after dd exits with systat iostat) As you can see here, 800KB/s is quite low, compared to raw read rate. What can be the cause? I don't know, but let's try formatting sd1a with frag-size of 8192 and block-size of 65536 ... (same exact sequence of steps as above, just adjust frag-size and block-size) ... $ time (sudo dd if=/dev/zero of=./x bs=128k count=3200; sync) 3200+0 records in 3200+0 records out 419430400 bytes transferred in 21.194 secs (19789417 bytes/sec) 0m42.44s real 0m0.00s user 0m0.33s syste 419430400/42.44=9,882,902.92 That's almost 10MB/s Perhaps people who are having problems with their slow USB flash drives can try the same experiment, see if it helps. indeed, this speeds up writes on my USB flash drives. However, during an _install_, there is no option to specify additional parameters to newfs - or is there? Besides escaping to a shell and running newfs myself? Would the installer run its own newfs over the newly disklabel'd partitions anyway? Write speed during an install to a USB flash matters - usually it takes ages just to untar the sets ... Jan
Re: writing to usb very slow
I don't know about installing on USB, but the copy to/from FAT ( -t msdos )has increased noticeable with new snapshots.
Re: writing to usb very slow
On 2012-05-03, Jan Stary h...@stare.cz wrote: However, during an _install_, there is no option to specify additional parameters to newfs - or is there? Not directly, but you /can/ set block/fragment sizes in the disklabel editor, which newfs honours if within bounds: enable [X]pert mode and modify/re-create your partitions. If you're looking to change other newfs options, e.g. reducing the number of inodes (which IMO is rather high on large filesystems) you would need to run newfs manually.
Re: writing to usb very slow
On Sat, Jan 16, 2010 at 01:44:49AM +0100, T. Tofus von Blisstein wrote: Hello, I have noticed that writing to a usb drive is slow. What does slow mean? It means that compared to other OS's. Which OS's? Yes, it hurts: the penguin. Hello there, I came across a cheap USB flash drive, particularly a SanDisk Cruzer U3: umass0 at uhub1 port 1 configuration 1 interface 0 SanDisk SanDisk Cruzer rev 2.00/2.00 addr 2 umass0: using SCSI over Bulk-Only scsibus3 at umass0: 2 targets, initiator 0 sd1 at scsibus3 targ 1 lun 0: SanDisk, SanDisk Cruzer, 8.02 SCSI0 0/direct removable sd1: 3863MB, 512 bytes/sec, 7913471 sec total Reading and writing from/to a raw disk /dev/rsd1c gives same exact rates as on other OSes: $ sudo dd if=/dev/rsd1c of=/dev/null bs=128k count=1600 1600+0 records in 1600+0 records out 209715200 bytes transferred in 8.822 secs (23771541 bytes/sec) $ sudo dd if=/dev/zero of=/dev/rsd1c bs=128k count=1600 1600+0 records in 1600+0 records out 209715200 bytes transferred in 19.253 secs (10892315 bytes/sec) Which are reasonable numbers. Now let's put a filesystem on it: $ sudo fdisk -i sd1 Do you wish to write new MBR and partition table? [n] y Writing MBR at offset 0. $ sudo disklabel -E sd1 Label editor (enter '?' for help at any prompt) a partition: [a] offset: [63] size: [7903917] FS type: [4.2BSD] w q No label changes. $ sudo newfs sd1a /dev/rsd1a: 3859.3MB in 7903916 sectors of 512 bytes 20 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each super-block backups (for fsck -b #) at: 32, 414688, 829344, 1244000, 1658656, 2073312, 2487968, 2902624, 3317280, 3731936, 4146592, 4561248, 4975904, 5390560, 5805216, 6219872, 6634528, 7049184, 7463840, 7878496, $ sudo mount /dev/sd1a /mnt/test; cd /mnt/test $ sudo dd if=/dev/zero of=./x bs=128k count=1600 1600+0 records in 1600+0 records out 209715200 bytes transferred in 254.224 secs (824923 bytes/sec) (confirmed that pengingwrites is 0 right after dd exits with systat iostat) As you can see here, 800KB/s is quite low, compared to raw read rate. What can be the cause? I don't know, but let's try formatting sd1a with frag-size of 8192 and block-size of 65536 ... (same exact sequence of steps as above, just adjust frag-size and block-size) ... $ time (sudo dd if=/dev/zero of=./x bs=128k count=3200; sync) 3200+0 records in 3200+0 records out 419430400 bytes transferred in 21.194 secs (19789417 bytes/sec) 0m42.44s real 0m0.00s user 0m0.33s syste 419430400/42.44=9,882,902.92 That's almost 10MB/s Perhaps people who are having problems with their slow USB flash drives can try the same experiment, see if it helps.
Re: writing to usb very slow
On Wed, Jan 20, 2010 at 12:34:19AM +0100, T. Tofus von Blisstein wrote: # mount -o async /dev/sd1a /mnt # date cp -r TEST/ /mnt umount /mnt date Wed Jan 20 00:17:54 CET 2010 Wed Jan 20 00:33:35 CET 2010 And you think because of all this noise, someone is going to sit down and fix it for you? Boy are you wrong... 2010/1/20 David Vasek va...@fido.cz: On Tue, 19 Jan 2010, T. Tofus von Blisstein wrote: Hello David, thanks. You're welcome. # mount /dev/sd1a /mnt # date cp -r TEST/ /mnt umount /mnt date Tue Jan 19 23:11:27 CET 2010 Tue Jan 19 23:29:12 CET 2010 So it's reduced a lot, but still it is much slower than... sorry guys, the penguin. Still, from 50 minutes to 18 to copy 256M is a significant improvement Try using softdep option for mount, possibly even the async option, if you like the risk, otherwise you are comparing two completely different modes of filesystem operation. Regards, David
Re: writing to usb very slow
Am 19.01.10 23:35, schrieb T. Tofus von Blisstein: disklabel -E /dev/rsd1c allocate all for partition a newfs /dev/rsd1c I hope it's just a typo in your email, since rsd1c is the whole disk and will erase any partitioning table you previously created.
Re: writing to usb very slow
Sounds like you are already on the right track, courtesy Peter Hansteen, so I'll simply support the direction you are going by telling you that I back up my systems (with a home-brew scheme that uses a combination of rsync and tar) to 7200 rpm SATA drives in USB shoeboxes with ext2 filesystems and have done so for years running Linux and for the last 10 months or so running OpenBSD on the same set of hardware (an assortment of Thinkpads, a Lenovo workstation, and a cheap HP desktop). I've had no performance problems with OpenBSD doing backups to these drives and therefore I haven't any measurements to quote. But I'd notice it if OpenBSD were a factor of 5 slower; it isn't. /Don Allen
Re: writing to usb very slow
Hello Donald, well, I cannot tell the same here. I still think it must be related to that usb 1.0 being reported in dmesg usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 Intel EHCI root hub rev 2.00/1.00 addr 1 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 Intel UHCI root hub rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 Intel UHCI root hub rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 Intel UHCI root hub rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 Intel UHCI root hub rev 1.00/1.00 addr 1 I would like to find out which is the one which is reported as 2.0 and I only have three USB ports... anyway, thanks to all of you for your help T. 2010/1/19 Donald Allen donaldcal...@gmail.com: Sounds like you are already on the right track, courtesy Peter Hansteen, so I'll simply support the direction you are going by telling you that I back up my systems (with a home-brew scheme that uses a combination of rsync and tar) to 7200 rpm SATA drives in USB shoeboxes with ext2 filesystems and have done so for years running Linux and for the last 10 months or so running OpenBSD on the same set of hardware (an assortment of Thinkpads, a Lenovo workstation, and a cheap HP desktop). I've had no performance problems with OpenBSD doing backups to these drives and therefore I haven't any measurements to quote. But I'd notice it if OpenBSD were a factor of 5 slower; it isn't. /Don Allen -- Pau
Re: writing to usb very slow
On Tue, Jan 19, 2010 at 1:58 PM, T. Tofus von Blisstein tuffst...@googlemail.com wrote: well, I cannot tell the same here. I still think it must be related to that usb 1.0 being reported in dmesg usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 Intel EHCI root hub rev 2.00/1.00 addr 1 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 Intel UHCI root hub rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 Intel UHCI root hub rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 Intel UHCI root hub rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 Intel UHCI root hub rev 1.00/1.00 addr 1 I would like to find out which is the one which is reported as 2.0 and I only have three USB ports... Plug it in. The kernel tells you.
Re: writing to usb very slow
Ted Unangst wrote: Plug it in. The kernel tells you. usbdevs -v also helps regards, Robert
Re: writing to usb very slow
Hello, this is an example. Attached is a 1GB (fat!) usb memory stick. It took 40 minutes to copy 285M. This one was Jan 19 21:18:04 hux /bsd: umass0: using SCSI over Bulk-Only Jan 19 21:18:04 hux /bsd: scsibus1 at umass0: 2 targets, initiator 0 Jan 19 21:18:04 hux /bsd: sd1 at scsibus1 targ 1 lun 0: -Pretec, 01GB, 2.00 SCSI2 0/direct removable Jan 19 21:18:04 hux /bsd: sd1: 983MB, 512 bytes/sec, 2015231 sec total I will repeat the test with all other ports now. Thanks again T. # du -hs TEST/ 285MTEST/ # date cp -r TEST/ /mnt umount /mnt date Tue Jan 19 21:20:01 CET 2010 Tue Jan 19 22:00:18 CET 2010 # usbdevs -v Controller /dev/usb0: addr 1: high speed, self powered, config 1, EHCI root hub(0x), Intel(0x8086), rev 1.00 port 1 powered port 2 powered port 3 powered port 4 powered port 5 powered port 6 powered Controller /dev/usb1: addr 1: high speed, self powered, config 1, EHCI root hub(0x), Intel(0x8086), rev 1.00 port 1 addr 2: high speed, power 80 mA, config 1, USB Mass Storage Device(0xba63), PRETEC 00, iSerialNumber 01AE000823 port 2 powered port 3 powered port 4 powered port 5 powered port 6 powered Controller /dev/usb2: addr 1: full speed, self powered, config 1, UHCI root hub(0x), Intel(0x8086), rev 1.00 port 1 powered port 2 powered Controller /dev/usb3: addr 1: full speed, self powered, config 1, UHCI root hub(0x), Intel(0x8086), rev 1.00 port 1 addr 2: full speed, power 100 mA, config 1, Fingerprint Sensor(0x2810), AuthenTec(0 port 2 powered Controller /dev/usb4: addr 1: full speed, self powered, config 1, UHCI root hub(0x), Intel(0x8086), rev 1.00 port 1 powered port 2 powered Controller /dev/usb5: addr 1: full speed, self powered, config 1, UHCI root hub(0x), Intel(0x8086), rev 1.00 port 1 powered port 2 powered 2010/1/19 Robert info...@die-optimisten.net: Ted Unangst wrote: Plug it in. The kernel tells you. usbdevs -v also helps regards, Robert -- Pau
Re: writing to usb very slow
On Tue, Jan 19, 2010 at 10:10:41PM +0100, T. Tofus von Blisstein wrote: this is an example. Attached is a 1GB (fat!) usb memory stick. It took 40 minutes to copy 285M. As I mentioned previously, the performance of USB memory sticks seems to vary greatly on OpenBSD depending on the stick's manufacturer. A stick that writes quickly on Windows might be terribly slow on OpenBSD. In that case, just buy a different memory stick. I have found the Kingston DataTraveler to write quite quickly on OpenBSD. http://www.kingston.com/flash/datatraveler.asp
Re: writing to usb very slow
On Tue, Jan 19, 2010 at 9:10 PM, T. Tofus von Blisstein tuffst...@googlemail.com wrote: this is an example. Attached is a 1GB (fat!) usb memory stick. It took 40 minutes to copy 285M. Mmmh, clearly your key. Just did a test on my OpenBSD laptop (Samsung Q35) with that key connected: umass0 at uhub0 port 1 configuration 1 interface 0 Corsair VoyagerGT rev 2.00/11.00 addr 2 umass0: using SCSI over Bulk-Only scsibus1 at umass0: 2 targets, initiator 0 sd0 at scsibus1 targ 1 lun 0: Corsair, VoyagerGT, 1100 SCSI0 0/direct removable Writing the same sized file took me 1m10s. This is using your test and with a 285M file. Steph
Re: writing to usb very slow
On Tue, 19 Jan 2010, T. Tofus von Blisstein wrote: Hello, this is an example. Attached is a 1GB (fat!) usb memory stick. It took 40 minutes to copy 285M. This one was Jan 19 21:18:04 hux /bsd: umass0: using SCSI over Bulk-Only Jan 19 21:18:04 hux /bsd: scsibus1 at umass0: 2 targets, initiator 0 Jan 19 21:18:04 hux /bsd: sd1 at scsibus1 targ 1 lun 0: -Pretec, 01GB, 2.00 SCSI2 0/direct removable Jan 19 21:18:04 hux /bsd: sd1: 983MB, 512 bytes/sec, 2015231 sec total I will repeat the test with all other ports now. There shouldn't be any difference among individual USB ports. First, try this: # dd if=/dev/rsd1c bs=64k count=1k of=/dev/null (or just hit ^C if it takes too long) and see what the reading speed of the _device_ is. If you don't have any data on the device and if you are willing to recreate the MBR and filesystem there, you can also test the writing speed: # dd if=/dev/zero bs=64k count=1k of=/dev/rsd1c Otherewise you are also measuring the filesystem performance and such. For some reason (which is unknown to me), foreign filesystems, such as ext2fs and msdos, are quite slow on OpenBSD, both for reading and for writing. The CPU is not the bottleneck in operations on these filesystems. Regards, David
Re: writing to usb very slow
which filesystem are you using on that device? T. 2010/1/19 FRLinux frli...@gmail.com: On Tue, Jan 19, 2010 at 9:10 PM, T. Tofus von Blisstein tuffst...@googlemail.com wrote: this is an example. Attached is a 1GB (fat!) usb memory stick. It took 40 minutes to copy 285M. Mmmh, clearly your key. Just did a test on my OpenBSD laptop (Samsung Q35) with that key connected: umass0 at uhub0 port 1 configuration 1 interface 0 Corsair VoyagerGT rev 2.00/11.00 addr 2 umass0: using SCSI over Bulk-Only scsibus1 at umass0: 2 targets, initiator 0 sd0 at scsibus1 targ 1 lun 0: Corsair, VoyagerGT, 1100 SCSI0 0/direct removable Writing the same sized file took me 1m10s. This is using your test and with a 285M file. Steph -- Pau
Re: writing to usb very slow
indeed... Tue Jan 19 22:11:38 CET 2010 Tue Jan 19 22:51:50 CET 2010 I will try your test now The point is that I have many huge USB drives which I would not like to throw away just like that I cannot afford looking for new usb drives which are fine on obsd. I would like to understand what is going on (at least within my limits of understanding!) Thanks T. 2010/1/19 David Vasek va...@fido.cz: On Tue, 19 Jan 2010, T. Tofus von Blisstein wrote: Hello, this is an example. Attached is a 1GB (fat!) usb memory stick. It took 40 minutes to copy 285M. This one was Jan 19 21:18:04 hux /bsd: umass0: using SCSI over Bulk-Only Jan 19 21:18:04 hux /bsd: scsibus1 at umass0: 2 targets, initiator 0 Jan 19 21:18:04 hux /bsd: sd1 at scsibus1 targ 1 lun 0: -Pretec, 01GB, 2.00 SCSI2 0/direct removable Jan 19 21:18:04 hux /bsd: sd1: 983MB, 512 bytes/sec, 2015231 sec total I will repeat the test with all other ports now. There shouldn't be any difference among individual USB ports. First, try this: # dd if=/dev/rsd1c bs=64k count=1k of=/dev/null (or just hit ^C if it takes too long) and see what the reading speed of the _device_ is. If you don't have any data on the device and if you are willing to recreate the MBR and filesystem there, you can also test the writing speed: # dd if=/dev/zero bs=64k count=1k of=/dev/rsd1c Otherewise you are also measuring the filesystem performance and such. For some reason (which is unknown to me), foreign filesystems, such as ext2fs and msdos, are quite slow on OpenBSD, both for reading and for writing. The CPU is not the bottleneck in operations on these filesystems. Regards, David -- Pau
Re: writing to usb very slow
On Tue, Jan 19, 2010 at 9:41 PM, T. Tofus von Blisstein tuffst...@googlemail.com wrote: which filesystem are you using on that device? Sorry realized I had forgotten to specify that in my reply. Key is a 8GB formatted with FAT. Steph
Re: writing to usb very slow
Hello David, thanks. # date dd if=/dev/rsd1c bs=64k count=1k of=/dev/null date Tue Jan 19 23:02:59 CET 2010 1024+0 records in 1024+0 records out 67108864 bytes transferred in 4.130 secs (16249108 bytes/sec) Tue Jan 19 23:03:03 CET 2010 # dd if=/dev/zero bs=64k count=1k of=/dev/rsd1c 1024+0 records in 1024+0 records out 67108864 bytes transferred in 15.053 secs (4458089 bytes/sec) Now I reformatted the drive to 4.4BSD disklabel -E /dev/rsd1c allocate all for partition a newfs /dev/rsd1c And run again my personal test... # tail -f /var/log/messages Jan 19 22:11:20 hux /bsd: sd1: 983MB, 512 bytes/sec, 2015231 sec total Jan 19 23:11:02 hux /bsd: sd1 detached Jan 19 23:11:02 hux /bsd: scsibus1 detached Jan 19 23:11:02 hux /bsd: umass0 detached Jan 19 23:11:05 hux /bsd: umass0 at uhub1 Jan 19 23:11:05 hux /bsd: port 2 configuration 1 interface 0 PRETEC Technology USB Mass Storage Device rev 2.00/1.00 addr 2 Jan 19 23:11:05 hux /bsd: umass0: using SCSI over Bulk-Only Jan 19 23:11:05 hux /bsd: scsibus1 at umass0: 2 targets, initiator 0 Jan 19 23:11:05 hux /bsd: sd1 at scsibus1 targ 1 lun 0: -Pretec, 01GB, 2.00 SCSI2 0/direct removable Jan 19 23:11:05 hux /bsd: sd1: 983MB, 512 bytes/sec, 2015231 sec total ^C # mount /dev/sd1a /mnt # date cp -r TEST/ /mnt umount /mnt date Tue Jan 19 23:11:27 CET 2010 Tue Jan 19 23:29:12 CET 2010 So it's reduced a lot, but still it is much slower than... sorry guys, the penguin. Still, from 50 minutes to 18 to copy 256M is a significant improvement I think the penguin takes some 2 minutes for that... but I can live with it Since I only do a _full_ backup once every 5 months or so and I have a script using rsync, I only have to transfer some 18M or so every time I do a normal backup Thanks, T. 2010/1/19 David Vasek va...@fido.cz: On Tue, 19 Jan 2010, T. Tofus von Blisstein wrote: Hello, this is an example. Attached is a 1GB (fat!) usb memory stick. It took 40 minutes to copy 285M. This one was Jan 19 21:18:04 hux /bsd: umass0: using SCSI over Bulk-Only Jan 19 21:18:04 hux /bsd: scsibus1 at umass0: 2 targets, initiator 0 Jan 19 21:18:04 hux /bsd: sd1 at scsibus1 targ 1 lun 0: -Pretec, 01GB, 2.00 SCSI2 0/direct removable Jan 19 21:18:04 hux /bsd: sd1: 983MB, 512 bytes/sec, 2015231 sec total I will repeat the test with all other ports now. There shouldn't be any difference among individual USB ports. First, try this: # dd if=/dev/rsd1c bs=64k count=1k of=/dev/null (or just hit ^C if it takes too long) and see what the reading speed of the _device_ is. If you don't have any data on the device and if you are willing to recreate the MBR and filesystem there, you can also test the writing speed: # dd if=/dev/zero bs=64k count=1k of=/dev/rsd1c Otherewise you are also measuring the filesystem performance and such. For some reason (which is unknown to me), foreign filesystems, such as ext2fs and msdos, are quite slow on OpenBSD, both for reading and for writing. The CPU is not the bottleneck in operations on these filesystems. Regards, David -- Pau
Re: writing to usb very slow
On Tue, Jan 19, 2010 at 10:54:16PM +0100, T. Tofus von Blisstein wrote: The point is that I have many huge USB drives which I would not like to throw away just like that this thread is really confusing. is the subject flash/memory sticks, or any USB storage? it seems to me the only thing that is slow is flash/memory sticks, or maybe I'm missing something? -- jake...@sdf.lonestar.org SDF Public Access UNIX System - http://sdf.lonestar.org
Re: writing to usb very slow
On Tue, 19 Jan 2010, T. Tofus von Blisstein wrote: Hello David, thanks. You're welcome. # mount /dev/sd1a /mnt # date cp -r TEST/ /mnt umount /mnt date Tue Jan 19 23:11:27 CET 2010 Tue Jan 19 23:29:12 CET 2010 So it's reduced a lot, but still it is much slower than... sorry guys, the penguin. Still, from 50 minutes to 18 to copy 256M is a significant improvement Try using softdep option for mount, possibly even the async option, if you like the risk, otherwise you are comparing two completely different modes of filesystem operation. Regards, David
Re: writing to usb very slow
sorry about that, but I also have noticed this problem with external usb drives I don't have any kind of test to report yet, though 2010/1/19 Jacob Meuser jake...@sdf.lonestar.org: On Tue, Jan 19, 2010 at 10:54:16PM +0100, T. Tofus von Blisstein wrote: The point is that I have many huge USB drives which I would not like to throw away just like that this thread is really confusing. is the subject flash/memory sticks, or any USB storage? it seems to me the only thing that is slow is flash/memory sticks, or maybe I'm missing something? -- jake...@sdf.lonestar.org SDF Public Access UNIX System - http://sdf.lonestar.org
Re: writing to usb very slow
# mount -o async /dev/sd1a /mnt # date cp -r TEST/ /mnt umount /mnt date Wed Jan 20 00:17:54 CET 2010 Wed Jan 20 00:33:35 CET 2010 2010/1/20 David Vasek va...@fido.cz: On Tue, 19 Jan 2010, T. Tofus von Blisstein wrote: Hello David, thanks. You're welcome. # mount /dev/sd1a /mnt # date cp -r TEST/ /mnt umount /mnt date Tue Jan 19 23:11:27 CET 2010 Tue Jan 19 23:29:12 CET 2010 So it's reduced a lot, but still it is much slower than... sorry guys, the penguin. Still, from 50 minutes to 18 to copy 256M is a significant improvement Try using softdep option for mount, possibly even the async option, if you like the risk, otherwise you are comparing two completely different modes of filesystem operation. Regards, David
Re: writing to usb very slow
From: Janne Johansson j...@it.su.se Subject: Re: writing to usb very slow Date: Mon, 18 Jan 2010 08:34:41 +0100 You need to include the time to sync (or umount) after writing and deleting the files, otherwise you arent benchmarking the USB speed but rather how fast your caches are, since the time for rm -rf * will be spent at the umount (which you didnt include in the timing) instead, when flushing the pending writes/deletes. Janne, thank you for your suggestion. I had a test again. The environment was the same as the previous test. But copied/deleted files were different (At this test, 4.6 kernel source tree and its tarball were used). Here's the result including the time of umount; WRITING TIME --- mountreal user system option write+umount write+umount write+umount -- --- --- none 9m05.93s+0m07.37s 0m0.38s+0m0.00s 0m5.61s+0m0.41s softdep 3m35.64s+0m35.71s 0m0.32s+0m0.00s 0m5.27s+0m0.49s async3m22.61s+0m05.90s 0m0.26s+0m0.00s 0m4.75s+0m0.37s DELETING TIME --- mountreal user system option delete+umount delete+umountdelete+umount -- --- --- none 4m21.23s+0m01.04s 0m0.03s+0m0.00s 0m1.71s+0m0.01s softdep 0m02.10s+0m27.49s 0m0.04s+0m0.00s 0m0.43s+0m0.75s async0m02.40s+0m01.54s 0m0.10s+0m0.00s 0m0.44s+0m0.00s However, mounting USB flash drive without any FFS option is very slow at all.
Re: writing to usb very slow
dmesg? results of testing with dd? What speed to do you get if you use FFS? I get around 9 MB/s to a slow USB hard disk with FFS, which would be around 2.5 hours for 72 GB, and I routinely backup many GBs to it. So I'm guessing either its ext2 or your hardware. On Sat, Jan 16, 2010 at 01:44:49AM +0100, T. Tofus von Blisstein wrote: Hello, I have noticed that writing to a usb drive is slow. What does slow mean? It means that compared to other OS's. Which OS's? Yes, it hurts: the penguin. Information: The external drive has been formated as ext2, so that I can mount it with obsd and the penguin Copying some 72G takes ~ 2hours on the penguin and some... 10 hours or more This is obsd -current snapshot of 12 Jan, i386 Actually it's even worse I start the copying of those 72G before I go to bed Then the system freezes ctrl+alt+f1 to get a console does nothing ctrl + alt + backspace does nothing Sounds like an APCI problem? It feels like USB 1 or worse _and_ the system freezes You can of course blame the device, but I have had the same speed problem on a number of devices You can now stone me, call me stupid, tell me that I have stated that linux gives free blowjobs, that I am a dumbass etc etc etc... it's fine, go for it but after that, please try to copy to an extermal drive some 3G with linux and obsd and tell me the difference. I am all against linux, I love obsd, I have obsd as my main production system, I buy the CDs at each release ... and I know this is misc and not exactly the most friendly place of the world I am simply asking: Has anybody else noticed this or am I the only one on the face of the world with this problem? Any (positive, polite) comment will be very much appreciated. T.
Re: writing to usb very slow
T. Tofus von Blisstein tuffst...@googlemail.com writes: I have noticed that writing to a usb drive is slow. One likely culprit is your system's BIOS, that likely reports only USB 1.something capabilities, confident that no operating system would ever believe what it says on the matter anyway. Anecdotal evidence suggests that OpenBSD is one rare bird that actually cares, and respects the device's reported capabilites. Imagine what would happen to your data if we wrote it too fast for the receiving device to actually handle. Take a peek at your dmesg for anything usb related. My dear old Thinkpad R60 apparently had five USB devices available, with only one reported as 2.0 capable (never found out which one that was or if it was indeed accessible): ~$ grep usb dmesg.thingy.mp usb0 at uhci0: USB revision 1.0 uhub0 at usb0 usb1 at uhci1: USB revision 1.0 uhub1 at usb1 usb2 at uhci2: USB revision 1.0 uhub2 at usb2 usb3 at uhci3: USB revision 1.0 uhub3 at usb3 usb4 at ehci0: USB revision 2.0 uhub4 at usb4 while my slightly newer ThinkPad SL500 apparently has a few more (still only one reported as 2.0 capable, though): $ dmesg | grep usb usb1 at ehci1: USB revision 2.0 uhub1 at usb1 Intel EHCI root hub rev 2.00/1.00 addr 1 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 Intel UHCI root hub rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 Intel UHCI root hub rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 Intel UHCI root hub rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 Intel UHCI root hub rev 1.00/1.00 addr 1 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 Intel EHCI root hub rev 2.00/1.00 addr 1 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 Intel UHCI root hub rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 Intel UHCI root hub rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 Intel UHCI root hub rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 Intel UHCI root hub rev 1.00/1.00 addr 1 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 Intel EHCI root hub rev 2.00/1.00 addr 1 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 Intel UHCI root hub rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 Intel UHCI root hub rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 Intel UHCI root hub rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 Intel UHCI root hub rev 1.00/1.00 addr 1 but performance with such things as a USB 3G modem (Huawei somethingorother) and a thumbdrive appears to be better on the newer system. Likely by sheer luck I've happened on the parts that operate on full 2.0 speed. I'm sure people with more wisdom on the subject can fill in on this via the list. - Peter -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: writing to usb very slow
On Sat, 16 Jan 2010, T. Tofus von Blisstein wrote: I am simply asking: Has anybody else noticed this or am I the only one on the face of the world with this problem? In my limited experience, USB HD are usually about alright but USB sticks are super slow (i.e. EHCI speed is slower than usb1.1). -- Antoine
Re: writing to usb very slow
On Sun, Jan 17, 2010 at 06:41:38PM +0100, Antoine Jacoutot wrote: In my limited experience, USB HD are usually about alright but USB sticks are super slow (i.e. EHCI speed is slower than usb1.1). My experience is that only certain USB sticks are slow on OpenBSD. Experiment with a few different brands and models until you find one that writes at a reasonable speed.
Re: writing to usb very slow
Thanks to all for your answers. Peter, thanks for your point. Look at this: hux(p3)| dmesg| grep usb usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 Intel EHCI root hub rev 2.00/1.00 addr 1 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 Intel UHCI root hub rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 Intel UHCI root hub rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 Intel UHCI root hub rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 Intel UHCI root hub rev 1.00/1.00 addr 1 I will have a look now to the BIOS and try to upgrade it. I will report later. Thanks again T. 2010/1/17 Peter N. M. Hansteen pe...@bsdly.net: T. Tofus von Blisstein tuffst...@googlemail.com writes: I have noticed that writing to a usb drive is slow. One likely culprit is your system's BIOS, that likely reports only USB 1.something capabilities, confident that no operating system would ever believe what it says on the matter anyway. Anecdotal evidence suggests that OpenBSD is one rare bird that actually cares, and respects the device's reported capabilites. Imagine what would happen to your data if we wrote it too fast for the receiving device to actually handle. Take a peek at your dmesg for anything usb related. My dear old Thinkpad R60 apparently had five USB devices available, with only one reported as 2.0 capable (never found out which one that was or if it was indeed accessible): ~$ grep usb dmesg.thingy.mp usb0 at uhci0: USB revision 1.0 uhub0 at usb0 usb1 at uhci1: USB revision 1.0 uhub1 at usb1 usb2 at uhci2: USB revision 1.0 uhub2 at usb2 usb3 at uhci3: USB revision 1.0 uhub3 at usb3 usb4 at ehci0: USB revision 2.0 uhub4 at usb4 while my slightly newer ThinkPad SL500 apparently has a few more (still only one reported as 2.0 capable, though): $ dmesg | grep usb usb1 at ehci1: USB revision 2.0 uhub1 at usb1 Intel EHCI root hub rev 2.00/1.00 addr 1 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 Intel UHCI root hub rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 Intel UHCI root hub rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 Intel UHCI root hub rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 Intel UHCI root hub rev 1.00/1.00 addr 1 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 Intel EHCI root hub rev 2.00/1.00 addr 1 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 Intel UHCI root hub rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 Intel UHCI root hub rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 Intel UHCI root hub rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 Intel UHCI root hub rev 1.00/1.00 addr 1 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 Intel EHCI root hub rev 2.00/1.00 addr 1 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 Intel UHCI root hub rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 Intel UHCI root hub rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 Intel UHCI root hub rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 Intel UHCI root hub rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 Intel UHCI root hub rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 Intel UHCI root hub rev 1.00/1.00 addr 1 but performance with such things as a USB 3G modem (Huawei somethingorother) and a thumbdrive appears to be better on the newer system. Likely by sheer luck I've happened on the parts that operate on full 2.0 speed. I'm sure people with more wisdom on the subject can fill in on this via the list. - Peter -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: writing to usb very slow
I have measured write/remove time with the USB flash drive; uhub4 at uhub0 port 5 NEC product 0x005a rev 2.00/1.00 addr 2 umass0 at uhub4 port 1 configuration 1 interface 0 TOSHIBA TransMemory rev 2.00/1.10 addr 3 umass0: using SCSI over Bulk-Only scsibus2 at umass0: 2 targets, initiator 0 sd0 at scsibus2 targ 1 lun 0: TOSHIBA, TransMemory, PMAP SCSI0 0/direct removable sd0: 961MB, 512 bytes/sec, 1968128 sec total First, I copied file tree (about 100MB) from mfs to such USB flash drive, then umount. Next, mount it again, then remove all files and directories. (test script at the end of this mail) Here's the data; write files remove files --- --- - mount option real usersystemreal usersystem --- - --- - --- --- none 24m18.37s 0m1.15s 0m16.58s 14m54.99s 0m0.26s 0m5.70s noatime 23m56.60s 0m1.03s 0m16.95s 14m32.22s 0m0.42s 0m5.59s async07m28.74s 0m0.79s 0m14.29s 00m06.91s 0m0.24s 0m1.90s async,noatime07m29.09s 0m0.87s 0m14.97s 00m07.02s 0m0.19s 0m2.06s softdep 09m55.40s 0m0.73s 0m15.59s 04m23.48s 0m0.24s 0m2.17s softdep,noatime 09m54.15s 0m0.90s 0m15.05s 04m27.81s 0m0.21s 0m1.99s result: write time: none softdep async remove time: none softdep async Both softdep and async options seem to affect write/remove times. And especially, async option very affects remove time. [test script]--- #!/bin/sh for opt in none noatime async noatime,async softdep softdep,async do if [ X$opt = Xnone ] then opt= else opt=-o $opt fi echo opt: $opt if mount $opt /dev/sd0d /mnt then cd /mfs/ time pax -rwpe . /mnt/tmp/. umount /mnt sleep 60 fi if mount $opt /dev/sd0d /mnt then cd /mnt/tmp time rm -rf * cd umount /mnt sleep 60 fi done
Re: writing to usb very slow
KAWAMATA Yoshihiro wrote: I have measured write/remove time with the USB flash drive; sd0 at scsibus2 targ 1 lun 0: TOSHIBA, TransMemory, PMAP SCSI0 0/direct removable sd0: 961MB, 512 bytes/sec, 1968128 sec total First, I copied file tree (about 100MB) from mfs to such USB flash drive, then umount. Next, mount it again, then remove all files and directories. (test script at the end of this mail) You need to include the time to sync (or umount) after writing and deleting the files, otherwise you arent benchmarking the USB speed but rather how fast your caches are, since the time for rm -rf * will be spent at the umount (which you didnt include in the timing) instead, when flushing the pending writes/deletes. Just like windows will write REALLY fast to your USB-stick, then you have to wait for the eject removable device instead if you try to take it out quickly after a large copy.
