Re: USB I/O draining my userspace (Ubuntu Natty 64b)
On Wed, Jun 15, 2011 at 06:42:48PM +0300, Ira Abramov wrote: Quoting Yedidyah Bar-David, from the post of Wed, 15 Jun: Did you try cp (or cp --sparse=always if you really want to) and see if it helps? I don't know of a similar option for dd. I have here an image with partitions in it, which is why I use DD. cp can't help me, sadly. Why not cp? cp image /dev/sdX should work with no problem. Indeed you won't have control on the block size, but I can't think of other important options of dd that you'll miss. -- Didi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: USB I/O draining my userspace (Ubuntu Natty 64b)
On Wed, 2011-06-15 at 18:42 +0300, Ira Abramov wrote: Quoting Yedidyah Bar-David, from the post of Wed, 15 Jun: Perhaps it uses USB1 and not 2? nope, I had that problem when I accidentally switched ports to a USB1 port, the 22 minute burn took over 113 minutes before I noticed it was still writing and killed it. also, to answer Geoff - nothing else is on the USB, the M/K are on PS/2 connectors. Maybe the same controller chip, but not the same bus or kernel module. at least with oflag=dsync it doesn't get the entire userspace stuck, whatever that does. a few side-notes: 1. what oflag=dsync does (oflag=direct will do it too) is bypass linux's buffer cache. this means that it does not polute the buffer cache with one-time garbage. dd without an oflag writes to the cache, and pages stored there for other apps get dumped or flushed to disk - causing the effects you see. using 'direct' is better for your flash (although it could be that on a USB drive, dsync and direct map to the same underlying operations). 2. regarding the block size - the optimal size will be the erase-unit of your flash. check out the specifications of the unit you have (you care about the flash controller it the unit - not about the flash memory itself) - and supply this as the block size to DD. erase-units are usually several 10s or 100s of KB. writes not done using these units, will cause several data-copy operations per operating system write I/Os, to be performed by the flash controller. note: since you already made writes using the old notion - which probably caused fragmentation of the flash memory - it might be a good idea to find whether you can reset your USB drive before doing the next copy. --guy bs=8M is the next parameter I'll try, or maybe I should go for 32M? Can you rmmod all *hci_ucd modules except for ehci_hcd and see what happens? nope, I assumed Ubuntu would have taken precautions not to insmod anything useless for my hardware... Did you try cp (or cp --sparse=always if you really want to) and see if it helps? I don't know of a similar option for dd. I have here an image with partitions in it, which is why I use DD. cp can't help me, sadly. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
USB I/O draining my userspace (Ubuntu Natty 64b)
First, an appology. I added another message on the RMS fiasco only to find out later the threads all died out and I was committing a faux pas. my appologies! please ignore I posted it if at all possible. Second, here's my problem: I have here a workstation running an Athlon 3700+, and part of my job is to occasionally write out an image file to USB universal card reader, testing the product of my builds. The writing takes forever (since I haven't discovered how to get dd to write out the sparse image to the CF card sparsely). Also, untill I moved the card reader to one of the backpanel ports, the write would drag my entire environment to a halt at the same time - even the mouse pointer gets stuck at some point, until dd would finish. switching from front to back panel and adding the oflag=dsync option solved the freַ¯ing of the userspace but not the horrible writing speeds. I have a feeling this is a major bug with the USBstorage driver or some related module, but as this is old hardware running on the latest kernel from Ubuntu, I am surprised. Anyone got a clue? Thanks, Ira. -- Fastest gun in the west Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: USB I/O draining my userspace (Ubuntu Natty 64b)
On Wed, 15 Jun 2011 16:19:08 +0300 Ira Abramov lists-linux...@ira.abramov.org wrote: First, an appology. I added another message on the RMS fiasco only to find out later the threads all died out and I was committing a faux pas. my appologies! please ignore I posted it if at all possible. It's too late! You've done it now ;-) Second, here's my problem: I have here a workstation running an Athlon 3700+, and part of my job is to occasionally write out an image file to USB universal card reader, testing the product of my builds. The writing takes forever (since I haven't discovered how to get dd to write out the sparse image to the CF card sparsely). Also, untill I moved the card reader to one of the backpanel ports, the write would drag my entire environment to a halt at the same time - even the mouse pointer gets stuck at some point, until dd would finish. switching from front to back panel and adding the oflag=dsync option solved the freַ¯ing of the userspace but not the horrible writing speeds. I do hourly rsyncs between two drives and it often causes my system to bog heavily. I won't speculate on the reason because that topic is apparently as inflammatory as the Stallman's boycott (!) but it does happen. It also happens on my system with dd between disks. System in question is E8400 with Seagate 16M Barracudas with Slackware and 2.6.29.6 kernel. I have seen this behavior in every desktop OS I have tried (NetBSD, FreeBSD, OpenBSD, Slackware, the W word) except for Slamd64 (no longer maintained). I have a feeling this is a major bug with the USBstorage driver or some related module, but as this is old hardware running on the latest kernel from Ubuntu, I am surprised. Anyone got a clue? Thanks, Ira. Are you using the default blocksize for dd? It may not be optimal for your USB device. I wonder if it would be possible to find out what blocksize the device uses and if it would make a difference. I know bs does make a big difference when copying big files to a physical disk. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: USB I/O draining my userspace (Ubuntu Natty 64b)
On Jun 15, 2011, at 4:19 PM, Ira Abramov wrote: Second, here's my problem: I have here a workstation running an Athlon 3700+, and part of my job is to occasionally write out an image file to USB universal card reader, testing the product of my builds. The writing takes forever (since I haven't discovered how to get dd to write out the sparse image to the CF card sparsely). Also, untill I moved the card reader to one of the backpanel ports, the write would drag my entire environment to a halt at the same time - even the mouse pointer gets stuck at some point, until dd would finish. switching from front to back panel and adding the oflag=dsync option solved the freַ¯ing of the userspace but not the horrible writing speeds. I have a feeling this is a major bug with the USBstorage driver or some related module, but as this is old hardware running on the latest kernel from Ubuntu, I am surprised. Anyone got a clue? No. It has to do with how USB is implemented in hardware. Almost every motherboard I have seen only has 2 USB ports. Motherboards with 6 USB ports on the back, and connectors for more on the front of the case still have only 2 USB ports, but they have a hub built into the motherboard. For example, on one system I have: lspci yields: 00:10.0 USB Controller: VIA Technologies, Inc. VT82x UHCI USB 1.1 Controller (rev 62) 00:10.1 USB Controller: VIA Technologies, Inc. VT82x UHCI USB 1.1 Controller (rev 62) 00:10.2 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 65) The third USB controller is a PCI card. Now if you do a lsusb: Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub But I don't have any hubs. They are on the motherboard and the PCI card. So if you plug the keyboard and mouse into the same port as the memory stick, you are doing two things. The first is you are slowing the memory stick down to the speed of the keyboard and mouse, usually 12mbits per second. The second is that you are blocking IO to the keyboard and mouse while the disk is busy. Geoff. -- Geoffrey S. Mendelson, N3OWJ/4X1GM Making your enemy reliant on software you support is the best revenge. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: USB I/O draining my userspace (Ubuntu Natty 64b)
On Wed, Jun 15, 2011 at 04:19:08PM +0300, Ira Abramov wrote: First, an appology. I added another message on the RMS fiasco only to find out later the threads all died out and I was committing a faux pas. my appologies! please ignore I posted it if at all possible. Second, here's my problem: I have here a workstation running an Athlon 3700+, and part of my job is to occasionally write out an image file to USB universal card reader, testing the product of my builds. The writing takes forever (since I haven't discovered how to get dd to write out the sparse image to the CF card sparsely). Also, untill I moved the card reader to one of the backpanel ports, the write would drag my entire environment to a halt at the same time - even the mouse pointer gets stuck at some point, until dd would finish. switching from front to back panel and adding the oflag=dsync option solved the freַ¯ing of the userspace but not the horrible writing speeds. Perhaps it uses USB1 and not 2? Can you rmmod all *hci_ucd modules except for ehci_hcd and see what happens? Did you try cp (or cp --sparse=always if you really want to) and see if it helps? I don't know of a similar option for dd. -- Didi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: USB I/O draining my userspace (Ubuntu Natty 64b)
Quoting Yedidyah Bar-David, from the post of Wed, 15 Jun: Perhaps it uses USB1 and not 2? nope, I had that problem when I accidentally switched ports to a USB1 port, the 22 minute burn took over 113 minutes before I noticed it was still writing and killed it. also, to answer Geoff - nothing else is on the USB, the M/K are on PS/2 connectors. Maybe the same controller chip, but not the same bus or kernel module. at least with oflag=dsync it doesn't get the entire userspace stuck, whatever that does. bs=8M is the next parameter I'll try, or maybe I should go for 32M? Can you rmmod all *hci_ucd modules except for ehci_hcd and see what happens? nope, I assumed Ubuntu would have taken precautions not to insmod anything useless for my hardware... Did you try cp (or cp --sparse=always if you really want to) and see if it helps? I don't know of a similar option for dd. I have here an image with partitions in it, which is why I use DD. cp can't help me, sadly. -- A fetus who made it Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: USB I/O draining my userspace (Ubuntu Natty 64b)
i suggest you trace it. use blktrace. On Wed, Jun 15, 2011 at 5:42 PM, Ira Abramov lists-linux...@ira.abramov.org wrote: Quoting Yedidyah Bar-David, from the post of Wed, 15 Jun: Perhaps it uses USB1 and not 2? nope, I had that problem when I accidentally switched ports to a USB1 port, the 22 minute burn took over 113 minutes before I noticed it was still writing and killed it. also, to answer Geoff - nothing else is on the USB, the M/K are on PS/2 connectors. Maybe the same controller chip, but not the same bus or kernel module. at least with oflag=dsync it doesn't get the entire userspace stuck, whatever that does. bs=8M is the next parameter I'll try, or maybe I should go for 32M? Can you rmmod all *hci_ucd modules except for ehci_hcd and see what happens? nope, I assumed Ubuntu would have taken precautions not to insmod anything useless for my hardware... Did you try cp (or cp --sparse=always if you really want to) and see if it helps? I don't know of a similar option for dd. I have here an image with partitions in it, which is why I use DD. cp can't help me, sadly. -- A fetus who made it Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: USB I/O draining my userspace (Ubuntu Natty 64b)
On Wed, 15 Jun 2011, geoffrey mendelson wrote: No. It has to do with how USB is implemented in hardware. Almost every motherboard I have seen only has 2 USB ports. Motherboards with 6 USB ports on the back, and connectors for more on the front of the case still have only 2 USB ports, but they have a hub built into the motherboard. For example, on one system I have: lspci yields: 00:10.0 USB Controller: VIA Technologies, Inc. VT82x UHCI USB 1.1 Controller (rev 62) 00:10.1 USB Controller: VIA Technologies, Inc. VT82x UHCI USB 1.1 Controller (rev 62) 00:10.2 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 65) The third USB controller is a PCI card. Now if you do a lsusb: Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub But I don't have any hubs. They are on the motherboard and the PCI card. So if you plug the keyboard and mouse into the same port as the memory stick, you are doing two things. This is incorrect. Those hubs you see are virtual root hubs which are merely a way for the linux kernel presents things. Note that the vendor is Linux Foundation. Actually, all your USB ports are connected directly to the host controller, which has (on a typical PC) 6 or more ports. The first is you are slowing the memory stick down to the speed of the keyboard and mouse, usually 12mbits per second. A USB 2.0 high speed hub may be connected to both low/full speed devices and high speed devices at the same time, and the high speed devices will be used at high speed. The second is that you are blocking IO to the keyboard and mouse while the disk is busy. That is not how USB works. The data to/from disk is done in bulk transfer which has lowest priority so it does not block HID data from keyboard or mouse that uses interrupt transfer. On the other hand the small amount of data that the mouse and keyboard generate is unlikely to noticeably reduce the bandwidth available to the disk. -- Matan. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
