Re: Safely remove USB drive
A great bit of work. Do you know if USB DAT drives are working on Sparc64? They were not when I last tested (probably OBSD 6.2). Andrew On Mon, 13 Feb 2023 at 13:02, Crystal Kolipe wrote: > On Thu, Feb 09, 2023 at 09:20:48PM -0300, Crystal Kolipe wrote: > > There is no reason why we couldn't implement support for 'eject -t' for > sd > > devices, (which is internally treated as a tape retension command). > > > > In fact, I just made a patch to do that out of curiosity and was able to > > access the usb device again after 'reloading' it. > > Over the weekend I received a few emails off-list asking me about this, so > I've written up and posted a brief explainer here: > > https://research.exoticsilicon.com/articles/eject_command_hacking > >
Re: Safely remove USB drive
On Thu, Feb 09, 2023 at 09:20:48PM -0300, Crystal Kolipe wrote: > There is no reason why we couldn't implement support for 'eject -t' for sd > devices, (which is internally treated as a tape retension command). > > In fact, I just made a patch to do that out of curiosity and was able to > access the usb device again after 'reloading' it. Over the weekend I received a few emails off-list asking me about this, so I've written up and posted a brief explainer here: https://research.exoticsilicon.com/articles/eject_command_hacking
Re: Safely remove USB drive
I want to thank you from here for the tour of the usb utils, appreciated.. -- Daniele Bonini Crystal Kolipe wrote: > On Thu, Feb 09, 2023 at 11:02:14AM -0500, Bryan Steele wrote: > > On Wed, Feb 08, 2023 at 10:34:07AM -0300, vitmau...@gmail.com wrote: > > > Hi, > > > > > > I'm not using my drives for anything more than copying files, dd > > > etc. I just got curious because you mentioned the act of > > > detaching a device after umounting it and I don't know how to do > > > that on OpenBSD. On Fedora I would issue "udisks --detach > > > /dev/sdX" (older versions) or "udisksctl poweroff -b /dev/sdX" > > > (newer versions). > > > > > > Best, > > > Vitor > > > > Umounting should be good enough, but you can also use eject(1), > > which should have the desired effect, e.g: > > Eject is slightly different. > > When you use eject against an sd device, it's basically setting a > flag that says, 'once any outstanding I/O is finished on this device, > then mark it offline', (I.E. no media present). > > So if you have running, for example: > > # dd if=/dev/rsd1c of=/dev/null bs=1m count=256 & > > and then do an eject from a different terminal, it should let the dd > process complete, and then off-line the device.
Re: Safely remove USB drive
On Thu, Feb 09, 2023 at 11:02:14AM -0500, Bryan Steele wrote: > On Wed, Feb 08, 2023 at 10:34:07AM -0300, vitmau...@gmail.com wrote: > > Hi, > > > > I'm not using my drives for anything more than copying files, dd etc. > > I just got curious because you mentioned the act of detaching a device > > after umounting it and I don't know how to do that on OpenBSD. On > > Fedora I would issue "udisks --detach /dev/sdX" (older versions) or > > "udisksctl poweroff -b /dev/sdX" (newer versions). > > > > Best, > > Vitor > > Umounting should be good enough, but you can also use eject(1), which > should have the desired effect, e.g: Eject is slightly different. When you use eject against an sd device, it's basically setting a flag that says, 'once any outstanding I/O is finished on this device, then mark it offline', (I.E. no media present). So if you have running, for example: # dd if=/dev/rsd1c of=/dev/null bs=1m count=256 & and then do an eject from a different terminal, it should let the dd process complete, and then off-line the device. > You'll need to physically reconnect the drive if you want to use it again. Yes, although it's still active at the usb level, as far as the sd driver is concerned, the device has no media present. There is no reason why we couldn't implement support for 'eject -t' for sd devices, (which is internally treated as a tape retension command). In fact, I just made a patch to do that out of curiosity and was able to access the usb device again after 'reloading' it. Probably pointless but a fun (*) programming exercise. (*) Depends on your definition of fun, obviously.
Re: Safely remove USB drive
On Wed, Feb 08, 2023 at 10:34:07AM -0300, vitmau...@gmail.com wrote: > Hi, > > I'm not using my drives for anything more than copying files, dd etc. > I just got curious because you mentioned the act of detaching a device > after umounting it and I don't know how to do that on OpenBSD. On > Fedora I would issue "udisks --detach /dev/sdX" (older versions) or > "udisksctl poweroff -b /dev/sdX" (newer versions). > > Best, > Vitor Umounting should be good enough, but you can also use eject(1), which should have the desired effect, e.g: # disklabel sd1 # /dev/rsd1c: type: SCSI disk: SCSI disk label: Flash Disk ... # eject /dev/rsd1c # disklabel sd1 disklabel: DIOCGDINFO: Input/output error # You'll need to physically reconnect the drive if you want to use it again. -Bryan.
Re: Safely remove USB drive
On Wed, Feb 08, 2023 at 07:15:27PM +0100, Paul de Weerd wrote: > On Wed, Feb 08, 2023 at 04:50:32PM +0100, Jan Stary wrote: > | On Feb 08 13:56:18, pe...@bsdly.net wrote: > | > 1) close any open files stored there > | > 2) make sure no process has the media as $PWD (as in, cd away from there, > | >and really a variation on the first) > | > 3) issue at least one sync command (some folklore will insist on three) > | > 4) umount the media from wherever it was mounted > | > | 4 takes care of 1,2,3, right? > > Not if the kernel has a file open on it (e.g. a swap file [1]) or if > there's another mount somewhere in the filesystem you're trying to > unmount (a special case of the first condition, I guess). > > [weerd@kale] $ doas mount /dev/sd3a /mnt > [weerd@kale] $ doas mkdir /mnt/deeper > [weerd@kale] $ doas mount /dev/sd3d /mnt/deeper > [weerd@kale] $ doas dd if=/dev/zero of=/mnt/swap bs=1M count=4096 > [weerd@kale] $ doas swapon /mnt/swap > > The kernel is not going to let you `umount /mnt`, and forcibly > removing the device is very likely going to lead to amazing new > learning opportunities. > > So, yeah, as people have stated .. the way to safely remove a USB > drive from your system depends on what you were using the USB drive > for. For the generic case, where we have no clue what OP was doing: > shut down your machine first and remove the device then. > Does shut down save you from the case above? Shut down is the same as umount -f ? Nola -- Nils Ola Nilsson, nils...@abc.se
Re: Safely remove USB drive
Just to keep the discussion and clarificarions in the mailing list. I'm sorry for all the typos of my my previous post. Paul de Weerd wrote: > and forcibly removing the device is very likely going to lead to > amazing new learning opportunities. I can confirm these statement and I invite you to double check all the "learning opportunities": knowing the situation an attacker can easily point to deattach all your devices, even the huband that's te magic.. > [1]: Putting swap on removable media is a tremendously bad idea - > don't do it, unless you're keen on those learning opportunities. I can confirm also this one for what I saw a tremendous improvement of performances looking at mfs, given that I create an msf more big than the swap partition, forcing msf to move on ram with, I believe with the save pros of msf. However, let me confirm I'm getting the thing right. -- Daniele Bonini
Re: Safely remove USB drive
Paul de Weerd wrote: > and forcibly removing the device is very likely going to lead to > amazing new learning opportunities. I can confirm these statement and I invite you to double check all the "learning opportunities": taken that, by "chance", they are not great happenings brr..than the magic become to attack you there > [1]: Putting swap on removable media is a tremendously bad idea - > don't do it, unless you're keen on those learning opportunities. I can confirm also this one for what I saw a tremendous improvement of performances looking at mfs, if I have taken the goon one.. -- Daniele Bonini
Re: Safely remove USB drive
On Wed, Feb 08, 2023 at 04:50:32PM +0100, Jan Stary wrote: | On Feb 08 13:56:18, pe...@bsdly.net wrote: | > 1) close any open files stored there | > 2) make sure no process has the media as $PWD (as in, cd away from there, | >and really a variation on the first) | > 3) issue at least one sync command (some folklore will insist on three) | > 4) umount the media from wherever it was mounted | | 4 takes care of 1,2,3, right? Not if the kernel has a file open on it (e.g. a swap file [1]) or if there's another mount somewhere in the filesystem you're trying to unmount (a special case of the first condition, I guess). [weerd@kale] $ doas mount /dev/sd3a /mnt [weerd@kale] $ doas mkdir /mnt/deeper [weerd@kale] $ doas mount /dev/sd3d /mnt/deeper [weerd@kale] $ doas dd if=/dev/zero of=/mnt/swap bs=1M count=4096 [weerd@kale] $ doas swapon /mnt/swap The kernel is not going to let you `umount /mnt`, and forcibly removing the device is very likely going to lead to amazing new learning opportunities. So, yeah, as people have stated .. the way to safely remove a USB drive from your system depends on what you were using the USB drive for. For the generic case, where we have no clue what OP was doing: shut down your machine first and remove the device then. Paul [1]: Putting swap on removable media is a tremendously bad idea - don't do it, unless you're keen on those learning opportunities. -- >[<++>-]<+++.>+++[<-->-]<.>+++[<+ +++>-]<.>++[<>-]<+.--.[-] http://www.weirdnet.nl/
Re: Safely remove USB drive
On Wed, Feb 08, 2023 at 05:59:12PM +0100, Peter N. M. Hansteen wrote:
> On Wed, Feb 08, 2023 at 04:50:32PM +0100, Jan Stary wrote:
> > On Feb 08 13:56:18, pe...@bsdly.net wrote:
> > > 1) close any open files stored there
> > > 2) make sure no process has the media as $PWD (as in, cd away from there,
> > >and really a variation on the first)
> > > 3) issue at least one sync command (some folklore will insist on three)
> > > 4) umount the media from wherever it was mounted
> >
> > 4 takes care of 1,2,3, right?
>
> It is a common assumption it does,
It certainly does the sync, from /src/sbin/umount/umount.c:
int
main(int argc, char *argv[])
{
int all, ch, errs;
/* Start disks transferring immediately. */
sync();
> but I have seen time and again applications
> either coredumping and hanging while doing so or just getting terribly
> confused
> when their presumed current directory disappeared from under them.
>
> Depending on how much force you put behind the umount (as in doas, sudo) it
> is not entirely certain you would be able to umount a file system that has
> open files.
>
> Then again, your mileage may vary. And the OP asked for safe removal.
I interpreted 'safe removal' as meaning that the data on the removable storage
is
kept intact.
If a userland program crashes or mis-behaves when a device is removed, then that
is really an issue specific to that program, and a somewhat different
discussion.
If unmounting a device, (and detaching any associated softraid device if there
is
one), is not enough to ensure that it's contents matches what the system
believed
it had written to it, then there is surely a kernel bug somewhere, (or a
hardware
issue).
Re: Safely remove USB drive
On Wed, Feb 08, 2023 at 04:50:32PM +0100, Jan Stary wrote: > On Feb 08 13:56:18, pe...@bsdly.net wrote: > > 1) close any open files stored there > > 2) make sure no process has the media as $PWD (as in, cd away from there, > >and really a variation on the first) > > 3) issue at least one sync command (some folklore will insist on three) > > 4) umount the media from wherever it was mounted > > 4 takes care of 1,2,3, right? It is a common assumption it does, but I have seen time and again applications either coredumping and hanging while doing so or just getting terribly confused when their presumed current directory disappeared from under them. Depending on how much force you put behind the umount (as in doas, sudo) it is not entirely certain you would be able to umount a file system that has open files. Then again, your mileage may vary. And the OP asked for safe removal. - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team https://bsdly.blogspot.com/ https://www.bsdly.net/ https://www.nuug.no/ "Remember to set the evil bit on all malicious network traffic" delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: Safely remove USB drive
On Feb 08 13:56:18, pe...@bsdly.net wrote: > 1) close any open files stored there > 2) make sure no process has the media as $PWD (as in, cd away from there, >and really a variation on the first) > 3) issue at least one sync command (some folklore will insist on three) > 4) umount the media from wherever it was mounted 4 takes care of 1,2,3, right?
Re: Safely remove USB drive
On Wed, Feb 08, 2023 at 10:44:20AM -0300, Crystal Kolipe wrote: > If you are just copying files, and/or using dd to, for example, overwrite > a device with zeros or random data, then you don't need to do anything > special to use usb storage devices on OpenBSD. In the case of dd-ing to a usb stick I'd say only to wait until you get the shell prompt back before you unplug it. Then you'll be fine. - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team https://bsdly.blogspot.com/ https://www.bsdly.net/ https://www.nuug.no/ "Remember to set the evil bit on all malicious network traffic" delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: Safely remove USB drive
Hi, thank you Crystal for the explanation. Best, Vito Em qua., 8 de fev. de 2023 às 10:44, Crystal Kolipe escreveu: > > On Wed, Feb 08, 2023 at 10:34:07AM -0300, vitmau...@gmail.com wrote: > > I'm not using my drives for anything more than copying files, dd etc. > > I just got curious because you mentioned the act of detaching a device > > after umounting it and I don't know how to do that on OpenBSD. > > If and only if you are using it with softraid, (usually as an encrypted > volume), then you would need to detach the softraid device using bioctl -d. > > But you are almost certainly not doing that. > > If you are just copying files, and/or using dd to, for example, overwrite > a device with zeros or random data, then you don't need to do anything > special to use usb storage devices on OpenBSD.
Re: Safely remove USB drive
On Wed, Feb 08, 2023 at 10:34:07AM -0300, vitmau...@gmail.com wrote: > I'm not using my drives for anything more than copying files, dd etc. > I just got curious because you mentioned the act of detaching a device > after umounting it and I don't know how to do that on OpenBSD. If and only if you are using it with softraid, (usually as an encrypted volume), then you would need to detach the softraid device using bioctl -d. But you are almost certainly not doing that. If you are just copying files, and/or using dd to, for example, overwrite a device with zeros or random data, then you don't need to do anything special to use usb storage devices on OpenBSD.
Re: Safely remove USB drive
Hi, I'm not using my drives for anything more than copying files, dd etc. I just got curious because you mentioned the act of detaching a device after umounting it and I don't know how to do that on OpenBSD. On Fedora I would issue "udisks --detach /dev/sdX" (older versions) or "udisksctl poweroff -b /dev/sdX" (newer versions). Best, Vitor Em qua., 8 de fev. de 2023 às 10:24, Crystal Kolipe escreveu: > > Hi, > > On Wed, Feb 08, 2023 at 10:13:59AM -0300, vitmau...@gmail.com wrote: > > thank you all for the replies. Crystal, what command would I use to > > detach a USB drive? I tried eject, but it doesn't seem to work with > > the kind of devices I have. > > Can you explain exactly what you are trying to do, and what devices > you are using? > > If you are just writing files to a usb flash drive from the command line, > then you mount it, copy the files, and umount, nothing more. > > (Actually, umount automatically does a sync as soon as you invoke it.) > > If you are doing something more complicated or using a usb device > other than a flash drive or external HD, then let us know so that we > can give you accurate advice.
Re: Safely remove USB drive
Hi, On Wed, Feb 08, 2023 at 10:13:59AM -0300, vitmau...@gmail.com wrote: > thank you all for the replies. Crystal, what command would I use to > detach a USB drive? I tried eject, but it doesn't seem to work with > the kind of devices I have. Can you explain exactly what you are trying to do, and what devices you are using? If you are just writing files to a usb flash drive from the command line, then you mount it, copy the files, and umount, nothing more. (Actually, umount automatically does a sync as soon as you invoke it.) If you are doing something more complicated or using a usb device other than a flash drive or external HD, then let us know so that we can give you accurate advice.
Re: Safely remove USB drive
In facts, there are people around who play very "nasty" with these sticks like booting from them, etc... I suggest to reach one of them and ask for a "safe unmounting" Indeed, sorry for this joke too! (time for "safe break")
Re: Safely remove USB drive
Hi, thank you all for the replies. Crystal, what command would I use to detach a USB drive? I tried eject, but it doesn't seem to work with the kind of devices I have. Best, Vitor Em qua., 8 de fev. de 2023 às 09:55, Crystal Kolipe escreveu: > > On Wed, Feb 08, 2023 at 09:27:08AM -0300, vitmau...@gmail.com wrote: > > quick and very basic question: is syncing and umounting a USB drive > > enough to safely remove it or should I execute other commands before > > unplugging these devices? > > Unless you are doing something 'unusual' (*), then just unmounting > it is fine, you don't even need to run sync. > > (*) Such as having it encrypted as a softraid device, in which case > you'd need to detach that after unmounting it. Or writing to > the raw device directly with tar, for example.
Re: Safely remove USB drive
On Wed, Feb 08, 2023 at 09:27:08AM -0300, vitmau...@gmail.com wrote: > quick and very basic question: is syncing and umounting a USB drive > enough to safely remove it or should I execute other commands before > unplugging these devices? My personal check list for safely removing removable media after use would be 1) close any open files stored there 2) make sure no process has the media as $PWD (as in, cd away from there, and really a variation on the first) 3) issue at least one sync command (some folklore will insist on three) 4) umount the media from wherever it was mounted then you can go ahead and unplug. - Peter -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team https://bsdly.blogspot.com/ https://www.bsdly.net/ https://www.nuug.no/ "Remember to set the evil bit on all malicious network traffic" delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: Safely remove USB drive
On Wed, Feb 08, 2023 at 09:27:08AM -0300, vitmau...@gmail.com wrote: > quick and very basic question: is syncing and umounting a USB drive > enough to safely remove it or should I execute other commands before > unplugging these devices? Unless you are doing something 'unusual' (*), then just unmounting it is fine, you don't even need to run sync. (*) Such as having it encrypted as a softraid device, in which case you'd need to detach that after unmounting it. Or writing to the raw device directly with tar, for example.
