Re: lsof alternative for listing open files?
Ingo, Thank you for the detailed explanation. I wasn't thinking about atomically handling both at the same time. I also see why lsof in Linux can be deceiving. @marc Yes I think it is of pretty poor design. With Ingo's explanation and the fact they both read /proc and/or use lsof. And the learning continues. On Thursday, August 9, 2018, Ingo Schwarze wrote: > Hi Edward, > > Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 06:29:04PM -0500: > >> I was looking to port bleachbit, system cleanup tool, to OpenBSD >> and one function is to make sure certain files are not in use before >> it proceeds. > > Strictly speaking, that is impossible due to a TOCTOU race condition. > You cannot do the check and the removal atomically in one step. > If you do the check and find that no process has it open, then by > the time you proceed to removing it, another process may have opened > it. Or even worse, someone may have deleted the old file or moved > it to a different name and a third person may have created a > completely new file for a completely different purpose with the old > name. None of that is OpenBSD-specific, by the way, the same > arguments hold on Linux. > > If you are willing to ignore the dangers posed by such race conditions, > then both fuser(1) and fstat(1) can be used: both take "file" > arguments. > > By the way, i just confirmed that the /proc/PID/fd/FDNUM filename > feature is indeed broken on Linux: > >$ uname -a > Linux donnerwolke.asta-kit.de 4.9.0-0.bpo.3-686 #1 SMP Debian > 4.9.30-2+deb9u5~bpo8+1 (2017-09-28) i686 GNU/Linux >$ cd /tmp >$ touch old.txt >$ tail -f old.txt > > In another terminal: > >$ cd /tmp >$ ln old.txt new.txt >$ rm old.txt >$ pgrep tail > 24052 >$ readlink /proc/24052/fd/3 > /tmp/old.txt (deleted) >$ lsof | grep new.txt >$ lsof | grep tail | grep 3r > /tmp/old.txt (deleted) > > So the kernel claims that "new.txt" is not open by any process, > and it also claims that the file open by tail(1) can no longer > be accessed via the file system. However, typing > >$ echo test >> new.txt > > in the second terminal makes "test" appear on the first terminal, > so it is a totally normal, fully functional file. > > So the description > > "Obsolete package: lsof (ancient software that doesn't work)" > > is indeed accurate. If lsof says a file isn't open, it may well > be open anyway. If lsof says a file was deleted, that may be an > outright lie. If lsof reports that a given process has a file open > with some name, then that name may be neither the name the process > used for opening the file nor any of the names the file has now, > though it usually is one of the names that the file may have had > at some undefined time in between. You cannot rely on any of those > statements from lsof because making such statements is just impossible > by the basic way how UNIX (including Linux) works, even without any > race conditions. And then you get the race conditions on top of > all that. Enjoy the mix! > > Yours, > Ingo >
Re: lsof alternative for listing open files?
On Thu, Aug 09, 2018 at 06:29:04PM -0500, Edward Lopez-Acosta wrote: > Hi Ingo, > > I was looking to port bleachbit, system cleanup tool, to OpenBSD and one > function is to make sure certain files are not in use before it proceeds. An > example would be cache files by a browser which would need closed. > > Beyond that though it was more of an educational exercise on my part as I > continue becoming familiar with OpenBSD and its workings. > > Edward Lopez-Acosta > > On 8/9/18 6:17 PM, Ingo Schwarze wrote: > >Hi Edward, > > > >Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 05:41:04PM -0500: > > > >>I am aware of fuser and fstat but these seem to only give me inodes. > >>Is there an equivalent to the Linux application `lsof`? So, how is this actually a problem for the case at hand ? Is bleachbit so badly designed that it can't be made to work with fstat output ? If I understand your problem correctly, bleachbit is going to build a list of files it wants to remove, then try its best to check whether these files are actually in use by the system. There, inode/devno information is exactly what you need. You just need to coerce bleachbit into pairing that information correctly.
Re: lsof alternative for listing open files?
Hi Theo, Theo de Raadt wrote on Thu, Aug 09, 2018 at 06:11:14PM -0600: > Ingo, you don't understand. Kids these days don't want something that > always works. It only has to work most of the time! Then they can > build a company upon the quicksand! What a pity that you already wasted weeks, if not months by making sure that what unveil(2) does with file and directory names has well-defined behaviour and is fully consistent, and that the behaviour is designed in a way that can be documented fully and clearly, in a way seeming so simple that the tricky design work preceding it is totally hidden from the user. All that time could have been saved by realizing earlier what it is that the kids actually want! (-; Ingo
Re: lsof alternative for listing open files?
Hello, > I am aware of fuser and fstat but these seem to only give me inodes. You can use # find /foo -inum 123 to search for the corresponding file.
Re: lsof alternative for listing open files?
Ingo Schwarze wrote: > Hi Edward, > > Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 06:29:04PM -0500: > > > I was looking to port bleachbit, system cleanup tool, to OpenBSD > > and one function is to make sure certain files are not in use before > > it proceeds. > > Strictly speaking, that is impossible due to a TOCTOU race condition. > You cannot do the check and the removal atomically in one step. > If you do the check and find that no process has it open, then by > the time you proceed to removing it, another process may have opened > it. Or even worse, someone may have deleted the old file or moved > it to a different name and a third person may have created a > completely new file for a completely different purpose with the old > name. None of that is OpenBSD-specific, by the way, the same > arguments hold on Linux. > > If you are willing to ignore the dangers posed by such race conditions, > then both fuser(1) and fstat(1) can be used: both take "file" > arguments. > > By the way, i just confirmed that the /proc/PID/fd/FDNUM filename > feature is indeed broken on Linux: > >$ uname -a > Linux donnerwolke.asta-kit.de 4.9.0-0.bpo.3-686 #1 SMP Debian > 4.9.30-2+deb9u5~bpo8+1 (2017-09-28) i686 GNU/Linux >$ cd /tmp >$ touch old.txt >$ tail -f old.txt > > In another terminal: > >$ cd /tmp >$ ln old.txt new.txt >$ rm old.txt >$ pgrep tail > 24052 >$ readlink /proc/24052/fd/3 > /tmp/old.txt (deleted) >$ lsof | grep new.txt >$ lsof | grep tail | grep 3r > /tmp/old.txt (deleted) > > So the kernel claims that "new.txt" is not open by any process, > and it also claims that the file open by tail(1) can no longer > be accessed via the file system. However, typing > >$ echo test >> new.txt > > in the second terminal makes "test" appear on the first terminal, > so it is a totally normal, fully functional file. > > So the description > > "Obsolete package: lsof (ancient software that doesn't work)" > > is indeed accurate. If lsof says a file isn't open, it may well > be open anyway. If lsof says a file was deleted, that may be an > outright lie. If lsof reports that a given process has a file open > with some name, then that name may be neither the name the process > used for opening the file nor any of the names the file has now, > though it usually is one of the names that the file may have had > at some undefined time in between. You cannot rely on any of those > statements from lsof because making such statements is just impossible > by the basic way how UNIX (including Linux) works, even without any > race conditions. And then you get the race conditions on top of > all that. Enjoy the mix! Ingo, you don't understand. Kids these days don't want something that always works. It only has to work most of the time! Then they can build a company upon the quicksand!
Re: lsof alternative for listing open files?
Hi Edward, Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 06:29:04PM -0500: > I was looking to port bleachbit, system cleanup tool, to OpenBSD > and one function is to make sure certain files are not in use before > it proceeds. Strictly speaking, that is impossible due to a TOCTOU race condition. You cannot do the check and the removal atomically in one step. If you do the check and find that no process has it open, then by the time you proceed to removing it, another process may have opened it. Or even worse, someone may have deleted the old file or moved it to a different name and a third person may have created a completely new file for a completely different purpose with the old name. None of that is OpenBSD-specific, by the way, the same arguments hold on Linux. If you are willing to ignore the dangers posed by such race conditions, then both fuser(1) and fstat(1) can be used: both take "file" arguments. By the way, i just confirmed that the /proc/PID/fd/FDNUM filename feature is indeed broken on Linux: $ uname -a Linux donnerwolke.asta-kit.de 4.9.0-0.bpo.3-686 #1 SMP Debian 4.9.30-2+deb9u5~bpo8+1 (2017-09-28) i686 GNU/Linux $ cd /tmp $ touch old.txt $ tail -f old.txt In another terminal: $ cd /tmp $ ln old.txt new.txt $ rm old.txt $ pgrep tail 24052 $ readlink /proc/24052/fd/3 /tmp/old.txt (deleted) $ lsof | grep new.txt $ lsof | grep tail | grep 3r /tmp/old.txt (deleted) So the kernel claims that "new.txt" is not open by any process, and it also claims that the file open by tail(1) can no longer be accessed via the file system. However, typing $ echo test >> new.txt in the second terminal makes "test" appear on the first terminal, so it is a totally normal, fully functional file. So the description "Obsolete package: lsof (ancient software that doesn't work)" is indeed accurate. If lsof says a file isn't open, it may well be open anyway. If lsof says a file was deleted, that may be an outright lie. If lsof reports that a given process has a file open with some name, then that name may be neither the name the process used for opening the file nor any of the names the file has now, though it usually is one of the names that the file may have had at some undefined time in between. You cannot rely on any of those statements from lsof because making such statements is just impossible by the basic way how UNIX (including Linux) works, even without any race conditions. And then you get the race conditions on top of all that. Enjoy the mix! Yours, Ingo
Re: lsof alternative for listing open files?
Have you looked at fstat? https://man.openbsd.org/fstat.1 Sent from mobile. Original Message From: Edward Lopez-Acosta Sent: Thursday, August 9, 2018 16:30 To: Ingo Schwarze Cc: misc@openbsd.org Subject: Re: lsof alternative for listing open files? Hi Ingo, I was looking to port bleachbit, system cleanup tool, to OpenBSD and one function is to make sure certain files are not in use before it proceeds. An example would be cache files by a browser which would need closed. Beyond that though it was more of an educational exercise on my part as I continue becoming familiar with OpenBSD and its workings. Edward Lopez-Acosta On 8/9/18 6:17 PM, Ingo Schwarze wrote: > Hi Edward, > > Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 05:41:04PM -0500: > >> I am aware of fuser and fstat but these seem to only give me inodes. >> Is there an equivalent to the Linux application `lsof`? > > $ pkg_add lsof > Obsolete package: lsof (ancient software that doesn't work) > > Once a process has a file open, there is no was to get back from > the file descriptor to a file name. Actually, the file name may > have changed since the file was opened, or the file may not have > any name whatsoever any longer, or the name might now point to a > different file that is not open. It is by design that you cannot > translate an inode number back to a filename. I have no idea what > the Linux kernel is doing with symbolic links like /proc/21325/fd/3, > but i doubt that it makes much sense. > > What is the actual problem you are trying to solve? > > Yours, > Ingo >
Re: lsof alternative for listing open files?
Hi Ingo, I was looking to port bleachbit, system cleanup tool, to OpenBSD and one function is to make sure certain files are not in use before it proceeds. An example would be cache files by a browser which would need closed. Beyond that though it was more of an educational exercise on my part as I continue becoming familiar with OpenBSD and its workings. Edward Lopez-Acosta On 8/9/18 6:17 PM, Ingo Schwarze wrote: Hi Edward, Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 05:41:04PM -0500: I am aware of fuser and fstat but these seem to only give me inodes. Is there an equivalent to the Linux application `lsof`? $ pkg_add lsof Obsolete package: lsof (ancient software that doesn't work) Once a process has a file open, there is no was to get back from the file descriptor to a file name. Actually, the file name may have changed since the file was opened, or the file may not have any name whatsoever any longer, or the name might now point to a different file that is not open. It is by design that you cannot translate an inode number back to a filename. I have no idea what the Linux kernel is doing with symbolic links like /proc/21325/fd/3, but i doubt that it makes much sense. What is the actual problem you are trying to solve? Yours, Ingo
Re: lsof alternative for listing open files?
Hi Edward, Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 05:41:04PM -0500: > I am aware of fuser and fstat but these seem to only give me inodes. > Is there an equivalent to the Linux application `lsof`? $ pkg_add lsof Obsolete package: lsof (ancient software that doesn't work) Once a process has a file open, there is no was to get back from the file descriptor to a file name. Actually, the file name may have changed since the file was opened, or the file may not have any name whatsoever any longer, or the name might now point to a different file that is not open. It is by design that you cannot translate an inode number back to a filename. I have no idea what the Linux kernel is doing with symbolic links like /proc/21325/fd/3, but i doubt that it makes much sense. What is the actual problem you are trying to solve? Yours, Ingo
lsof alternative for listing open files?
Hello, I am aware of fuser and fstat but these seem to only give me inodes. Is there an equivalent to the Linux application `lsof`? I also noticed there is no `/proc` filesystem so checking that is also out. Thank you, -- Edward Lopez-Acosta
