Re: lsof alternative for listing open files?
Hi Theo, Theo de Raadt wrote on Thu, Aug 09, 2018 at 06:11:14PM -0600: > Ingo, you don't understand. Kids these days don't want something that > always works. It only has to work most of the time! Then they can > build a company upon the quicksand! What a pity that you already wasted weeks, if not months by making sure that what unveil(2) does with file and directory names has well-defined behaviour and is fully consistent, and that the behaviour is designed in a way that can be documented fully and clearly, in a way seeming so simple that the tricky design work preceding it is totally hidden from the user. All that time could have been saved by realizing earlier what it is that the kids actually want! (-; Ingo
Re: lsof alternative for listing open files?
Hello, > I am aware of fuser and fstat but these seem to only give me inodes. You can use # find /foo -inum 123 to search for the corresponding file.
Re: lsof alternative for listing open files?
Ingo Schwarze wrote: > Hi Edward, > > Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 06:29:04PM -0500: > > > I was looking to port bleachbit, system cleanup tool, to OpenBSD > > and one function is to make sure certain files are not in use before > > it proceeds. > > Strictly speaking, that is impossible due to a TOCTOU race condition. > You cannot do the check and the removal atomically in one step. > If you do the check and find that no process has it open, then by > the time you proceed to removing it, another process may have opened > it. Or even worse, someone may have deleted the old file or moved > it to a different name and a third person may have created a > completely new file for a completely different purpose with the old > name. None of that is OpenBSD-specific, by the way, the same > arguments hold on Linux. > > If you are willing to ignore the dangers posed by such race conditions, > then both fuser(1) and fstat(1) can be used: both take "file" > arguments. > > By the way, i just confirmed that the /proc/PID/fd/FDNUM filename > feature is indeed broken on Linux: > >$ uname -a > Linux donnerwolke.asta-kit.de 4.9.0-0.bpo.3-686 #1 SMP Debian > 4.9.30-2+deb9u5~bpo8+1 (2017-09-28) i686 GNU/Linux >$ cd /tmp >$ touch old.txt >$ tail -f old.txt > > In another terminal: > >$ cd /tmp >$ ln old.txt new.txt >$ rm old.txt >$ pgrep tail > 24052 >$ readlink /proc/24052/fd/3 > /tmp/old.txt (deleted) >$ lsof | grep new.txt >$ lsof | grep tail | grep 3r > /tmp/old.txt (deleted) > > So the kernel claims that "new.txt" is not open by any process, > and it also claims that the file open by tail(1) can no longer > be accessed via the file system. However, typing > >$ echo test >> new.txt > > in the second terminal makes "test" appear on the first terminal, > so it is a totally normal, fully functional file. > > So the description > > "Obsolete package: lsof (ancient software that doesn't work)" > > is indeed accurate. If lsof says a file isn't open, it may well > be open anyway. If lsof says a file was deleted, that may be an > outright lie. If lsof reports that a given process has a file open > with some name, then that name may be neither the name the process > used for opening the file nor any of the names the file has now, > though it usually is one of the names that the file may have had > at some undefined time in between. You cannot rely on any of those > statements from lsof because making such statements is just impossible > by the basic way how UNIX (including Linux) works, even without any > race conditions. And then you get the race conditions on top of > all that. Enjoy the mix! Ingo, you don't understand. Kids these days don't want something that always works. It only has to work most of the time! Then they can build a company upon the quicksand!
Re: lsof alternative for listing open files?
Hi Edward, Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 06:29:04PM -0500: > I was looking to port bleachbit, system cleanup tool, to OpenBSD > and one function is to make sure certain files are not in use before > it proceeds. Strictly speaking, that is impossible due to a TOCTOU race condition. You cannot do the check and the removal atomically in one step. If you do the check and find that no process has it open, then by the time you proceed to removing it, another process may have opened it. Or even worse, someone may have deleted the old file or moved it to a different name and a third person may have created a completely new file for a completely different purpose with the old name. None of that is OpenBSD-specific, by the way, the same arguments hold on Linux. If you are willing to ignore the dangers posed by such race conditions, then both fuser(1) and fstat(1) can be used: both take "file" arguments. By the way, i just confirmed that the /proc/PID/fd/FDNUM filename feature is indeed broken on Linux: $ uname -a Linux donnerwolke.asta-kit.de 4.9.0-0.bpo.3-686 #1 SMP Debian 4.9.30-2+deb9u5~bpo8+1 (2017-09-28) i686 GNU/Linux $ cd /tmp $ touch old.txt $ tail -f old.txt In another terminal: $ cd /tmp $ ln old.txt new.txt $ rm old.txt $ pgrep tail 24052 $ readlink /proc/24052/fd/3 /tmp/old.txt (deleted) $ lsof | grep new.txt $ lsof | grep tail | grep 3r /tmp/old.txt (deleted) So the kernel claims that "new.txt" is not open by any process, and it also claims that the file open by tail(1) can no longer be accessed via the file system. However, typing $ echo test >> new.txt in the second terminal makes "test" appear on the first terminal, so it is a totally normal, fully functional file. So the description "Obsolete package: lsof (ancient software that doesn't work)" is indeed accurate. If lsof says a file isn't open, it may well be open anyway. If lsof says a file was deleted, that may be an outright lie. If lsof reports that a given process has a file open with some name, then that name may be neither the name the process used for opening the file nor any of the names the file has now, though it usually is one of the names that the file may have had at some undefined time in between. You cannot rely on any of those statements from lsof because making such statements is just impossible by the basic way how UNIX (including Linux) works, even without any race conditions. And then you get the race conditions on top of all that. Enjoy the mix! Yours, Ingo
Re: lsof alternative for listing open files?
Have you looked at fstat? https://man.openbsd.org/fstat.1 Sent from mobile. Original Message From: Edward Lopez-Acosta Sent: Thursday, August 9, 2018 16:30 To: Ingo Schwarze Cc: misc@openbsd.org Subject: Re: lsof alternative for listing open files? Hi Ingo, I was looking to port bleachbit, system cleanup tool, to OpenBSD and one function is to make sure certain files are not in use before it proceeds. An example would be cache files by a browser which would need closed. Beyond that though it was more of an educational exercise on my part as I continue becoming familiar with OpenBSD and its workings. Edward Lopez-Acosta On 8/9/18 6:17 PM, Ingo Schwarze wrote: > Hi Edward, > > Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 05:41:04PM -0500: > >> I am aware of fuser and fstat but these seem to only give me inodes. >> Is there an equivalent to the Linux application `lsof`? > > $ pkg_add lsof > Obsolete package: lsof (ancient software that doesn't work) > > Once a process has a file open, there is no was to get back from > the file descriptor to a file name. Actually, the file name may > have changed since the file was opened, or the file may not have > any name whatsoever any longer, or the name might now point to a > different file that is not open. It is by design that you cannot > translate an inode number back to a filename. I have no idea what > the Linux kernel is doing with symbolic links like /proc/21325/fd/3, > but i doubt that it makes much sense. > > What is the actual problem you are trying to solve? > > Yours, > Ingo >
Re: lsof alternative for listing open files?
Hi Ingo, I was looking to port bleachbit, system cleanup tool, to OpenBSD and one function is to make sure certain files are not in use before it proceeds. An example would be cache files by a browser which would need closed. Beyond that though it was more of an educational exercise on my part as I continue becoming familiar with OpenBSD and its workings. Edward Lopez-Acosta On 8/9/18 6:17 PM, Ingo Schwarze wrote: Hi Edward, Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 05:41:04PM -0500: I am aware of fuser and fstat but these seem to only give me inodes. Is there an equivalent to the Linux application `lsof`? $ pkg_add lsof Obsolete package: lsof (ancient software that doesn't work) Once a process has a file open, there is no was to get back from the file descriptor to a file name. Actually, the file name may have changed since the file was opened, or the file may not have any name whatsoever any longer, or the name might now point to a different file that is not open. It is by design that you cannot translate an inode number back to a filename. I have no idea what the Linux kernel is doing with symbolic links like /proc/21325/fd/3, but i doubt that it makes much sense. What is the actual problem you are trying to solve? Yours, Ingo
Re: lsof alternative for listing open files?
Hi Edward, Edward Lopez-Acosta wrote on Thu, Aug 09, 2018 at 05:41:04PM -0500: > I am aware of fuser and fstat but these seem to only give me inodes. > Is there an equivalent to the Linux application `lsof`? $ pkg_add lsof Obsolete package: lsof (ancient software that doesn't work) Once a process has a file open, there is no was to get back from the file descriptor to a file name. Actually, the file name may have changed since the file was opened, or the file may not have any name whatsoever any longer, or the name might now point to a different file that is not open. It is by design that you cannot translate an inode number back to a filename. I have no idea what the Linux kernel is doing with symbolic links like /proc/21325/fd/3, but i doubt that it makes much sense. What is the actual problem you are trying to solve? Yours, Ingo
lsof alternative for listing open files?
Hello, I am aware of fuser and fstat but these seem to only give me inodes. Is there an equivalent to the Linux application `lsof`? I also noticed there is no `/proc` filesystem so checking that is also out. Thank you, -- Edward Lopez-Acosta
Two Factor Authentication Prompt
Hi, I currently use login_ldap to authenticate local user accounts against LDAP. However, I would also like to deploy a two factor approach via SSH eg. using TOTP via login_oath. I can successfully have a user authenticate with either (switching the login class). However, is it possible to use both as consecutive login prompts?
Re: routing traffic to transparent squid cluster
On Thu, Aug 9, 2018 at 7:59 AM, Joerg Streckfuss wrote: > Dear list, > > i'm playing around with a squid setup, where the http traffic from a > client is transparently routed from the gateway (openbsd 6.3) to two squid > caches (squid 3.5.28). This means the caches are _not_ placed on the > gateway. > I'm not sure if it meets your requirements, but you may wish to consider using the Web Proxy Auto-Discovery Protocol (WPAD) to tell clients which proxy to use. It's not bump-in-the-wire transparent, but it may be an option.
routing traffic to transparent squid cluster
Dear list,
i'm playing around with a squid setup, where the http traffic from a client is
transparently routed from the gateway (openbsd 6.3) to two squid caches (squid
3.5.28). This means the caches are _not_ placed on the gateway.
With PF this is very easy to achieve:
pass in quick on $INT_IF inet proto tcp from $CLIENT to any port 80 \
route-to { ( $DMZ_IF $SQUID_1), (trunk2 SQUID_2) } least-states
So far, so good. My next goal is redundancy. In other words the gateway should
stop routing traffic to an unreachable cache. Imho I thought this is very easy
to achieve with the help of relayd.
To map the upper PF rule to a fully redundant setup, I tried something like
this:
PF:
pass in quick on $INT_IF inet proto tcp from $CLIENT to any port http \
divert-to 127.0.0.1 port 3130
Relayd:
relay webproxy_3130 {
listen on 127.0.0.1 port 3130
transparent forward to port 80 check tcp mode loadbalance
}
But of course this doesn't work because the relay translates the destination
address which it should not. I didn't found any options like a pf route-to for
relays and think it wouldn't make much sense in the context of relays.
Relayd supports a route-to option for redirects but I dind't found a working
configuration.
Perhaps this is all broken by design. If so could somebody point me out a better
solution (haproxy in front of the caches)?
Any help would be greatly appreciated.
Thanks
USB composite devices & OpenBSD
There are a lot of modern composite USB devices on the market now like Modems, cameras, etc. How to make them work in OpenBSD like penguin's usbmodeswitch works?
Re: openbsd port of Emby or Plex
On 08/08/18 16:14, Marc Espie wrote: On Wed, Aug 08, 2018 at 11:30:52AM -0700, Jordan Geoghegan wrote: I know minidlna is in ports, but it doesn't do on the fly media transcoding/remuxing with FFmpeg nor does it have an integrated HTML5 video player like Serviio does. I tried minidlna, but it unfortunately doesnt work for me as my devices need the transcoding/remuxing feature that Serviio offers. For example, my bluray player refuses to playback most video containers over DLNA (mp4, mkv etc) but will play most video formats (AVC, Xvid, mpeg2 etc) all it needs is for the video files to be remuxed into an M2TS or AVI container, which Serviio is able to do on the fly without transcoding. It's also useful for making my H265 content available on my older devices that don't natively support the newer codecs/containers. I also quite enjoy the HTML5 video player, as it allows me to watch my movies on the go. It's annoying that it's proprietary, but at least it works well on OpenBSD. Tip: devices like android tablets or the XBox One don't necessarily advertize it, but they DO have VLC, and it's fairly happy with most x265 content. The only issue I've had with ldna service so far is the need to remux subtitles into an mp4/mkv because separate .srt files do not work. That's fairly easy to do and rather permanent. Out of a fairly large sample of video files, I can say that VLC on those devices handles everything but maybe 0.05%. Yes VLC is an amazing program and I use it heavily. MiniDLNA is also a great project. Unfortunately VLC isn't available for Xbox 360. MiniDLNA doesn't work very good for my BDP S5200 bluray players or my Xbox 360 devices (of which my family owns over half a dozen) as they require AVI, M2TS or WMV containers respectively. It isn't feasible for me to remux/transcode my 10TB media library (which is comprised mostly of mkv) just so I can watch cartoons on my living room TV. Granted I could just build a dedicated OpenBSD media streaming box, but that isn't a feasible solution for my mother or my technophobe sister who both live on the other side of the country from me. It was easy enough for me to just set up a small machine next to their router and just scp new shows over to them as needed, where Serviio then detects the new content and automatically adds it to their library; From there it can then be accessed on their smart TV, bluray player, 360, computer or iPad either at home or abroad. MiniDLNA and VLC also don't really have an equivalent to Serviio's HTML5 streaming server. It is nice to be able to just pop open a browser and have access to all my media on my home server. I like that Serviio can parse movie/TV show names and connect to IMDB / TVDB to grab relevant metadata and cover art for my files. It is nice to see my files optionally presented with actual correct episode names, rather than "S01E03$TVShowName$TorrentersName". I find that the video metadata makes wading through 10,000+ movies and TV episodes a much more pleasant experience, as I can search by year, actor, genre etc. Another neat feature it offers is the ability to automatically download subtitles for all your videos and either on-the-fly transcode the file to burn them in, or remux on-the-fly to embed the subtitles. On my home server, I run Serviio in a vmm(4) virtual machine with read-only NFS access to my media library. Proprietary software is bad ju ju, but I find Serviio to be a preferable all in one solution to Plex, as I can at least run it on my favourite OS.
