Re: An FS question perhaps... non blocking I/O.
John Polstra scribbled this message on Sep 12: > In article <[EMAIL PROTECTED]>, > Garrett Wollman <[EMAIL PROTECTED]> wrote: > > [POLLEXTEND, POLLATTRIB, POLLNLINK, POLLWRITE] > > > It is probably undocumented. I was a bit reluctant to document it > > since I know that the interface is not correct. One of these days, > > I (or more likely some enterprising young hacker) will fix it. > > Just to avoid duplicated effort: I currently have work in progress > on a "fslog" pseudo-device. It enables you to monitor a filesystem > and receive notifications for all interesting changes to files and > directories. This includes reads, writes, renames, file creations, > unlinks, links, etc. -- anything that changes the stat(2) results > for a file, or causes directory entries to be created, destroyed, or > changed. The device itself is working, but so far I have implemented > the support for only a few of the event types. It won't take much > more work to finish it. ugh, why aren't you extending poll to work on files and directories to get this info?? it would make MUCH more sense to extend poll to do this.. any specific reason why it wasn't done this way? > My personal interest is to allow a CVSup master server to avoid > doing a tree walk whenever a client connects. I want to provide the > functionality of the old "supscan" utility, but in real time. I completely agree that something like this should be implemented.. then we can out shine winXX in where file managers have files appeare INSTANTLY after they are created... :) -- John-Mark Gurney Voice: +1 541 684 8449 Cu Networking P.O. Box 5693, 97405 "The soul contains in itself the event that shall presently befall it. The event is only the actualizing of its thought." -- Ralph Waldo Emerson To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
> In message <[EMAIL PROTECTED]>, John Polstra writes: > > >> ugh, why aren't you extending poll to work on files and directories to > >> get this info?? it would make MUCH more sense to extend poll to do this.. > >> > >> any specific reason why it wasn't done this way? > > > >Yes. Last time I checked, our CVS repository contained 50,000 files > >in 13,000 directories. Somehow the thought of a 63,000-element pollfd > >array leaves me cold. > > Sounds like something Bruce would do :-) He can try it on the repository here: $ find . -type d | wc -l 8764 $ find . -type f | wc -l 586382 $ It would be one expensive call with that many elements to scan... > > - -- > Poul-Henning Kamp FreeBSD coreteam member > [EMAIL PROTECTED] "Real hackers run -current on their laptop." > FreeBSD -- It will take a long time before progress goes too far! > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-current" in the body of the message > > -- > To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
In message <[EMAIL PROTECTED]> John Polstra writes: : There are now 63000 files and directories in the repository. : That's 2**3 * 3**2 * 5**3 * 7. If we concatenate the exponents, : we get 3231, which is 3**2 * 359. Repeating, we get 21, which : is 3 * 7. One more repetition and we get 11, which is itself a : prime, as is 37. Isn't that amazing? : : ;-) Stop. The Illuminati "Law of 5's" is spreading. After all we have 2**3 (2+3 is 5), 3**2 (ditto) and 5**3 (three of the sacred 5's). The 7 is put in to confuse The Man[tm], but true afficionados know about the deception and how to deal with that. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
Poul-Henning Kamp wrote: >>Somehow the thought of a 63,000-element pollfd array leaves me cold. > > Sounds like something Bruce would do :-) It could be worse. Satoshi, for example, would say something like: There are now 63000 files and directories in the repository. That's 2**3 * 3**2 * 5**3 * 7. If we concatenate the exponents, we get 3231, which is 3**2 * 359. Repeating, we get 21, which is 3 * 7. One more repetition and we get 11, which is itself a prime, as is 37. Isn't that amazing? ;-) John --- John Polstra [EMAIL PROTECTED] John D. Polstra & Co., Inc.Seattle, Washington USA "No matter how cynical I get, I just can't keep up."-- Nora Ephron To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
In message <[EMAIL PROTECTED]>, John Polstra writes: >> ugh, why aren't you extending poll to work on files and directories to >> get this info?? it would make MUCH more sense to extend poll to do this.. >> >> any specific reason why it wasn't done this way? > >Yes. Last time I checked, our CVS repository contained 50,000 files >in 13,000 directories. Somehow the thought of a 63,000-element pollfd >array leaves me cold. Sounds like something Bruce would do :-) -- Poul-Henning Kamp FreeBSD coreteam member [EMAIL PROTECTED] "Real hackers run -current on their laptop." FreeBSD -- It will take a long time before progress goes too far! To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
John-Mark Gurney wrote: > John Polstra scribbled this message on Sep 12: >> >> Just to avoid duplicated effort: I currently have work in progress >> on a "fslog" pseudo-device. It enables you to monitor a filesystem >> and receive notifications for all interesting changes to files and >> directories. This includes reads, writes, renames, file creations, >> unlinks, links, etc. -- anything that changes the stat(2) results >> for a file, or causes directory entries to be created, destroyed, or >> changed. The device itself is working, but so far I have implemented >> the support for only a few of the event types. It won't take much >> more work to finish it. > > ugh, why aren't you extending poll to work on files and directories to > get this info?? it would make MUCH more sense to extend poll to do this.. > > any specific reason why it wasn't done this way? Yes. Last time I checked, our CVS repository contained 50,000 files in 13,000 directories. Somehow the thought of a 63,000-element pollfd array leaves me cold. Sometimes you want to know about all changes in a whole tree of files. Poll isn't well-suited for that. John --- John Polstra [EMAIL PROTECTED] John D. Polstra & Co., Inc.Seattle, Washington USA "No matter how cynical I get, I just can't keep up."-- Nora Ephron To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
In article <[EMAIL PROTECTED]>, Tony Finch <[EMAIL PROTECTED]> wrote: > > Are there any standard APIs for doing file meta-operations > asynchronously? (open, close, creat, link, unlink, etc.) I only know of one way to do that. Hand the operation off to a separate process, using the FD-passing feature of Unix domain sockets to send the file descriptor back and forth. John -- John Polstra [EMAIL PROTECTED] John D. Polstra & Co., Inc.Seattle, Washington USA "No matter how cynical I get, I just can't keep up."-- Nora Ephron To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
In message <[EMAIL PROTECTED]> John Polstra writes: : My personal interest is to allow a CVSup master server to avoid : doing a tree walk whenever a client connects. I want to provide the : functionality of the old "supscan" utility, but in real time. Also useful for tasty /dev/ persistance in a devfs world. A devfsd could certainly use this feature. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
On Sun, Sep 12, 1999 at 10:56:42AM -0700, John Polstra wrote: > Just to avoid duplicated effort: I currently have work in progress > on a "fslog" pseudo-device. It enables you to monitor a filesystem > and receive notifications for all interesting changes to files and > directories. This includes reads, writes, renames, file creations, > unlinks, links, etc. -- anything that changes the stat(2) results > for a file, or causes directory entries to be created, destroyed, or > changed. The device itself is working, but so far I have implemented > the support for only a few of the event types. It won't take much > more work to finish it. Very nice; this is the way that Microsoft's web indexing engine works. It opens a whole host of interesting stuff... -- Dom Mitchell -- Palmer & Harvey McLane -- Unix Systems Administrator "vi has two modes the one in which it beeps and the one in which it doesnt." -- Anon. ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ** To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
Garrett Wollman wrote: > < said: > >> Just to avoid duplicated effort: I currently have work in progress >> on a "fslog" pseudo-device. It enables you to monitor a filesystem >> and receive notifications for all interesting changes to files and >> directories. > > However, this is not necessarily duplicative of making polling on > files work. Yes, I agree. I just thought things might migrate in that direction and wanted to avoid wasted work. John To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
< said: > Just to avoid duplicated effort: I currently have work in progress > on a "fslog" pseudo-device. It enables you to monitor a filesystem > and receive notifications for all interesting changes to files and > directories. However, this is not necessarily duplicative of making polling on files work. For one thing, it's unlikely that we will want to require fslog be universally configured just to make `tail -f' and `biff' work. -GAWollman -- Garrett A. Wollman | O Siem / We are all family / O Siem / We're all the same [EMAIL PROTECTED] | O Siem / The fires of freedom Opinions not those of| Dance in the burning flame MIT, LCS, CRS, or NSA| - Susan Aglukark and Chad Irschick To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
Rodney W. Grimes wrote: > > COOL!! And these mods could make a really nice addition to system > auditing tools required by high security systems. Do you pass anytype > of ``who done it'' via fslog? As of now, no. But it would be easy to add, and I think it's a great idea. Thanks! John To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
Julian Elischer <[EMAIL PROTECTED]> wrote: > >The Posix AIO calls that john implememted are the best way of doing this. Are there any standard APIs for doing file meta-operations asynchronously? (open, close, creat, link, unlink, etc.) Tony. -- f.a.n.finch[EMAIL PROTECTED][EMAIL PROTECTED]e pluribus unix To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
> In article <[EMAIL PROTECTED]>, > Garrett Wollman <[EMAIL PROTECTED]> wrote: > > [POLLEXTEND, POLLATTRIB, POLLNLINK, POLLWRITE] > > > It is probably undocumented. I was a bit reluctant to document it > > since I know that the interface is not correct. One of these days, > > I (or more likely some enterprising young hacker) will fix it. > > Just to avoid duplicated effort: I currently have work in progress > on a "fslog" pseudo-device. It enables you to monitor a filesystem > and receive notifications for all interesting changes to files and > directories. This includes reads, writes, renames, file creations, > unlinks, links, etc. -- anything that changes the stat(2) results > for a file, or causes directory entries to be created, destroyed, or > changed. The device itself is working, but so far I have implemented > the support for only a few of the event types. It won't take much > more work to finish it. > > My personal interest is to allow a CVSup master server to avoid > doing a tree walk whenever a client connects. I want to provide the > functionality of the old "supscan" utility, but in real time. COOL!! And these mods could make a really nice addition to system auditing tools required by high security systems. Do you pass anytype of ``who done it'' via fslog? -- Rod Grimes - KD7CAX - (RWG25)[EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
In article <[EMAIL PROTECTED]>, Garrett Wollman <[EMAIL PROTECTED]> wrote: [POLLEXTEND, POLLATTRIB, POLLNLINK, POLLWRITE] > It is probably undocumented. I was a bit reluctant to document it > since I know that the interface is not correct. One of these days, > I (or more likely some enterprising young hacker) will fix it. Just to avoid duplicated effort: I currently have work in progress on a "fslog" pseudo-device. It enables you to monitor a filesystem and receive notifications for all interesting changes to files and directories. This includes reads, writes, renames, file creations, unlinks, links, etc. -- anything that changes the stat(2) results for a file, or causes directory entries to be created, destroyed, or changed. The device itself is working, but so far I have implemented the support for only a few of the event types. It won't take much more work to finish it. My personal interest is to allow a CVSup master server to avoid doing a tree walk whenever a client connects. I want to provide the functionality of the old "supscan" utility, but in real time. John -- John Polstra [EMAIL PROTECTED] John D. Polstra & Co., Inc.Seattle, Washington USA "No matter how cynical I get, I just can't keep up."-- Nora Ephron To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
Luigi Rizzo <[EMAIL PROTECTED]> writes: > Is there any way to guarantee (more or less strictly, see below) > that when i issue a read() on a file (a real file coming from a > UFS i mean) such read will not block because data from the disk is > not in memory yet, yet avoid that i end up in a busy loop. There is no way to guarantee that a read will not block on disk I/O. The same applies to writes. The read/write calls should never return EAGAIN for local files, even if they are set for non-blocking I/O. This actually makes sense. Given virtual memory, you also don't know if your call to read or the memory you're copying to is actually in memory, it might block to page something in, which is also disk I/O and is comparable to the I/O for the actual file. So even if non-blocking would "work" on local files (how could it, meaningfully?), it would be a hint at best. > Any suggestion ? Would async I/O help in a case like this, considering > the possible overhead for handling signals when a transfer is > complete ? Yes, aio is probably the best alternative. It is quite expensive, though. Another alternative is to use multiple processes, but it also has considerable overhead. FreeBSD and typical Unix-like systems in general aren't perfect. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
RE: An FS question perhaps... non blocking I/O.
< said: > I just checked, the poll(2) listed at > http://www.freebsd.org/cgi/man.cgi?query=poll&apropos=0&sektion=0&manpath=Fr > eeBSD+4.0-current&format=html as being from 4.0-current is dated Sept 7, > 1996; either the man page search on the web is out of date or this is an > undocumented bit? It is probably undocumented. I was a bit reluctant to document it since I know that the interface is not correct. One of these days, I (or more likely some enterprising young hacker) will fix it. -GAWollman -- Garrett A. Wollman | O Siem / We are all family / O Siem / We're all the same [EMAIL PROTECTED] | O Siem / The fires of freedom Opinions not those of| Dance in the burning flame MIT, LCS, CRS, or NSA| - Susan Aglukark and Chad Irschick To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
RE: An FS question perhaps... non blocking I/O.
Hmm. I guess this was added to -current; I don't actually run -current, just read the list :) poll(2) on my -stable system doesn't mention POLLEXTEND. I just checked, the poll(2) listed at http://www.freebsd.org/cgi/man.cgi?query=poll&apropos=0&sektion=0&manpath=Fr eeBSD+4.0-current&format=html as being from 4.0-current is dated Sept 7, 1996; either the man page search on the web is out of date or this is an undocumented bit? Kelly ~[EMAIL PROTECTED]~ FreeBSD - The Power To Serve - http://www.freebsd.org/ Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD/ "The ultimate result of shielding men from the effects of folly is to fill the world with fools." - Herbert Spencer > -Original Message- > From: Juha Nurmela [mailto:[EMAIL PROTECTED]] > Sent: Thursday, September 09, 1999 3:26 PM > To: Kelly Yancey > Subject: Re: An FS question perhaps... non blocking I/O. > > > > Hello. > > Check out poll(2) and POLLEXTEND. > Don't know anything about it, have just seen it > when looking at other things. > > Juha > > To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
> Date: Thu, 9 Sep 1999 16:18:57 +0200 (MET DST) > From: Luigi Rizzo <[EMAIL PROTECTED]> > Subject: An FS question perhaps... non blocking I/O. > [ ... snip ... ] > > The app i have in mind is squid-like, which, if i understand > well, is a > single process looping around a select. If i get things > right, select() > on a file descriptor will return the read bit set if i am not at the > end of file, yet the block might not be in memory yet even if the UFS > seems to do readahead. Maybe i can turn NONBLOCK on for these > descriptors, but still would have select returning essentially useless > info as I'd need to try the read() to be sure... > [ ... snip ... ] Since you are talking about files, I presume when you say "select() on a file descriptor will return the read bit set if I am not at the end of file" you are implying the file descriptor is for a file, not a socket. Admittingly, this has nothing to do with your question directly, but I wanted to clarify something (if nothing else then for my sake): select() will not block on file descriptors referring to files. The logic being that select is only supposed to wait until the operation can occur without blocking (namely a read operation). But reads on files never block, they just return a short count, hence select() always indicates success for file descriptors for open files (and won't block on them). And, we mentioned the the last sentence, select() will always return success for reading file descriptors for open files, EOF or not, since the read would just return a short count. So again, it has nothing to do with the question of finding out whether the data for the read in already in memory, but how can you tell if you are already at EOF for a file using select(), as you describe. I was basing the above statements off of various books I have read, and have never tried select()ing a file's file descriptor for reading as I have always been told it is pointless. If there is a way to do it, I would love to hear it (I have a project that requires tail -f like functionality and I would much prefer to use select() than just checking the file size every so often like tail does). Thanks, Kelly ~[EMAIL PROTECTED]~ FreeBSD - The Power To Serve - http://www.freebsd.org/ Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD/ "The ultimate result of shielding men from the effects of folly is to fill the world with fools." - Herbert Spencer To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: An FS question perhaps... non blocking I/O.
The Posix AIO calls that john implememted are the best way of doing this. On Thu, 9 Sep 1999, Luigi Rizzo wrote: > Hi, > > please redirect to the appropriate forum if appropriate. > > There is one thing i don't completely understand with non-blocking > FS operation. > > Is there any way to guarantee (more or less strictly, see below) > that when i issue a read() on a file (a real file coming from a > UFS i mean) such read will not block because data from the disk is > not in memory yet, yet avoid that i end up in a busy loop. > > The app i have in mind is squid-like, which, if i understand well, is a > single process looping around a select. If i get things right, select() > on a file descriptor will return the read bit set if i am not at the > end of file, yet the block might not be in memory yet even if the UFS > seems to do readahead. Maybe i can turn NONBLOCK on for these > descriptors, but still would have select returning essentially useless > info as I'd need to try the read() to be sure... > > Maybe poll() has some way to indicate "yes you can read and the page is > in memory" ? I can tolerate some occasional failures when because of > memory shortage the readahead page is discarded after the select/poll > in favour of some other one... > > Any suggestion ? Would async I/O help in a case like this, considering > the possible overhead for handling signals when a transfer is > complete ? > > thanks > luigi > ---+- > Luigi RIZZO, [EMAIL PROTECTED] . Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > TEL/FAX: +39-050-568.533/522 . via Diotisalvi 2, 56126 PISA (Italy) > > http://www.iet.unipi.it/~luigi/ngc99/ > First International Workshop on Networked Group Communication > ---+- > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-current" in the body of the message > To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
