Re: how tail waits for file to appear again?
Hello, Thank you for your advice! After I ran this tool and again read tail sources and `man kqueue` more carefully I hope I now have a better understanding how it is done it tail. While all the files opened with tail are in place kqueue is called with zero timeout which makes it just wait for chosen events forever. When one of the files being watched is deleted or renamed kqueue is repeatedly called with a 1 second timeout which will obviously return every time with 0 after which it is checked if the file which previously disappeared is back. If the file is back the timeout for kqueue is again set to zero, old file stats are replaced with new ones and normal data watching is resumed. On Fri Feb 17 15:17:17 2023, Stuart Henderson wrote: > On 2023-02-17, Maksim Rodin wrote: > >> > I was able to reproduce watching for new data and truncation of the > >> > file using "kqueue" but I do not quite understand how the original tail > >> > watches when the file appears again after deletion or renaming. > > I am sorry that I could not be clear enough in my words above. > > I meant I already understood a little how "kqueue" > > magic works. And I replicated some event watching logic in my learning > > task. > > I see how file truncating, deleting and renaming is watched by waiting > > NOTE_DELETE, NOTE_RENAME and NOTE_TRUNCATE events in EVFILT_VNODE > > filter. > > But I still do not see where and what event is watched for to make sure the > > file with initial file name is back (e.g. after deletion). After deleting a > > file there is no initial file > > descriptor, so there is nothing to watch using the old EVFILT_VNODE > > filter. > > I also see that after NOTE_DELETE | NOTE_RENAME events are caught only > > tfreopen function > > is called and I do not see any event watching actions in that function. > > It will probably be easier to grok if you see for yourself rather > than have it explained. > > I suggest running tail under ktrace, and watch the output live e.g. > with something like "kdump -l | ts %.T". In particular note the last > parameter to kevent(). > > > So my primary question might be: how can I monitor file creation (using > > events) > > by only knowing its name? > > While events can be used as part of it, you do need a little more > than events to watch for file creation. > > -- С уважением, Родин Максим
Re: how tail waits for file to appear again?
Hi, for the OpenBSD version, see kevent(2) and grep the source for kevent, kqueue, and EV_SET. BSD 4.4 used select(2) because it was faster than sleep (you can find the sources e.g. on github). Best regards Robert On Fri, 17 Feb 2023 08:23:13 +0300 Maksim Rodin wrote: > Hello, > Sorry if I chose the wrong place to ask such a question. > I have been learning C for a couple of months and along with reading > "C Primer Plus" by Stephen Prata and doing some exercises from it I > took a hard (for me) task to replicate a tail program in its simplest > form. I was able to reproduce watching for new data and truncation of > the file using kqueue but I do not quite understand how the original > tail watches when the file appears again after deletion or renaming. > By reading the original tail sources downloaded from OpenBSD mirror I > see that this is done by calling tfreopen function which seems to use > a "for" loop to (continuously?) stat(2) the file name till stat(2) > successfully returns and it does not seem to load a CPU as a simple > continuous "for" loop would do. > Can someone explain how it is done? > May be there is a better way to watch for the file to appear > correctly? Is inserting a sleep(3) in a loop an appropriate way?
Re: how tail waits for file to appear again?
On 2023-02-17, Maksim Rodin wrote: >> > I was able to reproduce watching for new data and truncation of the >> > file using "kqueue" but I do not quite understand how the original tail >> > watches when the file appears again after deletion or renaming. > I am sorry that I could not be clear enough in my words above. > I meant I already understood a little how "kqueue" > magic works. And I replicated some event watching logic in my learning > task. > I see how file truncating, deleting and renaming is watched by waiting > NOTE_DELETE, NOTE_RENAME and NOTE_TRUNCATE events in EVFILT_VNODE > filter. > But I still do not see where and what event is watched for to make sure the > file with initial file name is back (e.g. after deletion). After deleting a > file there is no initial file > descriptor, so there is nothing to watch using the old EVFILT_VNODE > filter. > I also see that after NOTE_DELETE | NOTE_RENAME events are caught only > tfreopen function > is called and I do not see any event watching actions in that function. It will probably be easier to grok if you see for yourself rather than have it explained. I suggest running tail under ktrace, and watch the output live e.g. with something like "kdump -l | ts %.T". In particular note the last parameter to kevent(). > So my primary question might be: how can I monitor file creation (using > events) > by only knowing its name? While events can be used as part of it, you do need a little more than events to watch for file creation.
Re: how tail waits for file to appear again?
> > I was able to reproduce watching for new data and truncation of the > > file using "kqueue" but I do not quite understand how the original tail > > watches when the file appears again after deletion or renaming. I am sorry that I could not be clear enough in my words above. I meant I already understood a little how "kqueue" magic works. And I replicated some event watching logic in my learning task. I see how file truncating, deleting and renaming is watched by waiting NOTE_DELETE, NOTE_RENAME and NOTE_TRUNCATE events in EVFILT_VNODE filter. But I still do not see where and what event is watched for to make sure the file with initial file name is back (e.g. after deletion). After deleting a file there is no initial file descriptor, so there is nothing to watch using the old EVFILT_VNODE filter. I also see that after NOTE_DELETE | NOTE_RENAME events are caught only tfreopen function is called and I do not see any event watching actions in that function. So my primary question might be: how can I monitor file creation (using events) by only knowing its name? On Fri Feb 17 11:47:03 2023, Mike Fischer wrote: > > > Am 17.02.2023 um 06:23 schrieb Maksim Rodin : > > > > Hello, > > Sorry if I chose the wrong place to ask such a question. > > I have been learning C for a couple of months and along with reading > > "C Primer Plus" by Stephen Prata and doing some exercises from it I took > > a hard (for me) task to replicate a tail program in its simplest form. > > I was able to reproduce watching for new data and truncation of the > > file using kqueue but I do not quite understand how the original tail > > watches when the file appears again after deletion or renaming. > > By reading the original tail sources downloaded from OpenBSD mirror I > > see that this is done by calling tfreopen function which seems to use a > > "for" loop to (continuously?) stat(2) the file name till stat(2) > > successfully > > returns and it does not seem to load a CPU as a simple continuous "for" > > loop would do. > > No, the for loop in line 362 of forward.c > (https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/tail/forward.c?annotate=1.33) > iterates over the files. Note that tail allows you to monitor more than one > file at a time, see tail(1). > > > > Can someone explain how it is done? > > tfreopen is called in line 224 of the same file inside a while(1) loop. At > the top of this loop kevent is called (L191). See kevent(2) for details on > how that works. That is the real _magic_ here ;-) > > tfqueue sets up the event mechanism for a single file so you may want to look > at that as well. > > > > May be there is a better way to watch for the file to appear correctly? > > The way tail(1) does this seems pretty optimal to me. > > > > Is inserting a sleep(3) in a loop an appropriate way? > > You could do this, but it’s less optimal than using kqueue/kevent because > sleep(3) will wait longer than necessary in some cases and wake up sooner > than required in others. It is basically a way to do polling which is always > worse than event driven code. > > > > > > Below is the function how it is done in tail: > > It would have been better to cite the file name and line numbers, very easy > with https://cvsweb.openbsd.org as I did above. There is also a mirror of the > repo on Github, which also makes this sort of thing very easy: > https://github.com/openbsd. E.g.: > https://github.com/openbsd/src/blob/master/usr.bin/tail/forward.c#L224 > > The links to the repositories are right on the https://www.openbsd.org home > page, so not hard to find at all. > > > HTH > Mike > > PS. Note that I am not an expert on kqueue/kevent programming. So followups > for details on these functions would probably need to be answered by someone > else. -- Maksim Rodin
Re: how tail waits for file to appear again?
> Am 17.02.2023 um 06:23 schrieb Maksim Rodin : > > Hello, > Sorry if I chose the wrong place to ask such a question. > I have been learning C for a couple of months and along with reading > "C Primer Plus" by Stephen Prata and doing some exercises from it I took > a hard (for me) task to replicate a tail program in its simplest form. > I was able to reproduce watching for new data and truncation of the > file using kqueue but I do not quite understand how the original tail > watches when the file appears again after deletion or renaming. > By reading the original tail sources downloaded from OpenBSD mirror I > see that this is done by calling tfreopen function which seems to use a > "for" loop to (continuously?) stat(2) the file name till stat(2) successfully > returns and it does not seem to load a CPU as a simple continuous "for" > loop would do. No, the for loop in line 362 of forward.c (https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/tail/forward.c?annotate=1.33) iterates over the files. Note that tail allows you to monitor more than one file at a time, see tail(1). > Can someone explain how it is done? tfreopen is called in line 224 of the same file inside a while(1) loop. At the top of this loop kevent is called (L191). See kevent(2) for details on how that works. That is the real _magic_ here ;-) tfqueue sets up the event mechanism for a single file so you may want to look at that as well. > May be there is a better way to watch for the file to appear correctly? The way tail(1) does this seems pretty optimal to me. > Is inserting a sleep(3) in a loop an appropriate way? You could do this, but it’s less optimal than using kqueue/kevent because sleep(3) will wait longer than necessary in some cases and wake up sooner than required in others. It is basically a way to do polling which is always worse than event driven code. > > Below is the function how it is done in tail: It would have been better to cite the file name and line numbers, very easy with https://cvsweb.openbsd.org as I did above. There is also a mirror of the repo on Github, which also makes this sort of thing very easy: https://github.com/openbsd. E.g.: https://github.com/openbsd/src/blob/master/usr.bin/tail/forward.c#L224 The links to the repositories are right on the https://www.openbsd.org home page, so not hard to find at all. HTH Mike PS. Note that I am not an expert on kqueue/kevent programming. So followups for details on these functions would probably need to be answered by someone else.
how tail waits for file to appear again?
Hello,
Sorry if I chose the wrong place to ask such a question.
I have been learning C for a couple of months and along with reading
"C Primer Plus" by Stephen Prata and doing some exercises from it I took
a hard (for me) task to replicate a tail program in its simplest form.
I was able to reproduce watching for new data and truncation of the
file using kqueue but I do not quite understand how the original tail
watches when the file appears again after deletion or renaming.
By reading the original tail sources downloaded from OpenBSD mirror I
see that this is done by calling tfreopen function which seems to use a
"for" loop to (continuously?) stat(2) the file name till stat(2) successfully
returns and it does not seem to load a CPU as a simple continuous "for"
loop would do.
Can someone explain how it is done?
May be there is a better way to watch for the file to appear correctly?
Is inserting a sleep(3) in a loop an appropriate way?
Below is the function how it is done in tail:
"""
#define AFILESINCR 8
static const struct timespec *
tfreopen(struct tailfile *tf) {
static struct tailfile **reopen = NULL;
static intnfiles = 0, afiles = 0;
static const struct timespec ts = {1, 0};
struct stat sb;
struct tailfile **treopen, *ttf;
int i;
if (tf && !(tf->fp == stdin) &&
((stat(tf->fname, ) != 0) || sb.st_ino != tf->sb.st_ino)) {
if (afiles < ++nfiles) {
afiles += AFILESINCR;
treopen = reallocarray(reopen, afiles, sizeof(*reopen));
if (treopen)
reopen = treopen;
else
afiles -= AFILESINCR;
}
if (nfiles <= afiles) {
for (i = 0; i < nfiles - 1; i++)
if (strcmp(reopen[i]->fname, tf->fname) == 0)
break;
if (i < nfiles - 1)
nfiles--;
else
reopen[nfiles-1] = tf;
} else {
warnx("Lost track of %s", tf->fname);
nfiles--;
}
}
for (i = 0; i < nfiles; i++) {
ttf = reopen[i];
if (stat(ttf->fname, ) == -1)
continue;
if (sb.st_ino != ttf->sb.st_ino) {
(void) memcpy(&(ttf->sb), , sizeof(ttf->sb));
ttf->fp = freopen(ttf->fname, "r", ttf->fp);
if (ttf->fp == NULL)
ierr(ttf->fname);
else {
warnx("%s has been replaced, reopening.",
ttf->fname);
tfqueue(ttf);
}
}
reopen[i] = reopen[--nfiles];
}
return nfiles ? : NULL;
}
"""
--
Maksim Rodin
