kqueue and EVFILT_USER
I'm trying to "port" an application that currently works on FreeBSD and NetBSD, but it fails to compile on OpenBSD due to lack of EVFILT_USER in kqueue. Is there any plans to add support it for OpenBSD? --- Mika T. Lindqvist / Talleo Project
Kqueue priorities feature?
I recall tedu@ saying my that there was no reason for there to be multiple kqueues in a program. Well I beg to differ. I have a program with 3 kqueues because I want three distinct priorities on each worker process. The parent program initializes and fork()s a process that listens and passes internet sockets to one of the worker processes which is the last worker kqueue priority. I want the workers to operate upon the socket upload and download operations within the already received internet sockets (memory and limited stalling or latency permitting) before accepting new internet sockets. This is the second kqueue priority. There are a couple other fork()ed processes; I’m not going into the weeds into why (it’s dealing with a patented process dealing with multiple clients interacting on the same data in the server), all the worker processes are socket connected to the two other processes. The first process receives data from the workers and manages timeout periods and the second receives data from the first process and sends back data to be processed back to the workers to indicate that a socket transaction is complete or otherwise needs to be terminated and I want that procedure to be completed without interruption by other kqueue events. This is the first priority. To do this at this point, I have to have what I call “the backbone” to jump around between a bunch of different kevent calls to mediate all of these distinct kqueues where each priority must be exhausted before the next priority should be dealt with. At the very least the second and third priorities need to be distinct to prevent session memory from exploding from the creation of new connections while nothing gets done on existing sockets. I suspect this is something that would be very useful to other systems. Maybe my program is a unique case which doesn’t deserve any attention, but perhaps I’m not the only one who could benefit from this…eh?!!! Or is FreeBSD the OpenBSD upstream on kqueue/kevent ? By the way, whoever unlocked kevent in -current, AMAZING -- -Luke
kqueue(2) EVFILT_DEVICE inquiry
Hello, Misc; I've been playing around with the kqueue(2) syscall and have a question about the EVFILT_DEVICE filter; what does the the descriptor in the identifier refer to? I've tried opening device files in /dev/ however upon invoking kqueue it returns immediately with 0 in fflgas. I noticed in the release notes for 6.3 that it was added to make drm(4) work, so I'm not sure whether or not it's limited to that particular driver, or if I can use it with any other device; has anyone worked with EVFILT_DEVICE filter and have they gotten it to work for other devices? Thank you in advance. Ben Raskin
Re: [OpenBSD -current] Change event timer in main loop with kqueue
Moving to tech@.
On Fri, Feb 26, 2021 at 09:42:07PM +0100, martin mag wrote:
> I've been trying to use kqueue for the last couple of day but I keep
> having an issue with EVFILT_TIMER filter. (I'm running Openbsd
> -current)
>
> Right now, I'm trying to do the following:
> 1) Initilialize a timer event @ 200ms, periodically.
> 2) Inside the main event loop => If this event is retrieved, print
> elapsed time since last one
> 3) After 2 iterations, MODIFY the timer event to 1000ms and continue the loop
> 4) Code stops after 4 iterations as pb arise after the first timer
> change @ iteration 2.
>
> Reading the manpages kqueue(2), one sees that:
> ** ) An event is uniquely defined by the pair (ident, filter) ==>
> in the example below (TIMER1, EVFILT_TIMER)
> **) "" Re-adding an existing event will modify the parameters of
> the original event, and not result in a duplicate entry. "" => So
> re-adding the event (TIMER1, EVFILT_TIMER) with a modified field
> 'data' should update the timer from 200ms to 1000ms.
>
> => Apparently, timer is updated, but not in the way I expected.
The kernel does not reschedule the timer when the user changes the
timeout period. The new period will take effect only after the current
period has expired. This is not explained in the manual page, though.
With the recent kqueue changes, it is straightforward to make the kernel
modify an existing timer. I think the clearest behaviour is to reset the
timer completely when it is modified. If there are pending events, they
should be cancelled because they do not necessarily correspond to the
new settings.
When f_modify and f_process are present in kqread_filtops, filt_timer
is not used. filt_timerexpire() activates timer knotes directly using
knote_activate() instead of KNOTE().
However, the current behaviour has been around so long that one can
argue that it is an actual feature. BSDs are not consistent with this,
though. FreeBSD resets the timer immediately, whereas NetBSD and
DragonFly BSD apply the new period after expiry.
I guess the resetting is harmless in most cases but might wreak havoc
at least with software that keeps poking its timers before expiry.
Index: lib/libc/sys/kqueue.2
=======
RCS file: src/lib/libc/sys/kqueue.2,v
retrieving revision 1.43
diff -u -p -r1.43 kqueue.2
--- lib/libc/sys/kqueue.2 14 Nov 2020 10:16:15 - 1.43
+++ lib/libc/sys/kqueue.2 27 Feb 2021 12:54:27 -
@@ -468,6 +468,11 @@ contains the number of times the timeout
This filter automatically sets the
.Dv EV_CLEAR
flag internally.
+.Pp
+If an existing timer is re-added, the existing timer and related pending events
+will be cancelled.
+The timer will be re-started using the timeout period
+.Fa data .
.It Dv EVFILT_DEVICE
Takes a descriptor as the identifier and the events to watch for in
.Fa fflags ,
Index: sys/kern/kern_event.c
===
RCS file: src/sys/kern/kern_event.c,v
retrieving revision 1.161
diff -u -p -r1.161 kern_event.c
--- sys/kern/kern_event.c 24 Feb 2021 14:59:52 - 1.161
+++ sys/kern/kern_event.c 27 Feb 2021 12:54:27 -
@@ -135,7 +135,8 @@ int filt_fileattach(struct knote *kn);
void filt_timerexpire(void *knx);
intfilt_timerattach(struct knote *kn);
void filt_timerdetach(struct knote *kn);
-intfilt_timer(struct knote *kn, long hint);
+intfilt_timermodify(struct kevent *kev, struct knote *kn);
+intfilt_timerprocess(struct knote *kn, struct kevent *kev);
void filt_seltruedetach(struct knote *kn);
const struct filterops kqread_filtops = {
@@ -163,7 +164,9 @@ const struct filterops timer_filtops = {
.f_flags= 0,
.f_attach = filt_timerattach,
.f_detach = filt_timerdetach,
- .f_event= filt_timer,
+ .f_event= NULL,
+ .f_modify = filt_timermodify,
+ .f_process = filt_timerprocess,
};
struct pool knote_pool;
@@ -444,15 +447,48 @@ filt_timerdetach(struct knote *kn)
struct timeout *to;
to = (struct timeout *)kn->kn_hook;
- timeout_del(to);
+ timeout_del_barrier(to);
free(to, M_KEVENT, sizeof(*to));
kq_ntimeouts--;
}
int
-filt_timer(struct knote *kn, long hint)
+filt_timermodify(struct kevent *kev, struct knote *kn)
+{
+ struct timeout *to = kn->kn_hook;
+ int s;
+
+ /* Reset the timer. Any pending events are discarded. */
+
+ timeout_del_barrier(to);
+
+ s = splhigh();
+ if (kn->kn_status & KN_QUEUED)
+ knote_dequeue(kn);
+ kn->kn_status &= ~KN_ACTIVE;
+ splx(s);
+
+ kn->kn_data = 0;
+ knote_modify(kev, kn);
+ /* Reinit timeout to invoke tick adjustment again. */
+
[OpenBSD -current] Change event timer in main loop with kqueue
Hello everyone!
I've been trying to use kqueue for the last couple of day but I keep
having an issue with EVFILT_TIMER filter. (I'm running Openbsd
-current)
Right now, I'm trying to do the following:
1) Initilialize a timer event @ 200ms, periodically.
2) Inside the main event loop => If this event is retrieved, print
elapsed time since last one
3) After 2 iterations, MODIFY the timer event to 1000ms and continue the loop
4) Code stops after 4 iterations as pb arise after the first timer
change @ iteration 2.
Reading the manpages kqueue(2), one sees that:
** ) An event is uniquely defined by the pair (ident, filter) ==>
in the example below (TIMER1, EVFILT_TIMER)
**) "" Re-adding an existing event will modify the parameters of
the original event, and not result in a duplicate entry. "" => So
re-adding the event (TIMER1, EVFILT_TIMER) with a modified field
'data' should update the timer from 200ms to 1000ms.
=> Apparently, timer is updated, but not in the way I expected. See
below an example.
Here is the C program. I removed every 'error-checker' intentionally
as this is just a basic test:
#include
#include
#include
#include
#define TIMER1 202
int main(){
int kq=0, nev=0;
struct kevent evlist, chlist;
struct timespec start, stop, elapsed;
/* Initialize the queue */
kq = kqueue();
/* Register event to the queue */
EV_SET(&chlist, TIMER1, EVFILT_TIMER, EV_ADD | EV_ENABLE, 0, 200, 0);
kevent(kq, &chlist, 1, NULL, 0, NULL);
for (int i=0; i<4; i++){
clock_gettitme(CLOCK_MONOTONIC, &start);
nev = kevent(kq, NULL, 0, &evlist, 1, NULL);
printf("Iteration %d => nb events=%d\n", i+1, nev);
if (evlist.ident == TIMER1){
clock_gettime(CLOCK_MONOTONIC, &stop);
timespecsub(&stop, &start, &elapsed);
printf("Time elapsed since previous iteration: %lld.%09lds\n",
(long long) elapsed.tv_sec, (long long)
elapsed.tv_nsec);
/* > MODIFY TIMER <== */
if( (i+1)%2 == 0){
printf("Adjusting timer event ...\n");
EV_SET(&chlist, TIMER1, EVFILT_TIMER, EV_ADD |
EV_ENABLE, 0, 1000, 0);
/* I also tried this: chlist.data = 1000; but same
problem arise*/
/* Register modification within the queue */
kevent(kq, &chlist, 1, NULL, 0, NULL);
printf("Next event should happen %dms later", chlist.data);
} /* End i%4 == 0
} /* End evlist.ident == TIMER1 */
} /* End for loop */
return EXIT_SUCCESS;
}
*** Compiled with gcc-8.4.0
# egcc -o test_kqueue test_kqueue.c
*** OUTPUT of above program
Iteration 1 => nb events=1
Time elapsed since previous event:0.203417468s
==
Iteration 2 => nb events=1
Time elapsed since previous event:0.199534100s
Adjusting timer event
Next event in 1000ms<< ===
This is where TIMER is changed
<< === and kqueue is updated
==
Iteration 3 => nb events=1
Time elapsed since previous event:0.199848328<< === Problem here:
<< It should be ~1s not 0.2s (initial timer)
==
Iteration 4 => nb events=1
Time elapsed since previous event:0.999884957s << === Now it's OK
Adjusting timer event
Next event in 1000ms
==
*** END OF OUTPUT
So what I expected from my program was that Iteration 3 would be
retrieved 1second after iteration 2. But here, it is retrieved 0.2s
after only. This is AS IF the change wasn't taken into account yet
...? The expected behaviour is seen at iteration 4.
I'm pretty sure I'm not understanding correctly what happens but I
cannot figure out where I'm wrong in my example.
I did another test modifying the event timer (line 31 in program) in loop with:
TEST 1: (Added EV_ONESHOT)
EV_SET(&chlist, TIMER1, EVFILT_TIMER, EV_ADD | EV_ENABLE |
EV_ONESHOT, 0, 1000, 0); ===>> ONESHOT does not seem to be taken
into account as the event keeps beeing retrieved 1s apart. (the
expected behavior would that that only one event should be triggered
after this modification)
TEST 2 (Disabling event to see if it happens instantly or if it is
"delayed" as in the previous examples)
The ONLY change that work as expected is EV_DISABLE, which stops
events from being retrieved after iteration 2.
Could any one help me figure out what I'm doing wrong and how I can
manage modifying an existing timer event?
Thanks a lot!
PS: This is not a copy/paste program as I'm not sending the message
from the same PC. I hope I didn't do any typos rewritting
everything...
kqueue/kevent: EVFILT_TIMER inacurracy
Given the following code (timertest.c):
#include
#include
#include
#include
#include
#include
#include
#include
int
main(int argc, char *argv[])
{
long longtimeout, elapsed;
int kqfd;
struct timespec ts0, ts;
struct keventkev;
if (argc < 2)
errx(1, "missing number of seconds");
timeout = atoll(argv[1]) * 1000;
if (errno || timeout <= 0)
err(1, "invalid number of seconds");
if ((kqfd = kqueue()) == -1)
err(1, "kqueue");
EV_SET(&kev, , EVFILT_TIMER, EV_ADD, 0, timeout, NULL);
if (clock_gettime(CLOCK_REALTIME, &ts0) == -1)
err(1, "clock_gettime");
if (kevent(kqfd, &kev, 1, NULL, 0, NULL) == -1)
err(1, "kevent");
printf("timer [%lld msec] started ... ", timeout);
fflush(stdout);
memset(&kev, 0, sizeof(kev));
if (kevent(kqfd, NULL, 0, &kev, 1, NULL) == -1)
err(1, "kevent");
if (kev.flags & EV_ERROR)
errx(1, "event error (%llu)", kev.data);
if (clock_gettime(CLOCK_REALTIME, &ts) == -1)
err(1, "clock_gettime");
timespecsub(&ts, &ts0, &ts);
elapsed = ts.tv_sec * 1000 + ts.tv_nsec / 100L;
printf("done\ntime elapsed: %lld msec\n>>> drift: %lld msec\n",
elapsed, elapsed - timeout);
return (0);
}
# cc timertest.c -o timertest
On an older machine, there is a noticeable clock drift:
# ./timertest 600
timer [60 msec] started ... done
time elapsed: 624008 msec
>>> drift: 24008 msec
On an idle VM, the drift is below noise...
# ./timertest 600
timer [60 msec] started ... done
time elapsed: 68 msec
>>> drift: 8 msec
... the same VM under 50% CPU load:
# ./timertest 600
timer [60 msec] started ... done
time elapsed: 609704 msec
>>> drift: 9704 msec
Is it the way I'm using kevent/EVFILT_TIMER?
Any help is being appreciated.
kqueue EV_DISPATCH and EV_EOF interaction
Hello,
I've been re-writing the polling mechanisms in the Erlang VM and stumbled
across
something that might be a bug in the OpenBSD implementation of kqueue.
When using EV_DISPATCH, the event is never triggered again after the EV_EOF
flag has been delivered, even though there is more data to be read from the
socket.
I've attached a smallish program that shows the problem.
The shortened ktrace output looks like this on OpenBSD 6.2:
29672 a.out0.012883 CALL kevent(4,0x7f7e8220,1,0,0,0)
29672 a.out0.012888 STRU struct kevent { ident=5, filter=EVFILT_READ,
flags=0x81, fflags=0<>, data=0, udata=0x0 }
29672 a.out0.012895 RET kevent 0
29672 a.out0.012904 CALL kevent(4,0,0,0x7f7e7cf0,32,0)
29672 a.out0.013408 STRU struct kevent { ident=5, filter=EVFILT_READ,
flags=0x81, fflags=0<>, data=6, udata=0x0 }
29672 a.out0.013493 RET kevent 1
29672 a.out0.013548 CALL read(5,0x7f7e8286,0x2)
29672 a.out0.013562 RET read 2
29672 a.out0.013590 CALL kevent(4,0x7f7e8220,1,0,0,0)
29672 a.out0.013594 STRU struct kevent { ident=5, filter=EVFILT_READ,
flags=0x84, fflags=0<>, data=0, udata=0x0 }
29672 a.out0.013608 RET kevent 0
29672 a.out1.08 CALL kevent(4,0,0,0x7f7e7cf0,32,0)
29672 a.out1.022537 STRU struct kevent { ident=5, filter=EVFILT_READ,
flags=0x8081, fflags=0<>, data=4, udata=0x0 }
29672 a.out1.022572 RET kevent 1
29672 a.out1.022663 CALL read(5,0x7f7e8286,0x2)
29672 a.out1.022707 RET read 2
29672 a.out1.022816 CALL kevent(4,0x7f7e8220,1,0,0,0)
29672 a.out1.022822 STRU struct kevent { ident=5, filter=EVFILT_READ,
flags=0x84, fflags=0<>, data=0, udata=0x0 }
29672 a.out1.022835 RET kevent 0
29672 a.out2.032238 CALL kevent(4,0,0,0x7f7e7cf0,32,0)
29672 a.out5.277194 PSIG SIGINT SIG_DFL
In this example I would have expected the last kevent call to return with
EV_EOF and
data set to 2, but it does not trigger again. If I don't use EV_DISPATCH,
the event is
triggered again and the program terminates.
Does anyone know if this is the expected behavior or a bug?
I've worked around this issue by using EV_ONESHOT instead of EV_DISPATCH on
OpenBSD for now, but would like to use EV_DISPATCH in the future as I've
found
that it aligns better with the abstractions that I use, and could possibly
be a little bit
more performant.
Lukas
PS. If relevant, it seems like FreeBSD does behave the way that I expected,
i.e.
it triggers again for EV_DISPATCH after EV_EOF has been shown. DS.
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#define USE_DISPATCH 1
int main() {
struct addrinfo *addr;
struct addrinfo hints;
int kq, listen_s, fd = -1;
struct kevent evSet;
struct kevent evList[32];
/* open a TCP socket */
memset(&hints, 0, sizeof hints);
hints.ai_family = PF_UNSPEC; /* any supported protocol */
hints.ai_flags = AI_PASSIVE; /* result for bind() */
hints.ai_socktype = SOCK_STREAM; /* TCP */
int error = getaddrinfo ("127.0.0.1", "8080", &hints, &addr);
if (error)
errx(1, "getaddrinfo failed: %s", gai_strerror(error));
listen_s = socket(addr->ai_family, addr->ai_socktype, addr->ai_protocol);
if (setsockopt(listen_s, SOL_SOCKET, SO_REUSEADDR, &(int){ 1 }, sizeof(int)) < 0)
errx(1, "setsockopt(SO_REUSEADDR) failed");
bind(listen_s, addr->ai_addr, addr->ai_addrlen);
listen(listen_s, 5);
kq = kqueue();
system("echo -n abcdef | nc -v -w 1 127.0.0.1 8080 &");
EV_SET(&evSet, listen_s, EVFILT_READ, EV_ADD, 0, 0, NULL);
if (kevent(kq, &evSet, 1, NULL, 0, NULL) == -1)
err(1, "kevent");
while(1) {
int i;
int nev = kevent(kq, NULL, 0, evList, 32, NULL);
for (i = 0; i < nev; i++) {
if (evList[i].ident == listen_s) {
struct sockaddr_storage addr;
socklen_t socklen = sizeof(addr);
if (fd != -1)
close(fd);
fd = accept(evList[i].ident, (struct sockaddr *)&addr, &socklen);
printf("accepted %d\n", fd);
#if USE_DISPATCH
EV_SET(&evSet, fd, EVFILT_READ, EV_ADD|EV_DISPATCH, 0, 0, NULL);
#else
EV_SET(&evSet, fd, EVFILT_READ, EV_ADD, 0, 0, NULL);
#endif
if (kevent(kq, &evSet, 1, NULL, 0, NULL) == -1)
err(1, "kevent");
} else {
if (evList[i].flags & EV_EOF && evList[i].data == 0) {
printf("closing %d\n", fd);
close(fd);
fd = -1;
exit(0);
} else if (evList[i].filter ==
kqueue
Hi Folks, am i wrong or kqueue support for socket descriptor is limited to 2 ^ 16 open sockets ? Thanks in advance.
How do you use EV_DISPATCH in kqueue(2)
Is EV_DISPATCH somehow like EV_ONESHOT or EVDISABLE ? What is a use case? If you have an open socket file descriptor with a EVEFILT_READ, does it close the socket upon getting some data? I don't run current.
Re: kqueue
It looks like you will be limited to 4096 timers and to valid file descriptors that don't exceed INT_MAX. My guess is that if you need more, you could run another kqueue for more timers or different kevents on identical file descriptors. Otherwise, the man page says: kevent() returns the number of events placed in the eventlist, up to the value given by nevents. If an error occurs while processing an element of the changelist and there is enough room in the eventlist, then the event will be placed in the eventlist with EV_ERROR set in flags and the system error in data. Otherwise, -1 will be returned, and errno will be set to indicate the error condition.
Re: kqueue
On Tue, Apr 18, 2017 at 9:40 AM, Friedrich Locke wrote: > just started to play winth kqueue and i wonder what happens if i request to > add n events to the queue there is memory only for a subset of the events. > Will the subset be added or the whole operation will fail ? The former. Changes in the changelist are applied in order and are not 'unwound' if a later change can't be made. Indeed, if nevents > 0 then failure to perform a particular change will be reported via the eventlist and it'll continue to apply later changes in the list until it either reaches the end of the changelist or has no space in the eventlist to report an error. In the last case (no space to report a change error) it'll stop processing changelist entries and report the error by returning -1 with the error in errno. Philip Guenther
kqueue
I suspect that you will sooner run out of file descriptors. but I assume that if it runs into a problem, kevent() will return -1 and it may be unrecoverable. I suspect that it would first occur because the kernel is being overutilized. The information that is being created, I suspect, is being stored in the kernel. I may look into the source code and try to find out. You'd probably have to edit /etc/login.conf as root to allow enough file descriptors to be spawned in the first place. But maybe you could do it with EVFILT_TIMER calls, as they don't require a file descriptor as an 'ident'. I suspect that the whole kqueue is flushed if you exceed a specific level.
kqueue
Hi folks, just started to play winth kqueue and i wonder what happens if i request to add n events to the queue there is memory only for a subset of the events. Will the subset be added or the whole operation will fail ? Thanks in advance.
Re: sndio non-blocking with kqueue
On Wed, Feb 25, 2015 at 03:08:01AM +, Sam Good wrote: > >From sndio design, it looks like non-blocking (like in sio_write) can be > >done with poll. > sio_pollfd can be used to obtain the pollfd struct which contains the file > descriptor fd. > The fd descriptor seems to be equivalent to the unix stream for the sound > playing device. > > Would it be possible to use the file descriptor fd with kqueue > instead of poll? Not directly. It's a "private" file descriptor. The client code transmits meta-data (flow control information) that's hidden from the caller. For instance, the server may send timing information consumed internally by the library; we don't expose it with sio_read(). In this case, we don't want the POLLIN event to be set, as there are no samples to read. This is why the library exposes sio_pollfd() and sio_revents() instead of exposing the bare file descriptor(s). Typically sio_revents() consumes the meta-data and adjusts the POLLIN and POLLOUT flags. > (I wanted to adapt an application that uses kqueue for > internet/TCP server listening, and be able to add an audio > playing capability; I wanted to try to use kqueue for both. I > also wanted to learn more about using kqueue) > > I wrote a basic program that gets the sndio playing device file > descriptor fd from sio_pollfd, and then adds that fd to the > kevent with filter EVFILT_WRITE. Then loop with kevent appears to > return that the fd is ready for WRITE, and the kevent return data > shows '4096' (available space in buffer for write) but sio_write > returns with value 0, indicating that no bytes could be written > into the output buffer. If this is the socket fd, this is OK, as sio_write() and the sndio interface to poll(2) does flow control. They may be buffer space available locally, but the device may not be ready to accept it. > Is there something different about the file descriptor for the > sndio output device, that kqueue/kevent can't be used? > There's no interface to kqueue in the sndio api yet (i mean, there's no equivalent of sio_pollfd, sio_revents, etc) because we never needed it, probably because as programs use few descriptors, poll(2) did the job so far. Furthermore, as kqueue was never used for audio, I'm not sure that the corresponding audio(4) code was tested.
Re: sndio non-blocking with kqueue
Sam Good wrote: > From sndio design, it looks like non-blocking (like in sio_write) can be > done with poll. sio_pollfd can be used to obtain the pollfd struct which > contains the file descriptor fd. The fd descriptor seems to be equivalent > to the unix stream for the sound playing device. > > Would it be possible to use the file descriptor fd with kqueue instead of > poll? > > > (I wanted to adapt an application that uses kqueue for internet/TCP server > listening, and be able to add an audio playing capability; I wanted to try > to use kqueue for both. I also wanted to learn more about using kqueue) > > I wrote a basic program that gets the sndio playing device file descriptor > fd from sio_pollfd, and then adds that fd to the kevent with filter > EVFILT_WRITE. Then loop with kevent appears to return that the fd is ready > for WRITE, and the kevent return data shows '4096' (available space in > buffer for write) but sio_write returns with value 0, indicating that no > bytes could be written into the output buffer. > > > Is there something different about the file descriptor for the sndio > output device, that kqueue/kevent can't be used? > > thanks. Did you start the device before writing to it ? (read sio_start(3))
sndio non-blocking with kqueue
>From sndio design, it looks like non-blocking (like in sio_write) can be done >with poll. sio_pollfd can be used to obtain the pollfd struct which contains the file descriptor fd. The fd descriptor seems to be equivalent to the unix stream for the sound playing device. Would it be possible to use the file descriptor fd with kqueue instead of poll? (I wanted to adapt an application that uses kqueue for internet/TCP server listening, and be able to add an audio playing capability; I wanted to try to use kqueue for both. I also wanted to learn more about using kqueue) I wrote a basic program that gets the sndio playing device file descriptor fd from sio_pollfd, and then adds that fd to the kevent with filter EVFILT_WRITE. Then loop with kevent appears to return that the fd is ready for WRITE, and the kevent return data shows '4096' (available space in buffer for write) but sio_write returns with value 0, indicating that no bytes could be written into the output buffer. Is there something different about the file descriptor for the sndio output device, that kqueue/kevent can't be used? thanks.
Re: Programming Question: using kqueue() and kevent()
(First, sorry if anybody got a response earlier; my net connection dropped out in the middle of sending the message and I'm pretty sure the mail did not make it through.) On Mon, Apr 20, 2009 at 08:31:44AM -0700, patrick keshishian wrote: > I've not used kqueue but first (quick) glance at the man page > indicates that you look at the return value from kevent() and then > iterate over that value (if not -1) and process each eventlist. Oops, yeah that's a stupid mistake, kevent() would definitely get called too many times. > > ?? ?? ?? ??changelist[i].ident = i; > > > > Pretty sure this line is your problem. > > the man page agrees with above statement. Read section about EVFILT_READ > filter. Ah, that is indeed the problem. I don't know why I thought that I should be shoving the file descriptor into the user field, especially since the manual page clearly says otherwise. Must have somehow got it into my head that ident was supposed to be a unique changelist or event identifier. With this fixed everything is working. Thanks much to both you and Art for taking the time to look over this, I appreciate it. -- Taylor Christopher Venable http://real.metasyntax.net:2357/
Re: Programming Question: using kqueue() and kevent()
This looks not good:
for (i = 0; i < kevent(kq, NULL, 0, eventlist, EVENT_COUNT,
&ts_five_sec); i += 1) {
fprintf(stderr, ">> FOUND A KEVENT\n");
line = (char *)calloc(eventlist[i].data + 1, sizeof(char));
recv(sockfd[eventlist[i].ident], line, eventlist[i].data, 0);
fprintf(stderr, "MSG = %s\n", line);
free(line);
}
I've not used kqueue but first (quick) glance at the man page
indicates that you look at the return value from kevent() and then
iterate over that value (if not -1) and process each eventlist.
On Mon, Apr 20, 2009 at 5:11 AM, Artur Grabowski wrote:
> Taylor Venable writes:
>
>> The plain-text version is here:
>> B B http://real.metasyntax.net:2357/tmp/kevent.c
>
> B B B B changelist[i].ident = i;
>
> Pretty sure this line is your problem.
the man page agrees with above statement. Read section about EVFILT_READ
filter.
--patrick
Re: Programming Question: using kqueue() and kevent()
Taylor Venable writes: > The plain-text version is here: > http://real.metasyntax.net:2357/tmp/kevent.c changelist[i].ident = i; Pretty sure this line is your problem. //art
Re: Programming Question: using kqueue() and kevent()
On Mon, Apr 20, 2009 at 01:53:13PM +0200, Pierre-Yves Ritschard wrote: > * Taylor Venable (tay...@metasyntax.net) wrote: > > I'm looking at using kqueue() and kevent() for some high-load > > client-side socket work, writing part of a stress testing system for > > our product at work. I've got an example that I put together, using > > the read filter on the socket file descriptors, but kevent() doesn't > > tell me that any data is available to be read. There is data there, > > though, so I must have done something wrong in setup. If somebody > > could take a look and tell me what I'm doing wrong, I'd really > > appreciate it. > > > > The plain-text version is here: > > http://real.metasyntax.net:2357/tmp/kevent.c > > > > Or if you like some HTML coloring: > > http://real.metasyntax.net:2357/tmp/kevent_c.html > > > > I'm doing this on OpenBSD 4.5 BETA currently, while waiting for my CD > > set to arrive. :) > > > > Thanks, > > > > The way we do it is through libevent, it's reliable, simple and > portable. > You were faster than me, damn :-) -- Gilles Chehade http://www.poolp.org/~gilles/
Re: Programming Question: using kqueue() and kevent()
* Taylor Venable (tay...@metasyntax.net) wrote: > I'm looking at using kqueue() and kevent() for some high-load > client-side socket work, writing part of a stress testing system for > our product at work. I've got an example that I put together, using > the read filter on the socket file descriptors, but kevent() doesn't > tell me that any data is available to be read. There is data there, > though, so I must have done something wrong in setup. If somebody > could take a look and tell me what I'm doing wrong, I'd really > appreciate it. > > The plain-text version is here: > http://real.metasyntax.net:2357/tmp/kevent.c > > Or if you like some HTML coloring: > http://real.metasyntax.net:2357/tmp/kevent_c.html > > I'm doing this on OpenBSD 4.5 BETA currently, while waiting for my CD > set to arrive. :) > > Thanks, > The way we do it is through libevent, it's reliable, simple and portable.
Programming Question: using kqueue() and kevent()
I'm looking at using kqueue() and kevent() for some high-load client-side socket work, writing part of a stress testing system for our product at work. I've got an example that I put together, using the read filter on the socket file descriptors, but kevent() doesn't tell me that any data is available to be read. There is data there, though, so I must have done something wrong in setup. If somebody could take a look and tell me what I'm doing wrong, I'd really appreciate it. The plain-text version is here: http://real.metasyntax.net:2357/tmp/kevent.c Or if you like some HTML coloring: http://real.metasyntax.net:2357/tmp/kevent_c.html I'm doing this on OpenBSD 4.5 BETA currently, while waiting for my CD set to arrive. :) Thanks, -- Taylor Christopher Venable http://real.metasyntax.net:2357/
Re: poll(2) vs kqueue(2) performance
I found the doxygen docs far less useful than the man page. On Apr 23, 2008, at 10:31 PM, "Niels Provos" <[EMAIL PROTECTED]> wrote: On Mon, Apr 21, 2008 at 2:05 PM, Gilles Chehade <[EMAIL PROTECTED]> wrote: Yay, I too fell in love with it and it's various API's despite the lack of documentation for most of them, header help understanding how things work but I wasted quite some time on bufferevents ;-) The documentation has actually become much better over time: http://www.monkey.org/~provos/libevent/doxygen-1.4.3/ However, I'd be happy to see any patches to improve the documentation. Thanks, Niels.
Re: poll(2) vs kqueue(2) performance
On Mon, Apr 21, 2008 at 2:05 PM, Gilles Chehade <[EMAIL PROTECTED]> wrote: > Yay, I too fell in love with it and it's various API's despite the lack > of documentation for most of them, header help understanding how things > work but I wasted quite some time on bufferevents ;-) The documentation has actually become much better over time: http://www.monkey.org/~provos/libevent/doxygen-1.4.3/ However, I'd be happy to see any patches to improve the documentation. Thanks, Niels.
Re: poll(2) vs kqueue(2) performance
On Mon, Apr 21, 2008 at 09:43:43AM -0500, Marco Peereboom wrote: > On Mon, Apr 21, 2008 at 02:31:26PM +0200, Henning Brauer wrote: > > programming w/ libevent is convenient at times, the decision poll vs > > libevent should not be made based on performance considerations, > > exception beeing the above massive concurrent connection case. > > I spent a lot of time doing libevent stuff for a work project the last > few weeks and I am in love with the API. One negative is the poor > documentation; especially the buffered events docs are inadequate. > > Yes, I read the source :-) but it would have been a little less painful > to have a slightly better man page and some standalone examples. > > Negatives aside once you figure it out it allows one to write code that > doesn't require threading and other complexity inducing stuff. Yay > finite state machines! > Yay, I too fell in love with it and it's various API's despite the lack of documentation for most of them, header help understanding how things work but I wasted quite some time on bufferevents ;-) Gilles -- Gilles Chehade http://www.poolp.org/
Re: poll(2) vs kqueue(2) performance
On Mon, Apr 21, 2008 at 02:31:26PM +0200, Henning Brauer wrote: > programming w/ libevent is convenient at times, the decision poll vs > libevent should not be made based on performance considerations, > exception beeing the above massive concurrent connection case. I spent a lot of time doing libevent stuff for a work project the last few weeks and I am in love with the API. One negative is the poor documentation; especially the buffered events docs are inadequate. Yes, I read the source :-) but it would have been a little less painful to have a slightly better man page and some standalone examples. Negatives aside once you figure it out it allows one to write code that doesn't require threading and other complexity inducing stuff. Yay finite state machines!
Re: poll(2) vs kqueue(2) performance
Marc, Henning, thank you for the insight. On Mon, Apr 21, 2008 at 8:38 PM, Marc Espie <[EMAIL PROTECTED]> wrote: > > On Sat, Apr 19, 2008 at 11:43:20AM +0200, Jonathan Schleifer wrote: > > "Edwin Eyan Moragas" <[EMAIL PROTECTED]> wrote: > > > > > the question is, which one is more useful when writing new servers? > > > kqueue or poll? > > > > poll is more portable, while kqueue should be more performant (at > > least, that's why it was invented). If your app only needs to run on > > OpenBSD, NetBSD and FreeBSD, you're just fine with kqueue, otherwise > > use poll. Generally, I think it's better to use poll and sacrifice that > > unnoticable performance gain. > > As usual, depends what you want to do. > > poll() and select() give you control over file descriptors. kqueue > encompasses more events. It's not especially faster, it just leads to > simpler code in case you need the supplementary events. > > If all you need to do is watch over a set of file descriptors, poll > and select are the simplest ones to use... and the most portable. > > In many, many cases, poll() is better. The only case where select comes > close is when you want to watch over most of your file descriptors (because > you access less memory in such a case). > > And then, you should profile. I'm not even sure it makes a difference. > > Most of the places in the system where we have select() are legacies: it > it's not broken, don't fix it. > -- garnet:jasmin:beryllium:gluon 90-12264 90-B
Re: poll(2) vs kqueue(2) performance
On Sat, Apr 19, 2008 at 11:43:20AM +0200, Jonathan Schleifer wrote: > "Edwin Eyan Moragas" <[EMAIL PROTECTED]> wrote: > > > the question is, which one is more useful when writing new servers? > > kqueue or poll? > > poll is more portable, while kqueue should be more performant (at > least, that's why it was invented). If your app only needs to run on > OpenBSD, NetBSD and FreeBSD, you're just fine with kqueue, otherwise > use poll. Generally, I think it's better to use poll and sacrifice that > unnoticable performance gain. As usual, depends what you want to do. poll() and select() give you control over file descriptors. kqueue encompasses more events. It's not especially faster, it just leads to simpler code in case you need the supplementary events. If all you need to do is watch over a set of file descriptors, poll and select are the simplest ones to use... and the most portable. In many, many cases, poll() is better. The only case where select comes close is when you want to watch over most of your file descriptors (because you access less memory in such a case). And then, you should profile. I'm not even sure it makes a difference. Most of the places in the system where we have select() are legacies: it it's not broken, don't fix it.
Re: poll(2) vs kqueue(2) performance
* Edwin Eyan Moragas <[EMAIL PROTECTED]> [2008-04-19 07:34]: > been reading the select(2) man pages and it mentions poll(2) > being more efficient in most cases. this makes it obvious to > discard the use of select(2) in writing new servers. yes. poll is the way better API, easier to use, easier kernel-side, avoids a lot of problems (common!) improper use of select() brings, ... > i've come across some performance benchmarks which is trying > to use kqueue(2). using kqueue directly is painful. I'd recommend libevent if you really want to go that route. That said, kqueue really only pays out when you have _lots_ of concurrent connections, say, >1. > the question is, which one is more useful when writing new servers? > kqueue or poll? programming w/ libevent is convenient at times, the decision poll vs libevent should not be made based on performance considerations, exception beeing the above massive concurrent connection case. -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
Re: poll(2) vs kqueue(2) performance
On Sat, Apr 19, 2008 at 5:43 PM, Jonathan Schleifer <[EMAIL PROTECTED]> wrote: > "Edwin Eyan Moragas" <[EMAIL PROTECTED]> wrote: > > > > the question is, which one is more useful when writing new servers? > > kqueue or poll? > > poll is more portable, while kqueue should be more performant (at > least, that's why it was invented). If your app only needs to run on > OpenBSD, NetBSD and FreeBSD, you're just fine with kqueue, otherwise > use poll. Generally, I think it's better to use poll and sacrifice that > unnoticable performance gain. thank you Jonathan. it seems poll's the way to go. > > -- > Jonathan > -- garnet:jasmin:beryllium:gluon 90-12264 90-B
Re: poll(2) vs kqueue(2) performance
"Edwin Eyan Moragas" <[EMAIL PROTECTED]> wrote: > the question is, which one is more useful when writing new servers? > kqueue or poll? poll is more portable, while kqueue should be more performant (at least, that's why it was invented). If your app only needs to run on OpenBSD, NetBSD and FreeBSD, you're just fine with kqueue, otherwise use poll. Generally, I think it's better to use poll and sacrifice that unnoticable performance gain. -- Jonathan
Re: poll(2) vs kqueue(2) performance
Hi Eric, On Sat, Apr 19, 2008 at 4:17 PM, Eric Faurot <[EMAIL PROTECTED]> wrote: > > the question is, which one is more useful when writing new servers? > > kqueue or poll? > > The more useful is event(3). i've been looking also at libevent and libev, both of which are excellent libraries. however, i'm more interested in simpler system calls rather than the libraries. thank you for pointing this out. interesting that openbsd has libevent as a standard library. > > Eric. > -- garnet:jasmin:beryllium:gluon 90-12264 90-B
Re: poll(2) vs kqueue(2) performance
On Sat, 19 Apr 2008 13:27:34 +0800 "Edwin Eyan Moragas" <[EMAIL PROTECTED]> wrote: > Hi all, > > been reading the select(2) man pages and it mentions poll(2) > being more efficient in most cases. this makes it obvious to > discard the use of select(2) in writing new servers. > > i've come across some performance benchmarks which is trying > to use kqueue(2). > > the question is, which one is more useful when writing new servers? > kqueue or poll? The more useful is event(3). Eric.
Re: poll(2) vs kqueue(2) performance
thank you, Theo. On Sat, Apr 19, 2008 at 1:50 PM, Theo de Raadt <[EMAIL PROTECTED]> wrote: > > been reading the select(2) man pages and it mentions poll(2) > > being more efficient in most cases. this makes it obvious to > > discard the use of select(2) in writing new servers. > > select requires that you set up a bit array correctly. but often > people just use a fd_set, and cause a variety of strange buffer > overflow cases as soon as their fd's happen to be greater than the bit > size of the fd_set. > > the kernel has to iterate over these bit arrays a few times. > > for everyone involved, poll is just plain cheaper. > > finally, go look at the latest commit to lib/libc/net/res_send.c to > see how much easier poll() is to use. > > > > i've come across some performance benchmarks which is trying > > to use kqueue(2). > > shrug. performance is only a small part of the whole. > > > > the question is, which one is more useful when writing new servers? > > kqueue or poll? > > use poll. it is easier to use -- the behaviours are less surprising. > it is also much more portable. everyone has select and poll, and > quite honestly poll() is a better select(), even if it came out of > AT&T. > poll it is. again, many thanks. -- garnet:jasmin:beryllium:gluon 90-12264 90-B
Re: poll(2) vs kqueue(2) performance
> been reading the select(2) man pages and it mentions poll(2) > being more efficient in most cases. this makes it obvious to > discard the use of select(2) in writing new servers. select requires that you set up a bit array correctly. but often people just use a fd_set, and cause a variety of strange buffer overflow cases as soon as their fd's happen to be greater than the bit size of the fd_set. the kernel has to iterate over these bit arrays a few times. for everyone involved, poll is just plain cheaper. finally, go look at the latest commit to lib/libc/net/res_send.c to see how much easier poll() is to use. > i've come across some performance benchmarks which is trying > to use kqueue(2). shrug. performance is only a small part of the whole. > the question is, which one is more useful when writing new servers? > kqueue or poll? use poll. it is easier to use -- the behaviours are less surprising. it is also much more portable. everyone has select and poll, and quite honestly poll() is a better select(), even if it came out of AT&T.
poll(2) vs kqueue(2) performance
Hi all, been reading the select(2) man pages and it mentions poll(2) being more efficient in most cases. this makes it obvious to discard the use of select(2) in writing new servers. i've come across some performance benchmarks which is trying to use kqueue(2). the question is, which one is more useful when writing new servers? kqueue or poll? -- garnet:jasmin:beryllium:gluon 90-12264 90-B
Re: kqueue and determine if a file is still open
On 1/18/06, Stephan Leemburg <[EMAIL PROTECTED]> wrote: > I'm using kqueue events to trigger certain programs. The programs run > when a file is dropped into a certain directory. To avoid race > conditions, I want to know if the newly created or modified files are > still open by a process, so I can do what I need to do when no-one > else is possibly working on the file. Is there a way, other than > repeating the tricks in lsof, to find out if a file is still open > from within a c program? and if somebody opens the file after you determine it's not open? you're solving the problem the wrong way. don't put the file in the directory until you are done with it.
kqueue and determine if a file is still open
I'm using kqueue events to trigger certain programs. The programs run when a file is dropped into a certain directory. To avoid race conditions, I want to know if the newly created or modified files are still open by a process, so I can do what I need to do when no-one else is possibly working on the file. Is there a way, other than repeating the tricks in lsof, to find out if a file is still open from within a c program? If not would a systemcall like sys_isopen(path, O_WRONLY), where the call returns the or-ed value of the openmodes and-ed with the actual parameter, not be usefull? -- Stephan
