Re: [Chicken-users] Pipe and thread problem
On Thu, Oct 25, 2012 at 04:12:50AM -0400, Felix wrote: > From: Moritz Heidkamp > Subject: Re: [Chicken-users] Pipe and thread problem > Date: Wed, 24 Oct 2012 23:10:20 +0200 > > > Hey Chickeneers, > > > > out of curiosity I implemented (probably horribly inefficitent) > > thread-aware replacements for open-input-file* and open-output-file* > > (see below). While implementing them I came across something that I > > didn't quite understand: file-read and file-write raise errors whenever > > the underlying syscall returns -1. However, in this case I want to > > handle some error situations (i.e. EINTR, EAGAIN and EWOULDBLOCK). The > > only way I could find to do this is by dispatching on (errno) in the > > excpetion handler. However it looks like (errno) is not thread-safe so > > it might actually return a different value when my handler is run. Is > > this observation correct? Perhaps we should mention this in errno's > > documentation? Also, to make it possible to handle these errors I > > suggest to add the respective errno to the condition object. This would > > be thread-safe as the posix unit declares disable-interrupts. Thoughts? > > No, "errno" is not thread-safe. Posix-related errors (see "posix-error" > in "posix-common.scm") get attached the error-string (obtained via > strerror(3)), also adding errno would be fine, I'd say. > > I'm quite sure "##sys#custom-input-port" and > "##sys#custom-output-port" in "posixunix.scm" are alreay doing what > you want, and much more efficiently. I tried changing the program to use "##sys#custom-input-port", and the program still froze. Here is the version with "##sys#custom-input-port" included: (use srfi-18) (use posix) (define (produce writer) (thread-start! (lambda () (print "producer started") (let loop ((i 0)) (display (conc "hello" i " ") writer) (newline writer) (flush-output writer) (thread-sleep! 1) (loop (+ i 1)) (define (consume reader) (thread-start! (lambda () (print "consumer started") (let loop () (print (read-line reader)) (loop) (define (stream-test in-fd out-fd) (let ((reader (##sys#custom-input-port "reader" "reader" in-fd)) (writer (##sys#custom-output-port "writer" "writer" out-fd))) (produce writer) (thread-join! (consume reader (call-with-values create-pipe stream-test) I tried the same program, but with Moritz's changes, and they seemed to work. The code is kind of long, so I posted it here: https://gist.github.com/3953900 -- Aaron Patterson http://tenderlovemaking.com/ ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
From: Moritz Heidkamp Subject: Re: [Chicken-users] Pipe and thread problem Date: Wed, 24 Oct 2012 23:10:20 +0200 > Hey Chickeneers, > > out of curiosity I implemented (probably horribly inefficitent) > thread-aware replacements for open-input-file* and open-output-file* > (see below). While implementing them I came across something that I > didn't quite understand: file-read and file-write raise errors whenever > the underlying syscall returns -1. However, in this case I want to > handle some error situations (i.e. EINTR, EAGAIN and EWOULDBLOCK). The > only way I could find to do this is by dispatching on (errno) in the > excpetion handler. However it looks like (errno) is not thread-safe so > it might actually return a different value when my handler is run. Is > this observation correct? Perhaps we should mention this in errno's > documentation? Also, to make it possible to handle these errors I > suggest to add the respective errno to the condition object. This would > be thread-safe as the posix unit declares disable-interrupts. Thoughts? No, "errno" is not thread-safe. Posix-related errors (see "posix-error" in "posix-common.scm") get attached the error-string (obtained via strerror(3)), also adding errno would be fine, I'd say. I'm quite sure "##sys#custom-input-port" and "##sys#custom-output-port" in "posixunix.scm" are alreay doing what you want, and much more efficiently. cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
Hey Chickeneers, out of curiosity I implemented (probably horribly inefficitent) thread-aware replacements for open-input-file* and open-output-file* (see below). While implementing them I came across something that I didn't quite understand: file-read and file-write raise errors whenever the underlying syscall returns -1. However, in this case I want to handle some error situations (i.e. EINTR, EAGAIN and EWOULDBLOCK). The only way I could find to do this is by dispatching on (errno) in the excpetion handler. However it looks like (errno) is not thread-safe so it might actually return a different value when my handler is run. Is this observation correct? Perhaps we should mention this in errno's documentation? Also, to make it possible to handle these errors I suggest to add the respective errno to the condition object. This would be thread-safe as the posix unit declares disable-interrupts. Thoughts? OK, here we go. It seems to work with your program, Aaron: (define (open-input-fd fd) (make-input-port (lambda () (let loop () (thread-wait-for-i/o! fd #:input) (or (condition-case (string-ref (car (file-read fd 1)) 0) (exn (exn i/o file) (select (errno) ((errno/intr errno/again errno/wouldblock) #f) (else (signal exn) (loop (lambda () (nth-value 0 (file-select fd #f 0))) (lambda () (file-close fd (define (open-output-fd fd) (let ((port (open-output-file* fd))) (make-output-port (lambda (buffer) (let loop ((buffer buffer) (len (string-length buffer))) (thread-wait-for-i/o! fd #:output) (let ((written (condition-case (file-write fd buffer) (exn (exn i/o file) (select (errno) ((errno/intr errno/again errno/wouldblock) #f) (else (signal exn))) (if written (unless (= written len) (loop (substring buffer written len) (- len written))) (loop buffer len) (lambda () (close-output-port port)) (lambda () (flush-output port) Oh yeah, and note the lame solution of opening the output FD with open-output-file* -- this is only because there is no flush-file in posix. Perhaps worth adding? Moritz Aaron Patterson writes: > On Wed, Oct 24, 2012 at 08:04:37AM -0400, Felix wrote: >> From: Aaron Patterson >> Subject: [Chicken-users] Pipe and thread problem >> Date: Tue, 23 Oct 2012 14:30:57 -0700 >> >> > Hi, I'm trying to simulate reading from a TTY that writes every two >> > seconds. I want to do this with a pipe and two threads, one thread >> > writes every N seconds, while the other reads any data available on the >> > pipe. >> > >> > Unfortunately, my code just hangs. After speaking with the fine people >> > in #chicken, it seems that this may be a bug. We played with different >> > calls to put the threads to sleep, and different functions to read data, >> > but they all ended up freezing at some point. >> >> The ports obtained from calls to "open-[input|output]-file*" use >> internally the "stream-port" class (ports on FILE* streams, in this >> case created via fdopen(3)). These are not thread-aware, AFAICT. Ports >> created for socket-fd's (tcp.scm) and processes (posixunix.scm, see >> specifically ##sys#custom-[input|output]-port") apparently do. > > Is there a different way I should be opening the pipe so that it is > thread aware? ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
On Wed, Oct 24, 2012 at 08:04:37AM -0400, Felix wrote: > From: Aaron Patterson > Subject: [Chicken-users] Pipe and thread problem > Date: Tue, 23 Oct 2012 14:30:57 -0700 > > > Hi, I'm trying to simulate reading from a TTY that writes every two > > seconds. I want to do this with a pipe and two threads, one thread > > writes every N seconds, while the other reads any data available on the > > pipe. > > > > Unfortunately, my code just hangs. After speaking with the fine people > > in #chicken, it seems that this may be a bug. We played with different > > calls to put the threads to sleep, and different functions to read data, > > but they all ended up freezing at some point. > > The ports obtained from calls to "open-[input|output]-file*" use > internally the "stream-port" class (ports on FILE* streams, in this > case created via fdopen(3)). These are not thread-aware, AFAICT. Ports > created for socket-fd's (tcp.scm) and processes (posixunix.scm, see > specifically ##sys#custom-[input|output]-port") apparently do. Is there a different way I should be opening the pipe so that it is thread aware? -- Aaron Patterson http://tenderlovemaking.com/ ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
Without having seen the code in question: my guess would be that those FILE*-ports would do i/o-buffering. flush-output might already help. ??? On Oct 24 2012, Felix wrote: From: Mario Domenech Goulart Subject: Re: [Chicken-users] Pipe and thread problem Date: Wed, 24 Oct 2012 09:44:37 -0400 Hi Felix, On Wed, 24 Oct 2012 08:04:37 -0400 (EDT) Felix wrote: From: Aaron Patterson Subject: [Chicken-users] Pipe and thread problem Date: Tue, 23 Oct 2012 14:30:57 -0700 Hi, I'm trying to simulate reading from a TTY that writes every two seconds. I want to do this with a pipe and two threads, one thread writes every N seconds, while the other reads any data available on the pipe. Unfortunately, my code just hangs. After speaking with the fine people in #chicken, it seems that this may be a bug. We played with different calls to put the threads to sleep, and different functions to read data, but they all ended up freezing at some point. The ports obtained from calls to "open-[input|output]-file*" use internally the "stream-port" class (ports on FILE* streams, in this case created via fdopen(3)). These are not thread-aware, AFAICT. Ports created for socket-fd's (tcp.scm) and processes (posixunix.scm, see specifically ##sys#custom-[input|output]-port") apparently do. From a user perspective (i.e., without knowing how things are implemented by Chicken), how can we know what procedures can be used with threads? Only from the documentation. I'm not sure I understand the question. It also depends on how likely blocking is. Wouldn't such a misuse of I/O operations in the presence of threads be a possible cause of #858? #858 refers to socket ports, which are implemented differently from the ports that refer to files. Note that spiffy also involves sendfile, which _does_ non-blocking I/O for files. I think. cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-us ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
From: Mario Domenech Goulart Subject: Re: [Chicken-users] Pipe and thread problem Date: Wed, 24 Oct 2012 09:44:37 -0400 > Hi Felix, > > On Wed, 24 Oct 2012 08:04:37 -0400 (EDT) Felix > wrote: > >> From: Aaron Patterson >> Subject: [Chicken-users] Pipe and thread problem >> Date: Tue, 23 Oct 2012 14:30:57 -0700 >> >>> Hi, I'm trying to simulate reading from a TTY that writes every two >>> seconds. I want to do this with a pipe and two threads, one thread >>> writes every N seconds, while the other reads any data available on the >>> pipe. >>> >>> Unfortunately, my code just hangs. After speaking with the fine people >>> in #chicken, it seems that this may be a bug. We played with different >>> calls to put the threads to sleep, and different functions to read data, >>> but they all ended up freezing at some point. >> >> The ports obtained from calls to "open-[input|output]-file*" use >> internally the "stream-port" class (ports on FILE* streams, in this >> case created via fdopen(3)). These are not thread-aware, AFAICT. Ports >> created for socket-fd's (tcp.scm) and processes (posixunix.scm, see >> specifically ##sys#custom-[input|output]-port") apparently do. > > From a user perspective (i.e., without knowing how things are > implemented by Chicken), how can we know what procedures can be used > with threads? Only from the documentation. I'm not sure I understand the question. It also depends on how likely blocking is. > > Wouldn't such a misuse of I/O operations in the presence of threads be > a possible cause of #858? #858 refers to socket ports, which are implemented differently from the ports that refer to files. Note that spiffy also involves sendfile, which _does_ non-blocking I/O for files. I think. cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
Hi Felix, On Wed, 24 Oct 2012 08:04:37 -0400 (EDT) Felix wrote: > From: Aaron Patterson > Subject: [Chicken-users] Pipe and thread problem > Date: Tue, 23 Oct 2012 14:30:57 -0700 > >> Hi, I'm trying to simulate reading from a TTY that writes every two >> seconds. I want to do this with a pipe and two threads, one thread >> writes every N seconds, while the other reads any data available on the >> pipe. >> >> Unfortunately, my code just hangs. After speaking with the fine people >> in #chicken, it seems that this may be a bug. We played with different >> calls to put the threads to sleep, and different functions to read data, >> but they all ended up freezing at some point. > > The ports obtained from calls to "open-[input|output]-file*" use > internally the "stream-port" class (ports on FILE* streams, in this > case created via fdopen(3)). These are not thread-aware, AFAICT. Ports > created for socket-fd's (tcp.scm) and processes (posixunix.scm, see > specifically ##sys#custom-[input|output]-port") apparently do. >From a user perspective (i.e., without knowing how things are implemented by Chicken), how can we know what procedures can be used with threads? Wouldn't such a misuse of I/O operations in the presence of threads be a possible cause of #858? Best wishes. Mario -- http://parenteses.org/mario ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
From: Aaron Patterson Subject: [Chicken-users] Pipe and thread problem Date: Tue, 23 Oct 2012 14:30:57 -0700 > Hi, I'm trying to simulate reading from a TTY that writes every two > seconds. I want to do this with a pipe and two threads, one thread > writes every N seconds, while the other reads any data available on the > pipe. > > Unfortunately, my code just hangs. After speaking with the fine people > in #chicken, it seems that this may be a bug. We played with different > calls to put the threads to sleep, and different functions to read data, > but they all ended up freezing at some point. The ports obtained from calls to "open-[input|output]-file*" use internally the "stream-port" class (ports on FILE* streams, in this case created via fdopen(3)). These are not thread-aware, AFAICT. Ports created for socket-fd's (tcp.scm) and processes (posixunix.scm, see specifically ##sys#custom-[input|output]-port") apparently do. cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
On Wed, Oct 24, 2012 at 09:32:35AM +0200, Peter Bex wrote: > On Wed, Oct 24, 2012 at 03:21:15AM -0400, Felix wrote: > > > Anyone with a better understanding of Chicken's internals able to > > > comment on which cases poll() is called and how the timeout value is > > > selected? > > > > poll(2) is not used in the runtime system. > > I think some libcs might implement select() in terms of poll? > In that case, poll is what you'd see in a syscall trace. > That's correct, libpthread on (at least my) version of OpenBSD implements select in terms of poll. -Alan -- .i ma'a lo bradi cu penmi gi'e du ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
On Wed, Oct 24, 2012 at 03:21:15AM -0400, Felix wrote: > > Anyone with a better understanding of Chicken's internals able to > > comment on which cases poll() is called and how the timeout value is > > selected? > > poll(2) is not used in the runtime system. I think some libcs might implement select() in terms of poll? In that case, poll is what you'd see in a syscall trace. Cheers, Peter -- http://sjamaan.ath.cx -- "The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music." -- Donald Knuth ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
From: Alan Post Subject: Re: [Chicken-users] Pipe and thread problem Date: Tue, 23 Oct 2012 16:14:17 -0600 > Anyone with a better understanding of Chicken's internals able to > comment on which cases poll() is called and how the timeout value is > selected? poll(2) is not used in the runtime system. cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Pipe and thread problem
For me, that program is blocking on a call to poll(), which must be something that is happening in chicken's threading code. It is polling on two file descriptors with an infinite timeout, and clearly never coming back up for air. the produce thread is hanging on the thread-sleep call, and the consume thread is hanging on the read-line call. This means (and I confirm in the ktrace) that your pipe is being written to. If I add a (newline writer) between your write and your flush, read-line finishes once (I'd expect that) and then blocks again on read-line, but the writing thread never wakes up. I'm not sure why a (thread-sleep! 1) would ever cause poll to be called with an infinite timeout, but a quick analysis shows that some case is not waking up produce and you get to a point where both ends of the pipe are waiting for the other one. Anyone with a better understanding of Chicken's internals able to comment on which cases poll() is called and how the timeout value is selected? -Alan On Tue, Oct 23, 2012 at 02:30:57PM -0700, Aaron Patterson wrote: > Hi, I'm trying to simulate reading from a TTY that writes every two > seconds. I want to do this with a pipe and two threads, one thread > writes every N seconds, while the other reads any data available on the > pipe. > > Unfortunately, my code just hangs. After speaking with the fine people > in #chicken, it seems that this may be a bug. We played with different > calls to put the threads to sleep, and different functions to read data, > but they all ended up freezing at some point. > > Here is the program: > > (use srfi-18) > (use posix) > > (define (produce writer) > (thread-start! >(lambda () > (print "producer started") > (let loop ((i 0)) >(display (conc "hello" i " ") writer) >(flush-output writer) >(thread-sleep! 1) >(loop (+ i 1)) > > (define (consume reader) > (thread-start! >(lambda () > (print "consumer started") > (let loop () >(print (read-line reader)) >(loop) > > (define (stream-test in-fd out-fd) > (let ((reader (open-input-file* in-fd)) > (writer (open-output-file* out-fd))) > (produce writer) > (thread-join! (consume reader > > (call-with-values create-pipe stream-test) > > Any help would be greatly appreciated. Thanks! > > -- > Aaron Patterson > http://tenderlovemaking.com/ > > ___ > Chicken-users mailing list > Chicken-users@nongnu.org > https://lists.nongnu.org/mailman/listinfo/chicken-users -- .i ma'a lo bradi cu penmi gi'e du ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
[Chicken-users] Pipe and thread problem
Hi, I'm trying to simulate reading from a TTY that writes every two seconds. I want to do this with a pipe and two threads, one thread writes every N seconds, while the other reads any data available on the pipe. Unfortunately, my code just hangs. After speaking with the fine people in #chicken, it seems that this may be a bug. We played with different calls to put the threads to sleep, and different functions to read data, but they all ended up freezing at some point. Here is the program: (use srfi-18) (use posix) (define (produce writer) (thread-start! (lambda () (print "producer started") (let loop ((i 0)) (display (conc "hello" i " ") writer) (flush-output writer) (thread-sleep! 1) (loop (+ i 1)) (define (consume reader) (thread-start! (lambda () (print "consumer started") (let loop () (print (read-line reader)) (loop) (define (stream-test in-fd out-fd) (let ((reader (open-input-file* in-fd)) (writer (open-output-file* out-fd))) (produce writer) (thread-join! (consume reader (call-with-values create-pipe stream-test) Any help would be greatly appreciated. Thanks! -- Aaron Patterson http://tenderlovemaking.com/ ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
