Re: [Chicken-users] All threads are blocking by I/O
Pedro Melendez scripsit: > One question, would this issue be solved too if the threads were native > and not based in continuations? In principle, yes. But the way Chicken treats the stack would make it very hard to port to a multistack design: the stack grows steadily as different procedures are invoked, and then is popped all at once by the garbage collector, so the C stack is the nursery of the Scheme heap. -- John Cowan co...@ccil.orghttp://ccil.org/~cowan Big as a house, much bigger than a house, it looked to [Sam], a grey-clad moving hill. Fear and wonder, maybe, enlarged him in the hobbit's eyes, but the Mumak of Harad was indeed a beast of vast bulk, and the like of him does not walk now in Middle-earth; his kin that live still in latter days are but memories of his girth and his majesty. --"Of Herbs and Stewed Rabbit" ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
Guys... You rock! Cygwin worked as a charm. I will be using it that way. One question, would this issue be solved too if the threads were native and not based in continuations? Would that be a good idea? Sent from my Windows Phone -- From: Pedro Melendez Sent: 2013-09-25 1:31 PM To: John Cowan Cc: chicken-users Subject: Re: [Chicken-users] All threads are blocking by I/O Hey John, Yeah, I guess you are right... that's the easier solution at this point, it just sounded nicer to have it work without cygwin. Thanks! On Wed, Sep 25, 2013 at 1:05 PM, John Cowan wrote: > Pedro Melendez scripsit: > > > Oh well, that's unfortunately :( The server would run at the end in a > Linux > > box, but I need to be able to test it on windows > > during the development. > > In that case, I would develop it under Cygwin rather than using the > MinGW setup. The Cygwin build of Chicken is much more Linuxly correct > than MinGW is or can be, and since you aren't deploying there, you > don't need to care about the GPL. > > -- > There are three kinds of people in the world: John Cowan > those who can count,co...@ccil.org > and those who can't. > -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
On Wed, 25 Sep 2013, Pedro Melendez wrote: Oh well, that's unfortunately :( The server would run at the end in a Linux box, but I need to be able to test it on windows during the development. I am willing to spend time on fixing this, but I am not very knowledgeable on Chicken internals so I am not sure how useful (or fast enough) would be my help. This is where cygwin might help. It provides a POSIX compatibility layer. It is almost GNU/Linux without the Linux kernel. The installation takes time and downloads lots of stuff. Do not use their Chicken Scheme package; it is very old. Select the development tool packages (not the MinGW flavour) and other things you need. You may have trouble if you need a library which is not available as a cygwin package. Check the package list. Look here. http://cygwin.com/install.html Good luck. -- Claude Marinier ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
Hello, think best way is using IO Completion Ports, cause WaitForMultipleObjects have limit on maximum wait objects (MAXIMUM_WAIT_OBJECTS=64 as i know) and also require maintain extra handle (CreateEvent) for each file descriptor. I spent some time on research, what we have: pros no more hangs no more fd_set, unlimited descriptor count in theory increase web/file servers performance cons user should avoid direct access to file descriptors (port->fileno,file-select) posix unit heavily use c library descriptors, better not touch this small test auditory (maybe move this to pros, if we break something nobody will cry xD) require changes in the chicken core, core dev team may not approve More detail description. require changes in scheduler,extras,library,ports most significant: *C_port_file replace fopen with CreateFile(FILE_FLAG_OVERLAPPED) associate file handle with the scheduler global completion port CreateIoCompletionPort(C_completion_port,file_handle) return FILE* using "fdopen_osfhandle" *stdin,stdout,strerr reopen with OVERLAPPED attribute using "fdopen_osfhandle" *C_read_char,C_write_char,C_display replace calls to C_fprintf, C_fgetc with custom implementation caller should wait unitl opeartion completed using ##sys#thread-block-for-i/o! *scheduler remove select&fd_set, maintain only fd <-> thread mappings. declare C_completion_port global variable completion port handle C_ready_fds_timeout receive ready descriptor via GetQueuedCompletionStatus(C_completion_port) ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
Hey John, Yeah, I guess you are right... that's the easier solution at this point, it just sounded nicer to have it work without cygwin. Thanks! On Wed, Sep 25, 2013 at 1:05 PM, John Cowan wrote: > Pedro Melendez scripsit: > > > Oh well, that's unfortunately :( The server would run at the end in a > Linux > > box, but I need to be able to test it on windows > > during the development. > > In that case, I would develop it under Cygwin rather than using the > MinGW setup. The Cygwin build of Chicken is much more Linuxly correct > than MinGW is or can be, and since you aren't deploying there, you > don't need to care about the GPL. > > -- > There are three kinds of people in the world: John Cowan > those who can count,co...@ccil.org > and those who can't. > -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
Pedro Melendez scripsit: > Oh well, that's unfortunately :( The server would run at the end in a Linux > box, but I need to be able to test it on windows > during the development. In that case, I would develop it under Cygwin rather than using the MinGW setup. The Cygwin build of Chicken is much more Linuxly correct than MinGW is or can be, and since you aren't deploying there, you don't need to care about the GPL. -- There are three kinds of people in the world: John Cowan those who can count,co...@ccil.org and those who can't. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
Hey Peter, > You don't have to; like I said you can just run "csi -s SCRIPT.SCM" That's a nice hint, thanks! > No, sorry. I'm afraid the bug is known; we should be using something > like WaitForMultipleObjects() instead of select(). The former is > supposed to work on any object, not just sockets. Unfortunately, > I've been told this will require substantial rework of CHICKEN's > internals because WaitForMultipleObjects() is callback-based, which > is fundamentally different from the way POSIX select() and poll() work. > > We'll need someone who is knowledgeable enough about both Windows and > CHICKEN and willing to spend time on implementing/reworking this. > Unfortunately, AFAIK there's nobody in the community who is willing > to do this. Oh well, that's unfortunately :( The server would run at the end in a Linux box, but I need to be able to test it on windows during the development. I am willing to spend time on fixing this, but I am not very knowledgeable on Chicken internals so I am not sure how useful (or fast enough) would be my help. Thanks for everything Peter... Cheers, Pedro On Wed, Sep 25, 2013 at 10:31 AM, Peter Bex wrote: > On Wed, Sep 25, 2013 at 09:12:14AM -0400, Pedro Melendez wrote: > > Hi Peter, > > > > Thank you so much for the help! > > Hi Pedro, > > You're most welcome! > > > I just tried it on a linux box and I can > > confirm you that is a windows specific behaviour :( > > I was afraid of that. > > > I am not using the REPL because I noticed that it was blocking everything > > unless I did a thread-join, so I opted to compile and test all the time > > (like we do with our old friend C/C++ :) > > You don't have to; like I said you can just run "csi -s SCRIPT.SCM" > > > > It shouldn't be, but our Windows implementation uses Winsock select(), > > > which is really Microsoft's own "special" interpretation of POSIX > select() > > > which doesn't work on anything except sockets. However, this *should* > > > still work since you're only using tcp ports. > > > > This makes a lot of sense, so I guess my only option is too debug the C++ > > side of the tcp implementation on Windows? Or do you know a workaround I > > could try for this? > > No, sorry. I'm afraid the bug is known; we should be using something > like WaitForMultipleObjects() instead of select(). The former is > supposed to work on any object, not just sockets. Unfortunately, > I've been told this will require substantial rework of CHICKEN's > internals because WaitForMultipleObjects() is callback-based, which > is fundamentally different from the way POSIX select() and poll() work. > > We'll need someone who is knowledgeable enough about both Windows and > CHICKEN and willing to spend time on implementing/reworking this. > Unfortunately, AFAIK there's nobody in the community who is willing > to do this. > > Cheers, > Peter > -- > http://www.more-magic.net > -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
On Wed, Sep 25, 2013 at 09:12:14AM -0400, Pedro Melendez wrote: > Hi Peter, > > Thank you so much for the help! Hi Pedro, You're most welcome! > I just tried it on a linux box and I can > confirm you that is a windows specific behaviour :( I was afraid of that. > I am not using the REPL because I noticed that it was blocking everything > unless I did a thread-join, so I opted to compile and test all the time > (like we do with our old friend C/C++ :) You don't have to; like I said you can just run "csi -s SCRIPT.SCM" > > It shouldn't be, but our Windows implementation uses Winsock select(), > > which is really Microsoft's own "special" interpretation of POSIX select() > > which doesn't work on anything except sockets. However, this *should* > > still work since you're only using tcp ports. > > This makes a lot of sense, so I guess my only option is too debug the C++ > side of the tcp implementation on Windows? Or do you know a workaround I > could try for this? No, sorry. I'm afraid the bug is known; we should be using something like WaitForMultipleObjects() instead of select(). The former is supposed to work on any object, not just sockets. Unfortunately, I've been told this will require substantial rework of CHICKEN's internals because WaitForMultipleObjects() is callback-based, which is fundamentally different from the way POSIX select() and poll() work. We'll need someone who is knowledgeable enough about both Windows and CHICKEN and willing to spend time on implementing/reworking this. Unfortunately, AFAIK there's nobody in the community who is willing to do this. Cheers, Peter -- http://www.more-magic.net ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
Hi Peter, Thank you so much for the help! I just tried it on a linux box and I can confirm you that is a windows specific behaviour :( > I tested it out (via csi -s mini-tcp-server.scm) and it kept looping and > when I connected with telnet it started printing "Hola" to my telnet > client. > > Is that the correct behaviour? Yes it is, the thing is that "Hola" was a test and should happens no matter what because is in a separate thread, but unfortunately the behaviour on Windows is that it gets stopped by the read-line function. > Perhaps you're running it from the REPL? The standard input for csi is > usually line-buffered, so it's the OS itself which might be blocking the > process while reading from stdin for the next command to run. This can > be worked around on UNIX by installing the parley egg. However, I don't > know if this works on Windows, because of the above-mentioned FUBAR > select() implementation. I am not using the REPL because I noticed that it was blocking everything unless I did a thread-join, so I opted to compile and test all the time (like we do with our old friend C/C++ :) > It shouldn't be, but our Windows implementation uses Winsock select(), > which is really Microsoft's own "special" interpretation of POSIX select() > which doesn't work on anything except sockets. However, this *should* > still work since you're only using tcp ports. This makes a lot of sense, so I guess my only option is too debug the C++ side of the tcp implementation on Windows? Or do you know a workaround I could try for this? Thanks again for everything, All your comments had been very helpful Cheers, Pedro On Wed, Sep 25, 2013 at 8:42 AM, Peter Bex wrote: > On Wed, Sep 25, 2013 at 05:21:57AM -0700, Pedro Melendez wrote: > > Hi Peter! > > > > Thank you so much for your response! > > > > I don't understand then what I am doing wrong, this is what I have so > > far: > > > > > https://github.com/pmelendez/scheme-test-server/blob/master/mini-tcp-server.scm > > > > Please notice that the code is dirty because it is a work in progress > > and I am kinda stuck. > > I tested it out (via csi -s mini-tcp-server.scm) and it kept looping and > when I connected with telnet it started printing "Hola" to my telnet > client. > > Is that the correct behaviour? > > The code itself looks fine to me, from a quick glance. > > > One detail I forgot to mention is that I am testing this on windows. > > That's a very important little detail :) There are still some important > glitches left on Windows, unfortunately. > > > Now after your explanation... Is it possible that the behavior is > > platform-specific? > > It shouldn't be, but our Windows implementation uses Winsock select(), > which is really Microsoft's own "special" interpretation of POSIX select() > which doesn't work on anything except sockets. However, this *should* > still work since you're only using tcp ports. > > Perhaps you're running it from the REPL? The standard input for csi is > usually line-buffered, so it's the OS itself which might be blocking the > process while reading from stdin for the next command to run. This can > be worked around on UNIX by installing the parley egg. However, I don't > know if this works on Windows, because of the above-mentioned FUBAR > select() implementation. > > I hope this helps. > > Cheers, > Peter > -- > http://www.more-magic.net > -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
On Wed, Sep 25, 2013 at 05:21:57AM -0700, Pedro Melendez wrote: > Hi Peter! > > Thank you so much for your response! > > I don't understand then what I am doing wrong, this is what I have so > far: > > https://github.com/pmelendez/scheme-test-server/blob/master/mini-tcp-server.scm > > Please notice that the code is dirty because it is a work in progress > and I am kinda stuck. I tested it out (via csi -s mini-tcp-server.scm) and it kept looping and when I connected with telnet it started printing "Hola" to my telnet client. Is that the correct behaviour? The code itself looks fine to me, from a quick glance. > One detail I forgot to mention is that I am testing this on windows. That's a very important little detail :) There are still some important glitches left on Windows, unfortunately. > Now after your explanation... Is it possible that the behavior is > platform-specific? It shouldn't be, but our Windows implementation uses Winsock select(), which is really Microsoft's own "special" interpretation of POSIX select() which doesn't work on anything except sockets. However, this *should* still work since you're only using tcp ports. Perhaps you're running it from the REPL? The standard input for csi is usually line-buffered, so it's the OS itself which might be blocking the process while reading from stdin for the next command to run. This can be worked around on UNIX by installing the parley egg. However, I don't know if this works on Windows, because of the above-mentioned FUBAR select() implementation. I hope this helps. Cheers, Peter -- http://www.more-magic.net ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
Hi Peter! Thank you so much for your response! I don't understand then what I am doing wrong, this is what I have so far: https://github.com/pmelendez/scheme-test-server/blob/master/mini-tcp-server.scm Please notice that the code is dirty because it is a work in progress and I am kinda stuck. One detail I forgot to mention is that I am testing this on windows. Now after your explanation... Is it possible that the behavior is platform-specific? Thanks again for the help! Cheers, Pedro Sent from my Windows Phone From: Peter Bex Sent: 2013-09-25 7:50 AM To: Pedro Melendez Cc: chicken-users Subject: Re: [Chicken-users] All threads are blocking by I/O On Wed, Sep 25, 2013 at 07:40:39AM -0400, Pedro Melendez wrote: > After some experimentation I came out to the realization that the reading > thread is blocking all other threads. And I actually just found that stated > on the TCP Unit documentation: > >- Blocking I/O will block all threads, except for some socket operations >(see the section about the tcp unit). An exception is the >read-eval-print loop on UNIX platforms: waiting for input will not block >other threads, provided the current input port reads input from a console. > > So, I guess my question is if there is a way to workaround this or I have > to combine my code with some C++ code to achieve what I want to do? Hi Pedro! By default, the tcp unit's procedures will create nonblocking I/O ports, and the srfi-18 scheduler will multiplex threads over all open sockets using POSIX poll(). In other words, it should Just Work if you read in one thread and write in the other thread. Maybe I'm misunderstanding what you're trying to do? If so, could you please send us some sample code to avoid further confusion? Cheers, Peter -- http://www.more-magic.net ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] All threads are blocking by I/O
On Wed, Sep 25, 2013 at 07:40:39AM -0400, Pedro Melendez wrote: > After some experimentation I came out to the realization that the reading > thread is blocking all other threads. And I actually just found that stated > on the TCP Unit documentation: > >- Blocking I/O will block all threads, except for some socket operations >(see the section about the tcp unit). An exception is the >read-eval-print loop on UNIX platforms: waiting for input will not block >other threads, provided the current input port reads input from a console. > > So, I guess my question is if there is a way to workaround this or I have > to combine my code with some C++ code to achieve what I want to do? Hi Pedro! By default, the tcp unit's procedures will create nonblocking I/O ports, and the srfi-18 scheduler will multiplex threads over all open sockets using POSIX poll(). In other words, it should Just Work if you read in one thread and write in the other thread. Maybe I'm misunderstanding what you're trying to do? If so, could you please send us some sample code to avoid further confusion? Cheers, Peter -- http://www.more-magic.net ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
