Re: [Haskell-cafe] Libevent FFI problems
* Adam Langley [EMAIL PROTECTED] [080725 13:05]: 2008/7/25 Krzysztof Skrzętnicki [EMAIL PROTECTED]: Developing a high-performance web server in Concurrent Haskell http://www.haskell.org/~simonmar/papers/web-server-jfp.pdf (see page 15) Perhaps you might be interested in this paper also because of its topic. That's a good reference. Also note that the paper is 6 years old and GHC has come a long way since then. I'd suspect that the graph on page 15 would look much more favourable to Haskell these days. http://www.seas.upenn.edu/~lipeng/homepage/unify.html This paper (and included code) should be relevant and useful. There's an epoll wrapper and a lot of good stuff. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
On Sat, Jul 26, 2008 at 4:34 AM, Levi Greenspan [EMAIL PROTECTED] wrote: client: internal error: awaitEvent: descriptor out of range (GHC version 6.8.3 for i386_unknown_linux) Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug Aborted This is caused by having more file descriptors than FD_SETSIZE (which is 1024) on my system. So clearly select() doesn't scale to the numbers I need :-( Now, *thats* a good point. I'd still suggest that taking the time to add epoll support to the threaded runtime would be better for you(, and for everyone else!) than avoiding the use of GHC threads. But, with that error, you're correct that the current select based system is insufficient. AGL -- Adam Langley [EMAIL PROTECTED] http://www.imperialviolet.org ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
On Fri, 2008-07-25 at 13:45 -0700, Adam Langley wrote: I'd suggest that you write your server on the select() based system as-is for now. Then, when you need epoll you'll be sufficiently motivated to hack up the RTS to include it ;) The problem with a select() based approach is that I can not have more than 1024 parallel connections on my Linux system, since this is the upper limit for the number of file descriptors in the FD_SET used by select(). Eg. # ulimit -n 1 # start client with many open connections client: internal error: awaitEvent: descriptor out of range (GHC version 6.8.3 for i386_unknown_linux) Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug Aborted This is caused by having more file descriptors than FD_SETSIZE (which is 1024) on my system. So clearly select() doesn't scale to the numbers I need :-( Thanks, Levi ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
2008/7/23 Levi Greenspan [EMAIL PROTECTED]: I would be grateful for any advices, hints or comments. And I really look forward to the FFI section in the Real World Haskell book. Generally it looks pretty good. I think I'm missing some C code (function wrapper). However, libevent is an odd choice to wrap. The RTS already multiplexes IO for Haskell programs. In order not to block the RTS, the libevent using code would have to be in its own kernel thread. Really, the RTS needs to be ported to libevent rather than an FFI wrapping. For your specific problem, I see you're allocating xptr, but I don't see that you're ever poking a value into it. Indeed, I can't see that createEvent ever uses 'x'. Hope that helps. AGL -- Adam Langley [EMAIL PROTECTED] http://www.imperialviolet.org ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
Thanks Adam. It took me some time to realize what you recognized immediately - xptr is useless. Instead I now bind the additional value which might be provided on event creation in a closure since the C code never has to deal with it. And yes, there are many functions missing. For now I just need the event notification to detect file descritor (or socket) changes. Regarding your remark, that the RTS multiplexes IO already I am usure how this works out in practice. The reason I write this wrapper is that I want a network server which can handle thousands of connections. And to not require a permanent thread for each I would like to use something like select in C or Selector in Java. I haven't found something like this in Haskell, hence the libevent wrapper. If you have any information how to write something like this without this wrapper I would be more than happy. Finally the code still leaks FunPtrs as so far I never free them and honestly I don't know when to do this. Any ideas? For reference I have attached a newer version of the wrapper code (removed the TimeVal stuff since I don't need it). Thanks again, Levi On Fri, Jul 25, 2008 at 7:21 PM, Adam Langley [EMAIL PROTECTED] wrote: 2008/7/23 Levi Greenspan [EMAIL PROTECTED]: I would be grateful for any advices, hints or comments. And I really look forward to the FFI section in the Real World Haskell book. Generally it looks pretty good. I think I'm missing some C code (function wrapper). However, libevent is an odd choice to wrap. The RTS already multiplexes IO for Haskell programs. In order not to block the RTS, the libevent using code would have to be in its own kernel thread. Really, the RTS needs to be ported to libevent rather than an FFI wrapping. For your specific problem, I see you're allocating xptr, but I don't see that you're ever poking a value into it. Indeed, I can't see that createEvent ever uses 'x'. Hope that helps. AGL -- Adam Langley [EMAIL PROTECTED] http://www.imperialviolet.org hsevent.hsc Description: Binary data ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
2008/7/25 Levi Greenspan [EMAIL PROTECTED]: And to not require a permanent thread for each I would like to use something like select in C or Selector in Java. I haven't found something like this in Haskell, hence the libevent wrapper. As far as I know GHC's scheduler uses select() internally. At least it was using at the time of writing this paper: Developing a high-performance web server in Concurrent Haskell http://www.haskell.org/~simonmar/papers/web-server-jfp.pdf (see page 15) Perhaps you might be interested in this paper also because of its topic. Christopher Skrzętnicki ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
2008/7/25 Krzysztof Skrzętnicki [EMAIL PROTECTED]: Developing a high-performance web server in Concurrent Haskell http://www.haskell.org/~simonmar/papers/web-server-jfp.pdf (see page 15) Perhaps you might be interested in this paper also because of its topic. That's a good reference. Also note that the paper is 6 years old and GHC has come a long way since then. I'd suspect that the graph on page 15 would look much more favourable to Haskell these days. AGL -- Adam Langley [EMAIL PROTECTED] http://www.imperialviolet.org ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
agl: 2008/7/25 Krzysztof Skrzętnicki [EMAIL PROTECTED]: Developing a high-performance web server in Concurrent Haskell http://www.haskell.org/~simonmar/papers/web-server-jfp.pdf (see page 15) Perhaps you might be interested in this paper also because of its topic. That's a good reference. Also note that the paper is 6 years old and GHC has come a long way since then. I'd suspect that the graph on page 15 would look much more favourable to Haskell these days. Ask Johan about the minimum 5k/sec web server using heavy concurrency he's hacking on. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
Thank you (and Christopher) for the link. I have one question though - I read this ticket in the GHC trac: http://hackage.haskell.org/trac/ghc/ticket/635 which plans to use epoll instead of select. The reason I thought of libevent is exactly the support for epoll and other better-than-select mechanisms. Is there progress in GHC with regard to this? This is very important for me since I awill not write a web-server, but rather want to play with Comet (i.e. having thousands of open connections as it is common for long-polling in addition to many relatively short request/response based connections). However it seems no quite attractive to go with what is offered by GHC currently instead of writing the libevent wrapper. Just for the sake of completeness - any insights into the freeFunPtr issue I wrote about? ;-) Many thanks to all of you. - Levi On Fri, Jul 25, 2008 at 10:05 PM, Adam Langley [EMAIL PROTECTED] wrote: 2008/7/25 Krzysztof Skrzętnicki [EMAIL PROTECTED]: Developing a high-performance web server in Concurrent Haskell http://www.haskell.org/~simonmar/papers/web-server-jfp.pdf (see page 15) Perhaps you might be interested in this paper also because of its topic. That's a good reference. Also note that the paper is 6 years old and GHC has come a long way since then. I'd suspect that the graph on page 15 would look much more favourable to Haskell these days. AGL -- Adam Langley [EMAIL PROTECTED] http://www.imperialviolet.org ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
2008/7/25 Levi Greenspan [EMAIL PROTECTED]: Regarding your remark, that the RTS multiplexes IO already I am usure how this works out in practice. The reason I write this wrapper is that I want a network server which can handle thousands of connections. And to not require a permanent thread for each I would like to use something like select in C or Selector in Java. I haven't found something like this in Haskell, hence the libevent wrapper. If you have any information how to write something like this without this wrapper I would be more than happy. The current model for concurrency in Haskell is m lightweight user-threads working on n kernel threads, thus the threads are very cheap in Haskell and take advantage of a multicore. I think with the current model, the right choice for a web server in Haskell is a multi-threaded model. As a point of comparison, I remember that blog post which noted that its FastCGI was able to withstand more than 3000 hits by second (on a quite normal setting). -- Jedaï ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Libevent FFI problems
On Fri, Jul 25, 2008 at 1:32 PM, Levi Greenspan [EMAIL PROTECTED] wrote: Thank you (and Christopher) for the link. I have one question though - I read this ticket in the GHC trac: http://hackage.haskell.org/trac/ghc/ticket/635 which plans to use epoll instead of select. The reason I thought of libevent is exactly the support for epoll and other better-than-select mechanisms. Is there progress in GHC with regard to this? This is very important for me since I awill not write a web-server, but rather want to play with Comet (i.e. having thousands of open connections as it is common for long-polling in addition to many relatively short request/response based connections). Bryan O'Sullivan make an off-the-cuff remark about doing this work at a talk a few months back, however I haven't heard anything about him starting. I think it's safe to assume that no one has taken it up yet. Personally, some of my other projects are higher priorities at the moment. I'd suggest that you write your server on the select() based system as-is for now. Then, when you need epoll you'll be sufficiently motivated to hack up the RTS to include it ;) AGL -- Adam Langley [EMAIL PROTECTED] http://www.imperialviolet.org ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Libevent FFI problems
Dear list members,
This is my first attempt to create a FFI to libevent
(http://monkey.org/~provos/libevent/) which is an event notification
library. A simple usage example is for instance given here:
http://unx.ca/log/libevent_echosrv1c/ . In C one basically creates
struct event instances which are initialized with a callback function
and the kind of event (e.g. read) one is intested in. Optionally a
data argument can be given which will be passed to the callback
function when invoked eventually. An event dispatcher loop will invoke
the callback when the event occurs.
I have attached my first attempt at a FFI to libevent. However it
doesn't work correctly. In the client code I want to pass 99 as data
to the callback function, but what is printed out is 0. I guess there
is an issue with pointer derefencing, but I don't know.
Also I am not sure how to handle the FunPtr deallocation. Since the
FunPtr is stored in the foreign event structure I don't know when I
can free it.
I would be grateful for any advices, hints or comments. And I really
look forward to the FFI section in the Real World Haskell book.
Thank you very much.
- Levi
hsevent.hsc
Description: Binary data
module Main where
import System.IO
import Network.Socket
import Network.Stream
import Network.HTTP
import Network.URI
import Control.Monad
import Control.Exception
import Control.Concurrent
import System.Posix.IO
import System.Posix.Types
import Event as E
import qualified Data.Map as Map
callback :: Fd - E.EventType - Int - IO ()
callback fd e x = do
print fd
print e
print x
main :: IO ()
main =
withSocketsDo $ do
E.init
addrinfos - getAddrInfo (Just (defaultHints {addrFlags = [AI_PASSIVE]}))
Nothing
(Just 1234)
let serveraddr = head addrinfos
-- create server socket
sock - socket (addrFamily serveraddr) Stream defaultProtocol
-- bind it to the address we're listening to
bindSocket sock (addrAddress serveraddr)
-- start listening for connection requests
-- max. connection requests waiting to be accepted = 5 (max. queue size)
listen sock 5
ev - E.createEvent (Fd (fdSocket sock)) [E.ReadEvent, E.WriteEvent] callback 99
E.addEvent ev Nothing
E.dispatch
return ()
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
