Hi Brandon - We seem to have stopped discussing the Net patches you were making. I think you made some good progress and I'd like to see a version of those patches get committed at some point...
Also, I was recently reminded of my desire to enable programming of web server type things with user-space threads - by a Reddit link to http://static.usenix.org/events/hotos03/tech/full_papers/vonbehren/vonbehren_html/index.html (which is a paper from 2003). We discussed how this could work briefly before, but what I did not realize at that time is that as far as I can tell, Massivethreads already multiplexes socket I/O onto system calls like epoll. So - I think it would be really exciting to do some performance measurements with a simple socket server, written in Chapel, and using Massivethreads. You might graph request latency vs number of open connections. See http://www.kegel.com/c10k.html http://libev.schmorp.de/bench.html for some really old material on these topics (so you can understand what graphs might have been interesting in the past). Anyway, Brandon, I think you are the closest of anybody to making such measurements (because you already have a Chapel socket server)... I don't think I'll personally have time to get to it but I think it's a great project. Cheers, -michael ________________________________________ From: Brandon Ross [[email protected]] Sent: Wednesday, December 04, 2013 05:29 To: Michael P. Ferguson Cc: [email protected] Subject: Re: [Chapel-developers] Chapel network library Hello, sorry about the late response. I haven't used libevent or libcurl before, but I'm not against bundling them and using them for the Net module. Though, I wasn't thinking the Net module would extend much beyond the transport layer, just provide a mechanism to establish socket connections and provide utilities generally useful in network applications (e.g., DNS resolution, URL parsing, etc.). The net_connect("http://example.com";) example was to highlight that it could potentially be aware of the transport protocol and port numbers of certain well-known application protocols (per RFC 1700) and add a little more convenience to the interface. It would just open a TCP connection on port 80, and wouldn't actually be able to speak HTTP -- that functionality should probably go in a separate module. On the subject of libevent, support for an event-based asynchrony approach to handling network IO (maybe even all IO) would be nice. I don't know how libevent works, but if it works how I imagine, we could just start one thread running for libevent and have all IO channel calls return a promise on the Chapel side and let libevent fulfill it. Then there wouldn't be a need to spawn new tasks to handle long IO calls. What we could also do is have IO calls return/take a sync/single array. I.e., not an array where each element is a sync/single, but rather the array internally tracks which ranges have been filled, and allows access to those ranges, while blocking if the range is still waiting to be filled. Does Chapel have such a thing already? Doing it like that would let the application consume bytes from the kernel buffer as they become available, process bytes immediately as they come in instead of waiting for the whole read request to be fulfilled, and also be written in a traditional sequential style. The same thing could be done for writing. It would let you write a simple downloader like this: var buffer:sync [1..0x1000] uint(8); cobegin { while sock.isOpen() do sock->read(buffer); while sock.isOpen() do outfile->write(buffer); } Which would be pretty fast, much simpler than the equivalent in other languages, and not require spawning multiple threads if we use a global event queue. As for conditional compilation to avoid linking errors, I tried adding pragmas to allow for conditional compilation of expressions. It worked by removing nodes from the AST if they had a pragma that excluded them from being built on the target platform. I ran into an issue where using the pragma on module definitions was causing a segfault, though, because the known module list is built during the parsing stage, and freed space was being referenced later in the build process. :( I tried moving the module stuff around to a later part of the build process, but it's tricky and probably not worth the effort. I'm glad you told me about CHPL_TARGET_PLATFORM, I'll look into that. I know relying on compile-time dead branch optimization is safe, but somehow I'd feel warmer and fuzzier if, instead of that (or doing it in C), I could just put a compilation condition on a declaration like so: pragma "for linux" proc myLibProc(...) { extern proc linuxcall(...); ... } pragma "for windows" proc myLibProc(...) { extern proc AnalogousWindowsCallEx(...); ... } It's not that big of a deal at this point, though. I think getaddrinfo is pretty universal, it just can't be linked to statically. I'll try linking to it as an extern proc and see if that causes any problems, and if not we can either deal with the compiler warning or try c-ares as Jason suggested. getaddrinfo blocks anyway, so an asynchronous way of doing it ala c-ares would be nicer, especially if we can tie it into our proposed event queue. -- Brandon On 12/03/2013 09:46 AM, Michael Ferguson wrote: > Hi Brandon - > > Replies to things below, inline. > > On 12/02/2013 07:42 PM, Brandon Ross wrote: >> Hmm, what if getaddrinfo was used directly as an extern proc in the Net >> module code instead of through a wrapper in sys.c? Though that would be >> an issue on systems without a getaddrinfo. We'd be able to handle that >> if it was possible to have conditional compilation of Chapel code >> depending on the OS type. Is there a pragma for that? > > Hmm. Given the current situation - that sys.c only provides a > single implementation DNS resolution - getaddrinfo - I do believe that > you could improve the portability goal by making it an extern > proc in the Net module. I still think that forcing the C compiler > to inline a version in sys.h would have the same effect. But either > way, the Net module would not be a standard module, and so that > Chapel code would only be compiled if the program said 'use Net;'. > > Of course, the other way would be to add an optional runtime layer, > like CHPL_REGEXP or CHPL_GMP. That might make sense if the networking > library has other dependencies that might not always be available, > such as libevent or libcurl. If you agree with me that it would make > sense to build atop libcurl and libevent, this is probably the sensible > way forward. I think the #define - type mess for system support > is best left to C. > > Chapel doesn't have anything like #ifdef, but the generic programming > facility is very flexible and could support string comparison on > config params that indicate the current platform. I believe that > should work now, but havn't tried it. These are defined in > ChapelBase.chpl > line 1-123. You should be able to say something like (I'm intentionally > ignoring details): > proc mygetaddrinfo() { > extern proc getaddrinfo(); > if CHPL_TARGET_PLATFORM == "cray" { > // do something else > } else { > getaddrinfo() > } > } > > Since CHPL_TARGET_PLATFORM (and the other similar variables) are > config param > (ie compile-time) constants, the compiler is obligated to remove the > branch > of the if statement that does not apply. Now, of course > CHPL_TARGET_PLATFORM > has a variety of versions for Crays, so perhaps your strstr would be > useful at compile time, like > > if CHPL_TARGET_PLATFORM.contains("cray") > > but that's a bit of a separate issue, which you should feel free to > put off for the moment. > >> Go's net package provides a pretty intuitive high-level way to do >> connect and listen without all the low-level structures and call >> sequences. Connecting looks like net.Dial("tcp", "example.com:8080") and >> listening looks like net.Listen("tcp", "127.0.0.1:8080"). You pass >> strings, and all the low-level DNS lookup, socket binding, and >> translation to system-specific constants and whatnot is hidden. Dial >> returns a Conn, which you can read and write to directly and use >> wherever a Reader or Writer is expected, while Listen returns a Listener >> with an Accept() which returns Conns. Access to the low level stuff is >> still provided, but these functions are generally all you need. >> >> I imagine Chapel's net package would have similar facilities, and maybe >> let you pass URL strings for convenience. It would be nice to connect to >> a web server with net_connect("tcp://example.com:80") or even >> net_connect("http://example.com";) and have it return a channel (or pair >> of). Similarly, net_listen will let you specify a bind address in the >> URL and return an object which has an accept() method that returns >> channels. >> >> It would be nice to offer some expressive flexibility and sensible >> defaults with regard to accepted URL formats for these functions. E.g., >> net_listen("tcp://0.0.0.0:80"), net_listen("tcp://:80"), >> net_listen("0.0.0.0:80"), and net_listen(":80") should be equivalent. >> Using a different socket type is as easy as changing the URL, e.g. >> net_listen("udp://:80"), net_listen("unix:/path/to/socket"), >> net_listen("raw://:80"). net_connect and net_listen would of course be >> overloaded to let you pass the arguments separately, e.g., >> net_connect("tcp", "example.com:80") or net_connect("tcp", >> "example.com", 80) as well. Ideally it should be as easy as possible to >> hack together a working network application and incrementally optimize >> it with lower-level library functions (which will of course be provided >> as well) as necessary. > > These all sound pretty reasonable to me, with some stylistic points > (Chapel code usually uses camelCase instead of the underscores, but > last time I tried to push for an official Chapel modules style guide, > there was lots of noise but no consensus). > I have in the past considered including libcurl. I don't really think > it would be reasonable to write our own. libcurl could easily handle > most client-based situations. If you want a web-server though, we > might need something else. Turns out though that libevent (which I > had earlier described a desire to incorporate) has basic http > support (including running a server). My preference would be to > incorporate both libevent and libcurl, and use libevent for > anything that it can do and libcurl for everything else. (It turns > out that you can convince libcurl to run atop libevent, see > http://stackoverflow.com/questions/6927710/using-libcurl-with-libevent - > doing so, combined with libevent support I described earlier, would > give us processor-efficient implementations of a variety of protocols, > so that e.g. a web crawler would be easy to write and efficient. > >> I just finished a simple URL parser for this purpose, which is the >> reason behind the string-oriented patch I submitted. Even if we opt not >> to use URLs for describing network connections, it's always nice to have >> a good URL parser in a networking package. (At least it's something I'm >> always looking for.) > > Right, we'll discuss your review request separately. Improvements to the > Chapel modules are always appreciated. But if you agree > with me that incorporating libcurl and libevent is the most reasonable > way forward, we might want to rely on their existing URL parsing... > > Cheers, > > -michael ------------------------------------------------------------------------------ "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available. Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs _______________________________________________ Chapel-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-developers
