I didn't try to integrate with libevent yet, I have to read something about it first. but there's a thing I didn't understand.. can't I simply collect all the urls I need to fetch (for a single gadget) at once and then pass them to the curl_multi (without libevent) ? it could became a bottleneck, right? however I'll try to adapt to libevent my example.
thanks leo On Tue, Jun 24, 2008 at 6:48 PM, Kevin Brown <[EMAIL PROTECTED]> wrote: > On Tue, Jun 24, 2008 at 8:42 AM, Leonardo Foderaro <[EMAIL PROTECTED]> > wrote: > >> Hi Kevin, >> I followed your suggest and played with libcurl in multi mode.. it's cool ! >> I tried some examples found in the net.. but I need more flexibility >> (e.g. an arbitrary number of easy handles) >> So I wrote a simple wrapper to deal with the async curl calls with >> ease, 'cmhw' (Curl Multi Handle Wrapper) >> The idea is to have a component which take care of all the aspects of >> the multi curling, >> you just need to create an instance and add the urls you want to >> fetch, and it just do the job. >> it's an early beta version, there's a lot of work which should be done >> (e.g. all the possible error checking, >> code optimization / safety checks, a better api interface, a better >> name... ) but it works. >> could it be useful for the gadget processing library? >> >> an example: >> >> /* our main struct. it holds the curl multi handle and the hash of >> all the needed resources */ >> cmhw_context_t *ctx = NULL; >> >> /* represents a single remote resource. holds an 'easy' curl handle >> and a chunk of memory for the fetched content data */ >> cmhw_item_t *item = NULL; >> >> /* init... */ >> ctx = cmhw_init(); >> >> /* add some request (with an optional name, just to get them later >> with ease) */ >> cmhw_add_url(ctx, "google", "http://www.google.com";); >> cmhw_add_url(ctx, "myspace", "http://www.myspace.com";); >> >> /* download them all! */ >> cmhw_perform(ctx); >> >> /* make something interesting with our results */ >> item = cmhw_item_get(ctx, "google"); >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, >> item->url, item->chunk->memory); >> >> item = cmhw_item_get(ctx, "myspace"); >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, >> item->url, item->chunk->memory); >> >> /* cleanup.. like a nice application... */ >> cmhw_cleanup(ctx); >> >> >> please let me know what you think about this and how I could improve it. > > > One thing that might be an issue is that gadget specs frequently require > fetching additional files (message bundles, preloads, etc.) One nice aspect > of multi mode is that you can always add new requests at any time, so if you > set up an async dispatching loop (using libevent), you can get very high > concurrency rates. > > >> >> thanks >> leo >> >> >> >> >> >> On Mon, Jun 23, 2008 at 1:42 AM, Kevin Brown <[EMAIL PROTECTED]> wrote: >> > On Sun, Jun 22, 2008 at 3:16 PM, Leonardo Foderaro < >> [EMAIL PROTECTED]> >> > wrote: >> > >> >> Hi, >> >> I'm trying to write a simple prototype of the library we talked about >> >> some days ago. >> >> It is a really early version at the moment.. just some experiments (of >> >> course the lucky codename is test.c ) >> >> >> >> What I'm using: >> >> - libcurl for the remote content fetching >> >> - libxml2 for the gadget xml parsing >> >> - some data structures from the glibc2.0 (e.g. GHashTable) for storing >> >> dynamic variables >> >> - a lexical scanner (generated using flex) for the hangman substitution >> >> >> >> Actually I'm able to fetch a gadget from a remote URL (or a local >> >> file), load it into the XmlParser and perform the hangman vars >> >> substitution of the Content node through the flex scanner (only >> >> __MODULE_ID__ for now). >> >> >> >> Of course there's a lot of work to do. >> >> >> >> Let me know what could be the next logical step.. I think I should >> >> construct a C representation of the gadget with all its properties, >> >> right? (it just writes to stdout at the moment) >> > >> > >> > Yeah, you'll probably want a struct similar to the JSON output from >> > /metadata in Shindig. As a library user / module writer, that's what I'd >> > want to consume. >> > >> > Definitely try libcurl in multi mode -- it's quite possible to implement >> a >> > high concurrency http client in this model using async events -- you can >> use >> > libevent for this. >> > >> > The ideal library would probably look something like this: >> > >> > os_request *req = os_request_create(); >> > req->locale = ... >> > os_request_add_gadget(req, "http://example.org/gadget.xml";); >> > os_get_data(req, &callback); >> > >> > void callback(os_request *req, os_response *resp) {...} >> > >> > Rendering could then be implemented as a consumer of this library. >> > >> > >> > >> >> >> >> thanks to all, >> >> leonardo >> >> >> > >> >
