maybe I'm thinking in the wrong way... 1 gadget ---> 1 http request ( /gadgets/ifr?url=http://host/gadget.xml) ----> many http fetch (the gadget xml, bundles, messages, etc..)
however, now my example supports libevent. it looks like this: cmhw_add_url(ctx, "gadget1", "http://host/gadget1.xml";); cmhw_add_url(ctx, "gadget2", "http://host/gadget2.xml";); do { while (CURLM_CALL_MULTI_PERFORM == curl_multi_socket_all(ctx->multi_handle, (ctx->unfinished))); event_dispatch(); } while ((*ctx->unfinished) != -1); is this similar to your idea? if so, I try to go one step further. thanks leo On Wed, Jun 25, 2008 at 1:25 AM, Kevin Brown <[EMAIL PROTECTED]> wrote: > On Tue, Jun 24, 2008 at 3:57 PM, Leonardo Foderaro <[EMAIL PROTECTED]> > wrote: > >> I didn't try to integrate with libevent yet, I have to read something >> about it first. >> but there's a thing I didn't understand.. can't I simply collect all >> the urls I need to fetch (for a single gadget) at once and then pass >> them to the curl_multi (without libevent) ? >> it could became a bottleneck, right? >> however I'll try to adapt to libevent my example. > > > I suppose this depends on the overall model. You'll want to support > simultaneous processing of multiple gadgets, right? If so, then you'll want > to be able to do this: > > - add all gadgets to curl and start processing > - when a gadget is returned (not all files will come back at the same speed > due to remote host differences and the like -- some may even be cached if > you support that), you'll want to parse it and extract the referenced urls > - add the referenced urls to curl and continue processing. > > * keep processing until nothing remains in the queue, then render. > > >> >> >> thanks >> leo >> >> On Tue, Jun 24, 2008 at 6:48 PM, Kevin Brown <[EMAIL PROTECTED]> wrote: >> > On Tue, Jun 24, 2008 at 8:42 AM, Leonardo Foderaro < >> [EMAIL PROTECTED]> >> > wrote: >> > >> >> Hi Kevin, >> >> I followed your suggest and played with libcurl in multi mode.. it's >> cool ! >> >> I tried some examples found in the net.. but I need more flexibility >> >> (e.g. an arbitrary number of easy handles) >> >> So I wrote a simple wrapper to deal with the async curl calls with >> >> ease, 'cmhw' (Curl Multi Handle Wrapper) >> >> The idea is to have a component which take care of all the aspects of >> >> the multi curling, >> >> you just need to create an instance and add the urls you want to >> >> fetch, and it just do the job. >> >> it's an early beta version, there's a lot of work which should be done >> >> (e.g. all the possible error checking, >> >> code optimization / safety checks, a better api interface, a better >> >> name... ) but it works. >> >> could it be useful for the gadget processing library? >> >> >> >> an example: >> >> >> >> /* our main struct. it holds the curl multi handle and the hash of >> >> all the needed resources */ >> >> cmhw_context_t *ctx = NULL; >> >> >> >> /* represents a single remote resource. holds an 'easy' curl handle >> >> and a chunk of memory for the fetched content data */ >> >> cmhw_item_t *item = NULL; >> >> >> >> /* init... */ >> >> ctx = cmhw_init(); >> >> >> >> /* add some request (with an optional name, just to get them later >> >> with ease) */ >> >> cmhw_add_url(ctx, "google", "http://www.google.com";); >> >> cmhw_add_url(ctx, "myspace", "http://www.myspace.com";); >> >> >> >> /* download them all! */ >> >> cmhw_perform(ctx); >> >> >> >> /* make something interesting with our results */ >> >> item = cmhw_item_get(ctx, "google"); >> >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, >> >> item->url, item->chunk->memory); >> >> >> >> item = cmhw_item_get(ctx, "myspace"); >> >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, >> >> item->url, item->chunk->memory); >> >> >> >> /* cleanup.. like a nice application... */ >> >> cmhw_cleanup(ctx); >> >> >> >> >> >> please let me know what you think about this and how I could improve it. >> > >> > >> > One thing that might be an issue is that gadget specs frequently require >> > fetching additional files (message bundles, preloads, etc.) One nice >> aspect >> > of multi mode is that you can always add new requests at any time, so if >> you >> > set up an async dispatching loop (using libevent), you can get very high >> > concurrency rates. >> > >> > >> >> >> >> thanks >> >> leo >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Jun 23, 2008 at 1:42 AM, Kevin Brown <[EMAIL PROTECTED]> wrote: >> >> > On Sun, Jun 22, 2008 at 3:16 PM, Leonardo Foderaro < >> >> [EMAIL PROTECTED]> >> >> > wrote: >> >> > >> >> >> Hi, >> >> >> I'm trying to write a simple prototype of the library we talked about >> >> >> some days ago. >> >> >> It is a really early version at the moment.. just some experiments >> (of >> >> >> course the lucky codename is test.c ) >> >> >> >> >> >> What I'm using: >> >> >> - libcurl for the remote content fetching >> >> >> - libxml2 for the gadget xml parsing >> >> >> - some data structures from the glibc2.0 (e.g. GHashTable) for >> storing >> >> >> dynamic variables >> >> >> - a lexical scanner (generated using flex) for the hangman >> substitution >> >> >> >> >> >> Actually I'm able to fetch a gadget from a remote URL (or a local >> >> >> file), load it into the XmlParser and perform the hangman vars >> >> >> substitution of the Content node through the flex scanner (only >> >> >> __MODULE_ID__ for now). >> >> >> >> >> >> Of course there's a lot of work to do. >> >> >> >> >> >> Let me know what could be the next logical step.. I think I should >> >> >> construct a C representation of the gadget with all its properties, >> >> >> right? (it just writes to stdout at the moment) >> >> > >> >> > >> >> > Yeah, you'll probably want a struct similar to the JSON output from >> >> > /metadata in Shindig. As a library user / module writer, that's what >> I'd >> >> > want to consume. >> >> > >> >> > Definitely try libcurl in multi mode -- it's quite possible to >> implement >> >> a >> >> > high concurrency http client in this model using async events -- you >> can >> >> use >> >> > libevent for this. >> >> > >> >> > The ideal library would probably look something like this: >> >> > >> >> > os_request *req = os_request_create(); >> >> > req->locale = ... >> >> > os_request_add_gadget(req, "http://example.org/gadget.xml";); >> >> > os_get_data(req, &callback); >> >> > >> >> > void callback(os_request *req, os_response *resp) {...} >> >> > >> >> > Rendering could then be implemented as a consumer of this library. >> >> > >> >> > >> >> > >> >> >> >> >> >> thanks to all, >> >> >> leonardo >> >> >> >> >> > >> >> >> > >> >
