On Wed, Jun 25, 2008 at 9:14 AM, Leonardo Foderaro <[EMAIL PROTECTED]> wrote:
> maybe I'm thinking in the wrong way... 1 gadget ---> 1 http request ( > /gadgets/ifr?url=http://host/gadget.xml) ----> many http fetch (the > gadget xml, bundles, messages, etc..) > > however, now my example supports libevent. > > it looks like this: > > cmhw_add_url(ctx, "gadget1", "http://host/gadget1.xml";); > cmhw_add_url(ctx, "gadget2", "http://host/gadget2.xml";); > > do { > while (CURLM_CALL_MULTI_PERFORM == > curl_multi_socket_all(ctx->multi_handle, (ctx->unfinished))); > event_dispatch(); > } while ((*ctx->unfinished) != -1); > > > is this similar to your idea? Yep, that's essentially what I'd do. This will allow you to avoid blocking for all gadget downloads when fetching others. It's pretty much guaranteed to win over doing everything serially. > > if so, I try to go one step further. > > thanks > leo > > > > > On Wed, Jun 25, 2008 at 1:25 AM, Kevin Brown <[EMAIL PROTECTED]> wrote: > > On Tue, Jun 24, 2008 at 3:57 PM, Leonardo Foderaro < > [EMAIL PROTECTED]> > > wrote: > > > >> I didn't try to integrate with libevent yet, I have to read something > >> about it first. > >> but there's a thing I didn't understand.. can't I simply collect all > >> the urls I need to fetch (for a single gadget) at once and then pass > >> them to the curl_multi (without libevent) ? > >> it could became a bottleneck, right? > >> however I'll try to adapt to libevent my example. > > > > > > I suppose this depends on the overall model. You'll want to support > > simultaneous processing of multiple gadgets, right? If so, then you'll > want > > to be able to do this: > > > > - add all gadgets to curl and start processing > > - when a gadget is returned (not all files will come back at the same > speed > > due to remote host differences and the like -- some may even be cached if > > you support that), you'll want to parse it and extract the referenced > urls > > - add the referenced urls to curl and continue processing. > > > > * keep processing until nothing remains in the queue, then render. > > > > > >> > >> > >> thanks > >> leo > >> > >> On Tue, Jun 24, 2008 at 6:48 PM, Kevin Brown <[EMAIL PROTECTED]> wrote: > >> > On Tue, Jun 24, 2008 at 8:42 AM, Leonardo Foderaro < > >> [EMAIL PROTECTED]> > >> > wrote: > >> > > >> >> Hi Kevin, > >> >> I followed your suggest and played with libcurl in multi mode.. it's > >> cool ! > >> >> I tried some examples found in the net.. but I need more flexibility > >> >> (e.g. an arbitrary number of easy handles) > >> >> So I wrote a simple wrapper to deal with the async curl calls with > >> >> ease, 'cmhw' (Curl Multi Handle Wrapper) > >> >> The idea is to have a component which take care of all the aspects of > >> >> the multi curling, > >> >> you just need to create an instance and add the urls you want to > >> >> fetch, and it just do the job. > >> >> it's an early beta version, there's a lot of work which should be > done > >> >> (e.g. all the possible error checking, > >> >> code optimization / safety checks, a better api interface, a better > >> >> name... ) but it works. > >> >> could it be useful for the gadget processing library? > >> >> > >> >> an example: > >> >> > >> >> /* our main struct. it holds the curl multi handle and the hash of > >> >> all the needed resources */ > >> >> cmhw_context_t *ctx = NULL; > >> >> > >> >> /* represents a single remote resource. holds an 'easy' curl handle > >> >> and a chunk of memory for the fetched content data */ > >> >> cmhw_item_t *item = NULL; > >> >> > >> >> /* init... */ > >> >> ctx = cmhw_init(); > >> >> > >> >> /* add some request (with an optional name, just to get them later > >> >> with ease) */ > >> >> cmhw_add_url(ctx, "google", "http://www.google.com";); > >> >> cmhw_add_url(ctx, "myspace", "http://www.myspace.com";); > >> >> > >> >> /* download them all! */ > >> >> cmhw_perform(ctx); > >> >> > >> >> /* make something interesting with our results */ > >> >> item = cmhw_item_get(ctx, "google"); > >> >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, > >> >> item->url, item->chunk->memory); > >> >> > >> >> item = cmhw_item_get(ctx, "myspace"); > >> >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, > >> >> item->url, item->chunk->memory); > >> >> > >> >> /* cleanup.. like a nice application... */ > >> >> cmhw_cleanup(ctx); > >> >> > >> >> > >> >> please let me know what you think about this and how I could improve > it. > >> > > >> > > >> > One thing that might be an issue is that gadget specs frequently > require > >> > fetching additional files (message bundles, preloads, etc.) One nice > >> aspect > >> > of multi mode is that you can always add new requests at any time, so > if > >> you > >> > set up an async dispatching loop (using libevent), you can get very > high > >> > concurrency rates. > >> > > >> > > >> >> > >> >> thanks > >> >> leo > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> On Mon, Jun 23, 2008 at 1:42 AM, Kevin Brown <[EMAIL PROTECTED]> > wrote: > >> >> > On Sun, Jun 22, 2008 at 3:16 PM, Leonardo Foderaro < > >> >> [EMAIL PROTECTED]> > >> >> > wrote: > >> >> > > >> >> >> Hi, > >> >> >> I'm trying to write a simple prototype of the library we talked > about > >> >> >> some days ago. > >> >> >> It is a really early version at the moment.. just some experiments > >> (of > >> >> >> course the lucky codename is test.c ) > >> >> >> > >> >> >> What I'm using: > >> >> >> - libcurl for the remote content fetching > >> >> >> - libxml2 for the gadget xml parsing > >> >> >> - some data structures from the glibc2.0 (e.g. GHashTable) for > >> storing > >> >> >> dynamic variables > >> >> >> - a lexical scanner (generated using flex) for the hangman > >> substitution > >> >> >> > >> >> >> Actually I'm able to fetch a gadget from a remote URL (or a local > >> >> >> file), load it into the XmlParser and perform the hangman vars > >> >> >> substitution of the Content node through the flex scanner (only > >> >> >> __MODULE_ID__ for now). > >> >> >> > >> >> >> Of course there's a lot of work to do. > >> >> >> > >> >> >> Let me know what could be the next logical step.. I think I should > >> >> >> construct a C representation of the gadget with all its > properties, > >> >> >> right? (it just writes to stdout at the moment) > >> >> > > >> >> > > >> >> > Yeah, you'll probably want a struct similar to the JSON output from > >> >> > /metadata in Shindig. As a library user / module writer, that's > what > >> I'd > >> >> > want to consume. > >> >> > > >> >> > Definitely try libcurl in multi mode -- it's quite possible to > >> implement > >> >> a > >> >> > high concurrency http client in this model using async events -- > you > >> can > >> >> use > >> >> > libevent for this. > >> >> > > >> >> > The ideal library would probably look something like this: > >> >> > > >> >> > os_request *req = os_request_create(); > >> >> > req->locale = ... > >> >> > os_request_add_gadget(req, "http://example.org/gadget.xml";); > >> >> > os_get_data(req, &callback); > >> >> > > >> >> > void callback(os_request *req, os_response *resp) {...} > >> >> > > >> >> > Rendering could then be implemented as a consumer of this library. > >> >> > > >> >> > > >> >> > > >> >> >> > >> >> >> thanks to all, > >> >> >> leonardo > >> >> >> > >> >> > > >> >> > >> > > >> > > >
