On Tue, Jun 24, 2008 at 3:57 PM, Leonardo Foderaro <[EMAIL PROTECTED]> wrote:
> I didn't try to integrate with libevent yet, I have to read something > about it first. > but there's a thing I didn't understand.. can't I simply collect all > the urls I need to fetch (for a single gadget) at once and then pass > them to the curl_multi (without libevent) ? > it could became a bottleneck, right? > however I'll try to adapt to libevent my example. I suppose this depends on the overall model. You'll want to support simultaneous processing of multiple gadgets, right? If so, then you'll want to be able to do this: - add all gadgets to curl and start processing - when a gadget is returned (not all files will come back at the same speed due to remote host differences and the like -- some may even be cached if you support that), you'll want to parse it and extract the referenced urls - add the referenced urls to curl and continue processing. * keep processing until nothing remains in the queue, then render. > > > thanks > leo > > On Tue, Jun 24, 2008 at 6:48 PM, Kevin Brown <[EMAIL PROTECTED]> wrote: > > On Tue, Jun 24, 2008 at 8:42 AM, Leonardo Foderaro < > [EMAIL PROTECTED]> > > wrote: > > > >> Hi Kevin, > >> I followed your suggest and played with libcurl in multi mode.. it's > cool ! > >> I tried some examples found in the net.. but I need more flexibility > >> (e.g. an arbitrary number of easy handles) > >> So I wrote a simple wrapper to deal with the async curl calls with > >> ease, 'cmhw' (Curl Multi Handle Wrapper) > >> The idea is to have a component which take care of all the aspects of > >> the multi curling, > >> you just need to create an instance and add the urls you want to > >> fetch, and it just do the job. > >> it's an early beta version, there's a lot of work which should be done > >> (e.g. all the possible error checking, > >> code optimization / safety checks, a better api interface, a better > >> name... ) but it works. > >> could it be useful for the gadget processing library? > >> > >> an example: > >> > >> /* our main struct. it holds the curl multi handle and the hash of > >> all the needed resources */ > >> cmhw_context_t *ctx = NULL; > >> > >> /* represents a single remote resource. holds an 'easy' curl handle > >> and a chunk of memory for the fetched content data */ > >> cmhw_item_t *item = NULL; > >> > >> /* init... */ > >> ctx = cmhw_init(); > >> > >> /* add some request (with an optional name, just to get them later > >> with ease) */ > >> cmhw_add_url(ctx, "google", "http://www.google.com";); > >> cmhw_add_url(ctx, "myspace", "http://www.myspace.com";); > >> > >> /* download them all! */ > >> cmhw_perform(ctx); > >> > >> /* make something interesting with our results */ > >> item = cmhw_item_get(ctx, "google"); > >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, > >> item->url, item->chunk->memory); > >> > >> item = cmhw_item_get(ctx, "myspace"); > >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, > >> item->url, item->chunk->memory); > >> > >> /* cleanup.. like a nice application... */ > >> cmhw_cleanup(ctx); > >> > >> > >> please let me know what you think about this and how I could improve it. > > > > > > One thing that might be an issue is that gadget specs frequently require > > fetching additional files (message bundles, preloads, etc.) One nice > aspect > > of multi mode is that you can always add new requests at any time, so if > you > > set up an async dispatching loop (using libevent), you can get very high > > concurrency rates. > > > > > >> > >> thanks > >> leo > >> > >> > >> > >> > >> > >> On Mon, Jun 23, 2008 at 1:42 AM, Kevin Brown <[EMAIL PROTECTED]> wrote: > >> > On Sun, Jun 22, 2008 at 3:16 PM, Leonardo Foderaro < > >> [EMAIL PROTECTED]> > >> > wrote: > >> > > >> >> Hi, > >> >> I'm trying to write a simple prototype of the library we talked about > >> >> some days ago. > >> >> It is a really early version at the moment.. just some experiments > (of > >> >> course the lucky codename is test.c ) > >> >> > >> >> What I'm using: > >> >> - libcurl for the remote content fetching > >> >> - libxml2 for the gadget xml parsing > >> >> - some data structures from the glibc2.0 (e.g. GHashTable) for > storing > >> >> dynamic variables > >> >> - a lexical scanner (generated using flex) for the hangman > substitution > >> >> > >> >> Actually I'm able to fetch a gadget from a remote URL (or a local > >> >> file), load it into the XmlParser and perform the hangman vars > >> >> substitution of the Content node through the flex scanner (only > >> >> __MODULE_ID__ for now). > >> >> > >> >> Of course there's a lot of work to do. > >> >> > >> >> Let me know what could be the next logical step.. I think I should > >> >> construct a C representation of the gadget with all its properties, > >> >> right? (it just writes to stdout at the moment) > >> > > >> > > >> > Yeah, you'll probably want a struct similar to the JSON output from > >> > /metadata in Shindig. As a library user / module writer, that's what > I'd > >> > want to consume. > >> > > >> > Definitely try libcurl in multi mode -- it's quite possible to > implement > >> a > >> > high concurrency http client in this model using async events -- you > can > >> use > >> > libevent for this. > >> > > >> > The ideal library would probably look something like this: > >> > > >> > os_request *req = os_request_create(); > >> > req->locale = ... > >> > os_request_add_gadget(req, "http://example.org/gadget.xml";); > >> > os_get_data(req, &callback); > >> > > >> > void callback(os_request *req, os_response *resp) {...} > >> > > >> > Rendering could then be implemented as a consumer of this library. > >> > > >> > > >> > > >> >> > >> >> thanks to all, > >> >> leonardo > >> >> > >> > > >> > > >
