ok I'll try to put all this pieces together, maybe packaged in a simple apache module.
On Wed, Jun 25, 2008 at 6:52 PM, Kevin Brown <[EMAIL PROTECTED]> wrote: > On Wed, Jun 25, 2008 at 9:14 AM, Leonardo Foderaro <[EMAIL PROTECTED]> > wrote: > >> maybe I'm thinking in the wrong way... 1 gadget ---> 1 http request ( >> /gadgets/ifr?url=http://host/gadget.xml) ----> many http fetch (the >> gadget xml, bundles, messages, etc..) >> >> however, now my example supports libevent. >> >> it looks like this: >> >> cmhw_add_url(ctx, "gadget1", "http://host/gadget1.xml";); >> cmhw_add_url(ctx, "gadget2", "http://host/gadget2.xml";); >> >> do { >> while (CURLM_CALL_MULTI_PERFORM == >> curl_multi_socket_all(ctx->multi_handle, (ctx->unfinished))); >> event_dispatch(); >> } while ((*ctx->unfinished) != -1); >> >> >> is this similar to your idea? > > > Yep, that's essentially what I'd do. This will allow you to avoid blocking > for all gadget downloads when fetching others. It's pretty much guaranteed > to win over doing everything serially. > > >> >> if so, I try to go one step further. >> >> thanks >> leo >> >> >> >> >> On Wed, Jun 25, 2008 at 1:25 AM, Kevin Brown <[EMAIL PROTECTED]> wrote: >> > On Tue, Jun 24, 2008 at 3:57 PM, Leonardo Foderaro < >> [EMAIL PROTECTED]> >> > wrote: >> > >> >> I didn't try to integrate with libevent yet, I have to read something >> >> about it first. >> >> but there's a thing I didn't understand.. can't I simply collect all >> >> the urls I need to fetch (for a single gadget) at once and then pass >> >> them to the curl_multi (without libevent) ? >> >> it could became a bottleneck, right? >> >> however I'll try to adapt to libevent my example. >> > >> > >> > I suppose this depends on the overall model. You'll want to support >> > simultaneous processing of multiple gadgets, right? If so, then you'll >> want >> > to be able to do this: >> > >> > - add all gadgets to curl and start processing >> > - when a gadget is returned (not all files will come back at the same >> speed >> > due to remote host differences and the like -- some may even be cached if >> > you support that), you'll want to parse it and extract the referenced >> urls >> > - add the referenced urls to curl and continue processing. >> > >> > * keep processing until nothing remains in the queue, then render. >> > >> > >> >> >> >> >> >> thanks >> >> leo >> >> >> >> On Tue, Jun 24, 2008 at 6:48 PM, Kevin Brown <[EMAIL PROTECTED]> wrote: >> >> > On Tue, Jun 24, 2008 at 8:42 AM, Leonardo Foderaro < >> >> [EMAIL PROTECTED]> >> >> > wrote: >> >> > >> >> >> Hi Kevin, >> >> >> I followed your suggest and played with libcurl in multi mode.. it's >> >> cool ! >> >> >> I tried some examples found in the net.. but I need more flexibility >> >> >> (e.g. an arbitrary number of easy handles) >> >> >> So I wrote a simple wrapper to deal with the async curl calls with >> >> >> ease, 'cmhw' (Curl Multi Handle Wrapper) >> >> >> The idea is to have a component which take care of all the aspects of >> >> >> the multi curling, >> >> >> you just need to create an instance and add the urls you want to >> >> >> fetch, and it just do the job. >> >> >> it's an early beta version, there's a lot of work which should be >> done >> >> >> (e.g. all the possible error checking, >> >> >> code optimization / safety checks, a better api interface, a better >> >> >> name... ) but it works. >> >> >> could it be useful for the gadget processing library? >> >> >> >> >> >> an example: >> >> >> >> >> >> /* our main struct. it holds the curl multi handle and the hash of >> >> >> all the needed resources */ >> >> >> cmhw_context_t *ctx = NULL; >> >> >> >> >> >> /* represents a single remote resource. holds an 'easy' curl handle >> >> >> and a chunk of memory for the fetched content data */ >> >> >> cmhw_item_t *item = NULL; >> >> >> >> >> >> /* init... */ >> >> >> ctx = cmhw_init(); >> >> >> >> >> >> /* add some request (with an optional name, just to get them later >> >> >> with ease) */ >> >> >> cmhw_add_url(ctx, "google", "http://www.google.com";); >> >> >> cmhw_add_url(ctx, "myspace", "http://www.myspace.com";); >> >> >> >> >> >> /* download them all! */ >> >> >> cmhw_perform(ctx); >> >> >> >> >> >> /* make something interesting with our results */ >> >> >> item = cmhw_item_get(ctx, "google"); >> >> >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, >> >> >> item->url, item->chunk->memory); >> >> >> >> >> >> item = cmhw_item_get(ctx, "myspace"); >> >> >> fprintf(stderr, "content for '%s'(%s):\n%s\n#####\n\n", item->name, >> >> >> item->url, item->chunk->memory); >> >> >> >> >> >> /* cleanup.. like a nice application... */ >> >> >> cmhw_cleanup(ctx); >> >> >> >> >> >> >> >> >> please let me know what you think about this and how I could improve >> it. >> >> > >> >> > >> >> > One thing that might be an issue is that gadget specs frequently >> require >> >> > fetching additional files (message bundles, preloads, etc.) One nice >> >> aspect >> >> > of multi mode is that you can always add new requests at any time, so >> if >> >> you >> >> > set up an async dispatching loop (using libevent), you can get very >> high >> >> > concurrency rates. >> >> > >> >> > >> >> >> >> >> >> thanks >> >> >> leo >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Jun 23, 2008 at 1:42 AM, Kevin Brown <[EMAIL PROTECTED]> >> wrote: >> >> >> > On Sun, Jun 22, 2008 at 3:16 PM, Leonardo Foderaro < >> >> >> [EMAIL PROTECTED]> >> >> >> > wrote: >> >> >> > >> >> >> >> Hi, >> >> >> >> I'm trying to write a simple prototype of the library we talked >> about >> >> >> >> some days ago. >> >> >> >> It is a really early version at the moment.. just some experiments >> >> (of >> >> >> >> course the lucky codename is test.c ) >> >> >> >> >> >> >> >> What I'm using: >> >> >> >> - libcurl for the remote content fetching >> >> >> >> - libxml2 for the gadget xml parsing >> >> >> >> - some data structures from the glibc2.0 (e.g. GHashTable) for >> >> storing >> >> >> >> dynamic variables >> >> >> >> - a lexical scanner (generated using flex) for the hangman >> >> substitution >> >> >> >> >> >> >> >> Actually I'm able to fetch a gadget from a remote URL (or a local >> >> >> >> file), load it into the XmlParser and perform the hangman vars >> >> >> >> substitution of the Content node through the flex scanner (only >> >> >> >> __MODULE_ID__ for now). >> >> >> >> >> >> >> >> Of course there's a lot of work to do. >> >> >> >> >> >> >> >> Let me know what could be the next logical step.. I think I should >> >> >> >> construct a C representation of the gadget with all its >> properties, >> >> >> >> right? (it just writes to stdout at the moment) >> >> >> > >> >> >> > >> >> >> > Yeah, you'll probably want a struct similar to the JSON output from >> >> >> > /metadata in Shindig. As a library user / module writer, that's >> what >> >> I'd >> >> >> > want to consume. >> >> >> > >> >> >> > Definitely try libcurl in multi mode -- it's quite possible to >> >> implement >> >> >> a >> >> >> > high concurrency http client in this model using async events -- >> you >> >> can >> >> >> use >> >> >> > libevent for this. >> >> >> > >> >> >> > The ideal library would probably look something like this: >> >> >> > >> >> >> > os_request *req = os_request_create(); >> >> >> > req->locale = ... >> >> >> > os_request_add_gadget(req, "http://example.org/gadget.xml";); >> >> >> > os_get_data(req, &callback); >> >> >> > >> >> >> > void callback(os_request *req, os_response *resp) {...} >> >> >> > >> >> >> > Rendering could then be implemented as a consumer of this library. >> >> >> > >> >> >> > >> >> >> > >> >> >> >> >> >> >> >> thanks to all, >> >> >> >> leonardo >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> > >> >
