Hi Matej, mato konecny wrote: > Hi Olivier > > Hi Matej, > > mato konecny wrote: > > Hi all, > > > > First of all, I am not Twisted nor Python expert, so this question is > > maybe trivial. The question is related to an IMDB plugin. > > Anyway, I created a layer between my resource provider (that does some > > HTTTP operations and returns deferred) and the controllers. This is > > mainly for caching and convenience purposes and works quite OK. What I > > do, I create a singleton of my extra layer (retriever) which then > > internally calls resource provider. The problem is that if user > has for > > example 20 movies in library, that means 20 defereds are fired almost > > instantly, resulting in 20 HTTP GETs (and some more because of the > > resource provider implementation). > > This is also bit annoying for user because then the responses to > the GET > > requests arrive asynchronously and therefore I want to implement a > > queueing mechanism. I got some inspiration from MediaScanner, so > create > > an object, in that one create deferred and put this in the queue and > > return created deferred. When the time comes, the object is taken > out of > > the queue, processed and (here comes my problem) how can I signal the > > original callee that "it's finished". > > Maybe this needs some code example, probably it's bit too long to post > > it here, but I can add it later if it clarifies what I am talking > about :) > > What I don't really get is why you would want to queue requests and make > an asynchronous process synchronous. Elisa uses asynchrony to provide > the user a responsive UI and it is perfectly fine to receive delayed > answers when populating some part of the UI. Just make sure you display > some default image e.g. while waiting for the answer. > > I already have this mechanism in place, the UI updates when the deferred > is fired. The issue is that I just retrieve all deferreds and all of > them start at the same time. This means 20x HTTP GET (search query), > after that parsing, another HTTP GET (get movie query), parsing, another > HTTP GET (get poster query) and then the deferred is "finished". The > second and 3rd GETs are taking time randomly and instead of getting all > data and poster of movie 1, then movie 2 etc I get part of data from > movie 1, then search result from movie 2, poster result from movie 3. > This is not the bigges issue though, scan is done only once and the > results are cached. I would like to cancel the deferreds if some of the > nodes in the GridView is clicked. There are then 2 possibilities: keep > hash of all deferreds and cancel them; or create a "queue" of incomming > requests and defer them one after eachother, when we need to cancel, we > just clear the queue.
Now I understand your dilemma. What we chose to do in several places in the code of Elisa with a similar concern is the first solution: keep a dictionary of all the currently pending deferreds, and cancel them all when entering a new controller. The queuing solution looks ok too, I don't see any particular contraindication to implementing it. As far as I know we don't have any helper code to do this though, so you'll have to figure that out yourself (if you can come up with a good generic solution maybe we could merge it back into our development branch). > As far as I understand, you don't need a queuing mechanism, just to > connect callbacks to your deferreds that will populate your UI (or take > whatever action needed) when fired. > > If I misunderstood your problem don't hesitate to ask again, maybe with > a simple use case and/or code snippets. > > OK, let me write some pseudocode, the full code is in my bazaar branch > (lp:~konecm/elisa/movie-library) It's bit longer but I think it might > help explaining... The mentioned branch contains the "stable" code where > everything is done the neat "deferred way" :) > > class IMDBResourceProvider(...): .... get() .... > > class MovieDataRetriever(singleton): > def enqueue(item, signal_done_cb) > item.dfr=Deferred() > set item in a queue > start scanning the queue if not started > > def _scan() > def _done() > item.signal_done_cb() > if not queue empty: > next queue item > > queue.get(item) > dfr=resourceprovider.get(item)... > dfr.addCallback(_done) > > > class ...(hieararchycontroller) > def node_clicked() > movieDataRetriever.queue=Queue() > ... > > class ListViewMode(...): > def get_image() > dfr = Deferred() > > def _done_cb(res) > dfr.callbacks(res) > > def _update_ui(res) > ...update UI code... > movieDataRetriever.enqueue(item, _done_cb) > dfr.addCallback(_update_ui) > > So, this is what I was able to get up and running, everything is now > processed one after eachother, I can cancel everything, let user see > which element is being currently processed... I am not very happy with > the solution though, the original "all deferred" solution was cleaner, > but to cancel deferreds I would have to create hash of all deferreds in > the retriever and then operate on that one, ending up with incomplete > models etc. > > One last thing - after clicking a node, the movie is not played, but > detailed information is shown, so if there are 20 movies being > processed, then user will wait pretty long time just to get info about one. Yes, simply unacceptable from a user's standpoint. > > Cheers > Matej Cheers, Olivier
