Hello, I have a small modification to David's patch that improves the loading time about 5-fold compared to his latest version, from 5 seconds to less than 1 second on my laptop.
The trick used is to bring the whole list of dependencies for the entire recipe list in a single call, and do the match manually, instead of getting a call per recipe. This approach allows the code to perform constant lookup times independent of number of rows. Since I can't push to contrib on my laptop (something funky about keys), I'm attaching the patch for review. Please let me know how it goes. Cheers, Alex On Fri, Apr 11, 2014 at 5:26 PM, Reyna, David <[email protected]>wrote: > Hi Belén, > > > page still seems to be performing much worse than all other pages. Should > > we be trying to fix the root cause of the problem, ie. the "100*(2+2) > > foreign key lookups and filters/count"? > > If we stay with the current design of the page, then I think the solution > is along the lines: > > 1. Add two fields to the Recipe class to hold the respective forward and > reverse counts to support instant lookup. These values are fixed when the > build completes so they can be consider constant for our purposes, plus > they would add very little overhead to the database (if you can get this > proposal past Alex). An alternate implementation to keep the database > pristine is to add the ability to cache information like this in some > runtime table, so that the calculation time is only spent once. > > 2. Add a dynamic lookup of a given forward or reverse dependency list > when one of those buttons are clicked. In this manner we only spend the > list computation time when explicitly asked, not for every recipe whether > they want it or not. > > - David > > > -----Original Message----- > > From: Barros Pena, Belen [mailto:[email protected]] > > Sent: Friday, April 11, 2014 5:41 AM > > To: Reyna, David; Lerner, Dave > > Cc: [email protected] > > Subject: Re: [review-request] V2: 6137 excessive load time for All > Recipes > > page > > > > On 11/04/2014 01:44, "Reyna, David" <[email protected]> wrote: > > > > >Here are the timing results on my slow host for the rendering time: > > > > > > (a) Original: 13 seconds > > > (b) V1 : 7 seconds > > > (c) V2 : 4 seconds > > > > This is obviously a huge improvement. What bothers me is that the Recipes > > page still seems to be performing much worse than all other pages. Should > > we be trying to fix the root cause of the problem, ie. the "100*(2+2) > > foreign key lookups and filters/count"? > > > > Thanks! > > > > Belén > > > > -- > _______________________________________________ > toaster mailing list > [email protected] > https://lists.yoctoproject.org/listinfo/toaster > -- Alex Damian Yocto Project SSG / OTC
From 520ebcc860ce11cbb3751cb8cdc4e7a0c22a7bb5 Mon Sep 17 00:00:00 2001 From: David Reyna <[email protected]> Date: Tue, 8 Apr 2014 22:25:13 -0700 Subject: [PATCH 1/1] bitbake: toaster: reduce redundant foreign key lookups Replace redundant foreign key lookups with "with" to improve all recipes page load time. Modified by Alex to improve loading time. [YOCTO #6137] Signed-off-by: David Reyna <[email protected]> Signed-off-by: Alex D <[email protected]> --- .../lib/toaster/toastergui/templates/recipes.html | 20 ++++++++++++++------ bitbake/lib/toaster/toastergui/views.py | 18 ++++++++++++++++++ 2 files changed, 32 insertions(+), 6 deletions(-) diff --git a/bitbake/lib/toaster/toastergui/templates/recipes.html b/bitbake/lib/toaster/toastergui/templates/recipes.html index 907b83d..791a487 100755 --- a/bitbake/lib/toaster/toastergui/templates/recipes.html +++ b/bitbake/lib/toaster/toastergui/templates/recipes.html @@ -45,31 +45,39 @@ <td><a href="{% url "recipe" build.pk recipe.pk %}">{{recipe.version}}</a></td> <!-- Depends --> <td class="depends_on"> - {% if recipe.r_dependencies_recipe.all.count %} + {% with deps=recipe_deps|get_dict_value:recipe.pk %} + {% with count=deps|length %} + {% if count %} <a class="btn" title="<a href='{% url "recipe" build.pk recipe.pk %}#dependencies'>{{recipe.name}}</a> dependencies" data-content="<ul class='unstyled'> - {% for i in recipe.r_dependencies_recipe.all|dictsort:"depends_on.name"%} + {% for i in deps|dictsort:"depends_on.name"%} <li><a href='{% url "recipe" build.pk i.depends_on.pk %}'>{{i.depends_on.name}}</a></li> {% endfor %} </ul>"> - {{recipe.r_dependencies_recipe.all.count}} + {{count}} </a> {% endif %} + {% endwith %} + {% endwith %} </td> <!-- Brought in by --> <td class="depends_by"> - {% if recipe.r_dependencies_depends.all.count %} + {% with revs=recipe_revs|get_dict_value:recipe.pk %} + {% with count=revs|length %} + {% if count %} <a class="btn" title="<a href='{% url "recipe" build.pk recipe.pk %}#brought-in-by'>{{recipe.name}}</a> reverse dependencies" data-content="<ul class='unstyled'> - {% for i in recipe.r_dependencies_depends.all|dictsort:"recipe.name"%} + {% for i in revs|dictsort:"recipe.name" %} <li><a href='{% url "recipe" build.pk i.recipe.pk %}'>{{i.recipe.name}}</a></li> {% endfor %} </ul>"> - {{recipe.r_dependencies_depends.all.count}} + {{count}} </a> {% endif %} + {% endwith %} + {% endwith %} </td> <!-- Recipe file --> <td class="recipe_file">{{recipe.file_path}}</td> diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/toastergui/views.py index e4ada14..e22e87c 100644 --- a/bitbake/lib/toaster/toastergui/views.py +++ b/bitbake/lib/toaster/toastergui/views.py @@ -1037,10 +1037,28 @@ def recipes(request, build_id): recipes = _build_page_range(Paginator(queryset, request.GET.get('count', 100)),request.GET.get('page', 1)) + # prefetch the forward and reverse recipe dependencies + deps = { }; revs = { } + queryset_dependency=Recipe_Dependency.objects.filter(recipe__in=map(lambda x: x.id, recipes.object_list)); + for recipe in recipes: + deplist = [ ] + for recipe_dep in [x for x in queryset_dependency if x.recipe_id == recipe.id]: + deplist.append(recipe_dep) + deps[recipe.id] = deplist + + queryset_dependency=Recipe_Dependency.objects.filter(depends_on__in=map(lambda x: x.id, recipes.object_list)); + for recipe in recipes: + revlist = [ ] + for recipe_dep in [x for x in queryset_dependency if x.depends_on_id == recipe.id]: + revlist.append(recipe_dep) + revs[recipe.id] = revlist + context = { 'objectname': 'recipes', 'build': Build.objects.filter(pk=build_id)[0], 'objects': recipes, + 'recipe_deps' : deps, + 'recipe_revs' : revs, 'tablecols':[ { 'name':'Recipe', -- 1.8.3.2
-- _______________________________________________ toaster mailing list [email protected] https://lists.yoctoproject.org/listinfo/toaster
