The Recipe Reporting System (RRS - live instance at http://recipes.yoctoproject.org) was originally developed as a fork of the layer index (live instance at http://layers.openembedded.org) - this set of changes rebases it on top of master and makes the functionality possible to activate for other layers. I've introduced the concept of a "Maintenance Plan" to tie together the recipe maintenance information for one or more layers and used it to replace all the parts that assumed Poky or OE-Core were the repositories/layers being tracked. I've also made it possible again for the application to gather data going back to 1.6 across the python 2/3 divide, and made various cleanups and optimisations in the code.
Along the way I have added some capabilities to the layer index that the RRS makes use of, including source URL and patch tracking. These were re-implemented with a more detailed structure rather than using what had been done in the rrs branch previously. The patch tracking will import patch upstream status, and the RRS can present some simple graphs using the data. There are quite a lot of patches in this patchset, thus I have sent the cover letter only. The first part consists of rebased commits from the rrs branch, mostly untouched; then my changes follow. The RRS remains a separate logical application which is disabled by default in settings.py - see README.rrs for setup instructions. The following changes since commit b4cfb049d9fda7fe82f97f8532abdb7272cf0a76: requirements.txt: bump Django and other dependency versions (2018-05-01 10:10:21 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/rrs-rebase http://git.yoctoproject.org/cgit.cgi//log/?h=paule/rrs-rebase Aníbal Limón (120): rrs: Add application rrs: Add README.rrs to help in setup. rrs: Add milestone selection bar for recipes and milestone. rrs: Add support for store RecipeUpstreamHistory rrs: Add initial support of recipes page rrs: Recipes add url validation rrs: Add support for Recipe detail page rrs: Recipes page improve table display rrs: Add support for filter in recipes page. README.rrs: Add instruction for serve static content in development rrs: Add support for Recipe maintainership history rrs/models.py: fix RecipeMaintainerHistory get_by_end_date. rrs: RecipeDetail view fix rrs/views.py: Recipes modularize improve milestone statistics rrs: Add Maintainer statistics page. rrs: Recipes when upstream_status is Downgrade display as Unknown templates/rrs/maintainers.html: Change the link from maintainer ownership from name to assigned recipes. rrs: Maintainers add support for display statistics of recipe upgrades by week in Milestone. rrs: Now display recipes when not found upstream info. rrs: Frontend improvments rrs/fixtures/initial_data.json: Improve Milestone names. rrs: Add support for display Can't be updated recipe upstream status. rrs: Improve recipes filtering rrs: Maintainers page fix maintainer count rrs: Maintainer statistics frontend improvments templates/rrs/recipes.html: Add button for search rrs: Add support to display version by Milestone based on RecipeUpgrade's rrs: Recipes don't display if don't exit in Milestone rrs: Add support for Release rrs: Add support for export Recipes reports by Release/Milestone rrs/fixtures/initial_data.json: Fix end_date of 1.8All Milestone rrs: Maintainers add support for display Milestone intervals rrs: Add missing __init__.py in migrations templates/rrs/{recipes, maintainers}.html: Improvment in alert div no records rrs/models.py: RecipeMaintainerHistory if not found return the first record rrs/views.py: Now if a recipe have history but not upstream info set to Unknown state. rrs: Frontend improvments rrs: Recipes frontend improvments clean search when select filter rrs: Add support to display sort status Recipes and Maintainers UI rrs: Add img assets used to display sort status templates/rrs/base_toplevel.html: Remove Recipes word in statistics bar rrs: Add support to display no update reason rrs: UI add clear search button in Recipes and Maintainers rrs: Maintainers statistics fixed sorting of weeks rrs: Release/Milestone improvments rrs/views.py: Recipes remove unnedded extra_url_params templates/rrs/recipes.html: Fix url params in JS templates/rrs/recipes.html: Only apply sort to Recipe column at load rrs/fixtures/initial_data.json: Update dates from Release/Milestone rrs: RecipeDetail show milestone instead of all in upgrades table templates/rrs: Recipes and Maintainers UI search improvments rrs: Add wsgi file for serve with gunicorn rrs: Add database indexes in models.py rrs: Add missing date indexes in RecipeUpstreamHistory rrs: Update uitablefilter README.rrs: Add Maintaince infomation requirements: Add tabular. conf/local.conf: Add RRS configuration commented. rrs: jquery.tablesorter.js remove ^M characters. rrs/tools/common.py: Add library with common functionality. rrs/tools: Add recipe maintainer history script. rrs/models.py: RecipeUpgrade fix selection by milestone. rrs/tools/common.py: Add get_pv_type function. rrs/tools/rrs_unique_recipes.py: Add script for filter recipes. rrs/tools/rrs_upgrade_history.py: Add script for update the recipe upgrades history. rrs/tools/rrs_: Reorder imports. rrs/tools: Add script for send email to maintainers. rrs/tools/rrs_unique_recipes.py: Remove wrong info at top comment. rrs/tools/common.py: Add load_recipes function. rrs/tools/rrs_upgrade_history.py: Improve version detection. rrs/tools/common.py: Add get_logger function. rrs/tools/rrs_unique_recipes.py: Change to use get_logger from common. rrs/tools/common.py: Handle exception when load_recipes caused by expand PV variable. rrs/tools/rrs_upgrade_history.py: Only run upgrade detection when detect recipe files in commit. rrs/tools: Change to use get_logger from common. rrs/tools/rrs_unique_recipes.py: Also filter -source recipes. rrs/tools.py: Create standalone get_recipe_pv_without_srcpv function. rrs/tools/common.py: Add import of FetchError and display parse errors rrs/tools/rrs_upgrade_history.py: WORKAROUND Run recipe parse as different proccess. settings.py: Add TOOLS_LOG_DIR for store RRS logs. rrs/tools/rrs_unique_recipes.py: Now only filter same pn recipes. rrs/views.py: RecipeList always use recipe from Recipe upgrades. rrs/tools/rrs_upstream_history.py: Add recipe upstream script. rrs/tools: Remove call to tinfoil shutdown method. rrs/tools: Convert unicode strings from Django models to strings. rrs/tools/rrs_upstream_history.py: Remove unneeded histroy save at end. rrs/tools: Add rrs_distros script. README.rrs: Updated update scripts section with new ones. rrs/tools/rrs_maintainer_history.py: Set new recipes to No maintainer if don't have rrs/views.py: RecipeListView always use recipes from history. rrs/tools/rrs_upgrade_history.py: Don't take into account git recipes without versioning. rrs/tools: Add daily_run.sh script for provide updates. rrs: Fix last updated. rrs/tools: get_recipe_pv_without_srcrev add support for discard prefixes rrs/tools: UpstreamHistory only load current recipe files. rrs_upstream_history.py: Use regexes in SPECIAL_PKGSUFFIX packages templates/rrs/recipes.html: Re-add the sort property into Maintainer. rrs_upstream_history.py: Fix use regexes in packages rrs/urls.py: Disable permanent (301) redirection on main view. rrs/recipes: Add support for persist search and filters. rrs/tools/rrs_upstream_email.py: Update message to point recipe file instead of deprecated upstream tracking file. rrs: Fix frontpage redirect tools/rrs_distros.py: Fix distro_check.create_distro_packages_list call rrs/tools/rrs_maintainer_history.py: Update to point new meta-poky repository. rrs: Frontend update to work with Python3 and django layerindex,rrs: Minor changes to work with python3. rrs/tools/common.py: common_setup() Add layerindex to the path. rrs/tools/rrs_upgrade_history.py: Fix import of split_recipe_fn rrs/tools/common.py: When load a recipe use tinfoil cache templates/rrs/recipedetail.html: Fix unneeded quotes in url call rrs/tools: Upgrade to use transaction.atomic() in Django 1.6 rrs/tools: Upgrade to use new API of tinfoil rrs/tools/common.py: Fix typo when load recipes rrs/tools/rrs_upgrade_history.py: Remove mp process related code rrs/tools/rrs_distros.py: Don't open files in binary mode rrs/tools/rrs_upstream_history.py: Disable threads due to new tinfoil API rrs/views.py: When export to csv use release_milestone.csv templates/rrs/recipedetail.html: Change Maintainer to Committer rrs/tools/rrs_upstream_history.py: Change regex variables to tuple rrs/tools/rrs_{distros,upstream_history}: Add path for poky meta lib Belen Barros Pena (7): readme: add missing instruction Remove milestone name from recipe details page rrs: Set of UI improvments rrs: remove sorting from the 'No update reason' column rrs: avoid too many columns in the recipes table Update tool URL in status email Show the 'Upstream version' column only when relevant Mariano Lopez (21): rrs/views.py: Fix exception in Maintainer view templates/rrs/maintainers.html: Added total row at the end of the table. templates/rrs/maintainers.html: Added links to recipe page. rrs/static/css/rrs-additional.css: Added formating to the table footer. views.py: Added validation in recipe detail rrs/models.py: Added raw SQL calls rrs/views.py: Changed ORM to raw SQL calls views.py: Add how long a recipe hasn't been updated views.py: Fix version in current milestone views.py: Fix current up to date percentage models.py: Added Raw SQL for percentage updated views.py: Changed the behavior of percentage done base_toplevel.html: Changed format of percentage done views.py: Changed date format to iso recipes.html: Add Last Updated column models.py: Code cleanup views.py: Show last updated date for all recipes rrs-additional.css: Added min-width to last update column recipes.html: Format change for last updated column rrs/views.py: Added percentages for navbar rrs/base_toplevel.html: Navbar redesign rrs/views: Fix mismatch in summary and recipe list Paul Eggleton (94): layerindex: Add support for rrs in settings.py and urls.py about: add RRS contributors rrs/models: move Raw class to views.py rrs/tools/rrs_unique_recipes: drop Drop "load url from future" Rename RRS URLs to have rrs_ prefix Use this layer index's URL for layer branch rrs/tools: add dry-run option to each script rrs: drop old migrations rrs: add initial migration rrs: add maintenance plans rrs: add Release link to MaintenancePlan rrs: show full name in admin rrs/tools: avoid unnecessary checkouts rrs_upgrade_history.py: add maintenance plan handling rrs: fix model string representations for Python 3 utils: decode command output in runcmd() as UTF-8 rrs/tools/common.py: import sys/os at top rrs_maintainer_history.py: support maintenance plans and remove poky hardcoding rrs_upstream_history.py: drop threading rrs_upstream_history.py: support maintenance plans and remove poky hardcoding rrs_upgrade_history: drop unused import rrs_upstream_history: properly handle missing Recipe rrs/tools: print debug message when parsing recipes rrs_distros: support maintenance plans and remove poky hardcoding rrs/tools: drop update_repo() rrs/models: add fields for more flexible email handling rrs_upstream_email: rework rrs_upstream_email: Adapt to template rendering API change rrs: handle maintenance plans in views rrs/models: handle All missing in milestones rrs/views: fix case in table names rrs/views: fix SQL parameter in get_reup_by_last_updated() rrs: replace fixtures with initial data migration rrs: add maintenance plan selection rrs: use default milestone instead of "All" for release drop-down Add link to maintenance plan in layer detail rrs: handle dependency field differences rrs: add titles to maintainers/recipes page rrs: add breadcrumb to recipe detail page rrs/tools: run all tools scripts with python3 Move run_command_interruptible() to utils rrs_upgrade_history.py: enable to work across python2/3 versions rrs_upgrade_history.py: ensure we shut down tinfoil safely common: ensure we pass a string to loadDataFull() rrs_upgrade_history: add a --fullreload option rrs_upgrade_history: handle multiple recipes with same pn per layer rrs_upgrade_history: use date/commit of last recipe upgrade import rrs_upstream_history: use try...finally to ensure tinfoil is shut down rrs_upgrade_history: only look at commits that actually change recipes rrs/tools: use layer index lock rrs_upgrade_history: handle broken version numbers rrs_upgrade_history: show text of any exception that blocks parsing rrs_upgrade_history: ignore files outside of the layer rrs_upgrade_history: Add debug option to process specific commit README.rrs: update rrs/tools: ensure recipe parsing code deletes temporary dir rrs_upgrade_history: handle only .inc file changing rrs_upgrade_history: drop final clean rrs_maintainer_history: add --fullreload option rrs_maintainer_history: avoid unnecessary checkouts Replace mimetype with content_type rrs: handle linking maintainership rrs: Add link to layer detail to breadcrumb on recipe detail Add a link from the Tools drop-down to the RRS if enabled admin: use more dynamic method of setting recipe read-only fields Save recipe source URLs rrs: support new source structure in recipe detail Implement layer web repo commit URL models: implement layerindex method to get recursive dependencies Fix layer description not wrapping properly utils: add common function to check out a specific git revision rrs_upgrade_history: improve checkout logic Turn URLs into links in layer notes rrs: restore full-width pages Implement patch tracking rrs: add patch listing to recipe list and detail rrs: add charts page rrs: releases should be unique by plan and name, not just name rrs: default python2/3 environments for new maintenance plan layer branches rrs: duplicate releases from first plan when adding a new plan rrs: improve admin for Release/Milestone objects rrs/tools: add -p/--plan option rrs: drop a couple of unused functions from Raw class rrs: link maintenance/upstream history to layerbranch rrs/urls: Use new urlpatterns list syntax templates/rrs: replace use of = with == rrs: validate that a layerbranch is only part of one plan rrs_upgrade_history: skip commits that don't touch the layer rrs_upstream_history: make more robust rrs: add flag to MaintenancePlan to specify layer-wide maintainers rrs: admin: validate that email address fields are set rrs: fix unique constraint on RecipeMaintainerHistory sha1 field TODO: add RRS items README.rrs | 61 ++ TODO | 19 + conf/local.conf | 7 + layerindex/admin.py | 19 +- layerindex/context_processors.py | 4 +- layerindex/forms.py | 9 +- layerindex/migrations/0011_source.py | 22 + .../migrations/0012_layeritem_vcs_commit_url.py | 37 + layerindex/migrations/0013_patch.py | 28 + layerindex/models.py | 82 ++ layerindex/recipeparse.py | 6 +- layerindex/static/css/additional.css | 1 + layerindex/tools/import_layer.py | 10 +- layerindex/tools/import_wiki_layers.py | 11 +- layerindex/update.py | 35 +- layerindex/update_layer.py | 98 +- layerindex/utils.py | 63 +- layerindex/views.py | 7 +- rrs/__init__.py | 2 + rrs/admin.py | 186 ++++ rrs/apps.py | 6 + rrs/context_processors.py | 19 + rrs/migrations/0001_initial.py | 337 +++++++ rrs/migrations/0002_maintenanceplan.py | 33 + rrs/migrations/0003_release_plan.py | 19 + rrs/migrations/0004_maint_plan_default.py | 39 + rrs/migrations/0005_release_plan_nonnull.py | 19 + rrs/migrations/0006_maintplan_email.py | 41 + rrs/migrations/0007_python23.py | 31 + rrs/migrations/0008_upgrade_info.py | 24 + rrs/migrations/0009_rmh_layerbranch.py | 20 + rrs/migrations/0010_recipemaintenancelink.py | 40 + rrs/migrations/0011_release_name_unique.py | 23 + rrs/migrations/0012_reup_layerbranch_field.py | 19 + rrs/migrations/0013_reup_layerbranch_populate.py | 29 + rrs/migrations/0014_reup_layerbranch_nonnull.py | 19 + rrs/migrations/0015_rmh_layerbranch_populate.py | 29 + rrs/migrations/0016_rmh_layerbranch_nonnull.py | 19 + .../0017_maintenanceplan_maintainer_style.py | 20 + rrs/migrations/0018_rmh_unique.py | 25 + rrs/migrations/__init__.py | 0 rrs/models.py | 453 +++++++++ rrs/static/css/rrs-additional.css | 125 +++ rrs/static/img/small.gif | Bin 0 -> 60 bytes rrs/static/img/small_asc.gif | Bin 0 -> 53 bytes rrs/static/img/small_desc.gif | Bin 0 -> 53 bytes rrs/static/js/jquery.tablesorter.js | 1031 ++++++++++++++++++++ rrs/static/js/uitablefilter.js | 117 +++ rrs/tools/common.py | 154 +++ rrs/tools/daily_run.sh | 16 + rrs/tools/rrs_distros.py | 162 +++ rrs/tools/rrs_maintainer_history.py | 234 +++++ rrs/tools/rrs_upgrade_history.py | 230 +++++ rrs/tools/rrs_upstream_email.py | 187 ++++ rrs/tools/rrs_upstream_history.py | 236 +++++ rrs/tools/upgrade_history_internal.py | 295 ++++++ rrs/urls.py | 33 + rrs/views.py | 939 ++++++++++++++++++ rrs/wsgi.py | 6 + settings.py | 3 + templates/base.html | 5 +- templates/layerindex/about.html | 3 + templates/layerindex/detail.html | 15 +- templates/layerindex/editlayer.html | 14 + templates/layerindex/recipedetail.html | 22 + templates/rrs/base_toplevel.html | 148 +++ templates/rrs/maintainers.html | 265 +++++ templates/rrs/recipedetail.html | 233 +++++ templates/rrs/recipes.html | 353 +++++++ templates/rrs/report_email.txt | 26 + templates/rrs/rrs_stats.html | 48 + urls.py | 10 + 72 files changed, 6810 insertions(+), 71 deletions(-) create mode 100644 README.rrs create mode 100644 layerindex/migrations/0011_source.py create mode 100644 layerindex/migrations/0012_layeritem_vcs_commit_url.py create mode 100644 layerindex/migrations/0013_patch.py create mode 100644 rrs/__init__.py create mode 100644 rrs/admin.py create mode 100644 rrs/apps.py create mode 100644 rrs/context_processors.py create mode 100644 rrs/migrations/0001_initial.py create mode 100644 rrs/migrations/0002_maintenanceplan.py create mode 100644 rrs/migrations/0003_release_plan.py create mode 100644 rrs/migrations/0004_maint_plan_default.py create mode 100644 rrs/migrations/0005_release_plan_nonnull.py create mode 100644 rrs/migrations/0006_maintplan_email.py create mode 100644 rrs/migrations/0007_python23.py create mode 100644 rrs/migrations/0008_upgrade_info.py create mode 100644 rrs/migrations/0009_rmh_layerbranch.py create mode 100644 rrs/migrations/0010_recipemaintenancelink.py create mode 100644 rrs/migrations/0011_release_name_unique.py create mode 100644 rrs/migrations/0012_reup_layerbranch_field.py create mode 100644 rrs/migrations/0013_reup_layerbranch_populate.py create mode 100644 rrs/migrations/0014_reup_layerbranch_nonnull.py create mode 100644 rrs/migrations/0015_rmh_layerbranch_populate.py create mode 100644 rrs/migrations/0016_rmh_layerbranch_nonnull.py create mode 100644 rrs/migrations/0017_maintenanceplan_maintainer_style.py create mode 100644 rrs/migrations/0018_rmh_unique.py create mode 100644 rrs/migrations/__init__.py create mode 100644 rrs/models.py create mode 100644 rrs/static/css/rrs-additional.css create mode 100644 rrs/static/img/small.gif create mode 100644 rrs/static/img/small_asc.gif create mode 100644 rrs/static/img/small_desc.gif create mode 100644 rrs/static/js/jquery.tablesorter.js create mode 100644 rrs/static/js/uitablefilter.js create mode 100644 rrs/tools/common.py create mode 100755 rrs/tools/daily_run.sh create mode 100755 rrs/tools/rrs_distros.py create mode 100755 rrs/tools/rrs_maintainer_history.py create mode 100755 rrs/tools/rrs_upgrade_history.py create mode 100755 rrs/tools/rrs_upstream_email.py create mode 100755 rrs/tools/rrs_upstream_history.py create mode 100644 rrs/tools/upgrade_history_internal.py create mode 100644 rrs/urls.py create mode 100644 rrs/views.py create mode 100644 rrs/wsgi.py create mode 100644 templates/rrs/base_toplevel.html create mode 100644 templates/rrs/maintainers.html create mode 100644 templates/rrs/recipedetail.html create mode 100644 templates/rrs/recipes.html create mode 100644 templates/rrs/report_email.txt create mode 100644 templates/rrs/rrs_stats.html -- 2.14.3 -- _______________________________________________ yocto mailing list [email protected] https://lists.yoctoproject.org/listinfo/yocto
