Re: [yocto] is there a rationale for YP using sysvinit as default init manager?
On Friday, 22 November 2019 9:40:35 AM NZDT Richard Purdie wrote: > On Thu, 2019-11-21 at 14:02 -0500, Robert P. J. Day wrote: > > don't get me wrong, i have no problem with that, but a colleague > > asked me what the reason was for using sysvinit as the *default*. i > > hemmed and hawed and suggested it was for simplicity and reliability, > > and that a lot of embedded systems didn't need the flashy features of > > systemd, and so on. > > > > is there any short answer to give to that question? > > The project existed before systemd and we haven't changed the default. > > We can change the default, it just means someone going through and > fixing the build failures it generates and rewriting all the test > metadata to invert poky and poky-altcfg. Personally I've got other > things I'd prefer to do... Kai was working on changing this in the last cycle but AFAICT not all issues were able to be resolved in time. Kai, do you have a status update? Thanks Paul -- Paul Eggleton Intel System Software Products -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 0/6] Recipe search improvements, misc fixes [cover letter only]
Some minor improvements to recipe searching, plus fixes for a couple of issues I noticed while doing the RRS rework. Note: this patchset is on top of paule/recipesymbol. The following changes since commit 6f85a1b4589df08e849a106338fe03d72e2394c2: TODO: add some more tasks (2019-11-21 02:51:30 +1300) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes14 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes14 Paul Eggleton (6): Drop LICENSE.diff2html update: fix exception with -x/--nofetch option recipes: improved support for queries containing quotes recipes: support pn: query prefix recipes: allow searching for layer:oe-core recipes: add help button to explain search terms layerindex/static/LICENSE.diff2html | 20 layerindex/update.py| 50 ++--- layerindex/views.py | 23 +++-- templates/layerindex/recipes.html | 22 + 4 files changed, 68 insertions(+), 47 deletions(-) delete mode 100644 layerindex/static/LICENSE.diff2html -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 00/29] RRS: rework upgrade history collection [cover letter only]
Taking a hard look at the recipe upgrade functionality, there were a number of shortcomings in how the data was collected which led to upgrades being missed, history disappearing when recipes were removed or renamed and scenarios such as multiple recipes with the same name not being handled correctly. This patchset introduces the concept of a RecipeSymbol (acting as an umbrella for RRS records for the same named recipe but independent of actual Recipe records which may be transient) and also fixes a number of issues in the upgrade data collection for much greater accuracy. The following changes since commit 50fc6780e022a95b452fdb6f57b4e1c6b37084f2: requirements.txt: bump a couple more versions (2019-10-29 10:22:59 +1300) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/recipesymbol http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/recipesymbol Paul Eggleton (29): RRS: collect history independent of current recipes RRS: Add deleted recipe handling RRS: use more robust RFC2822 date conversion RRS: skip problematic OE-Core commits (when a dependency) RRS: handle downgrades RRS: use more robust method of getting last upgrade record rrs_upgrade_history: record start marker in log file RRS: handle recipe moves without overwriting data RRS: support grouping upgrades by version for multi-version recipes RRS: use RecipeUpgradeGroup to determine downgrades RRS: detect PN changing without move rrs_upgrade_history: implement file path filtering RRS: record previous version rrs_upgrade_history: add stop commit option RRS: fixup handling of upgrades where recipe moved to inc RRS: fix some more bad OE-Core commits RRS: ensure upgrades recorded at exact same time are correctly ordered RRS: avoid historical parsing bug in bitbake recipeparse: handle recipes at root of repository RRS: handle when recipes get deleted and later re-added RRS: exclude lib/ subdirectory of layers to avoid picking up templates RRS: ensure default URLs for release/milestone are the latest RRS: detect changes in SRCREV as upgrades RRS: Add tool to dump upgrades RRS: enable grouping recipe upgrades by license RRS: Handle two versions added on same day then later one deleted Add recipe dependencies tool RRS: do not ignore non-numeric characters in versions TODO: add some more tasks TODO | 16 + layerindex/admin.py | 1 + layerindex/forms.py | 13 + .../migrations/0044_extendedprovides.py | 46 ++ layerindex/models.py | 8 + layerindex/recipeparse.py | 19 +- layerindex/update_layer.py| 2 + layerindex/urls.py| 6 +- layerindex/views.py | 116 - rrs/admin.py | 26 +- rrs/migrations/0020_recipesymbol_initial.py | 63 +++ rrs/migrations/0021_recipesymbol_nonnull.py | 31 ++ rrs/migrations/0022_recipesymbol_finish.py| 27 ++ rrs/migrations/0023_recipeupgrade_deleted.py | 25 + .../0024_recipeupgrade_downgrade.py | 20 + rrs/migrations/0025_recipeupgrade_move.py | 25 + rrs/migrations/0026_recipeupgrade_grouping.py | 39 ++ .../0027_recipeupgrade_prev_version.py| 20 + rrs/migrations/0028_recipeupgrade_srcrev.py | 20 + rrs/migrations/0029_rrs_license_group.py | 44 ++ rrs/models.py | 145 +- rrs/tools/common.py | 12 +- rrs/tools/dump_upgrades.py| 89 rrs/tools/rrs_maintainer_history.py | 32 +- rrs/tools/rrs_upgrade_history.py | 68 ++- rrs/tools/rrs_upstream_history.py | 36 +- rrs/tools/upgrade_history_internal.py | 431 +++--- rrs/views.py | 177 --- templates/base.html | 1 + templates/layerindex/recipedeps.html | 209 + templates/rrs/recipedetail.html | 27 +- 31 files changed, 1598 insertions(+), 196 deletions(-) create mode 100644 layerindex/migrations/0044_extendedprovides.py create mode 100644 rrs/migrations/0020_recipesymbol_initial.py create mode 100644 rrs/migrations/0021_recipesymbol_nonnull.py create mode 100644 rrs/migrations/0022_recipesymbol_finish.py create mode 100644 rrs/migrations/0023_recipeupgrade_deleted.py create mode 100644 rrs/migrations/0024_recipeupgrade_downgrade.py create mode 100644 rrs/migrations/0025_recipeupgrade_move.py create mode 100644 rrs/migrations/0026_recipeupgrade_grouping.py create mode 100644 rrs/migrations/0027_recipeupgrade_prev_version.py create mode 100644 rrs/migrations/0028_recipeupgrade_srcrev.py create mode 100644 rrs/migrations/0029_rrs_license_group.py create mode 100755 rrs/tools/dump_upgrades.py create mode
Re: [yocto] [warrior] [meta-qt4] 3rdparty: javascriptcore: JITStubs.cpp: allow builds of JavaScriptCore with gcc v8
Hi Mike On Wednesday, 20 November 2019 9:00:10 PM NZDT Mike Looijmans wrote: > I ran into this issue yesterday, and this patch solves the compilation on > warrior. However, it's a patch for Qt itself, and not for OE. > > Is there already a warrior patch in the making, or should I submit one? Quentin sent one yesterday (with the same subject). If you could give it a try and confirm it fixes the failure that would be great. Thanks Paul -- Paul Eggleton Intel System Software Products -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [error-report-web][PATCH V2] Add local.conf and auto.conf into error details
Hi Changqing,
Some comments below.
On Tuesday, 12 November 2019 9:32:53 PM NZDT changqing...@windriver.com wrote:
> From: Changqing Li
>
> Support to display local.conf and auto.conf on error report web.
> Here is commit in oe-core, which add local.conf/auto.conf into error report
> https://git.openembedded.org/openembedded-core/commit/?id=7adf9707c04d8ef6bcd8d8bda555687f705e6ee6
>
> This commit is related to YOCTO #13252
>
> Signed-off-by: Changqing Li
> ---
> Post/0006_auto_20190917_0419.py | 24
> Post/models.py | 2 ++
> Post/parser.py | 2 ++
> Post/test.py| 2 ++
> templates/error-details.html| 10 ++
> test-data/test-payload.json | 4 +++-
> 6 files changed, 43 insertions(+), 1 deletion(-)
> create mode 100644 Post/0006_auto_20190917_0419.py
>
> diff --git a/Post/0006_auto_20190917_0419.py b/Post/0006_auto_20190917_0419.py
> new file mode 100644
> index 000..827944e
> --- /dev/null
> +++ b/Post/0006_auto_20190917_0419.py
Could you please give the migration a proper name (-n option to makemigrations)
e.g. local_conf_auto_conf
> --- a/Post/models.py
> +++ b/Post/models.py
> @@ -43,6 +43,8 @@ class Build(models.Model):
> LINK_BACK = models.TextField(max_length=300, blank=True, null=True)
> ERROR_TYPE = models.CharField(max_length=20, choices=ERROR_TYPE_CHOICES,
>default=ErrorType.RECIPE)
> +LOCAL_CONF = models.TextField(max_length=int(settings.MAX_UPLOAD_SIZE),
> default="")
> +AUTO_CONF = models.TextField(max_length=int(settings.MAX_UPLOAD_SIZE),
> default="")
I'm not sure this is practical, for two reasons:
1) Field sizes should not be variable like this; changing the MAX_UPLOAD_SIZE
value after the fact would not change the database structure
2) The value could never actually reach MAX_UPLOAD_SIZE because the overhead of
the surrounding JSON would block it from being uploaded if it did
However, since this is a TextField we don't actually have to specify a
max_length (for a TextField max_length only actually affects the frontend, and
we don't expose this field in a form) so it can just be removed.
Another thing, instead of default="" you should use blank=True.
> +{% if detail.BUILD.LOCAL_CONF != "" %}
> +Local Conf:
> +{{ detail.BUILD.LOCAL_CONF | safe
> }}
> +{% endif %}
> +
> +{% if detail.BUILD.AUTO_CONF != "" %}
> +Auto Conf:
> +{{ detail.BUILD.AUTO_CONF | safe
> }}
> +{% endif %}
We cannot use the safe filter here - doing so could open up an XSS
vulnerability, since anyone can upload anything to the error-report application
and the content could include links or other malicious HTML data. We should
allow it to be auto-escaped. Is there a particular issue you were using this to
solve?
Cheers
Paul
--
Paul Eggleton
Intel System Software Products
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Where is the documentation for things to fix when upgrading poky versions?
Hi Sean On Thursday, 31 October 2019 9:33:30 AM NZDT McKay, Sean wrote: > This is probably a super easy question, but I've been searching half the day > and decided it would be faster to ask for help. > I know I've seen documentation in one of the manuals that points out major > changes in syntax or behavior that will need to be fixed when upgrading the > underlying version of poky, but now that I have need of it, I can't seem to > find it anywhere. Can anyone point me in the right direction? It's in the reference manual: https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#migration BTW if you (or anyone else) finds anything we missed documenting here then please do let us know - things do slip by, especially upstream changes. Cheers, Paul -- Paul Eggleton Intel System Software Products -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] Add SPDX Identifier
Hi Yann On Wednesday, 23 October 2019 2:25:48 AM NZDT Yann CARDAILLAC wrote: > On Mon, Oct 21, 2019 at 9:37 PM Paul Eggleton > wrote: > > On Tuesday, 22 October 2019 4:15:03 AM NZDT Yann CARDAILLAC wrote: > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=13527 > > > > > > I'm beginning the work of updating the layerindex-web repo, this one is a > > > bit tricky, how should I treat the files generated using Django ? > > > > I believe the correct license for Django-derived files is BSD-3-Clause: > > > > https://github.com/django/django/blob/master/LICENSE > > > > > I've yet to do html, txt and sh files, do you have any advices before I > > > start? > > > > Note that some of the templates used for authentication originally came > > from > > here: > > > > https://github.com/macdhuibh/django-registration-templates > > > > The LICENSE file indicates MIT for those though. > > > > The txt files are tricky because any stray newlines will affect > > formatting. I would probably avoid touching them. > > I did not yet add MIT to html files, I don't really know if that makes > sense or not? I didn't do that because it seemed a bit weird to me, but I > don't know what to think of that. Most of the time you'll find copyright on > the footer of the page, but I'm not sure about Licensing those. > > So here is a first iteration of the patch without htmls file licensed. I think it's fine to add to the HTML as long as you add it as part of the Django template comment at the top and not in the actual text that gets rendered. Looking over the license definitions they look fine, except the migrations - if these are considered derived from Django (which is fine) then they should all be BSD-3-Clause rather than some being MIT. It looks like some of the earlier ones don't have a "generated by django" comment, but they were all initially generated with ./manage.py makemigrations; some then had additional code added to them. Cheers Paul -- Paul Eggleton Intel System Software Products -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 2/2] requirements.txt: bump a couple more versions
Update pytz and beautifulsoup4. Signed-off-by: Paul Eggleton --- requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/requirements.txt b/requirements.txt index 84f2ea54..07d8495e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ amqp==2.5.2 -beautifulsoup4==4.8.0 +beautifulsoup4==4.8.1 billiard==3.6.1.0 celery==4.3.0 confusable-homoglyphs==3.2.0 @@ -20,7 +20,7 @@ GitPython==2.1.13 kombu==4.6.3 mysqlclient==1.4.4 Pillow==6.2.1 -pytz==2019.2 +pytz==2019.3 six==1.12.0 smmap2==2.0.5 soupsieve==1.9.4 -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 1/2] requirements.txt: update to fix CVE-2019-16865
Update Pillow version to incorporate a fix for a denial-of-service vulnerability (which should not affect this application however, as it does not use Pillow to process external images): https://nvd.nist.gov/vuln/detail/CVE-2019-16865 Signed-off-by: Paul Eggleton --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 855b7344..84f2ea54 100644 --- a/requirements.txt +++ b/requirements.txt @@ -19,7 +19,7 @@ gitdb2==2.0.6 GitPython==2.1.13 kombu==4.6.3 mysqlclient==1.4.4 -Pillow==6.1.0 +Pillow==6.2.1 pytz==2019.2 six==1.12.0 smmap2==2.0.5 -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 0/2] dependency updates
Updates to python dependencies in requirements.txt, including one upstream CVE fix. The following changes since commit fae9771604097896f10a4aec2f2aee7fefd8ec86: Fix cgit commit URL setting (2019-10-21 12:02:44 +1300) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/requirements1 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/requirements1 Paul Eggleton (2): requirements.txt: update to fix CVE-2019-16865 requirements.txt: bump a couple more versions requirements.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] Add SPDX Identifier
Hi Yann Thanks for looking into this. On Tuesday, 22 October 2019 4:15:03 AM NZDT Yann CARDAILLAC wrote: > https://bugzilla.yoctoproject.org/show_bug.cgi?id=13527 > > I'm beginning the work of updating the layerindex-web repo, this one is a > bit tricky, how should I treat the files generated using Django ? > > *For now I've given them MIT as for the other hand written script. However > I'm kind of worried that it's not the proper licence. * I believe the correct license for Django-derived files is BSD-3-Clause: https://github.com/django/django/blob/master/LICENSE > For the css side, I've added MIT to : > > ./rrs/static/css/rrs-additional.css > and > ./layerindex/static/css/additional.css That should be fine, those files were authored as part of the project. > *I've let the bootstraps css untouched since I guess that's not for us to > licence. * Correct, and we'd be required to re-apply it every time we upgraded as well which is likely to get forgotten. You could always get them to add one upstream if you're really keen :) > I've yet to do html, txt and sh files, do you have any advices before I > start? Note that some of the templates used for authentication originally came from here: https://github.com/macdhuibh/django-registration-templates The LICENSE file indicates MIT for those though. The txt files are tricky because any stray newlines will affect formatting. I would probably avoid touching them. Cheers Paul -- Paul Eggleton Intel System Software Products -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 3/3] Fix cgit commit URL setting
Typo, there has to be a ? in front of the id or otherwise we don't get
linked to the right commit. This would have affected recently added layers
with a cgit web frontend.
Signed-off-by: Paul Eggleton
---
templates/layerindex/editlayer.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/templates/layerindex/editlayer.html
b/templates/layerindex/editlayer.html
index dd95ea30..e21e649d 100644
--- a/templates/layerindex/editlayer.html
+++ b/templates/layerindex/editlayer.html
@@ -286,7 +286,7 @@
if (e) {
$('#id_vcs_web_tree_base_url').val(vcs_web_url +
'/tree/%path%?h=%branch%')
$('#id_vcs_web_file_base_url').val(vcs_web_url +
'/tree/%path%?h=%branch%')
-$('#id_vcs_web_commit_url').val(vcs_web_url +
'/commit/id=%hash%')
+$('#id_vcs_web_commit_url').val(vcs_web_url +
'/commit/?id=%hash%')
}
}
else if (type == 'gitweb') {
--
2.20.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 2/3] Add branch comparison function
Add the ability to compare available recipes and their versions between
two branches for a selection of layers (default is just OE-Core). This
was mainly intended to help us with the Yocto Project release notes
preparation (hence the "Plain text" button at the bottom of the page)
but is also useful in its own right.
Note: for readability, SRCREVs are only shown when PV has not changed.
Signed-off-by: Paul Eggleton
---
layerindex/forms.py | 19 ++
layerindex/urls.py | 12 +-
layerindex/views.py | 113 +-
templates/base.html | 1 +
templates/layerindex/branchcompare.html | 214 +++
templates/layerindex/branchcompare_plain.txt | 17 ++
6 files changed, 374 insertions(+), 2 deletions(-)
create mode 100644 templates/layerindex/branchcompare.html
create mode 100644 templates/layerindex/branchcompare_plain.txt
diff --git a/layerindex/forms.py b/layerindex/forms.py
index 13ba3aad..51583c60 100644
--- a/layerindex/forms.py
+++ b/layerindex/forms.py
@@ -354,3 +354,22 @@ class PatchDispositionForm(StyledModelForm):
}
PatchDispositionFormSet = modelformset_factory(PatchDisposition,
form=PatchDispositionForm, extra=0)
+
+
+class BranchComparisonForm(StyledForm):
+from_branch = forms.ModelChoiceField(label='From',
queryset=Branch.objects.none())
+to_branch = forms.ModelChoiceField(label='To',
queryset=Branch.objects.none())
+layers = forms.CharField(widget=forms.HiddenInput())
+
+def __init__(self, *args, request=None, **kwargs):
+super(BranchComparisonForm, self).__init__(*args, **kwargs)
+qs = Branch.objects.filter(comparison=False,
hidden=False).order_by('sort_priority', 'name')
+self.fields['from_branch'].queryset = qs
+self.fields['to_branch'].queryset = qs
+self.request = request
+
+def clean(self):
+cleaned_data = super(BranchComparisonForm, self).clean()
+if cleaned_data['from_branch'] == cleaned_data['to_branch']:
+raise forms.ValidationError({'to_branch': 'From and to branches
cannot be the same'})
+return cleaned_data
diff --git a/layerindex/urls.py b/layerindex/urls.py
index 89e70a22..abeb0928 100644
--- a/layerindex/urls.py
+++ b/layerindex/urls.py
@@ -14,7 +14,8 @@ from layerindex.views import LayerListView,
LayerReviewListView, LayerReviewDeta
bulk_change_edit_view, bulk_change_patch_view, BulkChangeDeleteView,
RecipeDetailView, RedirectParamsView, \
ClassicRecipeSearchView, ClassicRecipeDetailView, ClassicRecipeStatsView,
LayerUpdateDetailView, UpdateListView, \
UpdateDetailView, StatsView, publish_view, LayerCheckListView,
BBClassCheckListView, TaskStatusView, \
-ComparisonRecipeSelectView, ComparisonRecipeSelectDetailView,
task_log_view, task_stop_view, email_test_view
+ComparisonRecipeSelectView, ComparisonRecipeSelectDetailView,
task_log_view, task_stop_view, email_test_view, \
+BranchCompareView
from layerindex.models import LayerItem, Recipe, RecipeChangeset
from rest_framework import routers
from . import restviews
@@ -185,6 +186,15 @@ urlpatterns = [
url(r'^stoptask/(?P[-\w]+)/$',
task_stop_view,
name='task_stop'),
+url(r'^branch_comparison/$',
+BranchCompareView.as_view(
+template_name='layerindex/branchcompare.html'),
+name='branch_comparison'),
+url(r'^branch_comparison_plain/$',
+BranchCompareView.as_view(
+content_type='text/plain',
+template_name='layerindex/branchcompare_plain.txt'),
+name='branch_comparison_plain'),
url(r'^ajax/layerchecklist/(?P[-.\w]+)/$',
LayerCheckListView.as_view(
template_name='layerindex/layerchecklist.html'),
diff --git a/layerindex/views.py b/layerindex/views.py
index 2dacf516..12054fe7 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -47,7 +47,8 @@ from layerindex.forms import (AdvancedRecipeSearchForm,
BulkChangeEditFormSet,
ComparisonRecipeSelectForm, EditLayerForm,
EditNoteForm, EditProfileForm,
LayerMaintainerFormSet, RecipeChangesetForm,
- PatchDispositionForm, PatchDispositionFormSet)
+ PatchDispositionForm, PatchDispositionFormSet,
+ BranchComparisonForm)
from layerindex.models import (BBAppend, BBClass, Branch, ClassicRecipe,
Distro, DynamicBuildDep, IncFile, LayerBranch,
LayerDependency, LayerItem, LayerMaintainer,
@@ -1705,3 +1706,113 @@ class ComparisonRecipeSelectDetailView(DetailView):
messages.error(request, 'Failed to save changes: %s' % form.errors)
return self.get(request, *args, **kwargs)
+
+
+class BranchCompareView(FormView):
+
[yocto] [layerindex-web][PATCH 1/3] Track SRCREV for each recipe
For the purposes of the branch comparison function I'm about to add it
would be useful to track the value of SRCREV, so we can see if it has
changed even if PV hasn't.
Signed-off-by: Paul Eggleton
---
layerindex/migrations/0043_recipe_srcrev.py | 20
layerindex/models.py| 1 +
layerindex/update_layer.py | 4
templates/layerindex/recipedetail.html | 2 +-
4 files changed, 26 insertions(+), 1 deletion(-)
create mode 100644 layerindex/migrations/0043_recipe_srcrev.py
diff --git a/layerindex/migrations/0043_recipe_srcrev.py
b/layerindex/migrations/0043_recipe_srcrev.py
new file mode 100644
index ..64293767
--- /dev/null
+++ b/layerindex/migrations/0043_recipe_srcrev.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.22 on 2019-10-20 22:15
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+('layerindex', '0042_recipe_pe_pr'),
+]
+
+operations = [
+migrations.AddField(
+model_name='recipe',
+name='srcrev',
+field=models.CharField(blank=True, max_length=64),
+),
+]
diff --git a/layerindex/models.py b/layerindex/models.py
index 332ba39e..253c725c 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -475,6 +475,7 @@ class Recipe(models.Model):
updated = models.DateTimeField(auto_now=True)
blacklisted = models.CharField(max_length=255, blank=True)
configopts = models.CharField(max_length=4096, blank=True)
+srcrev = models.CharField(max_length=64, blank=True)
def vcs_web_url(self):
url = self.layerbranch.file_url(os.path.join(self.filepath,
self.filename))
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index fcae54f8..17098379 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -110,6 +110,10 @@ def update_recipe_file(tinfoil, data, path, recipe,
layerdir_start, repodir, sto
recipe.pv = envdata.getVar("PV", True)
recipe.pr = envdata.getVar("PR", True) or ""
recipe.pe = envdata.getVar("PE", True) or ""
+recipe.srcrev = envdata.getVar('SRCREV', True) or ''
+if recipe.srcrev == 'INVALID':
+# INVALID is the default from bitbake.conf, but we don't want to
see it
+recipe.srcrev = ''
recipe.summary = envdata.getVar("SUMMARY", True)
recipe.description = envdata.getVar("DESCRIPTION", True)
recipe.section = envdata.getVar("SECTION", True)
diff --git a/templates/layerindex/recipedetail.html
b/templates/layerindex/recipedetail.html
index b3ba65eb..64463863 100644
--- a/templates/layerindex/recipedetail.html
+++ b/templates/layerindex/recipedetail.html
@@ -54,7 +54,7 @@
Version
-{{ recipe.pv }}
+{{ recipe.pv }}{% if recipe.srcrev %} ({{
recipe.srcrev }}){% endif %}
Summary
--
2.20.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 0/3] Branch comparison functionality
Add the ability to compare the available recipes and their versions in two different branches, plus one unrelated patch to fix an issue I noticed recently with cgit URLs. The following changes since commit 1ce4a85d257ac928307ac0de6edf4cb80b50adce: editlayer: Be more specific on the searches (2019-10-14 09:20:33 +1300) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/branch_comparison http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/branch_comparison Paul Eggleton (3): Track SRCREV for each recipe Add branch comparison function Fix cgit commit URL setting layerindex/forms.py | 19 ++ layerindex/migrations/0043_recipe_srcrev.py | 20 ++ layerindex/models.py | 1 + layerindex/update_layer.py | 4 + layerindex/urls.py | 12 +- layerindex/views.py | 113 +- templates/base.html | 1 + templates/layerindex/branchcompare.html | 214 +++ templates/layerindex/branchcompare_plain.txt | 17 ++ templates/layerindex/editlayer.html | 2 +- templates/layerindex/recipedetail.html | 2 +- 11 files changed, 401 insertions(+), 4 deletions(-) create mode 100644 layerindex/migrations/0043_recipe_srcrev.py create mode 100644 templates/layerindex/branchcompare.html create mode 100644 templates/layerindex/branchcompare_plain.txt -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] Enabled zeus, not working...
Hi Mark On Tuesday, 15 October 2019 10:21:29 AM NZDT Mark Hatle wrote: > I added the zeus branch on the layers.openembedded.org today and it's not > showing up and being indexed. Any idea why? The bitbake branch wasn't correctly specified - it needed to be set to "1.44" (grabbed from https://wiki.yoctoproject.org/wiki/Releases if you don't have it in your head - I didn't ;). > On my own personal layer index I did it and it worked fine. So it may be > something related to the configuration. I guess it worked in your case because (based on your patch of the other day) you're picking bitbake out of a repo where zeus is a valid branch. FYI on layers.openembedded.org you also need to select the Python environment for the branch - python3 in this case. I've made both of these changes so it should index correctly the next time around. Cheers Paul -- Paul Eggleton Intel System Software Products -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] [PATCH 0/3] Some misc changes/fixes..
Hi Mark On Sunday, 13 October 2019 2:56:30 PM NZDT Mark Hatle wrote: > A few misc changes/fixes. The first two are well tested. However, I suspect > the 3/3 may be incorrect and I've labeled it an RFC due to this. > > 1/3 - '.' wasn't allowed in branch names w/o an error. This turned out > to be a fairly simple fix. > > 2/3 - For people who want to use 'poky' repository and not bitbake + > openembedded-core. I've tested this locally in both configurations. > > 3/3 - When I was testing, my local git mirror is broken up with > directories that are called 'git.openembedded.org' and 'git.yoctoproject.org' > due to this, the system was matching and locking out the edit layer > vcs_web_url submissions... so I tried to make it better.. but I'm not > sure it's right. > > Mark Hatle (3): > layerindex/urls.py: Allow branches with a '.' in the name > update.py: Allow bitbake to live in a subdirectory of a repository > editlayer: Be more specific on the searches Thanks, I merged these - I did make a minor change to 2/3 to allow it to work if BITBAKE_PATH doesn't appear in settings.py (I try to do this when adding new settings in case the user doesn't add the default to their edited settings file). Cheers Paul -- Paul Eggleton Intel System Software Products -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] Having problems instantiating a docker image
Hi Mark On Wednesday, 9 October 2019 11:59:35 AM NZDT Mark Hatle wrote: > I've setup the layerindex in the past (without docker).. but I'm attempting to > follow the current instructions w/o much success. > > I'm trying to use: > > ./dockersetup.py -m 8080:80 --no-https > > It asks me for my email address and then builds the docker images... > > then I get a failure connecting to https://google.com, connectivity check. > Since my layerindex will never leave my network I don't care about google.. > so I > re-run it with: > > ./dockersetup.py -m 8080:80 --no-https --no-connectivity -r > > re-enter my email address, it rebuilds the docker images... > > it then sits and spins with what looks like a help message for 'exec' as well > as: > > Database server may not be ready; will try again. I guess that's the docker-compose version issue we talked about on IRC - if it is then I should really add a version check. > If I try to connect to that machine port 8080 with my browser, I get back a > "Bad > Request (400)" I wonder if something is going wrong with the -m option, since it's not something I've exercised much. Perhaps it's erroneously setting up SSL on that port? Let's see if it recurs once docker-compose is upgraded. > So something is running, but it's broken. > > Any suggestions on getting this fixed? Also what will the admin login be once > it DOES come alive? I know in the past it had asked me for admin login > information and such, is the admin login documented somewhere for this docker > container? The admin login for the application user is something you get to set at the end of the installation process, but you're getting stuck before that part. You can set it on the command line but there almost certainly will be other issues, so let's fix those first and then you might not have to deal with this part. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [OE-core] 3.0 release notes / migration guide
On Thursday, 3 October 2019 11:44:53 AM NZDT akuster808 wrote: > On 10/2/19 3:20 PM, Paul Eggleton wrote: > > So it's that time again - we need to start building up the raw material for > > the release notes and migration guide for the upcoming 3.0 release, and I'd > > like to request some help with some parts of it - read on for details. > > > > For the migration guide we have a wiki page serving as a staging area: > > > > https://wiki.yoctoproject.org/wiki/FutureMigrationGuide > > Thanks for putting this together. No problem! > > Things that should go in the migration guide: *anything* in the release > > that > > would require someone who is upgrading to take some form of action (i.e. > > change their configuration / recipes / etc.) Help with this from people who > > implemented the changes or have already thought about / dealt with their > > impact would be much appreciated. > > > > There is a process where I look through all the commits since the last > > release > > and pull out the ones that need to be mentioned in some form - other than > > the > > migration guide items, the following needs to be gathered for the release > > notes: > > > > 1) Features - new functionality. We need to capture what the feature is and > > hopefully express the impact to the user succinctly. > > We remove LSB support. Thanks - I've just added that to the migration guide and will note it in the release notes. > > 2) Recipe upgrades - straightforward > > > > 3) CVE fixes - fairly straightforward, I just look for commits that > > explicitly > > mention "CVE". If I can easily do so I'll also note where a recipe upgrade > > fixes a CVE, but that isn't often readily available information. > > So how can the community help in this case? Better commit messages? That would be great - but it does require the person doing the upgrade to look upstream, and of course every upstream is different. Unfortunately with some upstreams finding fixed CVE information is quite difficult indeed. > Well, aren't open defects not fixed by the time we release time but we > intend on fixing after release a form of known issues ? Yes, that's true - I will say we haven't been particularly systematic about the known issues in past releases so perhaps we should drive the list from bugzilla. I would want the list to be kept as short as possible though and each item should succinctly describe the issue. Cheers Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] 3.0 release notes / migration guide
Hi folks So it's that time again - we need to start building up the raw material for the release notes and migration guide for the upcoming 3.0 release, and I'd like to request some help with some parts of it - read on for details. For the migration guide we have a wiki page serving as a staging area: https://wiki.yoctoproject.org/wiki/FutureMigrationGuide Things that should go in the migration guide: *anything* in the release that would require someone who is upgrading to take some form of action (i.e. change their configuration / recipes / etc.) Help with this from people who implemented the changes or have already thought about / dealt with their impact would be much appreciated. There is a process where I look through all the commits since the last release and pull out the ones that need to be mentioned in some form - other than the migration guide items, the following needs to be gathered for the release notes: 1) Features - new functionality. We need to capture what the feature is and hopefully express the impact to the user succinctly. 2) Recipe upgrades - straightforward 3) CVE fixes - fairly straightforward, I just look for commits that explicitly mention "CVE". If I can easily do so I'll also note where a recipe upgrade fixes a CVE, but that isn't often readily available information. 4) Known issues - generally this is difficult to get from the commits since we either find out about them after the commit that introduced the issue, or we don't know that they aren't addressed until right at the end of the release. We don't want to note *all* open bugs, but if there's something that the user is likely to hit that wasn't an issue in the previous release then we should note it. For the release notes I need help with #4 and #1 in particular, so if you know about anything in this release that falls into these categories then let me know - just a pointer to the commit and any extra comments that you would want to make would be helpful. In the mean time I will start the process of looking through the commits and will add things to the migration guide page, but bear in mind I don't always have all of the context for every change. Thanks! Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 0/3] More minor fixes [cover letter only]
A few miscellaneous minor improvements from the Clear Linux Dissector. The following changes since commit db51830ab238ed1a7ce33d43519c630b1c74c70b: update_classic_status: ensure fields are correctly ordered (2019-10-02 14:13:47 +1300) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/dissector-backports3 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/dissector-backports3 Paul Eggleton (3): README: add basic documentation on db backup and restore requirements.txt: update Add a space between text and glyphicons README | 17 + requirements.txt | 21 ++--- templates/layerindex/detail.html | 2 +- templates/layerindex/recipedetail.html | 2 +- templates/layerindex/recipes.html | 2 +- 5 files changed, 30 insertions(+), 14 deletions(-) -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH] These changes will add the metadata for product revision and epoch if present in the bitbake recipe.
Hi Pranay, On Tuesday, 1 October 2019 3:52:59 PM NZDT Pranay Mankad wrote: > Signed-off-by: Pranay Mankad > --- > .../migrations/0042_recipe_property_addition.py | 20 > > layerindex/models.py | 2 ++ > layerindex/update_layer.py | 2 ++ > 3 files changed, 24 insertions(+) > create mode 100644 layerindex/migrations/0042_recipe_property_addition.py So the substance of this change is fine, however the commit message needs a bit of reworking. The shortlog (first line, which is used by git-send-email automatically as the subject) should be shorter and then you can have a fuller explanation of the change after a blank line. It should look something like this: -- snip -- Track PE and PR for recipes This will add the metadata for epoch and revision if present in the bitbake recipe. These will be automatically available through the REST API but won't (yet) be visible in the UI. -- snip -- (Removing snip lines obviously) Thanks, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Xilinx/meta-jupyter layer
FYI I've just changed the layer index entry to point to the new repo. New content should be indexed in a few hours. Thanks for working this out! Dmitry: sorry, for clarity did you mean a separate mailing list or one of the YP / OE ones ? Cheers, Paul On Saturday, 28 September 2019 9:52:26 AM NZDT Dmitry Kargin wrote: > Nice job! > > You have my permission, if it is needed to change > https://layers.openembedded.org/layerindex/branch/master/layer/meta-jupyter/ > links to your repository. > > >Would you recommend maintaining this layer separately or... > I'd like to keep it separately from meta- and openembedded. > > >Would you suggest having a separate mailing list > meta-jupyter mailing list is fine. > > On Fri, 27 Sep 2019 at 23:12, Chandana Kalluri wrote: > > > Hello all, > > > > https://github.com/Xilinx/meta-jupyter is a meta-jupyter layer containing > > recipes for jupyter notebook. The initial recipes are based of Dmitry > > Kargin's meta-jupyter layer > > https://layers.openembedded.org/layerindex/branch/master/layer/meta-jupyter/. > > This layer has not been updated for a while. > > > > The Xilinx/meta-jupyter layer also adds recipes for python3 based > > notebooks apart from existing python2 based notebooks. > > This layer has been tested using Yocto thud layer and by running jupyter > > notebooks on Ultra96 community boards. > > We would like to maintain Xilinx/meta-jupyter layer actively and welcome > > contributions to this layer either through pull requests or via patches > > sent to meta-xilinx mailing list until a mailing list for meta-jupyter is > > in place. In the next cycle we will deprecate python2 recipes. > > > > A question to community, > > - Would you recommend maintaining this layer separately in the current > > github.com/Xilinx location or be included under meta- openembedded layers. > > - Would you suggest having a separate mailing list ? > > > > Thanks, > > Chandana > > > > > -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 00/15] More dockersetup/import fixes [cover letter only]
A few more improvements from the Clear Linux Dissector [1] for dockersetup, import_layers, and update_classic_status. [1] https://github.com/intel/clear-linux-dissector-web The following changes since commit b78e498c2086e4cb6631d73f36db059d37ad70ad: import_layers: fix updating build dependencies (2019-09-17 15:35:44 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/dissector-backports2 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/dissector-backports2 Paul Eggleton (15): dockersetup.py: add connectivity check docker: fix comment at top of Dockerfile dockersetup: proxy support fixes dockersetup: add uninstall option dockersetup: handle ArgumentTypeError properly dockersetup: allow updating if proxy config set in environment dockersetup: minor command line comment fix update_classic_status: tweak single-object queries update_classic_status: switch to argparse update_classic_status: add cover status import/export dockersetup: ensure letsencrypt volume change gets undone import_layers: use vcs_last_rev to check if record is updated import_layers: tweak debug logging to be a bit more useful README.devel: touch up a few things update_classic_status: ensure fields are correctly ordered Dockerfile| 4 +- README.devel | 29 +- docker/connectivity_check.sh | 35 ++ docker/git-proxy | 10 +- dockersetup.py| 191 -- layerindex/tools/import_layers.py | 12 +- layerindex/tools/update_classic_status.py | 423 +- 7 files changed, 481 insertions(+), 223 deletions(-) create mode 100755 docker/connectivity_check.sh -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] PR / PE in layer index
Hi Pranay On Tuesday, 24 September 2019 10:00:46 AM NZST Pranay Mankad wrote: > Thank you for supporting the recipesExtended view over the last couple of > days. Here’s a small patch that includes values from two more variables from > the Bitbake cooker – PR and PE. > > I read through Paul’s explanation of why PE made sense but not PR, and > thinking about it, the PR in layerindexs’ case shows if the untouched recipe > had a revision associated with or not. Source metadata is important for > information, and hence I’m submitting a change with the same. I suppose so yes. Out of interest I did a search of OE-Core and there were more recipes than I would have guessed that set PR, so it could perhaps be useful. I'm not sure I'd bother adding it to the UI yet though (and you haven't, that's fine.) > Here’s the patch (this is my first time submitting one, hopefully it’s the > right way to do it): So unfortunately no, you need to send it as an actual patch and not just pasted into an email - git-format-patch / git-send-email are the most effective ways to do that (though it assumes you are able to send email directly from your linux system). Here's some info: http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded (Of course you'd need to change mailing lists / prefixes in the examples, and some of the metadata requirements don't apply, but the basics are the same.) One other thing - you're editing the initial migration (layerindex/migrations/ 0001_initial.py) - that's not right. You need to undo the change to 0001_initial.py and then generate a new migration using the following command: ./manage.py createmigrations layerindex That will give you a new migration in layerindex/migrations/. You can then apply that migration to the database in the normal way: ./manage.py migrate layerindex Of course if your database already has those new fields the latter will fail, in this instance you could simply add --fake since the result would have been the same. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Extension of package name by distro version
Hi Jan
On Monday, 16 September 2019 8:55:00 PM NZST Jan Kaisrlik wrote:
> I would like to ask if it is possible to extend all package names by
> some string f.e. by string corresponding to the version of the
> distribution (+fw2.6.1)
> * without distro string - kernel-5.3.0-rc3_5.3.0+git0+e21a712a96-2_arm64.deb
> * with distro string -
> kernel-5.3.0-rc3_5.3.0+git0+e21a712a96-2+fw2.6.1_arm64.deb
> I've checked manuals and code and I do not see any easy way how to
> append some string to the version of the package.
>
> What I've played with it so far I added the mentioned string to PR in
> bbclasses' in poky. Honestly, I do not believe this is the best idea
> (on other hands it is working xD ) ...
I would say PR_append = "${DISTRO_VERSION}" in your custom distro config work
- did you try that?
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 3/3] import_layers: fix output when importing layers from scratch
Fixes "Importing None" message since we had not yet initialised the
layerbranch variable.
Signed-off-by: Paul Eggleton
---
layerindex/tools/import_layers.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/layerindex/tools/import_layers.py
b/layerindex/tools/import_layers.py
index 0ce31968..1e49bbdf 100755
--- a/layerindex/tools/import_layers.py
+++ b/layerindex/tools/import_layers.py
@@ -329,10 +329,10 @@ def main():
continue
logger.info('Updating %s (%d/%d)' % (layerbranch, i+1,
layercount))
else:
-logger.info('Importing %s (%d/%d)' % (layerbranch, i+1,
layercount))
layerbranch = LayerBranch()
layerbranch.branch = branch
layerbranch.layer = layer
+logger.info('Importing %s (%d/%d)' % (layerbranch, i+1,
layercount))
for key, value in layerbranchjs.items():
--
2.20.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 2/3] import_layers: use recipesExtended viewset
The new viewset has a different URL and uses pagination, so we need to
take that into account.
Signed-off-by: Paul Eggleton
---
layerindex/tools/import_layers.py | 92 ++-
1 file changed, 55 insertions(+), 37 deletions(-)
diff --git a/layerindex/tools/import_layers.py
b/layerindex/tools/import_layers.py
index d40c391a..0ce31968 100755
--- a/layerindex/tools/import_layers.py
+++ b/layerindex/tools/import_layers.py
@@ -89,7 +89,7 @@ def main():
layerbranches_url = jsdata['layerBranches']
layermaintainers_url = jsdata.get('layerMaintainers', None)
layernotes_url = jsdata.get('layerNotes', None)
-recipes_url = jsdata.get('recipes', None)
+recipes_url = jsdata.get('recipesExtended', None)
machines_url = jsdata.get('machines', None)
distros_url = jsdata.get('distros', None)
classes_url = jsdata.get('classes', None)
@@ -186,55 +186,73 @@ def main():
# The parent field always needs to be part of the keys
keys = key_fields + [parentfield]
+def fetch_api_url(api_url):
+rq = urllib.request.Request(api_url)
+data = urllib.request.urlopen(rq).read()
+return json.loads(data.decode('utf-8'))
+
if url:
if parent_orig_id is None:
raise Exception('import_child_items: if url is
specified then parent_orig_id must also be specified')
-rq = urllib.request.Request(url + '?filter=%s:%s' %
(parentfield, parent_orig_id))
-data = urllib.request.urlopen(rq).read()
-childjslist = json.loads(data.decode('utf-8'))
+childjsdata = fetch_api_url(url + '?filter=%s:%s' %
(parentfield, parent_orig_id))
elif childlist is not None:
-childjslist = childlist
+childjsdata = childlist
else:
raise Exception('import_child_items: either url or
childlist must be specified')
manager = getattr(parentobj, objclass.__name__.lower() +
'_set')
existing_ids = list(manager.values_list('id', flat=True))
updated_ids = []
-for childjs in childjslist:
-vals = {}
-for key, value in childjs.items():
-if key in exclude:
-continue
-vals[key] = value
-vals[parentfield] = parentobj
-if keys:
-keyvals = {k: vals[k] for k in keys}
-else:
-keyvals = vals
-
-# In the case of multiple records with the same keys (e.g.
multiple recipes with same pn),
-# we need to skip ones we've already touched
-obj = None
-created = False
-for entry in manager.filter(**keyvals):
-if entry.id not in updated_ids:
-obj = entry
+def import_list(childjslist):
+for childjs in childjslist:
+vals = {}
+for key, value in childjs.items():
+if key in exclude:
+continue
+vals[key] = value
+vals[parentfield] = parentobj
+
+if keys:
+keyvals = {k: vals[k] for k in keys}
+else:
+keyvals = vals
+
+# In the case of multiple records with the same keys
(e.g. multiple recipes with same pn),
+# we need to skip ones we've already touched
+obj = None
+created = False
+for entry in manager.filter(**keyvals):
+if entry.id not in updated_ids:
+obj = entry
+break
+else:
+created = True
+obj = objclass(**keyvals)
+
+for key, value in vals.items():
+setattr(obj, key, value)
+# Need to have saved before calling custom_field_cb
since the function might be adding child objects
+obj.save()
+updated_ids.append(obj.id)
+if custom_field_cb is not None:
+custom_field_cb(obj, childjs)
+if not created:
+if obj.id in existing_ids:
+existing_ids.remove(obj.id)
+
+if 'results' in childjsdata
[yocto] [layerindex-web][PATCH 1/3] API: fix recipes API performance regression
Adding these extra child items to the standard "recipes" viewset (which
we did recently in 684a06a383fd2a8da80490dff5f0dbc47750934e) means that
some current usage is impractical due to the size of the returned list
of items. Instead, create a recipesExtended viewset, move the new child
items to that and add pagination to avoid result size issues.
Signed-off-by: Paul Eggleton
---
layerindex/restviews.py | 25 ++---
layerindex/urls.py | 1 +
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/layerindex/restviews.py b/layerindex/restviews.py
index 96c24ea3..7de4cd7d 100644
--- a/layerindex/restviews.py
+++ b/layerindex/restviews.py
@@ -1,7 +1,16 @@
+# OpenEmbedded Layer Index REST API implementation
+#
+# Copyright (C) 2014, 2016-2019 Intel Corporation
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+
from layerindex.models import Branch, LayerItem, LayerMaintainer,
YPCompatibleVersion, LayerNote, LayerBranch, LayerDependency, Recipe, Machine,
Distro, BBClass, Source, Patch, PackageConfig, StaticBuildDep, DynamicBuildDep,
RecipeFileDependency, BBAppend, IncFile
-from rest_framework import viewsets, serializers
+from rest_framework import viewsets, serializers, pagination
from layerindex.querysethelper import params_to_queryset, get_search_tuple
+class LayerIndexPagination(pagination.PageNumberPagination):
+page_size = 200
+
class DynamicFieldsModelSerializer(serializers.ModelSerializer):
"""
A ModelSerializer that takes an additional "fields" argument that
@@ -114,6 +123,15 @@ class RecipeSerializer(serializers.ModelSerializer):
model = Recipe
fields = '__all__'
+class RecipeViewSet(ParametricSearchableModelViewSet):
+queryset = Recipe.objects.all()
+serializer_class = RecipeSerializer
+
+class RecipeExtendedSerializer(serializers.ModelSerializer):
+class Meta:
+model = Recipe
+fields = '__all__'
+
sources = serializers.SerializerMethodField()
patches = serializers.SerializerMethodField()
package_configs = serializers.SerializerMethodField()
@@ -143,9 +161,10 @@ class RecipeSerializer(serializers.ModelSerializer):
serializer = RecipeFileDependencySerializer(instance=qs, many=True,
read_only=True, fields=('layerbranch', 'path'))
return serializer.data
-class RecipeViewSet(ParametricSearchableModelViewSet):
+class RecipeExtendedViewSet(ParametricSearchableModelViewSet):
queryset = Recipe.objects.all()
-serializer_class = RecipeSerializer
+serializer_class = RecipeExtendedSerializer
+pagination_class = LayerIndexPagination
class MachineSerializer(serializers.ModelSerializer):
class Meta:
diff --git a/layerindex/urls.py b/layerindex/urls.py
index dec99585..13f42c92 100644
--- a/layerindex/urls.py
+++ b/layerindex/urls.py
@@ -28,6 +28,7 @@ router.register(r'layerDependencies',
restviews.LayerDependencyViewSet)
router.register(r'layerMaintainers', restviews.LayerMaintainerViewSet)
router.register(r'layerNotes', restviews.LayerNoteViewSet)
router.register(r'recipes', restviews.RecipeViewSet)
+router.register(r'recipesExtended', restviews.RecipeExtendedViewSet)
router.register(r'machines', restviews.MachineViewSet)
router.register(r'distros', restviews.DistroViewSet)
router.register(r'classes', restviews.ClassViewSet)
--
2.20.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 0/3] REST API fixes
Some fixes relating to the REST API performance issues experienced with the recent upgrade on layers.openembedded.org. The following changes since commit 73198bd5483c16b9852121c6855dc6dac96a98a0: README: add reference to using import_layers to update (2019-08-06 12:04:17 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/restapi-fixes http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/restapi-fixes Paul Eggleton (3): API: fix recipes API performance regression import_layers: use recipesExtended viewset import_layers: fix output when importing layers from scratch layerindex/restviews.py | 25 +++- layerindex/tools/import_layers.py | 94 ++- layerindex/urls.py| 1 + 3 files changed, 79 insertions(+), 41 deletions(-) -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 0/4] dockersetup fixes (cover letter only)
Some fixes for the dockersetup.py script from the Clear Linux Dissector on top of the set that was merged recently. The following changes since commit 73198bd5483c16b9852121c6855dc6dac96a98a0: README: add reference to using import_layers to update (2019-08-06 12:04:17 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/dissector-backports2 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/dissector-backports2 Paul Eggleton (4): dockersetup.py: add connectivity check docker: fix comment at top of Dockerfile dockersetup: proxy support fixes dockersetup: add uninstall option Dockerfile | 4 +- docker/connectivity_check.sh | 35 docker/git-proxy | 10 ++- dockersetup.py | 167 --- 4 files changed, 181 insertions(+), 35 deletions(-) create mode 100755 docker/connectivity_check.sh -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 00/15] REST API / import_layers improvements (cover letter only)
Add some missing items to the REST API and extend the import_layers script in order to make it possible to create and update a clone of an existing layer index without needing to fetch and parse all of the layer repositories. The following changes since commit 9ce9f7d2d1870a90b95809f43bae5d4e9f1ebd93: dockersetup: fix regression with https enabled (2019-07-31 09:03:46 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/importlayers http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/importlayers Paul Eggleton (15): API: add sources, patches, dependencies to recipes API: add BBAppend and IncFile objects to REST API import_layers: avoid importing duplicate objects import_layers: enable incremental LayerItem/LayerBranch import import_layers: import recipes import_layers: import other missing items admin: add layerbranch to BBAppend string representation Add simple script for testing web API import_layers: switch to argparse import_layers: allow filtering branches import_layers: allow filtering layers import_layers: delete non-matching LayerBranch objects import_layers: add reload option import_layers: show basic progress README: add reference to using import_layers to update README| 10 +- layerindex/models.py | 2 +- layerindex/restviews.py | 74 +- layerindex/tools/import_layers.py | 405 +++--- layerindex/tools/webapi-test.py | 71 ++ layerindex/urls.py| 2 + 6 files changed, 469 insertions(+), 95 deletions(-) create mode 100755 layerindex/tools/webapi-test.py -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH v2 000/129] Docker setup / misc fixes (cover letter only)
The Clear Linux* Dissector [1] is a special-purpose fork of the OE Layer Index codebase. During development a number of general-purpose fixes have been made, so I am now sending a slightly reworked set of these back to be merged into the layer index. Highlights: * Enhanced docker/docker-compose setup with script for one-command installation - this is now the preferred method of installation for uses other than development, so the README documentation has been updated accordingly. * User-selectable security questions for password resets * Fixes for spec file importing * Various minor security fixes [1] https://github.com/intel/clear-linux-dissector-web Changes since v1: * Add a few more fixes: import_otherdistro: handle non-UTF8 encoded spec files import_otherdistro: try-specfile: handle files in current dir dockersetup: fix error when printing URL with https enabled The following changes since commit dba1fbe5d1c5d8714d7ca3ca86c42972ebde128e: RRS: add missing migration (2019-05-28 09:57:58 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/dissector-backports http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/dissector-backports Amber Elliot (6): docker: add setup script admin.py: Add custom SiteAdmin model. Upgrade django-registration to version 3.0. Add user security questions layerindex/urls.py: improve formatting requirements.txt: Require secure version of Django. Paul Eggleton (122): Add ability to hide branches Split out recipe dependency handling to its own function docker: Add docker-compose file update_classic_status: recognise pythonhosted.org as python update_classic_status: set category for KDE packages update_classic_status: categorise perl packages import_otherdistro: add description option import_otherdistro: improve display of deleted items dockersetup: formatting fixes dockersetup: add some error checking to the setup script dockersetup: tweak portmapping option and display port docker/nginx*.conf: fix up indentation docker: use quoted values dockersetup: add HTTPS support and use by default dockersetup: Show intro message dockersetup: add letsencrypt support nginx: set some limits for DDOS protection docker: set mariadb wait_timeout to upstream default dockerignore: add docker files and tests docker: enable user/password for RabbitMQ server dockersetup: move HTTPS code to its own function dockersetup: support update/reinstall mode dockersetup: fix auto-generated passwords dockersetup: warn if http proxy specified without https dockersetup: use separate db account with lower privileges dockersetup: use with open in readfile/writefile .gitignore: add some more files/dirs dockersetup: support importing gzip compressed database dumps docker: pass through options for migrate script docker: restart services automatically unless stopped Update TableSorter to latest version of active fork Disable caching on auth views docker: enable SECURE_BROWSER_XSS_FILTER Use shell=False where possible with utils.runcmd() dockersetup: import database dump before running migrations dockersetup: move superuser account creation to the end dockersetup: set site info and email host Rework README documents import_layers: fix URL construction Add a script to create initial db dump dockersetup: add option to skip database migrations dockersetup: enable error report emails Disable autocomplete on sensitive fields docker: Increase nginx max upload size docker: increase gunicorn worker timeout to handle large images ClassicRecipeDetailView does not need a form Add ability to disposition comparison patches Add access controls to PatchDispositionAdmin import_otherdistro: refactor spec file import import_otherdistro.py: avoid deleting all records if no spec files found import_otherdistro.py: create layer/branch if they don't exist Add sha256sum to Source model Record sha256sum of other distro source files dockersetup: require email address import_otherdistro: optionally store local path import_otherdistro: fix handling of garbage in description values Determine if spec file patches are actually applied and striplevel docker-compose: fix missing DATABASE_USER for layerscelery service docker: add dependencies for derivative import tasks: support running non-shell commands tasks: fix incorrect closing tag on button Enable task log/progress to work within docker tasks: handle carriage returns in task output Record configure options Fix erroneously importing package fields from spec files docker: use python3 in migrate script docker: improve nginx https configuration security docker: Increase max packet size to 128M tasks: disable broker heartbeat to workaround connection issues dockersetup: 'easy' subprocess conversions to not use shell=True dockersetup: require python 3.4.3 or later
[yocto] [layerindex-web][PATCH 000/126] Docker setup / misc fixes (cover letter only)
The Clear Linux* Dissector [1] is a special-purpose fork of the OE Layer Index codebase. During development a number of general-purpose fixes have been made, so I am now sending a slightly reworked set of these back to be merged into the layer index. Highlights: * Enhanced docker/docker-compose setup with script for one-command installation - this is now the preferred method of installation for uses other than development, so the README documentation has been updated accordingly. * User-selectable security questions for password resets * Fixes for spec file importing * Various minor security fixes [1] https://github.com/intel/clear-linux-dissector-web The following changes since commit dba1fbe5d1c5d8714d7ca3ca86c42972ebde128e: RRS: add missing migration (2019-05-28 09:57:58 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/dissector-backports http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/dissector-backports Amber Elliot (6): docker: add setup script admin.py: Add custom SiteAdmin model. Upgrade django-registration to version 3.0. Add user security questions layerindex/urls.py: improve formatting requirements.txt: Require secure version of Django. Paul Eggleton (119): Add ability to hide branches Split out recipe dependency handling to its own function docker: Add docker-compose file update_classic_status: recognise pythonhosted.org as python update_classic_status: set category for KDE packages update_classic_status: categorise perl packages import_otherdistro: add description option import_otherdistro: improve display of deleted items dockersetup: formatting fixes dockersetup: add some error checking to the setup script dockersetup: tweak portmapping option and display port docker/nginx*.conf: fix up indentation docker: use quoted values dockersetup: add HTTPS support and use by default dockersetup: Show intro message dockersetup: add letsencrypt support nginx: set some limits for DDOS protection docker: set mariadb wait_timeout to upstream default dockerignore: add docker files and tests docker: enable user/password for RabbitMQ server dockersetup: move HTTPS code to its own function dockersetup: support update/reinstall mode dockersetup: fix auto-generated passwords dockersetup: warn if http proxy specified without https dockersetup: use separate db account with lower privileges dockersetup: use with open in readfile/writefile .gitignore: add some more files/dirs dockersetup: support importing gzip compressed database dumps docker: pass through options for migrate script docker: restart services automatically unless stopped Update TableSorter to latest version of active fork Disable caching on auth views docker: enable SECURE_BROWSER_XSS_FILTER Use shell=False where possible with utils.runcmd() dockersetup: import database dump before running migrations dockersetup: move superuser account creation to the end dockersetup: set site info and email host Rework README documents import_layers: fix URL construction Add a script to create initial db dump dockersetup: add option to skip database migrations dockersetup: enable error report emails Disable autocomplete on sensitive fields docker: Increase nginx max upload size docker: increase gunicorn worker timeout to handle large images ClassicRecipeDetailView does not need a form Add ability to disposition comparison patches Add access controls to PatchDispositionAdmin import_otherdistro: refactor spec file import import_otherdistro.py: avoid deleting all records if no spec files found import_otherdistro.py: create layer/branch if they don't exist Add sha256sum to Source model Record sha256sum of other distro source files dockersetup: require email address import_otherdistro: optionally store local path import_otherdistro: fix handling of garbage in description values Determine if spec file patches are actually applied and striplevel docker-compose: fix missing DATABASE_USER for layerscelery service docker: add dependencies for derivative import tasks: support running non-shell commands tasks: fix incorrect closing tag on button Enable task log/progress to work within docker tasks: handle carriage returns in task output Record configure options Fix erroneously importing package fields from spec files docker: use python3 in migrate script docker: improve nginx https configuration security docker: Increase max packet size to 128M tasks: disable broker heartbeat to workaround connection issues dockersetup: 'easy' subprocess conversions to not use shell=True dockersetup: require python 3.4.3 or later More shell quoting dockersetup: remove redundant code to update non-HTTPS lines in nginx-ssl.conf dockersetup: update nginx-ssl.conf in update mode Reimplement simplesearch admin: allow searching ClassicRecipes by cover_pn
[yocto] [layerindex-web][PATCH 0/7] Recipe reporting system fixes (cover letter only)
A bunch of fixes mostly for the Recipe Reporting System (RRS) part of the layer index. In particular, 5/7 fixes the RRS for layers in meta-openembedded. The following changes since commit ce5141fc39b2b171383fca002b2bc5cf082b31f4: RRS: add tool to import/export upstream history data (2019-02-14 10:05:53 +1300) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes13 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes13 Paul Eggleton (7): Split out patch status read to a method on the Patch model Record patch application order and use as default sorting Add admin handling for Patch objects RRS: Fix raw SQL statements to use parameters RRS: fix showing all recipes for multi-layer maintenance plans RRS: move Export button and restore Tools drop-down menu RRS: add missing migration layerindex/admin.py | 11 +- .../migrations/0027_patch_apply_order.py | 24 +++ layerindex/models.py | 31 layerindex/tools/import_otherdistro.py| 5 +- layerindex/update_layer.py| 34 + .../0019_maintplan_admin_delete_null.py | 22 +++ rrs/views.py | 143 +- templates/base.html | 4 + templates/rrs/base_toplevel.html | 5 +- templates/rrs/recipes.html| 2 + 10 files changed, 173 insertions(+), 108 deletions(-) create mode 100644 layerindex/migrations/0027_patch_apply_order.py create mode 100644 rrs/migrations/0019_maintplan_admin_delete_null.py -- 2.20.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [oe] [OE-core] Git commit process question.
On Thursday, 4 April 2019 5:46:04 AM NZDT Khem Raj wrote: > On Wed, Apr 3, 2019 at 7:41 AM Tom Rini wrote: > > > > On Wed, Apr 03, 2019 at 11:30:39AM +0100, Burton, Ross wrote: > > > On Tue, 2 Apr 2019 at 20:46, Tom Rini wrote: > > > > > The kernel does not have "upgrade foo to the latest upstream > > > > > version" commits. > > > > > > > > > > With the Automatic Upgrade Helper this is a semi-automatic task, and > > > > > most of the time there is no specific motivation other than > > > > > upgrading > > > > > to the latest upstream version. > > > > > > > > But since that's just filling in a template the body can also be a > > > > template perhaps with useful AUH data (run at ... by ... ?) ? > > > > > > Apart from making the commit message longer what does this achieve? > > > The commit already has a timestamp and author. > > > > It's an etiquette thing. Subject+Sign-off+Empty body is bad form. AUH > > updates are a form of "trivial update" that every project has. "Update > > $X from version $Y to $Z" is what a human would normally put. It's > > weird looking at git log of nothing but subject+signed-off-by. I'm not > > going to object further on this point, but I don't get it. > > if the content of subject is being repeated in body then I would > prefer an empty body > redundant information in commits should be avoided since it can create > impression that body does not have > useful information and skip reading it. We should strive to make commits > concise and useful. There is often (I won't say always, but often) something useful you can put in the commit message. If it's a recipe upgrade, you could put a pointer to the upstream changelog in it, for example. As the person doing the upgrade if your prior review of that changelog or other upstream release documentation indicated any backwards-compatibility issues or CVEs fixed then those really ought to be mentioned as well; if you're feeling especially generous you might mention highlights of any new functionality. (I have a proposal that might help us automate part of that which I've not yet fully fleshed out, hopefully one day soon I will get around to it.) The issue of empty commit messages is something I've complained about in the past, and not just about recipe upgrades. If I - as someone who is relatively familiar with OE - have to actually read beyond the shortlog / commit message to understand the basics of why a change has been made, then it's likely that the commit message wasn't good enough. Unlike other issues, once a commit goes in the message is set in stone within the git history, so if you are working on a change, *please* take a minute or two to document it adequately in the commit message so that others looking back can understand it. Thanks, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Removing busybox
On Thursday, 28 February 2019 9:09:02 AM NZDT Tom Rini wrote: > Somewhere on my (and other > folks too) TODO list is a packagegroup that includes procps and psmisc > and grep and tar and various network packages and less and more and... That is what packagegroup-core-full-cmdline (well, the packagegroup packages that it produces) was intended to be, although I guess there might be some argument about what should and should not be in there. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Using SDK/ESDK to build images
Hi Kristupas, On Wednesday, 13 February 2019 12:55:57 AM NZDT Kristupas Savickas wrote: > we're looking into providing our customers with SDK/ESDK packages to > develop custom solutions on our boards. We don't want to provide the > whole project itself as it would leak our intellectual property, so > precompiled packages are a must. Looking around, it seems like both SDK > and ESDK are suited to build single packages, rather than complete > images. Am I correct about this? Is there some kind of method to allow > images be built using SDK packages? The eSDK can build images, yes, via "devtool build-image". You won't get the same ability to customise the OS as with the full build system (unless you start editing the software contents or modifying the configuration, which isn't really supported) - the purpose is really to allow building an image to test one or more components that you're developing with the SDK. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [OE-core] Yocto Project Status WW07'19
Hi Alex On Wednesday, 13 February 2019 6:22:23 AM NZDT Alexander Kanavin wrote: > On Tue, 12 Feb 2019 at 17:10, wrote: > > The recipe reporting system integration with the layer index is available > > at: https://layers.openembedded.org/rrs/recipes/OE-Core/2.7/M3/ and the > > historical data has now been imported. > > Sadly the new RRS still seems to be suffering from the 'outdated info' > problem. For example glibc, or python3 recipes have been updated to > latest versions, yet RRS still shows that they haven't been. We're reliant upon the layer index updating and it appears that's not working at the moment. I have asked Michael to look into it. Thanks for the report, I'll reply back here when things are sorted out. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH V2] patchwork: Add a dockerfile for deploy patchwork and patchtest
Not actively, no. I will try to take a look at your patch this week though. Cheers, Paul On Monday, 11 February 2019 2:17:16 PM NZDT Changqing Li wrote: > Hi, Paul > > Are you maintaining this repo? > > On 2/10/19 2:43 AM, Jose Lamego wrote: > > Hi Randy, > > > > Sorry for the late response. I'm no longer involved with the Yocto > > Project. Last word I heard was that point of contact for Patchwork- > > related work was Paul Eggleton. > > > > Hope this helps. > > > > Jose > Thanks. > > > > On Mon, 2019-01-28 at 20:30 -0500, Randy MacLeod wrote: > >> On 1/16/19 12:37 AM, changqing...@windriver.com wrote: > >>> From: Changqing Li > >> Ping? > >> > >> Jose, are you maintaining this repo? > >> http://git.yoctoproject.org/cgit/cgit.cgi/patchwork/about/ > >> You were the committer for most of the recent work. > >> > >> ../Randy > >>> Add a dockerfile for easy deploy patchwork and patchtest in > >>> docker > >>> > >>> Signed-off-by: Changqing Li > >>> --- > >>>Dockerfile.pw | 52 > >>> +++ > >>>README.pw | 32 ++ > >>>requirements.txt | 12 ++ > >>>scripts/pw_createsuperuser.sh | 5 + > >>>scripts/pw_getmail.sh | 11 + > >>>scripts/pw_migrate.sh | 5 + > >>>scripts/pw_runwebserver.sh| 12 ++ > >>>7 files changed, 129 insertions(+) > >>>create mode 100644 Dockerfile.pw > >>>create mode 100644 README.pw > >>>create mode 100644 requirements.txt > >>>create mode 100755 scripts/pw_createsuperuser.sh > >>>create mode 100755 scripts/pw_getmail.sh > >>>create mode 100755 scripts/pw_migrate.sh > >>>create mode 100755 scripts/pw_runwebserver.sh > >>> > >>> diff --git a/Dockerfile.pw b/Dockerfile.pw > >>> new file mode 100644 > >>> index 000..ae3c1ee > >>> --- /dev/null > >>> +++ b/Dockerfile.pw > >>> @@ -0,0 +1,52 @@ > >>> +FROM ubuntu:latest > >>> +MAINTAINER owner > >>> + > >>> +EXPOSE 8080 > >>> + > >>> +ENV PYTHONUNBUFFERED=1 \ > >>> +LANG=en_US.UTF-8 \ > >>> +LC_ALL=en_US.UTF-8 \ > >>> +LC_CTYPE=en_US.UTF-8 \ > >>> +PATH="$PATH:/opt/patchwork/git- > >>> pw:/opt/patchtest:/opt/patchtest/scripts" > >>> + > >>> +RUN apt-get update \ > >>> +&& apt-get dist-upgrade -y \ > >>> +&& apt-get install -y --no-install-recommends \ > >>> +autoconf \ > >>> +gcc \ > >>> +g++ \ > >>> +make \ > >>> +locales \ > >>> +wget \ > >>> +curl \ > >>> +cron \ > >>> +python-pip \ > >>> +python-dev \ > >>> +python-setuptools \ > >>> +python3-pip \ > >>> +python3-dev \ > >>> +libmysqlclient-dev \ > >>> +git-core \ > >>> +getmail4 \ > >>> +&& sh -c "echo \"en_US.UTF-8 UTF-8\" >> /etc/locale.gen" \ > >>> +&& locale-gen en_US.UTF-8 \ > >>> +&& update-locale \ > >>> +&& pip2 install wheel \ > >>> +&& pip3 install wheel \ > >>> +&& pip3 install setuptools \ > >>> +&& apt-get autoremove --purge -y \ > >>> +&& rm -rf /var/lib/apt/lists/* \ > >>> +&& apt-get clean > >>> +ADD ./patchwork /opt/patchwork > >>> +ADD ./patchtest /opt/patchtest > >>> +ADD ./patchtest-oe /opt/patchtest-oe > >>> +ADD ./getmailrc /opt/getmail/getmailrc > >>> +RUN pip2 install -r /opt/patchwork/requirements.txt \ > >>> +&& pip3 install -r /opt/patchtest/requirements.txt \ > >>> +&& pip3 install -r /opt/patchtest-oe/requirements.txt \ > >>> +&& mkdir -p /opt/getmail \ > >>> +&& mkdir -p /opt/pw-logs \ > >>> +&& git config --global user.email "
Re: [yocto] [PATCH 00/10] Misc fixes (cover letter only)
Oops, yes, I forgot the [layerindex-web], my apologies. I can cc the oe-devel list if people really want to see it there as well; up to this point we've been using this list only. The other option would be a dedicated list but I suspect that would have a very limited subscription. Cheers, Paul On Thursday, 7 February 2019 11:24:24 AM NZDT akuster wrote: > > Hello Paul. > > On 2/6/19 2:00 PM, Paul Eggleton wrote: > > The following changes since commit 61e9b048591adab88d1996646286e1d7df024bea: > > > > Fix drop-down alignment on duplicates page (2018-11-20 11:58:42 +1300) > > > > are available in the Git repository at: > > > > git://git.yoctoproject.org/layerindex-web paule/fixes12 > > http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes12 > > > > Paul Eggleton (10): > > Send people an email when another user adds them as a maintainer > > Add links to other branch recipes in recipe detail > > Fix errors due to races deleting bitbake temp files > > local.conf: allow for upstream_tracking.inc to be missing > > Use try...finally or with to ensure files get closed > > Replace use of assert with exceptions > > docker: add wget to dependencies > > RRS: fix sorting arrow positioning > > rrs_maintainer_history: check out layer branch before looking for > > maintainers.inc > > RRS: add tool to import/export upstream history data > > Thanks for the updates and fixes. This is a very useful tool to both YP > and OE. > > I had to look at what was actual changes to figure out what this patch > set was against. Should we change the "subject" to help clarify? ie > [layer-index][PATCH ...? > > The layer index is hosted on OE. Should one of the OE mailing be CC'ed?? > > Yes I realize the sources are hosted on YP but the url for the tool is > http://layers.openembedded.org. can be confusing. > > kind regards, > Armin > > > > Dockerfile | 1 + > > conf/local.conf| 2 +- > > layerindex/bulkchange.py | 18 +- > > layerindex/update.py | 4 +- > > layerindex/update_layer.py | 14 +- > > layerindex/utils.py| 21 ++- > > layerindex/views.py| 70 ++-- > > rrs/tools/historytool.py | 196 + > > rrs/tools/rrs_distros.py | 19 +- > > rrs/tools/rrs_maintainer_history.py| 45 ++--- > > rrs/tools/rrs_upgrade_history.py | 4 +- > > rrs/tools/rrs_upstream_history.py | 3 +- > > rrs/tools/upgrade_history_internal.py | 3 +- > > rrs/views.py | 2 + > > templates/layerindex/maintemail.txt| 9 + > > templates/layerindex/maintemailsubject.txt | 1 + > > templates/layerindex/recipedetail.html | 27 +++ > > templates/rrs/maintainers.html | 2 +- > > templates/rrs/recipedetail.html| 27 +++ > > templates/rrs/recipes.html | 2 +- > > 20 files changed, 392 insertions(+), 78 deletions(-) > > create mode 100755 rrs/tools/historytool.py > > create mode 100644 templates/layerindex/maintemail.txt > > create mode 100644 templates/layerindex/maintemailsubject.txt > > > > -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [PATCH 00/10] Misc fixes (cover letter only)
The following changes since commit 61e9b048591adab88d1996646286e1d7df024bea: Fix drop-down alignment on duplicates page (2018-11-20 11:58:42 +1300) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes12 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes12 Paul Eggleton (10): Send people an email when another user adds them as a maintainer Add links to other branch recipes in recipe detail Fix errors due to races deleting bitbake temp files local.conf: allow for upstream_tracking.inc to be missing Use try...finally or with to ensure files get closed Replace use of assert with exceptions docker: add wget to dependencies RRS: fix sorting arrow positioning rrs_maintainer_history: check out layer branch before looking for maintainers.inc RRS: add tool to import/export upstream history data Dockerfile | 1 + conf/local.conf| 2 +- layerindex/bulkchange.py | 18 +- layerindex/update.py | 4 +- layerindex/update_layer.py | 14 +- layerindex/utils.py| 21 ++- layerindex/views.py| 70 ++-- rrs/tools/historytool.py | 196 + rrs/tools/rrs_distros.py | 19 +- rrs/tools/rrs_maintainer_history.py| 45 ++--- rrs/tools/rrs_upgrade_history.py | 4 +- rrs/tools/rrs_upstream_history.py | 3 +- rrs/tools/upgrade_history_internal.py | 3 +- rrs/views.py | 2 + templates/layerindex/maintemail.txt| 9 + templates/layerindex/maintemailsubject.txt | 1 + templates/layerindex/recipedetail.html | 27 +++ templates/rrs/maintainers.html | 2 +- templates/rrs/recipedetail.html| 27 +++ templates/rrs/recipes.html | 2 +- 20 files changed, 392 insertions(+), 78 deletions(-) create mode 100755 rrs/tools/historytool.py create mode 100644 templates/layerindex/maintemail.txt create mode 100644 templates/layerindex/maintemailsubject.txt -- 2.17.2 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [Recipe reporting system] Upgradable recipe name list
.1 Derek Straka > > > python-numpy 1.14.5 1.15.4Derek Straka > > > python3-numpy 1.14.5 1.15.4Derek Straka > > > python-setuptools 40.0.0 40.6.3Derek Straka > > > python3-six 1.11.0 1.12.0Derek Straka > > > python3-gitdb 2.0.42.0.5 Derek Straka > > > python3-pycairo 1.15.6 1.18.0Derek Straka > > > python3-native3.5.63.7.1 Derek Straka > > > acpica20180508 20181213 Fathi Boudra > > > libgpg-error 1.32 1.33 Hongxu Jia > > > gnupg 2.2.11 2.2.12Hongxu Jia > > > help2man-native 1.47.6 1.47.8Hongxu Jia > > > libxcrypt 4.1.14.4.1 Khem Raj > > > go-runtime1.11.1 1.11.4Khem Raj > > > go1.11.1 1.11.4Khem Raj > > > binutils 2.31 2.31.1Khem Raj > > > go-native 1.11.1 1.11.4Khem Raj > > > prelink 1.0+gitX 1.0-new-commits-a... Mark Hatle > > > db5.3.28 5.3.28+dfsg1 Mark Hatle > > > binutils-cross-ca... 2.31 2.31.1No maintainer > > > binutils-cross-i586 2.31 2.31.1No maintainer > > > go-cross-canadian... 1.11.1 1.11.4No maintainer > > > go-crosssdk-x86_6... 1.11.1 1.11.4No maintainer > > > binutils-crosssdk... 2.31 2.31.1No maintainer > > > go-cross-i586 1.11.1 1.11.4No maintainer > > > mesa 18.1.9 18.3.1Otavio > > > Salvador > > > linux-firmware0.0+gitX 0.0-new-commits-a... Otavio > > > Salvador > > > mesa-gl 18.1.9 18.3.1Otavio > > > Salvador > > > cmake 3.12.2 3.13.2Pascal Bach > > > cmake-native 3.12.2 3.13.2Pascal Bach > > > build-compare 2015.02.10+gitX 2015.02.10-new-co... Paul Eggleton > > > build-appliance-i... 15.0.0 20.0.0Richard > > > Purdie > > > gnu-config20180713+gitX20180713-new-comm... Robert Yang > > > squashfs-tools4.3+gitrX4.3-new-commits-a... Robert Yang > > > e2fsprogs 1.44.3 1.44.5Robert Yang > > > ccache3.4.23.5 Robert Yang > > > git 2.18.1 2.20.1Robert Yang > > > mklibs-native 0.1.43 0.1.44Robert Yang > > > piglit1.0+gitrX1.0-new-commits-a... Ross Burton > > > acpid 2.0.30 2.0.31Ross Burton > > > libinput 1.12.3 1.12.4Ross Burton > > > vulkan-demos git git-new-commits-a... Ross Burton > > > perl 5.24.4 5.28.1Ross Burton > > > gtk-icon-utils-na... 3.22.30 3.24.2Ross Burton > > > ofono 1.25 1.26 Ross Burton > > > libical 2.0.03.0.4 Ross Burton > > > vulkan1.1.73.0 1.1.92.1 Ross Burton > > > lsbinitscripts9.79 10.01 Ross Burton > > > sysvinit 2.88dsf 2.93 Ross Burton > > > adwaita-icon-theme3.28.0 3.30.1Ross Burton > > > fontconfig2.12.6 2.13.1Ross Burton > > > gdk-pixbuf2.36.11 2.38.0Ross Burton > > > gtk+3 3.24.1 3.24.2Ross Burton > > > pkgconf 1.5.31.5.4 Ross Burton > > > perl-native 5.24.4 5.28.1Ross Burton > > > alsa-tools1.1.61.1.7 Tanu Kaskinen > > > alsa-plugins 1.1.61.1.7 Tanu Kaskinen > > > alsa-utils-scripts1.1.61.1.7 Tanu Kaskinen > > > alsa-lib 1.1.61.1.7 Tanu Kaskinen > > > alsa-utils1.1.61.1.7 Tanu Kaskinen > > > gnu-efi 3.0.83.0.9 Yi Zhao > > > libcap2.25 2.26 Yi Zhao > > > chkconfig 1.3.58 1.8 Yi Zhao > > > Version 1.5 requires selinux > > > logrotate 3.14.0 3.15.0Yi Zhao > > > file 5.34 5.35 Yi Zhao > > > wget 1.19.5 1.20 Yi Zhao > > > json-glib 1.4.21.4.4 Yi Zhao > > > libsdl2 2.0.82.0.9 Yi Zhao > > > > > > Upgradable count: 147 > > > Upgradable total count: 151 > > > > > > The based commit is: > > > > > > commit c22b0bf66a28324da66caf0660f171cc279a1f2b > > > Author: Robert Yang > > > Date: Tue Dec 18 17:48:22 2018 +0800 > > > > > > cmake-native: Set --parallel for configure > > > > > > This can save do_configure's time from 330s to 60s on my host. > > > > > > Signed-off-by: Robert Yang > > > Signed-off-by: Richard Purdie > > > > > > Any problem, please contact Jose Lamego > > > > > > -- > > > ___ > > > yocto mailing list > > > yocto@yoctoproject.org > > > https://lists.yoctoproject.org/listinfo/yocto > -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 00/30] Misc improvements (cover letter only)
A bunch of assorted improvements for the layer index. The following changes since commit 24b6e13442c403ae6d4a2b1782a5b5f06b643e22: docker: add TASK_LOG_DIR to docker settings file (2018-10-04 11:15:39 +1300) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes11 http://git.yoctoproject.org/cgit.cgi//log/?h=paule/fixes11 Paul Eggleton (30): Track and enable reporting on duplicate inc files Replace nvd3 with Chart.js update: handle if there is no origin/master branch in bitbake repo settings: add settings for CAPTCHA accessibility views: fix incorrect class usage in ClassSearchView Add CAPTCHA to registration/password reset forms Implement "delete account" function Allow user to edit their username in the Edit Profile form Restrict the number of username change attempts Use django-axes to lockout after multiple failed logins requirements.txt: update GitPython dependencies Add a script to help keep requirements.txt up to date Enable password strength validation by default Add some text to the password reset email Improve password form formatting Require re-registration if user changes email address Tweak activation complete page tools/github-fetch: handle non-fast-forward merges tools/github-fetch: fix marking deleted Exclude "Distro-specific" from "unknown/not available" in other distro search Use prop() instead of attr() to set checked property Replace uitablefilter usage with jQuery Add search to duplicates/comparison layer filter dialog Add search to layer dependencies list on edit/submit layer form README: add missing mention of TableSorter requirements.txt: bump versions Return to previous page after Edit profile Make login button return user to previous page Fix Submit/Edit layer form field rendering issue Fix drop-down alignment on duplicates page README| 6 +- TODO | 1 - check_requirements.sh |39 + docker/settings.py|43 +- layerindex/admin.py |10 + layerindex/auth_forms.py |34 + layerindex/auth_views.py |63 + layerindex/context_processors.py | 5 + layerindex/forms.py |28 +- layerindex/migrations/0026_incfile.py |24 + layerindex/models.py |12 + layerindex/recipeparse.py | 4 +- layerindex/static/css/additional.css | 7 +- layerindex/static/css/nv.d3.css | 645 - layerindex/static/js/Chart.js | 14384 layerindex/static/js/Chart.min.js |10 + layerindex/static/js/d3.js| 9275 -- layerindex/static/js/nv.d3.js | 14369 --- layerindex/static/js/uitablefilter.js |91 - layerindex/tools/github-fetch.py | 6 +- layerindex/update.py | 8 +- layerindex/update_layer.py|24 +- layerindex/views.py |83 +- password_validation.py|25 + requirements.txt |44 +- rrs/models.py | 2 +- rrs/static/js/uitablefilter.js| 117 - rrs/views.py |22 +- settings.py |43 +- templates/base.html | 6 +- templates/layerindex/about.html | 3 +- templates/layerindex/classicrecipes.html |36 +- templates/layerindex/classicstats.html|70 +- templates/layerindex/deleteaccount.html |56 + templates/layerindex/detail.html |33 +- templates/layerindex/duplicates.html |72 +- templates/layerindex/editlayer.html |27 +- templates/layerindex/layerchecklist.html | 2 +- templates/layerindex/layers.html |83 +- templates/layerindex/profile.html | 5 +- .../registration/activation_complete.html | 8 +- templates/registration/login.html | 3 + .../registration/password_change_form.html|23 +- .../registration/password_reset_confirm.html |23 +- .../registration/password_reset_done.html | 2 +- .../registration/password_reset_email.txt | 7 +- .../registration/password_reset_form.html |23 +- templates/registration/registration_form.html |23 +- templates/registration/reregister.html| 6 + templates/rrs/maintainers.html|13 +- templates/rrs/recipes.html| 6 +- templates/rrs/rrs_stats.html |75 +- urls.py
Re: [yocto] Yocto layers missing thud branches
On Monday, 19 November 2018 12:11:35 AM NZDT Max Krummenacher wrote: > Am Samstag, den 17.11.2018, 15:50 -0800 schrieb akuster808: > > Can the maintainers of meta-qt3, meta-qt4, meta-selinux, and meta-cgl > > please add a "Thud" branch > > > While at it, the following patches declaring thud compatibility are not > yet applied: > https://lists.yoctoproject.org/pipermail/yocto/2018-October/042780.html > https://lists.yoctoproject.org/pipermail/yocto/2018-October/042922.html > https://lists.yoctoproject.org/pipermail/yocto/2018-October/042923.html I have just taken care of meta-qt3 and meta-qt4, FWIW. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [OE-core] 2.6 migration guide
On Friday, 9 November 2018 11:41:42 AM NZDT Paul Eggleton wrote: > You could accomplish the latter using PACKAGECONFIG_pn-python3 = "pgo" at > the configuration level Oops, I meant PACKAGECONFIG_remove_pn-python3 = "pgo" here. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] 2.6 migration guide
On Monday, 5 November 2018 4:32:26 PM NZDT Paul Eggleton wrote: > On Wednesday, 31 October 2018 11:06:31 AM NZDT Scott Rifenbark wrote: > > I have an initial section at > > https://yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#moving-to-the-yocto-project-2.6-release, > > which is based on Richard's input. I am sure there are more items. > > OK, so I have removed Richard's items from the wiki page (since they're now > in the manual) and added items I have gathered from reviewing all of the > git commits in the release. Let me know if you need clarification on anything > - everyone else, let us know if I missed or messed up something. > > https://wiki.yoctoproject.org/wiki/FutureMigrationGuide I forgot one thing: --- snip --- Python 3 profile-guided optimisation --- The python3 recipe now enables profile-guided optimisation; this requires a little extra build time in exchange for improved performance on the target at runtime, and is only enabled if the current MACHINE has support for user-mode emulation in QEMU (i.e. "qemu-usermode" is in MACHINE_FEATURES, which it is by default, so the machine configuration would need to have opted out for it not to be). If you wish to disable Python profile-guided optimisation regardless of the value of MACHINE_FEATURES, then ensure that PACKAGECONFIG for python3 does not contain "pgo". You could accomplish the latter using PACKAGECONFIG_pn-python3 = "pgo" at the configuration level or by setting PACKAGECONFIG using a bbappend for the python3 recipe. --- snip --- > BTW I think the section on postinstalls (from Richard's input) is a little > terse > and needs expansion - most importantly it needs to describe what actions the > user might need to take. I'll take care of this on review if nobody else does. So what I think we need here is to copy part of the "Using exit 1 to explicitly defer a postinstall script until first boot..." paragraph that's mentioned in the 2.5 migration section. Would it make sense to actually break this out to its own place in the documentation (on "Deferring postinstalls to first boot") and then link to that from both places? While I think of it, could you also move the _remove item that's in the "Bitbake Changes" section into the "Override Changes" section? Thanks, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] 2.6 migration guide
On Wednesday, 31 October 2018 11:06:31 AM NZDT Scott Rifenbark wrote: > I have an initial section at > https://yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#moving-to-the-yocto-project-2.6-release, > which is based on Richard's input. I am sure there are more items. OK, so I have removed Richard's items from the wiki page (since they're now in the manual) and added items I have gathered from reviewing all of the git commits in the release. Let me know if you need clarification on anything - everyone else, let us know if I missed or messed up something. https://wiki.yoctoproject.org/wiki/FutureMigrationGuide BTW I think the section on postinstalls (from Richard's input) is a little terse and needs expansion - most importantly it needs to describe what actions the user might need to take. I'll take care of this on review if nobody else does. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] adding new layer
Hi Zoli
On Thursday, 25 October 2018 2:36:31 AM NZDT Zolee K wrote:
> I created a new layer, modified the conf file content according to the
> decumentation:
> # We have a conf and classes directory, add to BBPATH
> BBPATH .= ":${LAYERDIR}"
>
> # We have recipes-* directories, add to BBFILES
> BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
> ${LAYERDIR}/recipes-*/*/*.bbappend"
>
> BBFILE_COLLECTIONS += "saxo"
> BBFILE_PATTERN_saxo = "^${LAYERDIR}/"
> BBFILE_PRIORITY_saxo = "4"
>
> I copied the proftpd folder with a working bbappend file into the meta-saxo
> folder. But during bitbake I got an error massage:
> Warning: No bb files matched BBFILE_PATTERN_saxo
> '^/home/kz/oe-core/build/../stuff/meta-saxo/'
>
> I played around with the priority without success.
>
> Do you have any idea what am I doing wrong? I'd like to collect all my
> modifications in 1 layer.
If you have .bbappend file(s) but no .bb files in your layer you would expect
that message, it's telling you one of the entries in BBFILES is not matching.
There are several options here:
1) You can ignore the warning (it's just a warning)
2) Remove the .bb entry from BBFILES in meta-saxo/conf/layer.conf (you can add
it back if and when you need it - of course that does mean you could forget to
do that and then wonder why adding a .bb file doesn't work)
3) Set BBFILE_PATTERN_IGNORE_EMPTY_saxo = "1" to disable the warning.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] 2.6 migration guide
Hi folks When Scott prepares the migration guide for a release, he usually looks to me to collate the notable changes for it (those requiring user intervention). I haven't prepared anything yet though and the 2.6 release is fast approaching. Can anyone who has been involved in or is aware of such changes in master for upcoming 2.6 (thud) make a note of it on the wiki in raw form on this page: https://wiki.yoctoproject.org/wiki/FutureMigrationGuide Scott and I will take care of cleaning it up, so things added there don't need to be perfect. I will also be doing my usual trawl through the commits in the release for the more general changelog, so most things will get picked up that way, but it's really helpful if people with first-hand knowledge of the implications of some of these changes are involved in how they get documented. Thanks, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] NAME missing in mpc8315e-rdb.conf
On Tuesday, 23 October 2018 11:12:34 PM NZDT Jon Mason wrote: > A quick grep over all of the native config files shows that of all the > "#@TYPE: Machine", only one lacks a "#@NAME", mpc8315e-rdb.conf. > > It appears that the file OE version of the file has a real name and > description. See > http://git.openembedded.org/openembedded/plain/conf/machine/mpc8315e-rdb.conf > > Is there a reason to not at least copy this and have it provide some > useful information? Yes please - this would show up in the layer index so it is worth having: http://layers.openembedded.org/layerindex/branch/master/machines/? q=mpc8315e=1 Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Working on Dependent Recipes in eSDK
I think we would *like* this to be possible, from what I can tell I think the issue is that the copies of the library in the sysroots of other dependent recipes aren't being updated. The eSDK (and devtool) were originally written when there weren't recipe-specific sysroots, and this stuff was a bit simpler. Moving back to the full build system won't automatically fix this, though it does make workarounds possible - you'd need to rebuild the dependent recipes to get their sysroots updated. Cheers, Paul On Friday, 19 October 2018 7:30:17 AM NZDT Alexander Kanavin wrote: > I don’t think esdk is designed for this kind of ‘full stack’ development. > You can go back to working directly with yocto layers, and skip the esdk > altogether. Devtool will work exactly same (minus the sdk specific > commands). > > Alex > > > On 18 Oct 2018, at 19.37, aaron_wri...@selinc.com wrote: > > > > My team recently started using the eSDK for day-to-day development. > > Previously, they were using the bitbake command to build, which works, but > > has overhead. Yesterday three different people came up to me a question > > about workflow with the eSDK, and they all had to do with dependent > > recipes in the eSDK. > > > > For example, a developer would sit down to make something new that relies > > on a library provided by a recipe in the eSDK. So they "devtool > > sdk-install needed-library", and get to work. While working, they > > discover, I need to add something to this library I depend on so that it > > does what I want in the way I want. So they "devtool modify > > needed-library" and change the source of that library to do what they > > want, and build that library using its build system (CMake or Autotools). > > Then they return to working on the new something and find that none of > > their changes to the library they depend on are available. > > > > At this point, they are lost. They just start trying things to make it > > work, like "devtool build needed-library", which successfully completes, > > but doesn't help. Some people also try to use the build system of the > > library they modified to install it into the machine sysroot of the eSDK > > with a command like, "make install > > DESTDIR=/tmp/sysroots/", and that actually does help. > > They can now use the changes they made to the library in the new something > > they are working on. However, when they use the "devtool sdk-install" > > command again, their changes to the needed library that they installed > > into the machine sysroot are reverted. > > > > So--what is the workflow they are supposed to use? > > > > Pushing the changes to the library upstream, updating the recipe, and > > waiting for a new eSDK is not practical given that the developers may have > > to iterate on the changes to the library quite a bit before they get it > > just right for their new something. > > > > Is there a way to update the eSDK sysroots with recipes that are in the > > workspace? Could "devtool build" also update the eSDK sysroots? Or is > > there a different way to get this done? > -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH] Fix broken regex for recipes with names containing + signs
In order to show bbappends on the recipe detail page we are doing a
regex query to find any whose names match up with the recipe. In the
layer index instance at layers.openembedded.org viewing the recipe
detail page for any recipe whose name contains ++ (e.g. libsigc++-2.0 in
meta-oe) results in an invalid regex and causes a database error. Escape
any + signs in the name used within the regex in order to fix this.
(I wasn't actually able to reproduce this on my own setup despite also
using MariaDB, but I did find that the unescaped query was not correctly
matching records so it needed to be fixed anyway.)
Signed-off-by: Paul Eggleton
---
layerindex/views.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/layerindex/views.py b/layerindex/views.py
index 4eed0cef..5ac7ffab 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -869,6 +869,7 @@ class RecipeDetailView(DetailView):
if recipe:
verappendprefix = recipe.filename.split('.bb')[0]
appendprefix = verappendprefix.split('_')[0]
+appendprefix = appendprefix.replace('+', r'\+')
#context['verappends'] =
BBAppend.objects.filter(layerbranch__branch=recipe.layerbranch.branch).filter(filename='%s.bbappend'
% verappendprefix)
context['appends'] =
BBAppend.objects.filter(layerbranch__branch=recipe.layerbranch.branch).filter(filename__regex=r'^%s(_[^_]*)?\.bbappend'
% appendprefix)
verappends = []
--
2.17.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [OE-core] layers.openembedded.org upgraded
On Sunday, 30 September 2018 6:40:30 AM NZDT Randy MacLeod wrote: > It works for me now but it's not clear why. It seems that > it was a problem with my firefox profile. > > Work log below in case it might help someone else. > > ../Randy > > > I rebooted: >- still no menu. > I logged out and then into another account on the same laptop: >- it worked. > I created another firefox profile in my usual account: >- it worked. > I deleted that profile and used my default profile: >- still no menu. > While visiting layers.openembedded.org, I opened the firefox > console using Shift+Control+K >- I re-loaded the site. >- it worked! > I closed the console: >- it worked! > I restarted firefox: >- it still works! Thanks - I'm also still puzzled by this, but I guess it's good that it apparently fixed itself. If the issue returns or anyone else sees anything strange like this happening then please let me know. Cheers Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 00/12] More minor fixes (cover letter only)
A bunch of fixes for minor issues / missing functionality I've noticed since we did the update of layers.openembedded.org last week. The following changes since commit 2554449baa5dece6b3b2ca027f1a368823e43b00: update: fix regression caused by previous temp dir fix (2018-09-27 15:35:38 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes10 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes10 Paul Eggleton (12): update: fix error on LAYERRECOMMENDS failure admin: allow changing branch on layerbranch records update: use lists for git clone/checkout command parameters forms: fix validation of repository URLs in layer edit form Auto-determine web/file/tree/commit URLs for gitlab.com update: fix confusing debug message on renaming bbappend admin: allow searching on vcs_url on layer/layerbranch Add object string representation to truncation warning message update: fix commands.getstatusoutput errors when parsing python2 branch update: allow patch parsing to work with python 2 Fix wrapping on update detail page Add an email test function layerindex/admin.py | 5 +-- layerindex/bulkchange.py | 2 +- layerindex/forms.py | 2 +- layerindex/models.py | 2 +- layerindex/recipeparse.py| 2 +- layerindex/static/css/additional.css | 1 + layerindex/tools/import_classic.py | 2 +- layerindex/tools/import_layer.py | 7 + layerindex/update.py | 8 - layerindex/update_layer.py | 7 +++-- layerindex/urls.py | 5 ++- layerindex/utils.py | 6 ++-- layerindex/views.py | 46 +++- rrs/tools/common.py | 2 +- templates/base.html | 2 ++ templates/layerindex/editlayer.html | 9 ++ templates/layerindex/testemail.txt | 7 + 17 files changed, 92 insertions(+), 23 deletions(-) create mode 100644 templates/layerindex/testemail.txt -- 2.17.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [OE-core] layers.openembedded.org upgraded
Hi Randy On Friday, 28 September 2018 11:22:29 AM NZST Randy MacLeod wrote: > On 09/27/2018 05:28 PM, Nicolas Dechesne wrote: > > On Thu, Sep 27, 2018 at 10:43 PM Paul Eggleton > > wrote: > >> I'm very happy to announce that we've finally been able to upgrade the > >> layer > >> index at http://layers.openembedded.org to the latest revision on master, > >> incorporating quite a bit of work that's been going on for the past few > >> months. Improvements now visible: > > Nice new features, thanks! > > The "Branch:" and "Filter layers" selection menus don't work > for me when using Firefox 62.0 (64-bit) on Ubuntu-18.04. > > Google Chrome Version 69.0.3497.100 (Official Build) (64-bit) > works fine. Hmm, I do my development with Firefox and I just checked - with Firefox 62.0 (64-bit) here on Fedora 28 both work. Do you perhaps have an add-on enabled that might prevent these from working? I believe they both rely on javascript (the filter dropdown definitely does). Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] layers.openembedded.org upgraded
Hi all, I'm very happy to announce that we've finally been able to upgrade the layer index at http://layers.openembedded.org to the latest revision on master, incorporating quite a bit of work that's been going on for the past few months. Improvements now visible: * Patch tracking - each recipe detail page now lists any patches being applied by the recipe along with upstream status for each - see attached screenshot. You can click through to view/download the actual patch, and any URLs in the supplemental status text are also made into clickable links. * Source tracking - remote entries in SRC_URI are now listed on the recipe detail page and made into clickable links where possible - see attached screenshot * Link to inc files - there is now a link in the recipe detail page to any inc files that a recipe includes as long as they are in the same directory, as a shortcut to see the rest of the definitions for the recipe. * Recipe list CSV export - there is now an "Export CSV" button at the top of the recipe list on the layer detail page. This currently includes the recipe name and version - we could look at extending this, but note that the REST API provides access to all of the information programmatically and may be better suited for many applications that need this data. * Site-wide notice support - admins can now add notifications to appear at the top of the page across the entire site. This is useful in the case where there is some problem with the update process or maintenance is going on as happens from time to time. * Bootstrap 3 - the UI has been updated to use Bootstrap 3 from version 2 that we were using previously. This has made a fairly minor difference to the UI (padding/spacing/fonts have changed a little) but has allowed us to tidy up a few things in the code. * The "Base" layer type is no longer selectable for layer submissions. I noticed people sometimes selected this erroneously; it's only applicable for openembedded-core and meta-oe basically so that they show up at the top of the layer list. Only admins can now select this type for a layer. * Numerous other bugfixes, robustness improvements and code cleanups. Thanks very much to everyone who has contributed to the layer index code up to now (and to BitBake / tinfoil, which we rely upon to extract the information from the metadata), but I'd like to give particular thanks to Michael Halstead, Yi Zhao, Konrad Scherer, Robert Yang and Aníbal Limón for making this upgrade possible. Also integrated were the Recipe Reporting System (RRS) which powers http://recipes.yoctoproject.org and other distro comparison support, but these will take a bit more time to properly enable so I'll send out a separate email with further details when they are ready. As always, please let me know if you have any comments or notice any issues. (I've already seen a few minor problems in the update logs so I'll be looking into those.) Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre-- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH] update: fix regression caused by previous temp dir fix
In c26604146a74149487a1a2dfc40d40d53aa68bdf I made a fix to change where
the bitbake code writes out bitbake.lock and other files it creates
during parsing, but didn't adequately test it and it turns out our
call to delete the temp directory races against bitbake deleting
bitbake.lock and bitbake.sock. For now the simplest way to deal with
this is to ignore the errors since we don't care about these files,
we just want the temp dir gone.
Signed-off-by: Paul Eggleton
---
layerindex/update_layer.py | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index eb04fc44..97b84372 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -16,6 +16,7 @@ from datetime import datetime
import re
import tempfile
import shutil
+import errno
from distutils.version import LooseVersion
import itertools
import utils
@@ -39,6 +40,14 @@ class DryRunRollbackException(Exception):
pass
+def rm_tempdir_onerror(fn, fullname, exc_info):
+# Avoid errors when we're racing against bitbake deleting
bitbake.lock/bitbake.sock
+# (and anything else it happens to create in our temporary build directory
in future)
+if isinstance(exc_info[1], OSError) and exc_info[1].errno == errno.ENOENT:
+pass
+else:
+raise
+
def check_machine_conf(path, subdir_start):
subpath = path[len(subdir_start):]
res = conf_re.match(subpath)
@@ -828,7 +837,7 @@ def main():
logger.debug('Preserving temp directory %s' % tempdir)
else:
logger.debug('Deleting temp directory')
-shutil.rmtree(tempdir)
+shutil.rmtree(tempdir, onerror=rm_tempdir_onerror)
sys.exit(0)
--
2.17.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 0/5] More minor fixes (cover letter only)
A tweak for the other distro comparison modals, some fixes for the browse/search forms on the machines/distros/classes pages, and a fix for where bitbake-cookerdaemon.log gets written out when parsing. The following changes since commit 5edeb03cb1a50d68dc2b60c21f807c1f5c2db295: TODO: remove completed item (2018-09-20 16:04:49 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes9 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes9 Paul Eggleton (5): comparison: use style on hidden select instead of CSS class to hide comparisonrecipeselect: fix logic for setting modal title Select input box on machine/distro/class pages Ensure browse buttons clear input text recipeparse: fix bitbake-cookerdaemon.log being written out to meta/ TODO | 2 -- layerindex/recipeparse.py | 5 + layerindex/static/css/additional.css | 4 templates/layerindex/classes.html | 19 +++-- .../layerindex/comparisonrecipeselect.html| 21 --- .../comparisonrecipeselectdetail.html | 2 +- templates/layerindex/distros.html | 19 +++-- templates/layerindex/machines.html| 19 +++-- templates/layerindex/recipes.html | 4 ++-- 9 files changed, 68 insertions(+), 27 deletions(-) -- 2.17.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to debug unbuildable module error?
Hi Clay Somewhat late, but I stumbled across this this morning On Monday, 16 July 2018 9:29:07 AM NZST Clay D. Montgomery wrote: > > Is there a way to get bitbake to provide more information about why a > module is unbuildable? > > I am trying to build this example: > > poky/meta-skeleton/recipes-kernel/hello-mod/hello-mod_0.1.bb > > However, bitbake always reports: > > "Runtime target 'hello-mod' is unbuildable, removing..." The actual issue will be printed above this message. Usually it's because the recipe depends upon something else that could not be built (and the reason for that will be in the message above). Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] runtime: add BSP test case for usb storage
On Monday, 24 September 2018 3:02:28 PM NZST Hussin, Mohamad Noor Alim wrote: > This patch is my supersede of earlier "manualbsp" which I split into 2 parts > called usb test and microsd test. OK thanks. > I would use MACHINE_FEATURES = "usbhost" instead of introducing new variable > as it would eliminate problem in case someone try to execute the usb test > case without enable the "HARDWARE_TEST = 1" Sorry I don't understand - do you mean you will use it or you can't? > or download the image form public autobuilder. The test won't do anything in an image by default, only when you're running the test. In any case if the test isn't writing to the device there shouldn't be any major risk. > > Otherwise I agree with Mike's reply, we should avoid writing to the storage > > device as part of the test. > > It is mean that just do test like mount and unmount only? To read something > in storage device we need to write at first place? That's true - but you can still do a read test if you make it a precondition of the test that you write some known file to the storage device before running the test (as part of setup, just as you need to set up the board/device before running any tests - you just need to ensure this gets documented somewhere). Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] runtime: add BSP test case for usb storage
Hi Alim, On Monday, 20 August 2018 3:22:09 AM NZST mohamad.noor.alim.hus...@intel.com wrote: > From: Mohamad Noor Alim Hussin > > Contain test cases to test mount/unmount the usb stick on target > platform such as minnowboard and beaglebone. The test assume that > the usb storage device such as usb thumb drive was plugged into > the target device otherwise the test for would failed. It also test > to read and write from usb thumb drive. Usb test cases start with > mount the usb thumb drive then write and read from it. Lastly, it > will unmount it. If the usb thumb drive unable to mount due to corrupt > of partition or not exists, then the mount test will failed and the > following test would skip. Does this supersede part of your earlier "manualbsp" patch? It's not made clear here or in the cover letter but it looks like it might. The helper script patch needs to come before this patch rather than after it, since this patch won't work without it. > This test need to enable flag 'HARDWARE_TEST = "1"' on conf/local.conf > file in order to run on target device. This test should be skip on qemu. Can we use "usbhost" being in MACHINE_FEATURES and MACHINE not starting with "qemu" to enable this instead of adding another variable? (Assuming that sounds reasonable to everyone else) Otherwise I agree with Mike's reply, we should avoid writing to the storage device as part of the test. Also, as this is patching OE-Core, the patch needs to be sent to the openembedded-c...@lists.openembedded.org mailing list, if you could please use that for the next revision that would be great. Thanks, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 00/10] Bootstrap 3, update task enhancements, etc. (cover letter only)
A bit of a mixed bag: one bugfix, move to Bootstrap 3 for the frontend, some minor features and significant work on how update tasks for the other distro comparison functionality works. The following changes since commit 3aa6bf964d81a04b652499069f0454c546a5e296: import_layer.py: add -t option for layer_type (2018-09-20 15:53:07 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes8 http://git.yoctoproject.org/cgit.cgi//log/?h=paule/fixes8 Paul Eggleton (10): import_otherdistro: fix update recording code Upgrade to Bootstrap 3 Add admin link to tools dropdown menu models: add a get_checkout_branch() function Show actual branch on layer detail Show update task output more smoothly Show progress when running comparison update tasks Properly show update task success/failure Allow stopping update task TODO: remove completed item README| 4 +- TODO | 1 - docker/settings.py| 4 +- layerindex/forms.py |35 +- layerindex/migrations/0025_update_retcode.py |20 + layerindex/models.py |16 +- layerindex/static/css/additional.css |59 +- .../static/css/bootstrap-responsive.css | 1109 -- .../static/css/bootstrap-responsive.min.css | 9 - layerindex/static/css/bootstrap-theme.css | 587 + layerindex/static/css/bootstrap-theme.css.map | 1 + layerindex/static/css/bootstrap-theme.min.css | 6 + .../static/css/bootstrap-theme.min.css.map| 1 + layerindex/static/css/bootstrap.css | 10591 layerindex/static/css/bootstrap.css.map | 1 + layerindex/static/css/bootstrap.min.css |13 +- layerindex/static/css/bootstrap.min.css.map | 1 + .../fonts/glyphicons-halflings-regular.eot| Bin 0 -> 20127 bytes .../fonts/glyphicons-halflings-regular.svg| 288 + .../fonts/glyphicons-halflings-regular.ttf| Bin 0 -> 45404 bytes .../fonts/glyphicons-halflings-regular.woff | Bin 0 -> 23424 bytes .../fonts/glyphicons-halflings-regular.woff2 | Bin 0 -> 18028 bytes .../static/img/glyphicons-halflings-white.png | Bin 8777 -> 0 bytes .../static/img/glyphicons-halflings.png | Bin 12799 -> 0 bytes layerindex/static/js/bootstrap.js | 3331 ++--- layerindex/static/js/bootstrap.min.js |11 +- layerindex/static/js/jquery-1.7.2.js | 9404 -- layerindex/static/js/jquery-3.3.1.js | 10364 +++ layerindex/tasks.py |23 +- layerindex/templatetags/extrafilters.py |21 +- layerindex/tools/import_otherdistro.py|14 +- layerindex/urls.py| 8 +- layerindex/utils.py | 117 +- layerindex/views.py |52 +- rrs/static/css/rrs-additional.css | 7 +- settings.py | 7 +- templates/base.html |48 +- templates/base_toplevel.html |12 +- templates/layerindex/about.html | 4 +- templates/layerindex/bulkchange.html |10 +- templates/layerindex/bulkchangeedit.html | 8 +- templates/layerindex/bulkchangereview.html| 8 +- templates/layerindex/bulkchangesearch.html|22 +- templates/layerindex/classes.html |19 +- templates/layerindex/classic_base.html| 4 +- templates/layerindex/classicrecipedetail.html | 4 +- templates/layerindex/classicrecipes.html | 166 +- templates/layerindex/classicstats.html| 2 +- .../layerindex/comparisonrecipebase.html |10 +- .../layerindex/comparisonrecipeselect.html|97 +- .../comparisonrecipeselectdetail.html |90 +- templates/layerindex/deleteconfirm.html | 4 +- templates/layerindex/detail.html | 122 +- templates/layerindex/distros.html |20 +- templates/layerindex/duplicates.html |92 +- templates/layerindex/editlayer.html |32 +- templates/layerindex/editlayernote.html | 4 +- templates/layerindex/layers.html |50 +- templates/layerindex/machines.html|20 +- templates/layerindex/profile.html | 8 +- templates/layerindex/recipedetail.html|30 +- templates/layerindex/recipes.html |25 +- templates/layerindex/reviewdetail.html|32 +- templates/layerindex/reviewlist.html | 6 +- templates/layerindex/stats.html | 2 +- templates/layerindex/task.html|94 +- templates/layerindex/updatedetail.html| 6 +- templates/layerindex/updatelist.html |
[yocto] [layerindex-web][PATCH 00/15] More minor fixes (cover letter only)
The following changes since commit f82f5f2754dd2d41db8d62f455dd380c26635ee1: update: add option to stop immediately on parse errors (2018-08-24 16:41:59 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes7 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes7 Paul Eggleton (15): docker: remove one step from proxy setup docker: correct comment in .gitconfig update: do not try to parse a layer that failed to fetch Disable git interactive password prompts Limit selection of "Base" layer type reviewdetail: show newlines in description rrs/tools: handle dry-run properly rrs_distros: ensure we only run distro processing once rrs: fix recipe count not showing up properly if no per-recipe maintainers rrs: fix broken links when there are no milestones in a release rrs: show warnings for missing current release/milestone rrs_distros: match recipe on filename not PN rrs: admin: add in-line for Milestones to Releases rrs/tools/daily_run.sh: don't reload when updating TODO: add some more items Dockerfile | 8 +-- TODO | 13 ++-- docker/.gitconfig | 2 +- layerindex/forms.py| 9 ++- layerindex/update.py | 9 +++ layerindex/views.py| 7 ++- rrs/admin.py | 19 ++ rrs/models.py | 2 +- rrs/tools/common.py| 5 ++ rrs/tools/daily_run.sh | 2 +- rrs/tools/rrs_distros.py | 63 ++- rrs/tools/rrs_upgrade_history.py | 15 - rrs/tools/rrs_upstream_history.py | 83 +- rrs/views.py | 10 templates/layerindex/editlayer.html| 4 ++ templates/layerindex/reviewdetail.html | 2 +- templates/rrs/recipes.html | 4 ++ 17 files changed, 170 insertions(+), 87 deletions(-) -- 2.17.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] RFC: layer index docker fixes
On Thursday, 30 August 2018 11:44:10 AM NZST Paul Eggleton wrote: > On Wednesday, 29 August 2018 10:46:27 PM NZST Paul Eggleton wrote: > > I know Konrad promised to update the docker-compose setup to be closer > > to what we'd done recently with the docker setup, but I needed something > > urgently and wanted to learn how to use docker-compose anyway, so I've > > put this together: > > > > > > http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose > > > > Let me know what you think - it works well enough for me here, although the > > setup commands are now a bit ugly on their own. > > I realised this morning that "docker-compose run" can simplify a lot of the > docker run commands, so I've just converted most of the commands in > docker/README and pushed the branch again. So, any comments? Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [Yocto-bsp] [poky] Yocto kernel compilation issues
On Wednesday, 5 September 2018 1:44:22 PM NZST srinivasan wrote: > Adding the "DEPENDS += "xz-native bc-native" in the "linux-srini_4.14.bb" > did the trick FYI the main issue with your kernel recipe is it does not "inherit kernel" which takes care of this (and many other things) for you. I guess you based your recipe on linux-dummy.bb, but that recipe is special and shouldn't be used as a template for a real kernel. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto-bsp mailing list yocto-bsp@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto-bsp
Re: [yocto] [layerindex-web][PATCH 1/1] import_layer.py: add -t option for layer_type
Hi Robert
Sorry I meant to reply earlier.
I'm not especially happy with moving the constants to utils.py. Is there any
way we can avoid that?
Thanks
Paul
On Thursday, 30 August 2018 2:05:11 PM NZST Robert Yang wrote:
> Hi Paul,
>
> Do you have any comments on this, please ?
>
> // Robert
>
> On 07/31/2018 03:57 PM, Robert Yang wrote:
> > Hi Paul,
> >
> > On 07/24/2018 04:33 PM, Paul Eggleton wrote:
> >> Hi Robert,
> >>
> >> On Monday, 23 July 2018 12:30:01 PM CEST Robert Yang wrote:
> >>> Now the logic is:
> >>> Use options.layer_type if specified, guess if not, default to 'M'.
> >>>
> >>> Note choices=['A', 'B', 'S', 'D', 'M', ''], the '' is for default='', we
can't
> >>> use default='M' here, otherwise we don't know whether the 'M' is
specified by
> >>> user or not, we don't guess if it is specified by user, otherwise,
guess.
> >>>
> >>> Signed-off-by: Robert Yang
> >>> ---
> >>> layerindex/tools/import_layer.py | 18 +-
> >>> 1 file changed, 13 insertions(+), 5 deletions(-)
> >>>
> >>> diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/
import_layer.py
> >>> index 2413cff..0d501f6 100755
> >>> --- a/layerindex/tools/import_layer.py
> >>> +++ b/layerindex/tools/import_layer.py
> >>> @@ -189,6 +189,10 @@ def main():
> >>> parser.add_option("-s", "--subdir",
> >>> help = "Specify subdirectory",
> >>> action="store", dest="subdir")
> >>> +parser.add_option("-t", "--type",
> >>> +help = "Specify layer type. A: Base, B: Machine(BSP), S:
> >>> Software, D: Distribution, M: Miscellaneous",
> >>> +choices=['A', 'B', 'S', 'D', 'M', ''],
> >>> +action="store", dest="layer_type", default='')
> >>
> >> If it's practical to do, could you make this part of the code use
> >> LayerItem.LAYER_TYPE_CHOICES from models.py so that this will work
> >> if that is extended in future? Of course that will mean initialising
Django
> >> earlier, I'm not sure if that will have any side-effects that we wouldn't
want.
> >> If you could give that a try though that would be great.
> >
> > I think that the easies way is move the definition of LAYER_TYPE_CHOICES
> > to utils.py, then we can use utils.LAYER_TYPE_CHOICES in both models.py
> > and tools/import_layer.py
> >
> > I've updated it in the repo:
> >
> >git://git.pokylinux.org/poky-contrib rbt/li
> >http://git.pokylinux.org/cgit.cgi//log/?h=rbt/li
> >
> > commit aeb1fd6a8f10b40637121643fc7df2f17c9cd87c
> > Author: Robert Yang
> > Date: Mon Jul 9 15:28:10 2018 +0800
> >
> > import_layer.py: add -t option for layer_type
> >
> > Now the logic is:
> > Use options.layer_type if specified, and guess if not. Default to
'M'.
> >
> > Note choices=['A', 'B', 'S', 'D', 'M', ''], the '' is for default='',
we can't
> > use default='M' here, otherwise we don't know whether the 'M' is
specified by
> > user or is the default value, we don't guess if it is specified by
user,
> > otherwise, guess.
> >
> > Signed-off-by: Robert Yang
> >
> > diff --git a/layerindex/models.py b/layerindex/models.py
> > index e5c8052..1ff653f 100644
> > --- a/layerindex/models.py
> > +++ b/layerindex/models.py
> > @@ -102,14 +102,8 @@ class LayerItem(models.Model):
> > ('P', 'Published'),
> > ('X', 'No update'),
> > )
> > -LAYER_TYPE_CHOICES = (
> > -('A', 'Base'),
> > -('B', 'Machine (BSP)'),
> > -('S', 'Software'),
> > -('D', 'Distribution'),
> > -('W', 'WRTemplates'),
> > -('M', 'Miscellaneous'),
> > -)
> > +LAYER_TYPE_CHOICES = utils.LAYER_TYPE_CHOICES
> > +
> > name = models.CharField('Layer name', max_length=40, unique=True,
> > help_text='Name of the layer - must be unique and can only contain
letters,
> > numbers and dashes')
> > status = models.CharField(max_length=1,
choices=LAYER_STATUS_CHOICES,
> > default='N')
> > layer_type = models.CharField(max_length=1,
choices=LAYER_TYPE_CHOICES)
> > diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/
import_layer.py
> > index 2413
Re: [yocto] [layerindex-web] RFC: layer index docker fixes
On Wednesday, 29 August 2018 10:46:27 PM NZST Paul Eggleton wrote: > Hi folks > > I know Konrad promised to update the docker-compose setup to be closer > to what we'd done recently with the docker setup, but I needed something > urgently and wanted to learn how to use docker-compose anyway, so I've > put this together: > > > http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose > > Let me know what you think - it works well enough for me here, although the > setup commands are now a bit ugly on their own. I realised this morning that "docker-compose run" can simplify a lot of the docker run commands, so I've just converted most of the commands in docker/README and pushed the branch again. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] RFC: layer index docker fixes
Hi folks I know Konrad promised to update the docker-compose setup to be closer to what we'd done recently with the docker setup, but I needed something urgently and wanted to learn how to use docker-compose anyway, so I've put this together: http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/docker-compose Let me know what you think - it works well enough for me here, although the setup commands are now a bit ugly on their own. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 3/3] update: add option to stop immediately on parse errors
During debugging a parsing issue, we don't really want to continue if a
parsing error occurs, and in that situation I usually end up using
Ctrl+C to exit early. Add an option to exit immediately upon error to
avoid having to do that.
Signed-off-by: Paul Eggleton
---
layerindex/update.py | 8
layerindex/update_layer.py | 39 +++---
2 files changed, 32 insertions(+), 15 deletions(-)
diff --git a/layerindex/update.py b/layerindex/update.py
index aea7cd09..1b2bb153 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -57,6 +57,8 @@ def prepare_update_layer_command(options, branch, layer,
initial=False):
cmd += ' -q'
if options.keep_temp:
cmd += ' --keep-temp'
+if options.stop_on_error:
+cmd += ' --stop-on-error'
return cmd
def update_actual_branch(layerquery, fetchdir, branch, options,
update_bitbake, bitbakepath):
@@ -159,6 +161,9 @@ def main():
parser.add_option("", "--nocheckout",
help = "Don't check out branches",
action="store_true", dest="nocheckout")
+parser.add_option("", "--stop-on-error",
+help = "Stop on first parsing error",
+action="store_true", default=False, dest="stop_on_error")
parser.add_option("-a", "--actual-branch",
help = "Update actual branch for layer and bitbake",
action="store", dest="actual_branch", default='')
@@ -528,6 +533,9 @@ def main():
# Interrupted by user, break out of loop
logger.info('Update interrupted, exiting')
sys.exit(254)
+if options.stop_on_error and ret != 0:
+logger.info('Layer update failed with --stop-on-error,
stopping')
+sys.exit(1)
if failed_layers:
for branch, err_msg_list in failed_layers.items():
if err_msg_list:
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index d9c93360..eb04fc44 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -57,7 +57,7 @@ def split_recipe_fn(path):
patch_status_re = re.compile(r"^[\t ]*(Upstream[-_ ]Status:?)[\t ]*(\w+)([\t
]+.*)?", re.IGNORECASE | re.MULTILINE)
-def collect_patch(recipe, patchfn, layerdir_start):
+def collect_patch(recipe, patchfn, layerdir_start, stop_on_error):
from django.db import DatabaseError
from layerindex.models import Patch
@@ -93,10 +93,13 @@ def collect_patch(recipe, patchfn, layerdir_start):
except DatabaseError:
raise
except Exception as e:
-logger.error("Unable to read patch %s: %s", patchfn, str(e))
-patchrec.save()
+if stop_on_error:
+raise
+else:
+logger.error("Unable to read patch %s: %s", patchfn, str(e))
+patchrec.save()
-def collect_patches(recipe, envdata, layerdir_start):
+def collect_patches(recipe, envdata, layerdir_start, stop_on_error):
from layerindex.models import Patch
try:
@@ -111,9 +114,9 @@ def collect_patches(recipe, envdata, layerdir_start):
if not patch.startswith(layerdir_start):
# Likely a remote patch, skip it
continue
-collect_patch(recipe, patch, layerdir_start)
+collect_patch(recipe, patch, layerdir_start, stop_on_error)
-def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir,
skip_patches=False):
+def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir,
stop_on_error, skip_patches=False):
from django.db import DatabaseError
fn = str(os.path.join(path, recipe.filename))
@@ -197,7 +200,7 @@ def update_recipe_file(tinfoil, data, path, recipe,
layerdir_start, repodir, ski
if not skip_patches:
# Handle patches
-collect_patches(recipe, envdata, layerdir_start)
+collect_patches(recipe, envdata, layerdir_start, stop_on_error)
# Get file dependencies within this layer
deps = envdata.getVar('__depends', True)
@@ -235,9 +238,12 @@ def update_recipe_file(tinfoil, data, path, recipe,
layerdir_start, repodir, ski
except DatabaseError:
raise
except BaseException as e:
-if not recipe.pn:
-recipe.pn = recipe.filename[:-3].split('_')[0]
-logger.error("Unable to read %s: %s", fn, str(e))
+if stop_on_error:
+raise
+else:
+if not recipe.pn:
+recipe.pn = recipe.filename[:-3].split('_')[0]
+logger.error("Unable to read %s: %s", fn, str(e))
def update_machine_conf_file(path, machine):
logger.debug('Updating machine %s' % path)
@@ -304,6 +310,9 @@
[yocto] [layerindex-web][PATCH 0/3] Minor layer index fixes
A couple of minor fixes for the recent RRS and other distro integration, plus one additional debugging feature for the update script. The following changes since commit b3e9cb05d22e0961341d87dce3128eeb273544f6: Include errors/warnings in main log in error/warning counts for updates (2018-08-14 15:09:36 +0200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes6 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes6 Paul Eggleton (3): rrs: fix recipe/maintainer count showing at top Hide navbar on comparison select page update: add option to stop immediately on parse errors layerindex/update.py | 8 layerindex/update_layer.py| 39 --- templates/base_toplevel.html | 2 + .../layerindex/comparisonrecipeselect.html| 3 ++ templates/rrs/maintainers.html| 6 +-- templates/rrs/recipes.html| 6 +-- 6 files changed, 43 insertions(+), 21 deletions(-) -- 2.17.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 2/3] Hide navbar on comparison select page
We don't want to see the navbar on this page because otherwise it lets
you select branches and click on Stats etc. which doesn't make any sense
in this context.
Signed-off-by: Paul Eggleton
---
templates/base_toplevel.html | 2 ++
templates/layerindex/comparisonrecipeselect.html | 3 +++
2 files changed, 5 insertions(+)
diff --git a/templates/base_toplevel.html b/templates/base_toplevel.html
index 1dcac775..afcc19bf 100644
--- a/templates/base_toplevel.html
+++ b/templates/base_toplevel.html
@@ -15,6 +15,7 @@
+{% block navbar %}
@@ -53,6 +54,7 @@
{% block navs_extra %}{% endblock %}
+{% endblock %}
{% block content_inner %}{% endblock %}
diff --git a/templates/layerindex/comparisonrecipeselect.html
b/templates/layerindex/comparisonrecipeselect.html
index 1646ae3d..47b299be 100644
--- a/templates/layerindex/comparisonrecipeselect.html
+++ b/templates/layerindex/comparisonrecipeselect.html
@@ -14,6 +14,9 @@
{% block title_append %} - comparison - select match for {{ select_for.name
}}{% endblock %}
-->
+{% block navbar %}
+{% endblock %}
+
{% block page_heading %}
Select match for {{ select_for.pn }} in {% if
branch.comparison %}{{ branch }}{% else %}OpenEmbedded{% endif %}
--
2.17.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 1/3] rrs: fix recipe/maintainer count showing at top
We were using the main badge class as a selector here to show the recipe
or maintainer count, and that meant that it also showed up at the top
next to the login button if there were some layers to review. Use a
proper id to stop that from happening.
Signed-off-by: Paul Eggleton
---
templates/rrs/maintainers.html | 6 +++---
templates/rrs/recipes.html | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/templates/rrs/maintainers.html b/templates/rrs/maintainers.html
index c237000f..cee02006 100644
--- a/templates/rrs/maintainers.html
+++ b/templates/rrs/maintainers.html
@@ -23,7 +23,7 @@
-
+
@@ -188,9 +188,9 @@ $(document).ready(function() {
}
if (count == 2) {
-$('.badge').html("1 maintainer");
+$('#maintainer-count').html("1 maintainer");
} else {
-$('.badge').html((count - 1) + " maintainers")
+$('#maintainer-count').html((count - 1) + " maintainers")
}
updateTotals()
}
diff --git a/templates/rrs/recipes.html b/templates/rrs/recipes.html
index af3c08d6..d917f19d 100644
--- a/templates/rrs/recipes.html
+++ b/templates/rrs/recipes.html
@@ -23,7 +23,7 @@
-
+
@@ -203,9 +203,9 @@ $(document).ready(function() {
}
if (count == 2) {
-$('.badge').html("1 recipe");
+$('#recipe-count').html("1 recipe");
} else {
-$('.badge').html((count - 1) + " recipes");
+$('#recipe-count').html((count - 1) + " recipes");
}
}
--
2.17.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 5/5] Include errors/warnings in main log in error/warning counts for updates
When showing the error/warning counts for update records we need to
include any errors/warnings that are shown only in the main update log,
so we need to adjust how these are collected. Use a function rather than
pure aggregation to give a bit more control, and a {% with ... %} block
in the template to avoid the functions being called more than necessary.
Signed-off-by: Paul Eggleton
---
layerindex/models.py | 8
layerindex/views.py | 2 +-
templates/layerindex/updatelist.html | 6 --
3 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/layerindex/models.py b/layerindex/models.py
index 0dc4bd7c..379fdb6b 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -98,6 +98,14 @@ class Update(models.Model):
task_id = models.CharField(max_length=50, blank=True, db_index=True)
triggered_by = models.ForeignKey(User, blank=True, null=True,
on_delete=models.SET_NULL)
+def error_count(self):
+sums = self.layerupdate_set.aggregate(errors=models.Sum('errors'))
+return (sums['errors'] or 0) + self.log.count('ERROR:')
+
+def warning_count(self):
+sums = self.layerupdate_set.aggregate(warnings=models.Sum('warnings'))
+return (sums['warnings'] or 0) + self.log.count('WARNING:')
+
def __str__(self):
return '%s' % self.started
diff --git a/layerindex/views.py b/layerindex/views.py
index 69165c48..8adebed9 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -708,7 +708,7 @@ class UpdateListView(ListView):
paginate_by = 50
def get_queryset(self):
-return
Update.objects.all().order_by('-started').annotate(errors=Sum('layerupdate__errors'),
warnings=Sum('layerupdate__warnings'))
+return Update.objects.all().order_by('-started')
class UpdateDetailView(DetailView):
diff --git a/templates/layerindex/updatelist.html
b/templates/layerindex/updatelist.html
index d58d175b..5a207ce8 100644
--- a/templates/layerindex/updatelist.html
+++ b/templates/layerindex/updatelist.html
@@ -35,12 +35,14 @@
{% for update in updates %}
+{% with error_count=update.error_count
warning_count=update.warning_count %}
{{
update.started }}{% if update.reload %} (reload){% endif %}
{% if update.finished %}{{
update.started|timesince2:update.finished }}{% else %}(in progress){% endif
%}
-{% if update.errors %}{{ update.errors }}{% endif %}
-{% if update.warnings %}{{ update.warnings }}{% endif %}
+{% if error_count %}{{ error_count }}{% endif %}
+{% if warning_count %}{{ warning_count }}{% endif %}
+{% endwith %}
{% endfor %}
--
2.17.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 4/5] update: ensure that exceptions during the update process are logged
If an exception occurred during the update then we were managing to save
the update record, but we did not include the exception traceback in the
log for the update. Catch the exception and log it which ensures it gets
captured in the update record and still gets printed as well.
Signed-off-by: Paul Eggleton
---
layerindex/update.py | 4
1 file changed, 4 insertions(+)
diff --git a/layerindex/update.py b/layerindex/update.py
index 14529498..aea7cd09 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -540,6 +540,10 @@ def main():
except KeyboardInterrupt:
logger.info('Update interrupted, exiting')
sys.exit(254)
+except Exception:
+import traceback
+logger.error(traceback.format_exc().rstrip())
+sys.exit(1)
finally:
update.log = ''.join(listhandler.read())
update.finished = datetime.now()
--
2.17.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 2/5] update_layer: improve exception handling
When an exception occurs during the main part of update_layer, we were
catching and printing it but that's not enough - we need to do the
following as well:
* Use logger.error() to print the exception information, so that it gets
logged and highlighted as an error in the layer update
* Exit with a non-zero return code so that update.py knows it has failed
Signed-off-by: Paul Eggleton
---
layerindex/update_layer.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 2b39e8ef..d9c93360 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -808,7 +808,8 @@ def main():
pass
except:
import traceback
-traceback.print_exc()
+logger.error(traceback.format_exc().rstrip())
+sys.exit(1)
finally:
if tinfoil and (LooseVersion(bb.__version__) > LooseVersion("1.27")):
tinfoil.shutdown()
--
2.17.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 0/5] Update / error handling fixes
The layer index has recently been failing to update, and that wasn't picked up earlier as the errors weren't actually being collected in the update records. Makesure they are collected and sufficiently highlighted in the UI. The following changes since commit d17080d325b9b82e8b844c207053eb2b3ac111ad: Show links to include files on recipe detail (2018-08-13 16:04:16 +0200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/layerupdate http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/layerupdate Paul Eggleton (5): update: avoid errors if values could not be found in initial output update_layer: improve exception handling Improve collection/display of LayerUpdate records update: ensure that exceptions during the update process are logged Include errors/warnings in main log in error/warning counts for updates TODO | 1 - layerindex/admin.py | 2 +- .../0021_layerupdate_add_layer_branch.py | 26 +++ .../0022_layerupdate_set_layer_branch.py | 41 +++ .../0023_layerupdate_layer_branch_finalise.py | 29 .../migrations/0024_layerupdate_vcs_revs.py | 35 ++ layerindex/models.py | 36 +- layerindex/static/css/additional.css | 10 +++ layerindex/templatetags/extrafilters.py | 4 ++ layerindex/update.py | 68 +-- layerindex/update_layer.py| 3 +- layerindex/views.py | 6 +- templates/layerindex/layerupdate.html | 19 +- templates/layerindex/updatedetail.html| 34 -- templates/layerindex/updatelist.html | 6 +- 15 files changed, 284 insertions(+), 36 deletions(-) create mode 100644 layerindex/migrations/0021_layerupdate_add_layer_branch.py create mode 100644 layerindex/migrations/0022_layerupdate_set_layer_branch.py create mode 100644 layerindex/migrations/0023_layerupdate_layer_branch_finalise.py create mode 100644 layerindex/migrations/0024_layerupdate_vcs_revs.py -- 2.17.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH v2 00/31] Distro comparisons (cover letter only)
After running some more tests with Fedora metadata I've made the
following fixes:
- Turns out fedora golang rpms already have the processed name so we don't
need the stuff in rpmmacros.py, so drop that
- Add missing __grep definition
- Handle unbracketed expressions e.g. %name instead of %{name}
- Handle parentheses around conditional values as some spec files use
these
- Support %undefine
- Fix logic for marking recipes as deleted, since pn might be different
than the name we got from the spec file name
- Drop unused excludevalues parameter to expand() inner function
The following changes since commit fca74f3ac3c6536c50cb0a518fe78447a284dc55:
Update TODO list (2018-08-06 12:36:31 +0200)
are available in the Git repository at:
git://git.yoctoproject.org/layerindex-web paule/otherdistro
http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/otherdistro
Paul Eggleton (31):
forms: clear out comparison recipe/layerbranch if status doesn't
accept them
Add a script to import layers from an existing layer index
update_classic_status: fix matching on wrong layer
Add partial path macro for vcs URL fields
Add support for other distro comparisons
Add script to import from other distros
update_classic_status: update for other distro comparisons
update_classic_status: set cover_pn from pn for direct match
update_classic_status: use source URLs to match python/perl recipes
update_classic_status: look for certain prefixes in order to match
update_classic_status: add skip option
Allow searching for uncategorised recipes in the comparison search
Enhance filtering comparisons
views: fix unknown categories comparison stats graph to include "Not
available"
Add "needs attention" flag for comparison recipes
Add support for importing Debian package info
Support reversed other distro recipe list / filtering
Support selecting more than one layer in other distro comparison
search form
Support excluding inherited classes from other distro comparison
reversed query
Add basic CSV export for other distro comparisons
Add ability to store extra URLs to be displayed for comparison recipes
rrs: show link to other distro comparison pages
Rename LayerItem classic field to comparison
Fix comparison layers showing up in dependencies list in edit layer
page
Add ability to trigger comparison updates manually from UI
Add script to fetch all repos under a github organisation
Add script to fetch Fedora package sources
Show source URLs on recipe detail page
README: add setup info for other distro comparisons
Add side-by-side comparison detail and enhanced selection
Show links to include files on recipe detail
README| 82 ++-
TODO | 2 -
layerindex/admin.py | 6 +
layerindex/context_processors.py | 4 +-
layerindex/forms.py | 37 +-
layerindex/migrations/0015_comparison.py | 24 +
.../migrations/0016_classicrecipe_delete.py | 19 +
.../0017_classicrecipe_needs_attention.py | 19 +
.../migrations/0018_layerrecipeextraurl.py| 26 +
.../0019_layeritem_classic_comparison.py | 25 +
layerindex/migrations/0020_update_manual.py | 50 ++
layerindex/models.py | 113 +++-
layerindex/static/css/additional.css | 20 +
layerindex/tasks.py | 23 +
layerindex/templatetags/extrafilters.py | 35 +
layerindex/tools/fedora-fetch.py | 170 +
layerindex/tools/github-fetch.py | 135
layerindex/tools/import_classic.py| 2 +-
layerindex/tools/import_classic_wiki.py | 2 +-
layerindex/tools/import_layers.py | 251
layerindex/tools/import_otherdistro.py| 605 ++
layerindex/tools/update_classic_status.py | 187 +-
layerindex/update.py | 8 +-
layerindex/urls.py| 48 +-
layerindex/urls_branch.py | 5 +-
layerindex/utils.py | 3 +
layerindex/views.py | 502 ++-
templates/base.html | 2 +
templates/base_toplevel.html | 16 +-
templates/layerindex/classchecklist.html | 21 +
templates/layerindex/classic_base.html| 33 +-
templates/layerindex/classicrecipedetail.html | 184 ++
templates/layerindex/classicrecipes.html | 324 +-
templates/layerindex/classicrecipes_csv.txt | 2 +
templates/layerindex/classicstats.html| 16 +-
.../layerindex/comparisonrecipebase.html | 229 +++
.../layerindex/comparisonrecipeselect.html| 119
.../comparisonrecipeselectdetail.html | 98 +++
templates/layerindex/layerchecklist.html | 21 +
Re: [yocto] Copy git hashes of recipes during the image build
Hi Rajesh, On Wednesday, 8 August 2018 8:52:10 AM CEST Yuvarajesh Valleru wrote: > I am building the custom yocto image regularly so that I do have so may > versions of the image. But when an bug occured in older version images, > I could not find the changes in source code because of AUTOREV. Is there > any possibility to copy the hashes during the build and store in another > recipe, so that I can have all my hashes in my target system. There is indeed a way to collect the revisions, and since the question has come up before I just wrote a FAQ entry about it: https://wiki.yoctoproject.org/wiki/Technical_FAQ#How_do_I_collect_the_source_revisions_fetched_by_each_recipe.3F This wouldn't actually store the revisions in a form that you could then put into the image itself however. If you really wanted to do that you could in theory run buildhistory-collect-srcrevs from a recipe and then package its output though, I think that should work. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 00/31] Distro comparisons (cover letter only)
This set of changes builds upon the OE-Classic support we had previously to enable importing data from other Linux distributions (or indeed other operating systems) and linking and comparing the packages there to recipes in OE. The initial support is just for importing RPM spec files, using the assumption that they can be brought down locally (and scripts to fetch them are provided for Fedora and any nominated Github organisation). There is also basic support for importing Debian package info from a running system (i.e. from the output of apt-cache show "*"). However, the data is not specific and in theory a script could be written to import from just about any system. The following changes since commit fca74f3ac3c6536c50cb0a518fe78447a284dc55: Update TODO list (2018-08-06 12:36:31 +0200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/otherdistro http://git.yoctoproject.org/cgit.cgi//log/?h=paule/otherdistro Paul Eggleton (31): forms: clear out comparison recipe/layerbranch if status doesn't accept them Add a script to import layers from an existing layer index update_classic_status: fix matching on wrong layer Add partial path macro for vcs URL fields Add support for other distro comparisons Add script to import from other distros update_classic_status: update for other distro comparisons update_classic_status: set cover_pn from pn for direct match update_classic_status: use source URLs to match python/perl recipes update_classic_status: look for certain prefixes in order to match update_classic_status: add skip option Allow searching for uncategorised recipes in the comparison search Enhance filtering comparisons views: fix unknown categories comparison stats graph to include "Not available" Add "needs attention" flag for comparison recipes Add support for importing Debian package info Support reversed other distro recipe list / filtering Support selecting more than one layer in other distro comparison search form Support excluding inherited classes from other distro comparison reversed query Add basic CSV export for other distro comparisons Add ability to store extra URLs to be displayed for comparison recipes rrs: show link to other distro comparison pages Rename LayerItem classic field to comparison Fix comparison layers showing up in dependencies list in edit layer page Add ability to trigger comparison updates manually from UI Add script to fetch all repos under a github organisation Add script to fetch Fedora package sources Show source URLs on recipe detail page README: add setup info for other distro comparisons Add side-by-side comparison detail and enhanced selection Show links to include files on recipe detail README| 82 ++- TODO | 2 - layerindex/admin.py | 6 + layerindex/context_processors.py | 4 +- layerindex/forms.py | 37 +- layerindex/migrations/0015_comparison.py | 24 + .../migrations/0016_classicrecipe_delete.py | 19 + .../0017_classicrecipe_needs_attention.py | 19 + .../migrations/0018_layerrecipeextraurl.py| 26 + .../0019_layeritem_classic_comparison.py | 25 + layerindex/migrations/0020_update_manual.py | 50 ++ layerindex/models.py | 113 +++- layerindex/static/css/additional.css | 20 + layerindex/tasks.py | 23 + layerindex/templatetags/extrafilters.py | 35 ++ layerindex/tools/fedora-fetch.py | 170 ++ layerindex/tools/github-fetch.py | 135 + layerindex/tools/import_classic.py| 2 +- layerindex/tools/import_classic_wiki.py | 2 +- layerindex/tools/import_layers.py | 251 layerindex/tools/import_otherdistro.py| 564 ++ layerindex/tools/lib/__init__.py | 0 layerindex/tools/lib/rpmmacros.py | 53 ++ layerindex/tools/update_classic_status.py | 187 +- layerindex/update.py | 8 +- layerindex/urls.py| 48 +- layerindex/urls_branch.py | 5 +- layerindex/utils.py | 3 + layerindex/views.py | 502 ++-- templates/base.html | 2 + templates/base_toplevel.html | 16 +- templates/layerindex/classchecklist.html | 21 + templates/layerindex/classic_base.html| 33 +- templates/layerindex/classicrecipedetail.html | 184 ++ templates/layerindex/classicrecipes.html | 324 +- templates/layerindex/classicrecipes_csv.txt | 2 + templates/layerindex/classicstats.html| 16 +- .../layerindex/comparisonrecipebase.html
[yocto] [layerindex-web][PATCH 0/9] Layer index fixes (cover letter only)
Some fixes for the Docker setup and bulkchange, plus a couple of unrelated patches. The following changes since commit 65d058237f96a151c913a70ba4d639c7aa0fb0b5: Dockerfile: Reduce image size by merging RUN stages together (2018-07-23 08:40:00 +0200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes5 http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes5 Paul Eggleton (9): docker: drop names/emails from settings.py docker: fix copying .gitconfig for proxy setup README: tweak to make Docker setup more prominent docker/README: use -v option for volume mounts docker/README: fix setup to allow bulk change to work bulkchange: make script more robust bulkchange: fix missing closing div for form errors admin: improve layer maintainer handling Update TODO list Dockerfile | 12 ++--- README | 64 TODO | 22 +--- docker/README| 36 +++-- docker/settings.py | 3 +- layerindex/admin.py | 1 + layerindex/bulkchange.py | 21 +--- layerindex/models.py | 2 +- templates/layerindex/bulkchangeedit.html | 3 ++ 9 files changed, 104 insertions(+), 60 deletions(-) -- 2.17.1 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Cannot use any TEST_TARGET besides simpleremote or qemu
Hi Paulo, On Thursday, 2 August 2018 12:11:37 AM BST Paulo Neves wrote: > I am trying to create my own test controller. After copy pasting and > adapting the skeleton from BeagleBoneTarget to > $MYLAYER/lib/oeqa/controllers I set the appropriate TEST_TARGET. > > When I run the testimage task I always have errors like in [1]. The > actual keyword that is unexpected changes from invocation to > invocation. There was a similar error previously reported in this > mailing list but the presented solution was to actually stop using the > TEST_TARGET variable, which is not applicable. > > I have tried setting TEST_TARGET to any of the proposed options in the > manual [2] and the same unhelping error shows. I know that some of > that these controller targets, are not appropriate for my target > (odroid [cortex-arm15]), but the errors seems to not be related to any > functionality or these boards. > > I have analyzed and cannot really understand also how all the example > targets classes have a constructor which takes the d variable, while > the context.py call has some more variables including a dictionary, > kwargs, which does not actually have the d variable. I discovered this as well: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12842 This code got broken during the refactoring a while ago, and I think the person doing the work considered it to be legacy functionality and was going to remove it. Personally I'd like to see it stay and get fixed up, but the issue is having the time to fix it and maintain it - if you do by any chance then that would be great. At the moment I believe "simpleremote" does still work, it's just the actual classes that are broken, but I didn't give it extensive testing. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 1/1] import_layer.py: add -t option for layer_type
Hi Robert,
On Monday, 23 July 2018 12:30:01 PM CEST Robert Yang wrote:
> Now the logic is:
> Use options.layer_type if specified, guess if not, default to 'M'.
>
> Note choices=['A', 'B', 'S', 'D', 'M', ''], the '' is for default='', we can't
> use default='M' here, otherwise we don't know whether the 'M' is specified by
> user or not, we don't guess if it is specified by user, otherwise, guess.
>
> Signed-off-by: Robert Yang
> ---
> layerindex/tools/import_layer.py | 18 +-
> 1 file changed, 13 insertions(+), 5 deletions(-)
>
> diff --git a/layerindex/tools/import_layer.py
> b/layerindex/tools/import_layer.py
> index 2413cff..0d501f6 100755
> --- a/layerindex/tools/import_layer.py
> +++ b/layerindex/tools/import_layer.py
> @@ -189,6 +189,10 @@ def main():
> parser.add_option("-s", "--subdir",
> help = "Specify subdirectory",
> action="store", dest="subdir")
> +parser.add_option("-t", "--type",
> +help = "Specify layer type. A: Base, B: Machine(BSP), S:
> Software, D: Distribution, M: Miscellaneous",
> +choices=['A', 'B', 'S', 'D', 'M', ''],
> +action="store", dest="layer_type", default='')
If it's practical to do, could you make this part of the code use
LayerItem.LAYER_TYPE_CHOICES from models.py so that this will work
if that is extended in future? Of course that will mean initialising Django
earlier, I'm not sure if that will have any side-effects that we wouldn't want.
If you could give that a try though that would be great.
Thanks,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to force build of a blacklisted recipe?
On Thursday, 26 July 2018 11:16:02 AM CEST MOHAMMAD RASIM wrote: > I'm trying to build dvb-apps recipe which is blacklisted and was removed > from the newer openembedded releases, but it exist in the release I work > with > > Now when I try to build the recipe I get this error: > > >ERROR: > /home/oealliancebuilder/openpli-oe-core/meta-openembedded/meta-multimedia/ > recipes-dvb/dvb-apps/dvb-apps_1.1.1.bb > was skipped: Recipe is blacklisted: Fails to build with RSS > http://errors.yoctoproject.org/Errors/Details/130603/ - the recipe will > be removed on 2017-09-01 unless the issue is fixed > > But I want to tell openembedded to override the blacklisting and build > the recipe anyway so I can investigate and hopefully fix the recipe. How > can I do that ? You should be able to just set PNBLACKLIST[dvb-apps] = "" in a dvb-apps_%.bbappend for the recipe in your own layer, and then the blacklisting will be disabled for that recipe. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] can bitbake build offline ?
On Thursday, 26 July 2018 12:09:46 PM CEST MOHAMMAD RASIM wrote:
> Well, actually I used the find command in that wiki page to find the
> recipe that uses tag name to chose github revision and it found only one
> recipe
>
> >meta-openembedded/meta-initramfs/recipes-devtools/mtd/ubi-utils-
> >klibc_1.5.1.bb
Hmm, right, it seems like you're using a previous branch - this got fixed
in a later update of that recipe.
> I masked this recipe in the local.conf file and bitbake failed at
> parsing other recipes, looking at those recipes I saw that they don't
> use tag names but they use
>
> >SRCREV = "${AUTOREV}"
>
> which is logical since bitbake has to issue ls-remote to know what is
> the latest revision on the remote git repo (maybe the wiki page needs
> update to point this ?)
>
> Now I can't remove this AUTOREV in these recipes since I need bitbake to
> fetch the latest updates ( I push updates regularly on that repo and I
> don't want to update the recipe file each time to add the latest revision)
>
> So, here comes the hard part, Is it possible to tell bitbake to issue
> ls-remote if there is network and to use the latest fetched revision in
> the sources directory if the network is disabled ? that way I can use
> ${AUTOREV} in my recipes and bitbake will build from the local repo in
> the sources directory unless there is network to update the local repo,
> probably there is no such thing :) .
So there isn't a mechanism to do exactly that, no, however you can sort of
achieve the same thing if you set up a .inc file that sets the SRCREV values
for each recipe and sets BB_NO_NETWORK at the same time (though the
latter could still be separate). e.g. let's call it no_network.inc:
BB_NO_NETWORK = "1"
SRCREV_pn-abc = "b4c2bd84ee6f699e348d602a82d2d0963384cdea"
SRCREV_pn-xyz = "e3b30def2cd1c9ede7630489c3949a45b6eba6ee"
..
Then to build offline you would just add the following to your config:
require no_network.inc
FYI you can enable buildhistory and use the buildhistory-collect-srcrevs
script to generate all those SRCREV lines so you don't have to do that
by hand:
https://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#maintaining-build-output-quality
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] can bitbake build offline ?
Hi Mohammad, If it's failing during parsing that means you still have a recipe that does not have a proper SRCREV set, which you need to fix as described by the page that Alex linked. The error will be reporting which recipe that is. Cheers, Paul On Thursday, 26 July 2018 5:44:26 AM CEST MOHAMMAD RASIM wrote: > I tried that and bitbake still fails when disabling the network > > > On 07/21/2018 11:42 AM, Alexander Kanavin wrote: > > This page has a tip on what might be causing 'git ls-remote': > > > > https://wiki.yoctoproject.org/wiki/ How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F > > > > Alex > > > > 2018-07-21 10:32 GMT+02:00 MOHAMMAD RASIM : > >> Hi, is there a way I can run a bitbake build offline? I have all the > >> required sources of the target in the sources directory, and I have built > >> the target multiple times with internet connection, but when I build offline > >> it fails because it needs internet access during parsing recipes. > >> Can I tell bitbake to skip running `git ls-remote` during parsing recipes so > >> I can build offline? > >> > >> Thanks > >> -- > >> ___ > >> yocto mailing list > >> yocto@yoctoproject.org > >> https://lists.yoctoproject.org/listinfo/yocto > > -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] RFC: layer index docker fixes
On Monday, 16 July 2018 10:16:55 PM CEST Paul Eggleton wrote: > On Monday, 16 July 2018 8:24:53 PM CEST Konrad Scherer wrote: > > I recently noticed that if gcc is removed in the 'apt-get purge' step, > > the bitbake sanity checker will fail due to missing 'gcc' when doing a > > layer update. > > Hmm, well that's a different problem - those sanity checks shouldn't be > running. We don't need those tools since we're not building anything. I will > have a look into this. I was wrong - I remembered that we explicitly set HOSTTOOLS = "gcc" in our own conf/local.conf because there are other places in the metadata where we check for gcc which we do not override, and thus it's expected to be there. I've fixed and pushed the branch again, thanks for pointing out the issue (this time I actually tested running an update ;). Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] how to download a recipe
On Thursday, 12 July 2018 7:51:35 PM CEST Burton, Ross wrote: > On 12 July 2018 at 18:49, Simon Chamlian wrote: > > for grpc, there are 2 of them: > > > > http://layers.openembedded.org/layerindex/recipe/72419/ > > Layer meta-oe > > <http://layers.openembedded.org/layerindex/branch/master/layer/meta-oe/> > > (master branch) > > That's a bug in the tool, as meta-networking is a sublayer of meta-oe. If > you look at the paths, they're the same file. That is indeed a bug in the layer index update script - at some point in the past the recipe was moved and the move wasn't handled properly. I have a fix and will be sending it shortly. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] RFC: layer index docker fixes
On Monday, 16 July 2018 8:24:53 PM CEST Konrad Scherer wrote: > On 07/13/2018 03:29 AM, Paul Eggleton wrote: > > On Thursday, 12 July 2018 10:16:12 PM CEST you wrote: > >> I just noticed that the image doesn't contain git. Was this > >> deliberate or an oversight? > > > > Ah, no, it wasn't deliberate - perhaps it was there earlier with > > buildpack-deps? It doesn't appear that we ever explicitly had it in there. > > I've added it and pushed the branch again. > > I recently noticed that if gcc is removed in the 'apt-get purge' step, > the bitbake sanity checker will fail due to missing 'gcc' when doing a > layer update. Hmm, well that's a different problem - those sanity checks shouldn't be running. We don't need those tools since we're not building anything. I will have a look into this. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] RFC: layer index docker fixes
On Thursday, 12 July 2018 10:16:12 PM CEST you wrote: > I just noticed that the image doesn't contain git. Was this > deliberate or an oversight? Ah, no, it wasn't deliberate - perhaps it was there earlier with buildpack- deps? It doesn't appear that we ever explicitly had it in there. I've added it and pushed the branch again. > >> I recommend docker-compose because much of the information in > >> docker/README can be captured in a compose yaml file. It handles the > >> network setup and makes managing shared volumes much simpler. > > > > Right, we'd just need to make sure everything we now have in the README is > > covered by the compose file (everything that's practical to cover > > , anyway). > > I am heading out on vacation soon. I will rework my docker-compose > configuration with the new image and make some documentation when I get > back. OK great, thanks. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web] RFC: layer index docker fixes
Hi Konrad On Wednesday, 11 July 2018 7:03:28 PM CEST Konrad Scherer wrote: > I have attached a patch to reduce the layerindex-web image size further. > Docker image layers are like git objects and RUN commands can only hide > files, not remove them once they have been committed. The pip upgrade > caused the following 'pip install' to fail and was removed. Right, overall you don't save any storage space with separate stages - but the downside of merging these is that you lose the ability to incrementally update the container within those stages; any time anything before changes the whole block will need to be re-run. Having said that we still have the separation between requirements.txt and everything else which is probably sufficient. Oddly I did not see a failure from pip when I built this yesterday, but I guess it's probably not strictly necessary to upgrade it anyway. I've pushed your patch to the paule/dockerfixes branch. > I recommend docker-compose because much of the information in > docker/README can be captured in a compose yaml file. It handles the > network setup and makes managing shared volumes much simpler. Right, we'd just need to make sure everything we now have in the README is covered by the compose file (everything that's practical to cover, anyway). Cheers, Pau; -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web] RFC: layer index docker fixes
Hi folks I've been trying to get the OE layer index example docker setup in shape, based upon improvements from myself, Michael and Konrad. For the moment I've taken most of what Michael and I did (with some fixes and tweaks, and broken up into smaller patches) and a few pieces of the Dockerfile from Konrad's recent patch and put it here: http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/dockerfixes Konrad, I know you were a little bit more ambitious with your changes but I wanted to get something closer to what Michael is using and then we can build upon it. In particular I haven't yet explored docker-compose as you have, although it looks like it would make deploying all of this a bit easier. Let me know what you think. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 1/6] Add docker-compose file to create full layerindex stack of MariaDB, RabbitMQ and Nginx
Hi Konrad On Tuesday, 26 June 2018 7:41:30 PM CEST Konrad Scherer wrote: > Lots of new features added: > > - Layerindex runs as unprivileged user inside container > > - Celery worker is started before gunicorn > > - Entrypoint script supports changing RabbitMQ location > > - Entrypoint script support initialization of database and superuser > > - Reverse Proxy uses https with self signed certs by default and > supports Let's Encrypt certs (not enabled by default) > > - Move docker image to debian stretch and python3 > > - Remove build tools after installation to reduce the image > to under 500MB in size This is quite nice, thanks! Coincidentally, both myself and Michael Halstead have had a go at cleaning up and improving the Docker setup, so I have to do a little reconciliation between your and his changes - see here for his: https://github.com/halstead/layerindex-web/commit/b9791710ff97550fa9110ab89a70c42b1fc86581 I think we probably want to break all of this down into a set of discrete commits rather than one big one, and then it'll be a bit clearer. I'll do a first pass and come back to you both. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Keeping and modifying Source code in Yocto
On Tuesday, 10 July 2018 3:32:56 AM CEST ChenQi wrote: > I usually use the following steps when I need to modify source for some > purpose. > > 1. bitbake A (or at least bitbake A -c configure) > *2. cd tmp/work/x/A/** > **3. make the modification* > 4. bitbake A -c compile -f > 5. bitbake A > > If you find the modification useful, make a patch from it. This is the old way of doing it which can under some circumstances result in you losing your changes to the source tree, since the sources under tmp/work/ are only ever intended to be there temporarily. Devtool (as recommend by others on this thread) is designed to enable this workflow in a much safer manner. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout
On Monday, 9 July 2018 10:45:58 AM CEST Robert Yang wrote: > On 07/09/2018 04:36 PM, Paul Eggleton wrote: > > On Monday, 9 July 2018 10:34:07 AM CEST Paul Eggleton wrote: > >> On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote: > >>> Fixed: > >>> Assume there is no master branch in hello layer: > >>> $ update.py -l hello -b master > >>> INFO: Skipping update of layer hello - branch master doesn't exist > >>> > >>> This is correct since hello layer doesn't have master branch, but when > >>> --nocheckout: > >>> $ update.py -l hello -b master --nocheckout > >>> [snip] > >>> INFO: Sorting layers for branch mater: > >>> WARNING: Cannot find required collections on branch master: > >>> WARNING: hello: LAYERDEPENDS: > >>> > >>> This is incorrect, this patch fixed the problem, now it skips it since the > >>> branch doesn't exists when --nocheckout. > >> > >> This fix doesn't seem right. The intention of --nocheckout is to not check > >> anything > >> out, i.e. use whatever HEAD is in the current repository. It wouldn't be > >> correct > >> to get the commit from the origin in this case. > > > > Actually, on a second reading of the change. It's not obvious why this is > > needed but I suppose the comment clarifies it. > > Do I need update commit message, please ? No, it's OK, I'll merge as-is. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout
On Monday, 9 July 2018 10:34:07 AM CEST Paul Eggleton wrote: > Hi Robert, > > On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote: > > Fixed: > > Assume there is no master branch in hello layer: > > $ update.py -l hello -b master > > INFO: Skipping update of layer hello - branch master doesn't exist > > > > This is correct since hello layer doesn't have master branch, but when > > --nocheckout: > > $ update.py -l hello -b master --nocheckout > > [snip] > > INFO: Sorting layers for branch mater: > > WARNING: Cannot find required collections on branch master: > > WARNING: hello: LAYERDEPENDS: > > > > This is incorrect, this patch fixed the problem, now it skips it since the > > branch doesn't exists when --nocheckout. > > This fix doesn't seem right. The intention of --nocheckout is to not check > anything > out, i.e. use whatever HEAD is in the current repository. It wouldn't be > correct > to get the commit from the origin in this case. Actually, on a second reading of the change. It's not obvious why this is needed but I suppose the comment clarifies it. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout
Hi Robert, On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote: > Fixed: > Assume there is no master branch in hello layer: > $ update.py -l hello -b master > INFO: Skipping update of layer hello - branch master doesn't exist > > This is correct since hello layer doesn't have master branch, but when > --nocheckout: > $ update.py -l hello -b master --nocheckout > [snip] > INFO: Sorting layers for branch mater: > WARNING: Cannot find required collections on branch master: > WARNING: hello: LAYERDEPENDS: > > This is incorrect, this patch fixed the problem, now it skips it since the > branch doesn't exists when --nocheckout. This fix doesn't seem right. The intention of --nocheckout is to not check anything out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct to get the commit from the origin in this case. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
Hi Robert On Wednesday, 4 July 2018 7:52:05 PM NZST you wrote: > I'm sorry to say that I met layerindex' loaddata problems yesterday and > today, > I still didn't find the root cause. Have you tried dumpdata and loaddata > recently, please ? > > What I did was: > > $ python3 manage.py dumpdata --settings settings --exclude=contenttypes > --exclude=auth.Permission --exclude=corsheaders >dumped.json > > On another environment: > Setup database to sqlite3 in settings.py. > $ python3 manage.py loaddata --settings settings dumped.json > > The first problem I got was: > [snip] >File > "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/revisions.py", > > line 410, in _assert_registered > model=model, > reversion.errors.RegistrationError: Problem installing fixture > '/buildarea1/lyang1/layerindex-web/dumped.json': 'layerindex.models.Distro'> has not been registered with django-reversion > [snip] > > I think it is because we didn't use @reversion.register() for the class, so I > added them to layerindex/models.py, then I got other errors: > > [snip] >File > "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/models.py", > > line 272, in _local_field_dict > field_dict[field.attname] = getattr(obj, field.attname) > AttributeError: Problem installing fixture > '/buildarea1/lyang1/layerindex-web/dumped.json': 'Branch' object has no > attribute 'layerbranch_id' Hmm, that's odd. Branch shouldn't have layerbranch_id, it's the other way around - LayerBranch has a branch_id. > I'm not sure what's wrong atm, need more investigations. > > I need loaddata on my localhost to do development testing, so I can't start > work on update.py until I fix the loaddata problem. I have used loaddata and dumpdata here (a couple of times) but not recently. I did not experience these issues before though. However these don't seem like issues that would have started as a result of this patchset (or indeed recent changes, other than perhaps an upgrade of django-reversion), have you been using loaddata/dumpdata prior to this? Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
Hi Robert On Tuesday, 3 July 2018 2:45:11 PM NZST Robert Yang wrote: > Thanks for let me know this, this patch might be incorrect, suppose we have > two > layers: core and hello: > > 1) LAYERRECOMMENDS_core = "hello" > 2) $ update.py -l hello,core > > Then core maybe added before hello layer since it ignores recs on hello, and > if > hello is a new layer, it would not be in core's recs in database since core > knows nothing about hello, I think that this is incorrect. > > If we really need this, I think that we should not ignore recs when the > layer is present, but only ignore it when the layer is not present, for > example, ignore it when hello layer doesn't exist, otherwise, don't ignore it. Can you come up with an alternative fix that doesn't break parsing like it does now? > But I'm not sure about patch 4 (error -> warning) either, since layerindex is > a central database, whether add recs to conf/bblayers.conf should depend on > end user rather than ignore it in database, otherwise the end user (especially > the api user) would have no way to choice, for example, we use api to make > conf/bblayers.conf have all or no recs layers according to user's choice, > if the database is wrong, then there might be only part of recs layers. > Though we can check update.py's warnings to fix the problem. Recommends are just that - recommendations. Someone might legitimately submit a layer that recommends another which they don't submit (perhaps a commercial layer?). The system shouldn't refuse to handle it or indicate that it's broken (it isn't, there might be reduced functionality but the layer will still be parseable by bitbake). I'll leave these changes unmerged for a bit in case you have a better fix for the current problems. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 7/7] Add site-wide notice support
Add the ability to show a notice at the top of every page; this provides
the ability for admins to display a message to visitors in the case of
infrastructure or index data issues. Notices can have an expiry date and
can be disabled and re-enabled if needed. A subset of HTML can be used
for formatting the text, URLs will be made into clickable links, and
four "levels" are supported (info, success, warning and error).
Signed-off-by: Paul Eggleton
---
layerindex/admin.py | 1 +
layerindex/context_processors.py | 7 +--
layerindex/migrations/0014_sitenotice.py | 24 +++
layerindex/models.py | 25 +++-
layerindex/utils.py | 14 +
requirements.txt | 1 +
templates/base.html | 5 +
7 files changed, 74 insertions(+), 3 deletions(-)
create mode 100644 layerindex/migrations/0014_sitenotice.py
diff --git a/layerindex/admin.py b/layerindex/admin.py
index 3cb59691..312f7a30 100644
--- a/layerindex/admin.py
+++ b/layerindex/admin.py
@@ -198,3 +198,4 @@ admin.site.register(Patch)
admin.site.register(RecipeChangeset, RecipeChangesetAdmin)
admin.site.register(ClassicRecipe, ClassicRecipeAdmin)
admin.site.register(PythonEnvironment)
+admin.site.register(SiteNotice)
diff --git a/layerindex/context_processors.py b/layerindex/context_processors.py
index db8e3fa3..7cf20ede 100644
--- a/layerindex/context_processors.py
+++ b/layerindex/context_processors.py
@@ -1,11 +1,13 @@
# layerindex-web - custom context processor
#
-# Copyright (C) 2013 Intel Corporation
+# Copyright (C) 2013, 2018 Intel Corporation
#
# Licensed under the MIT license, see COPYING.MIT for details
-from layerindex.models import Branch, LayerItem
+from layerindex.models import Branch, LayerItem, SiteNotice
from django.contrib.sites.models import Site
+from django.db.models import Q
+from datetime import datetime
def layerindex_context(request):
import settings
@@ -20,4 +22,5 @@ def layerindex_context(request):
'oe_classic': Branch.objects.filter(name='oe-classic'),
'site_name': site_name,
'rrs_enabled': 'rrs' in settings.INSTALLED_APPS,
+'notices':
SiteNotice.objects.filter(disabled=False).filter(Q(expires__isnull=True) |
Q(expires__gte=datetime.now())),
}
diff --git a/layerindex/migrations/0014_sitenotice.py
b/layerindex/migrations/0014_sitenotice.py
new file mode 100644
index ..630700cf
--- /dev/null
+++ b/layerindex/migrations/0014_sitenotice.py
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+('layerindex', '0013_patch'),
+]
+
+operations = [
+migrations.CreateModel(
+name='SiteNotice',
+fields=[
+('id', models.AutoField(auto_created=True, serialize=False,
verbose_name='ID', primary_key=True)),
+('text', models.TextField(help_text='Text to show in the
notice. A limited subset of HTML is supported for formatting.')),
+('level', models.CharField(choices=[('I', 'Info'), ('S',
'Success'), ('W', 'Warning'), ('E', 'Error')], help_text='Level of notice to
display', default='I', max_length=1)),
+('disabled', models.BooleanField(verbose_name='Disabled',
help_text='Use to temporarily disable this notice', default=False)),
+('expires', models.DateTimeField(blank=True,
help_text='Optional date/time when this notice will stop showing', null=True)),
+],
+),
+]
diff --git a/layerindex/models.py b/layerindex/models.py
index ff164baa..891f5dfb 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -1,6 +1,6 @@
# layerindex-web - model definitions
#
-# Copyright (C) 2013-2016 Intel Corporation
+# Copyright (C) 2013-2018 Intel Corporation
#
# Licensed under the MIT license, see COPYING.MIT for details
@@ -658,3 +658,26 @@ class RecipeChange(models.Model):
def reset_fields(self):
for fieldname in self.RECIPE_VARIABLE_MAP:
setattr(self, fieldname, getattr(self.recipe, fieldname))
+
+class SiteNotice(models.Model):
+NOTICE_LEVEL_CHOICES = [
+('I', 'Info'),
+('S', 'Success'),
+('W', 'Warning'),
+('E', 'Error'),
+]
+text = models.TextField(help_text='Text to show in the notice. A limited
subset of HTML is supported for formatting.')
+level = models.CharField(max_length=1, choices=NOTICE_LEVEL_CHOICES,
default='I', help_text='Level of notice to display')
+disabled = models.BooleanField('Disabled', default=False, help_text='Use
to temporarily disable this notice')
+expires = models.DateTimeField(blank=True, null=True, help_text='Optional
date/time when this notice will stop showing')
+
+def __str__(self):
+
