Re: [yocto] [[PATCH][error-report-web] 5/8] Post/models.py: Build model add support for Error type.

[Michael Wood]

On 14/06/16 00:32, Aníbal Limón wrote:

In order to support other errors not only Recipe ones adds
a ERROR_TYPE field to the Build model defaults to "Recipe".

Add a class for store BuildErrorType currently supported
Recipe, Core, Bitbake selftest and OE selftest.

[YOCTO #7583]

Signed-off-by: Aníbal Limón 
---
  Post/migrations/0005_build_error_type.py | 19 +++
  Post/models.py   |  7 +++
  2 files changed, 26 insertions(+)
  create mode 100644 Post/migrations/0005_build_error_type.py

diff --git a/Post/migrations/0005_build_error_type.py 
b/Post/migrations/0005_build_error_type.py
new file mode 100644
index 000..96cdf8c
--- /dev/null
+++ b/Post/migrations/0005_build_error_type.py
@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+dependencies = [
+('Post', '0004_auto_20160530_1126'),
+]
+
+operations = [
+migrations.AddField(
+model_name='build',
+name='ERROR_TYPE',
+field=models.CharField(default=b'Recipe', max_length=64),
+),
+]
diff --git a/Post/models.py b/Post/models.py
index 84f8abf..f8d9916 100644
--- a/Post/models.py
+++ b/Post/models.py
@@ -11,6 +11,12 @@ from datetime import datetime
  
  import Levenshtein
  
+class BuildErrorType(object):

+RECIPE = "Recipe"
+BITBAKE_CORE = "Core"
+BITBAKE_SELFTEST = "Bitbake selftest"
+OE_SELFTEST = "OE selftest"
+
  # Create your models here.
  class Build(models.Model):
  DATE = models.DateTimeField('Submit date', blank=True, null=True)
@@ -25,6 +31,7 @@ class Build(models.Model):
  NAME = models.CharField(max_length=50)
  EMAIL = models.CharField(max_length=50)
  LINK_BACK = models.TextField(max_length=300, blank=True, null=True)
+ERROR_TYPE = models.CharField(max_length=64, default=BuildErrorType.RECIPE)
  
  class BuildFailure(models.Model):

  TASK = models.CharField(max_length=1024)


Thanks for the patches.

Ideally we wouldn't use a char field here as if the type string ever 
changed the database could end up with multiple versions of the type 
strings depending on when the type was saved, it would be possible to 
handle that with migrations but it would be pretty messy. It also allows 
any arbitrary chars which we probably don't want if it's something we're 
going to be filtering on.  Here is an example from Toaster on how it can 
be done with the choices option and an enum style.


|OUTCOME_NA = -1 OUTCOME_SUCCESS = 0 OUTCOME_COVERED = 1 OUTCOME_CACHED = 
2 OUTCOME_PREBUILT = 3 OUTCOME_FAILED = 4 OUTCOME_EMPTY = 5 TASK_OUTCOME 
= ( (OUTCOME_NA, 'Not Available'), (OUTCOME_SUCCESS, 'Succeeded'), 
(OUTCOME_COVERED, 'Covered'), (OUTCOME_CACHED, 'Cached'), 
(OUTCOME_PREBUILT, 'Prebuilt'), (OUTCOME_FAILED, 'Failed'), 
(OUTCOME_EMPTY, 'Empty'), ) ||outcome = models.IntegerField(choices=TASK_OUTCOME, default=OUTCOME_NA)|




Michael

--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] [yocto-autobuilder][PATCH] buildbot/templates: Fix link to error reporting tool

[Michael Wood]

Fix link so that we don't perform a full text search for the
SHA/revision and just filter for it using the commit field. This should
drastically reduce page loading time on the error reporting site.

[YOCTO #9295]

Signed-off-by: Michael Wood <michael.g.w...@intel.com>
---
 .../buildbot/status/web/templates/grid_macros.html  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/web/templates/grid_macros.html
 
b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/web/templates/grid_macros.html
index 0479927..8876422 100644
--- 
a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/web/templates/grid_macros.html
+++ 
b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/web/templates/grid_macros.html
@@ -26,7 +26,7 @@
   {%- else %}Commit: latest{% endif %}
   {%- if ss.branch %} in {{ ss.branch|e }}{% endif %}
   {%- if ss.hasPatch %} [patch]{% endif -%}
-  (http://errors.yoctoproject.org/Errors/Search/Args/?items=10={{ss.revision}};>errors)
+  (http://errors.yoctoproject.org/Errors/Latest/Autobuilder/?filter={{ss.revision}}=commit;>errors)
 
 {%- endfor %}
   
-- 
2.5.0

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [error-report-web][PATCH] parser: Add Unicode support to the error log

[Michael Wood]

On 30/09/15 14:54, mariano.lo...@linux.intel.com wrote:

From: Mariano Lopez 

If the server receives a error log with Unicode
character in it, it will throw and internal server
error, this is caused because the server doesn't
try to convert everything to ASCII

This patch changes the log encoding to UTF-8 so
it will allow Unicode characters.

[YOCTO #8225]

Signed-off-by: Mariano Lopez 
---
  Post/parser.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Post/parser.py b/Post/parser.py
index d33a968..599afde 100644
--- a/Post/parser.py
+++ b/Post/parser.py
@@ -85,7 +85,7 @@ class Parser:
  recipe = package
  recipe_version = "unknown"
  
-f = BuildFailure(TASK = str(fail['task']), RECIPE = recipe, RECIPE_VERSION = recipe_version, ERROR_DETAILS = str(fail['log']), BUILD = b)

+f = BuildFailure(TASK = str(fail['task']), RECIPE = recipe, 
RECIPE_VERSION = recipe_version, ERROR_DETAILS = fail['log'].encode('utf-8'), 
BUILD = b)
  
  f.save()
  


Thanks for the patch. Now committed.

Michael



--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] [RFC] error-report-web error matching

[Michael Wood]

I've recently added a feature which was requested for the 
error-report-web project[1] to add some kind of error matching 
mechanism, this is now live on http://errors.yoctoproject.org


I'd be interested to hear If anyone has feedback regarding the similar 
errors matching, whether it's useful or has too many false positives etc.


The matching threshold value can be tweaked, I've currently just set it 
to a threshold that seemed OK to me, but would be good if those with 
more experienced eyes could have a glance over.


Thanks,

Michael

[1] http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] [error-report-web] New errors-report-web

[Michael Wood]

Hi,

I've updated the errors.yoctoproject.org site with the latest version of 
error-report-web [1] that myself and Belén have been working on.


error-report-web is a project to provide a web service that can be used 
to collect and display build error reports. Error report generation can 
be enabled in your build's local.conf [2]


Here are some of the highlights for the latest version:

- Option to have a JSON response on error report submission
- Optional Special submitter allowing an extra tab in the UI for 
quick access to a defined submitter (e.g autobuilder)

- Filtering and ordering on columns
- Searching improved to allow search of the contents of error 
reports and other useful fields

- Improved page loading speed
- Non blocking graph generation
- Unit tests
- Update to Django 1.7
- Add support for Django migrations
- View errors by build
- Many UI tweaks
- Better error handling
- Simpler url structure

Thanks,

Michael

[1] http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/
[2] 
http://www.yoctoproject.org/docs/1.7.1/dev-manual/dev-manual.html#using-the-error-reporting-tool



--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] [error-report-web][PATCH] parser: Check for tag markup in the metadata fields

[Michael Wood]

Before we commit the error report metadata to the database do a
rudimentary check on all fields that are passed to the graphs page to
avoid any XSS happening.

Signed-off-by: Michael Wood michael.g.w...@intel.com
---
 Post/parser.py | 13 +
 1 file changed, 13 insertions(+)

diff --git a/Post/parser.py b/Post/parser.py
index fae9194..b180165 100644
--- a/Post/parser.py
+++ b/Post/parser.py
@@ -18,8 +18,21 @@ class Parser:
 def __init__(self, data):
 self.data = data
 
+# returns true if the values contain '' char
+# Ignore the failures field (which is an array anyway)
+def contains_tags (self, data):
+for key,val in data.items():
+if key == 'failures':
+continue
+
+if '' in val:
+return True
+return False
+
 def parse(self):
 jsondata = json.loads(self.data)
+if self.contains_tags(jsondata) == True:
+return
 
 MACHINE_NAME = str(jsondata['machine'])
 NATIVELSBSTRING = str(jsondata['nativelsb'])
-- 
1.9.1

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto