Hi Shakeel,
1. Since all of the offending objects are Target_File , you could change the
log message so show the object’s actual value by adding “.path”.
logger.info(" FOUND:'%s'" % (obj.path))
That will show us to look inside of the Target_File objects and observe the
actual multiple entries.
Since we are seeing a lot of upper/lower case similar files, I wonder if it is
a case problem:
(0-2-/usr/share/terminfo/a-18)({'target': <Target: Target object>, 'path':
'/usr/share/terminfo/a', 'inodetype': 2})
(0-2-/usr/share/terminfo/A-18)({'target': <Target: Target object>, 'path':
'/usr/share/terminfo/A', 'inodetype': 2})
The only cost of these errors should be perhaps missing entries in the target
file tree.
2. The second error set looks a lot nastier.
_mysql_exceptions.OperationalError: (1267, "Illegal mix of collations
(latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='")
Where did the Swedish text come from?
Do you perhaps have files with Unicode in their names?
I am wondering if we should have Toaster normalize all file paths/names for
TargetFile objects to UTF8 or something. That is certainly doable.
- David
From: Muhammad Shakeel Khan [mailto:[email protected]]
Sent: Monday, August 27, 2018 6:17 AM
To: Reyna, David
Cc: [email protected]
Subject: Re: [Toaster] Toaster Error - Rocko
Hi David,
Thanks for looking into this.
OK, I will ignore 1) as you said that the messages are largely passive.
For 2) I have added following slightly modified code to _cached_get
def _cached_get(self, clazz, **kwargs):
""" This is a memory-cached get. We assume that the objects will not
change in the database between gets.
"""
assert issubclass(clazz, models.Model), "_cached_get needs to get the
class as first argument"
key = ORMWrapper._build_key(**kwargs)
dictname = "objects_%s" % clazz.__name__
if not dictname in vars(self).keys():
vars(self)[dictname] = {}
if not key in vars(self)[dictname].keys():
try:
vars(self)[dictname][key] = clazz.objects.get(**kwargs)
except (ObjectDoesNotExist, MultipleObjectsReturned) as e:
logger.info<http://logger.info/>("ERROR:buildinfohelper:_cached_get: '%s'
(%s)(%s)" % (e, key, kwargs))
for obj in clazz.objects.filter(**kwargs):
logger.info<http://logger.info/>(" FOUND:'%s'" % (obj))
vars(self)[dictname][key] =
clazz.objects.filter(**kwargs).first()
return vars(self)[dictname][key]
I see a lot of queries returning "get() returned more than one Target_File --
it returned 2". Please see the log attached. Unfortunately there was another
error and bitbake still returned error code.
I am interested to know what could be the reason for this error? Can it happen
if a user starts a new build without first getting completed using same Toaster
databases? How can I reset the database entries? Do I need to completely remove
mysql database or there is something I can do within current build directory?
Best Regards,
Shakeel
On Mon, Aug 27, 2018 at 2:14 PM Reyna, David
<[email protected]<mailto:[email protected]>> wrote:
Hi Shakeel,
1) “WARNING: Could not add dependency to the package libc6 because
ld-linux.so.2 is an unknown package”
Yes, I have seen those messages. I have not debugged them deeply since the
errors are largely passive, but it have been on my list of things to look at
because there may be some details in the build data that we could be missing
out.
2) From the looks of the trace, it appears that a found target file that has
_two_ registered parents, at least using the cache/key mechanism of
‘_cached_get’.
I have not myself seen this error.
Since you have a reproducer, here is some sample code that you could in place
of the existing “_cached_get” in “bitbake/lib/bb/ui/buildinfohelper.py”. It
will:
(a) catch the exception,
(b) return the first found value so that the build can continue, and
(c) send to the log the value(s) that broke it.
If you could insert this code (once it compiles), then we will have a better
idea what happened.
def _cached_get(self, clazz, **kwargs):
""" This is a memory-cached get. We assume that the objects will not
change in the database between gets.
"""
assert issubclass(clazz, models.Model), "_cached_get needs to get the
class as first argument"
key = ORMWrapper._build_key(**kwargs)
dictname = "objects_%s" % clazz.__name__
if not dictname in vars(self).keys():
vars(self)[dictname] = {}
if not key in vars(self)[dictname].keys():
try:
vars(self)[dictname][key] = clazz.objects.get(**kwargs)
except KeyError as e:
logger.debug(1, "ERROR:buildinfohelper:_cached_get: '%s'
(%s)(%s)" % (e,key,**kwargs))
for obj in clazz.objects.filter(**kwargs):
logger.debug(1, " FOUND:'%s'" % (obj))
vars(self)[dictname][key] = clazz.objects.filter(**kwargs)[0]
return vars(self)[dictname][key]
BTW: this is the call in the trace that led to the error:
parent_obj = self._cached_get(Target_File, target = target_obj,
path = parent_path, inodetype = Target_File.ITYPE_DIRECTORY)
The second thing we could try is to directly look in the database and scan for
duplicates for the above values. Any duplicates would cause the observed crash.
Thanks,
David
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Muhammad Shakeel Khan
Sent: Monday, August 27, 2018 1:18 AM
To: [email protected]<mailto:[email protected]>
Subject: [Toaster] Toaster Error - Rocko
Hi,
We are running the Toaster production instance based on Rocko (2.4.3) and
seeing a lot of warnings when the build is completed. Our setup is:
1) Start Toaster production instance in isolation (toaster user) with nobuild
option, just to monitor command line builds
2) A different user then fetches 'poky' separately and then sets up the build
environment
3) Start toaster: $ source toaster start noweb
4) Run build: $ bitbake core-image-minimal
5) Stop toaster: $ source toaster stop
This result in following warnings/notes when the build is completed (complete
log attached)
WARNING: Could not add dependency to the package libc6 because ld-linux.so.2 is
an unknown package
NOTE: We did not find one recipe for theconfiguration data package libc6 Recipe
matching query does not exist.
WARNING: Could not add dependency to the package libc6 because ld-linux.so.2 is
an unknown package
What might be going wrong here? The Toaster web UI shows 100% progress.
Also in one other custom image build we see the following error at the end of
the build:
ERROR: get() returned more than one Target_File -- it returned 2!
Traceback (most recent call last):
File
"/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/toasterui.py",
line 416, in main
buildinfohelper.store_target_package_data(event)
File
"/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/buildinfohelper.py",
line 1383, in store_target_package_data
self.orm_wrapper.save_target_file_information(self.internal_state['build'],
target, filedata)
File
"/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/buildinfohelper.py",
line 536, in save_target_file_information
parent_obj = self._cached_get(Target_File, target = target_obj, path =
parent_path, inodetype = Target_File.ITYPE_DIRECTORY)
File
"/home/mshakeel/build-areas/build-area-rocko/sources/poky/bitbake/lib/bb/ui/buildinfohelper.py",
line 123, in _cached_get
vars(self)[dictname][key] = clazz.objects.get(**kwargs)
File
"/home/mshakeel/.local/lib/python3.6/site-packages/django/db/models/manager.py",
line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"/home/mshakeel/.local/lib/python3.6/site-packages/django/db/models/query.py",
line 384, in get
(self.model._meta.object_name, num)
orm.models.MultipleObjectsReturned: get() returned more than one Target_File --
it returned 2!
NOTE: No commit since BUILDHISTORY_COMMIT != '1'
NOTE: Build exited with errorcode 0
NOTE: ToasterUI build done, brbe: None
WARNING: Return value is 1
Any thoughts about this error? We are using mysql (django.db.backends.mysql)
for Toaster database.
Best Regards,
Shakeel
--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster
