Re: [yocto] [layerindex-web][PATCH 4/5] update_layer.py: use it as a module
Hi Paul, Thank you very much, and please see my comments inline. On 02/05/2018 07:27 AM, Paul Eggleton wrote: Hi Robert, On Wednesday, 3 January 2018 6:42:25 PM NZDT Robert Yang wrote: It had been split to a separate tool and use subprocess to run it to avoid tinfoil issues (e.g., can't be shutdown correctly in old version), but the utils.setup_django() cost a lot of time which made it a little slow, use mulitprocessing.Process() to run it as function can avoid both issues (tinfoil and slow issues). And remove bitbake modules when swith branch since it is may not be compatible between branches. This can save about 5 minutes in my testing when "update.py -b ", with parallel fetch and this, we can reduce the time from more than 9mins to less than 3 mins. Signed-off-by: Robert Yang --- layerindex/update.py | 115 +++-- layerindex/update_layer.py | 103 +--- 2 files changed, 70 insertions(+), 148 deletions(-) diff --git a/layerindex/update.py b/layerindex/update.py index d6488f0..e13c9ab 100755 --- a/layerindex/update.py +++ b/layerindex/update.py @@ -21,6 +21,9 @@ import utils import operator import re import multiprocessing +import update_layer +import copy +import imp import warnings warnings.filterwarnings("ignore", category=DeprecationWarning) @@ -34,61 +37,15 @@ except ImportError: logger.error("Please install PythonGit 0.3.1 or later in order to use this script") sys.exit(1) - -def reenable_sigint(): -signal.signal(signal.SIGINT, signal.SIG_DFL) - -def run_command_interruptible(cmd): -""" -Run a command with output displayed on the console, but ensure any Ctrl+C is -processed only by the child process. -""" -signal.signal(signal.SIGINT, signal.SIG_IGN) +def call_update_layer(options, queue): +ret = 1 +output = "" try: -process = subprocess.Popen( -cmd, cwd=os.path.dirname(sys.argv[0]), shell=True, preexec_fn=reenable_sigint, stdout=subprocess.PIPE, stderr=subprocess.STDOUT -) - -reader = codecs.getreader('utf-8')(process.stdout, errors='surrogateescape') -buf = '' -while True: -out = reader.read(1, 1) -if out: -sys.stdout.write(out) -sys.stdout.flush() -buf += out -elif out == '' and process.poll() != None: -break - +ret, output = update_layer.main(options, queue) +except Exception: +raise finally: -signal.signal(signal.SIGINT, signal.SIG_DFL) -return process.returncode, buf - - -def prepare_update_layer_command(options, branch, layer, initial=False): -"""Prepare the update_layer.py command line""" -if branch.update_environment: -cmdprefix = branch.update_environment.get_command() -else: -cmdprefix = 'python3' -cmd = '%s update_layer.py -l %s -b %s' % (cmdprefix, layer.name, branch.name) -if options.reload: -cmd += ' --reload' -if options.fullreload: -cmd += ' --fullreload' -if options.nocheckout: -cmd += ' --nocheckout' -if options.dryrun: -cmd += ' -n' -if initial: -cmd += ' -i' -if options.loglevel == logging.DEBUG: -cmd += ' -d' -elif options.loglevel == logging.ERROR: -cmd += ' -q' -if options.keep_temp: -cmd += ' --keep-temp' -return cmd +queue.put([ret, output]) def update_actual_branch(layerquery, fetchdir, branch, options, update_bitbake, bitbakepath): """Update actual branch for layers and bitbake in database""" @@ -266,6 +223,7 @@ def main(): try: lockfn = os.path.join(fetchdir, "layerindex.lock") lockfile = utils.lock_file(lockfn) +update_log_path = update_layer.update_log_path if not lockfile: logger.error("Layer index lock timeout expired") sys.exit(1) @@ -315,8 +273,15 @@ def main(): # We now do this by calling out to a separate script; doing otherwise turned out to be # unreliable due to leaking memory (we're using bitbake internals in a manner in which # they never get used during normal operation). +update_layer_options = copy.copy(options) last_rev = {} for branch in branches: +# The bitbake modules are not compatible between branches, so +# remove them, they will be imported again when needed. +for name, module in sys.modules.copy().items(): +if '__file__' in module.__dict__ and module.__file__.startswith(bitbakepath): +logger.debug('Removing module %s' % name) +del sys.modules[name] This part worried me, so I did some quick searching and came across this: http://justus.science/blog/2015/04/19/sys.modules-is
Re: [yocto] [layerindex-web] Updating layers?
Hi Steve On Friday, 12 January 2018 4:52:23 AM NZDT Steve Bedford wrote: > Is the layerindex not running the update script? I see end of September as > last updates, and a bunch of branches in our layer (meta-timesys) were > never picked up. FYI we've fixed this as of a few hours after your email - apologies for the downtime, we should now have things in place to avoid that kind of outage. Let me know if you see any other anomalies. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 0/5] update.py: performance improve and 2 fixes
Hi Robert On Thursday, 1 February 2018 5:59:42 PM NZDT Robert Yang wrote: > On 01/03/2018 01:42 PM, Robert Yang wrote: > > This email contains two parts: > > - The first 2 are bug fixes > > - The other 3 are performance improvements, I have 124 layers, > >the "update.py -b " needs about 9m20s to finish the running > >when everything is > >update, and we have several branches, we need them to run in > >periodically > >(e.g., per 30 mins, the short the better), so one branch 9m20s is > >really a problem. > >Only about 1m43s are needed with these improvements when set > >PARALLEL_JOBS to 10: > >update.py: fetch repos parallelly # Saved 2 mins > >update_layer.py: use it as a module # Saved 5 mins > >update_layer.py: only call init_parser when needed # Saved 1 min > > Any comments, please ? My apologies for the delay - I had concerns about the update_layer.py module stuff but hadn't made the time to review it properly until now. Aside from that the other patches are fine so I've merged those (with some minor tweaks to the commit messages). Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 4/5] update_layer.py: use it as a module
Hi Robert, On Wednesday, 3 January 2018 6:42:25 PM NZDT Robert Yang wrote: > It had been split to a separate tool and use subprocess to run it to avoid > tinfoil issues (e.g., can't be shutdown correctly in old version), but the > utils.setup_django() cost a lot of time which made it a little slow, use > mulitprocessing.Process() to run it as function can avoid both issues (tinfoil > and slow issues). > > And remove bitbake modules when swith branch since it is may not be compatible > between branches. > > This can save about 5 minutes in my testing when "update.py -b ", > with parallel fetch and this, we can reduce the time from more than 9mins to > less than 3 mins. > > Signed-off-by: Robert Yang > --- > layerindex/update.py | 115 > +++-- > layerindex/update_layer.py | 103 +--- > 2 files changed, 70 insertions(+), 148 deletions(-) > > diff --git a/layerindex/update.py b/layerindex/update.py > index d6488f0..e13c9ab 100755 > --- a/layerindex/update.py > +++ b/layerindex/update.py > @@ -21,6 +21,9 @@ import utils > import operator > import re > import multiprocessing > +import update_layer > +import copy > +import imp > > import warnings > warnings.filterwarnings("ignore", category=DeprecationWarning) > @@ -34,61 +37,15 @@ except ImportError: > logger.error("Please install PythonGit 0.3.1 or later in order to use > this script") > sys.exit(1) > > - > -def reenable_sigint(): > -signal.signal(signal.SIGINT, signal.SIG_DFL) > - > -def run_command_interruptible(cmd): > -""" > -Run a command with output displayed on the console, but ensure any > Ctrl+C is > -processed only by the child process. > -""" > -signal.signal(signal.SIGINT, signal.SIG_IGN) > +def call_update_layer(options, queue): > +ret = 1 > +output = "" > try: > -process = subprocess.Popen( > -cmd, cwd=os.path.dirname(sys.argv[0]), shell=True, > preexec_fn=reenable_sigint, stdout=subprocess.PIPE, stderr=subprocess.STDOUT > -) > - > -reader = codecs.getreader('utf-8')(process.stdout, > errors='surrogateescape') > -buf = '' > -while True: > -out = reader.read(1, 1) > -if out: > -sys.stdout.write(out) > -sys.stdout.flush() > -buf += out > -elif out == '' and process.poll() != None: > -break > - > +ret, output = update_layer.main(options, queue) > +except Exception: > +raise > finally: > -signal.signal(signal.SIGINT, signal.SIG_DFL) > -return process.returncode, buf > - > - > -def prepare_update_layer_command(options, branch, layer, initial=False): > -"""Prepare the update_layer.py command line""" > -if branch.update_environment: > -cmdprefix = branch.update_environment.get_command() > -else: > -cmdprefix = 'python3' > -cmd = '%s update_layer.py -l %s -b %s' % (cmdprefix, layer.name, > branch.name) > -if options.reload: > -cmd += ' --reload' > -if options.fullreload: > -cmd += ' --fullreload' > -if options.nocheckout: > -cmd += ' --nocheckout' > -if options.dryrun: > -cmd += ' -n' > -if initial: > -cmd += ' -i' > -if options.loglevel == logging.DEBUG: > -cmd += ' -d' > -elif options.loglevel == logging.ERROR: > -cmd += ' -q' > -if options.keep_temp: > -cmd += ' --keep-temp' > -return cmd > +queue.put([ret, output]) > > def update_actual_branch(layerquery, fetchdir, branch, options, > update_bitbake, bitbakepath): > """Update actual branch for layers and bitbake in database""" > @@ -266,6 +223,7 @@ def main(): > try: > lockfn = os.path.join(fetchdir, "layerindex.lock") > lockfile = utils.lock_file(lockfn) > +update_log_path = update_layer.update_log_path > if not lockfile: > logger.error("Layer index lock timeout expired") > sys.exit(1) > @@ -315,8 +273,15 @@ def main(): > # We now do this by calling out to a separate script; doing > otherwise turned out to be > # unreliable due to leaking memory (we're using bitbake > internals in a manner in which > # they never get used during normal operation). > +update_layer_options = copy.copy(options) > last_rev = {} > for branch in branches: > +# The bitbake modules are not compatible between branches, so > +# remove them, they will be imported again when needed. > +for name, module in sys.modules.copy().items(): > +if '__file__' in module.__dict__ and > module.__file__.startswith(bitbakepath): > +logger.debug('Removing module %s' % name) > +del sys.modules[name] This part
[yocto] [layerindex-web][PATCH] Handle __isnull in API query filtering
If you query on a boolean field you can use the string "False" to match False in the database; however if you try the same with __isnull then the query will match every record which is obviously undesirable. If __isnull is being used, then convert the value to a boolean so that the query works properly. An example of this type of query: http://127.0.0.1:8000/layerindex/api/layerBranches/?filter=yp_compatible_version__isnull:false Signed-off-by: Paul Eggleton --- layerindex/querysethelper.py | 24 ++-- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/layerindex/querysethelper.py b/layerindex/querysethelper.py index b4d30c2..4e7cac5 100644 --- a/layerindex/querysethelper.py +++ b/layerindex/querysethelper.py @@ -28,16 +28,16 @@ VALUE_SEPARATOR = "!" DESCENDING = "-" def __get_q_for_val(name, value): -if "OR" in value: -return functools.reduce(operator.or_, map(lambda x: __get_q_for_val(name, x), [ x for x in value.split("OR") ])) -if "AND" in value: -return functools.reduce(operator.and_, map(lambda x: __get_q_for_val(name, x), [ x for x in value.split("AND") ])) -if value.startswith("NOT"): -kwargs = { name : value.strip("NOT") } -return ~Q(**kwargs) -else: -kwargs = { name : value } -return Q(**kwargs) +if isinstance(value, str): +if "OR" in value: +return functools.reduce(operator.or_, map(lambda x: __get_q_for_val(name, x), [ x for x in value.split("OR") ])) +if "AND" in value: +return functools.reduce(operator.and_, map(lambda x: __get_q_for_val(name, x), [ x for x in value.split("AND") ])) +if value.startswith("NOT"): +kwargs = { name : value.strip("NOT") } +return ~Q(**kwargs) +kwargs = { name : value } +return Q(**kwargs) def _get_filtering_query(filter_string): @@ -46,6 +46,10 @@ def _get_filtering_query(filter_string): values = search_terms[1].split(VALUE_SEPARATOR) querydict = dict(zip(keys, values)) +for key in keys: +if key.endswith('__isnull'): +querydict[key] = (querydict[key].lower() == 'true') + return functools.reduce(operator.and_, map(lambda x: __get_q_for_val(x, querydict[x]), [k for k in querydict])) # we check that the input comes in a valid form that we can recognize -- 2.9.5 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Yocto Project Technical Team Meeting
On 02/03/2018 09:25 PM, Stephano Cetola wrote: > On 2/2/18 11:04 AM, Cetola, Stephano wrote: >> Cetola, Stephano has invited you to Yocto Project Technical Team Meeting >> >> When: >> >> Occurs the first Tuesday of every month effective 2/6/18 from 8:00 AM to >> 8:30 AM. >> >> Organizer: >> >> Cetola, Stephano >> >> Description: >> >> When: Occurs every month on the first Tuesday of the month from 8:00 AM >> to 8:30 AM effective 2/6/2018. (UTC-08:00) Pacific Time (US & Canada) > It has been mentioned to me that there is some confusion on when this > meeting occurs. I sent this invite using Outlook, which is a necessary > evil here. > > The meeting is from 8:00 - 8:30 (UTC -08:00). I'm happy to send a new > invite if enough people need one, but I want to minimize the noise here. Same time, Same Bat Channel only hosted by Robin? Is this link still correct? https://www.yoctoproject.org/tools-resources/community/monthly-technical-call - Armin We > > Cheers, > Stephano -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] runqemu looking for qemuboot.conf file...
I've written a bsp for the adzs-sc589-ezlite board from analog devices. I am trying to use qemu to emulate the board, but I get the following runqemu - INFO - Can't find qemuboot conf file, DEPLOY_DIR_IMAGE is NULL! runqemu - INFO - Running MACHINE=adzs-sc589-ezlite bitbake -e... runqemu - INFO - Overriding conf file setting of STAGING_DIR_NATIVE to /home/peter/yocto-build/build/tmp/work/armv5e-poky-linux-gnueabi/defaultpkgname/1.0-r0/recipe-sysroot-native from Bitbake environment Traceback (most recent call last): File "/home/peter/yocto-build/poky/scripts/runqemu", line 1243, in ret = main() File "/home/peter/yocto-build/poky/scripts/runqemu", line 1230, in main config.check_and_set() File "/home/peter/yocto-build/poky/scripts/runqemu", line 652, in check_and_set self.check_fstype() File "/home/peter/yocto-build/poky/scripts/runqemu", line 513, in check_fstype raise Exception("FSTYPE is NULL!") Exception: FSTYPE is NULL! I've tried forcing: $ bitbake core-image-minimal -c do_write_qemuboot_conf but with no success. Peter. This communication, including any attached documentation, is intended only for the person or entity to which it is addressed, and may contain confidential, personal, and/or privileged information. Any unauthorized disclosure, copying, or taking action on the contents is strictly prohibited. If you have received this message in error, please contact us immediately so we may correct our records. Please then delete or destroy the original transmission and any subsequent reply. Thank you. -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto