Re: [yocto] [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch

[Liam R. Howlett]

* Paul Eggleton  [161003 18:54]:
> On Mon, 26 Sep 2016 14:25:36 Liam R. Howlett wrote:
> > Collection and version will be pulled from the layer.conf if it exists
> > and dependencies will be resolved by first checking for layers with the
> > dependency name and then checking for collections.  It is necessary to
> > shutdown tinfoil to avoid bitbake complaining about multiple instances.
> 
> I'm not sure I understand why though, and there's not an explanation given 
> here. What you're doing as a workaround only works by chance and I suspect it 
> will break horribly when we do the tinfoil2 refactor in the next release 
> (it's 
> not expected that you'd have datastores still around and be parsing recipes 
> when tinfoil has been shut down), so we *really* need to try to avoid 
> shutting 
> down like this. Can you explain what you saw?


I received messages about having multiple copies of bitbake running if I
did not shut down tinfoil.  I also hit `too many files opened` in the
update.py loop.  I have since reworked this patch to pass in the tinfoil
instance to layerconfparser in this patch set.  I was almost done v2
when I received your comments.  I also noticed that I had shut down
tinfoil before it should have been shut down - in v2, I had moved my
shutdown call next to sys.exit(0).  I feel much better about the way I
handle this in v2.

> 
> Cheers,
> Paul
> 
> -- 
> 
> Paul Eggleton
> Intel Open Source Technology Centre
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch

[Paul Eggleton]

On Mon, 26 Sep 2016 14:25:36 Liam R. Howlett wrote:
> Collection and version will be pulled from the layer.conf if it exists
> and dependencies will be resolved by first checking for layers with the
> dependency name and then checking for collections.  It is necessary to
> shutdown tinfoil to avoid bitbake complaining about multiple instances.

I'm not sure I understand why though, and there's not an explanation given 
here. What you're doing as a workaround only works by chance and I suspect it 
will break horribly when we do the tinfoil2 refactor in the next release (it's 
not expected that you'd have datastores still around and be parsing recipes 
when tinfoil has been shut down), so we *really* need to try to avoid shutting 
down like this. Can you explain what you saw?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch

[Mark Hatle]

On 9/26/16 1:25 PM, Liam R. Howlett wrote:
> Collection and version will be pulled from the layer.conf if it exists
> and dependencies will be resolved by first checking for layers with the
> dependency name and then checking for collections.  It is necessary to
> shutdown tinfoil to avoid bitbake complaining about multiple instances.

I think I found a bug in this patch.  The collection/version apparently needs to
be 'unique' in the layerbranch.

So if you have two layerbranch entries, associated with different branches (say
master and krogoth) you will get an error that they both have the same
collection name.

(This happened when I tried to use the admin interface to set the values.)

I will look into a solution for this problem.

--Mark

> Signed-off-by: Liam R. Howlett 
> ---
>  layerindex/models.py |  2 ++
>  layerindex/tools/import_layer.py |  3 ---
>  layerindex/update.py |  1 +
>  layerindex/update_layer.py   | 25 +++--
>  layerindex/utils.py  | 16 ++--
>  5 files changed, 36 insertions(+), 11 deletions(-)
> 
> diff --git a/layerindex/models.py b/layerindex/models.py
> index 2db8818..dcccb1f 100644
> --- a/layerindex/models.py
> +++ b/layerindex/models.py
> @@ -130,6 +130,8 @@ class LayerItem(models.Model):
>  class LayerBranch(models.Model):
>  layer = models.ForeignKey(LayerItem)
>  branch = models.ForeignKey(Branch)
> +collection = models.CharField('Layer Collection', max_length=40, 
> unique=True, null=True, help_text='Name of the layer that could be used in 
> the list of dependencies - must be unique and can only contain letters, 
> numbers and dashes')
> +version = models.CharField('Layer Version', max_length=10, null=True, 
> help_text='The layer version for this particular branch.')
>  vcs_subdir = models.CharField('Repository subdirectory', max_length=40, 
> blank=True, help_text='Subdirectory within the repository where the layer is 
> located, if not in the root (usually only used if the repository contains 
> more than one layer)')
>  vcs_last_fetch = models.DateTimeField('Last successful fetch', 
> blank=True, null=True)
>  vcs_last_rev = models.CharField('Last revision fetched', max_length=80, 
> blank=True)
> diff --git a/layerindex/tools/import_layer.py 
> b/layerindex/tools/import_layer.py
> index 0a13d21..21dd802 100755
> --- a/layerindex/tools/import_layer.py
> +++ b/layerindex/tools/import_layer.py
> @@ -383,9 +383,6 @@ def main():
>  layerdep.layerbranch = layerbranch
>  layerdep.dependency = core_layer
>  layerdep.save()
> -layerconfparser = LayerConfParse(logger=logger)
> -config_data = layerconfparser.parse_layer(layerdir)
> -layerconfparser.shutdown()
>  utils.add_dependencies(layerbranch, config_data, 
> logger=logger)
>  
>  
> diff --git a/layerindex/update.py b/layerindex/update.py
> index ecd2380..e15caf6 100755
> --- a/layerindex/update.py
> +++ b/layerindex/update.py
> @@ -213,6 +213,7 @@ def main():
>  layerbranch = layer.get_layerbranch(branch)
>  if layerbranch.vcs_subdir:
>  repodir = os.path.join(repodir, layerbranch.vcs_subdir)
> +
>  config_data = layerconfparser.parse_layer(repodir)
>  utils.add_dependencies(layerbranch, config_data, 
> logger=logger)
>  
> diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
> index a937000..22c0dab 100644
> --- a/layerindex/update_layer.py
> +++ b/layerindex/update_layer.py
> @@ -196,16 +196,18 @@ def main():
>  except recipeparse.RecipeParseError as e:
>  logger.error(str(e))
>  sys.exit(1)
> +config_data = bb.data.createCopy(tinfoil.config_data)
> +tinfoil.shutdown()
>  
>  # Clear the default value of SUMMARY so that we can use DESCRIPTION 
> instead if it hasn't been set
> -tinfoil.config_data.setVar('SUMMARY', '')
> +config_data.setVar('SUMMARY', '')
>  # Clear the default value of DESCRIPTION so that we can see where it's 
> not set
> -tinfoil.config_data.setVar('DESCRIPTION', '')
> +config_data.setVar('DESCRIPTION', '')
>  # Clear the default value of HOMEPAGE ('unknown')
> -tinfoil.config_data.setVar('HOMEPAGE', '')
> +config_data.setVar('HOMEPAGE', '')
>  # Set a blank value for LICENSE so that it doesn't cause the parser to 
> die (e.g. with meta-ti -
>  # why won't they just fix that?!)
> -tinfoil.config_data.setVar('LICENSE', '')
> +config_data.setVar('LICENSE', '')
>  
>  try:
>  with transaction.atomic():
> @@ -244,7 +246,7 @@ def main():
>  layerbranch = LayerBranch()
>  layerbranch.layer = layer
>  layerbranch.branch = branch
> -layerbranch_source = 

[yocto] [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch

[Liam R. Howlett]

Collection and version will be pulled from the layer.conf if it exists
and dependencies will be resolved by first checking for layers with the
dependency name and then checking for collections.  It is necessary to
shutdown tinfoil to avoid bitbake complaining about multiple instances.

Signed-off-by: Liam R. Howlett 
---
 layerindex/models.py |  2 ++
 layerindex/tools/import_layer.py |  3 ---
 layerindex/update.py |  1 +
 layerindex/update_layer.py   | 25 +++--
 layerindex/utils.py  | 16 ++--
 5 files changed, 36 insertions(+), 11 deletions(-)

diff --git a/layerindex/models.py b/layerindex/models.py
index 2db8818..dcccb1f 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -130,6 +130,8 @@ class LayerItem(models.Model):
 class LayerBranch(models.Model):
 layer = models.ForeignKey(LayerItem)
 branch = models.ForeignKey(Branch)
+collection = models.CharField('Layer Collection', max_length=40, 
unique=True, null=True, help_text='Name of the layer that could be used in the 
list of dependencies - must be unique and can only contain letters, numbers and 
dashes')
+version = models.CharField('Layer Version', max_length=10, null=True, 
help_text='The layer version for this particular branch.')
 vcs_subdir = models.CharField('Repository subdirectory', max_length=40, 
blank=True, help_text='Subdirectory within the repository where the layer is 
located, if not in the root (usually only used if the repository contains more 
than one layer)')
 vcs_last_fetch = models.DateTimeField('Last successful fetch', blank=True, 
null=True)
 vcs_last_rev = models.CharField('Last revision fetched', max_length=80, 
blank=True)
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 0a13d21..21dd802 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -383,9 +383,6 @@ def main():
 layerdep.layerbranch = layerbranch
 layerdep.dependency = core_layer
 layerdep.save()
-layerconfparser = LayerConfParse(logger=logger)
-config_data = layerconfparser.parse_layer(layerdir)
-layerconfparser.shutdown()
 utils.add_dependencies(layerbranch, config_data, 
logger=logger)
 
 
diff --git a/layerindex/update.py b/layerindex/update.py
index ecd2380..e15caf6 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -213,6 +213,7 @@ def main():
 layerbranch = layer.get_layerbranch(branch)
 if layerbranch.vcs_subdir:
 repodir = os.path.join(repodir, layerbranch.vcs_subdir)
+
 config_data = layerconfparser.parse_layer(repodir)
 utils.add_dependencies(layerbranch, config_data, logger=logger)
 
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index a937000..22c0dab 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -196,16 +196,18 @@ def main():
 except recipeparse.RecipeParseError as e:
 logger.error(str(e))
 sys.exit(1)
+config_data = bb.data.createCopy(tinfoil.config_data)
+tinfoil.shutdown()
 
 # Clear the default value of SUMMARY so that we can use DESCRIPTION 
instead if it hasn't been set
-tinfoil.config_data.setVar('SUMMARY', '')
+config_data.setVar('SUMMARY', '')
 # Clear the default value of DESCRIPTION so that we can see where it's not 
set
-tinfoil.config_data.setVar('DESCRIPTION', '')
+config_data.setVar('DESCRIPTION', '')
 # Clear the default value of HOMEPAGE ('unknown')
-tinfoil.config_data.setVar('HOMEPAGE', '')
+config_data.setVar('HOMEPAGE', '')
 # Set a blank value for LICENSE so that it doesn't cause the parser to die 
(e.g. with meta-ti -
 # why won't they just fix that?!)
-tinfoil.config_data.setVar('LICENSE', '')
+config_data.setVar('LICENSE', '')
 
 try:
 with transaction.atomic():
@@ -244,7 +246,7 @@ def main():
 layerbranch = LayerBranch()
 layerbranch.layer = layer
 layerbranch.branch = branch
-layerbranch_source = layer.get_layerbranch('master')
+
 if not layerbranch_source:
 layerbranch_source = layer.get_layerbranch(None)
 if layerbranch_source:
@@ -262,6 +264,9 @@ def main():
 dep.layerbranch = layerbranch
 dep.save()
 
+layerbranch_source = layer.get_layerbranch('master')
+
+
 if layerbranch.vcs_subdir and not options.nocheckout:
 # Find latest commit in subdirectory
 # A bit odd to do it this way but apparently there's no other 
way in the GitPython API
@@ -280,6 +285,14 @@ def main():
 
 layerdir =