Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
Hi Paul, I've sent the patches to mailing list: [yocto] [layerindex-web][PATCH 0/4] update.py: several fixes The one for recommends is: update.py: add layers when RECOMMENDS isn't satisfied Now you can drop [PATCH 5/7], others are still needed. // Robert On 07/06/2018 02:49 PM, Robert Yang wrote: On 07/06/2018 01:28 PM, Paul Eggleton wrote: Hi Robert On Wednesday, 4 July 2018 7:52:05 PM NZST you wrote: I'm sorry to say that I met layerindex' loaddata problems yesterday and today, I still didn't find the root cause. Have you tried dumpdata and loaddata recently, please ? What I did was: $ python3 manage.py dumpdata --settings settings --exclude=contenttypes --exclude=auth.Permission -- exclude=corsheaders >dumped.json On another environment: Setup database to sqlite3 in settings.py. $ python3 manage.py loaddata --settings settings dumped.json The first problem I got was: [snip] File "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/revisions.py", line 410, in _assert_registered model=model, reversion.errors.RegistrationError: Problem installing fixture '/buildarea1/lyang1/layerindex-web/dumped.json': has not been registered with django-reversion [snip] I think it is because we didn't use @reversion.register() for the class, so I added them to layerindex/models.py, then I got other errors: [snip] File "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/models.py", line 272, in _local_field_dict field_dict[field.attname] = getattr(obj, field.attname) AttributeError: Problem installing fixture '/buildarea1/lyang1/layerindex-web/dumped.json': 'Branch' object has no attribute 'layerbranch_id' Hmm, that's odd. Branch shouldn't have layerbranch_id, it's the other way around - LayerBranch has a branch_id. I'm not sure what's wrong atm, need more investigations. I need loaddata on my localhost to do development testing, so I can't start work on update.py until I fix the loaddata problem. I have used loaddata and dumpdata here (a couple of times) but not recently. I did not experience these issues before though. However these don't seem like issues that would have started as a result of this patchset (or indeed recent changes, other than perhaps an upgrade of django-reversion), have you been using loaddata/dumpdata prior to this? dumpdata/loaddata worked well before March, Konrad (in CC) worked it around by: dumpdata --exclude=corsheaders --exclude=reversion.version --exclude=reversion.revision --exclude=captcha.captchastore --exclude=sessions.session So I can loaddata now. I've finished patch for 5/7, but I met other problems when testing on completely new branch which caused by recently changes, I will fix them and send out patches later // Robert Cheers, Paul -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
On 07/06/2018 01:28 PM, Paul Eggleton wrote: Hi Robert On Wednesday, 4 July 2018 7:52:05 PM NZST you wrote: I'm sorry to say that I met layerindex' loaddata problems yesterday and today, I still didn't find the root cause. Have you tried dumpdata and loaddata recently, please ? What I did was: $ python3 manage.py dumpdata --settings settings --exclude=contenttypes --exclude=auth.Permission --exclude=corsheaders >dumped.json On another environment: Setup database to sqlite3 in settings.py. $ python3 manage.py loaddata --settings settings dumped.json The first problem I got was: [snip] File "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/revisions.py", line 410, in _assert_registered model=model, reversion.errors.RegistrationError: Problem installing fixture '/buildarea1/lyang1/layerindex-web/dumped.json': has not been registered with django-reversion [snip] I think it is because we didn't use @reversion.register() for the class, so I added them to layerindex/models.py, then I got other errors: [snip] File "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/models.py", line 272, in _local_field_dict field_dict[field.attname] = getattr(obj, field.attname) AttributeError: Problem installing fixture '/buildarea1/lyang1/layerindex-web/dumped.json': 'Branch' object has no attribute 'layerbranch_id' Hmm, that's odd. Branch shouldn't have layerbranch_id, it's the other way around - LayerBranch has a branch_id. I'm not sure what's wrong atm, need more investigations. I need loaddata on my localhost to do development testing, so I can't start work on update.py until I fix the loaddata problem. I have used loaddata and dumpdata here (a couple of times) but not recently. I did not experience these issues before though. However these don't seem like issues that would have started as a result of this patchset (or indeed recent changes, other than perhaps an upgrade of django-reversion), have you been using loaddata/dumpdata prior to this? dumpdata/loaddata worked well before March, Konrad (in CC) worked it around by: dumpdata --exclude=corsheaders --exclude=reversion.version --exclude=reversion.revision --exclude=captcha.captchastore --exclude=sessions.session So I can loaddata now. I've finished patch for 5/7, but I met other problems when testing on completely new branch which caused by recently changes, I will fix them and send out patches later // Robert Cheers, Paul -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
Hi Robert On Wednesday, 4 July 2018 7:52:05 PM NZST you wrote: > I'm sorry to say that I met layerindex' loaddata problems yesterday and > today, > I still didn't find the root cause. Have you tried dumpdata and loaddata > recently, please ? > > What I did was: > > $ python3 manage.py dumpdata --settings settings --exclude=contenttypes > --exclude=auth.Permission --exclude=corsheaders >dumped.json > > On another environment: > Setup database to sqlite3 in settings.py. > $ python3 manage.py loaddata --settings settings dumped.json > > The first problem I got was: > [snip] >File > "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/revisions.py", > > line 410, in _assert_registered > model=model, > reversion.errors.RegistrationError: Problem installing fixture > '/buildarea1/lyang1/layerindex-web/dumped.json': 'layerindex.models.Distro'> has not been registered with django-reversion > [snip] > > I think it is because we didn't use @reversion.register() for the class, so I > added them to layerindex/models.py, then I got other errors: > > [snip] >File > "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/models.py", > > line 272, in _local_field_dict > field_dict[field.attname] = getattr(obj, field.attname) > AttributeError: Problem installing fixture > '/buildarea1/lyang1/layerindex-web/dumped.json': 'Branch' object has no > attribute 'layerbranch_id' Hmm, that's odd. Branch shouldn't have layerbranch_id, it's the other way around - LayerBranch has a branch_id. > I'm not sure what's wrong atm, need more investigations. > > I need loaddata on my localhost to do development testing, so I can't start > work on update.py until I fix the loaddata problem. I have used loaddata and dumpdata here (a couple of times) but not recently. I did not experience these issues before though. However these don't seem like issues that would have started as a result of this patchset (or indeed recent changes, other than perhaps an upgrade of django-reversion), have you been using loaddata/dumpdata prior to this? Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
Hi Paul, I'm sorry to say that I met layerindex' loaddata problems yesterday and today, I still didn't find the root cause. Have you tried dumpdata and loaddata recently, please ? What I did was: $ python3 manage.py dumpdata --settings settings --exclude=contenttypes --exclude=auth.Permission --exclude=corsheaders >dumped.json On another environment: Setup database to sqlite3 in settings.py. $ python3 manage.py loaddata --settings settings dumped.json The first problem I got was: [snip] File "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/revisions.py", line 410, in _assert_registered model=model, reversion.errors.RegistrationError: Problem installing fixture '/buildarea1/lyang1/layerindex-web/dumped.json': 'layerindex.models.Distro'> has not been registered with django-reversion [snip] I think it is because we didn't use @reversion.register() for the class, so I added them to layerindex/models.py, then I got other errors: [snip] File "/buildarea1/lyang1/layerindex-web/.venv/lib/python3.5/site-packages/reversion/models.py", line 272, in _local_field_dict field_dict[field.attname] = getattr(obj, field.attname) AttributeError: Problem installing fixture '/buildarea1/lyang1/layerindex-web/dumped.json': 'Branch' object has no attribute 'layerbranch_id' I'm not sure what's wrong atm, need more investigations. I need loaddata on my localhost to do development testing, so I can't start work on update.py until I fix the loaddata problem. // Robert On 07/03/2018 11:08 AM, Robert Yang wrote: On 07/03/2018 10:58 AM, Paul Eggleton wrote: Hi Robert On Tuesday, 3 July 2018 2:45:11 PM NZST Robert Yang wrote: Thanks for let me know this, this patch might be incorrect, suppose we have two layers: core and hello: 1) LAYERRECOMMENDS_core = "hello" 2) $ update.py -l hello,core Then core maybe added before hello layer since it ignores recs on hello, and if hello is a new layer, it would not be in core's recs in database since core knows nothing about hello, I think that this is incorrect. If we really need this, I think that we should not ignore recs when the layer is present, but only ignore it when the layer is not present, for example, ignore it when hello layer doesn't exist, otherwise, don't ignore it. Can you come up with an alternative fix that doesn't break parsing like it does now? OK, I will, maybe I can send you a patch by tomorrow, but I'm not sure since the implementation might be a little complicated. // Robert But I'm not sure about patch 4 (error -> warning) either, since layerindex is a central database, whether add recs to conf/bblayers.conf should depend on end user rather than ignore it in database, otherwise the end user (especially the api user) would have no way to choice, for example, we use api to make conf/bblayers.conf have all or no recs layers according to user's choice, if the database is wrong, then there might be only part of recs layers. Though we can check update.py's warnings to fix the problem. Recommends are just that - recommendations. Someone might legitimately submit a layer that recommends another which they don't submit (perhaps a commercial layer?). The system shouldn't refuse to handle it or indicate that it's broken (it isn't, there might be reduced functionality but the layer will still be parseable by bitbake). I'll leave these changes unmerged for a bit in case you have a better fix for the current problems. Cheers, Paul -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
On 07/03/2018 10:58 AM, Paul Eggleton wrote: Hi Robert On Tuesday, 3 July 2018 2:45:11 PM NZST Robert Yang wrote: Thanks for let me know this, this patch might be incorrect, suppose we have two layers: core and hello: 1) LAYERRECOMMENDS_core = "hello" 2) $ update.py -l hello,core Then core maybe added before hello layer since it ignores recs on hello, and if hello is a new layer, it would not be in core's recs in database since core knows nothing about hello, I think that this is incorrect. If we really need this, I think that we should not ignore recs when the layer is present, but only ignore it when the layer is not present, for example, ignore it when hello layer doesn't exist, otherwise, don't ignore it. Can you come up with an alternative fix that doesn't break parsing like it does now? OK, I will, maybe I can send you a patch by tomorrow, but I'm not sure since the implementation might be a little complicated. // Robert But I'm not sure about patch 4 (error -> warning) either, since layerindex is a central database, whether add recs to conf/bblayers.conf should depend on end user rather than ignore it in database, otherwise the end user (especially the api user) would have no way to choice, for example, we use api to make conf/bblayers.conf have all or no recs layers according to user's choice, if the database is wrong, then there might be only part of recs layers. Though we can check update.py's warnings to fix the problem. Recommends are just that - recommendations. Someone might legitimately submit a layer that recommends another which they don't submit (perhaps a commercial layer?). The system shouldn't refuse to handle it or indicate that it's broken (it isn't, there might be reduced functionality but the layer will still be parseable by bitbake). I'll leave these changes unmerged for a bit in case you have a better fix for the current problems. Cheers, Paul -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
Hi Robert On Tuesday, 3 July 2018 2:45:11 PM NZST Robert Yang wrote: > Thanks for let me know this, this patch might be incorrect, suppose we have > two > layers: core and hello: > > 1) LAYERRECOMMENDS_core = "hello" > 2) $ update.py -l hello,core > > Then core maybe added before hello layer since it ignores recs on hello, and > if > hello is a new layer, it would not be in core's recs in database since core > knows nothing about hello, I think that this is incorrect. > > If we really need this, I think that we should not ignore recs when the > layer is present, but only ignore it when the layer is not present, for > example, ignore it when hello layer doesn't exist, otherwise, don't ignore it. Can you come up with an alternative fix that doesn't break parsing like it does now? > But I'm not sure about patch 4 (error -> warning) either, since layerindex is > a central database, whether add recs to conf/bblayers.conf should depend on > end user rather than ignore it in database, otherwise the end user (especially > the api user) would have no way to choice, for example, we use api to make > conf/bblayers.conf have all or no recs layers according to user's choice, > if the database is wrong, then there might be only part of recs layers. > Though we can check update.py's warnings to fix the problem. Recommends are just that - recommendations. Someone might legitimately submit a layer that recommends another which they don't submit (perhaps a commercial layer?). The system shouldn't refuse to handle it or indicate that it's broken (it isn't, there might be reduced functionality but the layer will still be parseable by bitbake). I'll leave these changes unmerged for a bit in case you have a better fix for the current problems. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
Hi Paul,
Thanks for let me know this, this patch might be incorrect, suppose we have two
layers: core and hello:
1) LAYERRECOMMENDS_core = "hello"
2) $ update.py -l hello,core
Then core maybe added before hello layer since it ignores recs on hello, and if
hello is a new layer, it would not be in core's recs in database since core
knows nothing about hello, I think that this is incorrect.
If we really need this, I think that we should not ignore recs when the
layer is present, but only ignore it when the layer is not present, for
example, ignore it when hello layer doesn't exist, otherwise, don't ignore it.
But I'm not sure about patch 4 (error -> warning) either, since layerindex is
a central database, whether add recs to conf/bblayers.conf should depend on
end user rather than ignore it in database, otherwise the end user (especially
the api user) would have no way to choice, for example, we use api to make
conf/bblayers.conf have all or no recs layers according to user's choice,
if the database is wrong, then there might be only part of recs layers.
Though we can check update.py's warnings to fix the problem.
// Robert
On 07/03/2018 06:58 AM, Paul Eggleton wrote:
We don't actually need to consider recommended layers when preparing the
order - just the depends. If we do then we can get into circular
dependency situation e.g. currently with meta-intel and meta-intel-qat
where meta-intel recommends meta-intel-qat and meta-intel-qat depends on
meta-intel. (Likely the latter dependency is erroneous since the content
of meta-intel-qat doesn't appear to depend on meta-intel, but there
could be other scenarios where it is legitimate).
Signed-off-by: Paul Eggleton
---
layerindex/update.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/layerindex/update.py b/layerindex/update.py
index 06c61a79..a4b96e24 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -399,7 +399,7 @@ def main():
deps = re.search("^LAYERDEPENDS = \"(.*)\"", output,
re.M).group(1) or ''
recs = re.search("^LAYERRECOMMENDS = \"(.*)\"", output,
re.M).group(1) or ''
-deps_dict = utils.explode_dep_versions2(bitbakepath, deps + ' ' + recs)
+deps_dict = utils.explode_dep_versions2(bitbakepath, deps)
if len(deps_dict) == 0:
# No depends, add it firstly
layerquery_sorted.append(layer)
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers
We don't actually need to consider recommended layers when preparing the
order - just the depends. If we do then we can get into circular
dependency situation e.g. currently with meta-intel and meta-intel-qat
where meta-intel recommends meta-intel-qat and meta-intel-qat depends on
meta-intel. (Likely the latter dependency is erroneous since the content
of meta-intel-qat doesn't appear to depend on meta-intel, but there
could be other scenarios where it is legitimate).
Signed-off-by: Paul Eggleton
---
layerindex/update.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/layerindex/update.py b/layerindex/update.py
index 06c61a79..a4b96e24 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -399,7 +399,7 @@ def main():
deps = re.search("^LAYERDEPENDS = \"(.*)\"", output,
re.M).group(1) or ''
recs = re.search("^LAYERRECOMMENDS = \"(.*)\"", output,
re.M).group(1) or ''
-deps_dict = utils.explode_dep_versions2(bitbakepath, deps
+ ' ' + recs)
+deps_dict = utils.explode_dep_versions2(bitbakepath, deps)
if len(deps_dict) == 0:
# No depends, add it firstly
layerquery_sorted.append(layer)
--
2.17.1
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
