Re: [OE-core] [PATCH v2 3/6] yocto-compat-layer.py: apply test_signatures to all layers

2017-06-30 Thread Mark Hatle 
On 6/28/17 11:33 AM, Patrick Ohly wrote:
> On Wed, 2017-06-28 at 11:08 +0200, Mark Hatle wrote:
>> On 6/27/17 5:33 PM, Patrick Ohly wrote:
>>> Software layers were previously allowed to change signatures, but
>>> that's not desired for those layers either. The rule that a layer
>>> which is "Yocto Compatible 2.0" must not change signatures unless
>>> explicitly requested holds for all kinds of layers.
>>>
>>> However, as this is something that software layers might not be able
>>> to do right away, testing for signature changes in software layers can
>>> be disabled. It's on by default, as that was Richard's
>>> recommendation. Whether that should change needs further discussion as
>>> part of finalizing "Yocto Compatible 2.0".
>>>
>>> As it might still change, the tool now has both a with/without
>>> parameter so that users of the tool can choose the desired behavior
>>> without being affected by future changes to the default.
>>
>> How would you regulate the behavior of a software layer that is doing 
>> bbappends
>> or similar to a system provided component.
> 
> By adding a PACKAGECONFIG that is off by default?
> 
> But I haven't tried this and whether it influences task signatures. Do
> you have a specific example?
> 
> Regarding these patches, is it okay to merge them as they are now?
> Without them, we cannot test software layers for signature changes, so
> won't know how much of a problem it would be.
> 
> The tool and "Yocto Compatible 2.0" are work in progress, so there's
> still time to refine it after merging. My motivation for getting them
> merged already now is a) to make the change available to others and b)
> to use the strict version of the check in refkit (where we currently
> satisfy the criteria).
> 

Yes, this was a comment about the commit message, not the technical content of
the patch.

I'm not sure that PACKAGECONFIG can be used to control this type of stuff.  
Perhaps?

In many cases, the bbappends seem to add patches or change the way the
compile/install work.  Often these are integration specific for something -- so
there is no 'easy' way to use PACKAGECONFIG to control this.

The only way I can think of is using a PACKAGECONFIG to define a an override.
But as far as I know just adding the evaluation to the overrides and such will
affect the task hashes in some way.

--Mark
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 3/6] yocto-compat-layer.py: apply test_signatures to all layers

2017-06-28 Thread Patrick Ohly 
On Wed, 2017-06-28 at 11:08 +0200, Mark Hatle wrote:
> On 6/27/17 5:33 PM, Patrick Ohly wrote:
> > Software layers were previously allowed to change signatures, but
> > that's not desired for those layers either. The rule that a layer
> > which is "Yocto Compatible 2.0" must not change signatures unless
> > explicitly requested holds for all kinds of layers.
> > 
> > However, as this is something that software layers might not be able
> > to do right away, testing for signature changes in software layers can
> > be disabled. It's on by default, as that was Richard's
> > recommendation. Whether that should change needs further discussion as
> > part of finalizing "Yocto Compatible 2.0".
> > 
> > As it might still change, the tool now has both a with/without
> > parameter so that users of the tool can choose the desired behavior
> > without being affected by future changes to the default.
> 
> How would you regulate the behavior of a software layer that is doing 
> bbappends
> or similar to a system provided component.

By adding a PACKAGECONFIG that is off by default?

But I haven't tried this and whether it influences task signatures. Do
you have a specific example?

Regarding these patches, is it okay to merge them as they are now?
Without them, we cannot test software layers for signature changes, so
won't know how much of a problem it would be.

The tool and "Yocto Compatible 2.0" are work in progress, so there's
still time to refine it after merging. My motivation for getting them
merged already now is a) to make the change available to others and b)
to use the strict version of the check in refkit (where we currently
satisfy the criteria).

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.



-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 3/6] yocto-compat-layer.py: apply test_signatures to all layers

2017-06-28 Thread Mark Hatle 
On 6/27/17 5:33 PM, Patrick Ohly wrote:
> Software layers were previously allowed to change signatures, but
> that's not desired for those layers either. The rule that a layer
> which is "Yocto Compatible 2.0" must not change signatures unless
> explicitly requested holds for all kinds of layers.
> 
> However, as this is something that software layers might not be able
> to do right away, testing for signature changes in software layers can
> be disabled. It's on by default, as that was Richard's
> recommendation. Whether that should change needs further discussion as
> part of finalizing "Yocto Compatible 2.0".
> 
> As it might still change, the tool now has both a with/without
> parameter so that users of the tool can choose the desired behavior
> without being affected by future changes to the default.

How would you regulate the behavior of a software layer that is doing bbappends
or similar to a system provided component.

My understanding is that the inclusion of the software layer SHOULD be able to
modify the existing recipe behavior, as there currently is no other way to
'control' activation.

This is different then BSP and distro layers, which have a specific activation
(and thus override) behavior.

--Mark

> Signed-off-by: Patrick Ohly 
> ---
>  scripts/lib/compatlayer/cases/common.py |  5 +++--
>  scripts/lib/compatlayer/context.py  |  3 ++-
>  scripts/yocto-compat-layer.py   | 12 +---
>  3 files changed, 14 insertions(+), 6 deletions(-)
> 
> diff --git a/scripts/lib/compatlayer/cases/common.py 
> b/scripts/lib/compatlayer/cases/common.py
> index 2dfcbb1..a1cdbab 100644
> --- a/scripts/lib/compatlayer/cases/common.py
> +++ b/scripts/lib/compatlayer/cases/common.py
> @@ -27,8 +27,9 @@ class CommonCompatLayer(OECompatLayerTestCase):
>'bitbake -e')
>  
>  def test_signatures(self):
> -if self.tc.layer['type'] == LayerType.SOFTWARE:
> -raise unittest.SkipTest("Layer %s isn't BSP or DISTRO one." \
> +if self.tc.layer['type'] == LayerType.SOFTWARE and \
> +   not self.tc.test_software_layer_signatures:
> +raise unittest.SkipTest("Not testing for signature changes in a 
> software layer %s." \
>   % self.tc.layer['name'])
>  
>  # task -> (old signature, new signature)
> diff --git a/scripts/lib/compatlayer/context.py 
> b/scripts/lib/compatlayer/context.py
> index 4932238..7811d4a 100644
> --- a/scripts/lib/compatlayer/context.py
> +++ b/scripts/lib/compatlayer/context.py
> @@ -9,6 +9,7 @@ import re
>  from oeqa.core.context import OETestContext
>  
>  class CompatLayerTestContext(OETestContext):
> -def __init__(self, td=None, logger=None, layer=None):
> +def __init__(self, td=None, logger=None, layer=None, 
> test_software_layer_signatures=True):
>  super(CompatLayerTestContext, self).__init__(td, logger)
>  self.layer = layer
> +self.test_software_layer_signatures = test_software_layer_signatures
> diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
> index 30c55a9..a16974f 100755
> --- a/scripts/yocto-compat-layer.py
> +++ b/scripts/yocto-compat-layer.py
> @@ -30,12 +30,12 @@ CASES_PATHS = 
> [os.path.join(os.path.abspath(os.path.dirname(__file__)),
>  'lib', 'compatlayer', 'cases')]
>  logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
>  
> -def test_layer_compatibility(td, layer):
> +def test_layer_compatibility(td, layer, test_software_layer_signatures):
>  from compatlayer.context import CompatLayerTestContext
>  logger.info("Starting to analyze: %s" % layer['name'])
>  
> logger.info("--")
>  
> -tc = CompatLayerTestContext(td=td, logger=logger, layer=layer)
> +tc = CompatLayerTestContext(td=td, logger=logger, layer=layer, 
> test_software_layer_signatures=test_software_layer_signatures)
>  tc.loadTests(CASES_PATHS)
>  return tc.runTests()
>  
> @@ -53,6 +53,12 @@ def main():
>  help='List of MACHINEs to be used during testing', 
> action='store')
>  parser.add_argument('--additional-layers', nargs="+",
>  help='List of additional layers to add during testing', 
> action='store')
> +group = parser.add_mutually_exclusive_group()
> +group.add_argument('--with-software-layer-signature-check', 
> action='store_true', dest='test_software_layer_signatures',
> +   default=True,
> +   help='check that software layers do not change 
> signatures (on by default)')
> +group.add_argument('--without-software-layer-signature-check', 
> action='store_false', dest='test_software_layer_signatures',
> +   help='disable signature checking for software layers')
>  parser.add_argument('-n', '--no-auto', help='Disable auto layer 
> discovery',
>  action='store_true')
>

[OE-core] [PATCH v2 3/6] yocto-compat-layer.py: apply test_signatures to all layers

2017-06-27 Thread Patrick Ohly 
Software layers were previously allowed to change signatures, but
that's not desired for those layers either. The rule that a layer
which is "Yocto Compatible 2.0" must not change signatures unless
explicitly requested holds for all kinds of layers.

However, as this is something that software layers might not be able
to do right away, testing for signature changes in software layers can
be disabled. It's on by default, as that was Richard's
recommendation. Whether that should change needs further discussion as
part of finalizing "Yocto Compatible 2.0".

As it might still change, the tool now has both a with/without
parameter so that users of the tool can choose the desired behavior
without being affected by future changes to the default.

Signed-off-by: Patrick Ohly 
---
 scripts/lib/compatlayer/cases/common.py |  5 +++--
 scripts/lib/compatlayer/context.py  |  3 ++-
 scripts/yocto-compat-layer.py   | 12 +---
 3 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/compatlayer/cases/common.py 
b/scripts/lib/compatlayer/cases/common.py
index 2dfcbb1..a1cdbab 100644
--- a/scripts/lib/compatlayer/cases/common.py
+++ b/scripts/lib/compatlayer/cases/common.py
@@ -27,8 +27,9 @@ class CommonCompatLayer(OECompatLayerTestCase):
   'bitbake -e')
 
 def test_signatures(self):
-if self.tc.layer['type'] == LayerType.SOFTWARE:
-raise unittest.SkipTest("Layer %s isn't BSP or DISTRO one." \
+if self.tc.layer['type'] == LayerType.SOFTWARE and \
+   not self.tc.test_software_layer_signatures:
+raise unittest.SkipTest("Not testing for signature changes in a 
software layer %s." \
  % self.tc.layer['name'])
 
 # task -> (old signature, new signature)
diff --git a/scripts/lib/compatlayer/context.py 
b/scripts/lib/compatlayer/context.py
index 4932238..7811d4a 100644
--- a/scripts/lib/compatlayer/context.py
+++ b/scripts/lib/compatlayer/context.py
@@ -9,6 +9,7 @@ import re
 from oeqa.core.context import OETestContext
 
 class CompatLayerTestContext(OETestContext):
-def __init__(self, td=None, logger=None, layer=None):
+def __init__(self, td=None, logger=None, layer=None, 
test_software_layer_signatures=True):
 super(CompatLayerTestContext, self).__init__(td, logger)
 self.layer = layer
+self.test_software_layer_signatures = test_software_layer_signatures
diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index 30c55a9..a16974f 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -30,12 +30,12 @@ CASES_PATHS = 
[os.path.join(os.path.abspath(os.path.dirname(__file__)),
 'lib', 'compatlayer', 'cases')]
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
-def test_layer_compatibility(td, layer):
+def test_layer_compatibility(td, layer, test_software_layer_signatures):
 from compatlayer.context import CompatLayerTestContext
 logger.info("Starting to analyze: %s" % layer['name'])
 
logger.info("--")
 
-tc = CompatLayerTestContext(td=td, logger=logger, layer=layer)
+tc = CompatLayerTestContext(td=td, logger=logger, layer=layer, 
test_software_layer_signatures=test_software_layer_signatures)
 tc.loadTests(CASES_PATHS)
 return tc.runTests()
 
@@ -53,6 +53,12 @@ def main():
 help='List of MACHINEs to be used during testing', action='store')
 parser.add_argument('--additional-layers', nargs="+",
 help='List of additional layers to add during testing', 
action='store')
+group = parser.add_mutually_exclusive_group()
+group.add_argument('--with-software-layer-signature-check', 
action='store_true', dest='test_software_layer_signatures',
+   default=True,
+   help='check that software layers do not change 
signatures (on by default)')
+group.add_argument('--without-software-layer-signature-check', 
action='store_false', dest='test_software_layer_signatures',
+   help='disable signature checking for software layers')
 parser.add_argument('-n', '--no-auto', help='Disable auto layer discovery',
 action='store_true')
 parser.add_argument('-d', '--debug', help='Enable debug output',
@@ -173,7 +179,7 @@ def main():
 layers_tested = layers_tested + 1
 continue
 
-result = test_layer_compatibility(td, layer)
+result = test_layer_compatibility(td, layer, 
args.test_software_layer_signatures)
 results[layer['name']] = result
 results_status[layer['name']] = 'PASS' if 
results[layer['name']].wasSuccessful() else 'FAIL'
 layers_tested = layers_tested + 1
-- 
git-series 0.9.1
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org