Re: [OE-core] [PATCH] devtool-source.bbclass: Support kernel fragments not in SRC_URI

2018-08-09 Thread Anuj Mittal 
On 08/03/2018 04:55 AM, Alejandro Enedino Hernandez Samaniego wrote:
> We can give it a try but do you have a specific example of how it fails
> for you?
> Because I know theres lots of nested sccs on the yocto kernel cache, but
> in that case
> it wouldn't be a problem since this bug is specifically introduced by
> devtool when it
> copies local files from SRC_URI to a oe-local-files directory, it misses
> the corresponding cfg files (or patch files)
> since they're not listed in SRC_URI and it fails to build, in the case
> of the yocto kernel cache, it
> does not contain local files, so they wont be moved hence why it
> wouldn't be an issue.
> 
> From the kernel dev manual:
> https://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html#recipe-space-metadata
> :
> It is only necessary to specify the |.scc| files on the |SRC_URI|
> .

You are ignoring the line right next to it: "BitBake parses them and
fetches any files referenced in the .scc files by the include, patch, or
kconf commands." All files specified using 'include' will be parsed too
even if they are not in SRC_URI and this includes other scc files.

This change is good because it fixes something that is definitely
broken. My suggestion was meant to improve it further. I had provided
the test case in my reply to original patch.

And, I just tested it again using this change applied and I do see
errors when using a nested scc. Here it is again:

│   ├── linux
│   │   ├── linux-intel
│   │   │   ├── abc.scc
│   │   │   ├── def.scc

abc.scc only includes def.scc. And, def.scc includes a patch. SRC_URI
includes only abc.scc.

> 
> It specifies that only the scc files need to be on SRC_URI, which is why
> I'm saying the script
> will eventually run into them from the list.
> 
> I want to be clear that I'm not against doing this recursively, I just
> want to understand your testcase better.

I think if you'd not like to consider this use case as part of this
change, that should also be fine. But, do consider adding a new bug in
that case.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool-source.bbclass: Support kernel fragments not in SRC_URI

2018-08-02 Thread Alejandro Enedino Hernandez Samaniego 



On 08/02/2018 01:53 AM, Richard Purdie wrote:

On Thu, 2018-08-02 at 10:55 +0800, Anuj Mittal wrote:

On 08/02/2018 07:30 AM, Alejandro Enedino Hernandez Samaniego wrote:

Hey Anuj,


On 08/01/2018 04:25 AM, Anuj Mittal wrote:

On 07/31/2018 05:21 AM, Jaewon Lee wrote:

When using a recipe space kernel-meta, scc files are added
through
SRC_URI, but they may include corresponding kernel fragments
that are
not necessarily in SRC_URI.

For bitbake, this is not a problem because the kernel-yocto
class adds
the path where the .scc file was found to includes which
consequentially
makes the .cfg file available to the kernel build.

However, when using devtool, only files specified in SRC_URI
are copied
to oe-local-files in devtool's workspace. So if the cfg file is
not in
SRC_URI, it won't be copied, causing a kernel build failure
when trying
to find it.

This fix parses local .scc files in SRC_URI, copies the
corresponding .cfg
file to devtool's workdir, and also adds it to local_files so
it is
available when doing a devtool build for the kernel.

[YOCTO #12858]

Signed-off-by: Jaewon Lee 
Signed-off-by: Alejandro Enedino Hernandez Samaniego 
---
   meta/classes/devtool-source.bbclass | 12 
   1 file changed, 12 insertions(+)

diff --git a/meta/classes/devtool-source.bbclass
b/meta/classes/devtool-source.bbclass
index 56882a4..c70fea2 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -90,11 +90,23 @@ python devtool_post_unpack() {
   fname in files])
   return ret
   
+is_kernel_yocto = bb.data.inherits_class('kernel-yocto',

d)
   # Move local source files into separate subdir
   recipe_patches = [os.path.basename(patch) for patch in
   oe.recipeutils.get_recipe_patches(d)]
   local_files = oe.recipeutils.get_recipe_local_files(d)
   
+if is_kernel_yocto:

+  for key in local_files.copy():
+if key.endswith('scc'):
+  sccfile = open(local_files[key], 'r')
+  for l in sccfile:
+line = l.split()
+if line and line[0] == 'kconf' and line[-
1].endswith('.cfg'):
+  local_files[line[-1]] =
os.path.join(os.path.dirname(local_files[key]), line[-1])
+  shutil.copy2(os.path.join(os.path.dirname(local_
files[key]), line[-1]), workdir)
+  sccfile.close()
+

Would the patches included in these .scc files also need to be
handled
in the same way? Would this also work if there are other scc
files
included in a scc file?

Yes, I believe that the same mechanism should be used for patch
files as
well,
basically anything that may be needed to build but that its not
necessarily
explicitly listed on SRC_URI.

I believe it will work for other scc files and it doesnt have to
be
recursive,
because while cfg files arent required to be in SRC_URI , scc files
ARE
required
to be SRC_URI, which means that this will eventually find the other
scc
file on the list

I don't think they are required to be specified except for the top
level
one. At least, when I test it, I see problems. kernel-tools spp
script
parses them recursively and looks for a nested scc even if it is not
specified as part of SRC_URI. That is how the top level sccs from
kernel-cache are parsed too. Can you give it a try please?

Hey Anuj,
We can give it a try but do you have a specific example of how it fails 
for you?
Because I know theres lots of nested sccs on the yocto kernel cache, but 
in that case
it wouldn't be a problem since this bug is specifically introduced by 
devtool when it
copies local files from SRC_URI to a oe-local-files directory, it misses 
the corresponding cfg files (or patch files)
since they're not listed in SRC_URI and it fails to build, in the case 
of the yocto kernel cache, it
does not contain local files, so they wont be moved hence why it 
wouldn't be an issue.


From the kernel dev manual:
https://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html#recipe-space-metadata 
:
It is only necessary to specify the|.scc|files on the|SRC_URI| 
.


It specifies that only the scc files need to be on SRC_URI, which is why 
I'm saying the script

will eventually run into them from the list.

I want to be clear that I'm not against doing this recursively, I just 
want to understand your testcase better.

What would be really great here would be some test cases in the qa
framework! It'd be a good way to both illustrate the problems and then
test we've fixed it and that it stays fixed.

Hey Richard,
Sure that sounds like a good idea, well work on adding a testcase (or 
adding to the existing one) for devtool when

it is using a recipe-space kernel -meta.

Alejandro


Cheers,

Richard



-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org

Re: [OE-core] [PATCH] devtool-source.bbclass: Support kernel fragments not in SRC_URI

2018-08-02 Thread Richard Purdie 
On Thu, 2018-08-02 at 10:55 +0800, Anuj Mittal wrote:
> On 08/02/2018 07:30 AM, Alejandro Enedino Hernandez Samaniego wrote:
> > Hey Anuj,
> > 
> > 
> > On 08/01/2018 04:25 AM, Anuj Mittal wrote:
> > > On 07/31/2018 05:21 AM, Jaewon Lee wrote:
> > > > When using a recipe space kernel-meta, scc files are added
> > > > through
> > > > SRC_URI, but they may include corresponding kernel fragments
> > > > that are
> > > > not necessarily in SRC_URI.
> > > > 
> > > > For bitbake, this is not a problem because the kernel-yocto
> > > > class adds
> > > > the path where the .scc file was found to includes which
> > > > consequentially
> > > > makes the .cfg file available to the kernel build.
> > > > 
> > > > However, when using devtool, only files specified in SRC_URI
> > > > are copied
> > > > to oe-local-files in devtool's workspace. So if the cfg file is
> > > > not in
> > > > SRC_URI, it won't be copied, causing a kernel build failure
> > > > when trying
> > > > to find it.
> > > > 
> > > > This fix parses local .scc files in SRC_URI, copies the
> > > > corresponding .cfg
> > > > file to devtool's workdir, and also adds it to local_files so
> > > > it is
> > > > available when doing a devtool build for the kernel.
> > > > 
> > > > [YOCTO #12858]
> > > > 
> > > > Signed-off-by: Jaewon Lee 
> > > > Signed-off-by: Alejandro Enedino Hernandez Samaniego  > > > xilinx.com>
> > > > ---
> > > >   meta/classes/devtool-source.bbclass | 12 
> > > >   1 file changed, 12 insertions(+)
> > > > 
> > > > diff --git a/meta/classes/devtool-source.bbclass
> > > > b/meta/classes/devtool-source.bbclass
> > > > index 56882a4..c70fea2 100644
> > > > --- a/meta/classes/devtool-source.bbclass
> > > > +++ b/meta/classes/devtool-source.bbclass
> > > > @@ -90,11 +90,23 @@ python devtool_post_unpack() {
> > > >   fname in files])
> > > >   return ret
> > > >   
> > > > +is_kernel_yocto = bb.data.inherits_class('kernel-yocto',
> > > > d)
> > > >   # Move local source files into separate subdir
> > > >   recipe_patches = [os.path.basename(patch) for patch in
> > > >   oe.recipeutils.get_recipe_patches(d)]
> > > >   local_files = oe.recipeutils.get_recipe_local_files(d)
> > > >   
> > > > +if is_kernel_yocto:
> > > > +  for key in local_files.copy():
> > > > +if key.endswith('scc'):
> > > > +  sccfile = open(local_files[key], 'r')
> > > > +  for l in sccfile:
> > > > +line = l.split()
> > > > +if line and line[0] == 'kconf' and line[-
> > > > 1].endswith('.cfg'):
> > > > +  local_files[line[-1]] =
> > > > os.path.join(os.path.dirname(local_files[key]), line[-1])
> > > > +  shutil.copy2(os.path.join(os.path.dirname(local_
> > > > files[key]), line[-1]), workdir)
> > > > +  sccfile.close()
> > > > +
> > > 
> > > Would the patches included in these .scc files also need to be
> > > handled
> > > in the same way? Would this also work if there are other scc
> > > files
> > > included in a scc file?
> > 
> > Yes, I believe that the same mechanism should be used for patch
> > files as 
> > well,
> > basically anything that may be needed to build but that its not
> > necessarily
> > explicitly listed on SRC_URI.
> > 
> > I believe it will work for other scc files and it doesnt have to
> > be 
> > recursive,
> > because while cfg files arent required to be in SRC_URI , scc files
> > ARE 
> > required
> > to be SRC_URI, which means that this will eventually find the other
> > scc 
> > file on the list
> 
> I don't think they are required to be specified except for the top
> level
> one. At least, when I test it, I see problems. kernel-tools spp
> script
> parses them recursively and looks for a nested scc even if it is not
> specified as part of SRC_URI. That is how the top level sccs from
> kernel-cache are parsed too. Can you give it a try please?

What would be really great here would be some test cases in the qa
framework! It'd be a good way to both illustrate the problems and then
test we've fixed it and that it stays fixed.

Cheers,

Richard

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool-source.bbclass: Support kernel fragments not in SRC_URI

2018-08-01 Thread Anuj Mittal 
On 08/02/2018 07:30 AM, Alejandro Enedino Hernandez Samaniego wrote:
> Hey Anuj,
> 
> 
> On 08/01/2018 04:25 AM, Anuj Mittal wrote:
>> On 07/31/2018 05:21 AM, Jaewon Lee wrote:
>>> When using a recipe space kernel-meta, scc files are added through
>>> SRC_URI, but they may include corresponding kernel fragments that are
>>> not necessarily in SRC_URI.
>>>
>>> For bitbake, this is not a problem because the kernel-yocto class adds
>>> the path where the .scc file was found to includes which consequentially
>>> makes the .cfg file available to the kernel build.
>>>
>>> However, when using devtool, only files specified in SRC_URI are copied
>>> to oe-local-files in devtool's workspace. So if the cfg file is not in
>>> SRC_URI, it won't be copied, causing a kernel build failure when trying
>>> to find it.
>>>
>>> This fix parses local .scc files in SRC_URI, copies the corresponding .cfg
>>> file to devtool's workdir, and also adds it to local_files so it is
>>> available when doing a devtool build for the kernel.
>>>
>>> [YOCTO #12858]
>>>
>>> Signed-off-by: Jaewon Lee 
>>> Signed-off-by: Alejandro Enedino Hernandez Samaniego 
>>> ---
>>>   meta/classes/devtool-source.bbclass | 12 
>>>   1 file changed, 12 insertions(+)
>>>
>>> diff --git a/meta/classes/devtool-source.bbclass 
>>> b/meta/classes/devtool-source.bbclass
>>> index 56882a4..c70fea2 100644
>>> --- a/meta/classes/devtool-source.bbclass
>>> +++ b/meta/classes/devtool-source.bbclass
>>> @@ -90,11 +90,23 @@ python devtool_post_unpack() {
>>>   fname in files])
>>>   return ret
>>>   
>>> +is_kernel_yocto = bb.data.inherits_class('kernel-yocto', d)
>>>   # Move local source files into separate subdir
>>>   recipe_patches = [os.path.basename(patch) for patch in
>>>   oe.recipeutils.get_recipe_patches(d)]
>>>   local_files = oe.recipeutils.get_recipe_local_files(d)
>>>   
>>> +if is_kernel_yocto:
>>> +  for key in local_files.copy():
>>> +if key.endswith('scc'):
>>> +  sccfile = open(local_files[key], 'r')
>>> +  for l in sccfile:
>>> +line = l.split()
>>> +if line and line[0] == 'kconf' and line[-1].endswith('.cfg'):
>>> +  local_files[line[-1]] = 
>>> os.path.join(os.path.dirname(local_files[key]), line[-1])
>>> +  shutil.copy2(os.path.join(os.path.dirname(local_files[key]), 
>>> line[-1]), workdir)
>>> +  sccfile.close()
>>> +
>> Would the patches included in these .scc files also need to be handled
>> in the same way? Would this also work if there are other scc files
>> included in a scc file?
> Yes, I believe that the same mechanism should be used for patch files as 
> well,
> basically anything that may be needed to build but that its not necessarily
> explicitly listed on SRC_URI.
> 
> I believe it will work for other scc files and it doesnt have to be 
> recursive,
> because while cfg files arent required to be in SRC_URI , scc files ARE 
> required
> to be SRC_URI, which means that this will eventually find the other scc 
> file on the list

I don't think they are required to be specified except for the top level
one. At least, when I test it, I see problems. kernel-tools spp script
parses them recursively and looks for a nested scc even if it is not
specified as part of SRC_URI. That is how the top level sccs from
kernel-cache are parsed too. Can you give it a try please?

Thanks,

Anuj
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool-source.bbclass: Support kernel fragments not in SRC_URI

2018-08-01 Thread Alejandro Enedino Hernandez Samaniego 

Hey Anuj,


On 08/01/2018 04:25 AM, Anuj Mittal wrote:

On 07/31/2018 05:21 AM, Jaewon Lee wrote:

When using a recipe space kernel-meta, scc files are added through
SRC_URI, but they may include corresponding kernel fragments that are
not necessarily in SRC_URI.

For bitbake, this is not a problem because the kernel-yocto class adds
the path where the .scc file was found to includes which consequentially
makes the .cfg file available to the kernel build.

However, when using devtool, only files specified in SRC_URI are copied
to oe-local-files in devtool's workspace. So if the cfg file is not in
SRC_URI, it won't be copied, causing a kernel build failure when trying
to find it.

This fix parses local .scc files in SRC_URI, copies the corresponding .cfg
file to devtool's workdir, and also adds it to local_files so it is
available when doing a devtool build for the kernel.

[YOCTO #12858]

Signed-off-by: Jaewon Lee 
Signed-off-by: Alejandro Enedino Hernandez Samaniego 
---
  meta/classes/devtool-source.bbclass | 12 
  1 file changed, 12 insertions(+)

diff --git a/meta/classes/devtool-source.bbclass 
b/meta/classes/devtool-source.bbclass
index 56882a4..c70fea2 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -90,11 +90,23 @@ python devtool_post_unpack() {
  fname in files])
  return ret
  
+is_kernel_yocto = bb.data.inherits_class('kernel-yocto', d)

  # Move local source files into separate subdir
  recipe_patches = [os.path.basename(patch) for patch in
  oe.recipeutils.get_recipe_patches(d)]
  local_files = oe.recipeutils.get_recipe_local_files(d)
  
+if is_kernel_yocto:

+  for key in local_files.copy():
+if key.endswith('scc'):
+  sccfile = open(local_files[key], 'r')
+  for l in sccfile:
+line = l.split()
+if line and line[0] == 'kconf' and line[-1].endswith('.cfg'):
+  local_files[line[-1]] = 
os.path.join(os.path.dirname(local_files[key]), line[-1])
+  shutil.copy2(os.path.join(os.path.dirname(local_files[key]), 
line[-1]), workdir)
+  sccfile.close()
+

Would the patches included in these .scc files also need to be handled
in the same way? Would this also work if there are other scc files
included in a scc file?
Yes, I believe that the same mechanism should be used for patch files as 
well,

basically anything that may be needed to build but that its not necessarily
explicitly listed on SRC_URI.

I believe it will work for other scc files and it doesnt have to be 
recursive,
because while cfg files arent required to be in SRC_URI , scc files ARE 
required
to be SRC_URI, which means that this will eventually find the other scc 
file on the list


Cheers,

Alejandro



Thanks,

Anuj


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool-source.bbclass: Support kernel fragments not in SRC_URI

2018-08-01 Thread Anuj Mittal 
On 08/02/2018 08:27 AM, Jaewon Lee wrote:
> Hi Anuj,
> 
> Thanks for the reply
> That is a good point, will send a v2 that also deals with patches.
> Regarding other scc files,  according to the kernel dev manual, 
> "It is only necessary to specify the .scc files on the SRC_URI."
> 
> So from my understanding all scc files will be in SRC_URI which means they 
> get parsed by this fix, 
> Is this not the case?
If a scc file, foo.scc, includes any other scc file say bar.scc, then it
isn't mandatory to specify bar.scc as part of SRC_URI I think.

You can give it a try by creating empty sccs and including a malformed
patch in a nested scc to see if you get an error while patching.

Thanks,

Anuj
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool-source.bbclass: Support kernel fragments not in SRC_URI

2018-08-01 Thread Anuj Mittal 
On 07/31/2018 05:21 AM, Jaewon Lee wrote:
> When using a recipe space kernel-meta, scc files are added through
> SRC_URI, but they may include corresponding kernel fragments that are
> not necessarily in SRC_URI.
> 
> For bitbake, this is not a problem because the kernel-yocto class adds
> the path where the .scc file was found to includes which consequentially
> makes the .cfg file available to the kernel build.
> 
> However, when using devtool, only files specified in SRC_URI are copied
> to oe-local-files in devtool's workspace. So if the cfg file is not in
> SRC_URI, it won't be copied, causing a kernel build failure when trying
> to find it.
> 
> This fix parses local .scc files in SRC_URI, copies the corresponding .cfg
> file to devtool's workdir, and also adds it to local_files so it is
> available when doing a devtool build for the kernel.
> 
> [YOCTO #12858]
> 
> Signed-off-by: Jaewon Lee 
> Signed-off-by: Alejandro Enedino Hernandez Samaniego 
> ---
>  meta/classes/devtool-source.bbclass | 12 
>  1 file changed, 12 insertions(+)
> 
> diff --git a/meta/classes/devtool-source.bbclass 
> b/meta/classes/devtool-source.bbclass
> index 56882a4..c70fea2 100644
> --- a/meta/classes/devtool-source.bbclass
> +++ b/meta/classes/devtool-source.bbclass
> @@ -90,11 +90,23 @@ python devtool_post_unpack() {
>  fname in files])
>  return ret
>  
> +is_kernel_yocto = bb.data.inherits_class('kernel-yocto', d)
>  # Move local source files into separate subdir
>  recipe_patches = [os.path.basename(patch) for patch in
>  oe.recipeutils.get_recipe_patches(d)]
>  local_files = oe.recipeutils.get_recipe_local_files(d)
>  
> +if is_kernel_yocto:
> +  for key in local_files.copy():
> +if key.endswith('scc'):
> +  sccfile = open(local_files[key], 'r')
> +  for l in sccfile:
> +line = l.split()
> +if line and line[0] == 'kconf' and line[-1].endswith('.cfg'):
> +  local_files[line[-1]] = 
> os.path.join(os.path.dirname(local_files[key]), line[-1])
> +  shutil.copy2(os.path.join(os.path.dirname(local_files[key]), 
> line[-1]), workdir)
> +  sccfile.close()
> +

Would the patches included in these .scc files also need to be handled
in the same way? Would this also work if there are other scc files
included in a scc file?

Thanks,

Anuj
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] devtool-source.bbclass: Support kernel fragments not in SRC_URI

2018-07-30 Thread Jaewon Lee 
When using a recipe space kernel-meta, scc files are added through
SRC_URI, but they may include corresponding kernel fragments that are
not necessarily in SRC_URI.

For bitbake, this is not a problem because the kernel-yocto class adds
the path where the .scc file was found to includes which consequentially
makes the .cfg file available to the kernel build.

However, when using devtool, only files specified in SRC_URI are copied
to oe-local-files in devtool's workspace. So if the cfg file is not in
SRC_URI, it won't be copied, causing a kernel build failure when trying
to find it.

This fix parses local .scc files in SRC_URI, copies the corresponding .cfg
file to devtool's workdir, and also adds it to local_files so it is
available when doing a devtool build for the kernel.

[YOCTO #12858]

Signed-off-by: Jaewon Lee 
Signed-off-by: Alejandro Enedino Hernandez Samaniego 
---
 meta/classes/devtool-source.bbclass | 12 
 1 file changed, 12 insertions(+)

diff --git a/meta/classes/devtool-source.bbclass 
b/meta/classes/devtool-source.bbclass
index 56882a4..c70fea2 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -90,11 +90,23 @@ python devtool_post_unpack() {
 fname in files])
 return ret
 
+is_kernel_yocto = bb.data.inherits_class('kernel-yocto', d)
 # Move local source files into separate subdir
 recipe_patches = [os.path.basename(patch) for patch in
 oe.recipeutils.get_recipe_patches(d)]
 local_files = oe.recipeutils.get_recipe_local_files(d)
 
+if is_kernel_yocto:
+  for key in local_files.copy():
+if key.endswith('scc'):
+  sccfile = open(local_files[key], 'r')
+  for l in sccfile:
+line = l.split()
+if line and line[0] == 'kconf' and line[-1].endswith('.cfg'):
+  local_files[line[-1]] = 
os.path.join(os.path.dirname(local_files[key]), line[-1])
+  shutil.copy2(os.path.join(os.path.dirname(local_files[key]), 
line[-1]), workdir)
+  sccfile.close()
+
 # Ignore local files with subdir={BP}
 srcabspath = os.path.abspath(srcsubdir)
 local_files = [fname for fname in local_files if
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core