Re: [OE-core] [PATCH 3/4] kern-tools: fix processing for no branch meta-data

2016-12-08 Thread Bruce Ashfield 

On 2016-12-07 06:50 PM, Trevor Woerner wrote:

On Wed, Dec 7, 2016 at 1:05 PM, Trevor Woerner  wrote:

On Wed, Dec 7, 2016 at 11:18 AM, Bruce Ashfield
 wrote:

With the attached patch, I see nothing else that is named  in /tmp/

If you have the cycles, can you give it a try and let me know ?


Yes, I'm giving it a whirl right now. Thanks!


That patch looks good, how soon can it land? ;-)

I'm sure there's still something in the build that is creating
temporary files in /tmp. I'm not sure which process is doing it, I
have no reason to suspect it's the kernel tools (in fact I think it
happens too early in the build to be the kernel tools), and the tmp
files that are created are named with temporary names (e.g.
/tmp/tmp.2wavbhTlDU) so they shouldn't interfere with multiple users
building on the same machine. Each build appears to create two such
temp files that aren't cleaned up at the end of the build. Not a big
deal. I don't think it's something new that was introduced recently
(but I'd have to do more investigation to verify).


I did find one mktemp file that wasn't being cleaned up. I fixed
it and updated my commit to include that fix.

It'll all be out later today with my kernel version bumps + CVE
series.

Bruce





--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/4] kern-tools: fix processing for no branch meta-data

2016-12-07 Thread Bruce Ashfield 

On 2016-12-07 6:50 PM, Trevor Woerner wrote:

On Wed, Dec 7, 2016 at 1:05 PM, Trevor Woerner  wrote:

On Wed, Dec 7, 2016 at 11:18 AM, Bruce Ashfield
 wrote:

With the attached patch, I see nothing else that is named  in /tmp/

If you have the cycles, can you give it a try and let me know ?


Yes, I'm giving it a whirl right now. Thanks!


That patch looks good, how soon can it land? ;-)


For my part, I'll send it first thing tomorrow along with some version
bumps to the kernel.



I'm sure there's still something in the build that is creating
temporary files in /tmp. I'm not sure which process is doing it, I
have no reason to suspect it's the kernel tools (in fact I think it
happens too early in the build to be the kernel tools), and the tmp
files that are created are named with temporary names (e.g.
/tmp/tmp.2wavbhTlDU) so they shouldn't interfere with multiple users
building on the same machine. Each build appears to create two such
temp files that aren't cleaned up at the end of the build. Not a big
deal. I don't think it's something new that was introduced recently
(but I'd have to do more investigation to verify).


I noticed one mktemp file leaking when I was fixing the bug at hand
today. I made a note to loop back and have a look for where an early
exit is skipping clean up.

If it is the tools, I'll fix it along with the patch I did today.

Bruce





--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/4] kern-tools: fix processing for no branch meta-data

2016-12-07 Thread Trevor Woerner 
On Wed, Dec 7, 2016 at 1:05 PM, Trevor Woerner  wrote:
> On Wed, Dec 7, 2016 at 11:18 AM, Bruce Ashfield
>  wrote:
>> With the attached patch, I see nothing else that is named  in /tmp/
>>
>> If you have the cycles, can you give it a try and let me know ?
>
> Yes, I'm giving it a whirl right now. Thanks!

That patch looks good, how soon can it land? ;-)

I'm sure there's still something in the build that is creating
temporary files in /tmp. I'm not sure which process is doing it, I
have no reason to suspect it's the kernel tools (in fact I think it
happens too early in the build to be the kernel tools), and the tmp
files that are created are named with temporary names (e.g.
/tmp/tmp.2wavbhTlDU) so they shouldn't interfere with multiple users
building on the same machine. Each build appears to create two such
temp files that aren't cleaned up at the end of the build. Not a big
deal. I don't think it's something new that was introduced recently
(but I'd have to do more investigation to verify).
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/4] kern-tools: fix processing for no branch meta-data

2016-12-07 Thread Trevor Woerner 
On Wed, Dec 7, 2016 at 11:18 AM, Bruce Ashfield
 wrote:
> With the attached patch, I see nothing else that is named  in /tmp/
>
> If you have the cycles, can you give it a try and let me know ?

Yes, I'm giving it a whirl right now. Thanks!
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/4] kern-tools: fix processing for no branch meta-data

2016-12-07 Thread Bruce Ashfield 

On 2016-12-07 10:11 AM, Trevor Woerner wrote:

This patch does fix the raspberrypi issue, and it has been pushed, thankfully.

But I'm still seeing hard-coded filenames in /tmp that are messing up
the ability for more than one person to use a given build machine.
E.g.:

/tmp/patch.standard.arm-versatile-926ejs.queue
/tmp/patch.standard.mti-malta32.queue
/tmp/patch.standard.mti-malta64.queue
/tmp/patch.standard.preempt-rt.base.queue
/tmp/patch.standard.preempt-rt.queue
/tmp/patch.standard.queue
/tmp/patch.unused.queue

I think the malta ones are used for qemumips64, which is still failing
for this reason (multiple users)


I tracked this down to some of the sub commands using the dirname
of the global patch queue .. and since that is in /tmp/, they are
following along.

With the attached patch, I see nothing else that is named  in /tmp/

If you have the cycles, can you give it a try and let me know ?

Bruce



On Tue, Dec 6, 2016 at 6:02 PM, Paul Barker  wrote:

On Tue, 06 Dec 2016 22:14:41 +0100
Patrick Ohly  wrote:


On Fri, 2016-12-02 at 16:09 -0500, Bruce Ashfield wrote:

Lernel meta-data that has patches, but no branches, can trigger an
error due to no branch specific patch queue.

This error then cascades to more issues since the tools are using
a named file in /tmp to store and display error messages to the
user.

We fix both issues though the following kern tools tweaks:

  commit bd9e1d6c9b0a34ff3e19a06999aaf57ffadfd04c
  Author: Bruce Ashfield 
  Date:   Fri Dec 2 13:09:40 2016 -0500

scc: use mktemp for consolidated output capture

To provide useful error messages the tools dump pre-processed
files and messages to a temporary file. If multiple users are
doing builds, this means they either race, or can have permissions
issues.

By creating the temporary file via mktemp, we avoid both issues.
(We also make sure to clean these up on exit, or /tmp will get
polluted quickly).

  commit a287da4bfe0b4acb8f2b0627bd8e7abd1a1dde26
  Author: Bruce Ashfield 
  Date:   Fri Dec 2 13:08:08 2016 -0500

patch: do not assume a branch specific patch queue is needed

When processing input files per-branch and global patch queues are
generated. If the meta-data has not created any branches in the
repo, no branch specific queue is required.

The tools assumed that one is always valid, and hence would throw a
non-zero exit code and stop processing.

By testing for a named per-branch queue, we avoid this issue.


Ostro OS runs into the problem while trying to use current OE-core
master:

 .../patch.cmd: line 29: : No such file or directory

| ERROR: Function failed: do_kernel_metadata (log file is located ...)

This commit here fixed it for me. I see that it is already in Ross' mut2
branch, so hopefully that'll land in master soon.



Ditto for meta-raspberrypi, the kernel doesn't build with current
oe-core master as discussed on the yocto@ list. This patch is needed
to fix things.

Thanks,
Paul Barker
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


>From 707978ff47533c705eac42619988793f07a66b80 Mon Sep 17 00:00:00 2001
From: Bruce Ashfield 
Date: Wed, 7 Dec 2016 11:14:34 -0500
Subject: [PATCH] kern-tools: ensure that no shared directories are used

We need to avoid using shared/common directories for any files that are
part of specific build, since permissions issues in multi user
environments will cause issues.

Integrating the following commit to solve the issue:

   scc: move unused patch queue under output dir

Signed-off-by: Bruce Ashfield 
---
 meta/recipes-kernel/kern-tools/kern-tools-native_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb b/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
index 0f8a786176f9..3ca6ca22ef20 100644
--- a/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
+++ b/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
@@ -4,7 +4,7 @@ LIC_FILES_CHKSUM = "file://git/tools/kgit;beginline=5;endline=9;md5=a6c2fa8aef1b
 
 DEPENDS = "git-native"
 
-SRCREV = "bd9e1d6c9b0a34ff3e19a06999aaf57ffadfd04c"
+SRCREV = "626ceac135fa66277c2fa53197be33cc9d4d7614"
 PR = "r12"
 PV = "0.2+git${SRCPV}"
 
-- 
2.5.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/4] kern-tools: fix processing for no branch meta-data

2016-12-07 Thread Bruce Ashfield 

On 2016-12-07 10:11 AM, Trevor Woerner wrote:

This patch does fix the raspberrypi issue, and it has been pushed, thankfully.

But I'm still seeing hard-coded filenames in /tmp that are messing up
the ability for more than one person to use a given build machine.
E.g.:

/tmp/patch.standard.arm-versatile-926ejs.queue
/tmp/patch.standard.mti-malta32.queue
/tmp/patch.standard.mti-malta64.queue
/tmp/patch.standard.preempt-rt.base.queue
/tmp/patch.standard.preempt-rt.queue
/tmp/patch.standard.queue
/tmp/patch.unused.queue

I think the malta ones are used for qemumips64, which is still failing
for this reason (multiple users)


Those are never supposed to go to /tmp/, they are targeted to ${S}
of the kernel build.

It must be a side effect of the change I just did where the branch
isn't defined.

I'll whip up another patch.

Bruce



On Tue, Dec 6, 2016 at 6:02 PM, Paul Barker  wrote:

On Tue, 06 Dec 2016 22:14:41 +0100
Patrick Ohly  wrote:


On Fri, 2016-12-02 at 16:09 -0500, Bruce Ashfield wrote:

Lernel meta-data that has patches, but no branches, can trigger an
error due to no branch specific patch queue.

This error then cascades to more issues since the tools are using
a named file in /tmp to store and display error messages to the
user.

We fix both issues though the following kern tools tweaks:

  commit bd9e1d6c9b0a34ff3e19a06999aaf57ffadfd04c
  Author: Bruce Ashfield 
  Date:   Fri Dec 2 13:09:40 2016 -0500

scc: use mktemp for consolidated output capture

To provide useful error messages the tools dump pre-processed
files and messages to a temporary file. If multiple users are
doing builds, this means they either race, or can have permissions
issues.

By creating the temporary file via mktemp, we avoid both issues.
(We also make sure to clean these up on exit, or /tmp will get
polluted quickly).

  commit a287da4bfe0b4acb8f2b0627bd8e7abd1a1dde26
  Author: Bruce Ashfield 
  Date:   Fri Dec 2 13:08:08 2016 -0500

patch: do not assume a branch specific patch queue is needed

When processing input files per-branch and global patch queues are
generated. If the meta-data has not created any branches in the
repo, no branch specific queue is required.

The tools assumed that one is always valid, and hence would throw a
non-zero exit code and stop processing.

By testing for a named per-branch queue, we avoid this issue.


Ostro OS runs into the problem while trying to use current OE-core
master:

 .../patch.cmd: line 29: : No such file or directory

| ERROR: Function failed: do_kernel_metadata (log file is located ...)

This commit here fixed it for me. I see that it is already in Ross' mut2
branch, so hopefully that'll land in master soon.



Ditto for meta-raspberrypi, the kernel doesn't build with current
oe-core master as discussed on the yocto@ list. This patch is needed
to fix things.

Thanks,
Paul Barker
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/4] kern-tools: fix processing for no branch meta-data

2016-12-07 Thread Trevor Woerner 
This patch does fix the raspberrypi issue, and it has been pushed, thankfully.

But I'm still seeing hard-coded filenames in /tmp that are messing up
the ability for more than one person to use a given build machine.
E.g.:

/tmp/patch.standard.arm-versatile-926ejs.queue
/tmp/patch.standard.mti-malta32.queue
/tmp/patch.standard.mti-malta64.queue
/tmp/patch.standard.preempt-rt.base.queue
/tmp/patch.standard.preempt-rt.queue
/tmp/patch.standard.queue
/tmp/patch.unused.queue

I think the malta ones are used for qemumips64, which is still failing
for this reason (multiple users)

On Tue, Dec 6, 2016 at 6:02 PM, Paul Barker  wrote:
> On Tue, 06 Dec 2016 22:14:41 +0100
> Patrick Ohly  wrote:
>
>> On Fri, 2016-12-02 at 16:09 -0500, Bruce Ashfield wrote:
>> > Lernel meta-data that has patches, but no branches, can trigger an
>> > error due to no branch specific patch queue.
>> >
>> > This error then cascades to more issues since the tools are using
>> > a named file in /tmp to store and display error messages to the
>> > user.
>> >
>> > We fix both issues though the following kern tools tweaks:
>> >
>> >   commit bd9e1d6c9b0a34ff3e19a06999aaf57ffadfd04c
>> >   Author: Bruce Ashfield 
>> >   Date:   Fri Dec 2 13:09:40 2016 -0500
>> >
>> > scc: use mktemp for consolidated output capture
>> >
>> > To provide useful error messages the tools dump pre-processed
>> > files and messages to a temporary file. If multiple users are
>> > doing builds, this means they either race, or can have permissions
>> > issues.
>> >
>> > By creating the temporary file via mktemp, we avoid both issues.
>> > (We also make sure to clean these up on exit, or /tmp will get
>> > polluted quickly).
>> >
>> >   commit a287da4bfe0b4acb8f2b0627bd8e7abd1a1dde26
>> >   Author: Bruce Ashfield 
>> >   Date:   Fri Dec 2 13:08:08 2016 -0500
>> >
>> > patch: do not assume a branch specific patch queue is needed
>> >
>> > When processing input files per-branch and global patch queues are
>> > generated. If the meta-data has not created any branches in the
>> > repo, no branch specific queue is required.
>> >
>> > The tools assumed that one is always valid, and hence would throw a
>> > non-zero exit code and stop processing.
>> >
>> > By testing for a named per-branch queue, we avoid this issue.
>>
>> Ostro OS runs into the problem while trying to use current OE-core
>> master:
>>
>>  .../patch.cmd: line 29: : No such file or directory
>>
>> | ERROR: Function failed: do_kernel_metadata (log file is located ...)
>>
>> This commit here fixed it for me. I see that it is already in Ross' mut2
>> branch, so hopefully that'll land in master soon.
>>
>
> Ditto for meta-raspberrypi, the kernel doesn't build with current
> oe-core master as discussed on the yocto@ list. This patch is needed
> to fix things.
>
> Thanks,
> Paul Barker
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/4] kern-tools: fix processing for no branch meta-data

2016-12-06 Thread Paul Barker 
On Tue, 06 Dec 2016 22:14:41 +0100
Patrick Ohly  wrote:

> On Fri, 2016-12-02 at 16:09 -0500, Bruce Ashfield wrote:
> > Lernel meta-data that has patches, but no branches, can trigger an
> > error due to no branch specific patch queue.
> > 
> > This error then cascades to more issues since the tools are using
> > a named file in /tmp to store and display error messages to the
> > user.
> > 
> > We fix both issues though the following kern tools tweaks:
> > 
> >   commit bd9e1d6c9b0a34ff3e19a06999aaf57ffadfd04c
> >   Author: Bruce Ashfield 
> >   Date:   Fri Dec 2 13:09:40 2016 -0500
> > 
> > scc: use mktemp for consolidated output capture
> > 
> > To provide useful error messages the tools dump pre-processed
> > files and messages to a temporary file. If multiple users are
> > doing builds, this means they either race, or can have permissions
> > issues.
> > 
> > By creating the temporary file via mktemp, we avoid both issues.
> > (We also make sure to clean these up on exit, or /tmp will get
> > polluted quickly).
> > 
> >   commit a287da4bfe0b4acb8f2b0627bd8e7abd1a1dde26
> >   Author: Bruce Ashfield 
> >   Date:   Fri Dec 2 13:08:08 2016 -0500
> > 
> > patch: do not assume a branch specific patch queue is needed
> > 
> > When processing input files per-branch and global patch queues are
> > generated. If the meta-data has not created any branches in the
> > repo, no branch specific queue is required.
> > 
> > The tools assumed that one is always valid, and hence would throw a
> > non-zero exit code and stop processing.
> > 
> > By testing for a named per-branch queue, we avoid this issue.
> 
> Ostro OS runs into the problem while trying to use current OE-core
> master:
> 
>  .../patch.cmd: line 29: : No such file or directory
> 
> | ERROR: Function failed: do_kernel_metadata (log file is located ...)
> 
> This commit here fixed it for me. I see that it is already in Ross' mut2
> branch, so hopefully that'll land in master soon.
> 

Ditto for meta-raspberrypi, the kernel doesn't build with current
oe-core master as discussed on the yocto@ list. This patch is needed
to fix things.

Thanks,
Paul Barker
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/4] kern-tools: fix processing for no branch meta-data

2016-12-06 Thread Patrick Ohly 
On Fri, 2016-12-02 at 16:09 -0500, Bruce Ashfield wrote:
> Lernel meta-data that has patches, but no branches, can trigger an
> error due to no branch specific patch queue.
> 
> This error then cascades to more issues since the tools are using
> a named file in /tmp to store and display error messages to the
> user.
> 
> We fix both issues though the following kern tools tweaks:
> 
>   commit bd9e1d6c9b0a34ff3e19a06999aaf57ffadfd04c
>   Author: Bruce Ashfield 
>   Date:   Fri Dec 2 13:09:40 2016 -0500
> 
> scc: use mktemp for consolidated output capture
> 
> To provide useful error messages the tools dump pre-processed
> files and messages to a temporary file. If multiple users are
> doing builds, this means they either race, or can have permissions
> issues.
> 
> By creating the temporary file via mktemp, we avoid both issues.
> (We also make sure to clean these up on exit, or /tmp will get
> polluted quickly).
> 
>   commit a287da4bfe0b4acb8f2b0627bd8e7abd1a1dde26
>   Author: Bruce Ashfield 
>   Date:   Fri Dec 2 13:08:08 2016 -0500
> 
> patch: do not assume a branch specific patch queue is needed
> 
> When processing input files per-branch and global patch queues are
> generated. If the meta-data has not created any branches in the
> repo, no branch specific queue is required.
> 
> The tools assumed that one is always valid, and hence would throw a
> non-zero exit code and stop processing.
> 
> By testing for a named per-branch queue, we avoid this issue.

Ostro OS runs into the problem while trying to use current OE-core
master:

 .../patch.cmd: line 29: : No such file or directory

| ERROR: Function failed: do_kernel_metadata (log file is located ...)

This commit here fixed it for me. I see that it is already in Ross' mut2
branch, so hopefully that'll land in master soon.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.



-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core