[gem5-dev] Change in public/gem5[master]: scons: When spawning the linker process, don't involve the s...

[Gabe Black (Gerrit)]

Gabe Black has submitted this change and it was merged. (  
https://gem5-review.googlesource.com/2780 )


Change subject: scons: When spawning the linker process, don't involve the  
shell.

..

scons: When spawning the linker process, don't involve the shell.

The command line can be too long, causing bash to choke. This means we can't
use any shell syntax like shell variables or redirection when linking, but
that should be easy to avoid.

Change-Id: Ie6c8ecab337cef6bd3c7e403346ced06f46f0993
Reviewed-on: https://gem5-review.googlesource.com/2780
Reviewed-by: Jason Lowe-Power 
Reviewed-by: Steve Reinhardt 
Maintainer: Jason Lowe-Power 
---
M src/SConscript
1 file changed, 13 insertions(+), 1 deletion(-)

Approvals:
  Jason Lowe-Power: Looks good to me, approved; Looks good to me, approved
  Steve Reinhardt: Looks good to me, but someone else must approve



diff --git a/src/SConscript b/src/SConscript
index 521d73b..645b925 100755
--- a/src/SConscript
+++ b/src/SConscript
@@ -34,6 +34,7 @@
 import marshal
 import os
 import re
+import subprocess
 import sys
 import zlib

@@ -1194,7 +1195,18 @@
 if strip:
 progname += '.unstripped'

-targets = new_env.Program(progname, main_objs + static_objs)
+# When linking the gem5 binary, the command line can be too big for the
+# shell to handle. Use "subprocess" to spawn processes without passing
+# through the shell to avoid this problem. That means we also can't use
+# shell syntax in any of the commands this will run, but that isn't
+# currently an issue.
+def spawn_with_subprocess(sh, escape, cmd, args, env):
+return subprocess.call(args, env=env)
+
+# Since we're not running through a shell, no escaping is necessary  
either.

+targets = new_env.Program(progname, main_objs + static_objs,
+  SPAWN=spawn_with_subprocess,
+  ESCAPE=lambda x: x)

 if strip:
 if sys.platform == 'sunos5':

--
To view, visit https://gem5-review.googlesource.com/2780
To unsubscribe, visit https://gem5-review.googlesource.com/settings

Gerrit-Project: public/gem5
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie6c8ecab337cef6bd3c7e403346ced06f46f0993
Gerrit-Change-Number: 2780
Gerrit-PatchSet: 3
Gerrit-Owner: Gabe Black 
Gerrit-Reviewer: Gabe Black 
Gerrit-Reviewer: Jason Lowe-Power 
Gerrit-Reviewer: Steve Reinhardt 
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

[gem5-dev] Change in public/gem5[master]: scons: When spawning the linker process, don't involve the s...

[Gabe Black (Gerrit)]

Hello Steve Reinhardt,

I'd like you to reexamine a change. Please visit

https://gem5-review.googlesource.com/2780

to look at the new patch set (#2).

Change subject: scons: When spawning the linker process, don't involve the  
shell.

..

scons: When spawning the linker process, don't involve the shell.

The command line can be too long, causing bash to choke. This means we can't
use any shell syntax like shell variables or redirection when linking, but
that should be easy to avoid.

Change-Id: Ie6c8ecab337cef6bd3c7e403346ced06f46f0993
---
M src/SConscript
1 file changed, 13 insertions(+), 1 deletion(-)


--
To view, visit https://gem5-review.googlesource.com/2780
To unsubscribe, visit https://gem5-review.googlesource.com/settings

Gerrit-Project: public/gem5
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie6c8ecab337cef6bd3c7e403346ced06f46f0993
Gerrit-Change-Number: 2780
Gerrit-PatchSet: 2
Gerrit-Owner: Gabe Black 
Gerrit-Reviewer: Gabe Black 
Gerrit-Reviewer: Steve Reinhardt 
Gerrit-CC: Jason Lowe-Power 
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

Re: [gem5-dev] scons question

[Gabe Black]

In the meantime, this should circumvent command line problems during
linking:

https://gem5-review.googlesource.com/#/c/2780/

Gabe

On Thu, Apr 13, 2017 at 1:58 PM, Gabe Black  wrote:

> For reference, here's somebody also trying to figure out how to get '-r'
> working in scons from 9 years ago.
>
> http://scons-users.scons.narkive.com/OJRUXpKC/partial-linking-with-scons
>
> I don't think they got a great answer, but it's at least something.
>
> Gabe
>
> On Thu, Apr 13, 2017 at 1:52 PM, Gabe Black  wrote:
>
>> Yeah, I think the problem could have been using .a files. There's a thing
>> where the linker doesn't go back and look at stuff in an archive it's
>> already processed if it needs to, and will just error out due to unresolved
>> symbols. You have to solve that by putting the .as on the command line
>> multiple times so they'll be reprocessed, and if there are lots of
>> interdepedencies that can be difficult. I don't know if scons has built in
>> support for something like -r, but I don't think that has the same problem.
>>
>> Gabe
>>
>> On Thu, Apr 13, 2017 at 1:47 PM, Steve Reinhardt 
>> wrote:
>>
>>> I don't exactly remember why I thought it was hard... I probably only
>>> spent
>>> an hour or so looking at it before I realized it would take more time
>>> than
>>> I had (which wasn't much). Maybe I was overly pessimistic, I don't know.
>>> I
>>> was thinking of trying to do something implicit, e.g., if you could just
>>> hack SourceFile to add objects to a per-"module" list (base, sim, dev,
>>> etc.) instead of a global list then you wouldn't have to touch any of the
>>> other SConscript files.
>>>
>>> As far as the dependencies that Jason points out, I don't think they
>>> should
>>> be that problematic. It's not that we want to compile everything in
>>> 'base'
>>> using only the code in 'base'; we can compile the objects the same way
>>> we've always compiled the objects. The only difference is that we want to
>>> inject an intermediate linking step. We would still have to do all the
>>> code
>>> generation (ISA headers, debug headers, etc.) up front.
>>>
>>> Moving to the point where some of these intermediate objects/libraries
>>> are
>>> ISA-independent would be awesome, but I see that as a separate
>>> step---seems
>>> like too much to bite off to do that at the same time.
>>>
>>> Steve
>>>
>>>
>>> On Thu, Apr 13, 2017 at 1:38 PM, Gabe Black 
>>> wrote:
>>>
>>> > Yeah, I'm not surprised about how interlinked everything is. I looked
>>> into
>>> > what it would take to build multiple ISAs in at the same time a long
>>> time
>>> > ago, and it would have been a huge undertaking for the same reason.
>>> >
>>> > I think there are (at least) two different ways to do an incremental
>>> link.
>>> > First, we could create .a files with different groups of .o files, and
>>> then
>>> > link them together at the end. That might even be supported by scons,
>>> but
>>> > then ordering things correctly is a pain. This might be what you were
>>> > having trouble with.
>>> >
>>> > Another option is to use the -r or --relocatable flag on the linker
>>> which
>>> > makes it generate a file which can be used as an input to a subsequent
>>> run
>>> > of the linker. That could be used to gather up clusters of .o files and
>>> > generate another .o, and I think would alleviate the ordering issue.
>>> >
>>> > I was also thinking of doing something simple, like grouping things per
>>> > SConscript processed or something like that. That wouldn't be perfect,
>>> but
>>> > it would generally isolate things into relatively small clumps which
>>> would
>>> > get the job done. Explicitly grouping things would be give you better
>>> > groups I think, but it would be more cumbersome to manage.
>>> >
>>> > Gabe
>>> >
>>> > On Thu, Apr 13, 2017 at 1:20 PM, Jason Lowe-Power >> >
>>> > wrote:
>>> >
>>> > > Hi Gabe,
>>> > >
>>> > > Just today I was looking at incremental linking and breaking gem5
>>> into
>>> > > "components". I was specifically looking at it as a stepping stone
>>> to use
>>> > > Bazel (https://bazel.build/) as the build system instead of Scons.
>>> On a
>>> > > side note, it looks like Bazel has a lot of features that we could
>>> take
>>> > > advantage of. There are a couple of downsides, but that's a
>>> conversation
>>> > > for another time.
>>> > >
>>> > > What makes it hard to break gem5 into components is that everything
>>> is
>>> > very
>>> > > tightly coupled. For instance, I was trying to compile just "base"
>>> > thinking
>>> > > it was the simplest. But it depends on debug (which is everywhere,
>>> and
>>> > > nowhere), serializable (which depends on stuff in sim/), the event
>>> queue
>>> > > logic, and some stuff in ext/! Similarly, almost everything somehow
>>> > depends
>>> > > on the ISA. There are also a number of circular dependencies. It's
>>> far
>>> > from
>>> > > straightforward to 

[gem5-dev] Change in public/gem5[master]: scons: When spawning the linker process, don't involve the s...

[Gabe Black (Gerrit)]

Gabe Black has uploaded this change for review. (  
https://gem5-review.googlesource.com/2780



Change subject: scons: When spawning the linker process, don't involve the  
shell.

..

scons: When spawning the linker process, don't involve the shell.

The command line can be too long, causing bash to choke. This means we can't
use any shell syntax like shell variables or redirection when linking, but
that should be easy to avoid.

Change-Id: Ie6c8ecab337cef6bd3c7e403346ced06f46f0993
---
M src/SConscript
1 file changed, 13 insertions(+), 1 deletion(-)



diff --git a/src/SConscript b/src/SConscript
index 521d73b..2bb2956 100755
--- a/src/SConscript
+++ b/src/SConscript
@@ -34,6 +34,7 @@
 import marshal
 import os
 import re
+import subprocess
 import sys
 import zlib

@@ -1194,7 +1195,18 @@
 if strip:
 progname += '.unstripped'

-targets = new_env.Program(progname, main_objs + static_objs)
+# When linking the gem5 binary, the command line can be too big for the
+# shell to handle. Use "subprocess" to spawn processes without passing
+# through the shell to avoid this problem. That means we also can't use
+# shell syntax in any of the commands this will run, but that isn't
+# currently an issue.
+def spawn_with_subprocess(sh, escape, cmd, args, env):
+return subprocess.Popen(args, env=env).wait()
+
+# Since we're not running through a shell, no escaping is necessary  
either.

+targets = new_env.Program(progname, main_objs + static_objs,
+  SPAWN=spawn_with_subprocess,
+  ESCAPE=lambda x: x)

 if strip:
 if sys.platform == 'sunos5':

--
To view, visit https://gem5-review.googlesource.com/2780
To unsubscribe, visit https://gem5-review.googlesource.com/settings

Gerrit-Project: public/gem5
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie6c8ecab337cef6bd3c7e403346ced06f46f0993
Gerrit-Change-Number: 2780
Gerrit-PatchSet: 1
Gerrit-Owner: Gabe Black 
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

Re: [gem5-dev] scons question

[Gabe Black]

For reference, here's somebody also trying to figure out how to get '-r'
working in scons from 9 years ago.

http://scons-users.scons.narkive.com/OJRUXpKC/partial-linking-with-scons

I don't think they got a great answer, but it's at least something.

Gabe

On Thu, Apr 13, 2017 at 1:52 PM, Gabe Black  wrote:

> Yeah, I think the problem could have been using .a files. There's a thing
> where the linker doesn't go back and look at stuff in an archive it's
> already processed if it needs to, and will just error out due to unresolved
> symbols. You have to solve that by putting the .as on the command line
> multiple times so they'll be reprocessed, and if there are lots of
> interdepedencies that can be difficult. I don't know if scons has built in
> support for something like -r, but I don't think that has the same problem.
>
> Gabe
>
> On Thu, Apr 13, 2017 at 1:47 PM, Steve Reinhardt  wrote:
>
>> I don't exactly remember why I thought it was hard... I probably only
>> spent
>> an hour or so looking at it before I realized it would take more time than
>> I had (which wasn't much). Maybe I was overly pessimistic, I don't know. I
>> was thinking of trying to do something implicit, e.g., if you could just
>> hack SourceFile to add objects to a per-"module" list (base, sim, dev,
>> etc.) instead of a global list then you wouldn't have to touch any of the
>> other SConscript files.
>>
>> As far as the dependencies that Jason points out, I don't think they
>> should
>> be that problematic. It's not that we want to compile everything in 'base'
>> using only the code in 'base'; we can compile the objects the same way
>> we've always compiled the objects. The only difference is that we want to
>> inject an intermediate linking step. We would still have to do all the
>> code
>> generation (ISA headers, debug headers, etc.) up front.
>>
>> Moving to the point where some of these intermediate objects/libraries are
>> ISA-independent would be awesome, but I see that as a separate
>> step---seems
>> like too much to bite off to do that at the same time.
>>
>> Steve
>>
>>
>> On Thu, Apr 13, 2017 at 1:38 PM, Gabe Black  wrote:
>>
>> > Yeah, I'm not surprised about how interlinked everything is. I looked
>> into
>> > what it would take to build multiple ISAs in at the same time a long
>> time
>> > ago, and it would have been a huge undertaking for the same reason.
>> >
>> > I think there are (at least) two different ways to do an incremental
>> link.
>> > First, we could create .a files with different groups of .o files, and
>> then
>> > link them together at the end. That might even be supported by scons,
>> but
>> > then ordering things correctly is a pain. This might be what you were
>> > having trouble with.
>> >
>> > Another option is to use the -r or --relocatable flag on the linker
>> which
>> > makes it generate a file which can be used as an input to a subsequent
>> run
>> > of the linker. That could be used to gather up clusters of .o files and
>> > generate another .o, and I think would alleviate the ordering issue.
>> >
>> > I was also thinking of doing something simple, like grouping things per
>> > SConscript processed or something like that. That wouldn't be perfect,
>> but
>> > it would generally isolate things into relatively small clumps which
>> would
>> > get the job done. Explicitly grouping things would be give you better
>> > groups I think, but it would be more cumbersome to manage.
>> >
>> > Gabe
>> >
>> > On Thu, Apr 13, 2017 at 1:20 PM, Jason Lowe-Power 
>> > wrote:
>> >
>> > > Hi Gabe,
>> > >
>> > > Just today I was looking at incremental linking and breaking gem5 into
>> > > "components". I was specifically looking at it as a stepping stone to
>> use
>> > > Bazel (https://bazel.build/) as the build system instead of Scons.
>> On a
>> > > side note, it looks like Bazel has a lot of features that we could
>> take
>> > > advantage of. There are a couple of downsides, but that's a
>> conversation
>> > > for another time.
>> > >
>> > > What makes it hard to break gem5 into components is that everything is
>> > very
>> > > tightly coupled. For instance, I was trying to compile just "base"
>> > thinking
>> > > it was the simplest. But it depends on debug (which is everywhere, and
>> > > nowhere), serializable (which depends on stuff in sim/), the event
>> queue
>> > > logic, and some stuff in ext/! Similarly, almost everything somehow
>> > depends
>> > > on the ISA. There are also a number of circular dependencies. It's far
>> > from
>> > > straightforward to get any one directory to compile without all of the
>> > > others.
>> > >
>> > > Overall, I think what we would need to do to make this happen is
>> > completely
>> > > reorganize gem5's source. I think this would be good in the long run,
>> but
>> > > it would take me at least 2-4 weeks. Maybe I'm slower than others,
>> > though.
>> > > There is a huge downside to 

Re: [gem5-dev] scons question

[Gabe Black]

Yeah, I think the problem could have been using .a files. There's a thing
where the linker doesn't go back and look at stuff in an archive it's
already processed if it needs to, and will just error out due to unresolved
symbols. You have to solve that by putting the .as on the command line
multiple times so they'll be reprocessed, and if there are lots of
interdepedencies that can be difficult. I don't know if scons has built in
support for something like -r, but I don't think that has the same problem.

Gabe

On Thu, Apr 13, 2017 at 1:47 PM, Steve Reinhardt  wrote:

> I don't exactly remember why I thought it was hard... I probably only spent
> an hour or so looking at it before I realized it would take more time than
> I had (which wasn't much). Maybe I was overly pessimistic, I don't know. I
> was thinking of trying to do something implicit, e.g., if you could just
> hack SourceFile to add objects to a per-"module" list (base, sim, dev,
> etc.) instead of a global list then you wouldn't have to touch any of the
> other SConscript files.
>
> As far as the dependencies that Jason points out, I don't think they should
> be that problematic. It's not that we want to compile everything in 'base'
> using only the code in 'base'; we can compile the objects the same way
> we've always compiled the objects. The only difference is that we want to
> inject an intermediate linking step. We would still have to do all the code
> generation (ISA headers, debug headers, etc.) up front.
>
> Moving to the point where some of these intermediate objects/libraries are
> ISA-independent would be awesome, but I see that as a separate step---seems
> like too much to bite off to do that at the same time.
>
> Steve
>
>
> On Thu, Apr 13, 2017 at 1:38 PM, Gabe Black  wrote:
>
> > Yeah, I'm not surprised about how interlinked everything is. I looked
> into
> > what it would take to build multiple ISAs in at the same time a long time
> > ago, and it would have been a huge undertaking for the same reason.
> >
> > I think there are (at least) two different ways to do an incremental
> link.
> > First, we could create .a files with different groups of .o files, and
> then
> > link them together at the end. That might even be supported by scons, but
> > then ordering things correctly is a pain. This might be what you were
> > having trouble with.
> >
> > Another option is to use the -r or --relocatable flag on the linker which
> > makes it generate a file which can be used as an input to a subsequent
> run
> > of the linker. That could be used to gather up clusters of .o files and
> > generate another .o, and I think would alleviate the ordering issue.
> >
> > I was also thinking of doing something simple, like grouping things per
> > SConscript processed or something like that. That wouldn't be perfect,
> but
> > it would generally isolate things into relatively small clumps which
> would
> > get the job done. Explicitly grouping things would be give you better
> > groups I think, but it would be more cumbersome to manage.
> >
> > Gabe
> >
> > On Thu, Apr 13, 2017 at 1:20 PM, Jason Lowe-Power 
> > wrote:
> >
> > > Hi Gabe,
> > >
> > > Just today I was looking at incremental linking and breaking gem5 into
> > > "components". I was specifically looking at it as a stepping stone to
> use
> > > Bazel (https://bazel.build/) as the build system instead of Scons. On
> a
> > > side note, it looks like Bazel has a lot of features that we could take
> > > advantage of. There are a couple of downsides, but that's a
> conversation
> > > for another time.
> > >
> > > What makes it hard to break gem5 into components is that everything is
> > very
> > > tightly coupled. For instance, I was trying to compile just "base"
> > thinking
> > > it was the simplest. But it depends on debug (which is everywhere, and
> > > nowhere), serializable (which depends on stuff in sim/), the event
> queue
> > > logic, and some stuff in ext/! Similarly, almost everything somehow
> > depends
> > > on the ISA. There are also a number of circular dependencies. It's far
> > from
> > > straightforward to get any one directory to compile without all of the
> > > others.
> > >
> > > Overall, I think what we would need to do to make this happen is
> > completely
> > > reorganize gem5's source. I think this would be good in the long run,
> but
> > > it would take me at least 2-4 weeks. Maybe I'm slower than others,
> > though.
> > > There is a huge downside to this, too. All internal patches would be
> very
> > > hard to apply after making a change like this.
> > >
> > > Just a few thoughts.
> > >
> > > Cheers,
> > > Jason
> > >
> > > PS: For error 127, I would just do something hacky, personally. In the
> > long
> > > run, we should do something to make gem5 more modular and move away
> from
> > > SCons. But I don't think it's worth it just for this one issue.
> > >
> > > On Thu, Apr 13, 2017 at 3:10 PM Gabe Black 

Re: [gem5-dev] scons question

[Steve Reinhardt]

I don't exactly remember why I thought it was hard... I probably only spent
an hour or so looking at it before I realized it would take more time than
I had (which wasn't much). Maybe I was overly pessimistic, I don't know. I
was thinking of trying to do something implicit, e.g., if you could just
hack SourceFile to add objects to a per-"module" list (base, sim, dev,
etc.) instead of a global list then you wouldn't have to touch any of the
other SConscript files.

As far as the dependencies that Jason points out, I don't think they should
be that problematic. It's not that we want to compile everything in 'base'
using only the code in 'base'; we can compile the objects the same way
we've always compiled the objects. The only difference is that we want to
inject an intermediate linking step. We would still have to do all the code
generation (ISA headers, debug headers, etc.) up front.

Moving to the point where some of these intermediate objects/libraries are
ISA-independent would be awesome, but I see that as a separate step---seems
like too much to bite off to do that at the same time.

Steve


On Thu, Apr 13, 2017 at 1:38 PM, Gabe Black  wrote:

> Yeah, I'm not surprised about how interlinked everything is. I looked into
> what it would take to build multiple ISAs in at the same time a long time
> ago, and it would have been a huge undertaking for the same reason.
>
> I think there are (at least) two different ways to do an incremental link.
> First, we could create .a files with different groups of .o files, and then
> link them together at the end. That might even be supported by scons, but
> then ordering things correctly is a pain. This might be what you were
> having trouble with.
>
> Another option is to use the -r or --relocatable flag on the linker which
> makes it generate a file which can be used as an input to a subsequent run
> of the linker. That could be used to gather up clusters of .o files and
> generate another .o, and I think would alleviate the ordering issue.
>
> I was also thinking of doing something simple, like grouping things per
> SConscript processed or something like that. That wouldn't be perfect, but
> it would generally isolate things into relatively small clumps which would
> get the job done. Explicitly grouping things would be give you better
> groups I think, but it would be more cumbersome to manage.
>
> Gabe
>
> On Thu, Apr 13, 2017 at 1:20 PM, Jason Lowe-Power 
> wrote:
>
> > Hi Gabe,
> >
> > Just today I was looking at incremental linking and breaking gem5 into
> > "components". I was specifically looking at it as a stepping stone to use
> > Bazel (https://bazel.build/) as the build system instead of Scons. On a
> > side note, it looks like Bazel has a lot of features that we could take
> > advantage of. There are a couple of downsides, but that's a conversation
> > for another time.
> >
> > What makes it hard to break gem5 into components is that everything is
> very
> > tightly coupled. For instance, I was trying to compile just "base"
> thinking
> > it was the simplest. But it depends on debug (which is everywhere, and
> > nowhere), serializable (which depends on stuff in sim/), the event queue
> > logic, and some stuff in ext/! Similarly, almost everything somehow
> depends
> > on the ISA. There are also a number of circular dependencies. It's far
> from
> > straightforward to get any one directory to compile without all of the
> > others.
> >
> > Overall, I think what we would need to do to make this happen is
> completely
> > reorganize gem5's source. I think this would be good in the long run, but
> > it would take me at least 2-4 weeks. Maybe I'm slower than others,
> though.
> > There is a huge downside to this, too. All internal patches would be very
> > hard to apply after making a change like this.
> >
> > Just a few thoughts.
> >
> > Cheers,
> > Jason
> >
> > PS: For error 127, I would just do something hacky, personally. In the
> long
> > run, we should do something to make gem5 more modular and move away from
> > SCons. But I don't think it's worth it just for this one issue.
> >
> > On Thu, Apr 13, 2017 at 3:10 PM Gabe Black  wrote:
> >
> > > Ok, I think we're all pretty much in agreement then, since I was
> thinking
> > > about incremental linking too. What about it looked non-trivial,
> Steve? I
> > > haven't dug into it very much yet, but as long as it doesn't turn into
> > too
> > > much of an ordeal I'd like to take care of it.
> > >
> > > Potentially less ideal solutions which might also be easier would be to
> > > override the env['SPAWN'] setting so that it would skip going through
> the
> > > shell if it could, and seeing if we could load some things that are
> > > currently on the command line from files of options/input files.
> > >
> > > I think the reason scons is purposefully going through the shell is to
> > make
> > > things like redirection, etc., work. We take 

Re: [gem5-dev] scons question

[Gabe Black]

Yeah, I'm not surprised about how interlinked everything is. I looked into
what it would take to build multiple ISAs in at the same time a long time
ago, and it would have been a huge undertaking for the same reason.

I think there are (at least) two different ways to do an incremental link.
First, we could create .a files with different groups of .o files, and then
link them together at the end. That might even be supported by scons, but
then ordering things correctly is a pain. This might be what you were
having trouble with.

Another option is to use the -r or --relocatable flag on the linker which
makes it generate a file which can be used as an input to a subsequent run
of the linker. That could be used to gather up clusters of .o files and
generate another .o, and I think would alleviate the ordering issue.

I was also thinking of doing something simple, like grouping things per
SConscript processed or something like that. That wouldn't be perfect, but
it would generally isolate things into relatively small clumps which would
get the job done. Explicitly grouping things would be give you better
groups I think, but it would be more cumbersome to manage.

Gabe

On Thu, Apr 13, 2017 at 1:20 PM, Jason Lowe-Power 
wrote:

> Hi Gabe,
>
> Just today I was looking at incremental linking and breaking gem5 into
> "components". I was specifically looking at it as a stepping stone to use
> Bazel (https://bazel.build/) as the build system instead of Scons. On a
> side note, it looks like Bazel has a lot of features that we could take
> advantage of. There are a couple of downsides, but that's a conversation
> for another time.
>
> What makes it hard to break gem5 into components is that everything is very
> tightly coupled. For instance, I was trying to compile just "base" thinking
> it was the simplest. But it depends on debug (which is everywhere, and
> nowhere), serializable (which depends on stuff in sim/), the event queue
> logic, and some stuff in ext/! Similarly, almost everything somehow depends
> on the ISA. There are also a number of circular dependencies. It's far from
> straightforward to get any one directory to compile without all of the
> others.
>
> Overall, I think what we would need to do to make this happen is completely
> reorganize gem5's source. I think this would be good in the long run, but
> it would take me at least 2-4 weeks. Maybe I'm slower than others, though.
> There is a huge downside to this, too. All internal patches would be very
> hard to apply after making a change like this.
>
> Just a few thoughts.
>
> Cheers,
> Jason
>
> PS: For error 127, I would just do something hacky, personally. In the long
> run, we should do something to make gem5 more modular and move away from
> SCons. But I don't think it's worth it just for this one issue.
>
> On Thu, Apr 13, 2017 at 3:10 PM Gabe Black  wrote:
>
> > Ok, I think we're all pretty much in agreement then, since I was thinking
> > about incremental linking too. What about it looked non-trivial, Steve? I
> > haven't dug into it very much yet, but as long as it doesn't turn into
> too
> > much of an ordeal I'd like to take care of it.
> >
> > Potentially less ideal solutions which might also be easier would be to
> > override the env['SPAWN'] setting so that it would skip going through the
> > shell if it could, and seeing if we could load some things that are
> > currently on the command line from files of options/input files.
> >
> > I think the reason scons is purposefully going through the shell is to
> make
> > things like redirection, etc., work. We take advantage of that in at
> least
> > one place I've seen, but it might be feasible to remove that and
> specialize
> > env['SPAWN'] in a way that breaks redirection but allows arbitrarily long
> > command lines.
> >
> > But I still think incremental linking would be nicer for a number of
> other
> > reasons, as long as it's implementable.
> >
> > Gabe
> >
> > On Thu, Apr 13, 2017 at 10:16 AM, Steve Reinhardt 
> > wrote:
> >
> > > #3 is the traditional solution :)
> > >
> > > On Thu, Apr 13, 2017 at 10:13 AM, Gutierrez, Anthony <
> > > anthony.gutier...@amd.com> wrote:
> > >
> > > > There are a three ways to fix this as far as I can tell:
> > > >
> > > > 1) Modify our Scons setup to use staged linking.
> > > > 2) Recompile your kernel to allow for larger ARG_MAX.
> > > > 3) Modify your paths etc to avoid long names
> > > >
> > > > 1) seems to be the best option, but seems like it could be a lot of
> > work.
> > > >
> > > > -Original Message-
> > > > From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
> > > Black
> > > > Sent: Thursday, April 13, 2017 9:50 AM
> > > > To: gem5 Developer List 
> > > > Subject: Re: [gem5-dev] scons question
> > > >
> > > > Oh, I bet you're right. They actually spawn something like 'sh',
> '-c',
> > '
> > > > '.join(args), and I bet sh (which is 

Re: [gem5-dev] scons question

[Jason Lowe-Power]

Hi Gabe,

Just today I was looking at incremental linking and breaking gem5 into
"components". I was specifically looking at it as a stepping stone to use
Bazel (https://bazel.build/) as the build system instead of Scons. On a
side note, it looks like Bazel has a lot of features that we could take
advantage of. There are a couple of downsides, but that's a conversation
for another time.

What makes it hard to break gem5 into components is that everything is very
tightly coupled. For instance, I was trying to compile just "base" thinking
it was the simplest. But it depends on debug (which is everywhere, and
nowhere), serializable (which depends on stuff in sim/), the event queue
logic, and some stuff in ext/! Similarly, almost everything somehow depends
on the ISA. There are also a number of circular dependencies. It's far from
straightforward to get any one directory to compile without all of the
others.

Overall, I think what we would need to do to make this happen is completely
reorganize gem5's source. I think this would be good in the long run, but
it would take me at least 2-4 weeks. Maybe I'm slower than others, though.
There is a huge downside to this, too. All internal patches would be very
hard to apply after making a change like this.

Just a few thoughts.

Cheers,
Jason

PS: For error 127, I would just do something hacky, personally. In the long
run, we should do something to make gem5 more modular and move away from
SCons. But I don't think it's worth it just for this one issue.

On Thu, Apr 13, 2017 at 3:10 PM Gabe Black  wrote:

> Ok, I think we're all pretty much in agreement then, since I was thinking
> about incremental linking too. What about it looked non-trivial, Steve? I
> haven't dug into it very much yet, but as long as it doesn't turn into too
> much of an ordeal I'd like to take care of it.
>
> Potentially less ideal solutions which might also be easier would be to
> override the env['SPAWN'] setting so that it would skip going through the
> shell if it could, and seeing if we could load some things that are
> currently on the command line from files of options/input files.
>
> I think the reason scons is purposefully going through the shell is to make
> things like redirection, etc., work. We take advantage of that in at least
> one place I've seen, but it might be feasible to remove that and specialize
> env['SPAWN'] in a way that breaks redirection but allows arbitrarily long
> command lines.
>
> But I still think incremental linking would be nicer for a number of other
> reasons, as long as it's implementable.
>
> Gabe
>
> On Thu, Apr 13, 2017 at 10:16 AM, Steve Reinhardt 
> wrote:
>
> > #3 is the traditional solution :)
> >
> > On Thu, Apr 13, 2017 at 10:13 AM, Gutierrez, Anthony <
> > anthony.gutier...@amd.com> wrote:
> >
> > > There are a three ways to fix this as far as I can tell:
> > >
> > > 1) Modify our Scons setup to use staged linking.
> > > 2) Recompile your kernel to allow for larger ARG_MAX.
> > > 3) Modify your paths etc to avoid long names
> > >
> > > 1) seems to be the best option, but seems like it could be a lot of
> work.
> > >
> > > -Original Message-
> > > From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
> > Black
> > > Sent: Thursday, April 13, 2017 9:50 AM
> > > To: gem5 Developer List 
> > > Subject: Re: [gem5-dev] scons question
> > >
> > > Oh, I bet you're right. They actually spawn something like 'sh', '-c',
> '
> > > '.join(args), and I bet sh (which is symlinked to /bin/bash) is blowing
> > up
> > > because the command line is very long. I remember my terminal asking
> if I
> > > really wanted to copy/paste something like 129K characters when trying
> to
> > > copy the command line to run it outside of scons.
> > >
> > > Now to figure out how to fix it...
> > >
> > > On Thu, Apr 13, 2017 at 8:02 AM, Beckmann, Brad  >
> > > wrote:
> > >
> > > > Have you investigated the length of the linker command when building
> > > > from outside the gem5 directory?  In the past, we've seen that
> > > > mysterious error
> > > > 127 because the linker stage uses a shell command length that exceeds
> > > > the length supported by the OS.  64KB I believe.  I suspect that the
> > > > filenames are longer when building outside of gem5, thus it only
> > > > happens in that situation.  The linker command may be shorter using
> > > clang as well.
> > > >
> > > > Brad
> > > >
> > > >
> > > >
> > > > -Original Message-
> > > > From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
> > > > Black
> > > > Sent: Thursday, April 13, 2017 1:53 AM
> > > > To: gem5 Developer List 
> > > > Subject: [gem5-dev] scons question
> > > >
> > > > Hi folks. I'm fighting with a very confusing problem with scons at
> the
> > > > moment. For reasons I haven't determined, when I have things set up
> to
> > > > build when scons is run from outside 

Re: [gem5-dev] scons question

[Gabe Black]

Ok, I think we're all pretty much in agreement then, since I was thinking
about incremental linking too. What about it looked non-trivial, Steve? I
haven't dug into it very much yet, but as long as it doesn't turn into too
much of an ordeal I'd like to take care of it.

Potentially less ideal solutions which might also be easier would be to
override the env['SPAWN'] setting so that it would skip going through the
shell if it could, and seeing if we could load some things that are
currently on the command line from files of options/input files.

I think the reason scons is purposefully going through the shell is to make
things like redirection, etc., work. We take advantage of that in at least
one place I've seen, but it might be feasible to remove that and specialize
env['SPAWN'] in a way that breaks redirection but allows arbitrarily long
command lines.

But I still think incremental linking would be nicer for a number of other
reasons, as long as it's implementable.

Gabe

On Thu, Apr 13, 2017 at 10:16 AM, Steve Reinhardt  wrote:

> #3 is the traditional solution :)
>
> On Thu, Apr 13, 2017 at 10:13 AM, Gutierrez, Anthony <
> anthony.gutier...@amd.com> wrote:
>
> > There are a three ways to fix this as far as I can tell:
> >
> > 1) Modify our Scons setup to use staged linking.
> > 2) Recompile your kernel to allow for larger ARG_MAX.
> > 3) Modify your paths etc to avoid long names
> >
> > 1) seems to be the best option, but seems like it could be a lot of work.
> >
> > -Original Message-
> > From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
> Black
> > Sent: Thursday, April 13, 2017 9:50 AM
> > To: gem5 Developer List 
> > Subject: Re: [gem5-dev] scons question
> >
> > Oh, I bet you're right. They actually spawn something like 'sh', '-c', '
> > '.join(args), and I bet sh (which is symlinked to /bin/bash) is blowing
> up
> > because the command line is very long. I remember my terminal asking if I
> > really wanted to copy/paste something like 129K characters when trying to
> > copy the command line to run it outside of scons.
> >
> > Now to figure out how to fix it...
> >
> > On Thu, Apr 13, 2017 at 8:02 AM, Beckmann, Brad 
> > wrote:
> >
> > > Have you investigated the length of the linker command when building
> > > from outside the gem5 directory?  In the past, we've seen that
> > > mysterious error
> > > 127 because the linker stage uses a shell command length that exceeds
> > > the length supported by the OS.  64KB I believe.  I suspect that the
> > > filenames are longer when building outside of gem5, thus it only
> > > happens in that situation.  The linker command may be shorter using
> > clang as well.
> > >
> > > Brad
> > >
> > >
> > >
> > > -Original Message-
> > > From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
> > > Black
> > > Sent: Thursday, April 13, 2017 1:53 AM
> > > To: gem5 Developer List 
> > > Subject: [gem5-dev] scons question
> > >
> > > Hi folks. I'm fighting with a very confusing problem with scons at the
> > > moment. For reasons I haven't determined, when I have things set up to
> > > build when scons is run from outside the gem5 directory (using -C), it
> > > fails the final linker step with error 127 and no other output 100% of
> > > the time. If I run from within the gem5 directory everything works
> fine.
> > >
> > > I did some reading, and bash reports error 127 when it can't find the
> > > command you asked it to run. To determine if that might be the
> > > problem, I modified scons to run "which" on each command it was about
> > > to spawn before it did, to make sure it resolved to something. That
> > > worked just fine. If I run the command manually, it returns exit code
> > > 0. If I take the environment scons tries to run g++ under and
> > > partially duplicate that with a script and env -i, it still succeeds.
> > >
> > > If I run with clang instead of g++, I get the same behavior which
> > > makes me think it's not g++ doing something weird, it's scons. I can't
> > > for the life of me figure out what though, and I can't seem to get any
> > > information to work with other than this mysterious error 127.
> > >
> > > If any of you have any idea why it's doing what it's doing, or if
> > > there's any information I can gather that might help, I would be very
> > > happy to hear it.
> > >
> > > Gabe
> > > ___
> > > gem5-dev mailing list
> > > gem5-dev@gem5.org
> > > http://m5sim.org/mailman/listinfo/gem5-dev
> > > ___
> > > gem5-dev mailing list
> > > gem5-dev@gem5.org
> > > http://m5sim.org/mailman/listinfo/gem5-dev
> > ___
> > gem5-dev mailing list
> > gem5-dev@gem5.org
> > http://m5sim.org/mailman/listinfo/gem5-dev
> > ___
> > gem5-dev mailing list
> > 

Re: [gem5-dev] scons question

[Steve Reinhardt]

#3 is the traditional solution :)

On Thu, Apr 13, 2017 at 10:13 AM, Gutierrez, Anthony <
anthony.gutier...@amd.com> wrote:

> There are a three ways to fix this as far as I can tell:
>
> 1) Modify our Scons setup to use staged linking.
> 2) Recompile your kernel to allow for larger ARG_MAX.
> 3) Modify your paths etc to avoid long names
>
> 1) seems to be the best option, but seems like it could be a lot of work.
>
> -Original Message-
> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black
> Sent: Thursday, April 13, 2017 9:50 AM
> To: gem5 Developer List 
> Subject: Re: [gem5-dev] scons question
>
> Oh, I bet you're right. They actually spawn something like 'sh', '-c', '
> '.join(args), and I bet sh (which is symlinked to /bin/bash) is blowing up
> because the command line is very long. I remember my terminal asking if I
> really wanted to copy/paste something like 129K characters when trying to
> copy the command line to run it outside of scons.
>
> Now to figure out how to fix it...
>
> On Thu, Apr 13, 2017 at 8:02 AM, Beckmann, Brad 
> wrote:
>
> > Have you investigated the length of the linker command when building
> > from outside the gem5 directory?  In the past, we've seen that
> > mysterious error
> > 127 because the linker stage uses a shell command length that exceeds
> > the length supported by the OS.  64KB I believe.  I suspect that the
> > filenames are longer when building outside of gem5, thus it only
> > happens in that situation.  The linker command may be shorter using
> clang as well.
> >
> > Brad
> >
> >
> >
> > -Original Message-
> > From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
> > Black
> > Sent: Thursday, April 13, 2017 1:53 AM
> > To: gem5 Developer List 
> > Subject: [gem5-dev] scons question
> >
> > Hi folks. I'm fighting with a very confusing problem with scons at the
> > moment. For reasons I haven't determined, when I have things set up to
> > build when scons is run from outside the gem5 directory (using -C), it
> > fails the final linker step with error 127 and no other output 100% of
> > the time. If I run from within the gem5 directory everything works fine.
> >
> > I did some reading, and bash reports error 127 when it can't find the
> > command you asked it to run. To determine if that might be the
> > problem, I modified scons to run "which" on each command it was about
> > to spawn before it did, to make sure it resolved to something. That
> > worked just fine. If I run the command manually, it returns exit code
> > 0. If I take the environment scons tries to run g++ under and
> > partially duplicate that with a script and env -i, it still succeeds.
> >
> > If I run with clang instead of g++, I get the same behavior which
> > makes me think it's not g++ doing something weird, it's scons. I can't
> > for the life of me figure out what though, and I can't seem to get any
> > information to work with other than this mysterious error 127.
> >
> > If any of you have any idea why it's doing what it's doing, or if
> > there's any information I can gather that might help, I would be very
> > happy to hear it.
> >
> > Gabe
> > ___
> > gem5-dev mailing list
> > gem5-dev@gem5.org
> > http://m5sim.org/mailman/listinfo/gem5-dev
> > ___
> > gem5-dev mailing list
> > gem5-dev@gem5.org
> > http://m5sim.org/mailman/listinfo/gem5-dev
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
>
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

Re: [gem5-dev] scons question

[Gutierrez, Anthony]

There are a three ways to fix this as far as I can tell:

1) Modify our Scons setup to use staged linking.
2) Recompile your kernel to allow for larger ARG_MAX.
3) Modify your paths etc to avoid long names

1) seems to be the best option, but seems like it could be a lot of work.

-Original Message-
From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black
Sent: Thursday, April 13, 2017 9:50 AM
To: gem5 Developer List 
Subject: Re: [gem5-dev] scons question

Oh, I bet you're right. They actually spawn something like 'sh', '-c', '
'.join(args), and I bet sh (which is symlinked to /bin/bash) is blowing up 
because the command line is very long. I remember my terminal asking if I 
really wanted to copy/paste something like 129K characters when trying to copy 
the command line to run it outside of scons.

Now to figure out how to fix it...

On Thu, Apr 13, 2017 at 8:02 AM, Beckmann, Brad 
wrote:

> Have you investigated the length of the linker command when building 
> from outside the gem5 directory?  In the past, we've seen that 
> mysterious error
> 127 because the linker stage uses a shell command length that exceeds 
> the length supported by the OS.  64KB I believe.  I suspect that the 
> filenames are longer when building outside of gem5, thus it only 
> happens in that situation.  The linker command may be shorter using clang as 
> well.
>
> Brad
>
>
>
> -Original Message-
> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe 
> Black
> Sent: Thursday, April 13, 2017 1:53 AM
> To: gem5 Developer List 
> Subject: [gem5-dev] scons question
>
> Hi folks. I'm fighting with a very confusing problem with scons at the 
> moment. For reasons I haven't determined, when I have things set up to 
> build when scons is run from outside the gem5 directory (using -C), it 
> fails the final linker step with error 127 and no other output 100% of 
> the time. If I run from within the gem5 directory everything works fine.
>
> I did some reading, and bash reports error 127 when it can't find the 
> command you asked it to run. To determine if that might be the 
> problem, I modified scons to run "which" on each command it was about 
> to spawn before it did, to make sure it resolved to something. That 
> worked just fine. If I run the command manually, it returns exit code 
> 0. If I take the environment scons tries to run g++ under and 
> partially duplicate that with a script and env -i, it still succeeds.
>
> If I run with clang instead of g++, I get the same behavior which 
> makes me think it's not g++ doing something weird, it's scons. I can't 
> for the life of me figure out what though, and I can't seem to get any 
> information to work with other than this mysterious error 127.
>
> If any of you have any idea why it's doing what it's doing, or if 
> there's any information I can gather that might help, I would be very 
> happy to hear it.
>
> Gabe
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

Re: [gem5-dev] scons question

[Steve Reinhardt]

Agreed. I actually looked into doing this once briefly (primarily to reduce
link time), but it looked like it wasn't going to be trivial, so I punted.

On Thu, Apr 13, 2017 at 10:09 AM, Andreas Hansson 
wrote:

> The best solution, in my view, would be to link the various subdirectories
> into static libraries, libbase.a etc, and do so independent of the ISA
> that is being built. Everything besides the arch and dev tree should then
> be able to be built once, and the linking also ends up being hierarchical.
>
> Andreas
>
> On 13/04/2017, 17:50, "gem5-dev on behalf of Gabe Black"
>  wrote:
>
> >Oh, I bet you're right. They actually spawn something like 'sh', '-c', '
> >'.join(args), and I bet sh (which is symlinked to /bin/bash) is blowing up
> >because the command line is very long. I remember my terminal asking if I
> >really wanted to copy/paste something like 129K characters when trying to
> >copy the command line to run it outside of scons.
> >
> >Now to figure out how to fix it...
> >
> >On Thu, Apr 13, 2017 at 8:02 AM, Beckmann, Brad 
> >wrote:
> >
> >> Have you investigated the length of the linker command when building
> >>from
> >> outside the gem5 directory?  In the past, we've seen that mysterious
> >>error
> >> 127 because the linker stage uses a shell command length that exceeds
> >>the
> >> length supported by the OS.  64KB I believe.  I suspect that the
> >>filenames
> >> are longer when building outside of gem5, thus it only happens in that
> >> situation.  The linker command may be shorter using clang as well.
> >>
> >> Brad
> >>
> >>
> >>
> >> -Original Message-
> >> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
> >>Black
> >> Sent: Thursday, April 13, 2017 1:53 AM
> >> To: gem5 Developer List 
> >> Subject: [gem5-dev] scons question
> >>
> >> Hi folks. I'm fighting with a very confusing problem with scons at the
> >> moment. For reasons I haven't determined, when I have things set up to
> >> build when scons is run from outside the gem5 directory (using -C), it
> >> fails the final linker step with error 127 and no other output 100% of
> >>the
> >> time. If I run from within the gem5 directory everything works fine.
> >>
> >> I did some reading, and bash reports error 127 when it can't find the
> >> command you asked it to run. To determine if that might be the problem,
> >>I
> >> modified scons to run "which" on each command it was about to spawn
> >>before
> >> it did, to make sure it resolved to something. That worked just fine.
> >>If I
> >> run the command manually, it returns exit code 0. If I take the
> >>environment
> >> scons tries to run g++ under and partially duplicate that with a script
> >>and
> >> env -i, it still succeeds.
> >>
> >> If I run with clang instead of g++, I get the same behavior which makes
> >>me
> >> think it's not g++ doing something weird, it's scons. I can't for the
> >>life
> >> of me figure out what though, and I can't seem to get any information to
> >> work with other than this mysterious error 127.
> >>
> >> If any of you have any idea why it's doing what it's doing, or if
> >>there's
> >> any information I can gather that might help, I would be very happy to
> >>hear
> >> it.
> >>
> >> Gabe
> >> ___
> >> gem5-dev mailing list
> >> gem5-dev@gem5.org
> >> http://m5sim.org/mailman/listinfo/gem5-dev
> >> ___
> >> gem5-dev mailing list
> >> gem5-dev@gem5.org
> >> http://m5sim.org/mailman/listinfo/gem5-dev
> >___
> >gem5-dev mailing list
> >gem5-dev@gem5.org
> >http://m5sim.org/mailman/listinfo/gem5-dev
>
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
>
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

Re: [gem5-dev] scons question

[Andreas Hansson]

The best solution, in my view, would be to link the various subdirectories
into static libraries, libbase.a etc, and do so independent of the ISA
that is being built. Everything besides the arch and dev tree should then
be able to be built once, and the linking also ends up being hierarchical.

Andreas

On 13/04/2017, 17:50, "gem5-dev on behalf of Gabe Black"
 wrote:

>Oh, I bet you're right. They actually spawn something like 'sh', '-c', '
>'.join(args), and I bet sh (which is symlinked to /bin/bash) is blowing up
>because the command line is very long. I remember my terminal asking if I
>really wanted to copy/paste something like 129K characters when trying to
>copy the command line to run it outside of scons.
>
>Now to figure out how to fix it...
>
>On Thu, Apr 13, 2017 at 8:02 AM, Beckmann, Brad 
>wrote:
>
>> Have you investigated the length of the linker command when building
>>from
>> outside the gem5 directory?  In the past, we've seen that mysterious
>>error
>> 127 because the linker stage uses a shell command length that exceeds
>>the
>> length supported by the OS.  64KB I believe.  I suspect that the
>>filenames
>> are longer when building outside of gem5, thus it only happens in that
>> situation.  The linker command may be shorter using clang as well.
>>
>> Brad
>>
>>
>>
>> -Original Message-
>> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
>>Black
>> Sent: Thursday, April 13, 2017 1:53 AM
>> To: gem5 Developer List 
>> Subject: [gem5-dev] scons question
>>
>> Hi folks. I'm fighting with a very confusing problem with scons at the
>> moment. For reasons I haven't determined, when I have things set up to
>> build when scons is run from outside the gem5 directory (using -C), it
>> fails the final linker step with error 127 and no other output 100% of
>>the
>> time. If I run from within the gem5 directory everything works fine.
>>
>> I did some reading, and bash reports error 127 when it can't find the
>> command you asked it to run. To determine if that might be the problem,
>>I
>> modified scons to run "which" on each command it was about to spawn
>>before
>> it did, to make sure it resolved to something. That worked just fine.
>>If I
>> run the command manually, it returns exit code 0. If I take the
>>environment
>> scons tries to run g++ under and partially duplicate that with a script
>>and
>> env -i, it still succeeds.
>>
>> If I run with clang instead of g++, I get the same behavior which makes
>>me
>> think it's not g++ doing something weird, it's scons. I can't for the
>>life
>> of me figure out what though, and I can't seem to get any information to
>> work with other than this mysterious error 127.
>>
>> If any of you have any idea why it's doing what it's doing, or if
>>there's
>> any information I can gather that might help, I would be very happy to
>>hear
>> it.
>>
>> Gabe
>> ___
>> gem5-dev mailing list
>> gem5-dev@gem5.org
>> http://m5sim.org/mailman/listinfo/gem5-dev
>> ___
>> gem5-dev mailing list
>> gem5-dev@gem5.org
>> http://m5sim.org/mailman/listinfo/gem5-dev
>___
>gem5-dev mailing list
>gem5-dev@gem5.org
>http://m5sim.org/mailman/listinfo/gem5-dev

IMPORTANT NOTICE: The contents of this email and any attachments are 
confidential and may also be privileged. If you are not the intended recipient, 
please notify the sender immediately and do not disclose the contents to any 
other person, use it for any purpose, or store or copy the information in any 
medium. Thank you.
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

Re: [gem5-dev] scons question

[Gabe Black]

Oh, I bet you're right. They actually spawn something like 'sh', '-c', '
'.join(args), and I bet sh (which is symlinked to /bin/bash) is blowing up
because the command line is very long. I remember my terminal asking if I
really wanted to copy/paste something like 129K characters when trying to
copy the command line to run it outside of scons.

Now to figure out how to fix it...

On Thu, Apr 13, 2017 at 8:02 AM, Beckmann, Brad 
wrote:

> Have you investigated the length of the linker command when building from
> outside the gem5 directory?  In the past, we've seen that mysterious error
> 127 because the linker stage uses a shell command length that exceeds the
> length supported by the OS.  64KB I believe.  I suspect that the filenames
> are longer when building outside of gem5, thus it only happens in that
> situation.  The linker command may be shorter using clang as well.
>
> Brad
>
>
>
> -Original Message-
> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black
> Sent: Thursday, April 13, 2017 1:53 AM
> To: gem5 Developer List 
> Subject: [gem5-dev] scons question
>
> Hi folks. I'm fighting with a very confusing problem with scons at the
> moment. For reasons I haven't determined, when I have things set up to
> build when scons is run from outside the gem5 directory (using -C), it
> fails the final linker step with error 127 and no other output 100% of the
> time. If I run from within the gem5 directory everything works fine.
>
> I did some reading, and bash reports error 127 when it can't find the
> command you asked it to run. To determine if that might be the problem, I
> modified scons to run "which" on each command it was about to spawn before
> it did, to make sure it resolved to something. That worked just fine. If I
> run the command manually, it returns exit code 0. If I take the environment
> scons tries to run g++ under and partially duplicate that with a script and
> env -i, it still succeeds.
>
> If I run with clang instead of g++, I get the same behavior which makes me
> think it's not g++ doing something weird, it's scons. I can't for the life
> of me figure out what though, and I can't seem to get any information to
> work with other than this mysterious error 127.
>
> If any of you have any idea why it's doing what it's doing, or if there's
> any information I can gather that might help, I would be very happy to hear
> it.
>
> Gabe
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

Re: [gem5-dev] scons question

[Beckmann, Brad]

Have you investigated the length of the linker command when building from 
outside the gem5 directory?  In the past, we've seen that mysterious error 127 
because the linker stage uses a shell command length that exceeds the length 
supported by the OS.  64KB I believe.  I suspect that the filenames are longer 
when building outside of gem5, thus it only happens in that situation.  The 
linker command may be shorter using clang as well.

Brad



-Original Message-
From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black
Sent: Thursday, April 13, 2017 1:53 AM
To: gem5 Developer List 
Subject: [gem5-dev] scons question

Hi folks. I'm fighting with a very confusing problem with scons at the moment. 
For reasons I haven't determined, when I have things set up to build when scons 
is run from outside the gem5 directory (using -C), it fails the final linker 
step with error 127 and no other output 100% of the time. If I run from within 
the gem5 directory everything works fine.

I did some reading, and bash reports error 127 when it can't find the command 
you asked it to run. To determine if that might be the problem, I modified 
scons to run "which" on each command it was about to spawn before it did, to 
make sure it resolved to something. That worked just fine. If I run the command 
manually, it returns exit code 0. If I take the environment scons tries to run 
g++ under and partially duplicate that with a script and env -i, it still 
succeeds.

If I run with clang instead of g++, I get the same behavior which makes me 
think it's not g++ doing something weird, it's scons. I can't for the life of 
me figure out what though, and I can't seem to get any information to work with 
other than this mysterious error 127.

If any of you have any idea why it's doing what it's doing, or if there's any 
information I can gather that might help, I would be very happy to hear it.

Gabe
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

[gem5-dev] scons question

[Gabe Black]

Hi folks. I'm fighting with a very confusing problem with scons at the
moment. For reasons I haven't determined, when I have things set up to
build when scons is run from outside the gem5 directory (using -C), it
fails the final linker step with error 127 and no other output 100% of the
time. If I run from within the gem5 directory everything works fine.

I did some reading, and bash reports error 127 when it can't find the
command you asked it to run. To determine if that might be the problem, I
modified scons to run "which" on each command it was about to spawn before
it did, to make sure it resolved to something. That worked just fine. If I
run the command manually, it returns exit code 0. If I take the environment
scons tries to run g++ under and partially duplicate that with a script and
env -i, it still succeeds.

If I run with clang instead of g++, I get the same behavior which makes me
think it's not g++ doing something weird, it's scons. I can't for the life
of me figure out what though, and I can't seem to get any information to
work with other than this mysterious error 127.

If any of you have any idea why it's doing what it's doing, or if there's
any information I can gather that might help, I would be very happy to hear
it.

Gabe
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

[gem5-dev] Cron <m5test@zizzer> /z/m5/regression/do-regression quick

[Cron Daemon]

* 
build/MIPS/tests/opt/quick/se/03.learning-gem5/mips/linux/learning-gem5-p1-two-level:
 CHANGED!
*** diff[simout]: SKIPPED* 
build/MIPS/tests/opt/quick/se/00.hello/mips/linux/simple-atomic: CHANGED!
*** diff[simout]: SKIPPED* 
build/MIPS/tests/opt/quick/se/00.hello/mips/linux/o3-timing: CHANGED!
 --- quick/se/00.hello/mips/linux/simple-timing-ruby ---* 
build/MIPS/tests/opt/quick/se/00.hello/mips/linux/simple-timing: CHANGED!
* build/MIPS/tests/opt/quick/se/00.hello/mips/linux/simple-timing-ruby: 
CHANGED!--- quick/se/03.learning-gem5/mips/linux/learning-gem5-p1-simple ---
* 
build/MIPS/tests/opt/quick/se/03.learning-gem5/mips/linux/learning-gem5-p1-simple:
 CHANGED!
* build/POWER/tests/opt/quick/se/00.hello/power/linux/o3-timing: CHANGED!
* build/POWER/tests/opt/quick/se/00.hello/power/linux/simple-atomic: 
CHANGED!
* build/SPARC/tests/opt/quick/se/00.hello/sparc/linux/simple-timing: 
CHANGED!
*** diff[simout]: SKIPPED* 
build/SPARC/tests/opt/quick/se/02.insttest/sparc/linux/simple-atomic: CHANGED!
* build/SPARC/tests/opt/quick/se/00.hello/sparc/linux/simple-timing-ruby: 
CHANGED!
* build/SPARC/tests/opt/quick/se/00.hello/sparc/linux/simple-atomic: 
CHANGED!*** diff[simout]: SKIPPED
* build/SPARC/tests/opt/quick/se/02.insttest/sparc/linux/o3-timing: CHANGED!
* build/SPARC/tests/opt/quick/se/10.mcf/sparc/linux/simple-atomic: CHANGED!
* 
build/SPARC/tests/opt/quick/se/03.learning-gem5/sparc/linux/learning-gem5-p1-simple:
 CHANGED!
* 
build/SPARC/tests/opt/quick/se/40.m5threads-test-atomic/sparc/linux/simple-atomic-mp:
 CHANGED!--- quick/se/40.m5threads-test-atomic/sparc/linux/o3-timing-mp ---
* 
build/SPARC/tests/opt/quick/se/40.m5threads-test-atomic/sparc/linux/o3-timing-mp:
 CHANGED!
* 
build/SPARC/tests/opt/quick/se/03.learning-gem5/sparc/linux/learning-gem5-p1-two-level:
 CHANGED!
* 
build/SPARC/tests/opt/quick/se/40.m5threads-test-atomic/sparc/linux/simple-timing-mp:
 CHANGED!
* build/SPARC/tests/opt/quick/se/02.insttest/sparc/linux/simple-timing: 
CHANGED!
*** gem5: OK* 
build/SPARC/tests/opt/quick/se/50.vortex/sparc/linux/simple-timing: CHANGED!
* build/SPARC/tests/opt/quick/se/50.vortex/sparc/linux/simple-atomic: 
CHANGED!
 * build/SPARC/tests/opt/quick/se/70.twolf/sparc/linux/simple-atomic: 
CHANGED!--- quick/se/70.twolf/sparc/linux/simple-timing ---
* build/SPARC/tests/opt/quick/se/70.twolf/sparc/linux/simple-timing: 
CHANGED!
* build/X86/tests/opt/quick/se/10.mcf/x86/linux/simple-atomic: CHANGED!
* build/X86/tests/opt/quick/se/70.twolf/x86/linux/simple-atomic: CHANGED!
* build/X86/tests/opt/quick/se/70.twolf/x86/linux/simple-timing: CHANGED!
* 
build/X86/tests/opt/quick/se/03.learning-gem5/x86/linux/learning-gem5-p1-simple:
 CHANGED!
* build/X86/tests/opt/quick/se/00.hello/x86/linux/simple-atomic: CHANGED!
* build/X86/tests/opt/quick/se/00.hello/x86/linux/o3-timing: CHANGED!
* build/X86/tests/opt/quick/se/00.hello/x86/linux/simple-timing-ruby: 
CHANGED!
* build/X86/tests/opt/quick/se/00.hello/x86/linux/simple-timing: CHANGED!
* 
build/X86/tests/opt/quick/se/03.learning-gem5/x86/linux/learning-gem5-p1-two-level:
 CHANGED!
 * build/ARM/tests/opt/quick/se/00.hello/arm/linux/minor-timing: CHANGED!
* build/ARM/tests/opt/quick/se/00.hello/arm/linux/simple-atomic: 
CHANGED!* build/ARM/tests/opt/quick/se/00.hello/arm/linux/simple-timing: 
CHANGED!*** diff[simout]: SKIPPED
*** diff[simout]: SKIPPED* 
build/ARM/tests/opt/quick/se/00.hello/arm/linux/o3-timing-checker: CHANGED!
* build/ARM/tests/opt/quick/se/00.hello/arm/linux/o3-timing: CHANGED!* 
build/ARM/tests/opt/quick/se/03.learning-gem5/arm/linux/learning-gem5-p1-simple:
 CHANGED!
* 
build/ARM/tests/opt/quick/se/00.hello/arm/linux/simple-atomic-dummychecker: 
CHANGED!
* 
build/ARM/tests/opt/quick/se/03.learning-gem5/arm/linux/learning-gem5-p1-two-level:
 CHANGED!
* build/ARM/tests/opt/quick/se/10.mcf/arm/linux/simple-atomic: CHANGED!
* build/ARM/tests/opt/quick/se/10.mcf/arm/linux/simple-timing: CHANGED!
* build/ARM/tests/opt/quick/se/50.vortex/arm/linux/simple-atomic: CHANGED!
* build/ARM/tests/opt/quick/se/50.vortex/arm/linux/simple-timing: CHANGED!
* build/ARM/tests/opt/quick/se/70.twolf/arm/linux/simple-atomic: CHANGED!
* build/ARM/tests/opt/quick/se/70.twolf/arm/linux/simple-timing: CHANGED!
* 
build/ARM/tests/opt/quick/fs/10.linux-boot/arm/linux/realview-simple-timing: 
CHANGED!
* 
build/ARM/tests/opt/quick/fs/10.linux-boot/arm/linux/realview-simple-timing-dual:
 CHANGED!
* build/RISCV/tests/opt/quick/se/00.hello/riscv/linux/simple-atomic: 
CHANGED!
*** diff[config.ini]: SKIPPED* 
build/RISCV/tests/opt/quick/se/00.hello/riscv/linux/o3-timing: CHANGED!*** 
gem5: OK
*** diff[simout]: SKIPPED* 
build/RISCV/tests/opt/quick/se/00.hello/riscv/linux/simple-timing: 
CHANGED!Statistics mismatch
*