Constantine,
On 01.02.2015 22:25, Constantine wrote:
Good results, Dirk.
It's definitely worth to use posix_spawn.
When do you plan to integrate the experimental version into the mainline?
next step is switching the Node class to using __slots__, then we want to do a full release 2.4. After that, the version 2.5 will
contain the stubprocess/posix_spawn wrapper...if all goes well.
/> .. including some hacks to exploit the uniform structure of the benchmark
sources//../
What kind of hacks?
How do they affect performance?
I pre-expanded the CXXCOM action string like this:
DefaultEnvironment(tools=[])
env = Environment(tools=['g++','ar','link','fastcpp'],
CPPFLAGS=['-Wall'], CPPDEFINES=['LINUX'],
CPPPATH=[Dir('#')])
env['CXXCOM'] = 'g++ -o $TARGET -c -Wall -I. -DLINUX $SOURCES'
because it stays the same all the time. This shortcuts a lot of subst()ing of strings. As you can see, I also used the "fastcpp"
Tool which additionally replaces the default suffixes for CPP/object files with their actual strings.
For some classes like Action, I found several places where a lot of checking is done, in order to provide maximum flexibility. It's
normally allowed to pass a sequence of mixed strings and Nodes as "target" or "source" parameter. Here, I assume that it's always a
list of strings...so the check is commented out. For the full changes see the attached patch, they cut the complete reading/parsing
of all SConscripts down to 30 seconds.
This is only to show that the basic design of SCons is able to provide fast results, when turning the right knobs...and also to
identify the places where most of the runtime is spent. Whether to actually include some of my hacks, and in which form
(command-line option?, Tool?) would still have to be discussed.
Thank you for testing the clean targets case.
I haven't tested this case on large projects.
You should definitely have a look at this. It seems to get worse when the number of total files for the build increases. By the way,
do you support custom commands and generated files in Aqualid? Like, let's say:
- "foo.c" is compiled to the "foo" executable
- "foo" is then used to create the header "bar.h"
- "bar.h" is an implicit dependency of "bar.c"
- "bar.c" is compiled to "bar"
It would also be interesting to see how well Aqualid is doing in the build
system shootout:
https://github.com/ndmitchell/build-shootout
by Neil Mitchell. Have a look at it, if you find the time.
Best regards,
Dirk
# HG changeset patch
# Parent 35dbc21379ec919c648e12e8822962d6991a585b
diff -r 35dbc21379ec -r 23b208e01c06 src/engine/SCons/Action.py
--- a/src/engine/SCons/Action.py Sun Dec 21 13:13:44 2014 +0100
+++ b/src/engine/SCons/Action.py Sat Jan 31 00:33:20 2015 +0100
@@ -99,8 +99,6 @@
__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__"
-import SCons.compat
-
import dis
import os
# compat layer imports "cPickle" for us if it's available.
@@ -112,7 +110,6 @@
import SCons.Debug
from SCons.Debug import logInstanceCreation
import SCons.Errors
-import SCons.Executor
import SCons.Util
import SCons.Subst
@@ -357,21 +354,6 @@
if isinstance(act, ActionBase):
return act
- if is_List(act):
- return CommandAction(act, **kw)
-
- if callable(act):
- try:
- gen = kw['generator']
- del kw['generator']
- except KeyError:
- gen = 0
- if gen:
- action_type = CommandGeneratorAction
- else:
- action_type = FunctionAction
- return action_type(act, kw)
-
if is_String(act):
var=SCons.Util.get_environment_var(act)
if var:
@@ -388,6 +370,22 @@
# The list of string commands may include a LazyAction, so we
# reprocess them via _do_create_list_action.
return _do_create_list_action(commands, kw)
+
+ if is_List(act):
+ return CommandAction(act, **kw)
+
+ if callable(act):
+ try:
+ gen = kw['generator']
+ del kw['generator']
+ except KeyError:
+ gen = 0
+ if gen:
+ action_type = CommandGeneratorAction
+ else:
+ action_type = FunctionAction
+ return action_type(act, kw)
+
# Catch a common error case with a nice message:
if isinstance(act, int) or isinstance(act, float):
raise TypeError("Don't know how to create an Action from a number (%s)"%act)
diff -r 35dbc21379ec -r 23b208e01c06 src/engine/SCons/Environment.py
--- a/src/engine/SCons/Environment.py Sun Dec 21 13:13:44 2014 +0100
+++ b/src/engine/SCons/Environment.py Sat Jan 31 00:33:20 2015 +0100
@@ -456,35 +456,38 @@
if not args:
return []
- args = SCons.Util.flatten(args)
+ #args = SCons.Util.flatten(args)
nodes = []
+ kw['raw'] = 1
for v in args:
if SCons.Util.is_String(v):
- n = None
- for l in lookup_list:
- n = l(v)
- if n is not None:
- break
- if n is not None:
- if SCons.Util.is_String(n):
- # n = self.subst(n, raw=1, **kw)
- kw['raw'] = 1
- n = self.subst(n, **kw)
- if node_factory:
- n = node_factory(n)
- if SCons.Util.is_List(n):
- nodes.extend(n)
- else:
- nodes.append(n)
- elif node_factory:
+# n = None
+# for l in lookup_list:
+# n = l(v)
+# if n is not None:
+# break
+# if n is not None:
+# if SCons.Util.is_String(n):
+# # n = self.subst(n, raw=1, **kw)
+# kw['raw'] = 1
+# n = self.subst(n, **kw)
+# if node_factory:
+# n = node_factory(n)
+# if SCons.Util.is_List(n):
+# nodes.extend(n)
+# else:
+# nodes.append(n)
+# elif node_factory:
+ if node_factory:
# v = node_factory(self.subst(v, raw=1, **kw))
- kw['raw'] = 1
- v = node_factory(self.subst(v, **kw))
- if SCons.Util.is_List(v):
- nodes.extend(v)
- else:
- nodes.append(v)
+ #kw['raw'] = 1
+ #v = node_factory(self.subst(v, **kw))
+ #if SCons.Util.is_List(v):
+ # nodes.extend(v)
+ #else:
+ # nodes.append(v)
+ nodes.append(node_factory(v))
else:
nodes.append(v)
diff -r 35dbc21379ec -r 23b208e01c06 src/engine/SCons/Executor.py
--- a/src/engine/SCons/Executor.py Sun Dec 21 13:13:44 2014 +0100
+++ b/src/engine/SCons/Executor.py Sat Jan 31 00:33:20 2015 +0100
@@ -577,20 +577,19 @@
nullenv = None
+import SCons.Util
+class NullEnvironment(SCons.Util.Null):
+ import SCons.CacheDir
+ _CacheDir_path = None
+ _CacheDir = SCons.CacheDir.CacheDir(None)
+ def get_CacheDir(self):
+ return self._CacheDir
def get_NullEnvironment():
"""Use singleton pattern for Null Environments."""
global nullenv
- import SCons.Util
- class NullEnvironment(SCons.Util.Null):
- import SCons.CacheDir
- _CacheDir_path = None
- _CacheDir = SCons.CacheDir.CacheDir(None)
- def get_CacheDir(self):
- return self._CacheDir
-
- if not nullenv:
+ if nullenv is None:
nullenv = NullEnvironment()
return nullenv
diff -r 35dbc21379ec -r 23b208e01c06 src/engine/SCons/Node/FS.py
--- a/src/engine/SCons/Node/FS.py Sun Dec 21 13:13:44 2014 +0100
+++ b/src/engine/SCons/Node/FS.py Sat Jan 31 00:33:20 2015 +0100
@@ -2322,23 +2322,13 @@
raise SCons.Errors.UserError(msg)
# There is no Node for this path name, and we're allowed
# to create it.
- # (note: would like to use p.rsplit('/',1) here but
- # that's not in python 2.3)
- # e.g.: dir_name, file_name = p.rsplit('/',1)
- last_slash = p.rindex('/')
- if (last_slash >= 0):
- dir_name = p[:last_slash]
- file_name = p[last_slash+1:]
- else:
- dir_name = p # shouldn't happen, just in case
- file_name = ''
-
+ dir_name, file_name = p.rsplit('/',1)
dir_node = self._lookup_abs(dir_name, Dir)
result = klass(file_name, dir_node, self.fs)
# Double-check on disk (as configured) that the Node we
# created matches whatever is out there in the real world.
- result.diskcheck_match()
+ #result.diskcheck_match()
self._lookupDict[k] = result
dir_node.entries[_my_normcase(file_name)] = result
@@ -2346,7 +2336,8 @@
else:
# There is already a Node for this path name. Allow it to
# complain if we were looking for an inappropriate type.
- result.must_be_same(klass)
+ pass
+ #result.must_be_same(klass)
return result
def __str__(self):
diff -r 35dbc21379ec -r 23b208e01c06 src/engine/SCons/Platform/posix.py
--- a/src/engine/SCons/Platform/posix.py Sun Dec 21 13:13:44 2014 +0100
+++ b/src/engine/SCons/Platform/posix.py Sat Jan 31 00:33:20 2015 +0100
@@ -36,6 +36,7 @@
import os
import os.path
import subprocess
+import stubprocess
import sys
import select
_______________________________________________
Scons-dev mailing list
[email protected]
https://pairlist2.pair.net/mailman/listinfo/scons-dev
