Re: [edk2] [Patch] BaseTools: Support multi thread build Basetool on Windows
Jaben: NmakeSubdirs.bat will be removed once NmakeSubdirs.py is stable. Now, bat is kept for a while. Thanks Liming > -Original Message- > From: Carsey, Jaben > Sent: Wednesday, August 29, 2018 12:10 AM > To: Gao, Liming > Cc: edk2-devel@lists.01.org; Guo, Dongao > Subject: Re: [edk2] [Patch] BaseTools: Support multi thread build Basetool on > Windows > > This looks like a great change. Why not delete the bat file? > > Reviewed-by Jaben Carsey: > > Jaben > > > On Aug 28, 2018, at 8:28 AM, Liming Gao wrote: > > > > From: Dongao Guo > > > > Add NmakeSubdirs.py to replace NmakeSubdirs.bat in VS Makefile. This script > > will > > invoke nmake in multi thread mode. It can save more than half time of > > BaseTools > > C clean build. > > GCC make supports multiple thread in make phase. So, GNNmakefile doesn't > > need apply > > this script. > > > > single task or job=1: > >just single thread and invoke subprocess,subprocess will use > >system.stdout to print output. > > multi task: > >thread number is logic cpu count.All subprocess output will pass to > >python script by PIPE and then script print it to system.stdout. > > > > Contributed-under: TianoCore Contribution Agreement 1.1 > > Signed-off-by: Dongao Guo > > Reviewed-by: Liming Gao > > Test-by: Liming Gao > > --- > > BaseTools/Makefile | 12 +- > > BaseTools/Source/C/Makefile | 14 +-- > > BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 169 > > +++ > > 3 files changed, 182 insertions(+), 13 deletions(-) > > create mode 100644 BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > > > diff --git a/BaseTools/Makefile b/BaseTools/Makefile > > index 3736d85..b98cd85 100644 > > --- a/BaseTools/Makefile > > +++ b/BaseTools/Makefile > > @@ -1,7 +1,7 @@ > > ## @file > > # Windows makefile for Base Tools project build. > > # > > -# Copyright (c) 2007 - 2017, Intel Corporation. All rights reserved. > > +# Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved. > > # This program and the accompanying materials > > # are licensed and made available under the terms and conditions of the BSD > > License > > # which accompanies this distribution. The full text of the license may be > > found at > > @@ -20,19 +20,19 @@ SUBDIRS = $(BASE_TOOLS_PATH)\Source\C > > $(BASE_TOOLS_PATH)\Source\Python > > all: c python > > > > c : > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all > > $(BASE_TOOLS_PATH)\Source\C > > + @$(PYTHON_HOME)\python.exe > > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all > $(BASE_TOOLS_PATH)\Source\C > > > > python: > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all > > $(BASE_TOOLS_PATH)\Source\Python > > + @$(PYTHON_HOME)\python.exe > > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all > $(BASE_TOOLS_PATH)\Source\Python > > > > subdirs: $(SUBDIRS) > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $** > > + @$(PYTHON_HOME)\python.exe > > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all $** > > > > .PHONY: clean > > clean: > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat clean $(SUBDIRS) > > + $(PYTHON_HOME)\python.exe > > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py clean $(SUBDIRS) > > > > .PHONY: cleanall > > cleanall: > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat cleanall > > $(SUBDIRS) > > + $(PYTHON_HOME)\python.exe > > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py cleanall $(SUBDIRS) > > > > diff --git a/BaseTools/Source/C/Makefile b/BaseTools/Source/C/Makefile > > index 5428180..1246d23 100644 > > --- a/BaseTools/Source/C/Makefile > > +++ b/BaseTools/Source/C/Makefile > > @@ -1,7 +1,7 @@ > > ## @file > > # Windows makefile for C tools build. > > # > > -# Copyright (c) 2009 - 2017, Intel Corporation. All rights reserved. > > +# Copyright (c) 2009 - 2018, Intel Corporation. All rights reserved. > > # This program and the accompanying materials > > # are licensed and made available under the terms and conditions of the BSD > > License > > # which accompanies this distribution. The full text of the license may be > > found at > > @@ -16,7 +16,7 @@ HOST_ARCH = IA32 > > > > LIBRARIES = Common > > APPLICATIONS = \ > > - BootSectImage \ > > +
Re: [edk2] [Patch] BaseTools: Support multi thread build Basetool on Windows
This looks like a great change. Why not delete the bat file? Reviewed-by Jaben Carsey: Jaben > On Aug 28, 2018, at 8:28 AM, Liming Gao wrote: > > From: Dongao Guo > > Add NmakeSubdirs.py to replace NmakeSubdirs.bat in VS Makefile. This script > will > invoke nmake in multi thread mode. It can save more than half time of > BaseTools > C clean build. > GCC make supports multiple thread in make phase. So, GNNmakefile doesn't need > apply > this script. > > single task or job=1: >just single thread and invoke subprocess,subprocess will use >system.stdout to print output. > multi task: >thread number is logic cpu count.All subprocess output will pass to >python script by PIPE and then script print it to system.stdout. > > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Dongao Guo > Reviewed-by: Liming Gao > Test-by: Liming Gao > --- > BaseTools/Makefile | 12 +- > BaseTools/Source/C/Makefile | 14 +-- > BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 169 +++ > 3 files changed, 182 insertions(+), 13 deletions(-) > create mode 100644 BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > diff --git a/BaseTools/Makefile b/BaseTools/Makefile > index 3736d85..b98cd85 100644 > --- a/BaseTools/Makefile > +++ b/BaseTools/Makefile > @@ -1,7 +1,7 @@ > ## @file > # Windows makefile for Base Tools project build. > # > -# Copyright (c) 2007 - 2017, Intel Corporation. All rights reserved. > +# Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved. > # This program and the accompanying materials > # are licensed and made available under the terms and conditions of the BSD > License > # which accompanies this distribution. The full text of the license may be > found at > @@ -20,19 +20,19 @@ SUBDIRS = $(BASE_TOOLS_PATH)\Source\C > $(BASE_TOOLS_PATH)\Source\Python > all: c python > > c : > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all > $(BASE_TOOLS_PATH)\Source\C > + @$(PYTHON_HOME)\python.exe > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all > $(BASE_TOOLS_PATH)\Source\C > > python: > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all > $(BASE_TOOLS_PATH)\Source\Python > + @$(PYTHON_HOME)\python.exe > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all > $(BASE_TOOLS_PATH)\Source\Python > > subdirs: $(SUBDIRS) > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $** > + @$(PYTHON_HOME)\python.exe > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all $** > > .PHONY: clean > clean: > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat clean $(SUBDIRS) > + $(PYTHON_HOME)\python.exe > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py clean $(SUBDIRS) > > .PHONY: cleanall > cleanall: > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat cleanall $(SUBDIRS) > + $(PYTHON_HOME)\python.exe > $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py cleanall $(SUBDIRS) > > diff --git a/BaseTools/Source/C/Makefile b/BaseTools/Source/C/Makefile > index 5428180..1246d23 100644 > --- a/BaseTools/Source/C/Makefile > +++ b/BaseTools/Source/C/Makefile > @@ -1,7 +1,7 @@ > ## @file > # Windows makefile for C tools build. > # > -# Copyright (c) 2009 - 2017, Intel Corporation. All rights reserved. > +# Copyright (c) 2009 - 2018, Intel Corporation. All rights reserved. > # This program and the accompanying materials > # are licensed and made available under the terms and conditions of the BSD > License > # which accompanies this distribution. The full text of the license may be > found at > @@ -16,7 +16,7 @@ HOST_ARCH = IA32 > > LIBRARIES = Common > APPLICATIONS = \ > - BootSectImage \ > + VfrCompile \ > BrotliCompress \ > EfiLdrImage \ > EfiRom \ > @@ -32,7 +32,7 @@ APPLICATIONS = \ > Split \ > TianoCompress \ > VolInfo \ > - VfrCompile \ > + BootSectImage \ > DevicePath > > all: libs apps install > @@ -43,7 +43,7 @@ libs: $(LIBRARIES) >@echo # Build libraries >@echo ## >@if not exist $(LIB_PATH) mkdir $(LIB_PATH) > -@Makefiles\NmakeSubdirs.bat all $** > +@$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py all $** > > apps: $(APPLICATIONS) >@echo. > @@ -51,7 +51,7 @@ apps: $(APPLICATIONS) >@echo # Build executables >@echo ## >@if not exist $(BIN_PATH) mkdir $(BIN_PATH) > -@Makefiles\NmakeSubdirs.bat all $** > +@$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py all $** > > install: $(LIB_PATH) $(BIN_PATH) >@echo. > @@ -65,11 +65,11 @@ install: $(LIB_PATH) $(BIN_PATH) > > .PHONY: clean > clean: > - @Makefiles\NmakeSubdirs.bat clean $(LIBRARIES) $(APPLICATIONS) > + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py clean $(LIBRARIES) > $(APPLICATIONS) > > .PHONY: cleanall > cleanall: > - @Makefiles\NmakeSubdirs.bat cleanall $(LIBRARIES) $(APPLICATIONS) > + @$(PYTHON_HOME)\python.exe
[edk2] [Patch] BaseTools: Support multi thread build Basetool on Windows
From: Dongao Guo Add NmakeSubdirs.py to replace NmakeSubdirs.bat in VS Makefile. This script will invoke nmake in multi thread mode. It can save more than half time of BaseTools C clean build. GCC make supports multiple thread in make phase. So, GNNmakefile doesn't need apply this script. single task or job=1: just single thread and invoke subprocess,subprocess will use system.stdout to print output. multi task: thread number is logic cpu count.All subprocess output will pass to python script by PIPE and then script print it to system.stdout. Contributed-under: TianoCore Contribution Agreement 1.1 Signed-off-by: Dongao Guo Reviewed-by: Liming Gao Test-by: Liming Gao --- BaseTools/Makefile | 12 +- BaseTools/Source/C/Makefile | 14 +-- BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 169 +++ 3 files changed, 182 insertions(+), 13 deletions(-) create mode 100644 BaseTools/Source/C/Makefiles/NmakeSubdirs.py diff --git a/BaseTools/Makefile b/BaseTools/Makefile index 3736d85..b98cd85 100644 --- a/BaseTools/Makefile +++ b/BaseTools/Makefile @@ -1,7 +1,7 @@ ## @file # Windows makefile for Base Tools project build. # -# Copyright (c) 2007 - 2017, Intel Corporation. All rights reserved. +# Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved. # This program and the accompanying materials # are licensed and made available under the terms and conditions of the BSD License # which accompanies this distribution. The full text of the license may be found at @@ -20,19 +20,19 @@ SUBDIRS = $(BASE_TOOLS_PATH)\Source\C $(BASE_TOOLS_PATH)\Source\Python all: c python c : - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $(BASE_TOOLS_PATH)\Source\C + @$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all $(BASE_TOOLS_PATH)\Source\C python: - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $(BASE_TOOLS_PATH)\Source\Python + @$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all $(BASE_TOOLS_PATH)\Source\Python subdirs: $(SUBDIRS) - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $** + @$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py all $** .PHONY: clean clean: - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat clean $(SUBDIRS) + $(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py clean $(SUBDIRS) .PHONY: cleanall cleanall: - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat cleanall $(SUBDIRS) + $(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.py cleanall $(SUBDIRS) diff --git a/BaseTools/Source/C/Makefile b/BaseTools/Source/C/Makefile index 5428180..1246d23 100644 --- a/BaseTools/Source/C/Makefile +++ b/BaseTools/Source/C/Makefile @@ -1,7 +1,7 @@ ## @file # Windows makefile for C tools build. # -# Copyright (c) 2009 - 2017, Intel Corporation. All rights reserved. +# Copyright (c) 2009 - 2018, Intel Corporation. All rights reserved. # This program and the accompanying materials # are licensed and made available under the terms and conditions of the BSD License # which accompanies this distribution. The full text of the license may be found at @@ -16,7 +16,7 @@ HOST_ARCH = IA32 LIBRARIES = Common APPLICATIONS = \ - BootSectImage \ + VfrCompile \ BrotliCompress \ EfiLdrImage \ EfiRom \ @@ -32,7 +32,7 @@ APPLICATIONS = \ Split \ TianoCompress \ VolInfo \ - VfrCompile \ + BootSectImage \ DevicePath all: libs apps install @@ -43,7 +43,7 @@ libs: $(LIBRARIES) @echo # Build libraries @echo ## @if not exist $(LIB_PATH) mkdir $(LIB_PATH) - @Makefiles\NmakeSubdirs.bat all $** + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py all $** apps: $(APPLICATIONS) @echo. @@ -51,7 +51,7 @@ apps: $(APPLICATIONS) @echo # Build executables @echo ## @if not exist $(BIN_PATH) mkdir $(BIN_PATH) - @Makefiles\NmakeSubdirs.bat all $** + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py all $** install: $(LIB_PATH) $(BIN_PATH) @echo. @@ -65,11 +65,11 @@ install: $(LIB_PATH) $(BIN_PATH) .PHONY: clean clean: - @Makefiles\NmakeSubdirs.bat clean $(LIBRARIES) $(APPLICATIONS) + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py clean $(LIBRARIES) $(APPLICATIONS) .PHONY: cleanall cleanall: - @Makefiles\NmakeSubdirs.bat cleanall $(LIBRARIES) $(APPLICATIONS) + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py cleanall $(LIBRARIES) $(APPLICATIONS) !INCLUDE Makefiles\ms.rule diff --git a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py new file mode 100644 index 000..29bb5df --- /dev/null +++ b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py @@ -0,0 +1,169 @@ +# @file