[edk2-devel] [PATCH] BaseTools:Fix the library dependency missing in Binary Cache

2019-05-28 Thread Steven Shi
: Liming Gao Cc: Bob Feng Cc: Christian Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python/build/build.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py index 80ceb98310..673a9379ba

[edk2-devel] [PATCH] BaseTools:Extend the binary cache to support library cache

2019-05-28 Thread Steven Shi
library. Cc: Liming Gao Cc: Bob Feng Cc: Christian Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++--- BaseTools/Source/Python/build/build.py | 3 ++- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/BaseTools/Source

Re: [edk2-devel] [PATCH] BaseTools:Fix the library dependency missing in Binary Cache

2019-05-28 Thread Steven Shi
OK, will solve this issue through the fix patch of BZ 1797. Thanks Steven Shi Intel\SSG\FID\Firmware Infrastructure > -Original Message- > From: Rodriguez, Christian > Sent: Tuesday, May 28, 2019 11:06 PM > To: devel@edk2.groups.io; Shi, Steven > Cc: Gao, Limin

Re: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to support library cache

2019-05-28 Thread Steven Shi
OK, will send the new version patch to simplify it. Thanks Steven Shi Intel\SSG\FID\Firmware Infrastructure > -Original Message- > From: Rodriguez, Christian > Sent: Tuesday, May 28, 2019 11:16 PM > To: devel@edk2.groups.io; Shi, Steven > Cc: Gao, Liming ; Feng, Bob C

[edk2-devel] [PATCH v2] BaseTools:Extend the binary cache to support library cache

2019-05-28 Thread Steven Shi
module. If a driver module cache miss happen, all its dependency library modules need rebuild which is very time-consuming. This patch is to entend the binary cache to support library. Cc: Liming Gao Cc: Bob Feng Cc: Christian Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python

Re: [edk2-devel] [PATCH v2] BaseTools:Extend the binary cache to support library cache

2019-05-29 Thread Steven Shi
OK, will move the change notices from commit message to cover letter. Thanks Steven Shi > -Original Message- > From: Leif Lindholm [mailto:leif.lindh...@linaro.org] > Sent: Wednesday, May 29, 2019 7:14 PM > To: devel@edk2.groups.io; Shi, Steven > Cc: Gao, Limin

[edk2-devel] [PATCH v3 0/1] BaseTools:Extend the binary cache to support library

2019-05-29 Thread Steven Shi
Steven Shi (1): BaseTools:Extend the binary cache to support library cache BaseTools/Source/Python/AutoGen/AutoGen.py | 6 ++ 1 file changed, 6 insertions(+) -- 2.17.1.windows.2 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#41654

[edk2-devel] [PATCH v3 1/1] BaseTools:Extend the binary cache to support library cache

2019-05-29 Thread Steven Shi
library. Cc: Liming Gao Cc: Bob Feng Cc: Christian Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGen.py | 6 ++ 1 file changed, 6 insertions(+) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py index

Re: [edk2-devel] [PATCH v3 1/1] BaseTools:Extend the binary cache to support library cache

2019-05-30 Thread Steven Shi
Yes. The "entend" should be "extend", please help to fix it when pushing. Thanks! Steven Shi > -Original Message- > From: Feng, Bob C > Sent: Thursday, May 30, 2019 10:44 AM > To: devel@edk2.groups.io; Shi, Steven > Cc: Gao, Liming ; Rodriguez, Chris

[edk2-devel] [PATCH] BaseTools:Update binary cache restore time to current time

2019-05-28 Thread Steven Shi
on the time stamp. Enhance to update the restored file creation time to current time. Cc: Liming Gao Cc: Bob Feng Cc: Zhiju Fan Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGen.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/BaseTools/Source

Re: [edk2-devel] [PATCH] BaseTools: Cannot store library cache of different arch together

2019-06-14 Thread Steven Shi
Hi Christian, I think about BZ 1895 again, and your suggestion of "change the hash to include the arch" is better. I've sent the v2 patch to enhance it. BTW, the CopyFileOnChange() is still necessary for BZ 1894. Thanks Steven Shi Intel\SSG\FID\Firmware Infrastructure From:

[edk2-devel] [PATCH v2 1/1] BaseTools:Build Cache output notification message

2019-06-14 Thread Steven Shi
message about the cache miss or hit when build consumes the cache. Cc: Liming Gao Cc: Bob Feng Cc: Zhiju.Fan Signed-off-by: Steven Shi --- BaseTools/Source/Python/build/build.py | 10 ++ 1 file changed, 10 insertions(+) diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools

[edk2-devel] [PATCH v2 0/1] BaseTools:Build Cache output notification message

2019-06-14 Thread Steven Shi
V2: Only output message for the driver module and not for library, which can filter duplicated library messages from different driver modules, and make the cache messages look more concise. V1: Initial fix from Zhiju Steven Shi (1): BaseTools:Build Cache output notification message

[edk2-devel] [PATCH v2 0/1] BaseTools: Cannot store library cache of different arch together

2019-06-14 Thread Steven Shi
Steven Shi (1): BaseTools: Cannot store library cache of different arch together BaseTools/Source/Python/AutoGen/AutoGen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.17.1.windows.2 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply

[edk2-devel] [PATCH] BaseTools:Introduce CopyFileOnChange() function to copy cache files

2019-06-14 Thread Steven Shi
. This patch introduces an atomic copy function to avoid duplicated cache files copy. Cc: Liming Gao Cc: Bob Feng Cc: Christian Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGen.py | 11 ++- BaseTools/Source/Python/Common/LongFilePathOs.py | 4

[edk2-devel] [PATCH v2 1/1] BaseTools: Cannot store library cache of different arch together

2019-06-14 Thread Steven Shi
: Christian Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py index 3f41fbb507..1fe8312d22 100644 --- a/BaseTools

Re: [edk2-devel] [PATCH] BaseTools: Cannot store library cache of different arch together

2019-06-11 Thread Steven Shi
h avoid to use the set() as the container to save the library and > >module objects because the objects might have same __hash__ value. > > > >Cc: Liming Gao mailto:liming@intel.com>> > >Cc: Bob Feng mailto:bob.c.f...@intel.com>> > >Cc: Christian R

Re: [edk2-devel] [PATCH] BaseTools: Cannot store library cache of different arch together

2019-06-11 Thread Steven Shi
Sorry, the CopyFileOnChange() will ensure only once IO store/restore writing for each library. The extra IO read is ok. Thanks Steven Shi Intel\SSG\FID\Firmware Infrastructure From: Shi, Steven Sent: Wednesday, June 12, 2019 11:24 AM To: Rodriguez, Christian ; devel@edk2.groups.io Cc: Gao

[edk2-devel] [PATCH v2 0/1] BaseTools:Build cache cannot store the cache files for library package

2019-06-11 Thread Steven Shi
V2: As Bob suggestion to remove the "pass" in except block. Enhance the comments as well V1: Initial fix from Zhiju Steven Shi (1): BaseTools:Build cache cannot store the cache files for library package BaseTools/Source/Python/AutoGen/AutoGen.py | 5 - 1 file changed, 4 insert

[edk2-devel] [PATCH v2 1/1] BaseTools:Build cache cannot store the cache files for library package

2019-06-11 Thread Steven Shi
build cache files (e.g. .hash and .lib) are stored in the BinCache folder. But the BinCache folder is not generated at all. This patch is going to fix that issue. Cc: Liming Gao Cc: Bob Feng Cc: Zhiju.Fan Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGen.py | 5 - 1

[edk2-devel] [PATCH] BaseTools: Cannot store library cache of different arch together

2019-06-10 Thread Steven Shi
Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python/build/build.py | 13 +++-- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py index 0855d4561c..f7a79cbbab 100644 --- a/BaseTools/Source

[edk2-devel] [PATCH] BaseTools:Change build cache option name from --binary- to --cache-

2019-06-17 Thread Steven Shi
https://bugzilla.tianocore.org/show_bug.cgi?id=1896 Current build cache is enabled by the below two options: --binary-destination, which is to produce the cache. --binary-source, which is to consume the cache. Since current build cache does not only support to store the binary file (e.g.

Re: [edk2-devel] [Patch 2/2] BaseTools: Move Build Cache related function out of CreateAsBuiltInf

2019-06-24 Thread Steven Shi
Reviewed-by: Steven Shi > -Original Message- > From: Feng, Bob C > Sent: Monday, June 24, 2019 9:52 AM > To: devel@edk2.groups.io > Cc: Feng, Bob C ; Shi, Steven ; > Gao, Liming ; Rodriguez, Christian > > Subject: [Patch 2/2] BaseTools: Move Build Ca

Re: [edk2-devel] [Patch 1/2] BaseTools: refine CreateAsBuiltInf function

2019-06-24 Thread Steven Shi
Reviewed-by: Steven Shi > -Original Message- > From: Feng, Bob C > Sent: Monday, June 24, 2019 9:51 AM > To: devel@edk2.groups.io > Cc: Gao, Liming ; Shi, Steven ; > Rodriguez, Christian ; Feng, Bob C > > Subject: [Patch 1/2] BaseTools: refine CreateAsBuiltInf

[edk2-devel] [PATCH v2 0/1] BaseTools:Introduce CopyFileOnChange() function to copy cache files

2019-06-17 Thread Steven Shi
V2: set shallow=False in filecmp.cmp() to compare the src and dsc files content use shutil.copy() replace the shutil.copyfile() to do the file copy, which is better to reserve the dsc file metadata V1: Initial fix Steven Shi (1): BaseTools:Introduce CopyFileOnChange() function to copy cache

[edk2-devel] [PATCH v2 1/1] BaseTools:Introduce CopyFileOnChange() function to copy cache files

2019-06-17 Thread Steven Shi
. This patch introduces an atomic copy function to avoid duplicated cache files copy. Cc: Liming Gao Cc: Bob Feng Cc: Christian Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGen.py | 10 +- BaseTools/Source/Python/Common/LongFilePathOs.py | 4

[edk2-devel] [PATCH v3 0/1] BaseTools: Cannot store library cache of different arch together

2019-06-17 Thread Steven Shi
suggestion, Still keep the set() usage, but instead to enahnce the AutoGen object __hash__ definition to be unique by including the AutoGen object arch string into the __hash_ values. V1: Avoid to use the set() as the container to save the library and module objects Steven Shi (1): BaseTools

[edk2-devel] [PATCH v3 1/1] BaseTools: Cannot store library cache of different arch together

2019-06-17 Thread Steven Shi
: Liming Gao Cc: Bob Feng Cc: Christian Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGen.py | 21 + 1 file changed, 21 insertions(+) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py index

Re: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain for new LLVM/CLANG8

2019-04-30 Thread Steven Shi
equirements. See the background detail here: http://lists.llvm.org/pipermail/llvm-dev/2019-January/129749.html Thanks Steven Shi Intel\SSG\FID\Firmware Infrastructure > -Original Message- > From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of Leif > Lindholm > S

[edk2-devel] [PATCH 0/2] Add batch option to support build Binary Cache

2019-05-08 Thread Steven Shi
This patch serial is to extend the options in the Kabylake and Purley build batch file to support build Binary Cache produce and consume switch. Steven Shi (2): KabylakeOpenBoardPkg/KabylakeRvp3: Extend build to support Binary Cache PurleyOpenBoardPkg/BoardMtOlympus: Extend build

[edk2-devel] [PATCH 1/2] KabylakeOpenBoardPkg/KabylakeRvp3: Extend build to support Binary Cache

2019-05-08 Thread Steven Shi
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1784 Need extend the options in the Kabylake build bld.bat file to support Binary Cache. --- .../KabylakeOpenBoardPkg/KabylakeRvp3/bld.bat | 24 +-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git

[edk2-devel] [PATCH 2/2] PurleyOpenBoardPkg/BoardMtOlympus: Extend build to support Binary Cache

2019-05-08 Thread Steven Shi
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=1785 Extend the options in the Purley build batch file to support Binary Cache produce and consume switch. --- .../PurleyOpenBoardPkg/BoardMtOlympus/bld.bat | 18 -- .../BoardMtOlympus/prebuild.bat| 4 ++-- 2

Re: [edk2-devel] [PATCH 0/2] Add batch option to support build Binary Cache

2019-05-08 Thread Steven Shi
as default + SET BINARY_CACHE_PATH=BinCache +) + Thanks Steven Shi > -Original Message- > From: Gao, Liming > Sent: Thursday, May 9, 2019 11:37 AM > To: Shi, Steven ; devel@edk2.groups.io > Cc: Chiu, Chasel ; Lu, Shifei A > ; > Bi, Dandan ; Kubacki, Michael A > > S

[edk2-devel] [PATCH] BaseTools: Skip .mcb file module in Binary Cache

2019-05-08 Thread Steven Shi
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1723 Current Kabylake open platform will build fail if enabled to consume the binary cache, because the binary cache doesn't support to recovery the .mcb microcode file, e.g. m80406E8_0026.mcb, in a platform level folder which is outside of

Re: [edk2-devel] [PATCH] BaseTools: Skip .mcb file module in Binary Cache

2019-05-08 Thread Steven Shi
ild Cache directly. Thanks Steven Shi > -Original Message- > From: Gao, Liming > Sent: Thursday, May 9, 2019 1:47 PM > To: Yao, Jiewen ; devel@edk2.groups.io; Shi, Steven > > Cc: Feng, Bob C ; Rodriguez, Christian > ; Johnson, Michael > > Subject: RE: [edk2-deve

[edk2-devel] [PATCH v6 2/5] BaseTools: Print first cache missing file for build cachle

2019-08-15 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1925 When a module build cache miss, add support to print the first cache missing file path and name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWork

[edk2-devel] [PATCH v6 0/5] Build cache enhancement

2019-08-15 Thread Steven Shi
From: "Shi, Steven" This patch set is for the 201908 stable tag Enhance the edk2 build cache with below patches: Patch 01/05: Improve the cache hit rate through new cache checkpoint and hash algorithm Patch 02/05: Print more info to explain why a module build cache miss Patch 03/05: Fix the

[edk2-devel] [PATCH v6 3/5] BaseTools: Change the [Arch][Name] module key in Build cache

2019-08-15 Thread Steven Shi
j __hash__ and __repr__ attributes already contain the full path and arch name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/GenMake.py | 6 +- BaseTools/Source/Python/build/build.py | 48 ---

[edk2-devel] [PATCH v6 5/5] BaseTools: Improve the file saving and copying reliability

2019-08-15 Thread Steven Shi
ge() might be invoked in multiple sub-processes simultaneously, and this patch adds global locks to sync these functions invoking which can harden their reliability. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 6 -- BaseTo

[edk2-devel] [PATCH v6 1/5] BaseTools: Improve the cache hit in the edk2 build cache

2019-08-15 Thread Steven Shi
leverages the module makefile to achieve more accurate and precise dependency info for a module. When the build cache miss with the first quick hash, the Basetool will caculate new one after makefile is generated and then check again. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTo

[edk2-devel] [PATCH v6 4/5] BaseTools: Add GenFds multi-thread support in build cache

2019-08-15 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1923 Fix the issue that the GenFds multi-thread will build fail if enable the build cache together. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/ModuleAutoG

[edk2-devel] [PATCH v3 1/5] BaseTools: Improve the cache hit in the edk2 build cache

2019-08-13 Thread Steven Shi
leverages the module makefile to achieve more accurate and precise dependency info for a module. When the build cache miss with the first quick hash, the Basetool will caculate new one after makefile is generated and then check again. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTo

[edk2-devel] [PATCH v3 5/5] BaseTools: Improve the file saving and copying reliability

2019-08-13 Thread Steven Shi
ge() might be invoked in multiple sub-processes simultaneously, and this patch adds a global lock to sync these functions invoking which can harden their reliability. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/GenC.py | 2 +- BaseTools/Sou

[edk2-devel] [PATCH v3 3/5] BaseTools: Change the [Arch][Name] module key in Build cache

2019-08-13 Thread Steven Shi
j __hash__ and __repr__ attributes already contain the full path and arch name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/GenMake.py | 6 +- BaseTools/Source/Python/build/build.py | 49 +---

[edk2-devel] [PATCH v3 2/5] BaseTools: Print first cache missing file for build cachle

2019-08-13 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1925 When a module build cache miss, add support to print the first cache missing file path and name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWork

[edk2-devel] [PATCH v3 0/5] Build cache enhancement

2019-08-13 Thread Steven Shi
From: "Shi, Steven" Enhance the edk2 build cache with below patches: Patch 01/05: Improve the cache hit rate through new cache checkpoint and hash algorithm Patch 02/05: Print more info to explain why a module build cache miss Patch 03/05: Fix the unsafe [self.Arch][self.Name] key usage in

[edk2-devel] [PATCH v3 4/5] BaseTools: Add GenFds multi-thread support in build cache

2019-08-13 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1923 Fix the issue that the GenFds multi-thread will build fail if enable the build cache together. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/ModuleAutoG

[edk2-devel] [PATCH v2 1/4] BaseTools: Improve the cache hit in the edk2 build cache

2019-08-12 Thread Steven Shi
leverages the module makefile to achieve more accurate and precise dependency info for a module. When the build cache miss with the first quick hash, the Basetool will caculate new one after makefile is generated and then check again. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTo

[edk2-devel] [PATCH v2 4/4] BaseTools: Add GenFds multi-thread support in build cache

2019-08-12 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1923 Fix the issue that the GenFds multi-thread will build fail if enable the build cache together. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/ModuleAutoG

[edk2-devel] [PATCH v2 2/4] BaseTools: Print first cache missing file for build cachle

2019-08-12 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1925 When a module build cache miss, add support to print the first cache missing file path and name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWork

[edk2-devel] [PATCH v2 0/4] Build cache enhancement

2019-08-12 Thread Steven Shi
From: "Shi, Steven" Enhance the edk2 build cache with below patches: Patch 01/04: Improve the cache hit rate through new cache checkpoint and hash algorithm Patch 02/04: Print more info to explain why a module build cache miss Patch 03/04: Fix the unsafe [self.Arch][self.Name] key usage in

[edk2-devel] [PATCH v2 3/4] BaseTools: Change the [Arch][Name] module key in Build cache

2019-08-12 Thread Steven Shi
j __hash__ and __repr__ attributes already contain the full path and arch name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/GenMake.py | 6 +- BaseTools/Source/Python/build/build.py | 49 +---

[edk2-devel] [PATCH v5 5/5] BaseTools: Improve the file saving and copying reliability

2019-08-14 Thread Steven Shi
ge() might be invoked in multiple sub-processes simultaneously, and this patch adds global locks to sync these functions invoking which can harden their reliability. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 4 +++- BaseTo

[edk2-devel] [PATCH v5 1/5] BaseTools: Improve the cache hit in the edk2 build cache

2019-08-14 Thread Steven Shi
leverages the module makefile to achieve more accurate and precise dependency info for a module. When the build cache miss with the first quick hash, the Basetool will caculate new one after makefile is generated and then check again. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTo

[edk2-devel] [PATCH v5 3/5] BaseTools: Change the [Arch][Name] module key in Build cache

2019-08-14 Thread Steven Shi
j __hash__ and __repr__ attributes already contain the full path and arch name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/GenMake.py | 6 +- BaseTools/Source/Python/build/build.py | 48 ---

[edk2-devel] [PATCH v5 4/5] BaseTools: Add GenFds multi-thread support in build cache

2019-08-14 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1923 Fix the issue that the GenFds multi-thread will build fail if enable the build cache together. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/ModuleAutoG

[edk2-devel] [PATCH v5 2/5] BaseTools: Print first cache missing file for build cachle

2019-08-14 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1925 When a module build cache miss, add support to print the first cache missing file path and name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWork

[edk2-devel] [PATCH v5 0/5] Build cache enhancement

2019-08-14 Thread Steven Shi
From: "Shi, Steven" This patch set is for the 201908 stable tag Enhance the edk2 build cache with below patches: Patch 01/05: Improve the cache hit rate through new cache checkpoint and hash algorithm Patch 02/05: Print more info to explain why a module build cache miss Patch 03/05: Fix the

[edk2-devel] [PATCH v4 0/5] Build cache enhancement

2019-08-14 Thread Steven Shi
From: "Shi, Steven" Enhance the edk2 build cache with below patches: Patch 01/05: Improve the cache hit rate through new cache checkpoint and hash algorithm Patch 02/05: Print more info to explain why a module build cache miss Patch 03/05: Fix the unsafe [self.Arch][self.Name] key usage in

[edk2-devel] [PATCH v4 3/5] BaseTools: Change the [Arch][Name] module key in Build cache

2019-08-14 Thread Steven Shi
j __hash__ and __repr__ attributes already contain the full path and arch name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/GenMake.py | 6 +- BaseTools/Source/Python/build/build.py | 48 ---

[edk2-devel] [PATCH v4 1/5] BaseTools: Improve the cache hit in the edk2 build cache

2019-08-14 Thread Steven Shi
leverages the module makefile to achieve more accurate and precise dependency info for a module. When the build cache miss with the first quick hash, the Basetool will caculate new one after makefile is generated and then check again. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTo

[edk2-devel] [PATCH v4 5/5] BaseTools: Improve the file saving and copying reliability

2019-08-14 Thread Steven Shi
ge() might be invoked in multiple sub-processes simultaneously, and this patch adds a global lock to sync these functions invoking which can harden their reliability. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 4 +++- BaseTo

[edk2-devel] [PATCH v4 4/5] BaseTools: Add GenFds multi-thread support in build cache

2019-08-14 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1923 Fix the issue that the GenFds multi-thread will build fail if enable the build cache together. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/ModuleAutoG

[edk2-devel] [PATCH v4 2/5] BaseTools: Print first cache missing file for build cachle

2019-08-14 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1925 When a module build cache miss, add support to print the first cache missing file path and name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWork

Re: [edk2-devel] [Patch 0/9] Enable multiple process AutoGen

2019-07-21 Thread Steven Shi
is multi-process build tool fails to re-generate the new autogen code (CreateCodeFile) and makefile (CreateMakeFile). The new build tool wrongly skip these steps in incremental build. Thanks Steven Shi > -Original Message- > From: devel@edk2.groups.io [mailto:devel@edk2.grou

[edk2-devel] [PATCH 1/4] BaseTools: Improve the cache hit in the edk2 build cache

2019-08-08 Thread Steven Shi
leverages the module makefile to achieve more accurate and precise dependency info for a module. When the build cache miss with the first quick hash, the Basetool will caculate new one after makefile is generated and then check again. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTo

[edk2-devel] [PATCH 3/4] BaseTools: Change the [Arch][Name] module key in Build cache

2019-08-08 Thread Steven Shi
j __hash__ and __repr__ attributes already contain the full path and arch name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/GenMake.py | 6 +- BaseTools/Source/Python/build/build.py | 49 +---

[edk2-devel] [PATCH 4/4] BaseTools: Add GenFds multi-thread support in build cache

2019-08-08 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1923 Fix the issue that the GenFds multi-thread will build fail if enable the build cache together. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/ModuleAutoG

[edk2-devel] [PATCH 0/4] Build cache enhancement

2019-08-08 Thread Steven Shi
From: "Shi, Steven" Enhance the edk2 build cache with below patches: Patch 01/04: Improve the cache hit rate through new cache checkpoint and hash algorithm Patch 02/04: Print more info to explain why a module build cache miss Patch 03/04: Fix the unsafe [self.Arch][self.Name] key usage in

[edk2-devel] [PATCH 2/4] BaseTools: Print first cache missing file for build cachle

2019-08-08 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1925 When a module build cache miss, add support to print the first cache missing file path and name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWork

[edk2-devel] [PATCH] BaseTools: Support long file path in windows for misc functions

2019-08-21 Thread Steven Shi
y and save in windows. This patch enhances them to support the long file path copy and save correctly. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/Common/Misc.py | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/BaseTools/Sou

[edk2-devel] [PATCH] [edk2-stable201908] BaseTools: Support long file path in windows for misc functions

2019-08-21 Thread Steven Shi
y and save in windows. This patch enhances them to support the long file path copy and save correctly. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/Common/Misc.py | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/BaseTools/Sou

[edk2-devel] [PATCH] BaseTools: Support more file types in build cache

2019-08-26 Thread Steven Shi
From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1958 Current build cache does not store and restore all types file of Hii/vfr, version and dpx. This patch adds more file types to support them in build cache. Cc: Liming Gao Cc: Bob Feng Signed-off-by:

Re: [edk2-devel] commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error

2019-09-06 Thread Steven Shi
Hi Vladimir, Did you enable the build cache? Could you let me know your build steps and commands? Thanks Steven From: Vladimir Olovyannikov [mailto:vladimir.olovyanni...@broadcom.com] Sent: Saturday, September 7, 2019 4:45 AM To: devel@edk2.groups.io; Shi, Steven Cc: Gao, Liming ; Feng, Bob C

Re: [edk2-devel] commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error

2019-09-09 Thread Steven Shi
--binary-destination=BinCache · Remove the Build folder $ rm Build/ -r –f · Consume the build cache: $ build -p OvmfPkg/OvmfPkgIa32.dsc -a IA32 -t GCC5 -b DEBUG --hash --binary-source=BinCache Thanks Steven Shi Intel\SSG\FID\Firmware Infrastructure From: Vladimir Olovyannikov