Re: [edk2-devel] [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker for parallel

2020-08-13 Thread Liming Gao 
A little better is also OK. Reviewed-by: Liming Gao 

-Original Message-
From: Feng, Bob C  
Sent: 2020年8月13日 17:01
To: Gao, Liming ; devel@edk2.groups.io
Cc: Chen, Christine 
Subject: RE: [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker for 
parallel

Liming, I did the test on Ovmf, there is some improvement but not much.

Thanks,
Bob

-Original Message-
From: Gao, Liming 
Sent: Wednesday, August 12, 2020 10:35 PM
To: Feng, Bob C ; devel@edk2.groups.io
Cc: Chen, Christine 
Subject: RE: [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker for 
parallel

I like this idea. Is there any build performance improvement?

> -Original Message-
> From: Feng, Bob C 
> Sent: Wednesday, August 12, 2020 2:09 PM
> To: devel@edk2.groups.io
> Cc: Gao, Liming ; Chen, Christine 
> 
> Subject: [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker 
> for parallel
> 
> AsBuiltInf can be created during AutoGen phase. Move CreateAsBuiltInf 
> into AutoGenWorker to make this step run in parallel.
> 
> Signed-off-by: Bob Feng 
> Cc: Liming Gao 
> Cc: Yuwei Chen 
> ---
>  BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 2 +-
>  BaseTools/Source/Python/build/build.py   | 4 +---
>  2 files changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> index 017f676399..0425f1ab0b 100755
> --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> @@ -278,11 +278,11 @@ class AutoGenWorkerInProcess(mp.Process):
>  else:
>  self.cache_q.put((Ma.MetaFile.Path, Ma.Arch, 
> "PreMakeCache", False))
> 
>  Ma.CreateCodeFile(False)
>
> Ma.CreateMakeFile(False,GenFfsList=FfsCmd.get((Ma.MetaFile.Path,
> Ma.Arch),[]))
> -
> +Ma.CreateAsBuiltInf()
>  if GlobalData.gBinCacheSource and CommandTarget in [None, 
> "", "all"]:
>  try:
>  CacheResult = Ma.CanSkipbyMakeCache()
>  except:
>  CacheResult = False diff --git 
> a/BaseTools/Source/Python/build/build.py
> b/BaseTools/Source/Python/build/build.py
> index 59ceacfed0..c4cfe38ad9 100755
> --- a/BaseTools/Source/Python/build/build.py
> +++ b/BaseTools/Source/Python/build/build.py
> @@ -871,11 +871,11 @@ class Build():
>  if GlobalData.gUseHashCache and not 
> GlobalData.gBinCacheDest and self.Target in [None, "", "all"]:
>  cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch, 
> "PreMakeCache", False))
> 
>  PcdMa.CreateCodeFile(False)
>  PcdMa.CreateMakeFile(False,GenFfsList = 
> DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.Path, PcdMa.Arch),[]))
> -
> +PcdMa.CreateAsBuiltInf()
>  # Force cache miss for PCD driver
>  if GlobalData.gBinCacheSource and self.Target in [None, 
> "", "all"]:
>  cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch, 
> "MakeCache", False))
> 
>  self.AutoGenMgr.join()
> @@ -1263,11 +1263,10 @@ class Build():
> 
>  # build modules
>  if BuildModule:
>  BuildCommand = BuildCommand + [Target]
>  LaunchCommand(BuildCommand, AutoGenObject.MakeFileDir)
> -self.CreateAsBuiltInf()
>  if GlobalData.gBinCacheDest:
>  self.GenDestCache()
>  elif GlobalData.gUseHashCache and not GlobalData.gBinCacheSource:
>  # Only for --hash
>  # Update PreMakeCacheChain files @@ -2274,11 +2273,10 
> @@ class Build():
>  # All modules have been put in build tasks queue. Tell task 
> scheduler
>  # to exit if all tasks are completed
>  #
>  ExitFlag.set()
>  BuildTask.WaitForComplete()
> -self.CreateAsBuiltInf()
>  if GlobalData.gBinCacheDest:
>  self.GenDestCache()
>  elif GlobalData.gUseHashCache and not 
> GlobalData.gBinCacheSource:
>  # Only for --hash
>  # Update PreMakeCacheChain files
> --
> 2.20.1.windows.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#64156): https://edk2.groups.io/g/devel/message/64156
Mute This Topic: https://groups.io/mt/76142326/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker for parallel

2020-08-13 Thread Bob Feng 
Liming, I did the test on Ovmf, there is some improvement but not much.

Thanks,
Bob

-Original Message-
From: Gao, Liming  
Sent: Wednesday, August 12, 2020 10:35 PM
To: Feng, Bob C ; devel@edk2.groups.io
Cc: Chen, Christine 
Subject: RE: [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker for 
parallel

I like this idea. Is there any build performance improvement?

> -Original Message-
> From: Feng, Bob C 
> Sent: Wednesday, August 12, 2020 2:09 PM
> To: devel@edk2.groups.io
> Cc: Gao, Liming ; Chen, Christine 
> 
> Subject: [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker 
> for parallel
> 
> AsBuiltInf can be created during AutoGen phase. Move CreateAsBuiltInf 
> into AutoGenWorker to make this step run in parallel.
> 
> Signed-off-by: Bob Feng 
> Cc: Liming Gao 
> Cc: Yuwei Chen 
> ---
>  BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 2 +-
>  BaseTools/Source/Python/build/build.py   | 4 +---
>  2 files changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py 
> b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> index 017f676399..0425f1ab0b 100755
> --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> @@ -278,11 +278,11 @@ class AutoGenWorkerInProcess(mp.Process):
>  else:
>  self.cache_q.put((Ma.MetaFile.Path, Ma.Arch, 
> "PreMakeCache", False))
> 
>  Ma.CreateCodeFile(False)
>
> Ma.CreateMakeFile(False,GenFfsList=FfsCmd.get((Ma.MetaFile.Path, 
> Ma.Arch),[]))
> -
> +Ma.CreateAsBuiltInf()
>  if GlobalData.gBinCacheSource and CommandTarget in [None, 
> "", "all"]:
>  try:
>  CacheResult = Ma.CanSkipbyMakeCache()
>  except:
>  CacheResult = False diff --git 
> a/BaseTools/Source/Python/build/build.py 
> b/BaseTools/Source/Python/build/build.py
> index 59ceacfed0..c4cfe38ad9 100755
> --- a/BaseTools/Source/Python/build/build.py
> +++ b/BaseTools/Source/Python/build/build.py
> @@ -871,11 +871,11 @@ class Build():
>  if GlobalData.gUseHashCache and not 
> GlobalData.gBinCacheDest and self.Target in [None, "", "all"]:
>  cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch, 
> "PreMakeCache", False))
> 
>  PcdMa.CreateCodeFile(False)
>  PcdMa.CreateMakeFile(False,GenFfsList = 
> DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.Path, PcdMa.Arch),[]))
> -
> +PcdMa.CreateAsBuiltInf()
>  # Force cache miss for PCD driver
>  if GlobalData.gBinCacheSource and self.Target in [None, 
> "", "all"]:
>  cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch, 
> "MakeCache", False))
> 
>  self.AutoGenMgr.join()
> @@ -1263,11 +1263,10 @@ class Build():
> 
>  # build modules
>  if BuildModule:
>  BuildCommand = BuildCommand + [Target]
>  LaunchCommand(BuildCommand, AutoGenObject.MakeFileDir)
> -self.CreateAsBuiltInf()
>  if GlobalData.gBinCacheDest:
>  self.GenDestCache()
>  elif GlobalData.gUseHashCache and not GlobalData.gBinCacheSource:
>  # Only for --hash
>  # Update PreMakeCacheChain files @@ -2274,11 +2273,10 
> @@ class Build():
>  # All modules have been put in build tasks queue. Tell task 
> scheduler
>  # to exit if all tasks are completed
>  #
>  ExitFlag.set()
>  BuildTask.WaitForComplete()
> -self.CreateAsBuiltInf()
>  if GlobalData.gBinCacheDest:
>  self.GenDestCache()
>  elif GlobalData.gUseHashCache and not 
> GlobalData.gBinCacheSource:
>  # Only for --hash
>  # Update PreMakeCacheChain files
> --
> 2.20.1.windows.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#64146): https://edk2.groups.io/g/devel/message/64146
Mute This Topic: https://groups.io/mt/76142326/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker for parallel

2020-08-12 Thread Carsey, Jaben 
Reviewed-by: Jaben Carsey 

> -Original Message-
> From: devel@edk2.groups.io  On Behalf Of Bob
> Feng
> Sent: Tuesday, August 11, 2020 11:09 PM
> To: devel@edk2.groups.io
> Cc: Gao, Liming ; Chen, Christine
> 
> Subject: [edk2-devel] [Patch] BaseTools: Move CreateAsBuiltInf into
> AutoGenWorker for parallel
> 
> AsBuiltInf can be created during AutoGen phase. Move CreateAsBuiltInf into
> AutoGenWorker to make this step run in parallel.
> 
> Signed-off-by: Bob Feng 
> Cc: Liming Gao 
> Cc: Yuwei Chen 
> ---
>  BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 2 +-
>  BaseTools/Source/Python/build/build.py   | 4 +---
>  2 files changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> index 017f676399..0425f1ab0b 100755
> --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> @@ -278,11 +278,11 @@ class AutoGenWorkerInProcess(mp.Process):
>  else:
>  self.cache_q.put((Ma.MetaFile.Path, Ma.Arch,
> "PreMakeCache", False))
> 
>  Ma.CreateCodeFile(False)
> 
> Ma.CreateMakeFile(False,GenFfsList=FfsCmd.get((Ma.MetaFile.Path,
> Ma.Arch),[]))
> -
> +Ma.CreateAsBuiltInf()
>  if GlobalData.gBinCacheSource and CommandTarget in [None, "",
> "all"]:
>  try:
>  CacheResult = Ma.CanSkipbyMakeCache()
>  except:
>  CacheResult = False diff --git
> a/BaseTools/Source/Python/build/build.py
> b/BaseTools/Source/Python/build/build.py
> index 59ceacfed0..c4cfe38ad9 100755
> --- a/BaseTools/Source/Python/build/build.py
> +++ b/BaseTools/Source/Python/build/build.py
> @@ -871,11 +871,11 @@ class Build():
>  if GlobalData.gUseHashCache and not 
> GlobalData.gBinCacheDest
> and self.Target in [None, "", "all"]:
>  cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch,
> "PreMakeCache", False))
> 
>  PcdMa.CreateCodeFile(False)
>  PcdMa.CreateMakeFile(False,GenFfsList =
> DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.Path, PcdMa.Arch),[]))
> -
> +PcdMa.CreateAsBuiltInf()
>  # Force cache miss for PCD driver
>  if GlobalData.gBinCacheSource and self.Target in [None, 
> "", "all"]:
>  cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch, 
> "MakeCache",
> False))
> 
>  self.AutoGenMgr.join()
> @@ -1263,11 +1263,10 @@ class Build():
> 
>  # build modules
>  if BuildModule:
>  BuildCommand = BuildCommand + [Target]
>  LaunchCommand(BuildCommand, AutoGenObject.MakeFileDir)
> -self.CreateAsBuiltInf()
>  if GlobalData.gBinCacheDest:
>  self.GenDestCache()
>  elif GlobalData.gUseHashCache and not GlobalData.gBinCacheSource:
>  # Only for --hash
>  # Update PreMakeCacheChain files @@ -2274,11 +2273,10 @@ 
> class
> Build():
>  # All modules have been put in build tasks queue. Tell task 
> scheduler
>  # to exit if all tasks are completed
>  #
>  ExitFlag.set()
>  BuildTask.WaitForComplete()
> -self.CreateAsBuiltInf()
>  if GlobalData.gBinCacheDest:
>  self.GenDestCache()
>  elif GlobalData.gUseHashCache and not
> GlobalData.gBinCacheSource:
>  # Only for --hash
>  # Update PreMakeCacheChain files
> --
> 2.20.1.windows.1
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#64124): https://edk2.groups.io/g/devel/message/64124
Mute This Topic: https://groups.io/mt/76142326/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker for parallel

2020-08-12 Thread Liming Gao 
I like this idea. Is there any build performance improvement?

> -Original Message-
> From: Feng, Bob C 
> Sent: Wednesday, August 12, 2020 2:09 PM
> To: devel@edk2.groups.io
> Cc: Gao, Liming ; Chen, Christine 
> Subject: [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker for 
> parallel
> 
> AsBuiltInf can be created during AutoGen phase. Move CreateAsBuiltInf
> into AutoGenWorker to make this step run in parallel.
> 
> Signed-off-by: Bob Feng 
> Cc: Liming Gao 
> Cc: Yuwei Chen 
> ---
>  BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 2 +-
>  BaseTools/Source/Python/build/build.py   | 4 +---
>  2 files changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py 
> b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> index 017f676399..0425f1ab0b 100755
> --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> @@ -278,11 +278,11 @@ class AutoGenWorkerInProcess(mp.Process):
>  else:
>  self.cache_q.put((Ma.MetaFile.Path, Ma.Arch, 
> "PreMakeCache", False))
> 
>  Ma.CreateCodeFile(False)
>  
> Ma.CreateMakeFile(False,GenFfsList=FfsCmd.get((Ma.MetaFile.Path, Ma.Arch),[]))
> -
> +Ma.CreateAsBuiltInf()
>  if GlobalData.gBinCacheSource and CommandTarget in [None, 
> "", "all"]:
>  try:
>  CacheResult = Ma.CanSkipbyMakeCache()
>  except:
>  CacheResult = False
> diff --git a/BaseTools/Source/Python/build/build.py 
> b/BaseTools/Source/Python/build/build.py
> index 59ceacfed0..c4cfe38ad9 100755
> --- a/BaseTools/Source/Python/build/build.py
> +++ b/BaseTools/Source/Python/build/build.py
> @@ -871,11 +871,11 @@ class Build():
>  if GlobalData.gUseHashCache and not 
> GlobalData.gBinCacheDest and self.Target in [None, "", "all"]:
>  cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch, 
> "PreMakeCache", False))
> 
>  PcdMa.CreateCodeFile(False)
>  PcdMa.CreateMakeFile(False,GenFfsList = 
> DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.Path, PcdMa.Arch),[]))
> -
> +PcdMa.CreateAsBuiltInf()
>  # Force cache miss for PCD driver
>  if GlobalData.gBinCacheSource and self.Target in [None, 
> "", "all"]:
>  cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch, 
> "MakeCache", False))
> 
>  self.AutoGenMgr.join()
> @@ -1263,11 +1263,10 @@ class Build():
> 
>  # build modules
>  if BuildModule:
>  BuildCommand = BuildCommand + [Target]
>  LaunchCommand(BuildCommand, AutoGenObject.MakeFileDir)
> -self.CreateAsBuiltInf()
>  if GlobalData.gBinCacheDest:
>  self.GenDestCache()
>  elif GlobalData.gUseHashCache and not GlobalData.gBinCacheSource:
>  # Only for --hash
>  # Update PreMakeCacheChain files
> @@ -2274,11 +2273,10 @@ class Build():
>  # All modules have been put in build tasks queue. Tell task 
> scheduler
>  # to exit if all tasks are completed
>  #
>  ExitFlag.set()
>  BuildTask.WaitForComplete()
> -self.CreateAsBuiltInf()
>  if GlobalData.gBinCacheDest:
>  self.GenDestCache()
>  elif GlobalData.gUseHashCache and not 
> GlobalData.gBinCacheSource:
>  # Only for --hash
>  # Update PreMakeCacheChain files
> --
> 2.20.1.windows.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#64076): https://edk2.groups.io/g/devel/message/64076
Mute This Topic: https://groups.io/mt/76142326/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [Patch] BaseTools: Move CreateAsBuiltInf into AutoGenWorker for parallel

2020-08-11 Thread Bob Feng 
AsBuiltInf can be created during AutoGen phase. Move CreateAsBuiltInf
into AutoGenWorker to make this step run in parallel.

Signed-off-by: Bob Feng 
Cc: Liming Gao 
Cc: Yuwei Chen 
---
 BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 2 +-
 BaseTools/Source/Python/build/build.py   | 4 +---
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py 
b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
index 017f676399..0425f1ab0b 100755
--- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
@@ -278,11 +278,11 @@ class AutoGenWorkerInProcess(mp.Process):
 else:
 self.cache_q.put((Ma.MetaFile.Path, Ma.Arch, 
"PreMakeCache", False))
 
 Ma.CreateCodeFile(False)
 
Ma.CreateMakeFile(False,GenFfsList=FfsCmd.get((Ma.MetaFile.Path, Ma.Arch),[]))
-
+Ma.CreateAsBuiltInf()
 if GlobalData.gBinCacheSource and CommandTarget in [None, "", 
"all"]:
 try:
 CacheResult = Ma.CanSkipbyMakeCache()
 except:
 CacheResult = False
diff --git a/BaseTools/Source/Python/build/build.py 
b/BaseTools/Source/Python/build/build.py
index 59ceacfed0..c4cfe38ad9 100755
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -871,11 +871,11 @@ class Build():
 if GlobalData.gUseHashCache and not 
GlobalData.gBinCacheDest and self.Target in [None, "", "all"]:
 cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch, 
"PreMakeCache", False))
 
 PcdMa.CreateCodeFile(False)
 PcdMa.CreateMakeFile(False,GenFfsList = 
DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.Path, PcdMa.Arch),[]))
-
+PcdMa.CreateAsBuiltInf()
 # Force cache miss for PCD driver
 if GlobalData.gBinCacheSource and self.Target in [None, 
"", "all"]:
 cqueue.put((PcdMa.MetaFile.Path, PcdMa.Arch, 
"MakeCache", False))
 
 self.AutoGenMgr.join()
@@ -1263,11 +1263,10 @@ class Build():
 
 # build modules
 if BuildModule:
 BuildCommand = BuildCommand + [Target]
 LaunchCommand(BuildCommand, AutoGenObject.MakeFileDir)
-self.CreateAsBuiltInf()
 if GlobalData.gBinCacheDest:
 self.GenDestCache()
 elif GlobalData.gUseHashCache and not GlobalData.gBinCacheSource:
 # Only for --hash
 # Update PreMakeCacheChain files
@@ -2274,11 +2273,10 @@ class Build():
 # All modules have been put in build tasks queue. Tell task 
scheduler
 # to exit if all tasks are completed
 #
 ExitFlag.set()
 BuildTask.WaitForComplete()
-self.CreateAsBuiltInf()
 if GlobalData.gBinCacheDest:
 self.GenDestCache()
 elif GlobalData.gUseHashCache and not 
GlobalData.gBinCacheSource:
 # Only for --hash
 # Update PreMakeCacheChain files
-- 
2.20.1.windows.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#64046): https://edk2.groups.io/g/devel/message/64046
Mute This Topic: https://groups.io/mt/76142326/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-