Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908

2019-10-20 Thread Steven Shi 
Hi Derek,
I figure out a new solution for –hash cache as below. Please try it again.
https://github.com/shijunjing/edk2/tree/hashcache_v2

It is based on latest edk2 master branch, and what you need porting are below 
files. Let me know if you need me to create a patch based on edk2-stable201908.

$ git status
On branch hashcache_v2
$ git show HEAD --name-only
BaseTools/Conf/build_rule.template
BaseTools/Conf/tools_def.template
BaseTools/Source/Python/AutoGen/AutoGenWorker.py
BaseTools/Source/Python/AutoGen/DataPipe.py
BaseTools/Source/Python/AutoGen/GenMake.py
BaseTools/Source/Python/AutoGen/IncludesAutoGen.py
BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
BaseTools/Source/Python/Common/GlobalData.py
BaseTools/Source/Python/build/build.py


Thanks

Steven Shi
Intel\SSG\SFE\FIE Firmware Infrastructure


From: Shi, Steven
Sent: Sunday, September 29, 2019 10:32 AM
To: Lin, Derek (HPS SW) ; devel@edk2.groups.io
Subject: RE: [edk2-devel] BaseTools --hash malfunction after migrate from 
stable201905 to stable201908

Hi Derek,
Thank you for the data. I will continue to enhance the –hash performance, and 
will let you know after I figure out new solution.


Thanks

Steven Shi
Intel\SSG\SFE\FIE Firmware Infrastructure


From: Lin, Derek (HPS SW) [mailto:derek.l...@hpe.com]
Sent: Friday, September 27, 2019 3:57 PM
To: Shi, Steven mailto:steven@intel.com>>; 
devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: Re: [edk2-devel] BaseTools --hash malfunction after migrate from 
stable201905 to stable201908

Hi Steven,

Thanks for the fix!
I've tried your patch with a minor fix as attached.
   12   3   
 4
Build Duration:00:05:51  00:03:47  00:05:15  00:04:31
AutoGen Duration:  00:03:32  00:03:08  00:03:14  00:03:16
Make Duration:   00:01:50  00:00:16  00:01:29  00:00:50
GenFds Duration:   00:00:30  00:00:22  00:00:31  00:00:23

1: clean build with patch and --hash
2: incremental build
3. clean build without patch and without --hash
4. incremental build

BTW, this data cannot compare with previous data I provided because they are 
built in different environment.

In general, the result is as you said that it fix --hash functionality, but not 
as quick as before.
Also the cost of --hash in clean build is increased.

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#49276): https://edk2.groups.io/g/devel/message/49276
Mute This Topic: https://groups.io/mt/34227864/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908

2019-09-28 Thread Steven Shi 
Hi Derek,
Thank you for the data. I will continue to enhance the –hash performance, and 
will let you know after I figure out new solution.


Thanks

Steven Shi
Intel\SSG\SFE\FIE Firmware Infrastructure


From: Lin, Derek (HPS SW) [mailto:derek.l...@hpe.com]
Sent: Friday, September 27, 2019 3:57 PM
To: Shi, Steven ; devel@edk2.groups.io
Subject: Re: [edk2-devel] BaseTools --hash malfunction after migrate from 
stable201905 to stable201908

Hi Steven,

Thanks for the fix!
I've tried your patch with a minor fix as attached.
   12   3   
 4
Build Duration:00:05:51  00:03:47  00:05:15  00:04:31
AutoGen Duration:  00:03:32  00:03:08  00:03:14  00:03:16
Make Duration:   00:01:50  00:00:16  00:01:29  00:00:50
GenFds Duration:   00:00:30  00:00:22  00:00:31  00:00:23

1: clean build with patch and --hash
2: incremental build
3. clean build without patch and without --hash
4. incremental build

BTW, this data cannot compare with previous data I provided because they are 
built in different environment.

In general, the result is as you said that it fix --hash functionality, but not 
as quick as before.
Also the cost of --hash in clean build is increased.

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#48255): https://edk2.groups.io/g/devel/message/48255
Mute This Topic: https://groups.io/mt/34227864/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908

2019-09-27 Thread Lin, Derek (HPS SW) 
Hi Steven,

Thanks for the fix!
I've tried your patch with a minor fix as attached.
1            2           3                4
Build Duration:        00:05:51  00:03:47  00:05:15  00:04:31
AutoGen Duration:  00:03:32  00:03:08  00:03:14  00:03:16
Make Duration:       00:01:50  00:00:16  00:01:29  00:00:50
GenFds Duration:   00:00:30  00:00:22  00:00:31  00:00:23

1: clean build with patch and --hash
2: incremental build
3. clean build without patch and without --hash
4. incremental build

BTW, this data cannot compare with previous data I provided because they are 
built in different environment.

In general, the result is as you said that it fix --hash functionality, but not 
as quick as before.
Also the cost of --hash in clean build is increased.

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#48208): https://edk2.groups.io/g/devel/message/48208
Mute This Topic: https://groups.io/mt/34227864/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



build.patch
Description: Binary data

Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908

2019-09-24 Thread Steven Shi 
Hi Derek,
I took a look at this issue, and add back the -hash functionality in this 
branch: https://github.com/shijunjing/edk2/tree/hashcache_v1. The attachment is 
the patch based on latest edk2.
The current -hash fix performance is not as good as the edk2-stable201905,  
because the edk2-stable201905 -hash doesn't parse source code header files and 
not include the header files as the module's hash dependency which is not 
sound. After add the header files dependency, the -hash become slow.
Let me know whether the attached fix works for you. I might continue to tune 
its performance.


Thanks

Steven Shi
Intel\SSG\FID\Firmware Infrastructure

From: Shi, Steven
Sent: Monday, September 23, 2019 3:52 PM
To: 'Lin, Derek (HPS SW)' ; devel@edk2.groups.io
Cc: Feng, Bob C ; Gao, Liming (liming@intel.com) 

Subject: RE: BaseTools --hash malfunction after migrate from stable201905 to 
stable201908

Hi Derek,
Thank you to raise this issue. We will fix it and add back the --hash original 
functionality.


Thanks
Steven

From: Lin, Derek (HPS SW) [mailto:derek.l...@hpe.com]
Sent: Friday, September 20, 2019 2:54 PM
To: devel@edk2.groups.io
Cc: Feng, Bob C mailto:bob.c.f...@intel.com>>; Shi, 
Steven mailto:steven@intel.com>>
Subject: BaseTools --hash malfunction after migrate from stable201905 to 
stable201908

Hi BaseTools experts,

We saw the clean build performance improvement after updating from 
edk2-stable201905 to edk2-stable201908, which is promising.
But we found the incremental build time increase 25%~35%, which is not good.

We're building server platform. And we use --hash to speed up incremental build 
as described in https://github.com/BobCF/edk2/wiki/Incremental-Build
After upgrading to edk2-stable201908, none of AutoGen is skipped when I did an 
incremental build without any code change.

Dig into the issue, it is caused by commit 
https://github.com/tianocore/edk2/commit/0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
>From the code change logic, it seems like it require both --hash and 
>--binary-source to perform cache based incremental build. However, when we 
>only have --hash in build flag, it is not functional.


Here's a build time log retrieved from -y report.log.

201905 incremental |  201908 
incremental |  201905 clean | 201908 clean
Build Duration:00:03:4400:04:40 
   00:07:12 00:06:19
AutoGen Duration:  00:02:4000:03:15 
   00:04:28 00:03:09
Make Duration:00:00:3600:00:39  
  00:01:21 00:01:34
GenFds Duration: 00:00:2700:00:28   
 00:01:22 00:01:18

We could see the AutoGen Duration has almost no change between clean and 
incremental build in 201908.

Could we fix this?

Thanks,
Derek


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#47909): https://edk2.groups.io/g/devel/message/47909
Mute This Topic: https://groups.io/mt/34227864/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-



0001-Fix-the-hash-functionality-for-increamental-build.patch
Description: 0001-Fix-the-hash-functionality-for-increamental-build.patch

Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908

2019-09-23 Thread Steven Shi 
Hi Derek,
Thank you to raise this issue. We will fix it and add back the --hash original 
functionality.


Thanks
Steven

From: Lin, Derek (HPS SW) [mailto:derek.l...@hpe.com]
Sent: Friday, September 20, 2019 2:54 PM
To: devel@edk2.groups.io
Cc: Feng, Bob C ; Shi, Steven 
Subject: BaseTools --hash malfunction after migrate from stable201905 to 
stable201908

Hi BaseTools experts,

We saw the clean build performance improvement after updating from 
edk2-stable201905 to edk2-stable201908, which is promising.
But we found the incremental build time increase 25%~35%, which is not good.

We're building server platform. And we use --hash to speed up incremental build 
as described in https://github.com/BobCF/edk2/wiki/Incremental-Build
After upgrading to edk2-stable201908, none of AutoGen is skipped when I did an 
incremental build without any code change.

Dig into the issue, it is caused by commit 
https://github.com/tianocore/edk2/commit/0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
>From the code change logic, it seems like it require both --hash and 
>--binary-source to perform cache based incremental build. However, when we 
>only have --hash in build flag, it is not functional.


Here's a build time log retrieved from -y report.log.

201905 incremental |  201908 
incremental |  201905 clean | 201908 clean
Build Duration:00:03:4400:04:40 
   00:07:12 00:06:19
AutoGen Duration:  00:02:4000:03:15 
   00:04:28 00:03:09
Make Duration:00:00:3600:00:39  
  00:01:21 00:01:34
GenFds Duration: 00:00:2700:00:28   
 00:01:22 00:01:18

We could see the AutoGen Duration has almost no change between clean and 
incremental build in 201908.

Could we fix this?

Thanks,
Derek


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#47809): https://edk2.groups.io/g/devel/message/47809
Mute This Topic: https://groups.io/mt/34227864/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908

2019-09-20 Thread Lin, Derek (HPS SW) 
Hi BaseTools experts,

We saw the clean build performance improvement after updating from 
edk2-stable201905 to edk2-stable201908, which is promising.
But we found the incremental build time increase 25%~35%, which is not good.

We're building server platform. And we use --hash to speed up incremental build 
as described in https://github.com/BobCF/edk2/wiki/Incremental-Build
After upgrading to edk2-stable201908, none of AutoGen is skipped when I did an 
incremental build without any code change.

Dig into the issue, it is caused by commit 
https://github.com/tianocore/edk2/commit/0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
>From the code change logic, it seems like it require both --hash and 
>--binary-source to perform cache based incremental build. However, when we 
>only have --hash in build flag, it is not functional.


Here's a build time log retrieved from -y report.log.

201905 incremental |  201908 
incremental |  201905 clean | 201908 clean
Build Duration:00:03:4400:04:40 
   00:07:12 00:06:19
AutoGen Duration:  00:02:4000:03:15 
   00:04:28 00:03:09
Make Duration:00:00:3600:00:39  
  00:01:21 00:01:34
GenFds Duration: 00:00:2700:00:28   
 00:01:22 00:01:18

We could see the AutoGen Duration has almost no change between clean and 
incremental build in 201908.

Could we fix this?

Thanks,
Derek


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#47709): https://edk2.groups.io/g/devel/message/47709
Mute This Topic: https://groups.io/mt/34227864/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-