comaniac opened a new pull request #6713:
URL: https://github.com/apache/incubator-tvm/pull/6713
The previous implementation of initial population sampling treat a state to
be valid as long as we didn't encounter any problem when applying the initial
population rules. However, it's possible that all states outputted by this
phase cannot be successfully either lowered, extracted (features), or passed
the GPU code verification (due to invalid thread number or memory usage). In
this case, the evo search is trapped in a set of invalid states and it's
inefficient for it to find a valid point.
In this PR, we improved the initial population sampling to perform the same
process as we did in the evo search. Specifically, all states have to be
lowered and estimated to make sure they are valid. Note that this will also
increase the time of sampling initial population. Here is an example log of
tuning a forward conv2d (last layer in ResNet) on Nvidia T4:
```
------------------------------------------------------------
------------------------- [ Search ]
------------------------------------------------------------
Generate Sketches #s: 1
Encountered 331 errors during feature extraction, which are safely ignored.
Sample Iter: 1 #Pop: 66 #Target: 2048 fail_ct: 1982 Time
elapsed: 1.25
Encountered 345 errors during feature extraction, which are safely ignored.
Sample Iter: 2 #Pop: 133 #Target: 2048 fail_ct: 3963 Time
elapsed: 2.53
Encountered 338 errors during feature extraction, which are safely ignored.
Sample Iter: 3 #Pop: 181 #Target: 2048 fail_ct: 5963 Time
elapsed: 3.81
Encountered 376 errors during feature extraction, which are safely ignored.
Sample Iter: 4 #Pop: 242 #Target: 2048 fail_ct: 7950 Time
elapsed: 5.19
Encountered 351 errors during feature extraction, which are safely ignored.
Sample Iter: 5 #Pop: 303 #Target: 2048 fail_ct: 9937 Time
elapsed: 6.48
Encountered 350 errors during feature extraction, which are safely ignored.
Sample Iter: 6 #Pop: 361 #Target: 2048 fail_ct: 11927 Time
elapsed: 7.78
Encountered 338 errors during feature extraction, which are safely ignored.
Sample Iter: 7 #Pop: 415 #Target: 2048 fail_ct: 13921 Time
elapsed: 9.06
Encountered 344 errors during feature extraction, which are safely ignored.
Sample Iter: 8 #Pop: 485 #Target: 2048 fail_ct: 15899 Time
elapsed: 10.38
Encountered 343 errors during feature extraction, which are safely ignored.
Sample Iter: 9 #Pop: 539 #Target: 2048 fail_ct: 17893 Time
elapsed: 11.68
Encountered 361 errors during feature extraction, which are safely ignored.
Sample Iter: 10 #Pop: 613 #Target: 2048 fail_ct: 19867 Time
elapsed: 13.03
Encountered 357 errors during feature extraction, which are safely ignored.
Sample Iter: 11 #Pop: 670 #Target: 2048 fail_ct: 21858 Time
elapsed: 14.38
Encountered 349 errors during feature extraction, which are safely ignored.
Sample Iter: 12 #Pop: 739 #Target: 2048 fail_ct: 23837 Time
elapsed: 15.68
Encountered 347 errors during feature extraction, which are safely ignored.
Sample Iter: 13 #Pop: 799 #Target: 2048 fail_ct: 25825 Time
elapsed: 17.02
Encountered 358 errors during feature extraction, which are safely ignored.
Sample Iter: 14 #Pop: 842 #Target: 2048 fail_ct: 27830 Time
elapsed: 18.36
Encountered 339 errors during feature extraction, which are safely ignored.
Sample Iter: 15 #Pop: 893 #Target: 2048 fail_ct: 29827 Time
elapsed: 19.64
Encountered 364 errors during feature extraction, which are safely ignored.
Sample Iter: 16 #Pop: 947 #Target: 2048 fail_ct: 31821 Time
elapsed: 20.99
Encountered 337 errors during feature extraction, which are safely ignored.
Sample Iter: 17 #Pop: 997 #Target: 2048 fail_ct: 33819 Time
elapsed: 22.27
Encountered 344 errors during feature extraction, which are safely ignored.
Sample Iter: 18 #Pop: 1052 #Target: 2048 fail_ct: 35812 Time
elapsed: 23.63
Encountered 347 errors during feature extraction, which are safely ignored.
Sample Iter: 19 #Pop: 1112 #Target: 2048 fail_ct: 37800 Time
elapsed: 24.96
Encountered 330 errors during feature extraction, which are safely ignored.
Sample Iter: 20 #Pop: 1174 #Target: 2048 fail_ct: 39786 Time
elapsed: 26.23
Encountered 348 errors during feature extraction, which are safely ignored.
Sample Iter: 21 #Pop: 1232 #Target: 2048 fail_ct: 41776 Time
elapsed: 27.53
Encountered 346 errors during feature extraction, which are safely ignored.
Sample Iter: 22 #Pop: 1275 #Target: 2048 fail_ct: 43781 Time
elapsed: 28.82
Encountered 357 errors during feature extraction, which are safely ignored.
Sample Iter: 23 #Pop: 1319 #Target: 2048 fail_ct: 45785 Time
elapsed: 30.16
Encountered 349 errors during feature extraction, which are safely ignored.
Sample Iter: 24 #Pop: 1380 #Target: 2048 fail_ct: 47772 Time
elapsed: 31.48
Encountered 317 errors during feature extraction, which are safely ignored.
Sample Iter: 25 #Pop: 1424 #Target: 2048 fail_ct: 49776 Time
elapsed: 32.70
Encountered 323 errors during feature extraction, which are safely ignored.
Sample Iter: 26 #Pop: 1473 #Target: 2048 fail_ct: 51775 Time
elapsed: 33.96
Encountered 326 errors during feature extraction, which are safely ignored.
Sample Iter: 27 #Pop: 1541 #Target: 2048 fail_ct: 53755 Time
elapsed: 35.21
Encountered 311 errors during feature extraction, which are safely ignored.
Sample Iter: 28 #Pop: 1603 #Target: 2048 fail_ct: 55741 Time
elapsed: 36.42
Encountered 348 errors during feature extraction, which are safely ignored.
Sample Iter: 29 #Pop: 1662 #Target: 2048 fail_ct: 57730 Time
elapsed: 37.73
Encountered 322 errors during feature extraction, which are safely ignored.
Sample Iter: 30 #Pop: 1719 #Target: 2048 fail_ct: 59721 Time
elapsed: 39.00
Encountered 369 errors during feature extraction, which are safely ignored.
Sample Iter: 31 #Pop: 1782 #Target: 2048 fail_ct: 61706 Time
elapsed: 40.36
Encountered 351 errors during feature extraction, which are safely ignored.
Sample Iter: 32 #Pop: 1853 #Target: 2048 fail_ct: 63683 Time
elapsed: 41.68
Encountered 329 errors during feature extraction, which are safely ignored.
Sample Iter: 33 #Pop: 1922 #Target: 2048 fail_ct: 65662 Time
elapsed: 42.94
Encountered 328 errors during feature extraction, which are safely ignored.
Sample Iter: 34 #Pop: 1970 #Target: 2048 fail_ct: 67662 Time
elapsed: 44.17
Encountered 353 errors during feature extraction, which are safely ignored.
Sample Iter: 35 #Pop: 2020 #Target: 2048 fail_ct: 69660 Time
elapsed: 45.47
Encountered 331 errors during feature extraction, which are safely ignored.
Sample Iter: 36 #Pop: 2078 #Target: 2048 fail_ct: 71650 Time
elapsed: 46.76
Sample Initial Population #s: 2078 fail_ct: 71650 Time
elapsed: 46.78
/home/ubuntu/.local/lib/python3.6/site-packages/numpy/core/_methods.py:34:
RuntimeWarning: invalid value encountered in reduce
return umr_minimum(a, axis, None, out, keepdims, initial, where)
/home/ubuntu/.local/lib/python3.6/site-packages/numpy/core/_methods.py:30:
RuntimeWarning: invalid value encountered in reduce
return umr_maximum(a, axis, None, out, keepdims, initial, where)
GA Iter: 0 Max score: 0.9988 Min score: 0.9417 #Pop: 2078
#M+: 0 #M-: 0
Encountered 333 errors during feature extraction, which are safely ignored.
Encountered 310 errors during feature extraction, which are safely ignored.
Encountered 283 errors during feature extraction, which are safely ignored.
Encountered 290 errors during feature extraction, which are safely ignored.
Encountered 270 errors during feature extraction, which are safely ignored.
GA Iter: 5 Max score: 0.9999 Min score: 0.9880 #Pop: 2048
#M+: 1454 #M-: 0
Encountered 251 errors during feature extraction, which are safely ignored.
Encountered 266 errors during feature extraction, which are safely ignored.
Encountered 244 errors during feature extraction, which are safely ignored.
Encountered 250 errors during feature extraction, which are safely ignored.
Encountered 265 errors during feature extraction, which are safely ignored.
GA Iter: 10 Max score: 0.9999 Min score: 0.9930 #Pop: 2048
#M+: 1583 #M-: 0
EvolutionarySearch #s: 128 Time elapsed: 33.14
------------------------------------------------------------
------------------------- [ Measure ]
------------------------------------------------------------
Get 64 programs for measure. (This may take a while)
................................******************E**********E****
# skip logs. Max throughput 2383.68 GFlop/s
................................*************************E*T******
# skip logs. Max throughput 2383.68 GFlop/s
Median execution time: 2.911 ms
```
cc @merrymercy @jcf94
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
