Celve commented on PR #16338:
URL: https://github.com/apache/tvm/pull/16338#issuecomment-1879600743
> thanks for the clarification, do we have numbers comparing with cutlass?
cutlass:
```
======================= Encoding Profiling =======================
Name Time (ms) Count Total
time (ms) Pct (%) Memory (MB) Bandwidth (GB/s) Shape
fused_fused_decode4_NT_matmul2 0.3124 32 9.9961
34.48 48.67 152.1651 (22016, 512), (22016,
128), (1, 6, 4096), (1, 6, 22016)
fused_fused_decode5_fused_NT_matmul3_add 0.2669 32 8.5402
29.46 24.41 89.3100 (4096, 1376), (4096,
344), (1, 6, 11008), (1, 6, 4096), (1, 6, 4096)
fused_fused_decode2_NT_matmul 0.2039 32 6.5247
22.51 27.19 130.2134 (12288, 512), (12288,
128), (1, 6, 4096), (1, 6, 12288)
fused_fused_decode3_fused_NT_matmul1_add 0.1013 32 3.2423
11.18 9.14 88.0998 (4096, 512), (4096,
128), (1, 6, 4096), (1, 6, 4096), (1, 6, 4096)
fused_relax_nn_attention_cutlass1 0.0057 32 0.1828
0.63 0.20 33.3827 (1, 6, 32, 128), (1, 6,
32, 128), (1, 6, 32, 128), (8192,), (1, 6, 32, 128)
split 0.0052 32 0.1665
0.57 0.28 52.7735 (1, 6, 12288), (1, 6,
4096), (1, 6, 4096), (1, 6, 4096)
fused_rms_norm_cutlass 0.0025 65 0.1644
0.57 0.10 39.2137 (1, 6, 4096), (4096,),
(1, 6, 4096)
fused_fused_decode1_fused_NT_matmul4_cast 0.1064 1 0.1064
0.37 70.44 646.5570 (32000, 512), (32000,
128), (1, 1, 4096), (1, 1, 32000)
fused_split1_silu_multiply 0.0020 32 0.0626
0.22 0.38 188.7035 (1, 6, 22016), (1, 6,
11008)
fused_fused_decode1_take 0.0019 1 0.0019
0.01 70.36 35502.7417 (32000, 512), (32000,
128), (6,), (6, 4096)
slice 0.0018 1 0.0018
0.01 0.05 29.3516 (1, 6, 4096), (1, 1,
4096)
Total time: 28.9899 ms
======================= Decoding Profiling =======================
Name Time (ms) Count Total
time (ms) Pct (%) Memory (MB) Bandwidth (GB/s) Shape
fused_fused_decode4_NT_matmul7 0.0747 32 2.3908
37.73 48.42 632.9457 (22016, 512), (22016,
128), (1, 1, 4096), (1, 1, 22016)
fused_fused_decode5_fused_NT_matmul8_add1 0.0454 32 1.4527
22.93 24.22 521.0961 (4096, 1376), (4096,
344), (1, 1, 11008), (1, 1, 4096), (1, 1, 4096)
fused_fused_decode2_NT_matmul5 0.0437 32 1.3981
22.06 27.03 604.1956 (12288, 512), (12288,
128), (1, 1, 4096), (1, 1, 12288)
fused_fused_decode3_fused_NT_matmul6_add1 0.0182 32 0.5829
9.20 9.02 483.7281 (4096, 512), (4096,
128), (1, 1, 4096), (1, 1, 4096), (1, 1, 4096)
fused_relax_nn_attention1_cutlass1 0.0057 32 0.1822
2.88 0.13 22.7810 (1, 1, 32, 128), (1, 7,
32, 128), (1, 7, 32, 128), (8192,), (1, 1, 32, 128)
fused_rms_norm1_cutlass 0.0024 65 0.1580
2.49 0.02 9.4179 (1, 1, 4096), (4096,),
(1, 1, 4096)
fused_fused_decode1_fused_NT_matmul4_cast 0.1066 1 0.1066
1.68 70.44 645.1684 (32000, 512), (32000,
128), (1, 1, 4096), (1, 1, 32000)
fused_split2_silu1_multiply1 0.0020 32 0.0631
1.00 0.06 31.1784 (1, 1, 22016), (1, 1,
11008)
fused_fused_decode1_take1 0.0019 1 0.0019
0.03 70.32 36989.5363 (32000, 512), (32000,
128), (1,), (1, 4096)
Total time: 6.3364 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
