mengshyu opened a new pull request, #17663:
URL: https://github.com/apache/tvm/pull/17663
This PR adds Dlight CPU support with optimized GEMV scheduling, including
pattern detection, loop tiling, vectorization, and parallel execution. It
improves maintainability by refining target checks, reduction handling, and
scheduling logic.
CPU: AMD Ryzen 9 7950X 16-Core Processor
MODEL: Qwen2-0.5B-q4f16_1-MLC
Prompt: What is the meaning of life?
Results:
Baseline:
prompt_tokens=27 completion_tokens=235 total_tokens=262
extra={'prompt_tokens': 27, 'completion_tokens': 235, 'prefill_tokens': 27,
'decode_tokens': 234, 'jump_forward_tokens': 0, 'prefill_tokens_per_s':
0.9777329325367138,
'decode_tokens_per_s': 0.558195154052001,
'end_to_end_latency_s': 446.823128383, 'ttft_s': 27.614902906,
'inter_token_latency_s': 1.9013750143957446}
Optimized:
usage: prompt_tokens=27 completion_tokens=227 total_tokens=254
extra={'prompt_tokens': 27, 'completion_tokens': 227, 'prefill_tokens': 27,
'decode_tokens': 226, 'jump_forward_tokens': 0, 'prefill_tokens_per_s':
1.0010420333327994,
'decode_tokens_per_s': 2.9349053824023454,
'end_to_end_latency_s': 103.976080401, 'ttft_s': 26.971894387,
'inter_token_latency_s': 0.4580444070528635}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
