123123213weqw opened a new pull request, #5727:
URL: https://github.com/apache/rocketmq-dashboard/pull/5727
## Problem / Evidence
`estimateTokens` (`web/src/pages/ai/render/streamSpeed.ts`) implements "one
token per CJK character, one per four other characters" but computes the
non-CJK remainder from `text.length` — UTF-16 code units, not characters:
```ts
return cjkChars + Math.max(0, text.length - cjkChars) / 4;
```
A supplementary character (emoji, CJK extension) is two code units and one
character, so it is counted as two. Four rocket emoji estimate **2** tokens
instead of **1** — the same UTF-16/code-point confusion this codebase already
fixed for conversation titles (#5213) and server-side truncation
(`TextBounds`). The number feeds both the streamed tokens/s badge and the
context-usage estimate (`pages/ai/index.tsx`), so content with astral
characters over-reports both.
## Root cause / Fix
Count the non-CJK remainder in code points: a small `characterCount` helper
skips low surrogates so a surrogate pair contributes one character, matching
what the doc comment of `estimateTokens` already promises ("one per four other
**characters**").
## Tests
- New cases in `streamSpeed.test.ts`: `estimateTokens('🚀🚀🚀🚀')` is 1 (was 2)
and a mixed emoji+latin delta estimates on characters. Fails on the pre-fix
code.
- `npx vitest run src/pages/ai` → 195/195 passed; `npx tsc --noEmit -p
tsconfig.app.json` clean; eslint on touched files clean.
## Risk
Low: the estimate only changes for text containing supplementary characters,
in the direction of correctness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
