https://bugzilla.wikimedia.org/show_bug.cgi?id=44447
Web browser: ---
Bug ID: 44447
Summary: Top-level block-wise tokenization for better
performance
Product: Parsoid
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: JS/tokenizer
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected], [email protected],
[email protected]
Classification: Unclassified
Mobile Platform: ---
The current tokenizer performs a single pass and does not yield to other async
tasks until it is done. This queues up a lot of tokens and async actions at
once. It would be more efficient to cooperatively yield after each top-level
block or so, so that some async processing can already happen as soon as the
data becomes available. A simple process.nextTick call after each top-level
block and a new offset parameter to the tokenizer to re-start tokenization at
the given offset are probably all that is needed to achieve this.
--
You are receiving this mail because:
You are on the CC list for the bug.
You are watching all bug changes.
_______________________________________________
Wikibugs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
