I’m doing further tests and timings, comparing that 60K tiddler wiki before and after the improvements. One interesting datapoint is that using the advanced search box, the time to refresh the page on each keypress reduces a full order of magnitude, from approx 235ms to approx 22ms. For this wiki, this is a very consistent result, and makes an enormous difference in usability.
Again, I’ll be interested to see how well these improvements work with other peoples wikis. Best wishes Jeremy > On 18 May 2019, at 18:41, Jeremy Ruston <[email protected]> wrote: > > Apologies for being quiet on the groups over the last week or two. One thing > I’ve been working on is exploring performance improvements for large wikis. > As part of that work, there’s now a nearly-complete pull request for adding > indexes to the tiddler store. > > https://github.com/Jermolene/TiddlyWiki5/pull/3951 > <https://github.com/Jermolene/TiddlyWiki5/pull/3951> > > Currently, the core evaluates a lot of filter expressions like > [all[shadows+tiddlers]tag[$:/tags/ViewTemplate]!has[draft.of]] to construct > the user interface. These filters are slow because the “tag” operator has to > iterate through every tiddler in the store to find the tiddlers with that > tag. With this PR, the wiki store maintains an index of tiddlers with each > tag, updating it every time a tiddler changes. There’s also a fair amount of > refactoring to get the filter processing logic using the new indexes. > > The implementation introduces a new type of module called an “indexer”. It > has some standard methods: rebuild() for when the entire index must be > reconstructed, update(oldTiddler,newTiddler) when a tiddler is modified, > created or deleted. There are indexer-specific methods for performing > lookups. There is also a new convention whereby tiddler iterators can have > additional properties defining “index methods” that perform fast lookups on > the tiddlers in the iterator — this is how the filter operators find the > indexers that they know about. > > The TagIndexer speeds up the “tag” filter operator when it is used > immediately after one of [all[tiddlers]], [all[shadows]], > [all[shadows+tiddlers]], [all[tiddlers+shadows]] (note that using the tag > operator at the start of a filter run is equivalent to preceding it with > [all[tiddlers]]). > > There is also a FieldIndexer which speeds up the “field” and “has” operators, > with the same restrictions as to the input selection. The field indexer is > reasonably clever, and only constructs indexes for fields for which it has > encountered queries. > > On very large wikis (>60,000 tiddlers) I’m seeing startup times reduced by > 25% with these optimisations, and refresh time reduced by a factor of three. > It will be interesting to get reports from other people working with large or > complex wikis. > > Any comments/questions welcome, > > Best wishes > > Jeremy. -- You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/tiddlywikidev. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/274901A5-8A4E-4620-ACAB-E232B1E233A6%40gmail.com. For more options, visit https://groups.google.com/d/optout.
