Apologies for being quiet on the groups over the last week or two. One thing I’ve been working on is exploring performance improvements for large wikis. As part of that work, there’s now a nearly-complete pull request for adding indexes to the tiddler store.
https://github.com/Jermolene/TiddlyWiki5/pull/3951 <https://github.com/Jermolene/TiddlyWiki5/pull/3951> Currently, the core evaluates a lot of filter expressions like [all[shadows+tiddlers]tag[$:/tags/ViewTemplate]!has[draft.of]] to construct the user interface. These filters are slow because the “tag” operator has to iterate through every tiddler in the store to find the tiddlers with that tag. With this PR, the wiki store maintains an index of tiddlers with each tag, updating it every time a tiddler changes. There’s also a fair amount of refactoring to get the filter processing logic using the new indexes. The implementation introduces a new type of module called an “indexer”. It has some standard methods: rebuild() for when the entire index must be reconstructed, update(oldTiddler,newTiddler) when a tiddler is modified, created or deleted. There are indexer-specific methods for performing lookups. There is also a new convention whereby tiddler iterators can have additional properties defining “index methods” that perform fast lookups on the tiddlers in the iterator — this is how the filter operators find the indexers that they know about. The TagIndexer speeds up the “tag” filter operator when it is used immediately after one of [all[tiddlers]], [all[shadows]], [all[shadows+tiddlers]], [all[tiddlers+shadows]] (note that using the tag operator at the start of a filter run is equivalent to preceding it with [all[tiddlers]]). There is also a FieldIndexer which speeds up the “field” and “has” operators, with the same restrictions as to the input selection. The field indexer is reasonably clever, and only constructs indexes for fields for which it has encountered queries. On very large wikis (>60,000 tiddlers) I’m seeing startup times reduced by 25% with these optimisations, and refresh time reduced by a factor of three. It will be interesting to get reports from other people working with large or complex wikis. Any comments/questions welcome, Best wishes Jeremy. -- You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/tiddlywikidev. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/042EF5D2-F5BF-4682-ADF4-4646553009C8%40gmail.com. For more options, visit https://groups.google.com/d/optout.
