Hi Tony > Will this mechanism be effectively invisible to most users/designers? Yes. It’s a big change so in due course when the PR is merged we will need everyone to pay attention to testing it.
> With the code to support this mechanism in tiddlywiki, could it be also > surfaced to provide other indexing methods? > Such as indexing large data tiddlers, or a subset of tiddlers, tags? A kind > of indexedlookup operator? Yes. The current PR has two indexers for tag and field lookups, but the mechanism is extensible so that others can be added. > In many cases [is[current] is the first operator in filters especially on > view tiddler, should this be included in the application of the tagindexer? Yes, as you know it is much less efficient to start with [is[current]] than [all[current]] because the former scans each tiddler to check whether it is the current tiddler, while the second just selects the current tiddler without scanning. But, we might indeed be able to adapt the indexing mechanism to optimise the is[current] operator, and possibly some of the other variants. > Will the indexes be saved in tiddlers for the next reload?, perhaps this > could be optional allowing indexes to be forced to be regenerated in wikis on > reload in cases where there are external/federated wikis, and stored for > large indexes on wikis primarily used for search and lookups (with less > changes). No, I don’t think it’s worth it. These are very simple indexes that are pretty fast to build. It’s only if we were doing full text indexing that I think we might want to persistently cache the indexes. > Perhaps an option to store such indexes in local storage and regenerate them > if missing would help keep total size down for wikis. As I say, generating the indexes is pretty fast, even with 60,000 tiddlers. It’s only done once, and thereafter it is selectively updated as required. Best wishes Jeremy > Love your work > Tony > > On Sunday, May 19, 2019 at 3:41:23 AM UTC+10, Jeremy Ruston wrote: > Apologies for being quiet on the groups over the last week or two. One thing > I’ve been working on is exploring performance improvements for large wikis. > As part of that work, there’s now a nearly-complete pull request for adding > indexes to the tiddler store. > > https://github.com/Jermolene/TiddlyWiki5/pull/3951 > <https://github.com/Jermolene/TiddlyWiki5/pull/3951> > > Currently, the core evaluates a lot of filter expressions like > [all[shadows+tiddlers]tag[$:/tags/ViewTemplate]!has[draft.of]] to construct > the user interface. These filters are slow because the “tag” operator has to > iterate through every tiddler in the store to find the tiddlers with that > tag. With this PR, the wiki store maintains an index of tiddlers with each > tag, updating it every time a tiddler changes. There’s also a fair amount of > refactoring to get the filter processing logic using the new indexes. > > The implementation introduces a new type of module called an “indexer”. It > has some standard methods: rebuild() for when the entire index must be > reconstructed, update(oldTiddler,newTiddler) when a tiddler is modified, > created or deleted. There are indexer-specific methods for performing > lookups. There is also a new convention whereby tiddler iterators can have > additional properties defining “index methods” that perform fast lookups on > the tiddlers in the iterator — this is how the filter operators find the > indexers that they know about. > > The TagIndexer speeds up the “tag” filter operator when it is used > immediately after one of [all[tiddlers]], [all[shadows]], > [all[shadows+tiddlers]], [all[tiddlers+shadows]] (note that using the tag > operator at the start of a filter run is equivalent to preceding it with > [all[tiddlers]]). > > There is also a FieldIndexer which speeds up the “field” and “has” operators, > with the same restrictions as to the input selection. The field indexer is > reasonably clever, and only constructs indexes for fields for which it has > encountered queries. > > On very large wikis (>60,000 tiddlers) I’m seeing startup times reduced by > 25% with these optimisations, and refresh time reduced by a factor of three. > It will be interesting to get reports from other people working with large or > complex wikis. > > Any comments/questions welcome, > > Best wishes > > Jeremy. > > -- > You received this message because you are subscribed to the Google Groups > "TiddlyWikiDev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/tiddlywikidev > <https://groups.google.com/group/tiddlywikidev>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/tiddlywikidev/22681694-5fa1-47e9-9a9f-0c9df73d70a3%40googlegroups.com > > <https://groups.google.com/d/msgid/tiddlywikidev/22681694-5fa1-47e9-9a9f-0c9df73d70a3%40googlegroups.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/tiddlywikidev. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/EA656BCE-A3E1-4E2C-BD50-0ED4F7F570B6%40gmail.com. For more options, visit https://groups.google.com/d/optout.
