Jeremy,
Thanks for this development. I do see a need going forward. Can I please
ask?
- Will this mechanism be effectively invisible to most users/designers?
- With the code to support this mechanism in tiddlywiki, could it be
also surfaced to provide other indexing methods?
- Such as indexing large data tiddlers, or a subset of tiddlers,
tags? A kind of indexedlookup operator?
- In many cases [is[current] is the first operator in filters especially
on view tiddler, should this be included in the application of the
tagindexer?
- Will the indexes be saved in tiddlers for the next reload?, perhaps
this could be optional allowing indexes to be forced to be regenerated in
wikis on reload in cases where there are external/federated wikis, and
stored for large indexes on wikis primarily used for search and lookups
(with less changes).
- Perhaps an option to store such indexes in local storage and
regenerate them if missing would help keep total size down for wikis.
Love your work
Tony
On Sunday, May 19, 2019 at 3:41:23 AM UTC+10, Jeremy Ruston wrote:
>
> Apologies for being quiet on the groups over the last week or two. One
> thing I’ve been working on is exploring performance improvements for large
> wikis. As part of that work, there’s now a nearly-complete pull request for
> adding indexes to the tiddler store.
>
> https://github.com/Jermolene/TiddlyWiki5/pull/3951
>
> Currently, the core evaluates a lot of filter expressions like
> [all[shadows+tiddlers]tag[$:/tags/ViewTemplate]!has[draft.of]] to construct
> the user interface. These filters are slow because the “tag” operator has
> to iterate through every tiddler in the store to find the tiddlers with
> that tag. With this PR, the wiki store maintains an index of tiddlers with
> each tag, updating it every time a tiddler changes. There’s also a fair
> amount of refactoring to get the filter processing logic using the new
> indexes.
>
> The implementation introduces a new type of module called an “indexer”. It
> has some standard methods: rebuild() for when the entire index must be
> reconstructed, update(oldTiddler,newTiddler) when a tiddler is modified,
> created or deleted. There are indexer-specific methods for performing
> lookups. There is also a new convention whereby tiddler iterators can have
> additional properties defining “index methods” that perform fast lookups on
> the tiddlers in the iterator — this is how the filter operators find the
> indexers that they know about.
>
> The TagIndexer speeds up the “tag” filter operator when it is used
> immediately after one of [all[tiddlers]], [all[shadows]],
> [all[shadows+tiddlers]], [all[tiddlers+shadows]] (note that using the tag
> operator at the start of a filter run is equivalent to preceding it with
> [all[tiddlers]]).
>
> There is also a FieldIndexer which speeds up the “field” and “has”
> operators, with the same restrictions as to the input selection. The field
> indexer is reasonably clever, and only constructs indexes for fields for
> which it has encountered queries.
>
> On very large wikis (>60,000 tiddlers) I’m seeing startup times reduced by
> 25% with these optimisations, and refresh time reduced by a factor of
> three. It will be interesting to get reports from other people working with
> large or complex wikis.
>
> Any comments/questions welcome,
>
> Best wishes
>
> Jeremy.
>
--
You received this message because you are subscribed to the Google Groups
"TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/tiddlywikidev.
To view this discussion on the web visit
https://groups.google.com/d/msgid/tiddlywikidev/22681694-5fa1-47e9-9a9f-0c9df73d70a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
