Seems like the answer might be a separate tag filter operator for use on data with high use counts. Then the developer could decide when it was most appropriate to use whichever tag filter operator.
On Sunday, September 22, 2019 at 4:13:40 AM UTC-7, Jed Carty wrote: > > Each filter operator only works on the output of the previous operator, so > the order matters with later operators in a run generally working with > fewer inputs than earlier operators. > To my knowledge none of the operators do anything in parallel, that may be > difficult to set up because javascript is single threaded. > > The tags field is treated differently, the tags field is sorted and stored > as a different type of list than the title lists normally used in > tiddlywiki are, this means that accessing or manipulating the tags field > introduces an extra step or two. > > In the worst case it has to load the field, parse it as the list type it > uses, do whatever the operator does, re-encode it as the tags list and then > save it. > > Reading or searching the tags field doesn't need to re-encode it or save > the field, but the extra parsing can add some overhead. > > Other than that each filter step takes a title list and returns a title > list, there isn't any other memory that carries over between each step. > > The tag operator is a pretty heavy operator. It finds every tiddler in the > wiki that has the current tag then checks each input title to see if it is > on that list and returns the list of input titles that pass that test. > > So if you have a million tiddlers with the tag Vorto it is searching > through a list of a million tiddlers once for each input tiddler. > > I think that the current version has a better worst-case behaviour than > many alternatives, but it's speed is based on the number of tiddlers in the > input list and the number of tiddlers in the wiki that have the same tag. > > It could be changed so that the speed is based off of the number of input > tiddlers and the number of tags each input tiddler has which may be worse > in small wikis but a lot better in very large wikis. It depends on the cost > of checking the tags list of each tiddler in the input list individually vs > the cost of getting a list of all tiddlers in the wiki that have the input > tag. > > Most of the improvements that immediately come to mind don't preserve the > order of the input titles. In large wikis resorting the output could be far > faster than the current version, but much slower in smaller wikis. > > I don't know how much overhead would be added by adding a check to see > which one is faster, that check may be a complex task by itself. > > The tag operator may be a place where some significant improvements can be > made. > -- You received this message because you are subscribed to the Google Groups "TiddlyWiki" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/6c49003f-e6fe-4139-8c89-49220d4e3cdb%40googlegroups.com.
