Hi everyone! Diego was kind enough to bring this discussion to my attention - I plan on fixing up that issue he ran into.
Jeremy - if you decide to go with lunr.js but not use my plugin, please feel free to reach out if you have questions about lunr.js! One pitfall I ran into while upgrading from lunr 1.x to 2.x is that lunr's index is immutable by default in 2.x - I ended up writing a plugin to add mutable indexes. Of course, any collaboration on my plugin (along with advice on how to improve it) would be most welcome! -Rob On Thursday, June 28, 2018 at 9:14:27 AM UTC-5, Diego Mesa wrote: > > Hey Jeremy, > > Just wanted to let you know of this plugin: > > https://github.com/hoelzro/tw-full-text-search > > which incorporates lunr.js into TW for full text searching. I just tried > today and am somewhat able to do fuzzy searching with it (does not deal > with immediatel searching properly yet). I have let him know about this > discussion and the corresponding feature request on github, and have opened > an issue on his repo for the fuzzy matching: > > https://github.com/hoelzro/tw-full-text-search/issues/4 > > Perhaps you can reuse some of hoelzro's work? > > Best, > Diego > > On Wednesday, June 27, 2018 at 8:47:06 AM UTC-5, Diego Mesa wrote: >> >> Hey Jeremy, >> >> Thanks for your feedback - I agree search needs to performant. Im glad >> you're also looking at these issues! >> >> A priority for me is the inclusion of *fuzzy* searching - a quick look >> seems lunr does support it: >> >> https://lunrjs.com/guides/searching.html#fuzzy-matches >> >> but elastic does not. >> >> Best, >> Diego >> >> On Wednesday, June 27, 2018 at 2:28:18 AM UTC-5, Jeremy Ruston wrote: >>> >>> Hi Diego >>> >>> Filter operators are often re-executed and so it's important that they >>> perform well. For an expensive operation like a search we need to try to >>> cache as much of the work as we can. >>> >>> An approach that should work is to introduce a new startup module that >>> tracks changes to the store in order to keep the index up to date, and then >>> have the filter operator call a version of the search function that caches >>> the results (and clears the cache when it is invalidated by changes to the >>> tiddler store). >>> >>> Having said that, one of the client projects I'm working on for >>> Federatial looks like it's going to need an enhanced search capability. If >>> the project proceeds to the next phase then I'll be looking at adding a >>> third party search index library -- I've been looking at >>> https://lunrjs.com/ and http://elasticlunr.com/. The API that they >>> feature allows the index to be dynamically modified by adding and removing >>> data incrementally; fuse.js looks like it only accepts a monolithic block >>> of data, and would require a re-index each time the tiddler store changes. >>> >>> Best wishes >>> >>> Jeremy >>> >>> -- >>> Jeremy Ruston >>> jer...@jermolene.com >>> https://jermolene.com >>> >>> On 26 Jun 2018, at 20:45, Diego Mesa <dieg...@gmail.com> wrote: >>> >>> Hey all, >>> >>> Thanks Jeremy for sending that. The following is *more pseduo-code than >>> anything*, but reflects my quick hacking: >>> >>> (function(){ >>> >>> >>> /*jslint node: true, browser: true */ >>> /*global $tw: false */ >>> "use strict"; >>> >>> >>> var fuze = require("$:/plugins/fuzzy/fuze.js"); >>> >>> >>> var options = { >>> shouldSort: true, >>> matchAllTokens: true, >>> threshold: 0.6, >>> location: 0, >>> distance: 100, >>> maxPatternLength: 32, >>> minMatchCharLength: 1, >>> keys: undefined >>> }; >>> >>> >>> /* >>> Export our filter function >>> */ >>> exports.fuzzysearch = function(source,operator,options) { >>> var results = []; >>> >>> >>> source = options.source || $tw.wiki.each; >>> >>> >>> var searchTiddler = function(title){ >>> jsonData = $tw.wiki.getTiddlerAsJson(title) >>> var fuse = new Fuse(jsonData, options); // "list" is the item >>> array >>> var result = fuse.search(operator.operand); >>> return result; >>> }; >>> source(function(tiddler,title) { >>> if(searchTiddler(title) !== options.invert) { >>> results.push(title); >>> } >>> }); >>> >>> >>> console.log("e"); >>> return results; >>> }; >>> >>> >>> })(); >>> >>> I quickly generated this just by looking at: >>> >>> $:/core/modules/filters/search.js >>> >>> which is just a wrapper around: >>> >>> $:/core/modules/wiki.js >>> >>> where exports.search = is defined. >>> >>> The issue is that as you can see from >>> >>> http://fusejs.io/ >>> >>> Fuze works better if it would have access to the entire json string of >>> the wiki, and it can handle the searching and scoring itself. >>> >>> What do you guys think? Any tips to help me move forward? >>> >>> Best, >>> Diego >>> >>> On Tuesday, June 26, 2018 at 8:38:36 AM UTC-5, Jed Carty wrote: >>>> >>>> I hadn't realised it could be done that way. That makes including >>>> external libraries a lot easier than I thought. I am a bit embarrassed >>>> that >>>> I didn't notice you could do that. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "TiddlyWikiDev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to tiddlywikide...@googlegroups.com. >>> To post to this group, send email to tiddly...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/tiddlywikidev. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/tiddlywikidev/c0e08271-5dd4-4578-b2b3-d4693c27a8b4%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/tiddlywikidev/c0e08271-5dd4-4578-b2b3-d4693c27a8b4%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group. To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikidev+unsubscr...@googlegroups.com. To post to this group, send email to tiddlywikidev@googlegroups.com. Visit this group at https://groups.google.com/group/tiddlywikidev. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/1b2a50b6-1d9b-40c9-a4e1-d8b18cfbd910%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
