All, Rob has fixed the issue with his plugin, so we can now have fuzzy search!!!
On Thursday, June 28, 2018 at 11:09:21 AM UTC-5, Rob Hoelz wrote: > > Hi everyone! > > Diego was kind enough to bring this discussion to my attention - I plan on > fixing up that issue he ran into. > > Jeremy - if you decide to go with lunr.js but not use my plugin, please > feel free to reach out if you have questions about lunr.js! One pitfall I > ran into while upgrading from lunr 1.x to 2.x is that lunr's index is > immutable by default in 2.x - I ended up writing a plugin to add mutable > indexes. Of course, any collaboration on my plugin (along with advice on > how to improve it) would be most welcome! > > -Rob > > On Thursday, June 28, 2018 at 9:14:27 AM UTC-5, Diego Mesa wrote: >> >> Hey Jeremy, >> >> Just wanted to let you know of this plugin: >> >> https://github.com/hoelzro/tw-full-text-search >> >> which incorporates lunr.js into TW for full text searching. I just tried >> today and am somewhat able to do fuzzy searching with it (does not deal >> with immediatel searching properly yet). I have let him know about this >> discussion and the corresponding feature request on github, and have opened >> an issue on his repo for the fuzzy matching: >> >> https://github.com/hoelzro/tw-full-text-search/issues/4 >> >> Perhaps you can reuse some of hoelzro's work? >> >> Best, >> Diego >> >> On Wednesday, June 27, 2018 at 8:47:06 AM UTC-5, Diego Mesa wrote: >>> >>> Hey Jeremy, >>> >>> Thanks for your feedback - I agree search needs to performant. Im glad >>> you're also looking at these issues! >>> >>> A priority for me is the inclusion of *fuzzy* searching - a quick look >>> seems lunr does support it: >>> >>> https://lunrjs.com/guides/searching.html#fuzzy-matches >>> >>> but elastic does not. >>> >>> Best, >>> Diego >>> >>> On Wednesday, June 27, 2018 at 2:28:18 AM UTC-5, Jeremy Ruston wrote: >>>> >>>> Hi Diego >>>> >>>> Filter operators are often re-executed and so it's important that they >>>> perform well. For an expensive operation like a search we need to try to >>>> cache as much of the work as we can. >>>> >>>> An approach that should work is to introduce a new startup module that >>>> tracks changes to the store in order to keep the index up to date, and >>>> then >>>> have the filter operator call a version of the search function that caches >>>> the results (and clears the cache when it is invalidated by changes to the >>>> tiddler store). >>>> >>>> Having said that, one of the client projects I'm working on for >>>> Federatial looks like it's going to need an enhanced search capability. If >>>> the project proceeds to the next phase then I'll be looking at adding a >>>> third party search index library -- I've been looking at >>>> https://lunrjs.com/ and http://elasticlunr.com/. The API that they >>>> feature allows the index to be dynamically modified by adding and removing >>>> data incrementally; fuse.js looks like it only accepts a monolithic block >>>> of data, and would require a re-index each time the tiddler store changes. >>>> >>>> Best wishes >>>> >>>> Jeremy >>>> >>>> -- >>>> Jeremy Ruston >>>> [email protected] >>>> https://jermolene.com >>>> >>>> On 26 Jun 2018, at 20:45, Diego Mesa <[email protected]> wrote: >>>> >>>> Hey all, >>>> >>>> Thanks Jeremy for sending that. The following is *more pseduo-code >>>> than anything*, but reflects my quick hacking: >>>> >>>> (function(){ >>>> >>>> >>>> /*jslint node: true, browser: true */ >>>> /*global $tw: false */ >>>> "use strict"; >>>> >>>> >>>> var fuze = require("$:/plugins/fuzzy/fuze.js"); >>>> >>>> >>>> var options = { >>>> shouldSort: true, >>>> matchAllTokens: true, >>>> threshold: 0.6, >>>> location: 0, >>>> distance: 100, >>>> maxPatternLength: 32, >>>> minMatchCharLength: 1, >>>> keys: undefined >>>> }; >>>> >>>> >>>> /* >>>> Export our filter function >>>> */ >>>> exports.fuzzysearch = function(source,operator,options) { >>>> var results = []; >>>> >>>> >>>> source = options.source || $tw.wiki.each; >>>> >>>> >>>> var searchTiddler = function(title){ >>>> jsonData = $tw.wiki.getTiddlerAsJson(title) >>>> var fuse = new Fuse(jsonData, options); // "list" is the item >>>> array >>>> var result = fuse.search(operator.operand); >>>> return result; >>>> }; >>>> source(function(tiddler,title) { >>>> if(searchTiddler(title) !== options.invert) { >>>> results.push(title); >>>> } >>>> }); >>>> >>>> >>>> console.log("e"); >>>> return results; >>>> }; >>>> >>>> >>>> })(); >>>> >>>> I quickly generated this just by looking at: >>>> >>>> $:/core/modules/filters/search.js >>>> >>>> which is just a wrapper around: >>>> >>>> $:/core/modules/wiki.js >>>> >>>> where exports.search = is defined. >>>> >>>> The issue is that as you can see from >>>> >>>> http://fusejs.io/ >>>> >>>> Fuze works better if it would have access to the entire json string of >>>> the wiki, and it can handle the searching and scoring itself. >>>> >>>> What do you guys think? Any tips to help me move forward? >>>> >>>> Best, >>>> Diego >>>> >>>> On Tuesday, June 26, 2018 at 8:38:36 AM UTC-5, Jed Carty wrote: >>>>> >>>>> I hadn't realised it could be done that way. That makes including >>>>> external libraries a lot easier than I thought. I am a bit embarrassed >>>>> that >>>>> I didn't notice you could do that. >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "TiddlyWikiDev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/tiddlywikidev. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/tiddlywikidev/c0e08271-5dd4-4578-b2b3-d4693c27a8b4%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/tiddlywikidev/c0e08271-5dd4-4578-b2b3-d4693c27a8b4%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> -- You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/tiddlywikidev. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/9f196b0f-3628-4e4c-aac5-4cc4f967d9ea%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
