Re: [twdev] Re: Request for help with new filter operator and new tutorial
All,
Rob has fixed the issue with his plugin, so we can now have fuzzy search!!!
On Thursday, June 28, 2018 at 11:09:21 AM UTC-5, Rob Hoelz wrote:
>
> Hi everyone!
>
> Diego was kind enough to bring this discussion to my attention - I plan on
> fixing up that issue he ran into.
>
> Jeremy - if you decide to go with lunr.js but not use my plugin, please
> feel free to reach out if you have questions about lunr.js! One pitfall I
> ran into while upgrading from lunr 1.x to 2.x is that lunr's index is
> immutable by default in 2.x - I ended up writing a plugin to add mutable
> indexes. Of course, any collaboration on my plugin (along with advice on
> how to improve it) would be most welcome!
>
> -Rob
>
> On Thursday, June 28, 2018 at 9:14:27 AM UTC-5, Diego Mesa wrote:
>>
>> Hey Jeremy,
>>
>> Just wanted to let you know of this plugin:
>>
>> https://github.com/hoelzro/tw-full-text-search
>>
>> which incorporates lunr.js into TW for full text searching. I just tried
>> today and am somewhat able to do fuzzy searching with it (does not deal
>> with immediatel searching properly yet). I have let him know about this
>> discussion and the corresponding feature request on github, and have opened
>> an issue on his repo for the fuzzy matching:
>>
>> https://github.com/hoelzro/tw-full-text-search/issues/4
>>
>> Perhaps you can reuse some of hoelzro's work?
>>
>> Best,
>> Diego
>>
>> On Wednesday, June 27, 2018 at 8:47:06 AM UTC-5, Diego Mesa wrote:
>>>
>>> Hey Jeremy,
>>>
>>> Thanks for your feedback - I agree search needs to performant. Im glad
>>> you're also looking at these issues!
>>>
>>> A priority for me is the inclusion of *fuzzy* searching - a quick look
>>> seems lunr does support it:
>>>
>>> https://lunrjs.com/guides/searching.html#fuzzy-matches
>>>
>>> but elastic does not.
>>>
>>> Best,
>>> Diego
>>>
>>> On Wednesday, June 27, 2018 at 2:28:18 AM UTC-5, Jeremy Ruston wrote:
Hi Diego
Filter operators are often re-executed and so it's important that they
perform well. For an expensive operation like a search we need to try to
cache as much of the work as we can.
An approach that should work is to introduce a new startup module that
tracks changes to the store in order to keep the index up to date, and
then
have the filter operator call a version of the search function that caches
the results (and clears the cache when it is invalidated by changes to the
tiddler store).
Having said that, one of the client projects I'm working on for
Federatial looks like it's going to need an enhanced search capability. If
the project proceeds to the next phase then I'll be looking at adding a
third party search index library -- I've been looking at
https://lunrjs.com/ and http://elasticlunr.com/. The API that they
feature allows the index to be dynamically modified by adding and removing
data incrementally; fuse.js looks like it only accepts a monolithic block
of data, and would require a re-index each time the tiddler store changes.
Best wishes
Jeremy
--
Jeremy Ruston
jer...@jermolene.com
https://jermolene.com
On 26 Jun 2018, at 20:45, Diego Mesa wrote:
Hey all,
Thanks Jeremy for sending that. The following is *more pseduo-code
than anything*, but reflects my quick hacking:
(function(){
/*jslint node: true, browser: true */
/*global $tw: false */
"use strict";
var fuze = require("$:/plugins/fuzzy/fuze.js");
var options = {
shouldSort: true,
matchAllTokens: true,
threshold: 0.6,
location: 0,
distance: 100,
maxPatternLength: 32,
minMatchCharLength: 1,
keys: undefined
};
/*
Export our filter function
*/
exports.fuzzysearch = function(source,operator,options) {
var results = [];
source = options.source || $tw.wiki.each;
var searchTiddler = function(title){
jsonData = $tw.wiki.getTiddlerAsJson(title)
var fuse = new Fuse(jsonData, options); // "list" is the item
array
var result = fuse.search(operator.operand);
return result;
};
source(function(tiddler,title) {
if(searchTiddler(title) !== options.invert) {
results.push(title);
}
});
console.log("e");
return results;
};
})();
I quickly generated this just by looking at:
$:/core/modules/filters/search.js
which is just a wrapper around:
$:/core/modules/wiki.js
where exports.search = is defined.
The issue is that as you can see from
http://fusejs.io/
Fuze works better if it would have access to the entire json
Re: [twdev] Re: Request for help with new filter operator and new tutorial
Hi everyone!
Diego was kind enough to bring this discussion to my attention - I plan on
fixing up that issue he ran into.
Jeremy - if you decide to go with lunr.js but not use my plugin, please
feel free to reach out if you have questions about lunr.js! One pitfall I
ran into while upgrading from lunr 1.x to 2.x is that lunr's index is
immutable by default in 2.x - I ended up writing a plugin to add mutable
indexes. Of course, any collaboration on my plugin (along with advice on
how to improve it) would be most welcome!
-Rob
On Thursday, June 28, 2018 at 9:14:27 AM UTC-5, Diego Mesa wrote:
>
> Hey Jeremy,
>
> Just wanted to let you know of this plugin:
>
> https://github.com/hoelzro/tw-full-text-search
>
> which incorporates lunr.js into TW for full text searching. I just tried
> today and am somewhat able to do fuzzy searching with it (does not deal
> with immediatel searching properly yet). I have let him know about this
> discussion and the corresponding feature request on github, and have opened
> an issue on his repo for the fuzzy matching:
>
> https://github.com/hoelzro/tw-full-text-search/issues/4
>
> Perhaps you can reuse some of hoelzro's work?
>
> Best,
> Diego
>
> On Wednesday, June 27, 2018 at 8:47:06 AM UTC-5, Diego Mesa wrote:
>>
>> Hey Jeremy,
>>
>> Thanks for your feedback - I agree search needs to performant. Im glad
>> you're also looking at these issues!
>>
>> A priority for me is the inclusion of *fuzzy* searching - a quick look
>> seems lunr does support it:
>>
>> https://lunrjs.com/guides/searching.html#fuzzy-matches
>>
>> but elastic does not.
>>
>> Best,
>> Diego
>>
>> On Wednesday, June 27, 2018 at 2:28:18 AM UTC-5, Jeremy Ruston wrote:
>>>
>>> Hi Diego
>>>
>>> Filter operators are often re-executed and so it's important that they
>>> perform well. For an expensive operation like a search we need to try to
>>> cache as much of the work as we can.
>>>
>>> An approach that should work is to introduce a new startup module that
>>> tracks changes to the store in order to keep the index up to date, and then
>>> have the filter operator call a version of the search function that caches
>>> the results (and clears the cache when it is invalidated by changes to the
>>> tiddler store).
>>>
>>> Having said that, one of the client projects I'm working on for
>>> Federatial looks like it's going to need an enhanced search capability. If
>>> the project proceeds to the next phase then I'll be looking at adding a
>>> third party search index library -- I've been looking at
>>> https://lunrjs.com/ and http://elasticlunr.com/. The API that they
>>> feature allows the index to be dynamically modified by adding and removing
>>> data incrementally; fuse.js looks like it only accepts a monolithic block
>>> of data, and would require a re-index each time the tiddler store changes.
>>>
>>> Best wishes
>>>
>>> Jeremy
>>>
>>> --
>>> Jeremy Ruston
>>> jer...@jermolene.com
>>> https://jermolene.com
>>>
>>> On 26 Jun 2018, at 20:45, Diego Mesa wrote:
>>>
>>> Hey all,
>>>
>>> Thanks Jeremy for sending that. The following is *more pseduo-code than
>>> anything*, but reflects my quick hacking:
>>>
>>> (function(){
>>>
>>>
>>> /*jslint node: true, browser: true */
>>> /*global $tw: false */
>>> "use strict";
>>>
>>>
>>> var fuze = require("$:/plugins/fuzzy/fuze.js");
>>>
>>>
>>> var options = {
>>> shouldSort: true,
>>> matchAllTokens: true,
>>> threshold: 0.6,
>>> location: 0,
>>> distance: 100,
>>> maxPatternLength: 32,
>>> minMatchCharLength: 1,
>>> keys: undefined
>>> };
>>>
>>>
>>> /*
>>> Export our filter function
>>> */
>>> exports.fuzzysearch = function(source,operator,options) {
>>> var results = [];
>>>
>>>
>>> source = options.source || $tw.wiki.each;
>>>
>>>
>>> var searchTiddler = function(title){
>>> jsonData = $tw.wiki.getTiddlerAsJson(title)
>>> var fuse = new Fuse(jsonData, options); // "list" is the item
>>> array
>>> var result = fuse.search(operator.operand);
>>> return result;
>>>};
>>> source(function(tiddler,title) {
>>> if(searchTiddler(title) !== options.invert) {
>>> results.push(title);
>>> }
>>> });
>>>
>>>
>>> console.log("e");
>>> return results;
>>> };
>>>
>>>
>>> })();
>>>
>>> I quickly generated this just by looking at:
>>>
>>> $:/core/modules/filters/search.js
>>>
>>> which is just a wrapper around:
>>>
>>> $:/core/modules/wiki.js
>>>
>>> where exports.search = is defined.
>>>
>>> The issue is that as you can see from
>>>
>>> http://fusejs.io/
>>>
>>> Fuze works better if it would have access to the entire json string of
>>> the wiki, and it can handle the searching and scoring itself.
>>>
>>> What do you guys think? Any tips to help me move forward?
>>>
>>> Best,
>>> Diego
>>>
>>> On Tuesday, June 26, 2018 at 8:38:36 AM UTC-5, Jed Carty wrote:
I hadn't realised it could be done that way. That makes including
Re: [twdev] Re: Request for help with new filter operator and new tutorial
Hey Jeremy,
Just wanted to let you know of this plugin:
https://github.com/hoelzro/tw-full-text-search
which incorporates lunr.js into TW for full text searching. I just tried
today and am somewhat able to do fuzzy searching with it (does not deal
with immediatel searching properly yet). I have let him know about this
discussion and the corresponding feature request on github, and have opened
an issue on his repo for the fuzzy matching:
https://github.com/hoelzro/tw-full-text-search/issues/4
Perhaps you can reuse some of hoelzro's work?
Best,
Diego
On Wednesday, June 27, 2018 at 8:47:06 AM UTC-5, Diego Mesa wrote:
>
> Hey Jeremy,
>
> Thanks for your feedback - I agree search needs to performant. Im glad
> you're also looking at these issues!
>
> A priority for me is the inclusion of *fuzzy* searching - a quick look
> seems lunr does support it:
>
> https://lunrjs.com/guides/searching.html#fuzzy-matches
>
> but elastic does not.
>
> Best,
> Diego
>
> On Wednesday, June 27, 2018 at 2:28:18 AM UTC-5, Jeremy Ruston wrote:
>>
>> Hi Diego
>>
>> Filter operators are often re-executed and so it's important that they
>> perform well. For an expensive operation like a search we need to try to
>> cache as much of the work as we can.
>>
>> An approach that should work is to introduce a new startup module that
>> tracks changes to the store in order to keep the index up to date, and then
>> have the filter operator call a version of the search function that caches
>> the results (and clears the cache when it is invalidated by changes to the
>> tiddler store).
>>
>> Having said that, one of the client projects I'm working on for
>> Federatial looks like it's going to need an enhanced search capability. If
>> the project proceeds to the next phase then I'll be looking at adding a
>> third party search index library -- I've been looking at
>> https://lunrjs.com/ and http://elasticlunr.com/. The API that they
>> feature allows the index to be dynamically modified by adding and removing
>> data incrementally; fuse.js looks like it only accepts a monolithic block
>> of data, and would require a re-index each time the tiddler store changes.
>>
>> Best wishes
>>
>> Jeremy
>>
>> --
>> Jeremy Ruston
>> jer...@jermolene.com
>> https://jermolene.com
>>
>> On 26 Jun 2018, at 20:45, Diego Mesa wrote:
>>
>> Hey all,
>>
>> Thanks Jeremy for sending that. The following is *more pseduo-code than
>> anything*, but reflects my quick hacking:
>>
>> (function(){
>>
>>
>> /*jslint node: true, browser: true */
>> /*global $tw: false */
>> "use strict";
>>
>>
>> var fuze = require("$:/plugins/fuzzy/fuze.js");
>>
>>
>> var options = {
>> shouldSort: true,
>> matchAllTokens: true,
>> threshold: 0.6,
>> location: 0,
>> distance: 100,
>> maxPatternLength: 32,
>> minMatchCharLength: 1,
>> keys: undefined
>> };
>>
>>
>> /*
>> Export our filter function
>> */
>> exports.fuzzysearch = function(source,operator,options) {
>> var results = [];
>>
>>
>> source = options.source || $tw.wiki.each;
>>
>>
>> var searchTiddler = function(title){
>> jsonData = $tw.wiki.getTiddlerAsJson(title)
>> var fuse = new Fuse(jsonData, options); // "list" is the item
>> array
>> var result = fuse.search(operator.operand);
>> return result;
>>};
>> source(function(tiddler,title) {
>> if(searchTiddler(title) !== options.invert) {
>> results.push(title);
>> }
>> });
>>
>>
>> console.log("e");
>> return results;
>> };
>>
>>
>> })();
>>
>> I quickly generated this just by looking at:
>>
>> $:/core/modules/filters/search.js
>>
>> which is just a wrapper around:
>>
>> $:/core/modules/wiki.js
>>
>> where exports.search = is defined.
>>
>> The issue is that as you can see from
>>
>> http://fusejs.io/
>>
>> Fuze works better if it would have access to the entire json string of
>> the wiki, and it can handle the searching and scoring itself.
>>
>> What do you guys think? Any tips to help me move forward?
>>
>> Best,
>> Diego
>>
>> On Tuesday, June 26, 2018 at 8:38:36 AM UTC-5, Jed Carty wrote:
>>>
>>> I hadn't realised it could be done that way. That makes including
>>> external libraries a lot easier than I thought. I am a bit embarrassed that
>>> I didn't notice you could do that.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "TiddlyWikiDev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to tiddlywikide...@googlegroups.com.
>> To post to this group, send email to tiddly...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/tiddlywikidev.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/tiddlywikidev/c0e08271-5dd4-4578-b2b3-d4693c27a8b4%40googlegroups.com
>>
>>
Re: [twdev] Re: Request for help with new filter operator and new tutorial
Hey Jeremy,
Thanks for your feedback - I agree search needs to performant. Im glad
you're also looking at these issues!
A priority for me is the inclusion of *fuzzy* searching - a quick look
seems lunr does support it:
https://lunrjs.com/guides/searching.html#fuzzy-matches
but elastic does not.
Best,
Diego
On Wednesday, June 27, 2018 at 2:28:18 AM UTC-5, Jeremy Ruston wrote:
>
> Hi Diego
>
> Filter operators are often re-executed and so it's important that they
> perform well. For an expensive operation like a search we need to try to
> cache as much of the work as we can.
>
> An approach that should work is to introduce a new startup module that
> tracks changes to the store in order to keep the index up to date, and then
> have the filter operator call a version of the search function that caches
> the results (and clears the cache when it is invalidated by changes to the
> tiddler store).
>
> Having said that, one of the client projects I'm working on for Federatial
> looks like it's going to need an enhanced search capability. If the project
> proceeds to the next phase then I'll be looking at adding a third party
> search index library -- I've been looking at https://lunrjs.com/ and
> http://elasticlunr.com/. The API that they feature allows the index to be
> dynamically modified by adding and removing data incrementally; fuse.js
> looks like it only accepts a monolithic block of data, and would require a
> re-index each time the tiddler store changes.
>
> Best wishes
>
> Jeremy
>
> --
> Jeremy Ruston
> jer...@jermolene.com
> https://jermolene.com
>
> On 26 Jun 2018, at 20:45, Diego Mesa >
> wrote:
>
> Hey all,
>
> Thanks Jeremy for sending that. The following is *more pseduo-code than
> anything*, but reflects my quick hacking:
>
> (function(){
>
>
> /*jslint node: true, browser: true */
> /*global $tw: false */
> "use strict";
>
>
> var fuze = require("$:/plugins/fuzzy/fuze.js");
>
>
> var options = {
> shouldSort: true,
> matchAllTokens: true,
> threshold: 0.6,
> location: 0,
> distance: 100,
> maxPatternLength: 32,
> minMatchCharLength: 1,
> keys: undefined
> };
>
>
> /*
> Export our filter function
> */
> exports.fuzzysearch = function(source,operator,options) {
> var results = [];
>
>
> source = options.source || $tw.wiki.each;
>
>
> var searchTiddler = function(title){
> jsonData = $tw.wiki.getTiddlerAsJson(title)
> var fuse = new Fuse(jsonData, options); // "list" is the item
> array
> var result = fuse.search(operator.operand);
> return result;
>};
> source(function(tiddler,title) {
> if(searchTiddler(title) !== options.invert) {
> results.push(title);
> }
> });
>
>
> console.log("e");
> return results;
> };
>
>
> })();
>
> I quickly generated this just by looking at:
>
> $:/core/modules/filters/search.js
>
> which is just a wrapper around:
>
> $:/core/modules/wiki.js
>
> where exports.search = is defined.
>
> The issue is that as you can see from
>
> http://fusejs.io/
>
> Fuze works better if it would have access to the entire json string of the
> wiki, and it can handle the searching and scoring itself.
>
> What do you guys think? Any tips to help me move forward?
>
> Best,
> Diego
>
> On Tuesday, June 26, 2018 at 8:38:36 AM UTC-5, Jed Carty wrote:
>>
>> I hadn't realised it could be done that way. That makes including
>> external libraries a lot easier than I thought. I am a bit embarrassed that
>> I didn't notice you could do that.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWikiDev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywikide...@googlegroups.com .
> To post to this group, send email to tiddly...@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/tiddlywikidev.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tiddlywikidev/c0e08271-5dd4-4578-b2b3-d4693c27a8b4%40googlegroups.com
>
>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
--
You received this message because you are subscribed to the Google Groups
"TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to tiddlywikidev+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywikidev@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.
To view this discussion on the web visit
https://groups.google.com/d/msgid/tiddlywikidev/aab84e80-923e-4395-8387-604b23a3db61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [twdev] Re: Request for help with new filter operator and new tutorial
Hi Diego
Filter operators are often re-executed and so it's important that they perform
well. For an expensive operation like a search we need to try to cache as much
of the work as we can.
An approach that should work is to introduce a new startup module that tracks
changes to the store in order to keep the index up to date, and then have the
filter operator call a version of the search function that caches the results
(and clears the cache when it is invalidated by changes to the tiddler store).
Having said that, one of the client projects I'm working on for Federatial
looks like it's going to need an enhanced search capability. If the project
proceeds to the next phase then I'll be looking at adding a third party search
index library -- I've been looking at https://lunrjs.com/ and
http://elasticlunr.com/. The API that they feature allows the index to be
dynamically modified by adding and removing data incrementally; fuse.js looks
like it only accepts a monolithic block of data, and would require a re-index
each time the tiddler store changes.
Best wishes
Jeremy
--
Jeremy Ruston
jer...@jermolene.com
https://jermolene.com
> On 26 Jun 2018, at 20:45, Diego Mesa wrote:
>
> Hey all,
>
> Thanks Jeremy for sending that. The following is more pseduo-code than
> anything, but reflects my quick hacking:
>
> (function(){
>
>
> /*jslint node: true, browser: true */
> /*global $tw: false */
> "use strict";
>
>
> var fuze = require("$:/plugins/fuzzy/fuze.js");
>
>
> var options = {
> shouldSort: true,
> matchAllTokens: true,
> threshold: 0.6,
> location: 0,
> distance: 100,
> maxPatternLength: 32,
> minMatchCharLength: 1,
> keys: undefined
> };
>
>
> /*
> Export our filter function
> */
> exports.fuzzysearch = function(source,operator,options) {
> var results = [];
>
>
> source = options.source || $tw.wiki.each;
>
>
> var searchTiddler = function(title){
> jsonData = $tw.wiki.getTiddlerAsJson(title)
> var fuse = new Fuse(jsonData, options); // "list" is the item array
> var result = fuse.search(operator.operand);
> return result;
>};
> source(function(tiddler,title) {
> if(searchTiddler(title) !== options.invert) {
> results.push(title);
> }
> });
>
>
> console.log("e");
> return results;
> };
>
>
> })();
>
> I quickly generated this just by looking at:
>
> $:/core/modules/filters/search.js
>
> which is just a wrapper around:
>
> $:/core/modules/wiki.js
>
> where exports.search = is defined.
>
> The issue is that as you can see from
>
> http://fusejs.io/
>
> Fuze works better if it would have access to the entire json string of the
> wiki, and it can handle the searching and scoring itself.
>
> What do you guys think? Any tips to help me move forward?
>
> Best,
> Diego
>
>> On Tuesday, June 26, 2018 at 8:38:36 AM UTC-5, Jed Carty wrote:
>> I hadn't realised it could be done that way. That makes including external
>> libraries a lot easier than I thought. I am a bit embarrassed that I didn't
>> notice you could do that.
>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWikiDev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywikidev+unsubscr...@googlegroups.com.
> To post to this group, send email to tiddlywikidev@googlegroups.com.
> Visit this group at https://groups.google.com/group/tiddlywikidev.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tiddlywikidev/c0e08271-5dd4-4578-b2b3-d4693c27a8b4%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to tiddlywikidev+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywikidev@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.
To view this discussion on the web visit
https://groups.google.com/d/msgid/tiddlywikidev/193465C7-43F2-4CD2-8DBF-02593D190C36%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
