[tw5] Re: Intertwingling the TiddlyWiki - TF-IDF and tag inference

2019-01-22 Thread Kalmir
DevonThink app should be really good at this: "One of the key features of DevonThink Pro Office is its smart searching algorithms, its ability to suggest similar texts based on the contents of what you are looking at, etc. It does this by means of a proprietary algorithm, so I can't really

[tw5] Re: Intertwingling the TiddlyWiki - TF-IDF and tag inference

2019-01-22 Thread Rob Hoelz
That's a neat trick involving compression, Joe - I wonder if you could adapt a locality sensitive hash like simhash to create a specialized index for quick comparisons? Another thing that your mention of rsync reminded me of is word embeddings, in particular the latest and greatest in that

[tw5] Re: Intertwingling the TiddlyWiki - TF-IDF and tag inference

2019-01-22 Thread TonyM
Joe, We share the same aims, and as an experienced information/knowledge management professional I look forward to us finding effective tools to make these "inferences" using software and I am keen to contribute to this as well. Though given my own experience in this, I also find that every

[tw5] Re: Intertwingling the TiddlyWiki - TF-IDF and tag inference

2019-01-22 Thread Dave
Instead of comparing paragraphs or tiddlers, how hard would it be too detect unique sentences and compare them? I guess you'd almost need an AI to do that, hey? -- You received this message because you are subscribed to the Google Groups "TiddlyWiki" group. To unsubscribe from this group and

[tw5] Re: Intertwingling the TiddlyWiki - TF-IDF and tag inference

2019-01-22 Thread Joe Armstrong
YES - For a very long time I've wanted an assistant that watches what I do and helps me - this is my ultimate goal. I want to reduce entropy - I want to discover similar tiddlers and merge them to reduce entropy. I have been thinking about how to do this for 30 odd years (not for tiddlers -

[tw5] Re: Intertwingling the TiddlyWiki - TF-IDF and tag inference

2019-01-21 Thread Rob Hoelz
Again, thanks for sharing, Joe! I looked through the PDF and had a few thoughts: * Did you do any additional processing of the tiddler bodies, eg. stemming, chunking into bigrams/trigrams, or stripping out various wikitext elements like URLs? If you did, I'd be curious to hear how that

[tw5] Re: Intertwingling the TiddlyWiki - TF-IDF and tag inference

2019-01-21 Thread Rob Hoelz
Thanks, Joe! I'll read over that PDF you sent over; as far as the code goes, I think the PDF documentation describing the methodology should suffice. -Rob On Monday, January 21, 2019 at 11:33:31 AM UTC-6, Joe Armstrong wrote: > > The code I wrote was a bit messy and just as an experiment. >

[tw5] Re: Intertwingling the TiddlyWiki - TF-IDF and tag inference

2019-01-21 Thread Joe Armstrong
The code I wrote was a bit messy and just as an experiment. Good enough for proof of concept but not for production - it was just written to test a few ideas. I don't mind sending you a private copy - but explaining how it works would be low priority. A better idea would be for me to put it