I have been working on some robust algorithms for text summarization and 
matching. A very approximate (and misleading, but this is not important now) 
description is :

a) turn documents into lists of sentences
b) turn sentences into lists of words
c) estimate words statistics
d) turn sentences into lists of features, represented by numerical ids
c) compare sentences

For this purpose, dictionaries which keys are strings are used (implemented 
with tries). I have recently begun to study the database and have been 
wondering it would not be better to used it for that purpose. The idea would be 

a database containing document, sentences, words, features, ...

And it would be possible to get the number of occurrences of one word in all or 
one document, or the sentences which contain a certain word or feature for 

Does this sound reasonable?

Is the database fast enough? The basic statistic is word occurrence, which 
means that each word encountered would have to be searched within the external 
symbols : finding the value associated to a string in a trie remains fast 
whatever the numbers of keys. 

Is it possible to automatically propagate some information within the database? 
For example, when a word is read, its occurrence number have to be incremented, 
but also the occurrences of its related features.

Numerical ids have been only used for speed reasons, determining the equality 
of two numbers is much faster than for two strings. This makes nevertheless 
debugging unpleasant. What about comparing two external symbols?



UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Reply via email to