I have not looked at the code yet, but look for NovelAnalyzer in Lucene JIRA.
I believe it's supposed to do something similar.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
From: Cam Bazz [EMAIL PROTECTED]
To: solr-user@lucene.apache.org
Sent: Monday, June 9, 2008 3:55:16 PM
Subject: html to text based on some sort of uniqueness metric
Hello,
I am indexing newspaper articles as an excercise in solr. When dealing with
newspaper articles in previous experiences I always tried to get the div or
the table that contains the actual news, using nekohtml traversing tru the
dom tree and getting the text from the div or table that contains the
article. When dealing with many newspapers, it is a hassle to custom code to
extract relevant information. There is usually a lot of garbage in the html.
From categories to ads, and further more they change, so a static coding is
problematic.
I have been thinking if I could measure the frequency or uniqueness for each
node, and find the news automatically - but I have not come up with an
implementation.
Has anyone did/contemplated/used something similar? Maybe there is already a
way - using lucene, or even hadoop.
Best Regards,
-C.A.
