On Mar 7, 2008, at 7:28, Sebastian Steins <[EMAIL PROTECTED]>
wrote:
Hi there,
first of all, I want to introduce myself, because I have not posted
here before. My name is Sebastian and I am currently working as web
application developer. Search and Information Retrieval are not part
of my current work, but I am interested in those fields as a hobby.
I was triggered by the book "Programming Collective Intelligence"
which describes very complex algorithms like clustering in a very
easy way and shows the solutions in plain python code with SQLite
bindings.
I was very ardent by the solutions in the book, so I tried to modify
them for some experiments and I wanted to use Lucene instead of
PyLucene.
Did you mean to say "Lucene instead of SQLlite" ?
For now, I have a simple script which inserts articles from a RSS
feed into a Lucene index using PyLucene.
An article has also outgoing links, which I store this way:
#### Code ####
for link in params['links']:
doc.add(self.Lucene.Field("linksto", link,
self.Lucene.Field.Store.YES, self.Lucene.Field.Index.UN_TOKENIZED))
#### /Code ####
Is that a good way? Or is there another way in Lucene to store
"relational" data? How would it be possible to retrieve the document
with the most incoming links? Or the document with the greatest
number of outgoing links?
Lucene is not relational database but I assume you know that. It
indexes text tokens and returns documents that contain them.
Additionally, I want to calculate the similarity between documents
with my script, using K-Means, Dendograms and other things (mostly
described in the book mentioned above). Therefore, I would have to
compare a recently found (crawled) article, which is to be written
to the index with all articles in the Lucene index. How can that be
achieved in a more elegant way than doing a for-loop from 0 to
numDocs()? Is there a cheaper (in means of computer ressources) way?
Lucene can do that for you if you index your documents with term
vectors. The "Lucene in Action" book (recommended reading) has an
example on how to implement this. The sample code (called
MoreLikeThis) is available in PyLucene in Python
Unfortunatelly, I am not very familar with Java, so my reasearch for
the above questions in the sites around the Lucene-community did not
help really.
Try harder.
The [EMAIL PROTECTED] has a large community of users that
can help you with Lucene how-to questions that are independent of the
eventual implementation language.
Lucene is implemented in Java, PyLucene just wraps it with Python
wrappers. Familiarity with Java code and docs can be helpful. The
pylucene-dev list you just wrote to is about specific issues related
to that, not general Lucene how-to topics.
I found Mahout, a Java-prorgramm for k-means and other similar
algorithms for Lucene. However, this didn't help, because I want to
implement my experiments in Python, not Java.
Get the theory questions answered first with the book, the web and the
java-user list. Then, take a look at the many python samples that ship
with PyLucene to see how you could apply the solutions you found with
Python.
Andi..
Thank you very much for your help!
Sebastian
_______________________________________________
pylucene-dev mailing list
[email protected]
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
_______________________________________________
pylucene-dev mailing list
[email protected]
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
