Re: Installing PyLucene on Windows
Hi Christian, Out of curiosity, have you already tried using Django's interface to database-based full text search? http://docs.djangoproject.com/en/dev/ref/models/querysets/#search I ask because we are running (a slightly modified) Review Board on top of a Sybase SQL Anywhere database. This thread reminded me that I had wanted to modify the search to use SQL Anywhere's full text indexes. Is there was any specific reason you opted against this approach? Thanks! Nathan Auch Sybase iAnywhere Christian Hammond wrote: No, but good luck with it :) I really would like to find a good replacement for PyLucene. It's a royal pain to install. Christian -- Christian Hammond - chip...@chipx86.com mailto:chip...@chipx86.com Review Board - http://www.review-board.org VMware, Inc. - http://www.vmware.com On Wed, Oct 14, 2009 at 5:35 AM, Thilo-Alexander Ginkel th...@ginkel.com mailto:th...@ginkel.com wrote: Hello everyone, I was thinking about adding search support to our Review Board installation, which would require the installation of PyLucene. Has somebody succeeded in getting this working on a Windows-based platform? Thanks, Thilo To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com mailto:reviewboard%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~--- --~--~-~--~~~---~--~~ Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~---
Re: Installing PyLucene on Windows
Hi Nathan, At the time we wrote search, this functionality didn't exist. However, it's still too limited for our needs. First of all, not all databases that we support are supported by the full text search, meaning that some users would be left out. It makes development harder, since we develop and run nightly tests against sqlite, and doing any development/testing with search would mean we'd have to change our development/test infrastructure to use a database backend that supports it. Also, our search supports a number of search prefixes (file:, author:, etc), which we can't fully implement with the full text seach. The reason for this is that some of these are actually using pre-processed data that doesn't exist in the database as something easily indexable. We want all results to be tied to a particular review request. This is difficult in the cases of, say, file searching and bug number search. By default, we search a blob consisting of the summary, description, testing done, bugs (in a normalized form), submitter name (joined with the full name), and a normalized list of files. If we use full text search, we need to now parse the search string, do possibly expensive searches based on all those prefixes that we find in the string, and do a union of all those results with anything from that blob (which requires searching all those fields individually). This gets pretty hairy, fast. What we really want to do is pre-process all the results into a search index and search from that. PyLucene gives us this. Yes, it's a pain to install, and I hate it as much as anyone else who has tried to install it, but it does give us the functionality we need. I'm hoping to find a better solution in the long term, but I don't believe that the native database full text search is it. Now, maybe if we developed a custom solution that provided tables for storing the pre-processed search indexes and then searched from that using full text search. If we go this route, what I'd like to do is split off the search functionality into its own django app in our tree (reviewboard.search) and add a backends directory with different search backends (fulltextdb.py, pylucene.py). We'd then need a nice API that wraps the backends. I'd be all for a solution like this. Christian -- Christian Hammond - chip...@chipx86.com Review Board - http://www.reviewboard.org VMware, Inc. - http://www.vmware.com On Tue, Oct 20, 2009 at 1:07 PM, Nathan Auch [Sybase] na...@sybase.comwrote: Hi Christian, Out of curiosity, have you already tried using Django's interface to database-based full text search? http://docs.djangoproject.com/en/dev/ref/models/querysets/#search I ask because we are running (a slightly modified) Review Board on top of a Sybase SQL Anywhere database. This thread reminded me that I had wanted to modify the search to use SQL Anywhere's full text indexes. Is there was any specific reason you opted against this approach? Thanks! Nathan Auch Sybase iAnywhere Christian Hammond wrote: No, but good luck with it :) I really would like to find a good replacement for PyLucene. It's a royal pain to install. Christian -- Christian Hammond - chip...@chipx86.com mailto:chip...@chipx86.com Review Board - http://www.review-board.org VMware, Inc. - http://www.vmware.com On Wed, Oct 14, 2009 at 5:35 AM, Thilo-Alexander Ginkel th...@ginkel.com mailto:th...@ginkel.com wrote: Hello everyone, I was thinking about adding search support to our Review Board installation, which would require the installation of PyLucene. Has somebody succeeded in getting this working on a Windows-based platform? Thanks, Thilo To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com mailto:reviewboard%2bunsubscr...@googlegroups.comreviewboard%252bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~--- --~--~-~--~~~---~--~~ Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this
Re: Installing PyLucene on Windows
Christian Hammond wrote: At the time we wrote search, this functionality didn't exist. However, it's still too limited for our needs. . I'm hoping to find a better solution in the long term, but I don't believe that the native database full text search is it. Now, maybe if we developed a custom solution that provided tables for storing the pre-processed search indexes and then searched from that using full text search. If we go this route, what I'd like to do is split off the search functionality into its own django app in our tree (reviewboard.search) and add a backends directory with different search backends (fulltextdb.py, pylucene.py). We'd then need a nice API that wraps the backends. I'd be all for a solution like this. Have you seen Haystack? http://haystacksearch.org/ - specific to Django (models) with a few different search backends implemented. The closest/similar thing is http://swapoff.org/pyndexter which is not Django specific (but has not been updated in a while). Chris --~--~-~--~~~---~--~~ Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~---
Re: Installing PyLucene on Windows
I've seen Haystack but haven't looked into it very much. I'll take a look at these and see what we want to do from here on out. Maybe for 1.2. Thanks, Christian -- Christian Hammond - chip...@chipx86.com Review Board - http://www.reviewboard.org VMware, Inc. - http://www.vmware.com On Tue, Oct 20, 2009 at 3:58 PM, Chris Clark chris.cl...@ingres.com wrote: Christian Hammond wrote: At the time we wrote search, this functionality didn't exist. However, it's still too limited for our needs. . I'm hoping to find a better solution in the long term, but I don't believe that the native database full text search is it. Now, maybe if we developed a custom solution that provided tables for storing the pre-processed search indexes and then searched from that using full text search. If we go this route, what I'd like to do is split off the search functionality into its own django app in our tree (reviewboard.search) and add a backends directory with different search backends (fulltextdb.py, pylucene.py). We'd then need a nice API that wraps the backends. I'd be all for a solution like this. Have you seen Haystack? http://haystacksearch.org/ - specific to Django (models) with a few different search backends implemented. The closest/similar thing is http://swapoff.org/pyndexter which is not Django specific (but has not been updated in a while). Chris To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~--- --~--~-~--~~~---~--~~ Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~---
Re: Installing PyLucene on Windows
No, but good luck with it :) I really would like to find a good replacement for PyLucene. It's a royal pain to install. Christian -- Christian Hammond - chip...@chipx86.com Review Board - http://www.review-board.org VMware, Inc. - http://www.vmware.com On Wed, Oct 14, 2009 at 5:35 AM, Thilo-Alexander Ginkel th...@ginkel.comwrote: Hello everyone, I was thinking about adding search support to our Review Board installation, which would require the installation of PyLucene. Has somebody succeeded in getting this working on a Windows-based platform? Thanks, Thilo To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~--- --~--~-~--~~~---~--~~ Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~---
Re: Installing PyLucene on Windows
Apache Solr - http://lucene.apache.org/solr/ http://wiki.apache.org/solr/SolPython PySolr - http://code.google.com/p/pysolr/ Whether they will be less of a pain to install, I simply don't know. -Tom On Wed, Oct 14, 2009 at 5:09 PM, Christian Hammond chip...@chipx86.comwrote: No, but good luck with it :) I really would like to find a good replacement for PyLucene. It's a royal pain to install. Christian -- Christian Hammond - chip...@chipx86.com Review Board - http://www.review-board.org VMware, Inc. - http://www.vmware.com On Wed, Oct 14, 2009 at 5:35 AM, Thilo-Alexander Ginkel th...@ginkel.comwrote: Hello everyone, I was thinking about adding search support to our Review Board installation, which would require the installation of PyLucene. Has somebody succeeded in getting this working on a Windows-based platform? Thanks, Thilo To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~--- --~--~-~--~~~---~--~~ Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~--~~~~--~~--~--~---
