Re: GData Server - Lucene storage

2006-06-07 Thread Ian Boston
Simon, Im picking this thread up from the web archive, but I there was some talk of replication of indexes. This message may not be threaded correctly. I've just completed a custom FSDirectory implementation that is designed to work in a cluster with replication. The anatomy of this cluster i

Re: GData Server - Lucene storage

2006-06-02 Thread Andi Vajda
://www.sleepycat.com/docs/ref/lock/am_conv.html Andi.. - Original Message From: Andi Vajda <[EMAIL PROTECTED]> To: java-dev@lucene.apache.org; jason rutherglen <[EMAIL PROTECTED]> Sent: Friday, June 2, 2006 10:52:27 AM Subject: Re: GData Server - Lucene storage On Fri, 2 Jun

Re: GData Server - Lucene storage

2006-06-02 Thread jason rutherglen
10:52:27 AM Subject: Re: GData Server - Lucene storage On Fri, 2 Jun 2006, jason rutherglen wrote: > It might be interesting to merge using BDB into Solr, as an option to > provide better realtime updates. Perhaps the replication could be used as > well in place of rsync? I don&

Re: GData Server - Lucene storage

2006-06-02 Thread Simon Willnauer
On 6/2/06, Otis Gospodnetic <[EMAIL PROTECTED]> wrote: Simon, I look a quick look at the UML PDF. It seems to me that various *Services are overly complicated. Since you can have only 1 thread modifying the Lucene index, perhaps you should go the same route as IndexModifier (I never used it,

Re: GData Server - Lucene storage

2006-06-02 Thread Otis Gospodnetic
Simon, I look a quick look at the UML PDF. It seems to me that various *Services are overly complicated. Since you can have only 1 thread modifying the Lucene index, perhaps you should go the same route as IndexModifier (I never used it, but it looks like people are using it to manage write/d

Re: GData Server - Lucene storage

2006-06-02 Thread Andi Vajda
On Fri, 2 Jun 2006, jason rutherglen wrote: It might be interesting to merge using BDB into Solr, as an option to provide better realtime updates. Perhaps the replication could be used as well in place of rsync? I don't have any experience with BDB replication, anyone have thoughts on the m

Re: GData Server - Lucene storage

2006-06-02 Thread Yonik Seeley
On 6/2/06, jason rutherglen <[EMAIL PROTECTED]> wrote: It might be interesting to merge using BDB into Solr, as an option to provide better realtime updates. Perhaps the replication could be used as well in place of rsync? I don't have any experience with BDB replication, anyone have thought

Re: GData Server - Lucene storage

2006-06-02 Thread jason rutherglen
ssage From: Simon Willnauer <[EMAIL PROTECTED]> To: Yonik Seeley <[EMAIL PROTECTED]> Cc: java-dev@lucene.apache.org Sent: Friday, June 2, 2006 9:56:24 AM Subject: Re: GData Server - Lucene storage On 6/2/06, Yonik Seeley <[EMAIL PROTECTED]> wrote: > > On 6/2/06, Simo

Re: GData Server - Lucene storage

2006-06-02 Thread Erik Hatcher
On Jun 2, 2006, at 12:56 PM, Simon Willnauer wrote: I had a look at the licence of the sleepycat BerkleyDB for Java dist. and in my opinion it is alright to use and distribute it with the gdata server. Are there any experts on licencing? Is it ok for the ASF to use that? It's ok to use it,

Re: GData Server - Lucene storage

2006-06-02 Thread Simon Willnauer
On 6/2/06, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 6/2/06, Simon Willnauer <[EMAIL PROTECTED]> wrote: > So this would happen quiet often due to updates and inserts. > Hmm it is more and more a bad idea to use a lucene index as a storage. > Rather go straight to a Database. Yes, if you need

Re: GData Server - Lucene storage

2006-06-02 Thread Tatu Saloranta
--- Simon Willnauer <[EMAIL PROTECTED]> wrote: ... > Using the client thread as the indexing thread might > just cause some > performance drawback but that's considerable for Actually, I would not even assume that: handing tasks over between threads causes context switch, and more cache misses. I

Re: GData Server - Lucene storage

2006-06-02 Thread Yonik Seeley
On 6/2/06, Simon Willnauer <[EMAIL PROTECTED]> wrote: The problem here is that there is no action comparable to commit. The entry comes in and will be added to the storage. The delete will be queued but when should the delete operation start. Waiting for the writer to idle?! We could do it that w

Re: GData Server - Lucene storage

2006-06-02 Thread Simon Willnauer
On 6/2/06, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 6/2/06, Simon Willnauer <[EMAIL PROTECTED]> wrote: > This is also true. This problem is still the server response, if i queue > some updates / inserts or index them into a RamDir I still have the problem > of concurrent indexing. The client s

RE: GData Server - Lucene storage

2006-06-02 Thread Robert Engels
5 AM To: java-dev@lucene.apache.org Subject: Re: GData Server - Lucene storage On 6/2/06, Simon Willnauer <[EMAIL PROTECTED]> wrote: > This is also true. This problem is still the server response, if i > queue some updates / inserts or index them into a RamDir I still have > the problem

Re: GData Server - Lucene storage

2006-06-02 Thread Yonik Seeley
On 6/2/06, Simon Willnauer <[EMAIL PROTECTED]> wrote: This is also true. This problem is still the server response, if i queue some updates / inserts or index them into a RamDir I still have the problem of concurrent indexing. The client should wait for the writing process to finish correctly oth

Re: GData Server - Lucene storage

2006-06-02 Thread Simon Willnauer
On 6/2/06, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 6/1/06, Simon Willnauer <[EMAIL PROTECTED]> wrote: > So the results of the search are entry ids and a > corresponding feed. These entries will be retrieved from the storage > and send back to the client. In the simplest case of using a lucen

Re: GData Server - Lucene storage

2006-06-02 Thread Yonik Seeley
On 6/1/06, Simon Willnauer <[EMAIL PROTECTED]> wrote: So the results of the search are entry ids and a corresponding feed. These entries will be retrieved from the storage and send back to the client. In the simplest case of using a lucene stored field to store the original entry, it's a single

Re: GData Server - Lucene storage

2006-06-02 Thread Simon Willnauer
On 6/2/06, Ian Holsman <[EMAIL PROTECTED]> wrote: On 02/06/2006, at 9:37 AM, Simon Willnauer wrote: > The biggest problem with the lucene storage is to achieve a > transactional state. Imagine the following scenario: > An update request comes in. -> the entry to update will be added to > the lu

Re: GData Server - Lucene storage

2006-06-02 Thread Ian Holsman
On 02/06/2006, at 9:37 AM, Simon Willnauer wrote: The biggest problem with the lucene storage is to achieve a transactional state. Imagine the following scenario: An update request comes in. -> the entry to update will be added to the lucene writer who writes the update. But another delete re