uuuhhhh, take anything Otis says as *much* more informed than anything I say
on this topic <G>.

Erick

On Thu, Sep 18, 2008 at 2:32 AM, Tobias Larsson Hult <
[EMAIL PROTECTED]> wrote:

> Thanks for the quick responses!
>
> Good point about the warmup issues Erick, that's something we will
> consider. Good to know that this kind of setup has been proved working for
> at least one :) I think we will do a small setup and test the performance.
>
> Thanks again for valuable input!
>
> Best Regards
> Tobias
>
>
> On 16 sep 2008, at 18.10, Otis Gospodnetic wrote:
>
>  Tobias,
>>
>> That's the approach I took with Simpy.com and it's been working well for
>> several years now.  You'll have to keep track of searchers and close them
>> when appropriate, of course.
>>
>> Otis
>> --
>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>
>>
> On 16 sep 2008, at 17.17, Erick Erickson wrote:
>
>>
>> The main arguments against using many separate indexes are
>> 1> search warmup time. That is, each time you open an index
>>    the first few queries take much longer than subsequent searches.
>> 2> Managing a bazillion indexes is non-trivial.
>>
>>
>> That said, in your particular case these may not apply. I guess the
>> piece of information that really counts is "how often do you expect
>> to update/search a given index"? You could avoid the warmup issue
>> by keeping an index open for some period of time after the first
>> search on the assumption that the user is going to make multiple
>> searches rather than just one. I'm sure there are other tricks
>> you can try.
>>
>> So, how often do you expect
>> 1> users to backup date
>> 2> users to query data?
>> and what is acceptable search response time? and are your
>> users willing to live with a significant delay on the first couple
>> of queries?
>>
>> I'd only be comfortable with choosing an approach if I tried
>> it out with a single computer's content and generated a few
>> stats....
>>
>> Best
>> Erick
>>
>> ----- Original Message ----
>>
>>> From: Tobias Larsson Hult <[EMAIL PROTECTED]>
>>> To: java-user@lucene.apache.org
>>> Sent: Tuesday, September 16, 2008 10:55:09 AM
>>> Subject: Using separate index for each user
>>>
>>> Hi,
>>>
>>> We're thinking of using Lucene to integrate search in a backup service
>>> application. The background is that we have a bunch of users using a
>>> backup service, and we want them to be able to search their own, and
>>> only their own, backups.
>>>
>>> The total amount of data that's being backed up is very large (size in
>>> terabyte). Even though the index will probably be smaller due to only
>>> indexing relevant fields, it is still to much to incorporate in one
>>> index. But since a user will only search in his/her own files we're
>>> thinking of creating one index for each user. There will be a lot of
>>> indexes of course but each index will not span to more than a couple
>>> of gigabytes at the most.
>>>
>>> So when a user searches or adds new content to the backup we will open
>>> up his/her index and to a search/update in that particular index. That
>>> way, each query/update should not be so performance intense.
>>>
>>> Does this sound like a reasonable solution?  Of course this means
>>> creating a lot of IndexReaders/Writers but I prefer that to searching
>>> in a huge index everytime when a user only wants to search in a slice
>>> of the total index.
>>>
>>> Best Regards,
>>> Tobias Larsson Hult
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>

Reply via email to