30 million - that's feasible on a single (beefy) Solr server but whether
it's advisable to go distributed or not depends on other factors, like query
speed issues you may have with that many docs in a single server, expected
collection growth, and so on.
As for your questions further below
A quick add on to this -- we have over 30 million documents.
I take it that we should be looking @ Distributed Solr?
as in
http://www.lucidimagination.com/content/scaling-lucene-and-solr#d0e344
Thanks.
On Mon, Feb 27, 2012 at 2:33 PM, Memory Makers wrote:
> Many thanks for the response.
>
> H
Many thanks for the response.
Here is the revised questions:
For example if I have N processes that are producing documents to index:
1. Should I have them simultaneously submit documents to Solr (will this
improve the indexing throughput)?
2. Is there anything I can do Solr configuration wise th
My two cents:
- pulling is better than pushing -
http://wiki.apache.org/solr/Solrj#Streaming_documents_for_an_update
- DIH is not thread safe https://issues.apache.org/jira/browse/SOLR-3011 But
there are few patches for trunk which fix it.
Regards
On Mon, Feb 27, 2012 at 10:46 PM, Erik Hatcher
Yes, absolutely. Parallelizing indexing can make a huge difference. How you
do so will depend on your indexing environment. Most crudely, running multiple
indexing scripts on different subsets of data up to the the limitations of your
operating system and hardware is how many do it. SolrJ h
sday, October 13, 2009 11:07 AM
To:
Subject: Re: Tips on speeding up indexing needed...
: References: <4acb30d2.2010...@umich.edu>
: <69de18140910070109m27e50d2sc82a7c7bdd683...@mail.gmail.com>
: <4acc95a3.5000...@umich.edu>
:
: <4acfc943.4040...@umich.e
: References: <4acb30d2.2010...@umich.edu>
: <69de18140910070109m27e50d2sc82a7c7bdd683...@mail.gmail.com>
: <4acc95a3.5000...@umich.edu>
:
: <4acfc943.4040...@umich.edu>
: In-Reply-To: <4acfc943.4040...@umich.edu>
: Subject: Tips on speedi
Thanks, Lance. I already commit at the end. I will take a look at the data
import handler. Thanks again!
-- Bill
--
From: "Lance Norskog"
Sent: Saturday, October 10, 2009 7:58 PM
To:
Subject: Re: Tips on speeding up indexing nee
ur help!
>
> -- Bill
>
> --
> From: "William Pierce"
> Sent: Saturday, October 10, 2009 5:47 PM
> To:
> Subject: Tips on speeding up indexing needed...
>
>> Folks:
>>
>> I have a corpus of approx 6 M
y, October 10, 2009 5:47 PM
To:
Subject: Tips on speeding up indexing needed...
Folks:
I have a corpus of approx 6 M documents each of approx 4K bytes.
Currently, the way indexing is set up I read documents from a database and
issue solr post requests in batches (batches are set up s
Folks:
I have a corpus of approx 6 M documents each of approx 4K bytes.
Currently, the way indexing is set up I read documents from a database and
issue solr post requests in batches (batches are set up so that the
maxPostSize of tomcat which is set to 2MB is adhered to). This means that
in
--
>> From: Britske
>> To: solr-user@lucene.apache.org
>> Sent: Wednesday, March 25, 2009 10:05:17 AM
>> Subject: speeding up indexing with a LOT of indexed fields
>>
>>
>> hi,
>>
>> I'm having difficulty indexing a collection of documents
- Original Message
> From: Britske
> To: solr-user@lucene.apache.org
> Sent: Wednesday, March 25, 2009 10:05:17 AM
> Subject: speeding up indexing with a LOT of indexed fields
>
>
> hi,
>
> I'm having difficulty indexing a collection of documents in a reasonable
- lastly: should I be able to get more out of this box or am I just
complaining ;-)
Thanks for making it to here,
and hoping to receive some valuable info,
Cheers,
Britske
--
View this message in context:
http://www.nabble.com/speeding-up-indexing-with-a-LOT-of-indexed-fields-tp22702364p22702364.html
Sent from the Solr - User mailing list archive at Nabble.com.
14 matches
Mail list logo
