Thanks Mark!.
Do you have any comment regarding the performance differences between
indexing TSV files as opposed to directly indexing each document via
http post?
-----Original Message-----
From: Mark Miller [mailto:[EMAIL PROTECTED]
Sent: Wednesday, September 24, 2008 2:12 PM
To: solr-user@lucene.apache.org
Subject: Re: help required: how to design a large scale solr system
From my limited experience:
I think you might have a bit of trouble getting 60 mil docs on a single
machine. Cached queries will probably still be *very* fast, but non
cached queries are going to be very slow in many cases. Is that 5
seconds for all queries? You will never meet that on first run queries
with 60mil docs on that machine. The light query load might make things
workable...but your near the limits of a single machine (4 core or not)
with 60 mil. You want to use a very good stopword list...common term
queries will be killer. The docs being so small will be your only
possible savior if you go the one machine route - that and cached hits.
You don't have enough ram to get as much of the filesystem into RAM as
youd like for 60 mil docs either.
I think you might try two machines with 30, 3 with 20, or 4 with 15. The
more you spread, even with slower machines, the faster your likely to
index, which as you say, will take a long time for 60 mil docs (start
today <g>). Multiple machines will help the indexing speed the most for
sure - its still going to take a long time.
I don't think you will get much advantage using more than one solr
install on a single machine - if you do, that should be addressed in the
code, even with RAID.
So I say, spread if you can. Faster indexing, faster search, easy to
expand later. Distributed search is so easy with solr 1.3, you wont
regret it. I think there is a bug to be addressed if your needing this
in a week though - in my experience, with distributed search, for every
million docs on a machine beyond the first, you lose a doc in a search
across all machines (ie 1 mil on machine 1, 1 million on machine 2, a
*:* search will be missing 1 doc. 10 mil each on 3 machines, a *:*
search will be missing 30. Not a big deal, but could be a concern for
some with picky, look at everything customers.
- Mark
Ben Shlomo, Yatir wrote:
Hi!
I am already using solr 1.2 and happy with it.
In a new project with very tight dead line (10 development days from
today) I need to setup a more ambitious system in terms of scale
Here is the spec:
* I need to index about 60,000,000
documents
* Each document is has 11 textual fields to be indexed &
stored
and 4 more fields to be stored only
* Most fields are short (2-14 characters) however 2 indexed
fields can be up to 1KB and another stored field is up to 1KB
* On average every document is about 0.5 KB to be stored and
0.4KB to be indexed
* The SLA for data freshness is a full nightly re-index ( I
cannot obtain an incremental update/delete lists of the modified
documents)
* The SLA for query time is 5 seconds
* the number of expected queries is 2-3 queries per second
* the queries are simple a combination of Boolean operation
and
name searches (no fancy fuzzy searches and levinstien distances, no
faceting, etc)
* I have a 64 bit Dell 2950 4-cpu machine (2 dual cores )
with
RAID 10, 200 GB HD space, and 8GB RAM memory
* The documents are not given to me explicitly - I am given a
raw-documents in RAM - one by one, from which I create my document in
RAM.
and then I can either http-post is to index it directly or append it
to
a tsv file for later indexing
* Each document has a unique ID
I have a few directions I am thinking about
The simple approach
* Have one solr instance that will
index
the entire document set (from files). I am afraid this will take too
much time
Direction 1
* Create TSV files from all the
documents - this will take around 3-4 hours
* Have all the documents partitioned
into several subsets (how many should I choose? )
* Have multiple solr instances on the
same machine
* Let each solr instance concurrently
index the appropriate subset
* At the end merge all the indices
using
the IndexMergeTool - (how much time will it take ?)
Direction 2
* Like the previous but instead of
using the IndexMergeTool , use distributed search with shards
(upgrading
to solr 1.3)
Direction 3,4
* Like previous directions only avoid
using TSV files at all and directly index the documents from RAM
Questions:
* Which direction do you recommend in order to meet the SLAs
in
the fastest way?
* Since I have RAID on the machine can I gain performance by
using multiple solr instances on the same machine or only multiple
machines will help me
* What's the minimal number of machines I should require (I
might get more weaker machines)
* How many concurrent indexers are recommended?
* Do you agree that the bottle neck is the indexing time?
Any help is appreciated
Thanks in advance
yatir
