Hi.
I will as well head into a path like yours within some months from
now.
Currently I have an index of ~10M docs and only store id's in the
index
for
performance and distribution reasons. When we enter a new market I'm
assuming we will soon hit 100M and quite soon after that 1G
documents.
Each
document have in average about 3-5k data.
We will use a GlusterFS installation with RAID1 (or RAID10) SATA
enclosures
as shared storage (think of it as a SAN or shared storage at
least, one
mount point). Hope this will be the right choice, only future can
tell.
Since we are developing a search engine I frankly don't think even
having
100's of SOLR instances serving the index will cut it performance
wise if
we
have one big index. I totally agree with the others claiming that
you most
definitely will go OOE or hit some other constraints of SOLR if
you must
have the whole result in memory sort it and create a xml response.
I did
hit
such constraints when I couldn't afford the instances to have enough
memory
and I had only 1M of docs back then. And think of it... Optimizing
a TB
index will take a long long time and you really want to have an
optimized
index if you want to reduce search time.
I am thinking of a sharding solution where I fragment the index
over the
disk(s) and let each SOLR instance only have little piece of the
total
index. This will require a master database or namenode (or simpler
just a
properties file in each index dir) of some sort to know what docs is
located
on which machine or at least how many docs each shard have. This
is to
ensure that whenever you introduce a new SOLR instance with a new
shard
the
master indexer will know what shard to prioritize. This is
probably not
enough either since all new docs will go to the new shard until it
is
filled
(have the same size as the others) only then will all shards
receive docs
in
a loadbalanced fashion. So whenever you want to add a new indexer
you
probably need to initiate a "stealing" process where it steals
docs from
the
others until it reaches some sort of threshold (10 servers = each
shard
should have 1/10 of the docs or such).
I think this will cut it and enabling us to grow with the data. I
think
doing a distributed reindexing will as well be a good thing when
it comes
to
cutting both indexing and optimizing speed. Probably each indexer
should
buffer it's shard locally on RAID1 SCSI disks, optimize it and
then just
copy it to the main index to minimize the burden of the shared
storage.
Let's say the indexing part will be all fancy and working i TB
scale now
we
come to searching. I personally believe after talking to other
guys which
have built big search engines that you need to introduce a
controller like
searcher on the client side which itself searches in all of the
shards and
merges the response. Perhaps Distributed Solr solves this and will
love to
test it whenever my new installation of servers and enclosures is
finished.
Currently my idea is something like this.
public Page<Document> search(SearchDocumentCommand sdc)
{
Set<Integer> ids = documentIndexers.keySet();
int nrOfSearchers = ids.size();
int totalItems = 0;
Page<Document> docs = new Page(sdc.getPage(),
sdc.getPageSize());
for (Iterator<Integer> iterator = ids.iterator();
iterator.hasNext();)
{
Integer id = iterator.next();
List<DocumentIndexer> indexers = documentIndexers.get(id);
DocumentIndexer indexer =
indexers.get(random.nextInt(indexers.size()));
SearchDocumentCommand sdc2 = copy(sdc);
sdc2.setPage(sdc.getPage()/nrOfSearchers);
Page<Document> res = indexer.search(sdc);
totalItems += res.getTotalItems();
docs.addAll(res);
}
if(sdc.getComparator() != null)
{
Collections.sort(docs, sdc.getComparator());
}
docs.setTotalItems(totalItems);
return docs;
}
This is my RaidedDocumentIndexer which wraps a set of
DocumentIndexers. I
switch from Solr to raw Lucene back and forth benchmarking and
comparing
stuff so I have two implementations of DocumentIndexer
(SolrDocumentIndexer
and LuceneDocumentIndexer) to make the switch easy.
I think this approach is quite OK but the paging stuff is broken I
think.
However the searching speed will at best be constant proportional
to the
number of searchers, probably a lot worse. To get even more speed
each
document indexer should be put into a separate thread with
something like
EDU.oswego.cs.dl.util.concurrent.FutureResult in cojunction with a
thread
pool. The Future result times out after let's say 750 msec and the
client
ignores all searchers which are slower. Probably some performance
metrics
should be gathered about each searcher so the client knows which
indexers
to
prefer over the others.
But of course if you have 50 searchers, having each client thread
spawn
yet
another 50 threads isn't a good thing either. So perhaps a combo of
iterative and parallell search needs to be done with the ratio
configurable.
The controller patterns is used by Google I think I think Peter
Zaitzev
(mysqlperformanceblog) once told me.
Hope I gave some insights in how I plan to scale to TB size and
hopefully
someone smacks me on my head and says "Hey dude do it like this
instead".
Kindly
//Marcus
Phillip Farber wrote:
Hello everyone,
We are considering Solr 1.2 to index and search a terabyte-scale
dataset
of OCR. Initially our requirements are simple: basic tokenizing,
score
sorting only, no faceting. The schema is simple too. A document
consists of a numeric id, stored and indexed and a large text
field,
indexed not stored, containing the OCR typically ~1.4Mb. Some
limited
faceting or additional metadata fields may be added later.
The data in question currently amounts to about 1.1Tb of OCR
(about 1M
docs) which we expect to increase to 10Tb over time. Pilot tests
on the
desktop w/ 2.6 GHz P4 with 2.5 Gb memory, java 1Gb heap on ~180
Mb of
data via HTTP suggest we can index at a rate sufficient to keep
up with
the inputs (after getting over the 1.1 Tb hump). We envision
nightly
commits/optimizes.
We expect to have low QPS (<10) rate and probably will not need
millisecond query response.
Our environment makes available Apache on blade servers (Dell
1955 dual
dual-core 3.x GHz Xeons w/ 8GB RAM) connected to a *large*,
high-performance NAS system over a dedicated (out-of-band) GbE
switch
(Dell PowerConnect 5324) using a 9K MTU (jumbo packets). We are
starting
with 2 blades and will add as demands require.
While we have a lot of storage, the idea of master/slave Solr
Collection
Distribution to add more Solr instances clearly means duplicating
an
immense index. Is it possible to use one instance to update the
index
on NAS while other instances only read the index and commit to keep
their caches warm instead?
Should we expect Solr indexing time to slow significantly as we
scale
up? What kind of query performance could we expect? Is it totally
naive even to consider Solr at this kind of scale?
Given these parameters is it realistic to think that Solr could
handle
the task?
Any advice/wisdom greatly appreciated,
Phil
--
View this message in context:
http://www.nabble.com/Solr-feasibility-with-terabyte-scale-data-tp14963703p17142176.html
Sent from the Solr - User mailing list archive at Nabble.com.
