large scale Solr Deployment = how to do (Expert
Question)?
Hello Ephraim, hello Lance, hello Walter,
thanks for your replies:
Ephraim, thanks very much for the further detailed explanation. I will
try
to setup a demo system in the next few days and use your advice.
LoadBalancers
You might also want to look at the heritrix crawler too:
http://crawler.archive.org/
I have written three crawlers in the past, all for RSS feeds, it is not easy.
Happy to provide tips and help if you want to go down that route.
François
On Apr 8, 2011, at 1:53 AM, Andrea Campi wrote:
I can't view the document either -- it showed up empty.
Has anyone succeeded in viewing it?
Andy
--- On Fri, 4/8/11, Albert Vila a...@imente.com wrote:
From: Albert Vila a...@imente.com
Subject: Re: Very very large scale Solr Deployment = how to do (Expert
Question)?
To: solr-user
, Albert Vila a...@imente.com wrote:
From: Albert Vila a...@imente.com
Subject: Re: Very very large scale Solr Deployment = how to do (Expert
Question)?
To: solr-user@lucene.apache.org
Date: Friday, April 8, 2011, 3:43 AM
Ephraim, I still can't view the
document.
Don't know if I'm doing
the document either -- it showed up
empty.
Has anyone succeeded in viewing it?
Andy
--- On Fri, 4/8/11, Albert Vila a...@imente.com
wrote:
From: Albert Vila a...@imente.com
Subject: Re: Very very large scale Solr Deployment
= how to do (Expert Question)?
To: solr-user@lucene.apache.org
of the document in pdf or openoffice
format? I'm on Linux so there's no way for me to use MS Word.
Thanks.
--- On Fri, 4/8/11, Albert Vila a...@imente.com wrote:
From: Albert Vila a...@imente.com
Subject: Re: Very very large scale Solr Deployment = how to do (Expert
Question)?
To: solr-user
Perfect. Thank you very much.
Andy
--- On Fri, 4/8/11, Pascal Coupet pcou...@gmail.com wrote:
From: Pascal Coupet pcou...@gmail.com
Subject: Re: Very very large scale Solr Deployment = how to do (Expert
Question)?
To: solr-user@lucene.apache.org
Date: Friday, April 8, 2011, 10:20 AM
I
: Thursday, April 07, 2011 8:30 AM
To: solr-user@lucene.apache.org
Subject: Re: Very very large scale Solr Deployment = how to do (Expert
Question)?
Hello Ephraim, hello Lance, hello Walter,
thanks for your replies:
Ephraim, thanks very much for the further detailed explanation. I will
try
to setup
On Apr 6, 2011, at 10:29 PM, Jens Mueller wrote:
Walter, thanks for the advice: Well you are right, mentioning google. My
question was also to understand how such large systems like google/facebook
are actually working. So my numbers are just theoretical and made up. My
system will be
Hello all,
thanks for your generous help.
I think I now know everything: (What I want to do is to build a web crawler
and index the documents found). I will start with the setup as suggested by
Ephraim (Several sharded masters, each with at least one slave for reads and
some aggregators for
On Fri, Apr 8, 2011 at 6:23 AM, Jens Mueller supidupi...@googlemail.comwrote:
Hello all,
thanks for your generous help.
I think I now know everything: (What I want to do is to build a web
crawler
and index the documents found). I will start with the setup as suggested by
Write a web
]
Sent: Wednesday, April 06, 2011 8:58 AM
To: solr-user@lucene.apache.org
Cc: Isan Fulia; Tirthankar Chatterjee
Subject: Re: FW: Very very large scale Solr Deployment = how to do
(Expert Question)?
I third that request.
Would greatly appreciate taking a look at that diagram!
Regards,
Jonathan
I would not use replication. LinkedIn consumer search is a flat system
where one process indexes new entries and does queries simultaneously.
It's a custom Lucene app called Zoie. Their stuff is on Github..
I would get documents to indexers via a multicast IP-based queueing
system. This scales
The bigger answer is that you cannot get to this size by just configuring Solr.
You may have to invent a lot of stuff. Like all of Google.
Where did you get these numbers? The proposed query rate is twice as big as
Google (Feb 2010 estimate, 34K qps).
I work at MarkLogic, and we scale to 100's
Hello Ephraim, hello Lance, hello Walter,
thanks for your replies:
Ephraim, thanks very much for the further detailed explanation. I will try
to setup a demo system in the next few days and use your advice.
LoadBalancers are an important aspect of your design. Can you recommend one
LB
- Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
- Original Message
From: Jens Mueller supidupi...@googlemail.com
To: solr-user@lucene.apache.org
Sent: Thu, April 7, 2011 1:29:40 AM
Subject: Re: Very very large scale Solr Deployment = how to do (Expert
Question
might want to look into
SolrCloud as well.
Ephraim Ofir
-Original Message-
From: Jens Mueller [mailto:supidupi...@googlemail.com]
Sent: Tuesday, April 05, 2011 4:25 AM
To: solr-user@lucene.apache.org
Subject: Very very large scale Solr Deployment = how to do (Expert
Question)?
Hello
Message-
From: Jens Mueller [mailto:supidupi...@googlemail.com]
Sent: Tuesday, April 05, 2011 4:25 AM
To: solr-user@lucene.apache.org
Subject: Very very large scale Solr Deployment = how to do (Expert
Question)?
Hello Experts,
I am a Solr newbie but read quite a lot of docs. I still
, so here it is if you
want it...
Ephraim Ofir
-Original Message-
From: Ephraim Ofir
Sent: Tuesday, April 05, 2011 10:20 AM
To: 'solr-user@lucene.apache.org'
Subject: RE: Very very large scale Solr Deployment = how to do (Expert
Question)?
I'm not sure about the scale you're
...@icq.com
of course the attachment didn't get to the list, so here it is if you
want it...
Ephraim Ofir
-Original Message-
From: Ephraim Ofir
Sent: Tuesday, April 05, 2011 10:20 AM
To: 'solr-user@lucene.apache.org'
Subject: RE: Very very large scale Solr Deployment = how to do (Expert
-Original Message-
From: Jens Mueller [mailto:supidupi...@googlemail.com]
Sent: Tuesday, April 05, 2011 10:30 PM
To: solr-user@lucene.apache.org
Subject: Re: FW: Very very large scale Solr Deployment = how to do (Expert
Question)?
Hello Ephraim,
thank you so much for the great Document/Scaling
PM
To: solr-user@lucene.apache.org
Subject: Re: FW: Very very large scale Solr Deployment = how to do
(Expert
Question)?
Hello Ephraim,
thank you so much for the great Document/Scaling-Concept!!
First I think you really should publish this on the solr wiki. This
approach
Hello Experts,
I am a Solr newbie but read quite a lot of docs. I still do not understand
what would be the best way to setup very large scale deployments:
Goal (threoretical):
A.) Index-Size: 1 Petabyte (1 Document is about 5 KB in Size)
B) Queries: 10 Queries/ per Second
C)
23 matches
Mail list logo
