...@gmail.commailto:erickerick...@gmail.commailto:
erickerick...@gmail.commailto:erickerick...@gmail.com]
Date d'envoi : lundi 7 octobre 2013 14:33
À :
solr-user@lucene.apache.orgmailto:solr-user@lucene.apache.orgmailto:solr-user@lucene.apache.org
Objet : Re: feedback on Solr 4.x LotsOfCores feature
@lucene.apache.org
Objet : Re: feedback on Solr 4.x LotsOfCores feature
Thanks for the great writeup! It's always interesting to see how
a feature plays out in the real world. A couple of questions
though:
bq: We added 2 Cores options :
Do you mean you patched Solr? If so are you willing
[erickerick...@gmail.commailto:erickerick...@gmail.com]
Date d'envoi : lundi 7 octobre 2013 14:33
À : solr-user@lucene.apache.orgmailto:solr-user@lucene.apache.org
Objet : Re: feedback on Solr 4.x LotsOfCores feature
Thanks for the great writeup! It's always interesting to see how
a feature
@lucene.apache.org
Objet : Re: feedback on Solr 4.x LotsOfCores feature
Thanks for the great writeup! It's always interesting to see how
a feature plays out in the real world. A couple of questions
though:
bq: We added 2 Cores options :
Do you mean you patched Solr? If so are you willing to shard
@lucene.apache.org
Objet : Re: feedback on Solr 4.x LotsOfCores feature
Thanks for the great writeup! It's always interesting to see how
a feature plays out in the real world. A couple of questions
though:
bq: We added 2 Cores options :
Do you mean you patched Solr? If so are you willing to shard
decided to use the LotsOfCores feature.
Our goal was to change the current I/O usage : from lots of random I/O
access on huge segments to mostly sequential I/O access on small segments.
For our use case, it's not a big deal, that the first query to one not
yet loaded core will be slow.
And, we don’t
possible.
There is also an overhead on queries : too many results are filtered to
find only results concerning user.
For these reason and others, like not pooled users, hardware savings,
better scoring, some requests that do not support filtering, we have
decided to use the LotsOfCores feature
an overhead on queries : too many results are filtered to
find only results concerning user.
For these reason and others, like not pooled users, hardware savings,
better scoring, some requests that do not support filtering, we have
decided to use the LotsOfCores feature.
Our goal
mailboxes don’t have side effects to each other).
One important thing with the LotsOfCores feature is to take care of :
- the number of file descriptors, it used a lot (need to increase global
max and per process fd)
- the value of the transientCacheSize depending of the RAM size and the
PermGen
Aleksey,
It was a less than ideal situation. because we did not have a choice. We
had external systems/scripts to manage this. A new custom implementation is
being built on SolrCloud which would have taken care of most of hose
issues.
SolrReplication is a hidden once you move to cloud. But it
Thanks Paul. Just a little clarification:
You mention that you migrate data using built-in replication, but if
you map and route users yourself, doesn't that mean that you also need
to manage replication yourself? Your routing logic needs to be aware
of how to map both replicas for each user, and
On Fri, Jun 7, 2013, at 02:59 PM, Jack Krupansky wrote:
AFAICT, SolrCloud addresses the use case of distributed update for a
relatively smaller number of collections (dozens?) that have a relatively
larger number of rows - billions over a modest to moderate number of
nodes
(a handful to
: Re: LotsOfCores feature
On Fri, Jun 7, 2013, at 02:59 PM, Jack Krupansky wrote:
AFAICT, SolrCloud addresses the use case of distributed update for a
relatively smaller number of collections (dozens?) that have a relatively
larger number of rows - billions over a modest to moderate number of
nodes
A use case would a web site or service that had millions of users, each of
whom would have an active Solr core when they are active, but inactive
otherwise. Of course those cores would not all reside on one node and
ZooKeeper is out of the question for managing anything that is in the
The Wiki page was built not for Cloud Solr.
We have done such a deployment where less than a tenth of cores were active
at any given point in time. though there were tens of million indices they
were split among a large no:of hosts.
If you don't insist of Cloud deployment it is possible. I'm
I should have been clearer, and others have mentioned... the lots of cores
stuff is really outside Zookeeper/SolrCloud at present. I don't think it's
incompatible, but it wasn't part of the design so it'll need some effort to
make it play nice with SolrCloud. I'm not sure there's actually a
-user@lucene.apache.org
Subject: Re: LotsOfCores feature
The Wiki page was built not for Cloud Solr.
We have done such a deployment where less than a tenth of cores were active
at any given point in time. though there were tens of million indices they
were split among a large no:of hosts.
If you
Aleksey: What would you say is the average core size for your use case -
thousands or millions of rows? And how sharded would each of your
collections be, if at all?
Average core/collection size wouldn't even be thousands, hundreds more
like. And the largest would be half a million or so but
for short
periods of time.
-- Jack Krupansky
-Original Message-
From: Aleksey
Sent: Friday, June 07, 2013 3:44 PM
To: solr-user
Subject: Re: LotsOfCores feature
Aleksey: What would you say is the average core size for your use case -
thousands or millions of rows? And how sharded
We set it up like this
+ individual solr instances are setup
+ external mapping/routing to allocate users to instances. This information
can be stored in an external data store
+ all cores are created as transient and loadonstart as false
+ cores come online on demand
+ as and when users data get
I was looking at this wiki and linked issues:
http://wiki.apache.org/solr/LotsOfCores
they talk about a limit being 100K cores. Is that per server or per
entire fleet because zookeeper needs to manage that?
I was considering a use case where I have tens of millions of indices
but less that a
100K is really not the limit, it's just hard to imagine
100K cores on a single machine unless some were
really rarely used. And it's per node, not cluster-wide.
The current state is that everything is in place, including
transient cores, auto-discovery, etc. So you should be
able to go ahead and
: Erick Erickson
Sent: Thursday, June 06, 2013 3:53 PM
To: solr-user@lucene.apache.org
Subject: Re: LotsOfCores feature
100K is really not the limit, it's just hard to imagine
100K cores on a single machine unless some were
really rarely used. And it's per node, not cluster-wide
.
-- Jack Krupansky
-Original Message- From: Erick Erickson
Sent: Thursday, June 06, 2013 3:53 PM
To: solr-user@lucene.apache.org
Subject: Re: LotsOfCores feature
100K is really not the limit, it's just hard to imagine
100K cores on a single machine unless some were
really rarely
itself might be a traditional SolrCloud cluster,
except that it needs to be multi-data center.
-- Jack Krupansky
-Original Message-
From: Aleksey
Sent: Thursday, June 06, 2013 8:06 PM
To: solr-user
Subject: Re: LotsOfCores feature
I would not try putting tens of millions of cores
On 6/6/2013 6:32 PM, Jack Krupansky wrote:
big snip
This would be a lot more of a true Solr Cloud than the cluster
support that we have today.
And the CloudKeeper itself might be a traditional SolrCloud cluster,
except that it needs to be multi-data center.
I like a lot of what you said
-lotsofcores-feature-in
-
production-tp3193798p3193798.html Sent from the Solr - User mailing
list archive at Nabble.com.
--
If you reply to this email, your message will be added to the discussion
below:
http://lucene.472066.n3.nabble.com/Is-anobdy-using
will be decided.
Thanks Regards,
Umesh
--
View this message in context:
http://lucene.472066.n3.nabble.com/Is-anobdy-using-lotsofcores-feature-in-production-tp3193798p3236958.html
Sent from the Solr - User mailing list archive at Nabble.com.
scalable. I have around 1000 core and want to use this feature. Will
there
be any issue in production?
http://wiki.apache.org/solr/LotsOfCores
Thanks,
Umesh
--
View this message in context:
http://lucene.472066.n3.nabble.com/Is-anobdy-using-lotsofcores-feature-in-
production
this message in context:
http://lucene.472066.n3.nabble.com/Is-anobdy-using-lotsofcores-feature-in
-
production-tp3193798p3193798.html Sent from the Solr - User mailing
list archive at Nabble.com.
.472066.n3.nabble.com/Is-anobdy-using-lotsofcores-feature-in-
production-tp3193798p3193798.html Sent from the Solr - User mailing list
archive at Nabble.com.
.nabble.com/Is-anobdy-using-lotsofcores-feature-in-production-tp3193798p3193798.html
Sent from the Solr - User mailing list archive at Nabble.com.
32 matches
Mail list logo
