Thanks, I am no Solr expert so I may be over-simplifying things a bit in my
ignorance.
"No. The replicas are in a "down" state the Solr instances are brought back
up" Why can't I dictate (at least initially) the "up" state somehow? It
seems Solr keeps track of where replicas were deployed so that the cluster
'heals' itself when all nodes are back. At deployment, I know which nodes
should be available so the collection could be unavailable until all
expected nodes are up.
Thank you for the pointer to the createNodeSet parameter, that might prove
useful.
"I think the thing I'm getting stuck on is how in the world the
Solr code could know enough to "do the right thing". How many
docs do you have? How big are they? How much to you expect
to grow? What kinds of searches do you want to support?"
Solr can't know these things. But me as the deployer/developer might.
For example say I know my initial data size and can say the index will be
10 TB. If I have 2 nodes with 5 TB disks well then I have to have 2 shards
because it won't fit on one node. If instead I have 4 nodes with 5 TB
disks, well I could still have 2 shards but with replicas. Or I could
choose no replicas but more shards. This is what I mean by the
shard/replica decision being partially dependent on available hardware;
there are some decisions I could make knowing my planned deployment so that
when I start the cluster it can be immediately functional. Rather than
first starting the cluster, then creating the collection, then making it
available.
You may be right that it is a small and complicated concern because I
really only need to care about it once when I am first deploying my
cluster. But everyone who needs to stand up a SolrCloud cluster needs to do
it. My guess is most people either do it manually as a one-time operations
thing or they write a custom script to do it for them automatically as I am
attempting. Seems like a good candidate for a new feature.
- Frank
On Wed, Jan 9, 2019 at 4:18 PM Erick Erickson <erickerick...@gmail.com>
wrote:
> bq. do all 100 replicas move to the one remaining node?
>
> No. The replicas are in a "down" state the Solr instances
> are brought back up (I'm skipping autoscaling here, but
> even that wouldn't move all the replicas to the one remaining
> node).
>
> bq. what the collection *should* look like based on the
> hardware I am deploying to.
>
> With the caveat that the Solr instances have to be up, this
> is entirely possible. First of all, you can provide a "createNodeSet"
> to the create command to specify exactly what Solr nodes you
> want used for your collection. There's a special "EMPTY"
> value that _almost_ does what you want, that is it creates
> no replicas, just the configuration in ZooKeeper. Thereafter,
> though, you have to ADDREPLICA (which you can do with
> "node" parameter to place it exactly where you want.
>
> bq. how many shards are at least partially dependent on the
> available hardware
>
> Not if you're using compositeID routing. The number of shards
> is fixed at creation time, although you can split them later.
>
> I don't think you can use bin/solr create_collection with the
> EMPTY createNodeSet, so you need at least one
> Solr node running to create your skeleton collection.
>
> I think the thing I'm getting stuck on is how in the world the
> Solr code could know enough to "do the right thing". How many
> docs do you have? How big are they? How much to you expect
> to grow? What kinds of searches do you want to support?
>
> But more power to you if you can figure out how to support the kind
> of thing you want. Personally I think it's harder than you might
> think and not broadly useful. I've been wrong more times than I like
> to recall, so maybe you have an approach that would get around
> the tigers hiding in the grass I think are out there...
>
> Best,
> Erick
>
>
> On Wed, Jan 9, 2019 at 3:04 PM Frank Greguska <fg...@apache.org> wrote:
> >
> > Thanks for the response. You do raise good points.
> >
> > Say I reverse your example and I have a 10 node cluster with a 10-shard
> > collection and a replication factor of 10. Now I kill 9 of my nodes, do
> all
> > 100 replicas move to the one remaining node? I believe the answer is,
> well
> > that depends on the configuration.
> >
> > I'm thinking about it from the initial cluster planning side of things.
> The
> > decisions about auto-scaling, how many replicas, and even how many shards
> > are at least partially dependent on the available hardware. So at
> > deployment time I would think there would be a way of defining what the
> > collection *should* look like based on the hardware I am deploying to.
> > Obviously this could change during runtime and I may need to add nodes,
> > split shards, etc...
> >
> > As it is now it seems like I need to deploy my cluster then write a
> custom
> > script to ensure each node I expect to be there is running and only then
> > create my collection with desired shards and replication.
> >
> > - Frank
> >
> > On Wed, Jan 9, 2019 at 2:14 PM Erick Erickson <erickerick...@gmail.com>
> > wrote:
> >
> > > How would you envision that working? When would the
> > > replicas actually be created and under what heuristics?
> > >
> > > Imagine this is possible, and there are a bunch of
> > > placeholders in ZK for a 10-shard, collection with
> > > a replication factor of 10 (100 replicas all told). Now
> > > I bring up a single Solr instance. Should all 100 replicas
> > > be created immediately? Wait for N Solr nodes to be
> > > brought online? On some command?
> > >
> > > My gut feel is that this would be fraught with problems
> > > and not very valuable to many people. If you could create
> > > the "template" in ZK without any replicas actually being created,
> > > then at some other point say "make it so", I don't see the advantage
> > > over just the current setup. And I do think that it would be
> > > considerable effort.
> > >
> > > Net-net is I'd like to see a much stronger justification
> > > before anyone embarks on something like this. First as
> > > I mentioned above I think it'd be a lot of effort, second I
> > > virtually guarantee it'd introduce significant bugs. How
> > > would it interact with autoscaling for instance?
> > >
> > > Best,
> > > Erick
> > >
> > > On Wed, Jan 9, 2019 at 9:59 AM Frank Greguska <fg...@apache.org>
> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I am trying to bootstrap a SolrCloud installation and I ran into an
> issue
> > > > that seems rather odd. I see it is possible to bootstrap a
> configuration
> > > > set from an existing SOLR_HOME using
> > > >
> > > > ./server/scripts/cloud-scripts/zkcli.sh -zkhost ${ZK_HOST} -cmd
> bootstrap
> > > > -solrhome ${SOLR_HOME}
> > > >
> > > > but this does not create a collection, it just uploads a
> configuration
> > > set.
> > > >
> > > > Furthermore, I can not use
> > > >
> > > > bin/solr create
> > > >
> > > > to create a collection and link it to my bootstrapped configuration
> set
> > > > because it requires Solr to already be running.
> > > >
> > > > I'm hoping someone can shed some light on why this is the case? It
> seems
> > > > like a collection is just some znodes stored in zookeeper that
> contain
> > > > configuration settings and such. Why should I not be able to create
> those
> > > > nodes before Solr is running?
> > > >
> > > > I'd like to open a feature request for this if one does not already
> exist
> > > > and if I am not missing something obvious.
> > > >
> > > > Thank you,
> > > >
> > > > Frank Greguska
> > >
>
