Re: Could not find collection when upgrading to Solr 7

[Marvin Bredal Lillehaug]

Ah, didn't get that legacyCloud was related to bootstrap
Have a new script working without bootstrapping.
Thank you for your input!

On Mon, Feb 26, 2018 at 7:36 AM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> Bootstrapping is not broken but the default in Solr 7 is to set the cluster
> property legacyCloud=false which means that bootstrapped cores cannot
> register new collections automatically anymore. If you want to rely on the
> old behavior then you need to set the cluster property legacyCloud=true.
> This was also mentioned in the upgrade section for Solr 7 in CHANGES.txt
>
> On Mon, Feb 26, 2018 at 1:28 AM, Marvin Bredal Lillehaug <
> marvin.lilleh...@gmail.com> wrote:
>
> > Thank your for your reply.
> > I will take a look at both the code for bootstrapping, and look at what
> you
> > are suggesting to determine what to do.
> >
> > On Sun, Feb 25, 2018 at 5:17 PM, Shawn Heisey 
> > wrote:
> >
> > > On 2/24/2018 8:05 AM, Marvin Bredal Lillehaug wrote:
> > >
> > >> This is for doing local development and running tests.
> > >> So starting with embedded zookeeper in SolrCloud mode is intentional.
> > The
> > >> idea is to just run the script setting up the folder structure for the
> > >> cores and everything is up and running, ready for indexing and
> querying
> > >> locally.
> > >> The applications using Solr also use Zookeeper for configuration and
> > >> coordination of their own stuff.
> > >>
> > >> There are no persmission issues.
> > >>
> > >> I have put the scripts here https://github.com/
> computerlove/start-solr
> > >>
> > >> The file config specifies what version to use. When running
> > >> run-dev-solr.sh
> > >> that version is downloaded, solrhome is copied to the right place and
> > the
> > >> instance is started.
> > >>
> > >
> > > I grabbed the git repo and proceeded to fiddle with it.
> > >
> > > As you said, it works when the solr version that is downloaded is
> 6.6.2,
> > > but fails with 7.2.1.  Based on what gets logged and what I know about
> > how
> > > Solr does initialization, I'm betting the bootstrap feature is
> broken.  I
> > > can't tell what's wrong, but I think that's probably where the problem
> > is.
> > >
> > > But as I already said, I strongly recommend that you don't use that
> > > feature.  At all.
> > >
> > > The cores that are in the git repo are empty -- there's no index.  If
> the
> > > source cores are empty when you run the script "for real", then I have
> an
> > > alternate idea for you to try:
> > >
> > > Instead of copying the cores and bootstrapping a non-cloud install
> into a
> > > cloud install:  Remove the bootstrap option. Create the collections
> using
> > > "bin/solr create", with -replicationFactor 1 -shards 1, and the -d
> option
> > > pointing at the config you want uploaded for the collection.  You could
> > > even go with a higher shard count if you want.  Increasing
> > > replicationFactor is probably possible, but as all this would be
> running
> > on
> > > a single server, there's no point.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
> >
> > --
> > med vennlig hilsen,
> > Marvin B. Lillehaug
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



-- 
med vennlig hilsen,
Marvin B. Lillehaug

Re: Could not find collection when upgrading to Solr 7

[Shalin Shekhar Mangar]

Bootstrapping is not broken but the default in Solr 7 is to set the cluster
property legacyCloud=false which means that bootstrapped cores cannot
register new collections automatically anymore. If you want to rely on the
old behavior then you need to set the cluster property legacyCloud=true.
This was also mentioned in the upgrade section for Solr 7 in CHANGES.txt

On Mon, Feb 26, 2018 at 1:28 AM, Marvin Bredal Lillehaug <
marvin.lilleh...@gmail.com> wrote:

> Thank your for your reply.
> I will take a look at both the code for bootstrapping, and look at what you
> are suggesting to determine what to do.
>
> On Sun, Feb 25, 2018 at 5:17 PM, Shawn Heisey 
> wrote:
>
> > On 2/24/2018 8:05 AM, Marvin Bredal Lillehaug wrote:
> >
> >> This is for doing local development and running tests.
> >> So starting with embedded zookeeper in SolrCloud mode is intentional.
> The
> >> idea is to just run the script setting up the folder structure for the
> >> cores and everything is up and running, ready for indexing and querying
> >> locally.
> >> The applications using Solr also use Zookeeper for configuration and
> >> coordination of their own stuff.
> >>
> >> There are no persmission issues.
> >>
> >> I have put the scripts here https://github.com/computerlove/start-solr
> >>
> >> The file config specifies what version to use. When running
> >> run-dev-solr.sh
> >> that version is downloaded, solrhome is copied to the right place and
> the
> >> instance is started.
> >>
> >
> > I grabbed the git repo and proceeded to fiddle with it.
> >
> > As you said, it works when the solr version that is downloaded is 6.6.2,
> > but fails with 7.2.1.  Based on what gets logged and what I know about
> how
> > Solr does initialization, I'm betting the bootstrap feature is broken.  I
> > can't tell what's wrong, but I think that's probably where the problem
> is.
> >
> > But as I already said, I strongly recommend that you don't use that
> > feature.  At all.
> >
> > The cores that are in the git repo are empty -- there's no index.  If the
> > source cores are empty when you run the script "for real", then I have an
> > alternate idea for you to try:
> >
> > Instead of copying the cores and bootstrapping a non-cloud install into a
> > cloud install:  Remove the bootstrap option. Create the collections using
> > "bin/solr create", with -replicationFactor 1 -shards 1, and the -d option
> > pointing at the config you want uploaded for the collection.  You could
> > even go with a higher shard count if you want.  Increasing
> > replicationFactor is probably possible, but as all this would be running
> on
> > a single server, there's no point.
> >
> > Thanks,
> > Shawn
> >
> >
>
>
> --
> med vennlig hilsen,
> Marvin B. Lillehaug
>



-- 
Regards,
Shalin Shekhar Mangar.

Re: Could not find collection when upgrading to Solr 7

[Marvin Bredal Lillehaug]

Thank your for your reply.
I will take a look at both the code for bootstrapping, and look at what you
are suggesting to determine what to do.

On Sun, Feb 25, 2018 at 5:17 PM, Shawn Heisey  wrote:

> On 2/24/2018 8:05 AM, Marvin Bredal Lillehaug wrote:
>
>> This is for doing local development and running tests.
>> So starting with embedded zookeeper in SolrCloud mode is intentional. The
>> idea is to just run the script setting up the folder structure for the
>> cores and everything is up and running, ready for indexing and querying
>> locally.
>> The applications using Solr also use Zookeeper for configuration and
>> coordination of their own stuff.
>>
>> There are no persmission issues.
>>
>> I have put the scripts here https://github.com/computerlove/start-solr
>>
>> The file config specifies what version to use. When running
>> run-dev-solr.sh
>> that version is downloaded, solrhome is copied to the right place and the
>> instance is started.
>>
>
> I grabbed the git repo and proceeded to fiddle with it.
>
> As you said, it works when the solr version that is downloaded is 6.6.2,
> but fails with 7.2.1.  Based on what gets logged and what I know about how
> Solr does initialization, I'm betting the bootstrap feature is broken.  I
> can't tell what's wrong, but I think that's probably where the problem is.
>
> But as I already said, I strongly recommend that you don't use that
> feature.  At all.
>
> The cores that are in the git repo are empty -- there's no index.  If the
> source cores are empty when you run the script "for real", then I have an
> alternate idea for you to try:
>
> Instead of copying the cores and bootstrapping a non-cloud install into a
> cloud install:  Remove the bootstrap option. Create the collections using
> "bin/solr create", with -replicationFactor 1 -shards 1, and the -d option
> pointing at the config you want uploaded for the collection.  You could
> even go with a higher shard count if you want.  Increasing
> replicationFactor is probably possible, but as all this would be running on
> a single server, there's no point.
>
> Thanks,
> Shawn
>
>


-- 
med vennlig hilsen,
Marvin B. Lillehaug

Re: Could not find collection when upgrading to Solr 7

[Shawn Heisey]

On 2/24/2018 8:05 AM, Marvin Bredal Lillehaug wrote:

This is for doing local development and running tests.
So starting with embedded zookeeper in SolrCloud mode is intentional. The
idea is to just run the script setting up the folder structure for the
cores and everything is up and running, ready for indexing and querying
locally.
The applications using Solr also use Zookeeper for configuration and
coordination of their own stuff.

There are no persmission issues.

I have put the scripts here https://github.com/computerlove/start-solr

The file config specifies what version to use. When running run-dev-solr.sh
that version is downloaded, solrhome is copied to the right place and the
instance is started.


I grabbed the git repo and proceeded to fiddle with it.

As you said, it works when the solr version that is downloaded is 6.6.2, 
but fails with 7.2.1.  Based on what gets logged and what I know about 
how Solr does initialization, I'm betting the bootstrap feature is 
broken.  I can't tell what's wrong, but I think that's probably where 
the problem is.


But as I already said, I strongly recommend that you don't use that 
feature.  At all.


The cores that are in the git repo are empty -- there's no index.  If 
the source cores are empty when you run the script "for real", then I 
have an alternate idea for you to try:


Instead of copying the cores and bootstrapping a non-cloud install into 
a cloud install:  Remove the bootstrap option. Create the collections 
using "bin/solr create", with -replicationFactor 1 -shards 1, and the -d 
option pointing at the config you want uploaded for the collection.  You 
could even go with a higher shard count if you want.  Increasing 
replicationFactor is probably possible, but as all this would be running 
on a single server, there's no point.


Thanks,
Shawn

Re: Could not find collection when upgrading to Solr 7

[Marvin Bredal Lillehaug]

This is for doing local development and running tests.
So starting with embedded zookeeper in SolrCloud mode is intentional. The
idea is to just run the script setting up the folder structure for the
cores and everything is up and running, ready for indexing and querying
locally.
The applications using Solr also use Zookeeper for configuration and
coordination of their own stuff.

There are no persmission issues.

I have put the scripts here https://github.com/computerlove/start-solr

The file config specifies what version to use. When running run-dev-solr.sh
that version is downloaded, solrhome is copied to the right place and the
instance is started.


On Sat, Feb 24, 2018 at 1:20 PM, Shawn Heisey  wrote:

> On 2/24/2018 3:13 AM, Marvin Bredal Lillehaug wrote:
>
>> We have a multicore Solr-instance that currently is running version 6.6.2.
>> For local development we run
>>
>> ./solr start -v -d $solrDir/server -h localhost -p $SOLR_PORT -c -f -s
>> $solrHomeDir -a "-Dbootstrap_conf=true -Djetty.host=localhost" -m 4096m
>>
>>
> Are you aware that the -c option, without the -z option, will start an
> embedded zookeeper server and put Solr into SolrCloud mode?  The embedded
> ZK server is not recommended except for setting up a proof of concept.  For
> production, the ZK ensemble needs at least three servers.
>
> Setting jetty.host to localhost will make it so that Solr cannot be
> accessed remotely, it will only work on the local machine.  This defeats
> one of the primary purposes of SolrCloud -- a highly available fault
> tolerant cluster.
>
> The bootstrap_conf setting is intended to be used precisely once -- in a
> situation where you want to convert a non-cloud install to SolrCloud.  It
> is not meant to be defined every time you start Solr.  I would strongly
> recommended that you do not use bootstrap at all.  Instead, start up a new
> install, create your collection how you desire, and then run your indexing
> so you create the index fresh.
>
> On Solr 6 everything starts fine, and cores and collections are created on
>> startup.
>> When running the same setup on Solr 7.x.x it crash with Could not find
>> collection : nvdb-transactionlog when running 
>> CoreContainer.repairCoreProperty.
>> Logs for Solr 6 and 7 attached.
>>
>
> That error probably means that the collection was not found in the
> ZooKeeper database.  Are you doing any kind of configuration or changes on
> the embedded ZK server before you start Solr?  Does the user that's running
> Solr have enough permissions to create and write to the ZK data directory?
> The problem *might* be the bootstrap_conf setting, but I am not sure
> whether that is correct.  Unless something is very broken in your install,
> then I would suspect the bootstrap option is causing a problem.
>
> Thanks,
> Shawn
>
>


-- 
med vennlig hilsen,
Marvin B. Lillehaug

Re: Could not find collection when upgrading to Solr 7

[Shawn Heisey]

On 2/24/2018 3:13 AM, Marvin Bredal Lillehaug wrote:
We have a multicore Solr-instance that currently is running version 
6.6.2.

For local development we run

./solr start -v -d $solrDir/server -h localhost -p $SOLR_PORT -c -f -s 
$solrHomeDir -a "-Dbootstrap_conf=true -Djetty.host=localhost" -m 4096m




Are you aware that the -c option, without the -z option, will start an 
embedded zookeeper server and put Solr into SolrCloud mode?  The 
embedded ZK server is not recommended except for setting up a proof of 
concept.  For production, the ZK ensemble needs at least three servers.


Setting jetty.host to localhost will make it so that Solr cannot be 
accessed remotely, it will only work on the local machine.  This defeats 
one of the primary purposes of SolrCloud -- a highly available fault 
tolerant cluster.


The bootstrap_conf setting is intended to be used precisely once -- in a 
situation where you want to convert a non-cloud install to SolrCloud.  
It is not meant to be defined every time you start Solr.  I would 
strongly recommended that you do not use bootstrap at all.  Instead, 
start up a new install, create your collection how you desire, and then 
run your indexing so you create the index fresh.


On Solr 6 everything starts fine, and cores and collections are 
created on startup.
When running the same setup on Solr 7.x.x it crash with Could not find 
collection : nvdb-transactionlog when 
running CoreContainer.repairCoreProperty. Logs for Solr 6 and 7 attached.


That error probably means that the collection was not found in the 
ZooKeeper database.  Are you doing any kind of configuration or changes 
on the embedded ZK server before you start Solr?  Does the user that's 
running Solr have enough permissions to create and write to the ZK data 
directory?  The problem *might* be the bootstrap_conf setting, but I am 
not sure whether that is correct.  Unless something is very broken in 
your install, then I would suspect the bootstrap option is causing a 
problem.


Thanks,
Shawn