Re: How to get a solr core to persist

[Shawn Heisey]

On 11/20/2017 6:26 AM, Amanda Shuman wrote:
> I did as you suggested and created the core by hand - I copied the files
> from the existing core, including the index files (data directory) and
> changed the core.properties file to the new core name (core_new) and
> restarted. Now I'm having a different issue - it says it is Optimized but
> that Current is not (the console shows the red prohibited sign, which I
> guess means false or something?). So basically there's no content at all in
> there. Re-reading your instructions here: " If you want to relocate the
> data, you can add a dataDir property to core.properties.  If it has a
> relative path, it is relative to the core.properties location." - Did I
> miss a step to get the existing index to load?

If data/index is in your core's directory and actually contains a
complete index, then Solr will load that index for that core on startup.

If the way you handle commits is insufficient, then there is a
possibility that the most recent updates to the source index are sitting
in memory and haven't been written to the on-disk index, but it seems
unlikely that this would result in a completely empty index.

Thanks,
Shawn

Re: How to get a solr core to persist

[Amanda Shuman]

Hi Shawn,

I did as you suggested and created the core by hand - I copied the files
from the existing core, including the index files (data directory) and
changed the core.properties file to the new core name (core_new) and
restarted. Now I'm having a different issue - it says it is Optimized but
that Current is not (the console shows the red prohibited sign, which I
guess means false or something?). So basically there's no content at all in
there. Re-reading your instructions here: " If you want to relocate the
data, you can add a dataDir property to core.properties.  If it has a
relative path, it is relative to the core.properties location." - Did I
miss a step to get the existing index to load?

Thanks!
Amanda

--
Dr. Amanda Shuman
Post-doc researcher, University of Freiburg, The Maoist Legacy Project

PhD, University of California, Santa Cruz
http://www.amandashuman.net/
http://www.prchistoryresources.org/
Office: +49 (0) 761 203 4925


On Wed, Nov 15, 2017 at 1:32 PM, Shawn Heisey  wrote:

> On 11/15/2017 2:28 AM, Amanda Shuman wrote:
>
>> 1) so does this mean that on the back-end I should first create my new
>> core, e.g., core1 and then within that place a conf folder with all the
>> files? Same for the data folder? If so, is it fine to just use the
>> existing
>> config files that I've previously worked on (i.e. the config for search
>> that I already modified)? I presume this won't be an issue.
>>
>> 2) does it matter if I create this core through the admin console or at
>> command line?
>>
>
> You can create your cores however you like.  I actually create all my
> cores completely by hand, including the core.properties file, and let Solr
> discover them on startup.  Mostly I just copy an existing core, change
> core.properties to correct values, make any config changes I need, and
> restart Solr.
>
> If you want to use the admin UI (or the CoreAdmin API directly, which is
> what the admin UI calls), then the instanceDir must have a conf directory
> with all the config files you require for the core, and NOT have a
> core.properties file.  If you're adding a core that already has a an index,
> then you would also include the data directory in the core's instanceDir.
> If you want to relocate the data, you can add a dataDir property to
> core.properties.  If it has a relative path, it is relative to the
> core.properties location.
>
> The commandline creation works pretty well.  The way it works is by
> copying a configset (which may be in server/solr/configsets or in a custom
> location) to the "conf" directory in the core, then calling the CoreAdmin
> API to actually add the core to Solr (and create core.properties so it'll
> get picked up on restart).
>
> Thanks,
> Shawn
>

Re: How to get a solr core to persist

[Erick Erickson]

Well, if you must ;)

Basically whatever is easier. "Core discovery" starts at SOLR_HOME,
wherever you've defined that. Which can be done with a sysvar
"-Dwhatrever" at start time or the "-s" param if using the bin/solr
start scripts. That's the root for finding all "core.properties"
files. Each core.properties file defines a core.

_within_ a core.properties file you can specify a dataDir pointing
wherever you want which is where the index goes.

If using the core admin API to create a core, you can specify dataDir
as one of the params.

Really, about all the core admin CREATE API does is create the
core.properties file, so Shawn's process of setting it all up
_including_ a core.properties file and starting Solr works perfectly.

The only caution when specifying dataDir for a particular core is it
as A Very Bad Thing if two cores point to the _same_ dataDir. One of
them will refuse to load due to the index being locked.

A lot of this is at:
https://lucene.apache.org/solr/guide/6_6/defining-core-properties.html

Best,
Erick

On Wed, Nov 15, 2017 at 5:05 AM, Rick Leir  wrote:
> Hi Shawn, Amanda
> When we put the data under /var/lib, I feel a need to put the config under 
> /etc. Is this recommended, and would you use a symbolic link for the conf dir?
> Cheers--Rick
> --
> Sorry for being brief. Alternate email is rickleir at yahoo dot com

Re: How to get a solr core to persist

[Rick Leir]

Hi Shawn, Amanda
When we put the data under /var/lib, I feel a need to put the config under 
/etc. Is this recommended, and would you use a symbolic link for the conf dir?
Cheers--Rick
-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Re: How to get a solr core to persist

[Shawn Heisey]

On 11/15/2017 2:28 AM, Amanda Shuman wrote:

1) so does this mean that on the back-end I should first create my new
core, e.g., core1 and then within that place a conf folder with all the
files? Same for the data folder? If so, is it fine to just use the existing
config files that I've previously worked on (i.e. the config for search
that I already modified)? I presume this won't be an issue.

2) does it matter if I create this core through the admin console or at
command line?


You can create your cores however you like.  I actually create all my 
cores completely by hand, including the core.properties file, and let 
Solr discover them on startup.  Mostly I just copy an existing core, 
change core.properties to correct values, make any config changes I 
need, and restart Solr.


If you want to use the admin UI (or the CoreAdmin API directly, which is 
what the admin UI calls), then the instanceDir must have a conf 
directory with all the config files you require for the core, and NOT 
have a core.properties file.  If you're adding a core that already has a 
an index, then you would also include the data directory in the core's 
instanceDir.  If you want to relocate the data, you can add a dataDir 
property to core.properties.  If it has a relative path, it is relative 
to the core.properties location.


The commandline creation works pretty well.  The way it works is by 
copying a configset (which may be in server/solr/configsets or in a 
custom location) to the "conf" directory in the core, then calling the 
CoreAdmin API to actually add the core to Solr (and create 
core.properties so it'll get picked up on restart).


Thanks,
Shawn

Re: How to get a solr core to persist

[Amanda Shuman]

Ah, also, this is what the admin console says for location of core docs
when I created the core at command line:

CWD:/opt/solr-5.3.1/serverInstance:/var/solr/data/[corename]Data:
/var/solr/data/[corename]/dataIndex:/var/solr/data/[corename]/data/index


--
Dr. Amanda Shuman
Post-doc researcher, University of Freiburg, The Maoist Legacy Project

PhD, University of California, Santa Cruz
http://www.amandashuman.net/
http://www.prchistoryresources.org/
Office: +49 (0) 761 203 4925


On Wed, Nov 15, 2017 at 10:28 AM, Amanda Shuman 
wrote:

> Hi Shawn,
>
> Thanks a million for your response! I really do appreciate it... this
> helps clarify how things should be set up.
>
> As for why things are set up the way they are and the webapps location...
> honestly I think my predecessor did not really understand solr at all...
> I'm trying to pick up the pieces now on the back-end. (On the bright side,
> I did figure out how to modify the search relevance critieria in the config
> files for our core, but I'm more of a front-end developer and that seemed a
> lot more intuitive to me.)
>
> It does seem that the solr home is currently in /var/solr/data (not
> server/solr) because when I created a new core at command line, that's
> where it went. We start/restart solr using /etc/init.d/ rather than
> bin/solr.
>
> If can ask a few very small follow-up questions to this:
>
> "If you do a manual core creation, the core.properties file must NOT
> exist, but the conf directory *must* exist with the proper contents."
>
> 1) so does this mean that on the back-end I should first create my new
> core, e.g., core1 and then within that place a conf folder with all the
> files? Same for the data folder? If so, is it fine to just use the existing
> config files that I've previously worked on (i.e. the config for search
> that I already modified)? I presume this won't be an issue.
>
> 2) does it matter if I create this core through the admin console or at
> command line?
>
> Thanks again!
> Amanda
>
> --
> Dr. Amanda Shuman
> Post-doc researcher, University of Freiburg, The Maoist Legacy Project
> 
> PhD, University of California, Santa Cruz
> http://www.amandashuman.net/
> http://www.prchistoryresources.org/
> Office: +49 (0) 761 203 4925 <+49%20761%202034925>
>
>
> On Tue, Nov 14, 2017 at 3:15 PM, Shawn Heisey  wrote:
>
>> On 11/14/2017 2:14 AM, Amanda Shuman wrote:
>>
>>> We have just one solr core, which resides in the webapp foIder (in solr
>>> 5.3.1 this is at /opt/solr-5.3.1/server/webapps/[corename]/  -- the data
>>> folder is in the same place at /data).
>>>
>>
>> Why is your core there?  That is not a typical location, and does explain
>> the restart behavior you're seeing.
>>
>> Usually core directories go in the solr home.  If you start solr using
>> "bin/solr start" directly without any options, the solr home will be in
>> server/solr, not server/webapp.  How are you starting Solr?
>>
>> "Could not create a new core in
>>> /opt/solr-5.3.1/server/webapps/[corename]/as another core is already
>>> defined there"
>>>
>>
>> When Solr starts, it begins searching the coreRootDirectory (which
>> defaults to the solr home) for cores.  When it locates a core.properties
>> file, that location becomes the instanceDir for a core.
>>
>> If you do a manual core creation, the core.properties file must NOT
>> exist, but the conf directory *must* exist with the proper contents. The
>> core creation will create that file.  If it already exists, then Solr will
>> refuse to create the core, just as you have seen.
>>
>> The program directory location you have mentioned (/opt/solr-5.3.1)
>> sounds like somebody did a service installation.  The default solr home
>> when you install the service (and start Solr with /etc/init.d/ rather
>> than bin/solr) is /var/solr/data.  This location can be overridden, but
>> that's the default.
>>
>> Instead of having your core in webapp, move it to the solr home, wherever
>> that is.  Then when you start Solr, it will find the core.
>>
>> If a service installation has been done, then you should not start Solr
>> with "bin/solr" -- you should start the installed service.
>>
>> Thanks,
>> Shawn
>>
>
>

Re: How to get a solr core to persist

[Amanda Shuman]

Hi Shawn,

Thanks a million for your response! I really do appreciate it... this helps
clarify how things should be set up.

As for why things are set up the way they are and the webapps location...
honestly I think my predecessor did not really understand solr at all...
I'm trying to pick up the pieces now on the back-end. (On the bright side,
I did figure out how to modify the search relevance critieria in the config
files for our core, but I'm more of a front-end developer and that seemed a
lot more intuitive to me.)

It does seem that the solr home is currently in /var/solr/data (not
server/solr) because when I created a new core at command line, that's
where it went. We start/restart solr using /etc/init.d/ rather than
bin/solr.

If can ask a few very small follow-up questions to this:

"If you do a manual core creation, the core.properties file must NOT exist,
but the conf directory *must* exist with the proper contents."

1) so does this mean that on the back-end I should first create my new
core, e.g., core1 and then within that place a conf folder with all the
files? Same for the data folder? If so, is it fine to just use the existing
config files that I've previously worked on (i.e. the config for search
that I already modified)? I presume this won't be an issue.

2) does it matter if I create this core through the admin console or at
command line?

Thanks again!
Amanda

--
Dr. Amanda Shuman
Post-doc researcher, University of Freiburg, The Maoist Legacy Project

PhD, University of California, Santa Cruz
http://www.amandashuman.net/
http://www.prchistoryresources.org/
Office: +49 (0) 761 203 4925


On Tue, Nov 14, 2017 at 3:15 PM, Shawn Heisey  wrote:

> On 11/14/2017 2:14 AM, Amanda Shuman wrote:
>
>> We have just one solr core, which resides in the webapp foIder (in solr
>> 5.3.1 this is at /opt/solr-5.3.1/server/webapps/[corename]/  -- the data
>> folder is in the same place at /data).
>>
>
> Why is your core there?  That is not a typical location, and does explain
> the restart behavior you're seeing.
>
> Usually core directories go in the solr home.  If you start solr using
> "bin/solr start" directly without any options, the solr home will be in
> server/solr, not server/webapp.  How are you starting Solr?
>
> "Could not create a new core in
>> /opt/solr-5.3.1/server/webapps/[corename]/as another core is already
>> defined there"
>>
>
> When Solr starts, it begins searching the coreRootDirectory (which
> defaults to the solr home) for cores.  When it locates a core.properties
> file, that location becomes the instanceDir for a core.
>
> If you do a manual core creation, the core.properties file must NOT exist,
> but the conf directory *must* exist with the proper contents. The core
> creation will create that file.  If it already exists, then Solr will
> refuse to create the core, just as you have seen.
>
> The program directory location you have mentioned (/opt/solr-5.3.1) sounds
> like somebody did a service installation.  The default solr home when you
> install the service (and start Solr with /etc/init.d/ rather than
> bin/solr) is /var/solr/data.  This location can be overridden, but that's
> the default.
>
> Instead of having your core in webapp, move it to the solr home, wherever
> that is.  Then when you start Solr, it will find the core.
>
> If a service installation has been done, then you should not start Solr
> with "bin/solr" -- you should start the installed service.
>
> Thanks,
> Shawn
>

Re: How to get a solr core to persist

[Shawn Heisey]

On 11/14/2017 2:14 AM, Amanda Shuman wrote:

We have just one solr core, which resides in the webapp foIder (in solr
5.3.1 this is at /opt/solr-5.3.1/server/webapps/[corename]/  -- the data
folder is in the same place at /data).


Why is your core there?  That is not a typical location, and does 
explain the restart behavior you're seeing.


Usually core directories go in the solr home.  If you start solr using 
"bin/solr start" directly without any options, the solr home will be in 
server/solr, not server/webapp.  How are you starting Solr?



"Could not create a new core in
/opt/solr-5.3.1/server/webapps/[corename]/as another core is already
defined there"


When Solr starts, it begins searching the coreRootDirectory (which 
defaults to the solr home) for cores.  When it locates a core.properties 
file, that location becomes the instanceDir for a core.


If you do a manual core creation, the core.properties file must NOT 
exist, but the conf directory *must* exist with the proper contents. 
The core creation will create that file.  If it already exists, then 
Solr will refuse to create the core, just as you have seen.


The program directory location you have mentioned (/opt/solr-5.3.1) 
sounds like somebody did a service installation.  The default solr home 
when you install the service (and start Solr with /etc/init.d/ 
rather than bin/solr) is /var/solr/data.  This location can be 
overridden, but that's the default.


Instead of having your core in webapp, move it to the solr home, 
wherever that is.  Then when you start Solr, it will find the core.


If a service installation has been done, then you should not start Solr 
with "bin/solr" -- you should start the installed service.


Thanks,
Shawn

How to get a solr core to persist

[Amanda Shuman]

Hi all, I consider myself relatively new to server admin (it's something I
have to do on the side for the project I'm on and I inherited the current
solr setup from my predecessor), so please be kind.

The issue:

We have just one solr core, which resides in the webapp foIder (in solr
5.3.1 this is at /opt/solr-5.3.1/server/webapps/[corename]/  -- the data
folder is in the same place at /data). The core is used to index and search
content from just one webapp. The core "disappears" every time the server
restarts, making the webapp search break every time. By disappears I mean
that the folder and content is still in the webapps folder on the server,
but it needs to be remapped/re-linked via the solr admin console. The core
was and is added via the solr admin console, not at command line. Whenever
I add it back I get this message in solr or in the logs:

"Could not create a new core in
/opt/solr-5.3.1/server/webapps/[corename]/as another core is already
defined there"

Then I refresh the page and it's there again and everything works fine.

So my predecessor knew about this problem and never fixed it; he told me to
just re-add the core via admin console every time the server starts (not a
good solution).

I've done a lot of research on getting a core to persist and it probably
has to do with permissions, but where? All the folders in webapps are owned
by the solr user. If the issue is with the solr command at startup using
root, how can I change this?

I also added a new core at command line, which works in that it persists,
but the core is by default in var/solr folder. How can I create the core in
the webapps folder instead? Also, must I first delete everything in that
folder, including all the conf and data files, before creating the new
core? Will I need to re-build everything from scratch if I do this or can I
use the same conf files as before (I presume I can re-add them)?

Also, there's nothing wrong with the content or permissions of the core
properties files, it is owned by the solr user and contains one line with
the correct core name.

Thanks for your help,

Amanda