Re: All cores gone along with all solr configuration upon reboot

[yaswanth kumar]

Hi Erick,

Here is the latest most error that I captured which seems to be actually
deleting the cores ( I did noticed that the core folders under the path
../solr/server/solr were deleted one by one when the server came back from
reboot)

2020-08-24 04:41:27.424 ERROR
(coreContainerWorkExecutor-2-thread-1-processing-n:9.70.170.51:8080_solr) [
  ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on$
at
org.apache.solr.cloud.ZkController.checkStateInZk(ZkController.java:1875)

*org.apache.solr.cloud.ZkController$NotInClusterStateException:
coreNodeName core_node3 does not exist in shard shard1, ignore the
exception if the replica was deleted*at
org.apache.solr.cloud.ZkController.checkStateInZk(ZkController.java:1875)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19$
at
org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1774)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15$
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1238)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 201$
at
org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:756)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19$
at
org.apache.solr.core.CoreContainer$$Lambda$343/.call(Unknown
Source) ~[?:?]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202)
~[metrics-core-4.0.5.jar:4.0.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
~[solr-solrj-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e$
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/.run(Unknown
Source) ~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]

For some reason I believe solr is not able to find the replica in the
clusterstate and its causing the delete activity, not really sure on why
its not able to find it in clusterstate, I think due to some issue looks
like first clusterstate is getting wiped out and then slowly rest the cores
are getting deleted themselves.

As you asked I did cross checked once again on the port numbers and I am
using 2181 as a clientport and the same is what I see in the dashboard
screen of solr for ZKHOST., not really sure on how can I prevent this going
forward. One thing here is that I am using Solr basic AUTHENTICATION plugin
if it makes any difference.

On Sat, Aug 22, 2020 at 11:55 AM Erick Erickson 
wrote:

> Autopurge shouldn’t matter, that’s just cleaning up old snapshots. That
> is, it should be configured, but having it enabled or not should have no
> bearing on your data disappearing.
>
> Also, are you absolutely certain that you are using your external ZK?
> Check the port on the admin screen. 9983 is the default for embededded ZK.
>
> All that said, nothing in Solr just deletes all this. The fact that you
> only saw this on reboot is highly suspicious, some external-to-Solr
> process, anything from a startup script to restoring a disk image to…. is
> removing that data I suspect.
>
> Best,
> Erick
>
> > On Aug 22, 2020, at 9:24 AM, yaswanth kumar 
> wrote:
> >
> > Thanks Eric for looking into this..
> >
> > But as I said before I confirmed that the paths in zookeeper were
> changed to local path than the /tmp that comes default with package. Does
> the zoo.cfg need to have autopurge settings ??which I don’t have in my
> config
> >
> > Also I did make sure that zoo.cfg inside solr and my external zoo are
> pointing to the same and have same configs if it matters.
> >
> > Sent from my iPhone
> >
> >> On Aug 22, 2020, at 9:07 AM, Erick Erickson 
> wrote:
> >>
> >> Sounds like you didn’t change Zookeeper data dir. Zookeeper defaults
> to putting its data in /tmp/zookeeper, see the zookeeper config file. And,
> of course, when you reboot it goes away.
> >>
> >> I’ve always disliked this, but the Zookeeper folks did it that way. So
> if you just copy zoo_sample.cfg to zoo.cfg that’s what you get, not under
> Solr’s control.
> >>
> >> As for how to recover, assuming you put your configsets in some kind of
> version control as we recommend:
> >>
> >> 0> set up Zookeeper to keep it’s data somewhere permanent. You may want
> to archive snapshots upon occasion as well.
> >>
> >> 1> save away the data directory for _one_ replica from each shard of
> every collection somewhere. You should have a bunch of directories like
> SOLR_HOME/…./collection1_shard1_replica_n1/data.
> >>
> >> 2> recreate 

Re: Solr doesn't run after editing solr.in.sh

[Walter Underwood]

Also, what platform is this on and what editor did you use (especially if you 
are on Windows)?

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Aug 23, 2020, at 4:35 PM, Erick Erickson  wrote:
> 
> Well, first show exactly what you uncommented. I doubt you uncommented them 
> one by one and tried everything, so you leave us guessing. Uncommenting 
> SOLR_HOME for instance would be shooting yourself in the foot since Solr 
> wouldn’t know where to start. Uncommenting some the authorization parameters 
> without providing the proper values would cause Solr not to run. Uncommenting 
> #SOLR_OPTS="$SOLR_OPTS -Dsolr.environment=prod” should be fine.
> 
> 
> Second, show us the output when you _do_ try to run. You can use the -f 
> option to dump logging to the console.
> 
> Best,
> Erick
> 
>> On Aug 23, 2020, at 9:58 AM, Joe Doupnik  wrote:
>> 
>> On 22/08/2020 22:08, maciejpreg...@tutanota.com.INVALID wrote:
>>> Good morning.
>>> When I uncomment any of commands in solr.in.sh, Solr doesn't run. What do I 
>>> have to do to fix a problem?
>>> Best regards,
>>> Maciej Pregiel
>> On 22/08/2020 22:08, maciejpreg...@tutanota.com.INVALID wrote:
>>> Good morning.
>>> When I uncomment any of commands in solr.in.sh, Solr doesn't run. What do I 
>>> have to do to fix a problem?
>>> Best regards,
>>> Maciej Pregiel
>> -
>>My approach has been to add local configuration options to the end of the 
>> file and leave intact the original text. Here is the end of my file, which 
>> has no changes above this material:
>> 
>> #SOLR_SECURITY_MANAGER_ENABLED=false
>> ## JRD values
>> SOLR_ULIMIT_CHECKS=falseGC_TUNE=" \
>> -XX:SurvivorRatio=4 \
>> -XX:TargetSurvivorRatio=90 \
>> -XX:MaxTenuringThreshold=8 \
>> -XX:+UseConcMarkSweepGC \
>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
>> -XX:+CMSScavengeBeforeRemark \
>> -XX:PretenureSizeThreshold=64m \
>> -XX:+UseCMSInitiatingOccupancyOnly \
>> -XX:CMSInitiatingOccupancyFraction=50 \
>> -XX:CMSMaxAbortablePrecleanTime=6000 \
>> -XX:+CMSParallelRemarkEnabled \
>> -XX:+ParallelRefProcEnabled \
>> -XX:-OmitStackTraceInFastThrow"
>> #JRD give more memory
>> ##SOLR_HEAP="4096m"
>> SOLR_HEAP="2048m"
>> ##JRD enlarge this
>> #SOLR_OPTS="$SOLR_OPTS -Xss512k"
>> SOLR_OPTS="$SOLR_OPTS -Xss1024k"
>> SOLR_STOP_WAIT=30
>> SOLR_JAVA_HOME="/usr/java/latest/"
>> SOLR_PID_DIR="/home/search/solr"
>> SOLR_HOME="/home/search/solr/data"
>> SOLR_LOGS_DIR="/home/search/solr/logs"
>> SOLR_PORT="8983"
>> SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000"
>> SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=6"
>> SOLR_OPTS="$SOLR_OPTS -Djava.io.tmpdir=/home/search/tmp"
>> 
>>Thanks,
>>Joe D.
> 

Re: Solr doesn't run after editing solr.in.sh

[Erick Erickson]

Well, first show exactly what you uncommented. I doubt you uncommented them one 
by one and tried everything, so you leave us guessing. Uncommenting SOLR_HOME 
for instance would be shooting yourself in the foot since Solr wouldn’t know 
where to start. Uncommenting some the authorization parameters without 
providing the proper values would cause Solr not to run. Uncommenting 
#SOLR_OPTS="$SOLR_OPTS -Dsolr.environment=prod” should be fine.


Second, show us the output when you _do_ try to run. You can use the -f option 
to dump logging to the console.

Best,
Erick

> On Aug 23, 2020, at 9:58 AM, Joe Doupnik  wrote:
> 
> On 22/08/2020 22:08, maciejpreg...@tutanota.com.INVALID wrote:
>> Good morning.
>> When I uncomment any of commands in solr.in.sh, Solr doesn't run. What do I 
>> have to do to fix a problem?
>> Best regards,
>> Maciej Pregiel
> On 22/08/2020 22:08, maciejpreg...@tutanota.com.INVALID wrote:
>> Good morning.
>> When I uncomment any of commands in solr.in.sh, Solr doesn't run. What do I 
>> have to do to fix a problem?
>> Best regards,
>> Maciej Pregiel
> -
> My approach has been to add local configuration options to the end of the 
> file and leave intact the original text. Here is the end of my file, which 
> has no changes above this material:
> 
> #SOLR_SECURITY_MANAGER_ENABLED=false
> ## JRD values
> SOLR_ULIMIT_CHECKS=falseGC_TUNE=" \
> -XX:SurvivorRatio=4 \
> -XX:TargetSurvivorRatio=90 \
> -XX:MaxTenuringThreshold=8 \
> -XX:+UseConcMarkSweepGC \
> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> -XX:+CMSScavengeBeforeRemark \
> -XX:PretenureSizeThreshold=64m \
> -XX:+UseCMSInitiatingOccupancyOnly \
> -XX:CMSInitiatingOccupancyFraction=50 \
> -XX:CMSMaxAbortablePrecleanTime=6000 \
> -XX:+CMSParallelRemarkEnabled \
> -XX:+ParallelRefProcEnabled \
> -XX:-OmitStackTraceInFastThrow"
> #JRD give more memory
> ##SOLR_HEAP="4096m"
> SOLR_HEAP="2048m"
> ##JRD enlarge this
> #SOLR_OPTS="$SOLR_OPTS -Xss512k"
> SOLR_OPTS="$SOLR_OPTS -Xss1024k"
> SOLR_STOP_WAIT=30
> SOLR_JAVA_HOME="/usr/java/latest/"
> SOLR_PID_DIR="/home/search/solr"
> SOLR_HOME="/home/search/solr/data"
> SOLR_LOGS_DIR="/home/search/solr/logs"
> SOLR_PORT="8983"
> SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000"
> SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=6"
> SOLR_OPTS="$SOLR_OPTS -Djava.io.tmpdir=/home/search/tmp"
> 
> Thanks,
> Joe D.

Re: Solr 8.6.1: Can't round-trip nested document from SolrJ

[Alexandre Rafalovitch]

Thank you Nunedra,

That was very helpful. I am looking forward to that documentation Jira
to be merged into the next release.

I was able to get the example working by switching away from anonymous
children to the field approach. Which means hasChildren() call also
did not work. It seems the addChildren/hasChildren will need a
different schema, without _nest_path_ defined. I did not test.

Interestingly, I was forced to add children as an array even when the
child was alone and the field was already marked multivalued. It seems
the code does not do conversation to multi-value type, which means the
query code has to be a lot more careful about checking field return
type and having multi-path handling. That's not what Solr does for
string class (tested). Is that a known issue?
https://github.com/arafalov/SolrJTest/blob/master/src/com/solrstart/solrj/Main.java#L88-L89

If I switch commented/uncommented lines around, the retrieval will
fail part way through, because one 'children' field is returned as
array, but not the other one:
{responseHeader={status=0,QTime=0,params={q=id:p1,fl=*,[child],wt=javabin,version=2}},response={numFound=1,numFoundExact=true,start=0,docs=[SolrDocument{id=p1,
name=[parent1], class=[foo.bar.parent1.1, foo.bar.parent1.2],
_version_=1675826293154775040, children=[SolrDocument{id=c1,
name=[child1], class=[foo.bar.child1], _version_=1675826293154775040,
children=SolrDocument{id=gc1, name=[grandChild1],
class=[foo.bar.grandchild1], _version_=1675826293154775040}},
SolrDocument{id=c2, name=[child2], class=[foo.bar.child2],
_version_=1675826293154775040}]}]}}

Regards,
   Alex.

On Sun, 23 Aug 2020 at 01:38, Munendra S N  wrote:
>
> Hi Alex,
>
> Currently, Fixing the documentation for nested docs is under progress. More
> context is available in this JIRA -
> https://issues.apache.org/jira/browse/SOLR-14383.
>
> https://github.com/arafalov/SolrJTest/blob/master/src/com/solrstart/solrj/Main.java
>
> The child doc transformer needs to be specified as part of the fl parameter
> like fl=*,[child] so that the descendants are returned for each matching
> doc. As the query q=* matches all the documents, they are returned. If only
> parent doc needs to be returned with descendants then, we should either use
> block join query or query clause which matches only parent doc.
>
> Another thing I noticed in the code is that the child docs are indexed as
> anonymous docs (similar to old syntax) instead of indexing them in the new
> syntax. With this, the nested block will be indexed but since the schema
> has _nested_path_ defined [child] doc transformer won't return any docs.
> Anonymous child docs need parentFilter but specifying parentFilter with
> _nested_path_ will lead to error
> It is due to this check -
> https://github.com/apache/lucene-solr/blob/1c8f4c988a07b08f83d85e27e59b43eed5e2ca2a/solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformerFactory.java#L104
>
> Instead of indexing the docs this way,
>
> > SolrInputDocument parent1 = new SolrInputDocument();
> > parent1.addField("id", "p1");
> > parent1.addField("name", "parent1");
> > parent1.addField("class", "foo.bar.parent1");
> >
> > SolrInputDocument child1 = new SolrInputDocument();
> >
> > parent1.addChildDocument(child1);
> > child1.addField("id", "c1");
> > child1.addField("name", "child1");
> > child1.addField("class", "foo.bar.child1");
> >
> >
> modify it to indexing
>
> > SolrInputDocument parent1 = new SolrInputDocument();
> > parent1.addField("id", "p1");
> > parent1.addField("name", "parent1");
> > parent1.addField("class", "foo.bar.parent1");
> >
> > SolrInputDocument child1 = new SolrInputDocument();
> >
> > parent1.addField("sometag", Arrays.asList(child1));
> > child1.addField("id", "c1");
> > child1.addField("name", "child1");
> > child1.addField("class", "foo.bar.child1");
> >
> > I think, once the documentation fixes get merged to master, indexing and
> searching with the nested documents will become much clearer.
>
> Regards,
> Munendra S N
>
>
>
> On Sun, Aug 23, 2020 at 5:18 AM Alexandre Rafalovitch 
> wrote:
>
> > Hello,
> >
> > I am trying to get up to date with both SolrJ and Nested Document
> > implementation and not sure where I am failing with a basic test
> > (
> > https://github.com/arafalov/SolrJTest/blob/master/src/com/solrstart/solrj/Main.java
> > ).
> >
> > I am using Solr 8.6.1 with a core created with bin/solr create -c
> > solrj (schemaless is still on).
> >
> > I then index a nested parent/child/grandchild document in and then
> > query it back. Looking at debug it seems to go out fine as a nested
> > doc but come back as a 3 individual ones.
> >
> > Output is:
> > SolrInputDocument(fields: [id=p1, name=parent1,
> > class=foo.bar.parent1], children: [SolrInputDocument(fields: [id=c1,
> > name=child1, class=foo.bar.child1], children:
> > [SolrInputDocument(fields: [id=gc1, name=grandChild1,
> > class=foo.bar.grandchild1])])])
> >
> > 

Re: Solr doesn't run after editing solr.in.sh

[Joe Doupnik]

On 22/08/2020 22:08, maciejpreg...@tutanota.com.INVALID wrote:

Good morning.
When I uncomment any of commands in solr.in.sh, Solr doesn't run. What do I 
have to do to fix a problem?
Best regards,
Maciej Pregiel

On 22/08/2020 22:08, maciejpreg...@tutanota.com.INVALID wrote:

Good morning.
When I uncomment any of commands in solr.in.sh, Solr doesn't run. What 
do I have to do to fix a problem?

Best regards,
Maciej Pregiel

-
    My approach has been to add local configuration options to the end 
of the file and leave intact the original text. Here is the end of my 
file, which has no changes above this material:


#SOLR_SECURITY_MANAGER_ENABLED=false
## JRD values
SOLR_ULIMIT_CHECKS=falseGC_TUNE=" \
-XX:SurvivorRatio=4 \
-XX:TargetSurvivorRatio=90 \
-XX:MaxTenuringThreshold=8 \
-XX:+UseConcMarkSweepGC \
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
-XX:+CMSScavengeBeforeRemark \
-XX:PretenureSizeThreshold=64m \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:CMSInitiatingOccupancyFraction=50 \
-XX:CMSMaxAbortablePrecleanTime=6000 \
-XX:+CMSParallelRemarkEnabled \
-XX:+ParallelRefProcEnabled \
-XX:-OmitStackTraceInFastThrow"
#JRD give more memory
##SOLR_HEAP="4096m"
SOLR_HEAP="2048m"
##JRD enlarge this
#SOLR_OPTS="$SOLR_OPTS -Xss512k"
SOLR_OPTS="$SOLR_OPTS -Xss1024k"
SOLR_STOP_WAIT=30
SOLR_JAVA_HOME="/usr/java/latest/"
SOLR_PID_DIR="/home/search/solr"
SOLR_HOME="/home/search/solr/data"
SOLR_LOGS_DIR="/home/search/solr/logs"
SOLR_PORT="8983"
SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000"
SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=6"
SOLR_OPTS="$SOLR_OPTS -Djava.io.tmpdir=/home/search/tmp"

    Thanks,
    Joe D.

Solr doesn't run after editing solr.in.sh

[maciejpregiel]

Good morning.
When I uncomment any of commands in solr.in.sh, Solr doesn't run. What do I 
have to do to fix a problem?
Best regards,
Maciej Pregiel