Re: Outdated information on JVM heap sizes in Solr 8.3 documentation?
https://issues.apache.org/jira/browse/SOLR-14263 > On Feb 15, 2020, at 10:17 AM, Mike Drob wrote: > > Erick, > > Can you drop a link to that Jira here after you create it? > > Many thanks, > Mike > > On Fri, Feb 14, 2020 at 6:05 PM Erick Erickson > wrote: > >> I just read that page over and it looks way out of date. I’ll raise >> a JIRA. >> >>> On Feb 14, 2020, at 2:55 PM, Walter Underwood >> wrote: >>> >>> Yeah, that is pretty outdated. At Netflix, I was running an 8 GB heap >> with Solr 1.3. :-) >>> >>> Every GC I know about has a stop-the-world collector as a last ditch >> measure. >>> >>> G1GC limits the time that the world will stop. It gives up after >> MaxGCPauseMillis >>> milliseconds and leaves the rest of the garbage uncollected. If it has 5 >> seconds >>> worth of work to do that, it might take 10 seconds, but in 200 ms >> chunks. It does >>> a lot of other stuff outside of the pauses to make the major collections >> more effective. >>> >>> We wrote Ultraseek in Python+C because Python used reference counting and >>> did not do garbage collection. That is the only way to have no pauses >> with >>> automatic memory management. >>> >>> wunder >>> Walter Underwood >>> wun...@wunderwood.org >>> http://observer.wunderwood.org/ (my blog) >>> >>>> On Feb 14, 2020, at 11:35 AM, Tom Burton-West >> wrote: >>>> >>>> Hello, >>>> >>>> In the section on JVM tuning in the Solr 8.3 documentation ( >>>> https://lucene.apache.org/solr/guide/8_3/jvm-settings.html#jvm-settings >> ) >>>> there is a paragraph which cautions about setting heap sizes over 2 GB: >>>> >>>> "The larger the heap the longer it takes to do garbage collection. This >> can >>>> mean minor, random pauses or, in extreme cases, "freeze the world" >> pauses >>>> of a minute or more. As a practical matter, this can become a serious >>>> problem for heap sizes that exceed about **two gigabytes**, even if far >>>> more physical memory is available. On robust hardware, you may get >> better >>>> results running multiple JVMs, rather than just one with a large memory >>>> heap. " (** added by me) >>>> >>>> I suspect this paragraph is severely outdated, but am not a Java expert. >>>> It seems to be contradicted by the statement in " >>>> >> https://lucene.apache.org/solr/guide/8_3/taking-solr-to-production.html#memory-and-gc-settings >> " >>>> "...values between 10 and 20 gigabytes are not uncommon for production >>>> servers" >>>> >>>> Are "freeze the world" pauses still an issue with modern JVM's? >>>> Is it still advisable to avoid heap sizes over 2GB? >>>> >>>> Tom >>>> https://www.hathitrust.org/blogslarge-scale-search >>> >> >>
Re: Outdated information on JVM heap sizes in Solr 8.3 documentation?
Erick, Can you drop a link to that Jira here after you create it? Many thanks, Mike On Fri, Feb 14, 2020 at 6:05 PM Erick Erickson wrote: > I just read that page over and it looks way out of date. I’ll raise > a JIRA. > > > On Feb 14, 2020, at 2:55 PM, Walter Underwood > wrote: > > > > Yeah, that is pretty outdated. At Netflix, I was running an 8 GB heap > with Solr 1.3. :-) > > > > Every GC I know about has a stop-the-world collector as a last ditch > measure. > > > > G1GC limits the time that the world will stop. It gives up after > MaxGCPauseMillis > > milliseconds and leaves the rest of the garbage uncollected. If it has 5 > seconds > > worth of work to do that, it might take 10 seconds, but in 200 ms > chunks. It does > > a lot of other stuff outside of the pauses to make the major collections > more effective. > > > > We wrote Ultraseek in Python+C because Python used reference counting and > > did not do garbage collection. That is the only way to have no pauses > with > > automatic memory management. > > > > wunder > > Walter Underwood > > wun...@wunderwood.org > > http://observer.wunderwood.org/ (my blog) > > > >> On Feb 14, 2020, at 11:35 AM, Tom Burton-West > wrote: > >> > >> Hello, > >> > >> In the section on JVM tuning in the Solr 8.3 documentation ( > >> https://lucene.apache.org/solr/guide/8_3/jvm-settings.html#jvm-settings > ) > >> there is a paragraph which cautions about setting heap sizes over 2 GB: > >> > >> "The larger the heap the longer it takes to do garbage collection. This > can > >> mean minor, random pauses or, in extreme cases, "freeze the world" > pauses > >> of a minute or more. As a practical matter, this can become a serious > >> problem for heap sizes that exceed about **two gigabytes**, even if far > >> more physical memory is available. On robust hardware, you may get > better > >> results running multiple JVMs, rather than just one with a large memory > >> heap. " (** added by me) > >> > >> I suspect this paragraph is severely outdated, but am not a Java expert. > >> It seems to be contradicted by the statement in " > >> > https://lucene.apache.org/solr/guide/8_3/taking-solr-to-production.html#memory-and-gc-settings > " > >> "...values between 10 and 20 gigabytes are not uncommon for production > >> servers" > >> > >> Are "freeze the world" pauses still an issue with modern JVM's? > >> Is it still advisable to avoid heap sizes over 2GB? > >> > >> Tom > >> https://www.hathitrust.org/blogslarge-scale-search > > > >
Re: Outdated information on JVM heap sizes in Solr 8.3 documentation?
I just read that page over and it looks way out of date. I’ll raise a JIRA. > On Feb 14, 2020, at 2:55 PM, Walter Underwood wrote: > > Yeah, that is pretty outdated. At Netflix, I was running an 8 GB heap with > Solr 1.3. :-) > > Every GC I know about has a stop-the-world collector as a last ditch measure. > > G1GC limits the time that the world will stop. It gives up after > MaxGCPauseMillis > milliseconds and leaves the rest of the garbage uncollected. If it has 5 > seconds > worth of work to do that, it might take 10 seconds, but in 200 ms chunks. It > does > a lot of other stuff outside of the pauses to make the major collections more > effective. > > We wrote Ultraseek in Python+C because Python used reference counting and > did not do garbage collection. That is the only way to have no pauses with > automatic memory management. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > >> On Feb 14, 2020, at 11:35 AM, Tom Burton-West wrote: >> >> Hello, >> >> In the section on JVM tuning in the Solr 8.3 documentation ( >> https://lucene.apache.org/solr/guide/8_3/jvm-settings.html#jvm-settings) >> there is a paragraph which cautions about setting heap sizes over 2 GB: >> >> "The larger the heap the longer it takes to do garbage collection. This can >> mean minor, random pauses or, in extreme cases, "freeze the world" pauses >> of a minute or more. As a practical matter, this can become a serious >> problem for heap sizes that exceed about **two gigabytes**, even if far >> more physical memory is available. On robust hardware, you may get better >> results running multiple JVMs, rather than just one with a large memory >> heap. " (** added by me) >> >> I suspect this paragraph is severely outdated, but am not a Java expert. >> It seems to be contradicted by the statement in " >> https://lucene.apache.org/solr/guide/8_3/taking-solr-to-production.html#memory-and-gc-settings; >> "...values between 10 and 20 gigabytes are not uncommon for production >> servers" >> >> Are "freeze the world" pauses still an issue with modern JVM's? >> Is it still advisable to avoid heap sizes over 2GB? >> >> Tom >> https://www.hathitrust.org/blogslarge-scale-search >
Re: Outdated information on JVM heap sizes in Solr 8.3 documentation?
Yeah, that is pretty outdated. At Netflix, I was running an 8 GB heap with Solr 1.3. :-) Every GC I know about has a stop-the-world collector as a last ditch measure. G1GC limits the time that the world will stop. It gives up after MaxGCPauseMillis milliseconds and leaves the rest of the garbage uncollected. If it has 5 seconds worth of work to do that, it might take 10 seconds, but in 200 ms chunks. It does a lot of other stuff outside of the pauses to make the major collections more effective. We wrote Ultraseek in Python+C because Python used reference counting and did not do garbage collection. That is the only way to have no pauses with automatic memory management. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Feb 14, 2020, at 11:35 AM, Tom Burton-West wrote: > > Hello, > > In the section on JVM tuning in the Solr 8.3 documentation ( > https://lucene.apache.org/solr/guide/8_3/jvm-settings.html#jvm-settings) > there is a paragraph which cautions about setting heap sizes over 2 GB: > > "The larger the heap the longer it takes to do garbage collection. This can > mean minor, random pauses or, in extreme cases, "freeze the world" pauses > of a minute or more. As a practical matter, this can become a serious > problem for heap sizes that exceed about **two gigabytes**, even if far > more physical memory is available. On robust hardware, you may get better > results running multiple JVMs, rather than just one with a large memory > heap. " (** added by me) > > I suspect this paragraph is severely outdated, but am not a Java expert. > It seems to be contradicted by the statement in " > https://lucene.apache.org/solr/guide/8_3/taking-solr-to-production.html#memory-and-gc-settings; > "...values between 10 and 20 gigabytes are not uncommon for production > servers" > > Are "freeze the world" pauses still an issue with modern JVM's? > Is it still advisable to avoid heap sizes over 2GB? > > Tom > https://www.hathitrust.org/blogslarge-scale-search
Outdated information on JVM heap sizes in Solr 8.3 documentation?
Hello, In the section on JVM tuning in the Solr 8.3 documentation ( https://lucene.apache.org/solr/guide/8_3/jvm-settings.html#jvm-settings) there is a paragraph which cautions about setting heap sizes over 2 GB: "The larger the heap the longer it takes to do garbage collection. This can mean minor, random pauses or, in extreme cases, "freeze the world" pauses of a minute or more. As a practical matter, this can become a serious problem for heap sizes that exceed about **two gigabytes**, even if far more physical memory is available. On robust hardware, you may get better results running multiple JVMs, rather than just one with a large memory heap. " (** added by me) I suspect this paragraph is severely outdated, but am not a Java expert. It seems to be contradicted by the statement in " https://lucene.apache.org/solr/guide/8_3/taking-solr-to-production.html#memory-and-gc-settings; "...values between 10 and 20 gigabytes are not uncommon for production servers" Are "freeze the world" pauses still an issue with modern JVM's? Is it still advisable to avoid heap sizes over 2GB? Tom https://www.hathitrust.org/blogslarge-scale-search
Re: Handling All Replicas Down in Solr 8.3 Cloud Collection
Here's roughly what was going on: 1. set up three node cluster with a collection. The collection has one shard and three replicas for that shard. 2. Shut down two of the nodes and verify the remaining node is the leader. Verified the other two nodes are registered as dead in solr ui. 3. bulk import several million documents into solr from a CSV file. 4. shut down the remaining node 5. start up all three nodes Even after three minutes no leader was active. I executed the FORCELEADER API call which completed successfully and waited three minutes -- still no replica was elected leader. I then compared my solr 8 cluster to a different solr cluster. I noticed that the znode */collections/example/leaders/shard1 *existed on both clusters but in the solr 8 cluster the znode was empty. I manually uploaded a json document with the proper settings to that znode and then called the FORCELEADER API call again and waited 3 minutes. A leader still wasn't elected. Then, I removed the replica for the node that I imported all the documents into it and added the replica back in. At that point, a leader was elected. I am not sure i have exact steps to reproduce but I did get it working. Thanks, Joe On Tue, Feb 4, 2020 at 7:54 AM Erick Erickson wrote: > First, be sure to wait at least 3 minutes before concluding the replicas > are permanently down, that’s the default wait period for certain leader > election fallbacks. It’s easy to conclude it’s never going to recover, 180 > seconds is an eternity ;). > > You can try the collections API FORCELEADER command. Assuming a leader is > elected and becomes active, you _may_ have to restart the other two Solr > nodes. > > How did you stop the servers? You mention disaster recovery, so I’m > thinking you did a “kill -9” or similar? Were you actively indexing at the > time? Solr _should_ manage the recovery even in that case, I’m mostly > wondering what the sequence of events that lead up to this was… > > Best, > Erick > > > On Feb 4, 2020, at 8:38 AM, Joseph Lorenzini wrote: > > > > Hi all, > > > > I have a 3 node solr cloud instance with a single collection. The solr > > nodes are pointed to a 3-node zookeeper ensemble. I was doing some basic > > disaster recovery testing and have encountered a problem that hasn't been > > obvious to me on how to fix. > > > > After i started back up the three solr java processes, i can see that > they > > are registered back in the solr UI. However, each replica is in a down > > state permanently. there are no logs in either solr or zookeeper that may > > indicate what the the problem would be -- neither exceptions nor > warnings. > > > > So is there any way to collect more diagnostics to figure out what's > going > > on? Short of deleting and recreating the replicas is there any way to fix > > this? > > > > Thanks, > > Joe > >
Re: Handling All Replicas Down in Solr 8.3 Cloud Collection
First, be sure to wait at least 3 minutes before concluding the replicas are permanently down, that’s the default wait period for certain leader election fallbacks. It’s easy to conclude it’s never going to recover, 180 seconds is an eternity ;). You can try the collections API FORCELEADER command. Assuming a leader is elected and becomes active, you _may_ have to restart the other two Solr nodes. How did you stop the servers? You mention disaster recovery, so I’m thinking you did a “kill -9” or similar? Were you actively indexing at the time? Solr _should_ manage the recovery even in that case, I’m mostly wondering what the sequence of events that lead up to this was… Best, Erick > On Feb 4, 2020, at 8:38 AM, Joseph Lorenzini wrote: > > Hi all, > > I have a 3 node solr cloud instance with a single collection. The solr > nodes are pointed to a 3-node zookeeper ensemble. I was doing some basic > disaster recovery testing and have encountered a problem that hasn't been > obvious to me on how to fix. > > After i started back up the three solr java processes, i can see that they > are registered back in the solr UI. However, each replica is in a down > state permanently. there are no logs in either solr or zookeeper that may > indicate what the the problem would be -- neither exceptions nor warnings. > > So is there any way to collect more diagnostics to figure out what's going > on? Short of deleting and recreating the replicas is there any way to fix > this? > > Thanks, > Joe
Handling All Replicas Down in Solr 8.3 Cloud Collection
Hi all, I have a 3 node solr cloud instance with a single collection. The solr nodes are pointed to a 3-node zookeeper ensemble. I was doing some basic disaster recovery testing and have encountered a problem that hasn't been obvious to me on how to fix. After i started back up the three solr java processes, i can see that they are registered back in the solr UI. However, each replica is in a down state permanently. there are no logs in either solr or zookeeper that may indicate what the the problem would be -- neither exceptions nor warnings. So is there any way to collect more diagnostics to figure out what's going on? Short of deleting and recreating the replicas is there any way to fix this? Thanks, Joe
Re: Solr 8.3
Thanks, Erick The solution provided by you works for us. Regarding the reason to implement it is as below. Current we use Solr 6.1 version and plan to switch to 8.3. In 6.1 the behavior is replica and shards of all collection maintain on separate. On production, many times we face replica goes into recovery mode. After some time if it not respond we required to restart it. Now, consider same scenario for 8.3 version and on one ip few collections have shared and some have a replica, anything goes into recovery may be hard to restart/manage it. If you can suggest any better way to manage this or overcome such a scenario please suggest. Regards, Vishal From: Erick Erickson Sent: Friday, January 3, 2020 8:39 PM To: solr-user@lucene.apache.org Subject: Re: Solr 8.3 Well, you can’t do what you want. The Admin UI is intended as a way for new users to get started, it was never intended to support all of the possible options. Look at the documentation for the Collections CREATE API, for instance: https://lucene.apache.org/solr/guide/8_1/collections-api.html In particular createNodeSet, and createNodeSet.shuffle. If you want to control absolutely everything, the special value EMPTY for createNodeSet won’t create any replicas and you can place each one individually with ADDREPLICA. Of course you can script this if you need automation. What you haven’t explained is _why_ you want to do this. Very often taking this kind of control is unnecessary and a distraction. That said, when it _is_ necessary you need to use the Collections API rather than the admin UI. Best, Erick > On Jan 3, 2020, at 4:53 AM, vishal patel > wrote: > > Thanks for the reply. Actually, I don't want to change the IP of the replica > but I want to create a collection on a specific shard IP. > As per my production server configuration, we consider 10.38.33.28 is shard1, > 10.38.33.30 is shard2, 10.38.33.31 is replica of shard1 and 10.38.33.29 is > replica of shard2. > We need to create all new collections the same as above. > > > When I create a collection(documents) using the admin panel. It created like > shard1 10.38.33.30 > replica1 10.38.33.31 > shard2 10.38.33.28 > replica2 10.38.33.29 > > Actually we want like this > shard1 10.38.33.28 > replica1 10.38.33.31 > shard2 10.38.33.30 > replica2 10.38.33.29 > > Regards, > Vishal > > From: Sankar Panda > Sent: Friday, January 3, 2020 12:36 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr 8.3 > > Hi Vishal, > > You can .go to the collection in admin console.mannually changed the ip > address as you want.Remove the replica and add it as per your > requirements.This option provides by the solr cloud. > Thanks > Sankar panda > > On Fri, Jan 3, 2020, 11:33 vishal patel > mailto:vishalpatel200...@outlook.com>> wrote: > I do not want to change the IP of the existing replica. I want to fix the IP > for the first time creating a collection. > > I have 4 machines. my IP of each machine is below > machine1 10.38.33.28 > machine2 10.38.33.29 > machine3 10.38.33.30 > machine4 10.38.33.31 > > I have created solr instance on each machine. when I am creating a first > collection(documents) using admin panel my structure becomes like this > shard1 10.38.33.30 > replica1 10.38.33.31 > shard2 10.38.33.28 > replica2 10.38.33.29 > > When creating a second collection(forms), my structure becomes like this > shard1 10.38.33.28 > replica1 10.38.33.30 > shard2 10.38.33.29 > replica2 10.38.33.31 > > I have attached a screenshot. [https://ibb.co/Yb2TpVX] > > Randomly IP assigned to shard or replica when creating collection but I want > to make shard1 on 10.38.33.28, replica1 on 10.38.33.31, shard2 on 10.38.33.30 > and replica2 on 10.38.33.31 when first time creating a collection using admin > panel. > > Can I Fix the IP to shard OR replica when creating a new > [cid:4c560b03-a18d-4bc1-b592-6b2f8103f1ae] collection? > > Regards, > Vishal > > > ____ > From: Erick Erickson mailto:erickerick...@gmail.com>> > Sent: Thursday, January 2, 2020 7:40 PM > To: solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org> > mailto:solr-user@lucene.apache.org>> > Subject: Re: Solr 8.3 > > No, you cannot change the IP of an existing replica. Either do as Sankar > mentioned when you first create the collection or use the MOVREPLICA > collections API command. > > MOVEREPLICA has existed for quite a long time, but if it’s not available, you > can do the same with the ADDREPLICA command to add a replica to a specific > node, wait for the replica to become full
Re: Solr 8.3
Well, you can’t do what you want. The Admin UI is intended as a way for new users to get started, it was never intended to support all of the possible options. Look at the documentation for the Collections CREATE API, for instance: https://lucene.apache.org/solr/guide/8_1/collections-api.html In particular createNodeSet, and createNodeSet.shuffle. If you want to control absolutely everything, the special value EMPTY for createNodeSet won’t create any replicas and you can place each one individually with ADDREPLICA. Of course you can script this if you need automation. What you haven’t explained is _why_ you want to do this. Very often taking this kind of control is unnecessary and a distraction. That said, when it _is_ necessary you need to use the Collections API rather than the admin UI. Best, Erick > On Jan 3, 2020, at 4:53 AM, vishal patel > wrote: > > Thanks for the reply. Actually, I don't want to change the IP of the replica > but I want to create a collection on a specific shard IP. > As per my production server configuration, we consider 10.38.33.28 is shard1, > 10.38.33.30 is shard2, 10.38.33.31 is replica of shard1 and 10.38.33.29 is > replica of shard2. > We need to create all new collections the same as above. > > > When I create a collection(documents) using the admin panel. It created like > shard1 10.38.33.30 > replica1 10.38.33.31 > shard2 10.38.33.28 > replica2 10.38.33.29 > > Actually we want like this > shard1 10.38.33.28 > replica1 10.38.33.31 > shard2 10.38.33.30 > replica2 10.38.33.29 > > Regards, > Vishal > > From: Sankar Panda > Sent: Friday, January 3, 2020 12:36 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr 8.3 > > Hi Vishal, > > You can .go to the collection in admin console.mannually changed the ip > address as you want.Remove the replica and add it as per your > requirements.This option provides by the solr cloud. > Thanks > Sankar panda > > On Fri, Jan 3, 2020, 11:33 vishal patel > mailto:vishalpatel200...@outlook.com>> wrote: > I do not want to change the IP of the existing replica. I want to fix the IP > for the first time creating a collection. > > I have 4 machines. my IP of each machine is below > machine1 10.38.33.28 > machine2 10.38.33.29 > machine3 10.38.33.30 > machine4 10.38.33.31 > > I have created solr instance on each machine. when I am creating a first > collection(documents) using admin panel my structure becomes like this > shard1 10.38.33.30 > replica1 10.38.33.31 > shard2 10.38.33.28 > replica2 10.38.33.29 > > When creating a second collection(forms), my structure becomes like this > shard1 10.38.33.28 > replica1 10.38.33.30 > shard2 10.38.33.29 > replica2 10.38.33.31 > > I have attached a screenshot. [https://ibb.co/Yb2TpVX] > > Randomly IP assigned to shard or replica when creating collection but I want > to make shard1 on 10.38.33.28, replica1 on 10.38.33.31, shard2 on 10.38.33.30 > and replica2 on 10.38.33.31 when first time creating a collection using admin > panel. > > Can I Fix the IP to shard OR replica when creating a new > [cid:4c560b03-a18d-4bc1-b592-6b2f8103f1ae] collection? > > Regards, > Vishal > > > ____ > From: Erick Erickson mailto:erickerick...@gmail.com>> > Sent: Thursday, January 2, 2020 7:40 PM > To: solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org> > mailto:solr-user@lucene.apache.org>> > Subject: Re: Solr 8.3 > > No, you cannot change the IP of an existing replica. Either do as Sankar > mentioned when you first create the collection or use the MOVREPLICA > collections API command. > > MOVEREPLICA has existed for quite a long time, but if it’s not available, you > can do the same with the ADDREPLICA command to add a replica to a specific > node, wait for the replica to become fully active, then use DELETEREPLICA on > the one you no longer want. > > Best, > Erick > >> On Jan 2, 2020, at 8:10 AM, Sankar Panda >> mailto:panda.san...@gmail.com>> wrote: >> >> Hi Vishal, >> You can create a empty nodeset and manually configure in the collection as >> desired in the admin page >> Thanks >> Sankar Panda >> >> On Thu, Jan 2, 2020, 14:36 vishal patel >> mailto:vishalpatel200...@outlook.com>> >> wrote: >> >>> My created collection in solr cloud below >>> >>> 10.38.33.24 is shard and its replica is 10.38.33.27. >>> 10.38.33.227 is shard and its replica is 10.38.33.219. >>> >>> I want to create a new collectio
Re: Solr 8.3
Thanks for the reply. Actually, I don't want to change the IP of the replica but I want to create a collection on a specific shard IP. As per my production server configuration, we consider 10.38.33.28 is shard1, 10.38.33.30 is shard2, 10.38.33.31 is replica of shard1 and 10.38.33.29 is replica of shard2. We need to create all new collections the same as above. When I create a collection(documents) using the admin panel. It created like shard1 10.38.33.30 replica1 10.38.33.31 shard2 10.38.33.28 replica2 10.38.33.29 Actually we want like this shard1 10.38.33.28 replica1 10.38.33.31 shard2 10.38.33.30 replica2 10.38.33.29 Regards, Vishal From: Sankar Panda Sent: Friday, January 3, 2020 12:36 PM To: solr-user@lucene.apache.org Subject: Re: Solr 8.3 Hi Vishal, You can .go to the collection in admin console.mannually changed the ip address as you want.Remove the replica and add it as per your requirements.This option provides by the solr cloud. Thanks Sankar panda On Fri, Jan 3, 2020, 11:33 vishal patel mailto:vishalpatel200...@outlook.com>> wrote: I do not want to change the IP of the existing replica. I want to fix the IP for the first time creating a collection. I have 4 machines. my IP of each machine is below machine1 10.38.33.28 machine2 10.38.33.29 machine3 10.38.33.30 machine4 10.38.33.31 I have created solr instance on each machine. when I am creating a first collection(documents) using admin panel my structure becomes like this shard1 10.38.33.30 replica1 10.38.33.31 shard2 10.38.33.28 replica2 10.38.33.29 When creating a second collection(forms), my structure becomes like this shard1 10.38.33.28 replica1 10.38.33.30 shard2 10.38.33.29 replica2 10.38.33.31 I have attached a screenshot. [https://ibb.co/Yb2TpVX] Randomly IP assigned to shard or replica when creating collection but I want to make shard1 on 10.38.33.28, replica1 on 10.38.33.31, shard2 on 10.38.33.30 and replica2 on 10.38.33.31 when first time creating a collection using admin panel. Can I Fix the IP to shard OR replica when creating a new [cid:4c560b03-a18d-4bc1-b592-6b2f8103f1ae] collection? Regards, Vishal From: Erick Erickson mailto:erickerick...@gmail.com>> Sent: Thursday, January 2, 2020 7:40 PM To: solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org> mailto:solr-user@lucene.apache.org>> Subject: Re: Solr 8.3 No, you cannot change the IP of an existing replica. Either do as Sankar mentioned when you first create the collection or use the MOVREPLICA collections API command. MOVEREPLICA has existed for quite a long time, but if it’s not available, you can do the same with the ADDREPLICA command to add a replica to a specific node, wait for the replica to become fully active, then use DELETEREPLICA on the one you no longer want. Best, Erick > On Jan 2, 2020, at 8:10 AM, Sankar Panda > mailto:panda.san...@gmail.com>> wrote: > > Hi Vishal, > You can create a empty nodeset and manually configure in the collection as > desired in the admin page > Thanks > Sankar Panda > > On Thu, Jan 2, 2020, 14:36 vishal patel > mailto:vishalpatel200...@outlook.com>> > wrote: > >> My created collection in solr cloud below >> >> 10.38.33.24 is shard and its replica is 10.38.33.27. >> 10.38.33.227 is shard and its replica is 10.38.33.219. >> >> I want to create a new collection on the Same. can not change the shard IP >> for the new collection. How can I? >> >> >> -- >> *From:* sudhir kumar >> mailto:gogikarsud...@gmail.com>> >> *Sent:* Thursday, January 2, 2020 2:01 PM >> *To:* solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org> >> mailto:solr-user@lucene.apache.org>> >> *Subject:* Re: Solr 8.3 >> >> sample url to create collection: >> >> >> http//host:8080/solr/admin/collections?action=CREATE=collectionname=2=3=2= >> host:8080_solr,host:8080_solr,host:8080_solr,host:8080_solr >> =collectionconfig >> >> On Thu, Jan 2, 2020 at 1:56 PM sudhir kumar >> mailto:gogikarsud...@gmail.com>> >> wrote: >> >>> Hey Vishal, >>> >>> You can use createNodeSet property while creating collection which will >>> allows you to create shards on specified IP. >>> >>> /admin/collections?action=CREATE=*name*=*number* >>> =*number*&*maxShardsPerNode*=*number*&*createNodeSet*= >>> *nodelist*=*configname* >>> >>> >>> Thanks, >>> >>> Sudhir >>> >>> >>> On Thu, Jan 2, 2020 at 1:49 PM vishal patel < >> vishalpatel200...@outlook.com<mailto:vishalpatel200...@outlook.com>> >>> wrote: >>> >>>> When I am creating 2 shards and 2 replicas using admin panel, automatic >>>> assign a shard or replica to any IP. >>>> I want to make the specific shard or replica to solr instance at the >> time >>>> of creating a collection. Can I? >>>> >>>> Regards, >>>> Vishal >>>> >>> >>
Re: Solr 8.3
Hi Vishal, You can .go to the collection in admin console.mannually changed the ip address as you want.Remove the replica and add it as per your requirements.This option provides by the solr cloud. Thanks Sankar panda On Fri, Jan 3, 2020, 11:33 vishal patel wrote: > I do not want to change the IP of the existing replica. I want to fix the > IP for the first time creating a collection. > > I have 4 machines. my IP of each machine is below > machine1 10.38.33.28 > machine2 10.38.33.29 > machine3 10.38.33.30 > machine4 10.38.33.31 > > I have created solr instance on each machine. when I am creating a first > collection(documents) using admin panel my structure becomes like this > shard1 10.38.33.30 >replica1 10.38.33.31 > shard2 10.38.33.28 >replica2 10.38.33.29 > > When creating a second collection(forms), my structure becomes like this > shard1 10.38.33.28 >replica1 10.38.33.30 > shard2 10.38.33.29 >replica2 10.38.33.31 > > I have attached a screenshot. [https://ibb.co/Yb2TpVX] > > Randomly IP assigned to shard or replica when creating collection but I > want to make shard1 on 10.38.33.28, replica1 on 10.38.33.31, shard2 on > 10.38.33.30 and replica2 on 10.38.33.31 when first time creating a > collection using admin panel. > > Can I Fix the IP to shard OR replica when creating a new collection? > > Regards, > Vishal > > > -- > *From:* Erick Erickson > *Sent:* Thursday, January 2, 2020 7:40 PM > *To:* solr-user@lucene.apache.org > *Subject:* Re: Solr 8.3 > > No, you cannot change the IP of an existing replica. Either do as Sankar > mentioned when you first create the collection or use the MOVREPLICA > collections API command. > > MOVEREPLICA has existed for quite a long time, but if it’s not available, > you can do the same with the ADDREPLICA command to add a replica to a > specific node, wait for the replica to become fully active, then use > DELETEREPLICA on the one you no longer want. > > Best, > Erick > > > On Jan 2, 2020, at 8:10 AM, Sankar Panda wrote: > > > > Hi Vishal, > > You can create a empty nodeset and manually configure in the collection > as > > desired in the admin page > > Thanks > > Sankar Panda > > > > On Thu, Jan 2, 2020, 14:36 vishal patel > > wrote: > > > >> My created collection in solr cloud below > >> > >> 10.38.33.24 is shard and its replica is 10.38.33.27. > >> 10.38.33.227 is shard and its replica is 10.38.33.219. > >> > >> I want to create a new collection on the Same. can not change the shard > IP > >> for the new collection. How can I? > >> > >> > >> -- > >> *From:* sudhir kumar > >> *Sent:* Thursday, January 2, 2020 2:01 PM > >> *To:* solr-user@lucene.apache.org > >> *Subject:* Re: Solr 8.3 > >> > >> sample url to create collection: > >> > >> > >> > http//host:8080/solr/admin/collections?action=CREATE=collectionname=2=3=2= > >> host:8080_solr,host:8080_solr,host:8080_solr,host:8080_solr > >> =collectionconfig > >> > >> On Thu, Jan 2, 2020 at 1:56 PM sudhir kumar > >> wrote: > >> > >>> Hey Vishal, > >>> > >>> You can use createNodeSet property while creating collection which will > >>> allows you to create shards on specified IP. > >>> > >>> /admin/collections?action=CREATE=*name*=*number* > >>> > =*number*&*maxShardsPerNode*=*number*&*createNodeSet*= > >>> *nodelist*=*configname* > >>> > >>> > >>> Thanks, > >>> > >>> Sudhir > >>> > >>> > >>> On Thu, Jan 2, 2020 at 1:49 PM vishal patel < > >> vishalpatel200...@outlook.com> > >>> wrote: > >>> > >>>> When I am creating 2 shards and 2 replicas using admin panel, > automatic > >>>> assign a shard or replica to any IP. > >>>> I want to make the specific shard or replica to solr instance at the > >> time > >>>> of creating a collection. Can I? > >>>> > >>>> Regards, > >>>> Vishal > >>>> > >>> > >> > >
Re: Solr 8.3
I do not want to change the IP of the existing replica. I want to fix the IP for the first time creating a collection. I have 4 machines. my IP of each machine is below machine1 10.38.33.28 machine2 10.38.33.29 machine3 10.38.33.30 machine4 10.38.33.31 I have created solr instance on each machine. when I am creating a first collection(documents) using admin panel my structure becomes like this shard1 10.38.33.30 replica1 10.38.33.31 shard2 10.38.33.28 replica2 10.38.33.29 When creating a second collection(forms), my structure becomes like this shard1 10.38.33.28 replica1 10.38.33.30 shard2 10.38.33.29 replica2 10.38.33.31 I have attached a screenshot. [https://ibb.co/Yb2TpVX] Randomly IP assigned to shard or replica when creating collection but I want to make shard1 on 10.38.33.28, replica1 on 10.38.33.31, shard2 on 10.38.33.30 and replica2 on 10.38.33.31 when first time creating a collection using admin panel. Can I Fix the IP to shard OR replica when creating a new [cid:4c560b03-a18d-4bc1-b592-6b2f8103f1ae] collection? Regards, Vishal From: Erick Erickson Sent: Thursday, January 2, 2020 7:40 PM To: solr-user@lucene.apache.org Subject: Re: Solr 8.3 No, you cannot change the IP of an existing replica. Either do as Sankar mentioned when you first create the collection or use the MOVREPLICA collections API command. MOVEREPLICA has existed for quite a long time, but if it’s not available, you can do the same with the ADDREPLICA command to add a replica to a specific node, wait for the replica to become fully active, then use DELETEREPLICA on the one you no longer want. Best, Erick > On Jan 2, 2020, at 8:10 AM, Sankar Panda wrote: > > Hi Vishal, > You can create a empty nodeset and manually configure in the collection as > desired in the admin page > Thanks > Sankar Panda > > On Thu, Jan 2, 2020, 14:36 vishal patel > wrote: > >> My created collection in solr cloud below >> >> 10.38.33.24 is shard and its replica is 10.38.33.27. >> 10.38.33.227 is shard and its replica is 10.38.33.219. >> >> I want to create a new collection on the Same. can not change the shard IP >> for the new collection. How can I? >> >> >> -- >> *From:* sudhir kumar >> *Sent:* Thursday, January 2, 2020 2:01 PM >> *To:* solr-user@lucene.apache.org >> *Subject:* Re: Solr 8.3 >> >> sample url to create collection: >> >> >> http//host:8080/solr/admin/collections?action=CREATE=collectionname=2=3=2= >> host:8080_solr,host:8080_solr,host:8080_solr,host:8080_solr >> =collectionconfig >> >> On Thu, Jan 2, 2020 at 1:56 PM sudhir kumar >> wrote: >> >>> Hey Vishal, >>> >>> You can use createNodeSet property while creating collection which will >>> allows you to create shards on specified IP. >>> >>> /admin/collections?action=CREATE=*name*=*number* >>> =*number*&*maxShardsPerNode*=*number*&*createNodeSet*= >>> *nodelist*=*configname* >>> >>> >>> Thanks, >>> >>> Sudhir >>> >>> >>> On Thu, Jan 2, 2020 at 1:49 PM vishal patel < >> vishalpatel200...@outlook.com> >>> wrote: >>> >>>> When I am creating 2 shards and 2 replicas using admin panel, automatic >>>> assign a shard or replica to any IP. >>>> I want to make the specific shard or replica to solr instance at the >> time >>>> of creating a collection. Can I? >>>> >>>> Regards, >>>> Vishal >>>> >>> >>
Re: Solr 8.3
No, you cannot change the IP of an existing replica. Either do as Sankar mentioned when you first create the collection or use the MOVREPLICA collections API command. MOVEREPLICA has existed for quite a long time, but if it’s not available, you can do the same with the ADDREPLICA command to add a replica to a specific node, wait for the replica to become fully active, then use DELETEREPLICA on the one you no longer want. Best, Erick > On Jan 2, 2020, at 8:10 AM, Sankar Panda wrote: > > Hi Vishal, > You can create a empty nodeset and manually configure in the collection as > desired in the admin page > Thanks > Sankar Panda > > On Thu, Jan 2, 2020, 14:36 vishal patel > wrote: > >> My created collection in solr cloud below >> >> 10.38.33.24 is shard and its replica is 10.38.33.27. >> 10.38.33.227 is shard and its replica is 10.38.33.219. >> >> I want to create a new collection on the Same. can not change the shard IP >> for the new collection. How can I? >> >> >> -- >> *From:* sudhir kumar >> *Sent:* Thursday, January 2, 2020 2:01 PM >> *To:* solr-user@lucene.apache.org >> *Subject:* Re: Solr 8.3 >> >> sample url to create collection: >> >> >> http//host:8080/solr/admin/collections?action=CREATE=collectionname=2=3=2= >> host:8080_solr,host:8080_solr,host:8080_solr,host:8080_solr >> =collectionconfig >> >> On Thu, Jan 2, 2020 at 1:56 PM sudhir kumar >> wrote: >> >>> Hey Vishal, >>> >>> You can use createNodeSet property while creating collection which will >>> allows you to create shards on specified IP. >>> >>> /admin/collections?action=CREATE=*name*=*number* >>> =*number*&*maxShardsPerNode*=*number*&*createNodeSet*= >>> *nodelist*=*configname* >>> >>> >>> Thanks, >>> >>> Sudhir >>> >>> >>> On Thu, Jan 2, 2020 at 1:49 PM vishal patel < >> vishalpatel200...@outlook.com> >>> wrote: >>> >>>> When I am creating 2 shards and 2 replicas using admin panel, automatic >>>> assign a shard or replica to any IP. >>>> I want to make the specific shard or replica to solr instance at the >> time >>>> of creating a collection. Can I? >>>> >>>> Regards, >>>> Vishal >>>> >>> >>
Re: Solr 8.3
Hi Vishal, You can create a empty nodeset and manually configure in the collection as desired in the admin page Thanks Sankar Panda On Thu, Jan 2, 2020, 14:36 vishal patel wrote: > My created collection in solr cloud below > > 10.38.33.24 is shard and its replica is 10.38.33.27. > 10.38.33.227 is shard and its replica is 10.38.33.219. > > I want to create a new collection on the Same. can not change the shard IP > for the new collection. How can I? > > > -- > *From:* sudhir kumar > *Sent:* Thursday, January 2, 2020 2:01 PM > *To:* solr-user@lucene.apache.org > *Subject:* Re: Solr 8.3 > > sample url to create collection: > > > http//host:8080/solr/admin/collections?action=CREATE=collectionname=2=3=2= > host:8080_solr,host:8080_solr,host:8080_solr,host:8080_solr > =collectionconfig > > On Thu, Jan 2, 2020 at 1:56 PM sudhir kumar > wrote: > > > Hey Vishal, > > > > You can use createNodeSet property while creating collection which will > > allows you to create shards on specified IP. > > > > /admin/collections?action=CREATE=*name*=*number* > > =*number*&*maxShardsPerNode*=*number*&*createNodeSet*= > > *nodelist*=*configname* > > > > > > Thanks, > > > > Sudhir > > > > > > On Thu, Jan 2, 2020 at 1:49 PM vishal patel < > vishalpatel200...@outlook.com> > > wrote: > > > >> When I am creating 2 shards and 2 replicas using admin panel, automatic > >> assign a shard or replica to any IP. > >> I want to make the specific shard or replica to solr instance at the > time > >> of creating a collection. Can I? > >> > >> Regards, > >> Vishal > >> > > >
Re: Solr 8.3
My created collection in solr cloud below [cid:4461af25-67be-4647-b9e5-766d3e2a2602] 10.38.33.24 is shard and its replica is 10.38.33.27. 10.38.33.227 is shard and its replica is 10.38.33.219. I want to create a new collection on the Same. can not change the shard IP for the new collection. How can I? From: sudhir kumar Sent: Thursday, January 2, 2020 2:01 PM To: solr-user@lucene.apache.org Subject: Re: Solr 8.3 sample url to create collection: http//host:8080/solr/admin/collections?action=CREATE=collectionname=2=3=2= host:8080_solr,host:8080_solr,host:8080_solr,host:8080_solr =collectionconfig On Thu, Jan 2, 2020 at 1:56 PM sudhir kumar wrote: > Hey Vishal, > > You can use createNodeSet property while creating collection which will > allows you to create shards on specified IP. > > /admin/collections?action=CREATE=*name*=*number* > =*number*&*maxShardsPerNode*=*number*&*createNodeSet*= > *nodelist*=*configname* > > > Thanks, > > Sudhir > > > On Thu, Jan 2, 2020 at 1:49 PM vishal patel > wrote: > >> When I am creating 2 shards and 2 replicas using admin panel, automatic >> assign a shard or replica to any IP. >> I want to make the specific shard or replica to solr instance at the time >> of creating a collection. Can I? >> >> Regards, >> Vishal >> >
Re: Solr 8.3
sample url to create collection: http//host:8080/solr/admin/collections?action=CREATE=collectionname=2=3=2= host:8080_solr,host:8080_solr,host:8080_solr,host:8080_solr =collectionconfig On Thu, Jan 2, 2020 at 1:56 PM sudhir kumar wrote: > Hey Vishal, > > You can use createNodeSet property while creating collection which will > allows you to create shards on specified IP. > > /admin/collections?action=CREATE=*name*=*number* > =*number*&*maxShardsPerNode*=*number*&*createNodeSet*= > *nodelist*=*configname* > > > Thanks, > > Sudhir > > > On Thu, Jan 2, 2020 at 1:49 PM vishal patel > wrote: > >> When I am creating 2 shards and 2 replicas using admin panel, automatic >> assign a shard or replica to any IP. >> I want to make the specific shard or replica to solr instance at the time >> of creating a collection. Can I? >> >> Regards, >> Vishal >> >
Re: Solr 8.3
Hey Vishal, You can use createNodeSet property while creating collection which will allows you to create shards on specified IP. /admin/collections?action=CREATE=*name*=*number* =*number*&*maxShardsPerNode*=*number*&*createNodeSet*= *nodelist*=*configname* Thanks, Sudhir On Thu, Jan 2, 2020 at 1:49 PM vishal patel wrote: > When I am creating 2 shards and 2 replicas using admin panel, automatic > assign a shard or replica to any IP. > I want to make the specific shard or replica to solr instance at the time > of creating a collection. Can I? > > Regards, > Vishal >
Solr 8.3
When I am creating 2 shards and 2 replicas using admin panel, automatic assign a shard or replica to any IP. I want to make the specific shard or replica to solr instance at the time of creating a collection. Can I? Regards, Vishal
Re: solr 8.3 indexing wrong values in some fields
Hi Colvin, I updated my setup to 8.3.1-RC2 and so far it seems to work. I've converted my 7.7 index again and indexed a whole bunch of new doc and haven't detected any memory corruption. Thanks a lot! On Mon, Dec 2, 2019 at 6:40 AM Colvin Cowie wrote: > This sounds like https://issues.apache.org/jira/browse/SOLR-13963 > Solr 8.3.1 is likely to be available soon - RC2 is at > > https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.3.1-RC2-reva3d456fba2cd1b9892defbcf46a0eb4d4bb4d01f/solr/ > Re-index > <https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.3.1-RC2-reva3d456fba2cd1b9892defbcf46a0eb4d4bb4d01f/solr/Re-index> > on it, and see if you still have issues. > > On Sun, 1 Dec 2019 at 17:35, Odysci wrote: > > > Hi, > > I have a solr cloud setup using solr 8.3 and zookeeper, which I recently > > converted from solr 7.7. I converted the index using the index updater > and > > it all went fine. My index has about 40 million docs. > > I used a separate program to check the values of all fields in the solr > > docs, for consistency (e.g., fields which are supposed to have > > only numbers, or only alpha chars, etc.). I ran this program immediately > > after the index updating and it did not detect any problems. > > > > Then I started regular use of the system, indexing new documents, and I > > noticed that some fields were getting the wrong values. For example, a > > field which was supposed to be a string with only digits had a string > > containing parts of another field name. It looked as if memory was > getting > > corrupted. There were no error msgs in the solr logs. > > In other words, solr 8.3 seems to be indexing wrong values in some > fields. > > This happens very few times, but it's happening. > > Has anyone seen this happening? > > Thanks! > > > > Reinaldo > > >
Re: solr 8.3 indexing wrong values in some fields
This sounds like https://issues.apache.org/jira/browse/SOLR-13963 Solr 8.3.1 is likely to be available soon - RC2 is at https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.3.1-RC2-reva3d456fba2cd1b9892defbcf46a0eb4d4bb4d01f/solr/ Re-index on it, and see if you still have issues. On Sun, 1 Dec 2019 at 17:35, Odysci wrote: > Hi, > I have a solr cloud setup using solr 8.3 and zookeeper, which I recently > converted from solr 7.7. I converted the index using the index updater and > it all went fine. My index has about 40 million docs. > I used a separate program to check the values of all fields in the solr > docs, for consistency (e.g., fields which are supposed to have > only numbers, or only alpha chars, etc.). I ran this program immediately > after the index updating and it did not detect any problems. > > Then I started regular use of the system, indexing new documents, and I > noticed that some fields were getting the wrong values. For example, a > field which was supposed to be a string with only digits had a string > containing parts of another field name. It looked as if memory was getting > corrupted. There were no error msgs in the solr logs. > In other words, solr 8.3 seems to be indexing wrong values in some fields. > This happens very few times, but it's happening. > Has anyone seen this happening? > Thanks! > > Reinaldo >
solr 8.3 indexing wrong values in some fields
Hi, I have a solr cloud setup using solr 8.3 and zookeeper, which I recently converted from solr 7.7. I converted the index using the index updater and it all went fine. My index has about 40 million docs. I used a separate program to check the values of all fields in the solr docs, for consistency (e.g., fields which are supposed to have only numbers, or only alpha chars, etc.). I ran this program immediately after the index updating and it did not detect any problems. Then I started regular use of the system, indexing new documents, and I noticed that some fields were getting the wrong values. For example, a field which was supposed to be a string with only digits had a string containing parts of another field name. It looked as if memory was getting corrupted. There were no error msgs in the solr logs. In other words, solr 8.3 seems to be indexing wrong values in some fields. This happens very few times, but it's happening. Has anyone seen this happening? Thanks! Reinaldo
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
Thanks Colvin.
Can you share the details in the ticket?
I plan to debug this today.
It's unlikely to be a synchronization issue because
serialization/deserialization usually happens in single thread.
On Sun, Nov 24, 2019, 4:09 AM Colvin Cowie
wrote:
> https://issues.apache.org/jira/browse/SOLR-13963
>
> I'll see about modifying the test I have to fit in with the existing tests,
> and if there's a better option then open to whatever
>
> On Sat, 23 Nov 2019 at 16:43, Colvin Cowie
> wrote:
>
> > I've found the problem, JavaBinCodec has a CharArr,* arr*, which is
> > modified in two different locations, but only one of which is protected
> > with a synchronized block
> >
> > getStringProvider(), which is used when you call getValue() rather than
> > getRawValue() on the string based SolrInputFields, synchronizes:
> >
> >
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L966
> > but _readStr() doesn't:
> >
> >
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L930
> >
> > Adding a synchronized block into _readStr() fixes the problem. At least
> as
> > far as my test goes.
> >
> > I'll raise a JIRA issue and can provide a patch with the synchronized
> > block, but not sure what test(s) should be updated / added to cover this?
> >
> >
> >
> > On Thu, 21 Nov 2019 at 18:23, Colvin Cowie
> > wrote:
> >
> >> *> the difference is because the _default config has the dynamic schema
> >> building in it, which I assume is pushing it down a different code
> path. *
> >>
> >> Also to add to that, I assumed initially that this just meant that it
> was
> >> working because the corrupted field names would just cause it to create
> a
> >> field with the dodgy name (since that's the idea for the dynamic
> schema),
> >> but checking the documents on retrieval showed they all had the right
> field
> >> names...
> >> So I assume it's a result of going into a different branch of code
> >> instead.
> >>
> >>
> >> On an unrelated matter, I saw this in the logs when running with
> embedded
> >> zookeeper... I don't think I've seen it mentioned anywhere else, so I
> will
> >> raise an issue for it
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> *2019-11-21 17:25:14.292 INFO (main) [ ] o.a.s.c.SolrZkServer
> STARTING
> >> EMBEDDED STANDALONE ZOOKEEPER SERVER at port 99832019-11-21 17:25:14.792
> >> INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper
> >> client=localhost:99832019-11-21 17:25:18.833 WARN (Thread-13) [ ]
> >> o.a.z.s.a.AdminServerFactory Unable to load jetty, not starting
> >> JettyAdminServer => java.lang.NoClassDefFoundError:
> >> org/eclipse/jetty/server/Connector at java.lang.Class.forName0(Native
> >> Method)java.lang.NoClassDefFoundError:
> org/eclipse/jetty/server/Connector
> >> at java.lang.Class.forName0(Native Method) ~[?:1.8.0_191] at
> >> java.lang.Class.forName(Class.java:264) ~[?:1.8.0_191] at
> >>
> org.apache.zookeeper.server.admin.AdminServerFactory.createAdminServer(AdminServerFactory.java:43)
> >> ~[?:?] at
> >>
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:136)
> >> ~[?:?] at
> org.apache.solr.cloud.SolrZkServer$1.run(SolrZkServer.java:121)
> >> ~[?:?]Caused by: java.lang.ClassNotFoundException:
> >> org.eclipse.jetty.server.Connector at
> >>
> org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:577)
> >> ~[jetty-webapp-9.4.19.v20190610.jar:9.4.19.v20190610] at
> >> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_191]
> ... 5
> >> more2019-11-21 17:25:19.365 INFO (main) [ ]
> o.a.s.c.c.ConnectionManager
> >> Waiting for client to connect to ZooKeeper2019-11-21 17:25:19.396 INFO
> >> (zkConnectionManagerCallback-7-thread-1) [ ]
> o.a.s.c.c.ConnectionManager
> >> zkClient has connected2019-11-21 17:25:19.396 INFO (main) [ ]
> >> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper*
> >>
> >> On Thu, 21 Nov 2019 at 17:30, Colvin Cowie
> >> wrote:
> >>
> >>> I've been a bit snowed under, but I've found the difference is because
> >>> the _default config has the dynamic schema building in it, which I
> assume
> >>> is pushing it down a different code path.
> >>>
> >>>>>> default="${update.autoCreateFields:true}"
> >>>
> >>>
> processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
> >>>
> >>> I'm using the vanilla Solr 8.3.0 binary8.3.0
> >>> 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:15:22
> with
> >>> Eclipse OpenJ9 Eclipse OpenJ9 VM 1.8.0_232 openj9-0.17.0
> >>> and I've checked with Oracle Corporation Java HotSpot(TM) 64-Bit Server
> >>> VM 1.8.0_191 25.191-b12 as well
> >>>
> >>> I've put a testcase and configsets in Google Drive:
> >>> https://drive.google.com/open?id=1ibKNWvowT8cXTwSa3bcTwKYLSRNur86U
> >>> The configsets are a copy
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
https://issues.apache.org/jira/browse/SOLR-13963
I'll see about modifying the test I have to fit in with the existing tests,
and if there's a better option then open to whatever
On Sat, 23 Nov 2019 at 16:43, Colvin Cowie
wrote:
> I've found the problem, JavaBinCodec has a CharArr,* arr*, which is
> modified in two different locations, but only one of which is protected
> with a synchronized block
>
> getStringProvider(), which is used when you call getValue() rather than
> getRawValue() on the string based SolrInputFields, synchronizes:
>
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L966
> but _readStr() doesn't:
>
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L930
>
> Adding a synchronized block into _readStr() fixes the problem. At least as
> far as my test goes.
>
> I'll raise a JIRA issue and can provide a patch with the synchronized
> block, but not sure what test(s) should be updated / added to cover this?
>
>
>
> On Thu, 21 Nov 2019 at 18:23, Colvin Cowie
> wrote:
>
>> *> the difference is because the _default config has the dynamic schema
>> building in it, which I assume is pushing it down a different code path. *
>>
>> Also to add to that, I assumed initially that this just meant that it was
>> working because the corrupted field names would just cause it to create a
>> field with the dodgy name (since that's the idea for the dynamic schema),
>> but checking the documents on retrieval showed they all had the right field
>> names...
>> So I assume it's a result of going into a different branch of code
>> instead.
>>
>>
>> On an unrelated matter, I saw this in the logs when running with embedded
>> zookeeper... I don't think I've seen it mentioned anywhere else, so I will
>> raise an issue for it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *2019-11-21 17:25:14.292 INFO (main) [ ] o.a.s.c.SolrZkServer STARTING
>> EMBEDDED STANDALONE ZOOKEEPER SERVER at port 99832019-11-21 17:25:14.792
>> INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper
>> client=localhost:99832019-11-21 17:25:18.833 WARN (Thread-13) [ ]
>> o.a.z.s.a.AdminServerFactory Unable to load jetty, not starting
>> JettyAdminServer => java.lang.NoClassDefFoundError:
>> org/eclipse/jetty/server/Connector at java.lang.Class.forName0(Native
>> Method)java.lang.NoClassDefFoundError: org/eclipse/jetty/server/Connector
>> at java.lang.Class.forName0(Native Method) ~[?:1.8.0_191] at
>> java.lang.Class.forName(Class.java:264) ~[?:1.8.0_191] at
>> org.apache.zookeeper.server.admin.AdminServerFactory.createAdminServer(AdminServerFactory.java:43)
>> ~[?:?] at
>> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:136)
>> ~[?:?] at org.apache.solr.cloud.SolrZkServer$1.run(SolrZkServer.java:121)
>> ~[?:?]Caused by: java.lang.ClassNotFoundException:
>> org.eclipse.jetty.server.Connector at
>> org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:577)
>> ~[jetty-webapp-9.4.19.v20190610.jar:9.4.19.v20190610] at
>> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_191] ... 5
>> more2019-11-21 17:25:19.365 INFO (main) [ ] o.a.s.c.c.ConnectionManager
>> Waiting for client to connect to ZooKeeper2019-11-21 17:25:19.396 INFO
>> (zkConnectionManagerCallback-7-thread-1) [ ] o.a.s.c.c.ConnectionManager
>> zkClient has connected2019-11-21 17:25:19.396 INFO (main) [ ]
>> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper*
>>
>> On Thu, 21 Nov 2019 at 17:30, Colvin Cowie
>> wrote:
>>
>>> I've been a bit snowed under, but I've found the difference is because
>>> the _default config has the dynamic schema building in it, which I assume
>>> is pushing it down a different code path.
>>>
>>> >> default="${update.autoCreateFields:true}"
>>>
>>>
>>> processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
>>>
>>> I'm using the vanilla Solr 8.3.0 binary8.3.0
>>> 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:15:22 with
>>> Eclipse OpenJ9 Eclipse OpenJ9 VM 1.8.0_232 openj9-0.17.0
>>> and I've checked with Oracle Corporation Java HotSpot(TM) 64-Bit Server
>>> VM 1.8.0_191 25.191-b12 as well
>>>
>>> I've put a testcase and configsets in Google Drive:
>>> https://drive.google.com/open?id=1ibKNWvowT8cXTwSa3bcTwKYLSRNur86U
>>> The configsets are a copy of the _default configset, except the
>>> "problem" configset has autoCreateFields set to false.
>>> I created a collection with 4 shards, replication factor 1 for each
>>> configset. The test case reliably fails on the "problem" collection and
>>> reliably passes against the "no_problem" collection.
>>>
>>> The test (well it's not actually a @Test but still) has static data
>>> (though it was originally generated randomly). The data is a bit mad... but
>>> it was easier to reproduce the problem
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
I've found the problem, JavaBinCodec has a CharArr,* arr*, which is
modified in two different locations, but only one of which is protected
with a synchronized block
getStringProvider(), which is used when you call getValue() rather than
getRawValue() on the string based SolrInputFields, synchronizes:
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L966
but _readStr() doesn't:
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L930
Adding a synchronized block into _readStr() fixes the problem. At least as
far as my test goes.
I'll raise a JIRA issue and can provide a patch with the synchronized
block, but not sure what test(s) should be updated / added to cover this?
On Thu, 21 Nov 2019 at 18:23, Colvin Cowie
wrote:
> *> the difference is because the _default config has the dynamic schema
> building in it, which I assume is pushing it down a different code path. *
>
> Also to add to that, I assumed initially that this just meant that it was
> working because the corrupted field names would just cause it to create a
> field with the dodgy name (since that's the idea for the dynamic schema),
> but checking the documents on retrieval showed they all had the right field
> names...
> So I assume it's a result of going into a different branch of code instead.
>
>
> On an unrelated matter, I saw this in the logs when running with embedded
> zookeeper... I don't think I've seen it mentioned anywhere else, so I will
> raise an issue for it
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *2019-11-21 17:25:14.292 INFO (main) [ ] o.a.s.c.SolrZkServer STARTING
> EMBEDDED STANDALONE ZOOKEEPER SERVER at port 99832019-11-21 17:25:14.792
> INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper
> client=localhost:99832019-11-21 17:25:18.833 WARN (Thread-13) [ ]
> o.a.z.s.a.AdminServerFactory Unable to load jetty, not starting
> JettyAdminServer => java.lang.NoClassDefFoundError:
> org/eclipse/jetty/server/Connector at java.lang.Class.forName0(Native
> Method)java.lang.NoClassDefFoundError: org/eclipse/jetty/server/Connector
> at java.lang.Class.forName0(Native Method) ~[?:1.8.0_191] at
> java.lang.Class.forName(Class.java:264) ~[?:1.8.0_191] at
> org.apache.zookeeper.server.admin.AdminServerFactory.createAdminServer(AdminServerFactory.java:43)
> ~[?:?] at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:136)
> ~[?:?] at org.apache.solr.cloud.SolrZkServer$1.run(SolrZkServer.java:121)
> ~[?:?]Caused by: java.lang.ClassNotFoundException:
> org.eclipse.jetty.server.Connector at
> org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:577)
> ~[jetty-webapp-9.4.19.v20190610.jar:9.4.19.v20190610] at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_191] ... 5
> more2019-11-21 17:25:19.365 INFO (main) [ ] o.a.s.c.c.ConnectionManager
> Waiting for client to connect to ZooKeeper2019-11-21 17:25:19.396 INFO
> (zkConnectionManagerCallback-7-thread-1) [ ] o.a.s.c.c.ConnectionManager
> zkClient has connected2019-11-21 17:25:19.396 INFO (main) [ ]
> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper*
>
> On Thu, 21 Nov 2019 at 17:30, Colvin Cowie
> wrote:
>
>> I've been a bit snowed under, but I've found the difference is because
>> the _default config has the dynamic schema building in it, which I assume
>> is pushing it down a different code path.
>>
>> > default="${update.autoCreateFields:true}"
>>
>>
>> processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
>>
>> I'm using the vanilla Solr 8.3.0 binary8.3.0
>> 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:15:22 with
>> Eclipse OpenJ9 Eclipse OpenJ9 VM 1.8.0_232 openj9-0.17.0
>> and I've checked with Oracle Corporation Java HotSpot(TM) 64-Bit Server
>> VM 1.8.0_191 25.191-b12 as well
>>
>> I've put a testcase and configsets in Google Drive:
>> https://drive.google.com/open?id=1ibKNWvowT8cXTwSa3bcTwKYLSRNur86U
>> The configsets are a copy of the _default configset, except the "problem"
>> configset has autoCreateFields set to false.
>> I created a collection with 4 shards, replication factor 1 for each
>> configset. The test case reliably fails on the "problem" collection and
>> reliably passes against the "no_problem" collection.
>>
>> The test (well it's not actually a @Test but still) has static data
>> (though it was originally generated randomly). The data is a bit mad... but
>> it was easier to reproduce the problem reliably with this data, than with
>> the normal documents we use in our product.
>> Each document has a different (dynamically named) field to index data
>> into, but it's the same data in each field.
>> The problem only appears (or probably is just more likely to appear?)
>> when the field names in the request are of different lengths.
>> The length / value of
Re: Possible bug in cluster status - > solr 8.3
AFAIK these collection properties are not tracked that faithfully and can get
out of sync, mostly because they are used only during collection CREATE and
BACKUP / RESTORE and not during other collection operations or during searching
/ indexing. SPLITSHARD doesn’t trust them, instead it checks the actual counts
of existing replicas.
These out-of-sync counts may actually cause problems in BACKUP / RESTORE, which
is worth checking.
There are also conceptual issues here, eg. “replicationFactor” becomes
meaningless as soon as we have different counts of NRT / TLOG / PULL replicas.
> On 21 Nov 2019, at 13:40, Jason Gerlowski wrote:
>
> It seems like an issue to me. Can you open a JIRA with these details?
>
> On Fri, Nov 15, 2019 at 10:51 AM Jacek Kikiewicz wrote:
>>
>> I found interesting situation, I've created a collection with only one
>> replica.
>> Then I scaled solr-cloud cluster, and run 'addreplica' call to add 2 more.
>> So I have a collection with 3 tlog replicas, cluster status page shows
>> them but shows also this:
>> "core_node2":{
>>"core":"EDITED_NAME_shard1_replica_t1",
>>"base_url":"http://EDITED_NODE:8983/solr;,
>>"node_name":"EDITED_NODE:8983_solr",
>>"state":"active",
>>"type":"TLOG",
>>"force_set_state":"false",
>>"leader":"true"},
>> "core_node5":{
>>"core":"EDITED_NAME_shard1_replica_t3",
>>"base_url":"http://EDITED_NODE:8983/solr;,
>>"node_name":"EDITED_NODE:8983_solr",
>>"state":"active",
>>"type":"TLOG",
>>"force_set_state":"false"},
>> "core_node6":{
>>"core":"EDITED_NAME_shard1_replica_t4",
>>"base_url":"http://EDITED_NODE:8983/solr;,
>>"node_name":"EDITED_NODE:8983_solr",
>>"state":"active",
>>"type":"TLOG",
>>"force_set_state":"false",
>>"router":{"name":"compositeId"},
>>"maxShardsPerNode":"1",
>>"autoAddReplicas":"false",
>>"nrtReplicas":"1",
>>"tlogReplicas":"1",
>>"znodeVersion":11,
>>
>>
>> As you can see I have 3 replicas but then I have also: "tlogReplicas":"1"
>>
>> If I create collection with tlogReplicas=3 then cluster status shows
>> "tlogReplicas":"3"
>> IS that a bug or somehow 'works as it should' ?
>>
>> Regards,
>> Jacek
>
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
*> the difference is because the _default config has the dynamic schema
building in it, which I assume is pushing it down a different code path. *
Also to add to that, I assumed initially that this just meant that it was
working because the corrupted field names would just cause it to create a
field with the dodgy name (since that's the idea for the dynamic schema),
but checking the documents on retrieval showed they all had the right field
names...
So I assume it's a result of going into a different branch of code instead.
On an unrelated matter, I saw this in the logs when running with embedded
zookeeper... I don't think I've seen it mentioned anywhere else, so I will
raise an issue for it
*2019-11-21 17:25:14.292 INFO (main) [ ] o.a.s.c.SolrZkServer STARTING
EMBEDDED STANDALONE ZOOKEEPER SERVER at port 99832019-11-21 17:25:14.792
INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper
client=localhost:99832019-11-21 17:25:18.833 WARN (Thread-13) [ ]
o.a.z.s.a.AdminServerFactory Unable to load jetty, not starting
JettyAdminServer => java.lang.NoClassDefFoundError:
org/eclipse/jetty/server/Connector at java.lang.Class.forName0(Native
Method)java.lang.NoClassDefFoundError: org/eclipse/jetty/server/Connector
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_191] at
java.lang.Class.forName(Class.java:264) ~[?:1.8.0_191] at
org.apache.zookeeper.server.admin.AdminServerFactory.createAdminServer(AdminServerFactory.java:43)
~[?:?] at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:136)
~[?:?] at org.apache.solr.cloud.SolrZkServer$1.run(SolrZkServer.java:121)
~[?:?]Caused by: java.lang.ClassNotFoundException:
org.eclipse.jetty.server.Connector at
org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:577)
~[jetty-webapp-9.4.19.v20190610.jar:9.4.19.v20190610] at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_191] ... 5
more2019-11-21 17:25:19.365 INFO (main) [ ] o.a.s.c.c.ConnectionManager
Waiting for client to connect to ZooKeeper2019-11-21 17:25:19.396 INFO
(zkConnectionManagerCallback-7-thread-1) [ ] o.a.s.c.c.ConnectionManager
zkClient has connected2019-11-21 17:25:19.396 INFO (main) [ ]
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper*
On Thu, 21 Nov 2019 at 17:30, Colvin Cowie
wrote:
> I've been a bit snowed under, but I've found the difference is because the
> _default config has the dynamic schema building in it, which I assume is
> pushing it down a different code path.
>
>default="${update.autoCreateFields:true}"
>
>
> processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
>
> I'm using the vanilla Solr 8.3.0 binary8.3.0
> 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:15:22 with
> Eclipse OpenJ9 Eclipse OpenJ9 VM 1.8.0_232 openj9-0.17.0
> and I've checked with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM
> 1.8.0_191 25.191-b12 as well
>
> I've put a testcase and configsets in Google Drive:
> https://drive.google.com/open?id=1ibKNWvowT8cXTwSa3bcTwKYLSRNur86U
> The configsets are a copy of the _default configset, except the "problem"
> configset has autoCreateFields set to false.
> I created a collection with 4 shards, replication factor 1 for each
> configset. The test case reliably fails on the "problem" collection and
> reliably passes against the "no_problem" collection.
>
> The test (well it's not actually a @Test but still) has static data
> (though it was originally generated randomly). The data is a bit mad... but
> it was easier to reproduce the problem reliably with this data, than with
> the normal documents we use in our product.
> Each document has a different (dynamically named) field to index data
> into, but it's the same data in each field.
> The problem only appears (or probably is just more likely to appear?) when
> the field names in the request are of different lengths.
> The length / value of the data doesn't appear to matter. Or is less
> impactful than variations in the field names.
> *If you run the test 10 times you will see a variety of different errors.
> i.e. it's not the same error every time.*
> I've included some examples of the errors in the Drive folder. One of the
> most fundamental (and probably points at the root cause) is this:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *2019-11-21 17:02:53.720 ERROR
> (updateExecutor-3-thread-6-processing-x:problem_collection_shard2_replica_n2
> r:core_node5 null n:10.0.75.1:8983_solr c:problem_collection s:shard2)
> [c:problem_collection s:shard2 r:core_node5
> x:problem_collection_shard2_replica_n2]
> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> SolrCmdDistributor$Req: cmd=add{,id=(null)}; node=ForwardNode:
> http://10.0.75.1:8983/solr/problem_collection_shard3_replica_n4/
>
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
I've been a bit snowed under, but I've found the difference is because the
_default config has the dynamic schema building in it, which I assume is
pushing it down a different code path.
I'm using the vanilla Solr 8.3.0 binary8.3.0
2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:15:22 with
Eclipse OpenJ9 Eclipse OpenJ9 VM 1.8.0_232 openj9-0.17.0
and I've checked with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM
1.8.0_191 25.191-b12 as well
I've put a testcase and configsets in Google Drive:
https://drive.google.com/open?id=1ibKNWvowT8cXTwSa3bcTwKYLSRNur86U
The configsets are a copy of the _default configset, except the "problem"
configset has autoCreateFields set to false.
I created a collection with 4 shards, replication factor 1 for each
configset. The test case reliably fails on the "problem" collection and
reliably passes against the "no_problem" collection.
The test (well it's not actually a @Test but still) has static data (though
it was originally generated randomly). The data is a bit mad... but it was
easier to reproduce the problem reliably with this data, than with the
normal documents we use in our product.
Each document has a different (dynamically named) field to index data into,
but it's the same data in each field.
The problem only appears (or probably is just more likely to appear?) when
the field names in the request are of different lengths.
The length / value of the data doesn't appear to matter. Or is less
impactful than variations in the field names.
*If you run the test 10 times you will see a variety of different errors.
i.e. it's not the same error every time.*
I've included some examples of the errors in the Drive folder. One of the
most fundamental (and probably points at the root cause) is this:
*2019-11-21 17:02:53.720 ERROR
(updateExecutor-3-thread-6-processing-x:problem_collection_shard2_replica_n2
r:core_node5 null n:10.0.75.1:8983_solr c:problem_collection s:shard2)
[c:problem_collection s:shard2 r:core_node5
x:problem_collection_shard2_replica_n2]
o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
SolrCmdDistributor$Req: cmd=add{,id=(null)}; node=ForwardNode:
http://10.0.75.1:8983/solr/problem_collection_shard3_replica_n4/
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
Very curious what the config change that's related to reproducing this looks like. Maybe it's something that is worth adding test-randomization around? Just thinking aloud.
Re: Possible bug in cluster status - > solr 8.3
It seems like an issue to me. Can you open a JIRA with these details?
On Fri, Nov 15, 2019 at 10:51 AM Jacek Kikiewicz wrote:
>
> I found interesting situation, I've created a collection with only one
> replica.
> Then I scaled solr-cloud cluster, and run 'addreplica' call to add 2 more.
> So I have a collection with 3 tlog replicas, cluster status page shows
> them but shows also this:
> "core_node2":{
> "core":"EDITED_NAME_shard1_replica_t1",
> "base_url":"http://EDITED_NODE:8983/solr;,
> "node_name":"EDITED_NODE:8983_solr",
> "state":"active",
> "type":"TLOG",
> "force_set_state":"false",
> "leader":"true"},
> "core_node5":{
> "core":"EDITED_NAME_shard1_replica_t3",
> "base_url":"http://EDITED_NODE:8983/solr;,
> "node_name":"EDITED_NODE:8983_solr",
> "state":"active",
> "type":"TLOG",
> "force_set_state":"false"},
> "core_node6":{
> "core":"EDITED_NAME_shard1_replica_t4",
> "base_url":"http://EDITED_NODE:8983/solr;,
> "node_name":"EDITED_NODE:8983_solr",
> "state":"active",
> "type":"TLOG",
> "force_set_state":"false",
> "router":{"name":"compositeId"},
> "maxShardsPerNode":"1",
> "autoAddReplicas":"false",
> "nrtReplicas":"1",
> "tlogReplicas":"1",
> "znodeVersion":11,
>
>
> As you can see I have 3 replicas but then I have also: "tlogReplicas":"1"
>
> If I create collection with tlogReplicas=3 then cluster status shows
> "tlogReplicas":"3"
> IS that a bug or somehow 'works as it should' ?
>
> Regards,
> Jacek
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
Sure, looking forward to that
thnaks
On Thu, Nov 21, 2019 at 8:06 AM Colvin Cowie wrote:
>
> Hi, I'll share it when I'm back at work tomorrow.
>
> I've found that the issue appears to not be reproducible using the
> solrconfig.xml in the _default example, so it must be something in ours (or
> something missing from it) that is making the problem appear, in
> combination with the code change.
>
> Thanks
>
>
> On Wednesday, 20 November 2019, Noble Paul wrote:
>
> > Can you share the test please
> >
> > On Thu, Nov 21, 2019 at 7:02 AM Noble Paul wrote:
> > >
> > > Thanks Colvin, I'll take a look
> > >
> > > On Thu, Nov 21, 2019 at 4:24 AM Colvin Cowie
> > wrote:
> > > >
> > > > I've identified the change which has caused the problem to
> > materialize, but
> > > > it shouldn't itself cause a problem.
> > > >
> > > > https://github.com/apache/lucene-solr/commit/
> > e45e8127d5c17af4e4b87a0a4eaf0afaf4f9ff4b#diff-
> > 7f7f485122d8257bd5d3210c092b967fR52
> > > > for https://issues.apache.org/jira/browse/SOLR-13682
> > > >
> > > > In writeMap, the new BiConsumer unwraps the SolrInputField using
> > getValue
> > > > rather than getRawValue (which the JavaBinCodec calls):
> > > >
> > > >
> > > > * if (o instanceof SolrInputField) {o = ((SolrInputField)
> > > > o).getValue(); }*
> > > > As a result the JavaBinCodec will now be hitting different writer
> > methods
> > > > based on the value retrieved from the SolrInputField, rather than just
> > > > writing the org.apache.solr.common.util.JavaBinCodec.writeKnownType(
> > Object)
> > > >
> > > >
> > > > *if (val instanceof SolrInputField) { return
> > > > writeKnownType(((SolrInputField) val).getRawValue());}*
> > > > https://github.com/apache/lucene-solr/blob/branch_8_3/
> > solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L362
> > > >
> > > > SolrInputField getValue uses
> > > > org.apache.solr.common.util.ByteArrayUtf8CharSequence.
> > convertCharSeq(Object)
> > > > while getRawValue just returns whatever value the SolrInputField has,
> > so
> > > > the EntryWriter in the JavaBinCodec hits different paths from the ones
> > > > which must non-deterministically produce garbage data when getValue()
> > is
> > > > used.
> > > >
> > > > Changing *getValue()* to *getRawValue()* in the SolrInputDocument's
> > > > *writeMap()* appears to "fix" the problem. (With getValue() the test I
> > have
> > > > reliably fails within 50 iterations of indexing 2500 documents, with
> > > > getRawValue() it succeeds for the 500 iterations I'm running it for)
> > > >
> > > > I'll see about providing a test that can be shared that demonstrates
> > the
> > > > problem, and see if we can find what is going wrong in the codec...
> > > >
> > > >
> > > > On Tue, 19 Nov 2019 at 13:48, Colvin Cowie > >
> > > > wrote:
> > > >
> > > > > Hello
> > > > >
> > > > > Apologies for the lack of actual detail in this, we're still digging
> > into
> > > > > it ourselves. I will provide more detail, and maybe some logs, once
> > I have
> > > > > a better idea of what is actually happening.
> > > > > But I thought I might as well ask if anyone knows of changes that
> > were
> > > > > made in the Solr 8.3 release that are likely to have caused an issue
> > like
> > > > > this?
> > > > >
> > > > > We were on Solr 8.1.1 for several months and moved to 8.2.0 for
> > about 2
> > > > > weeks before moving to 8.3.0 last week.
> > > > > We didn't see this issue at all on the previous releases. Since
> > moving to
> > > > > 8.3 we have had a consistent (but non-deterministic) set of failing
> > tests,
> > > > > on Windows and Linux.
> > > > >
> > > > > The issue we are seeing as that during updates, the data we have
> > sent is
> > > > > *sometimes* corrupted, as though a buffer has been used incorrectly.
> > For
> > > > > example if the well formed data went was
> > > > > *'fieldName':"this is a long string"*
> > > > > The error we see from
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
Hi, I'll share it when I'm back at work tomorrow.
I've found that the issue appears to not be reproducible using the
solrconfig.xml in the _default example, so it must be something in ours (or
something missing from it) that is making the problem appear, in
combination with the code change.
Thanks
On Wednesday, 20 November 2019, Noble Paul wrote:
> Can you share the test please
>
> On Thu, Nov 21, 2019 at 7:02 AM Noble Paul wrote:
> >
> > Thanks Colvin, I'll take a look
> >
> > On Thu, Nov 21, 2019 at 4:24 AM Colvin Cowie
> wrote:
> > >
> > > I've identified the change which has caused the problem to
> materialize, but
> > > it shouldn't itself cause a problem.
> > >
> > > https://github.com/apache/lucene-solr/commit/
> e45e8127d5c17af4e4b87a0a4eaf0afaf4f9ff4b#diff-
> 7f7f485122d8257bd5d3210c092b967fR52
> > > for https://issues.apache.org/jira/browse/SOLR-13682
> > >
> > > In writeMap, the new BiConsumer unwraps the SolrInputField using
> getValue
> > > rather than getRawValue (which the JavaBinCodec calls):
> > >
> > >
> > > * if (o instanceof SolrInputField) {o = ((SolrInputField)
> > > o).getValue(); }*
> > > As a result the JavaBinCodec will now be hitting different writer
> methods
> > > based on the value retrieved from the SolrInputField, rather than just
> > > writing the org.apache.solr.common.util.JavaBinCodec.writeKnownType(
> Object)
> > >
> > >
> > > *if (val instanceof SolrInputField) { return
> > > writeKnownType(((SolrInputField) val).getRawValue());}*
> > > https://github.com/apache/lucene-solr/blob/branch_8_3/
> solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L362
> > >
> > > SolrInputField getValue uses
> > > org.apache.solr.common.util.ByteArrayUtf8CharSequence.
> convertCharSeq(Object)
> > > while getRawValue just returns whatever value the SolrInputField has,
> so
> > > the EntryWriter in the JavaBinCodec hits different paths from the ones
> > > which must non-deterministically produce garbage data when getValue()
> is
> > > used.
> > >
> > > Changing *getValue()* to *getRawValue()* in the SolrInputDocument's
> > > *writeMap()* appears to "fix" the problem. (With getValue() the test I
> have
> > > reliably fails within 50 iterations of indexing 2500 documents, with
> > > getRawValue() it succeeds for the 500 iterations I'm running it for)
> > >
> > > I'll see about providing a test that can be shared that demonstrates
> the
> > > problem, and see if we can find what is going wrong in the codec...
> > >
> > >
> > > On Tue, 19 Nov 2019 at 13:48, Colvin Cowie >
> > > wrote:
> > >
> > > > Hello
> > > >
> > > > Apologies for the lack of actual detail in this, we're still digging
> into
> > > > it ourselves. I will provide more detail, and maybe some logs, once
> I have
> > > > a better idea of what is actually happening.
> > > > But I thought I might as well ask if anyone knows of changes that
> were
> > > > made in the Solr 8.3 release that are likely to have caused an issue
> like
> > > > this?
> > > >
> > > > We were on Solr 8.1.1 for several months and moved to 8.2.0 for
> about 2
> > > > weeks before moving to 8.3.0 last week.
> > > > We didn't see this issue at all on the previous releases. Since
> moving to
> > > > 8.3 we have had a consistent (but non-deterministic) set of failing
> tests,
> > > > on Windows and Linux.
> > > >
> > > > The issue we are seeing as that during updates, the data we have
> sent is
> > > > *sometimes* corrupted, as though a buffer has been used incorrectly.
> For
> > > > example if the well formed data went was
> > > > *'fieldName':"this is a long string"*
> > > > The error we see from Solr might be that
> > > > unknown field * 'fieldNamis a long string" *
> > > >
> > > > And variations of that kind of behaviour, were part of the data is
> missing
> > > > or corrupted. The data we are indexing does include fields which
> store
> > > > (escaped) serialized JSON strings - if that might have any bearing -
> but
> > > > the error isn't always on those fields.
> > > > For example, given a valid document that looks like this (I've
> replaced
> > > > the values by hand, so if
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
Can you share the test please
On Thu, Nov 21, 2019 at 7:02 AM Noble Paul wrote:
>
> Thanks Colvin, I'll take a look
>
> On Thu, Nov 21, 2019 at 4:24 AM Colvin Cowie
> wrote:
> >
> > I've identified the change which has caused the problem to materialize, but
> > it shouldn't itself cause a problem.
> >
> > https://github.com/apache/lucene-solr/commit/e45e8127d5c17af4e4b87a0a4eaf0afaf4f9ff4b#diff-7f7f485122d8257bd5d3210c092b967fR52
> > for https://issues.apache.org/jira/browse/SOLR-13682
> >
> > In writeMap, the new BiConsumer unwraps the SolrInputField using getValue
> > rather than getRawValue (which the JavaBinCodec calls):
> >
> >
> > * if (o instanceof SolrInputField) {o = ((SolrInputField)
> > o).getValue(); }*
> > As a result the JavaBinCodec will now be hitting different writer methods
> > based on the value retrieved from the SolrInputField, rather than just
> > writing the org.apache.solr.common.util.JavaBinCodec.writeKnownType(Object)
> >
> >
> > *if (val instanceof SolrInputField) { return
> > writeKnownType(((SolrInputField) val).getRawValue());}*
> > https://github.com/apache/lucene-solr/blob/branch_8_3/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L362
> >
> > SolrInputField getValue uses
> > org.apache.solr.common.util.ByteArrayUtf8CharSequence.convertCharSeq(Object)
> > while getRawValue just returns whatever value the SolrInputField has, so
> > the EntryWriter in the JavaBinCodec hits different paths from the ones
> > which must non-deterministically produce garbage data when getValue() is
> > used.
> >
> > Changing *getValue()* to *getRawValue()* in the SolrInputDocument's
> > *writeMap()* appears to "fix" the problem. (With getValue() the test I have
> > reliably fails within 50 iterations of indexing 2500 documents, with
> > getRawValue() it succeeds for the 500 iterations I'm running it for)
> >
> > I'll see about providing a test that can be shared that demonstrates the
> > problem, and see if we can find what is going wrong in the codec...
> >
> >
> > On Tue, 19 Nov 2019 at 13:48, Colvin Cowie
> > wrote:
> >
> > > Hello
> > >
> > > Apologies for the lack of actual detail in this, we're still digging into
> > > it ourselves. I will provide more detail, and maybe some logs, once I have
> > > a better idea of what is actually happening.
> > > But I thought I might as well ask if anyone knows of changes that were
> > > made in the Solr 8.3 release that are likely to have caused an issue like
> > > this?
> > >
> > > We were on Solr 8.1.1 for several months and moved to 8.2.0 for about 2
> > > weeks before moving to 8.3.0 last week.
> > > We didn't see this issue at all on the previous releases. Since moving to
> > > 8.3 we have had a consistent (but non-deterministic) set of failing tests,
> > > on Windows and Linux.
> > >
> > > The issue we are seeing as that during updates, the data we have sent is
> > > *sometimes* corrupted, as though a buffer has been used incorrectly. For
> > > example if the well formed data went was
> > > *'fieldName':"this is a long string"*
> > > The error we see from Solr might be that
> > > unknown field * 'fieldNamis a long string" *
> > >
> > > And variations of that kind of behaviour, were part of the data is missing
> > > or corrupted. The data we are indexing does include fields which store
> > > (escaped) serialized JSON strings - if that might have any bearing - but
> > > the error isn't always on those fields.
> > > For example, given a valid document that looks like this (I've replaced
> > > the values by hand, so if the json is messed up here, that's not
> > > relevant:)
> > > when returned with the json response writer:
> > >
> > >
> > >
> > >
> > > *{"id": "abcd","testField": "blah","jsonField":
> > > "{\"thing\":{\"abcd\":\"value\",\"xyz\":[\"abc\",\"def\",\"ghi\"],\"nnn\":\"xyz\"},\"stuff\":[{\"qqq\":\"rrr\"}],\"ttt\":0,\"mmm\":\"Some
> > > string\",\"someBool\":true}"}*
> > > We've had errors during indexing like:
> > > *unknown field
> > > 'testField:"value","xyz":["abc","def","ghi&q
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
Thanks Colvin, I'll take a look
On Thu, Nov 21, 2019 at 4:24 AM Colvin Cowie wrote:
>
> I've identified the change which has caused the problem to materialize, but
> it shouldn't itself cause a problem.
>
> https://github.com/apache/lucene-solr/commit/e45e8127d5c17af4e4b87a0a4eaf0afaf4f9ff4b#diff-7f7f485122d8257bd5d3210c092b967fR52
> for https://issues.apache.org/jira/browse/SOLR-13682
>
> In writeMap, the new BiConsumer unwraps the SolrInputField using getValue
> rather than getRawValue (which the JavaBinCodec calls):
>
>
> * if (o instanceof SolrInputField) {o = ((SolrInputField)
> o).getValue(); }*
> As a result the JavaBinCodec will now be hitting different writer methods
> based on the value retrieved from the SolrInputField, rather than just
> writing the org.apache.solr.common.util.JavaBinCodec.writeKnownType(Object)
>
>
> *if (val instanceof SolrInputField) { return
> writeKnownType(((SolrInputField) val).getRawValue());}*
> https://github.com/apache/lucene-solr/blob/branch_8_3/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L362
>
> SolrInputField getValue uses
> org.apache.solr.common.util.ByteArrayUtf8CharSequence.convertCharSeq(Object)
> while getRawValue just returns whatever value the SolrInputField has, so
> the EntryWriter in the JavaBinCodec hits different paths from the ones
> which must non-deterministically produce garbage data when getValue() is
> used.
>
> Changing *getValue()* to *getRawValue()* in the SolrInputDocument's
> *writeMap()* appears to "fix" the problem. (With getValue() the test I have
> reliably fails within 50 iterations of indexing 2500 documents, with
> getRawValue() it succeeds for the 500 iterations I'm running it for)
>
> I'll see about providing a test that can be shared that demonstrates the
> problem, and see if we can find what is going wrong in the codec...
>
>
> On Tue, 19 Nov 2019 at 13:48, Colvin Cowie
> wrote:
>
> > Hello
> >
> > Apologies for the lack of actual detail in this, we're still digging into
> > it ourselves. I will provide more detail, and maybe some logs, once I have
> > a better idea of what is actually happening.
> > But I thought I might as well ask if anyone knows of changes that were
> > made in the Solr 8.3 release that are likely to have caused an issue like
> > this?
> >
> > We were on Solr 8.1.1 for several months and moved to 8.2.0 for about 2
> > weeks before moving to 8.3.0 last week.
> > We didn't see this issue at all on the previous releases. Since moving to
> > 8.3 we have had a consistent (but non-deterministic) set of failing tests,
> > on Windows and Linux.
> >
> > The issue we are seeing as that during updates, the data we have sent is
> > *sometimes* corrupted, as though a buffer has been used incorrectly. For
> > example if the well formed data went was
> > *'fieldName':"this is a long string"*
> > The error we see from Solr might be that
> > unknown field * 'fieldNamis a long string" *
> >
> > And variations of that kind of behaviour, were part of the data is missing
> > or corrupted. The data we are indexing does include fields which store
> > (escaped) serialized JSON strings - if that might have any bearing - but
> > the error isn't always on those fields.
> > For example, given a valid document that looks like this (I've replaced
> > the values by hand, so if the json is messed up here, that's not relevant:)
> > when returned with the json response writer:
> >
> >
> >
> >
> > *{"id": "abcd","testField": "blah","jsonField":
> > "{\"thing\":{\"abcd\":\"value\",\"xyz\":[\"abc\",\"def\",\"ghi\"],\"nnn\":\"xyz\"},\"stuff\":[{\"qqq\":\"rrr\"}],\"ttt\":0,\"mmm\":\"Some
> > string\",\"someBool\":true}"}*
> > We've had errors during indexing like:
> > *unknown field
> > 'testField:"value","xyz":["abc","def","ghi"],"nnn":"xyz"},"stuff":[{"qqq":"rrr"}],"ttt":0,"mmm":"Some
> > string","someBool":true}���'*
> > (those � unprintable characters are part of it)
> >
> > So far we've not been able to reproduce the problem on a collection with a
> > single shard, so it does seem like the problem is only happening internally
> > when updates are distributed to
Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
I've identified the change which has caused the problem to materialize, but
it shouldn't itself cause a problem.
https://github.com/apache/lucene-solr/commit/e45e8127d5c17af4e4b87a0a4eaf0afaf4f9ff4b#diff-7f7f485122d8257bd5d3210c092b967fR52
for https://issues.apache.org/jira/browse/SOLR-13682
In writeMap, the new BiConsumer unwraps the SolrInputField using getValue
rather than getRawValue (which the JavaBinCodec calls):
* if (o instanceof SolrInputField) {o = ((SolrInputField)
o).getValue(); }*
As a result the JavaBinCodec will now be hitting different writer methods
based on the value retrieved from the SolrInputField, rather than just
writing the org.apache.solr.common.util.JavaBinCodec.writeKnownType(Object)
*if (val instanceof SolrInputField) { return
writeKnownType(((SolrInputField) val).getRawValue());}*
https://github.com/apache/lucene-solr/blob/branch_8_3/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L362
SolrInputField getValue uses
org.apache.solr.common.util.ByteArrayUtf8CharSequence.convertCharSeq(Object)
while getRawValue just returns whatever value the SolrInputField has, so
the EntryWriter in the JavaBinCodec hits different paths from the ones
which must non-deterministically produce garbage data when getValue() is
used.
Changing *getValue()* to *getRawValue()* in the SolrInputDocument's
*writeMap()* appears to "fix" the problem. (With getValue() the test I have
reliably fails within 50 iterations of indexing 2500 documents, with
getRawValue() it succeeds for the 500 iterations I'm running it for)
I'll see about providing a test that can be shared that demonstrates the
problem, and see if we can find what is going wrong in the codec...
On Tue, 19 Nov 2019 at 13:48, Colvin Cowie
wrote:
> Hello
>
> Apologies for the lack of actual detail in this, we're still digging into
> it ourselves. I will provide more detail, and maybe some logs, once I have
> a better idea of what is actually happening.
> But I thought I might as well ask if anyone knows of changes that were
> made in the Solr 8.3 release that are likely to have caused an issue like
> this?
>
> We were on Solr 8.1.1 for several months and moved to 8.2.0 for about 2
> weeks before moving to 8.3.0 last week.
> We didn't see this issue at all on the previous releases. Since moving to
> 8.3 we have had a consistent (but non-deterministic) set of failing tests,
> on Windows and Linux.
>
> The issue we are seeing as that during updates, the data we have sent is
> *sometimes* corrupted, as though a buffer has been used incorrectly. For
> example if the well formed data went was
> *'fieldName':"this is a long string"*
> The error we see from Solr might be that
> unknown field * 'fieldNamis a long string" *
>
> And variations of that kind of behaviour, were part of the data is missing
> or corrupted. The data we are indexing does include fields which store
> (escaped) serialized JSON strings - if that might have any bearing - but
> the error isn't always on those fields.
> For example, given a valid document that looks like this (I've replaced
> the values by hand, so if the json is messed up here, that's not relevant:)
> when returned with the json response writer:
>
>
>
>
> *{"id": "abcd","testField": "blah","jsonField":
> "{\"thing\":{\"abcd\":\"value\",\"xyz\":[\"abc\",\"def\",\"ghi\"],\"nnn\":\"xyz\"},\"stuff\":[{\"qqq\":\"rrr\"}],\"ttt\":0,\"mmm\":\"Some
> string\",\"someBool\":true}"}*
> We've had errors during indexing like:
> *unknown field
> 'testField:"value","xyz":["abc","def","ghi"],"nnn":"xyz"},"stuff":[{"qqq":"rrr"}],"ttt":0,"mmm":"Some
> string","someBool":true}���'*
> (those � unprintable characters are part of it)
>
> So far we've not been able to reproduce the problem on a collection with a
> single shard, so it does seem like the problem is only happening internally
> when updates are distributed to the other shards... But that's not been
> totally verified.
>
> We've also only encountered the problem on one of the collections we build
> (the data within each collection is generally the same though. The ids are
> slightly different - but still strings. The main difference is that this
> problematic index is built using an Iterator to *solrj
> org.apache.solr.client.solrj.SolrClient.add(String,
> Iterator)* - the *SolrInputDocument*s are not being
> reused in the client, I checked that -, while the other index is built by
> streaming CSVs to Solr.)
>
>
> We will look into it further, but if anyone has any ideas of what might
> have changed in 8.3 from 8.1 / 8.2 that could cause this, that would be
> helpful.
>
> Cheers
> Colvin
>
>
Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?
Hello
Apologies for the lack of actual detail in this, we're still digging into
it ourselves. I will provide more detail, and maybe some logs, once I have
a better idea of what is actually happening.
But I thought I might as well ask if anyone knows of changes that were made
in the Solr 8.3 release that are likely to have caused an issue like this?
We were on Solr 8.1.1 for several months and moved to 8.2.0 for about 2
weeks before moving to 8.3.0 last week.
We didn't see this issue at all on the previous releases. Since moving to
8.3 we have had a consistent (but non-deterministic) set of failing tests,
on Windows and Linux.
The issue we are seeing as that during updates, the data we have sent is
*sometimes* corrupted, as though a buffer has been used incorrectly. For
example if the well formed data went was
*'fieldName':"this is a long string"*
The error we see from Solr might be that
unknown field * 'fieldNamis a long string" *
And variations of that kind of behaviour, were part of the data is missing
or corrupted. The data we are indexing does include fields which store
(escaped) serialized JSON strings - if that might have any bearing - but
the error isn't always on those fields.
For example, given a valid document that looks like this (I've replaced the
values by hand, so if the json is messed up here, that's not relevant:)
when returned with the json response writer:
*{"id": "abcd","testField": "blah","jsonField":
"{\"thing\":{\"abcd\":\"value\",\"xyz\":[\"abc\",\"def\",\"ghi\"],\"nnn\":\"xyz\"},\"stuff\":[{\"qqq\":\"rrr\"}],\"ttt\":0,\"mmm\":\"Some
string\",\"someBool\":true}"}*
We've had errors during indexing like:
*unknown field
'testField:"value","xyz":["abc","def","ghi"],"nnn":"xyz"},"stuff":[{"qqq":"rrr"}],"ttt":0,"mmm":"Some
string","someBool":true}���'*
(those � unprintable characters are part of it)
So far we've not been able to reproduce the problem on a collection with a
single shard, so it does seem like the problem is only happening internally
when updates are distributed to the other shards... But that's not been
totally verified.
We've also only encountered the problem on one of the collections we build
(the data within each collection is generally the same though. The ids are
slightly different - but still strings. The main difference is that this
problematic index is built using an Iterator to *solrj
org.apache.solr.client.solrj.SolrClient.add(String,
Iterator)* - the *SolrInputDocument*s are not being
reused in the client, I checked that -, while the other index is built by
streaming CSVs to Solr.)
We will look into it further, but if anyone has any ideas of what might
have changed in 8.3 from 8.1 / 8.2 that could cause this, that would be
helpful.
Cheers
Colvin
NPE For Atomic Update on Solr 8.3
On upgrading to solr 8.3 also I am facing the same issue as SOLR-13523.
I have added these on my schema
Document present in solr:
[{
"id": "parent1",
"isInStock": 1,
"parent": true,
"_childDocuments_": [
{
"id": "child 1",
"isInStock": 0,
"parent": false
}
]
}]
On doing atomic update I am getting this
<https://lucene.472066.n3.nabble.com/file/t495295/Screenshot_2019-11-19_at_5.png>
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Possible bug in cluster status - > solr 8.3
I found interesting situation, I've created a collection with only one replica.
Then I scaled solr-cloud cluster, and run 'addreplica' call to add 2 more.
So I have a collection with 3 tlog replicas, cluster status page shows
them but shows also this:
"core_node2":{
"core":"EDITED_NAME_shard1_replica_t1",
"base_url":"http://EDITED_NODE:8983/solr;,
"node_name":"EDITED_NODE:8983_solr",
"state":"active",
"type":"TLOG",
"force_set_state":"false",
"leader":"true"},
"core_node5":{
"core":"EDITED_NAME_shard1_replica_t3",
"base_url":"http://EDITED_NODE:8983/solr;,
"node_name":"EDITED_NODE:8983_solr",
"state":"active",
"type":"TLOG",
"force_set_state":"false"},
"core_node6":{
"core":"EDITED_NAME_shard1_replica_t4",
"base_url":"http://EDITED_NODE:8983/solr;,
"node_name":"EDITED_NODE:8983_solr",
"state":"active",
"type":"TLOG",
"force_set_state":"false",
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"nrtReplicas":"1",
"tlogReplicas":"1",
"znodeVersion":11,
As you can see I have 3 replicas but then I have also: "tlogReplicas":"1"
If I create collection with tlogReplicas=3 then cluster status shows
"tlogReplicas":"3"
IS that a bug or somehow 'works as it should' ?
Regards,
Jacek
Re: Solr 8.3 Solrj streaming expressions do not return all field values
I created a JIRA for this:
https://issues.apache.org/jira/browse/SOLR-13894
On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke wrote:
> I have checked now Solr 8.3 server in admin UI. Same issue.
>
> Reproduction:
> select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”,
> qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)
>
> In 8.3 it returns only the id field.
> In 8.2 it returns id,found field.
>
> Since found is generated by select (and not coming from the collection)
> there must be an issue with select.
>
> Any idea why this is happening.
>
> Debug logs do not show any error and the expression is correctly received
> by Solr.
>
> Thank you.
>
> Best regards
>
> > Am 05.11.2019 um 14:59 schrieb Jörn Franke :
> >
> > Thanks I will check and come back to you. As far as I remember (but
> have to check) the queries generated by Solr were correct
> >
> > Just to be clear the same thing works with Solr 8.2 server and Solr 8.2
> client.
> >
> > It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
> >
> >> Am 05.11.2019 um 14:49 schrieb Joel Bernstein :
> >>
> >> I'll probably need some more details. One thing that's useful is to
> look at
> >> the logs and see the underlying Solr queries that are generated. Then
> try
> >> those underlying queries against the Solr index and see what comes
> back. If
> >> you're not seeing the fields with the plain Solr queries then we know
> it's
> >> something going on below streaming expressions. If you are seeing the
> >> fields then it's the expressions themselves that are not handling the
> data
> >> as expected.
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
> >>>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke
> wrote:
> >>>
> >>> Most likely this issue can bei also reproduced in the admin UI for the
> >>> streaming handler of a collection.
> >>>
> >>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke :
> >>>>
> >>>> Hi,
> >>>>
> >>>> I use streaming expressions, e.g.
> >>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A
> asc”)
> >>>> (Using export handler, sort is not really mandatory , I will remove it
> >>> later anyway)
> >>>>
> >>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
> >>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
> >>>>
> >>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then
> the
> >>> above statement only returns the id field, but not the found field.
> >>>>
> >>>> Questions:
> >>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
> >>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix
> this?
> >>>> 2) has the syntax for the above expression changed? If so how?
> >>>> 3) is this not expected behavior and I should create a Jira for it?
> >>>>
> >>>> Thank you.
> >>>> Best regards
> >>>
>
Re: Solr 8.3 admin ui collection selector disabled
Whew! I often work in a private window to lessen these kinds of “surprises”….. > On Nov 6, 2019, at 4:35 AM, Jörn Franke wrote: > > Never mind. Restart of browser worked. > >> Am 06.11.2019 um 10:32 schrieb Jörn Franke : >> >> Hi, >> >> After upgrading to Solr 8.3 I observe that in the Admin UI the collection >> selector is greyed out. I am using Chrome. The core selector works as >> expected. >> >> Any idea why this is happening? >> >> Thank you. >> >> Best regards
Re: Solr 8.3 Solrj streaming expressions do not return all field values
I have checked now Solr 8.3 server in admin UI. Same issue.
Reproduction:
select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”,
qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)
In 8.3 it returns only the id field.
In 8.2 it returns id,found field.
Since found is generated by select (and not coming from the collection) there
must be an issue with select.
Any idea why this is happening.
Debug logs do not show any error and the expression is correctly received by
Solr.
Thank you.
Best regards
> Am 05.11.2019 um 14:59 schrieb Jörn Franke :
>
> Thanks I will check and come back to you. As far as I remember (but have to
> check) the queries generated by Solr were correct
>
> Just to be clear the same thing works with Solr 8.2 server and Solr 8.2
> client.
>
> It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
>
>> Am 05.11.2019 um 14:49 schrieb Joel Bernstein :
>>
>> I'll probably need some more details. One thing that's useful is to look at
>> the logs and see the underlying Solr queries that are generated. Then try
>> those underlying queries against the Solr index and see what comes back. If
>> you're not seeing the fields with the plain Solr queries then we know it's
>> something going on below streaming expressions. If you are seeing the
>> fields then it's the expressions themselves that are not handling the data
>> as expected.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>>>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke wrote:
>>>
>>> Most likely this issue can bei also reproduced in the admin UI for the
>>> streaming handler of a collection.
>>>
>>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke :
>>>>
>>>> Hi,
>>>>
>>>> I use streaming expressions, e.g.
>>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
>>>> (Using export handler, sort is not really mandatory , I will remove it
>>> later anyway)
>>>>
>>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
>>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>>>>
>>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
>>> above statement only returns the id field, but not the found field.
>>>>
>>>> Questions:
>>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
>>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
>>>> 2) has the syntax for the above expression changed? If so how?
>>>> 3) is this not expected behavior and I should create a Jira for it?
>>>>
>>>> Thank you.
>>>> Best regards
>>>
Re: Solr 8.3 admin ui collection selector disabled
Never mind. Restart of browser worked. > Am 06.11.2019 um 10:32 schrieb Jörn Franke : > > Hi, > > After upgrading to Solr 8.3 I observe that in the Admin UI the collection > selector is greyed out. I am using Chrome. The core selector works as > expected. > > Any idea why this is happening? > > Thank you. > > Best regards
Solr 8.3 admin ui collection selector disabled
Hi, After upgrading to Solr 8.3 I observe that in the Admin UI the collection selector is greyed out. I am using Chrome. The core selector works as expected. Any idea why this is happening? Thank you. Best regards
Re: Solr 8.3 Solrj streaming expressions do not return all field values
Thanks I will check and come back to you. As far as I remember (but have to
check) the queries generated by Solr were correct
Just to be clear the same thing works with Solr 8.2 server and Solr 8.2 client.
It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
> Am 05.11.2019 um 14:49 schrieb Joel Bernstein :
>
> I'll probably need some more details. One thing that's useful is to look at
> the logs and see the underlying Solr queries that are generated. Then try
> those underlying queries against the Solr index and see what comes back. If
> you're not seeing the fields with the plain Solr queries then we know it's
> something going on below streaming expressions. If you are seeing the
> fields then it's the expressions themselves that are not handling the data
> as expected.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke wrote:
>>
>> Most likely this issue can bei also reproduced in the admin UI for the
>> streaming handler of a collection.
>>
>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke :
>>>
>>> Hi,
>>>
>>> I use streaming expressions, e.g.
>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
>>> (Using export handler, sort is not really mandatory , I will remove it
>> later anyway)
>>>
>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>>>
>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
>> above statement only returns the id field, but not the found field.
>>>
>>> Questions:
>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
>>> 2) has the syntax for the above expression changed? If so how?
>>> 3) is this not expected behavior and I should create a Jira for it?
>>>
>>> Thank you.
>>> Best regards
>>
Re: Solr 8.3 Solrj streaming expressions do not return all field values
I'll probably need some more details. One thing that's useful is to look at
the logs and see the underlying Solr queries that are generated. Then try
those underlying queries against the Solr index and see what comes back. If
you're not seeing the fields with the plain Solr queries then we know it's
something going on below streaming expressions. If you are seeing the
fields then it's the expressions themselves that are not handling the data
as expected.
Joel Bernstein
http://joelsolr.blogspot.com/
On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke wrote:
> Most likely this issue can bei also reproduced in the admin UI for the
> streaming handler of a collection.
>
> > Am 04.11.2019 um 13:32 schrieb Jörn Franke :
> >
> > Hi,
> >
> > I use streaming expressions, e.g.
> > Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
> > (Using export handler, sort is not really mandatory , I will remove it
> later anyway)
> >
> > This works perfectly fine if I use Solr 8.2.0 (server + client). It
> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
> >
> > However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
> above statement only returns the id field, but not the found field.
> >
> > Questions:
> > 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
> > 2) has the syntax for the above expression changed? If so how?
> > 3) is this not expected behavior and I should create a Jira for it?
> >
> > Thank you.
> > Best regards
>
Re: Solr 8.3 Solrj streaming expressions do not return all field values
Most likely this issue can bei also reproduced in the admin UI for the
streaming handler of a collection.
> Am 04.11.2019 um 13:32 schrieb Jörn Franke :
>
> Hi,
>
> I use streaming expressions, e.g.
> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
> (Using export handler, sort is not really mandatory , I will remove it later
> anyway)
>
> This works perfectly fine if I use Solr 8.2.0 (server + client). It returns
> Tuples in the form { “id”,”12345”, “found”:”Y”}
>
> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
> above statement only returns the id field, but not the found field.
>
> Questions:
> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
> 2) has the syntax for the above expression changed? If so how?
> 3) is this not expected behavior and I should create a Jira for it?
>
> Thank you.
> Best regards
Solr 8.3 Solrj streaming expressions do not return all field values
Hi,
I use streaming expressions, e.g.
Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
(Using export handler, sort is not really mandatory , I will remove it later
anyway)
This works perfectly fine if I use Solr 8.2.0 (server + client). It returns
Tuples in the form { “id”,”12345”, “found”:”Y”}
However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the above
statement only returns the id field, but not the found field.
Questions:
1) is this expected behavior, ie Solr client 8.3.0 is in this case not
compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
2) has the syntax for the above expression changed? If so how?
3) is this not expected behavior and I should create a Jira for it?
Thank you.
Best regards
