ZK is likely in a corrupt state already, trying to write to a folder with a CR at the end. It may have succeeded doing so, in which case fixing the config will again point it to the correct folder which may then contain an older version of the data (solr's config).
So what I'd do is this 1. Stop all Zookeepers and Solr 2. Decide what dataDir and logDir is the most current / correct 3. If necessary, move/rename these folders to match what you have in your fixed zoo.cfg 4. Start all ZKs - watch them sync up and be healthy 5. Start Solr (if the on-disk state on Solr matches the ZK config then all will come up again) 6. Fix any issues that may have happened in Solr due to the misconfiguration You may also want to check with the Zookeeper project (https://zookeeper.apache.org/) if they have a fix for this issue (tolerate CR in config file). Jan > 18. mai 2021 kl. 15:12 skrev yaswanth kumar <[email protected]>: > > Thanks for explaining about dos2unix stuff, but here my main question is > that what could be the sequential steps fixing the bad characters on the > zoo.cfg while the services are running. > > Do we need to stop all the solr services on all the servers, stop the > zookeeper on one server, correct the config , make the zookeeper service up > and do the same steps on remaining servers as well. Are these steps good > enough? or am I missing any sequence that could mess up the configs read by > solr > > Thanks, > Yaswanth > > On Tue, May 18, 2021 at 2:45 AM Claude Warren <[email protected]> wrote: > >> A quick fix would be to run the linux command `dos2unix` on the config >> files. This will ensure that all the end of line (EOL) characters are >> unix/linux style. >> >> A process fix that identifies a Windows tool that will put the Linux EOL >> character rather than the Windows one might be helpful, but I think you >> would still need to use the `dos2linux` command just to be safe. >> >> You could put `dos2unix` in your startup scripts just to be sure. >> >> You could spend some time looking at the code that reads the configuration >> files and contribute a patch that will remove the Windows EOL and replace >> it with the Linux EOL. If you do this you might also consider replacing >> the Mac EOL with Linux EOL in the same code. >> >> Claude >> >> On Tue, May 18, 2021 at 3:43 AM yaswanth kumar <[email protected]> >> wrote: >> >>> Solr : 8.2 >>> Zookeeper:3.4 >>> >>> Recently found an issue inside the zoo.cfg where the datadir and logdir >>> are having a dos carriage return characters on the folder path when this >> is >>> in Linux and that created a folder with ^M characters upon zoo start >> where >>> we noticed the zookeeper pid file is in. This is creating issues when the >>> servers were rebooted for maintenance purpose and when we have auto start >>> scripts for both zookeeper and solr. Sometimes all the collections were >>> disappeared and also loosing zoo configs that were configured before >> reboot. >>> >>> So now this bad config is still in live and we need to correct it >>> carefully by running sed command that replaces these bad carriage returns >>> on these zoo.cfg >>> >>> Are there a sequence steps that we need to do carefully while we change >>> this config (zoo.cfg) on live severs without any disruption to the >> cloud?? >>> >>> No of solr severs are 3 and all three got zookeepers on them and we need >>> to fix this bad characters on all 3 solr severs zoo.cfg >>> >>> Sent from my iPhone >> >> >> >> -- >> I like: Like Like - The likeliest place on the web >> <http://like-like.xenei.com> >> LinkedIn: http://www.linkedin.com/in/claudewarren >> > > > -- > Thanks & Regards, > Yaswanth Kumar Konathala. > [email protected]
