Re: how to change configurations in solrcloud setup
On 3/11/2015 10:45 PM, Aman Tandon wrote: You may need to manually remove the 127.0.1.1 entries from zookeeper after you fix the IP address problem. How to do that? The zkcli script included with Solr should have everything you need -- getfile, putfile, and clear ... but that would be a rather frustrating way to handle it. You won't be able to accomplish your goal by only deleting znodes, you'll have to edit some json structures and replace them in zookeeper. The main thing you'll need to edit is the clusterstate.json ... this is a single file in Solr 4.x, in 5.0 it has changed to a clusterstate for every collection. There are not very many GUI clients for zookeeper. The only one that I've really found is the one that is a plugin for eclipse. I happen to use eclipse, so this is fairly convenient for me: http://www.massedynamic.org/mediawiki/index.php?title=Eclipse_Plug-in_for_ZooKeeper Thanks, Shawn
Re: how to change configurations in solrcloud setup
Hi, Thanks Nitin for replying, isn't it will be costly operation to restart all nodes. What i am doing in this is uploading the configurations again to zookeeper and then reloading my core. And it is working well. So am i missing something? With Regards Aman Tandon On Wed, Mar 11, 2015 at 11:21 AM, Nitin Solanki nitinml...@gmail.com wrote: Hi Aman, You can apply configuration on solr cloud by using this command - sudo path_of_solr/solr_folder_name/example/scripts/cloud-scripts/zkcli.sh -zkhost localhost:9983 -cmd upconfig -confdir path_of_solr/solr_folder_name/example/solr/collection1/conf -confname default and then restart all nodes of solrcloud. On Mon, Mar 9, 2015 at 11:43 AM, Aman Tandon amantandon...@gmail.com wrote: Please help. With Regards Aman Tandon On Sat, Mar 7, 2015 at 9:58 PM, Aman Tandon amantandon...@gmail.com wrote: Hi, Please tell me what is best way to apply configuration changes in solr cloud and how to do that. Thanks in advance. With Regards Aman Tandon
Re: how to change configurations in solrcloud setup
On 3/11/2015 12:43 AM, Aman Tandon wrote: Thanks Nitin for replying, isn't it will be costly operation to restart all nodes. What i am doing in this is uploading the configurations again to zookeeper and then reloading my core. And it is working well. So am i missing something? Yes, that is enough, except that you should reload the collection, which will reload all cores for that collection. Fully restarting all Solr instances is not required for most changes to a collection config. There is an exception - when you are adding config that uses new jars added to ${solr.solr.home}/lib after Solr startup. In that situation, a restart would be required so that the new jars get loaded. It is strongly recommended that you use an external zookeeper and that you do a rolling restart, where you restart one node, wait for the cloud graph in the admin UI to show 100% green, then restart the next node. Nitin's suggestion shows the zkcli.sh script starting with sudo which runs the command as root. This is not necessary. As long as the contents of the example directory (or server directory in 5.0) is accessible to a normal user and the script is marked executable for that user, no special permissions are required. Thanks, Shawn
Re: how to change configurations in solrcloud setup
Thanks Shawn. except that you should reload the collection, which will reload all cores for that collection So i could reload a collection via Collection API's http://localhost:8983/solr/admin/collections?action=RELOADname=newCollection right? With Regards Aman Tandon On Wed, Mar 11, 2015 at 1:48 PM, Shawn Heisey apa...@elyograg.org wrote: On 3/11/2015 12:43 AM, Aman Tandon wrote: Thanks Nitin for replying, isn't it will be costly operation to restart all nodes. What i am doing in this is uploading the configurations again to zookeeper and then reloading my core. And it is working well. So am i missing something? Yes, that is enough, except that you should reload the collection, which will reload all cores for that collection. Fully restarting all Solr instances is not required for most changes to a collection config. There is an exception - when you are adding config that uses new jars added to ${solr.solr.home}/lib after Solr startup. In that situation, a restart would be required so that the new jars get loaded. It is strongly recommended that you use an external zookeeper and that you do a rolling restart, where you restart one node, wait for the cloud graph in the admin UI to show 100% green, then restart the next node. Nitin's suggestion shows the zkcli.sh script starting with sudo which runs the command as root. This is not necessary. As long as the contents of the example directory (or server directory in 5.0) is accessible to a normal user and the script is marked executable for that user, no special permissions are required. Thanks, Shawn
Re: how to change configurations in solrcloud setup
Hi Shawn, I make the changes in my schema.xml uploaded the configuration from one of my server, which is now visible in all other servers (I confirmed it by checking from admin interface). *My Solr Cloud arch is :* I have two collections, mcat intent in my external zookeeper ensemble of 3. *On Machine1* I created the mcat_shard1 of mcat in core 1 intent_shard1 of intent in core 2. *On Machine2* I created the mcat_shard2 of mcat in core 1 intent_shard2 of intent in core 2. *On Machine 3* I created the replica of mcat_shard1 in core 1 I created the replica of mcat_shard2 in core 2 I created the replica of intent_shard1 in core 3 I created the replica of intent_shard2 in core 4 So to apply the changes i reloaded my collection using the url: http://localhost:4567/solr/admin/collections?action=RELOADname=intent Due to which the two shard present on one of my machine (Machine 2) goes in recovery mode and i am not able to understand what's wrong here. *Please see the solr logs of that machine:* INFO - 2015-03-11 12:12:41.630; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2onlyIfLeaderActive=truestate=recoveringnodeName=192.168.5.81:4567_solraction=PREPRECOVERYcheckLive=truecore=mcat_shard2_replica_corewt=javabinonlyIfLeader=trueversion=2} status=400 QTime=1 ERROR - 2015-03-11 12:12:41.643; org.apache.solr.common.SolrException; Error while trying to recover. core=mcat_core:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.1.1:4567/solr: I was asked to wait on state recovering for null in null on 192.168.5.81:4567_solr but I still do not see the requested state. I see state: null live:false leader from ZK: Not available due to: java.lang.NullPointerException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:597) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:369) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235) Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.1.1:4567/solr: I was asked to wait on state recovering for null in null on 192.168.5.81:4567_solr but I still do not see the requested state. I see state: null live:false leader from ZK: Not available due to: java.lang.NullPointerException at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:249) at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:245) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ERROR - 2015-03-11 12:12:41.644; org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (6) core=mcat_core INFO - 2015-03-11 12:12:41.644; org.apache.solr.cloud.RecoveryStrategy; Wait 60.0 seconds before trying to recover again (7) INFO - 2015-03-11 12:12:42.665; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) INFO - 2015-03-11 12:12:43.355; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) I restart my complete cluster but the problem still present. Please help. *Here is the screenshot url:* *http://i.imgur.com/QFdg89S.png http://i.imgur.com/QFdg89S.png* http://i.imgur.com/tS0yTNh.png With Regards Aman Tandon On Wed, Mar 11, 2015 at 2:02 PM, Aman Tandon amantandon...@gmail.com wrote: Thanks Shawn. except that you should reload the collection, which will reload all cores for that collection So i could reload a collection via Collection API's http://localhost:8983/solr/admin/collections?action=RELOADname=newCollection right? With Regards Aman Tandon On Wed, Mar 11, 2015 at 1:48 PM, Shawn Heisey apa...@elyograg.org wrote: On 3/11/2015 12:43 AM, Aman Tandon wrote: Thanks Nitin for replying, isn't it will be costly operation to restart all nodes. What i am doing in this is uploading the configurations again to zookeeper and then reloading my core. And it is working well. So am i missing something? Yes, that is enough, except that you should reload the collection, which will reload all cores for that collection. Fully restarting all Solr instances is
Re: how to change configurations in solrcloud setup
On 3/11/2015 6:32 AM, Aman Tandon wrote: I restart my complete cluster but the problem still present. Please help. *Here is the screenshot url:* *http://i.imgur.com/QFdg89S.png http://i.imgur.com/tS0yTNh.png The first screenshot actually shows the problem, but it may not be immediately obvious. In both cases where the shard leader is on 127.0.1.1, recovery is not able to complete. The reason for this is extremely simple -- 127.0.1.1 is a loopback address, not accessible remotely ... so when the machine at 192.168.6.217 tries to contact the other machine, it is only talking to itself, and is not able to find the leader to initiate recovery. I don't know why that machine chose to register itself in zookeeper using a loopback address, unless perhaps the hosts file on that machine is set up incorrectly so that the local hostname is associated with that IP address. If you can't fix the IP lookup problem, you can override the value that Solr uses with the host property, which you can set on the java commandline (-Dhost=) or in the solr.xml file. http://wiki.apache.org/solr/SolrCloud#SolrCloud_Instance_Params You may need to manually remove the 127.0.1.1 entries from zookeeper after you fix the IP address problem. Thanks, Shawn
Re: how to change configurations in solrcloud setup
Hi Shawn, As suggested i gave the -Dhost=192.168.5.236 in command line for the server which was showing 127.0.0.1. *./solr start -c -z 192.168.6.217:2181 http://192.168.6.217:2181,192.168.5.81:2181 http://192.168.5.81:2181,192.168.5.236:2181 http://192.168.5.236:2181 -p 4567 -Dhost=192.168.5.236 -V* Now everything is good. Thank you so much. You may need to manually remove the 127.0.1.1 entries from zookeeper after you fix the IP address problem. How to do that? With Regards Aman Tandon On Wed, Mar 11, 2015 at 8:02 PM, Shawn Heisey apa...@elyograg.org wrote: On 3/11/2015 6:32 AM, Aman Tandon wrote: I restart my complete cluster but the problem still present. Please help. *Here is the screenshot url:* *http://i.imgur.com/QFdg89S.png http://i.imgur.com/tS0yTNh.png The first screenshot actually shows the problem, but it may not be immediately obvious. In both cases where the shard leader is on 127.0.1.1, recovery is not able to complete. The reason for this is extremely simple -- 127.0.1.1 is a loopback address, not accessible remotely ... so when the machine at 192.168.6.217 tries to contact the other machine, it is only talking to itself, and is not able to find the leader to initiate recovery. I don't know why that machine chose to register itself in zookeeper using a loopback address, unless perhaps the hosts file on that machine is set up incorrectly so that the local hostname is associated with that IP address. If you can't fix the IP lookup problem, you can override the value that Solr uses with the host property, which you can set on the java commandline (-Dhost=) or in the solr.xml file. http://wiki.apache.org/solr/SolrCloud#SolrCloud_Instance_Params You may need to manually remove the 127.0.1.1 entries from zookeeper after you fix the IP address problem. Thanks, Shawn
Re: how to change configurations in solrcloud setup
Hi Aman, You can apply configuration on solr cloud by using this command - sudo path_of_solr/solr_folder_name/example/scripts/cloud-scripts/zkcli.sh -zkhost localhost:9983 -cmd upconfig -confdir path_of_solr/solr_folder_name/example/solr/collection1/conf -confname default and then restart all nodes of solrcloud. On Mon, Mar 9, 2015 at 11:43 AM, Aman Tandon amantandon...@gmail.com wrote: Please help. With Regards Aman Tandon On Sat, Mar 7, 2015 at 9:58 PM, Aman Tandon amantandon...@gmail.com wrote: Hi, Please tell me what is best way to apply configuration changes in solr cloud and how to do that. Thanks in advance. With Regards Aman Tandon
Re: how to change configurations in solrcloud setup
Please help. With Regards Aman Tandon On Sat, Mar 7, 2015 at 9:58 PM, Aman Tandon amantandon...@gmail.com wrote: Hi, Please tell me what is best way to apply configuration changes in solr cloud and how to do that. Thanks in advance. With Regards Aman Tandon