Re: Replication script file issues..
Matthew Runo wrote: It seems that as soon as I get a commit, snapshooter goes wild. I have 1107 running instances of snapshooter right now.. I suspect you've got pathing and/or permissions issues. First try running snapshooter -v, and it will be louder. I've often had to dig in deeper, tho. I'd kill them all off. Edit the snapshooter script and add set -x to line two of the script and run it by hand. Make sure to run it by hand as the user (which might be tomcat, I don't know your setup) that would be running it from cron. It might be that you have disk performance issue, or two much data to transfer in 5 minutes or whatever your cron period is set to. If you've got multiple snapshooters hogging the master rsync at once, you'll very likely run into some blockage.
Re: Replication script file issues..
FYI, in additional to the -v option, the latest version of all the scripts have a -V option which is equivalent to set -x. Bill On 7/20/07, Matthew Runo [EMAIL PROTECTED] wrote: Just an FYI.. it seems that the generated scripts.conf file had DOS format newlines. Once I did :set fileformat=unix in VIM, all was well. Everything is working like a champ now, with no other changes aside from that. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 19, 2007, at 11:24 PM, Jed Reynolds wrote: Matthew Runo wrote: It seems that as soon as I get a commit, snapshooter goes wild. I have 1107 running instances of snapshooter right now.. I suspect you've got pathing and/or permissions issues. First try running snapshooter -v, and it will be louder. I've often had to dig in deeper, tho. I'd kill them all off. Edit the snapshooter script and add set -x to line two of the script and run it by hand. Make sure to run it by hand as the user (which might be tomcat, I don't know your setup) that would be running it from cron. It might be that you have disk performance issue, or two much data to transfer in 5 minutes or whatever your cron period is set to. If you've got multiple snapshooters hogging the master rsync at once, you'll very likely run into some blockage.
Re: Replication script file issues..
Yes, I used that, and didn't see much else. But I did see that it was trying to execute something on the blank line at the top of scripts.conf, which eventually lead to the newlines. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 20, 2007, at 9:45 AM, Bill Au wrote: FYI, in additional to the -v option, the latest version of all the scripts have a -V option which is equivalent to set -x. Bill On 7/20/07, Matthew Runo [EMAIL PROTECTED] wrote: Just an FYI.. it seems that the generated scripts.conf file had DOS format newlines. Once I did :set fileformat=unix in VIM, all was well. Everything is working like a champ now, with no other changes aside from that. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 19, 2007, at 11:24 PM, Jed Reynolds wrote: Matthew Runo wrote: It seems that as soon as I get a commit, snapshooter goes wild. I have 1107 running instances of snapshooter right now.. I suspect you've got pathing and/or permissions issues. First try running snapshooter -v, and it will be louder. I've often had to dig in deeper, tho. I'd kill them all off. Edit the snapshooter script and add set -x to line two of the script and run it by hand. Make sure to run it by hand as the user (which might be tomcat, I don't know your setup) that would be running it from cron. It might be that you have disk performance issue, or two much data to transfer in 5 minutes or whatever your cron period is set to. If you've got multiple snapshooters hogging the master rsync at once, you'll very likely run into some blockage.
Re: Replication script file issues..
Also... [EMAIL PROTECTED]:/home/mruno]$ sudo /opt/solr/bin/snapinstaller -M search1.zappos.com -d /opt/solr/data -S /opt/solr/logs -u tomcat5 -v : command not foundpts.conf: line 15: : command not foundpts.conf: line 15: started by mruno command: /opt/solr/bin/snapinstaller -M search1.zappos.com -d /opt/ solr/data -S /opt/solr/logs -u tomcat5 -v installing snapshot /opt/solr/data/snapshot.20070718113034 notifing Solr to open a new Searcher : command not foundpts.conf: line 15: !udo: no passwd entry for tomcat5 failed to connect to Solr server snapshot installed but Solr server has not open a new Searcher I think that snapinstaller is failing because commit fails without the -u tomcat5 in it... ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 18, 2007, at 11:40 AM, Matthew Runo wrote: That seemed to work well. Thank you! I'm seeing some weird stuff from the code though, revolving, I think, around users.. [EMAIL PROTECTED]:/home/mruno]$ sudo /opt/solr/bin/commit : command not foundpts.conf: line 15: !udo: no passwd entry for tomcat5 [EMAIL PROTECTED]:/home/mruno]$ sudo /opt/solr/bin/commit -u tomcat5 : command not foundpts.conf: line 15: : command not foundpts.conf: line 15: [EMAIL PROTECTED]:/home/mruno]$ Why would the first fail, but the second work? I have the following set up in the scripts.conf file.. user=tomcat5 solr_hostname=search2.zappos.com solr_port=8080 rsyncd_port=18080 data_dir=/opt/solr/data webapp_name=solr master_host=search1.zappos.com master_data_dir=/opt/solr/data master_status_dir=/opt/solr/logs Once I get this all working, I'm going to update the wiki with a how to page for replication. All the information is there, but it's a bit hard to find, and I think a nice how-to would help lots of people. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 18, 2007, at 10:17 AM, Bill Au wrote: You should specify snapshooter by its full path, as in: str name=exe/opt/solr/bin/snapshooter/str Bill On 7/18/07, Matthew Runo [EMAIL PROTECTED] wrote: I'm sorry about this guys, but I'm having the strangest path errors with replication.. SEVERE: java.io.IOException: Cannot run program snapshooter (in directory /opt/solr/bin): java.io.IOException: error=2, No such file or directory but.. cd /opt/solr/bin and the snapshooter file is there. And owned by the tomcat user... so what's the deal? Obviously I'm overlooking something.. but I'm not sure what.. Here's the solrconfig.conf file section listener event=postCommit class=solr.RunExecutableListener str name=exesnapshooter/str str name=dir/opt/solr/bin/str bool name=waittrue/bool arr name=args str-u tomcat5/str str-d /opt/solr/ data/str /arr arr name=env/arr /listener I'd think it would work. I guess, since SOLR runs in tomcat, I don't really need the -u option in there. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++
Re: Replication script file issues..
Also... tomcat5 17003 0.0 0.0 3084 1456 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17008 0.0 0.0 3084 1456 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17013 0.0 0.0 3084 1456 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17018 0.0 0.0 3084 1456 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17023 0.0 0.0 3084 1456 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17028 0.0 0.0 3084 1452 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17033 0.0 0.0 3084 1452 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17038 0.0 0.0 3084 1452 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17043 0.0 0.0 3084 1452 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17048 0.0 0.0 3216 1456 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17053 0.0 0.0 3216 1460 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17058 0.0 0.0 3216 1460 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17063 0.0 0.0 3216 1460 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17068 0.0 0.0 3216 1460 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17073 0.0 0.0 3216 1460 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17078 0.0 0.0 2956 1444 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data tomcat5 17081 0.0 0.0 2960 1464 ?S10:32 0:00 /bin/ bash /opt/solr/bin/snapshooter -u tomcat5 -d /opt/solr/data Why are there so many seemingly hung snapshooter processes? ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 18, 2007, at 11:42 AM, Matthew Runo wrote: Also... [EMAIL PROTECTED]:/home/mruno]$ sudo /opt/solr/bin/snapinstaller -M search1.zappos.com -d /opt/solr/data -S /opt/solr/logs -u tomcat5 -v : command not foundpts.conf: line 15: : command not foundpts.conf: line 15: started by mruno command: /opt/solr/bin/snapinstaller -M search1.zappos.com -d /opt/ solr/data -S /opt/solr/logs -u tomcat5 -v installing snapshot /opt/solr/data/snapshot.20070718113034 notifing Solr to open a new Searcher : command not foundpts.conf: line 15: !udo: no passwd entry for tomcat5 failed to connect to Solr server snapshot installed but Solr server has not open a new Searcher I think that snapinstaller is failing because commit fails without the -u tomcat5 in it... ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 18, 2007, at 11:40 AM, Matthew Runo wrote: That seemed to work well. Thank you! I'm seeing some weird stuff from the code though, revolving, I think, around users.. [EMAIL PROTECTED]:/home/mruno]$ sudo /opt/solr/bin/commit : command not foundpts.conf: line 15: !udo: no passwd entry for tomcat5 [EMAIL PROTECTED]:/home/mruno]$ sudo /opt/solr/bin/commit -u tomcat5 : command not foundpts.conf: line 15: : command not foundpts.conf: line 15: [EMAIL PROTECTED]:/home/mruno]$ Why would the first fail, but the second work? I have the following set up in the scripts.conf file.. user=tomcat5 solr_hostname=search2.zappos.com solr_port=8080 rsyncd_port=18080 data_dir=/opt/solr/data webapp_name=solr master_host=search1.zappos.com master_data_dir=/opt/solr/data master_status_dir=/opt/solr/logs Once I get this all working, I'm going to update the wiki with a how to page for replication. All the information is there, but it's a bit hard to find, and I think a nice how-to would help lots of people. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 18, 2007, at 10:17 AM, Bill Au wrote: You should specify snapshooter by its full path, as in: str name=exe/opt/solr/bin/snapshooter/str Bill On 7/18/07, Matthew Runo [EMAIL PROTECTED] wrote: I'm sorry about this guys, but I'm having the strangest path errors with replication.. SEVERE:
Re: Replication script file issues..
It seems that as soon as I get a commit, snapshooter goes wild. I have 1107 running instances of snapshooter right now.. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 18, 2007, at 11:42 AM, Matthew Runo wrote: Also... [EMAIL PROTECTED]:/home/mruno]$ sudo /opt/solr/bin/snapinstaller -M search1.zappos.com -d /opt/solr/data -S /opt/solr/logs -u tomcat5 -v : command not foundpts.conf: line 15: : command not foundpts.conf: line 15: started by mruno command: /opt/solr/bin/snapinstaller -M search1.zappos.com -d /opt/ solr/data -S /opt/solr/logs -u tomcat5 -v installing snapshot /opt/solr/data/snapshot.20070718113034 notifing Solr to open a new Searcher : command not foundpts.conf: line 15: !udo: no passwd entry for tomcat5 failed to connect to Solr server snapshot installed but Solr server has not open a new Searcher I think that snapinstaller is failing because commit fails without the -u tomcat5 in it... ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 18, 2007, at 11:40 AM, Matthew Runo wrote: That seemed to work well. Thank you! I'm seeing some weird stuff from the code though, revolving, I think, around users.. [EMAIL PROTECTED]:/home/mruno]$ sudo /opt/solr/bin/commit : command not foundpts.conf: line 15: !udo: no passwd entry for tomcat5 [EMAIL PROTECTED]:/home/mruno]$ sudo /opt/solr/bin/commit -u tomcat5 : command not foundpts.conf: line 15: : command not foundpts.conf: line 15: [EMAIL PROTECTED]:/home/mruno]$ Why would the first fail, but the second work? I have the following set up in the scripts.conf file.. user=tomcat5 solr_hostname=search2.zappos.com solr_port=8080 rsyncd_port=18080 data_dir=/opt/solr/data webapp_name=solr master_host=search1.zappos.com master_data_dir=/opt/solr/data master_status_dir=/opt/solr/logs Once I get this all working, I'm going to update the wiki with a how to page for replication. All the information is there, but it's a bit hard to find, and I think a nice how-to would help lots of people. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Jul 18, 2007, at 10:17 AM, Bill Au wrote: You should specify snapshooter by its full path, as in: str name=exe/opt/solr/bin/snapshooter/str Bill On 7/18/07, Matthew Runo [EMAIL PROTECTED] wrote: I'm sorry about this guys, but I'm having the strangest path errors with replication.. SEVERE: java.io.IOException: Cannot run program snapshooter (in directory /opt/solr/bin): java.io.IOException: error=2, No such file or directory but.. cd /opt/solr/bin and the snapshooter file is there. And owned by the tomcat user... so what's the deal? Obviously I'm overlooking something.. but I'm not sure what.. Here's the solrconfig.conf file section listener event=postCommit class=solr.RunExecutableListener str name=exesnapshooter/str str name=dir/opt/solr/bin/str bool name=waittrue/bool arr name=args str-u tomcat5/str str-d /opt/solr/ data/str /arr arr name=env/arr /listener I'd think it would work. I guess, since SOLR runs in tomcat, I don't really need the -u option in there. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++
