Re: Replication broken.. no helpful errors?
As I had pointed out in my first reply to this thread, you had a directory named temp-snapshot.20070816120113 in your data directory on the slave. Snapinstaller was mistakenly treating that as the lastest snapshot and was installing that every time it was called. Snapinstaller didn't trigger a commit since the same snapshot had already been installed. latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already installed I have open a bug to improve snapinstaller: http://issues.apache.org/jira/browse/SOLR-346 Bill On 9/6/07, Matthew Runo [EMAIL PROTECTED] wrote: Well, I've been playing around with it (removed all the snapshots, restarted tomcat) and it seems like it works now.. maybe. I was noticing that search2 and search3, the slaves, had searchers that had been opened several days ago - when we do several 100 commits and 2 optimizes on search1, the master, every day. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Sep 6, 2007, at 12:37 PM, Yonik Seeley wrote: On 9/6/07, Matthew Runo [EMAIL PROTECTED] wrote: The thing is that a new searcher is not opened if I look in the stats.jsp page. The index version never changes. The index version is read from the index... hence if the lucene index doesn't change (even if a ew snapshot was taken), the version won't change even if a new searcher was opened. Is the problem on the master side now since it looks like the slave is pulling a temp-snapshot? -Yonik
Re: Replication broken.. no helpful errors?
The snapinstaller script opens a new searcher by calling commit. From the attached debug output it looks like that actually worked: + /opt/solr/bin/commit + [[ 0 != 0 ]] + logExit ended 0 Try running the /opt/solr/bin/commit directly with the -V option. Bill On 9/5/07, Matthew Runo [EMAIL PROTECTED] wrote: If it helps anyone, this index is around a gig in size. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote: It seems that the scripts cannot open new searchers at the end of the process, for some reason. Here's a message from cron, but I'm not sure what to make of it... It looks like the files properly copied over, but failed the install. I removed the temp* directory, but still SOLR could not launch a new searcher. I don't see any activity in catalina.out though... started by tomcat5 command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/ data -S /opt/solr/logs -d /opt/solr/data -v pulling snapshot temp-snapshot.20070905150504 receiving file list ... done deleting segments_1ine deleting _164h_1.del deleting _164h.tis deleting _164h.tii deleting _164h.prx deleting _164h.nrm deleting _164h.frq deleting _164h.fnm deleting _164h.fdx deleting _164h.fdt deleting _164g_1.del deleting _164g.tis deleting _164g.tii deleting _164g.prx deleting _164g.nrm deleting _164g.frq deleting _164g.fnm deleting _164g.fdx deleting _164g.fdt deleting _164f_1.del deleting _164f.tis deleting _164f.tii deleting _164f.prx deleting _164f.nrm deleting _164f.frq deleting _164f.fnm deleting _164f.fdx deleting _164f.fdt deleting _164e_1.del deleting _164e.tis deleting _164e.tii deleting _164e.prx deleting _164e.nrm deleting _164e.frq deleting _164e.fnm deleting _164e.fdx deleting _164e.fdt deleting _164d_1.del deleting _164d.tis deleting _164d.tii deleting _164d.prx deleting _164d.nrm deleting _164d.frq deleting _164d.fnm deleting _164d.fdx deleting _164d.fdt deleting _164c_1.del deleting _164c.tis deleting _164c.tii deleting _164c.prx deleting _164c.nrm deleting _164c.frq deleting _164c.fnm deleting _164c.fdx deleting _164c.fdt deleting _164b_1.del deleting _164b.tis deleting _164b.tii deleting _164b.prx deleting _164b.nrm deleting _164b.frq deleting _164b.fnm deleting _164b.fdx deleting _164b.fdt deleting _164a_1.del deleting _164a.tis deleting _164a.tii deleting _164a.prx deleting _164a.nrm deleting _164a.frq deleting _164a.fnm deleting _164a.fdx deleting _164a.fdt deleting _163z_3.del deleting _163z.tis deleting _163z.tii deleting _163z.prx deleting _163z.nrm deleting _163z.frq deleting _163z.fnm deleting _163z.fdx deleting _163z.fdt deleting _163o_3.del deleting _163o.tis deleting _163o.tii deleting _163o.prx deleting _163o.nrm deleting _163o.frq deleting _163o.fnm deleting _163o.fdx deleting _163o.fdt deleting _163d_4.del deleting _163d.tis deleting _163d.tii deleting _163d.prx deleting _163d.nrm deleting _163d.frq deleting _163d.fnm deleting _163d.fdx deleting _163d.fdt deleting _1632_6.del deleting _1632.tis deleting _1632.tii deleting _1632.prx deleting _1632.nrm deleting _1632.frq deleting _1632.fnm deleting _1632.fdx deleting _1632.fdt deleting _162r_7.del deleting _162r.tis deleting _162r.tii deleting _162r.prx deleting _162r.nrm deleting _162r.frq deleting _162r.fnm deleting _162r.fdx deleting _162r.fdt deleting _162g_d.del deleting _162g.tis deleting _162g.tii deleting _162g.prx deleting _162g.nrm deleting _162g.frq deleting _162g.fnm deleting _162g.fdx deleting _162g.fdt deleting _1625_m.del deleting _1625.tis deleting _1625.tii deleting _1625.prx deleting _1625.nrm deleting _1625.frq deleting _1625.fnm deleting _1625.fdx deleting _1625.fdt deleting _161u_w.del deleting _161u.tis deleting _161u.tii deleting _161u.prx deleting _161u.nrm deleting _161u.frq deleting _161u.fnm deleting _161u.fdx deleting _161u.fdt deleting _161j_16.del ./ _161j_17.del _164m.fdt _164m.fdx _164m.fnm _164m.frq _164m.nrm _164m.prx _164m.tii _164m.tis _164m_1.del _164x.fdt _164x.fdx _164x.fnm _164x.frq _164x.nrm _164x.prx _164x.tii _164x.tis _164x_1.del segments.gen segments_1inv sent 516 bytes received 105864302 bytes 30247090.86 bytes/sec total size is 966107226 speedup is 9.13 + [[ -z search1 ]] + [[ -z /opt/solr/logs ]] + fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V + [[ -z tomcat5 ]] ++ whoami + [[ tomcat5 != tomcat5 ]] ++ who -m ++ cut '-d ' -f1 ++ sed '-es/^.*!//' + oldwhoami= + [[ '' == '' ]] +++ pgrep -g0 snapinstaller
Re: Replication broken.. no helpful errors?
The thing is that a new searcher is not opened if I look in the stats.jsp page. The index version never changes. When I run.. sudo /opt/solr/bin/commit -V -u tomcat5 ..I get a new searcher opened, but even though it (in theory) installed the new index, I see no docs in there. During the snapinstaller... + echo 2007/09/06 11:43:49 command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/solr/data -V -u tomcat5 + [[ -n '' ]] ++ ls /opt/solr/data ++ grep 'snapshot\.' ++ grep -v wip ++ sort -r ++ head -1 + name=temp-snapshot.20070905150504 + trap 'echo caught INT/TERM, exiting now but partial installation may have already occured;/bin/rm -rf ${data_dir/index.tmp$$;logExit aborted 13' INT TERM + [[ temp-snapshot.20070905150504 == '' ]] + name=/opt/solr/data/temp-snapshot.20070905150504 ++ cat /opt/solr/logs/snapshot.current ...it would seem that snappuller might not be properly setting the directory name - or should it be temp-*? I had replication working for a few weeks, and then it broke, and has been down since. We're going live with this project in about a week, and I really need to get this going before then =p ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Sep 6, 2007, at 6:01 AM, Bill Au wrote: The snapinstaller script opens a new searcher by calling commit. From the attached debug output it looks like that actually worked: + /opt/solr/bin/commit + [[ 0 != 0 ]] + logExit ended 0 Try running the /opt/solr/bin/commit directly with the -V option. Bill On 9/5/07, Matthew Runo [EMAIL PROTECTED] wrote: If it helps anyone, this index is around a gig in size. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote: It seems that the scripts cannot open new searchers at the end of the process, for some reason. Here's a message from cron, but I'm not sure what to make of it... It looks like the files properly copied over, but failed the install. I removed the temp* directory, but still SOLR could not launch a new searcher. I don't see any activity in catalina.out though... started by tomcat5 command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/ data -S /opt/solr/logs -d /opt/solr/data -v pulling snapshot temp-snapshot.20070905150504 receiving file list ... done deleting segments_1ine deleting _164h_1.del deleting _164h.tis deleting _164h.tii deleting _164h.prx deleting _164h.nrm deleting _164h.frq deleting _164h.fnm deleting _164h.fdx deleting _164h.fdt deleting _164g_1.del deleting _164g.tis deleting _164g.tii deleting _164g.prx deleting _164g.nrm deleting _164g.frq deleting _164g.fnm deleting _164g.fdx deleting _164g.fdt deleting _164f_1.del deleting _164f.tis deleting _164f.tii deleting _164f.prx deleting _164f.nrm deleting _164f.frq deleting _164f.fnm deleting _164f.fdx deleting _164f.fdt deleting _164e_1.del deleting _164e.tis deleting _164e.tii deleting _164e.prx deleting _164e.nrm deleting _164e.frq deleting _164e.fnm deleting _164e.fdx deleting _164e.fdt deleting _164d_1.del deleting _164d.tis deleting _164d.tii deleting _164d.prx deleting _164d.nrm deleting _164d.frq deleting _164d.fnm deleting _164d.fdx deleting _164d.fdt deleting _164c_1.del deleting _164c.tis deleting _164c.tii deleting _164c.prx deleting _164c.nrm deleting _164c.frq deleting _164c.fnm deleting _164c.fdx deleting _164c.fdt deleting _164b_1.del deleting _164b.tis deleting _164b.tii deleting _164b.prx deleting _164b.nrm deleting _164b.frq deleting _164b.fnm deleting _164b.fdx deleting _164b.fdt deleting _164a_1.del deleting _164a.tis deleting _164a.tii deleting _164a.prx deleting _164a.nrm deleting _164a.frq deleting _164a.fnm deleting _164a.fdx deleting _164a.fdt deleting _163z_3.del deleting _163z.tis deleting _163z.tii deleting _163z.prx deleting _163z.nrm deleting _163z.frq deleting _163z.fnm deleting _163z.fdx deleting _163z.fdt deleting _163o_3.del deleting _163o.tis deleting _163o.tii deleting _163o.prx deleting _163o.nrm deleting _163o.frq deleting _163o.fnm deleting _163o.fdx deleting _163o.fdt deleting _163d_4.del deleting _163d.tis deleting _163d.tii deleting _163d.prx deleting _163d.nrm deleting _163d.frq deleting _163d.fnm deleting _163d.fdx deleting _163d.fdt deleting _1632_6.del deleting _1632.tis deleting _1632.tii deleting _1632.prx deleting _1632.nrm deleting _1632.frq deleting _1632.fnm deleting _1632.fdx deleting _1632.fdt deleting _162r_7.del deleting _162r.tis deleting _162r.tii deleting _162r.prx deleting _162r.nrm deleting _162r.frq deleting _162r.fnm deleting _162r.fdx deleting _162r.fdt deleting _162g_d.del deleting _162g.tis deleting _162g.tii deleting _162g.prx deleting _162g.nrm deleting
Re: Replication broken.. no helpful errors?
On 9/6/07, Matthew Runo [EMAIL PROTECTED] wrote: The thing is that a new searcher is not opened if I look in the stats.jsp page. The index version never changes. The index version is read from the index... hence if the lucene index doesn't change (even if a ew snapshot was taken), the version won't change even if a new searcher was opened. Is the problem on the master side now since it looks like the slave is pulling a temp-snapshot? -Yonik
Re: Replication broken.. no helpful errors?
It seems that the scripts cannot open new searchers at the end of the process, for some reason. Here's a message from cron, but I'm not sure what to make of it... It looks like the files properly copied over, but failed the install. I removed the temp* directory, but still SOLR could not launch a new searcher. I don't see any activity in catalina.out though... started by tomcat5 command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/ data -S /opt/solr/logs -d /opt/solr/data -v pulling snapshot temp-snapshot.20070905150504 receiving file list ... done deleting segments_1ine deleting _164h_1.del deleting _164h.tis deleting _164h.tii deleting _164h.prx deleting _164h.nrm deleting _164h.frq deleting _164h.fnm deleting _164h.fdx deleting _164h.fdt deleting _164g_1.del deleting _164g.tis deleting _164g.tii deleting _164g.prx deleting _164g.nrm deleting _164g.frq deleting _164g.fnm deleting _164g.fdx deleting _164g.fdt deleting _164f_1.del deleting _164f.tis deleting _164f.tii deleting _164f.prx deleting _164f.nrm deleting _164f.frq deleting _164f.fnm deleting _164f.fdx deleting _164f.fdt deleting _164e_1.del deleting _164e.tis deleting _164e.tii deleting _164e.prx deleting _164e.nrm deleting _164e.frq deleting _164e.fnm deleting _164e.fdx deleting _164e.fdt deleting _164d_1.del deleting _164d.tis deleting _164d.tii deleting _164d.prx deleting _164d.nrm deleting _164d.frq deleting _164d.fnm deleting _164d.fdx deleting _164d.fdt deleting _164c_1.del deleting _164c.tis deleting _164c.tii deleting _164c.prx deleting _164c.nrm deleting _164c.frq deleting _164c.fnm deleting _164c.fdx deleting _164c.fdt deleting _164b_1.del deleting _164b.tis deleting _164b.tii deleting _164b.prx deleting _164b.nrm deleting _164b.frq deleting _164b.fnm deleting _164b.fdx deleting _164b.fdt deleting _164a_1.del deleting _164a.tis deleting _164a.tii deleting _164a.prx deleting _164a.nrm deleting _164a.frq deleting _164a.fnm deleting _164a.fdx deleting _164a.fdt deleting _163z_3.del deleting _163z.tis deleting _163z.tii deleting _163z.prx deleting _163z.nrm deleting _163z.frq deleting _163z.fnm deleting _163z.fdx deleting _163z.fdt deleting _163o_3.del deleting _163o.tis deleting _163o.tii deleting _163o.prx deleting _163o.nrm deleting _163o.frq deleting _163o.fnm deleting _163o.fdx deleting _163o.fdt deleting _163d_4.del deleting _163d.tis deleting _163d.tii deleting _163d.prx deleting _163d.nrm deleting _163d.frq deleting _163d.fnm deleting _163d.fdx deleting _163d.fdt deleting _1632_6.del deleting _1632.tis deleting _1632.tii deleting _1632.prx deleting _1632.nrm deleting _1632.frq deleting _1632.fnm deleting _1632.fdx deleting _1632.fdt deleting _162r_7.del deleting _162r.tis deleting _162r.tii deleting _162r.prx deleting _162r.nrm deleting _162r.frq deleting _162r.fnm deleting _162r.fdx deleting _162r.fdt deleting _162g_d.del deleting _162g.tis deleting _162g.tii deleting _162g.prx deleting _162g.nrm deleting _162g.frq deleting _162g.fnm deleting _162g.fdx deleting _162g.fdt deleting _1625_m.del deleting _1625.tis deleting _1625.tii deleting _1625.prx deleting _1625.nrm deleting _1625.frq deleting _1625.fnm deleting _1625.fdx deleting _1625.fdt deleting _161u_w.del deleting _161u.tis deleting _161u.tii deleting _161u.prx deleting _161u.nrm deleting _161u.frq deleting _161u.fnm deleting _161u.fdx deleting _161u.fdt deleting _161j_16.del ./ _161j_17.del _164m.fdt _164m.fdx _164m.fnm _164m.frq _164m.nrm _164m.prx _164m.tii _164m.tis _164m_1.del _164x.fdt _164x.fdx _164x.fnm _164x.frq _164x.nrm _164x.prx _164x.tii _164x.tis _164x_1.del segments.gen segments_1inv sent 516 bytes received 105864302 bytes 30247090.86 bytes/sec total size is 966107226 speedup is 9.13 + [[ -z search1 ]] + [[ -z /opt/solr/logs ]] + fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V + [[ -z tomcat5 ]] ++ whoami + [[ tomcat5 != tomcat5 ]] ++ who -m ++ cut '-d ' -f1 ++ sed '-es/^.*!//' + oldwhoami= + [[ '' == '' ]] +++ pgrep -g0 snapinstaller ++ tail -1 ++ cut -f1 '-d ' ++ ps h -Hfp 3621 3629 3630 3631 + oldwhoami=tomcat5 + [[ -z /opt/solr/data ]] ++ echo /opt/solr/data ++ cut -c1 + [[ / != \/ ]] ++ echo /opt/solr/logs ++ cut -c1 + [[ / != \/ ]] ++ date +%s + start=1189030205 + logMessage started by tomcat5 ++ timeStamp ++ date '+%Y/%m/%d %H:%M:%S' + echo 2007/09/05 15:10:05 started by tomcat5 + [[ -n '' ]] + logMessage command: /opt/solr/bin/snapinstaller -M search1 -S /opt/ solr/logs -d /opt/solr/data -V ++ timeStamp ++ date '+%Y/%m/%d %H:%M:%S' + echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/solr/data -V + [[ -n '' ]] ++ ls /opt/solr/data ++ grep 'snapshot\.' ++ grep -v wip ++ sort -r ++ head -1 + name=temp-snapshot.20070905150504 + trap 'echo caught INT/TERM, exiting now but partial installation may have already occured;/bin/rm -rf ${data_dir/index.tmp$$;logExit aborted 13' INT TERM + [[ temp-snapshot.20070905150504 == '' ]] +
Re: Replication broken.. no helpful errors?
If it helps anyone, this index is around a gig in size. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote: It seems that the scripts cannot open new searchers at the end of the process, for some reason. Here's a message from cron, but I'm not sure what to make of it... It looks like the files properly copied over, but failed the install. I removed the temp* directory, but still SOLR could not launch a new searcher. I don't see any activity in catalina.out though... started by tomcat5 command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/ data -S /opt/solr/logs -d /opt/solr/data -v pulling snapshot temp-snapshot.20070905150504 receiving file list ... done deleting segments_1ine deleting _164h_1.del deleting _164h.tis deleting _164h.tii deleting _164h.prx deleting _164h.nrm deleting _164h.frq deleting _164h.fnm deleting _164h.fdx deleting _164h.fdt deleting _164g_1.del deleting _164g.tis deleting _164g.tii deleting _164g.prx deleting _164g.nrm deleting _164g.frq deleting _164g.fnm deleting _164g.fdx deleting _164g.fdt deleting _164f_1.del deleting _164f.tis deleting _164f.tii deleting _164f.prx deleting _164f.nrm deleting _164f.frq deleting _164f.fnm deleting _164f.fdx deleting _164f.fdt deleting _164e_1.del deleting _164e.tis deleting _164e.tii deleting _164e.prx deleting _164e.nrm deleting _164e.frq deleting _164e.fnm deleting _164e.fdx deleting _164e.fdt deleting _164d_1.del deleting _164d.tis deleting _164d.tii deleting _164d.prx deleting _164d.nrm deleting _164d.frq deleting _164d.fnm deleting _164d.fdx deleting _164d.fdt deleting _164c_1.del deleting _164c.tis deleting _164c.tii deleting _164c.prx deleting _164c.nrm deleting _164c.frq deleting _164c.fnm deleting _164c.fdx deleting _164c.fdt deleting _164b_1.del deleting _164b.tis deleting _164b.tii deleting _164b.prx deleting _164b.nrm deleting _164b.frq deleting _164b.fnm deleting _164b.fdx deleting _164b.fdt deleting _164a_1.del deleting _164a.tis deleting _164a.tii deleting _164a.prx deleting _164a.nrm deleting _164a.frq deleting _164a.fnm deleting _164a.fdx deleting _164a.fdt deleting _163z_3.del deleting _163z.tis deleting _163z.tii deleting _163z.prx deleting _163z.nrm deleting _163z.frq deleting _163z.fnm deleting _163z.fdx deleting _163z.fdt deleting _163o_3.del deleting _163o.tis deleting _163o.tii deleting _163o.prx deleting _163o.nrm deleting _163o.frq deleting _163o.fnm deleting _163o.fdx deleting _163o.fdt deleting _163d_4.del deleting _163d.tis deleting _163d.tii deleting _163d.prx deleting _163d.nrm deleting _163d.frq deleting _163d.fnm deleting _163d.fdx deleting _163d.fdt deleting _1632_6.del deleting _1632.tis deleting _1632.tii deleting _1632.prx deleting _1632.nrm deleting _1632.frq deleting _1632.fnm deleting _1632.fdx deleting _1632.fdt deleting _162r_7.del deleting _162r.tis deleting _162r.tii deleting _162r.prx deleting _162r.nrm deleting _162r.frq deleting _162r.fnm deleting _162r.fdx deleting _162r.fdt deleting _162g_d.del deleting _162g.tis deleting _162g.tii deleting _162g.prx deleting _162g.nrm deleting _162g.frq deleting _162g.fnm deleting _162g.fdx deleting _162g.fdt deleting _1625_m.del deleting _1625.tis deleting _1625.tii deleting _1625.prx deleting _1625.nrm deleting _1625.frq deleting _1625.fnm deleting _1625.fdx deleting _1625.fdt deleting _161u_w.del deleting _161u.tis deleting _161u.tii deleting _161u.prx deleting _161u.nrm deleting _161u.frq deleting _161u.fnm deleting _161u.fdx deleting _161u.fdt deleting _161j_16.del ./ _161j_17.del _164m.fdt _164m.fdx _164m.fnm _164m.frq _164m.nrm _164m.prx _164m.tii _164m.tis _164m_1.del _164x.fdt _164x.fdx _164x.fnm _164x.frq _164x.nrm _164x.prx _164x.tii _164x.tis _164x_1.del segments.gen segments_1inv sent 516 bytes received 105864302 bytes 30247090.86 bytes/sec total size is 966107226 speedup is 9.13 + [[ -z search1 ]] + [[ -z /opt/solr/logs ]] + fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V + [[ -z tomcat5 ]] ++ whoami + [[ tomcat5 != tomcat5 ]] ++ who -m ++ cut '-d ' -f1 ++ sed '-es/^.*!//' + oldwhoami= + [[ '' == '' ]] +++ pgrep -g0 snapinstaller ++ tail -1 ++ cut -f1 '-d ' ++ ps h -Hfp 3621 3629 3630 3631 + oldwhoami=tomcat5 + [[ -z /opt/solr/data ]] ++ echo /opt/solr/data ++ cut -c1 + [[ / != \/ ]] ++ echo /opt/solr/logs ++ cut -c1 + [[ / != \/ ]] ++ date +%s + start=1189030205 + logMessage started by tomcat5 ++ timeStamp ++ date '+%Y/%m/%d %H:%M:%S' + echo 2007/09/05 15:10:05 started by tomcat5 + [[ -n '' ]] + logMessage command: /opt/solr/bin/snapinstaller -M search1 -S / opt/solr/logs -d /opt/solr/data -V ++ timeStamp ++ date '+%Y/%m/%d %H:%M:%S' + echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/solr/data -V + [[ -n '' ]] ++ ls /opt/solr/data ++ grep 'snapshot\.' ++ grep -v
Re: Replication broken.. no helpful errors?
latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already installed It looks like you have a directory named temp-snapshot.20070816120113 in your data directory. You should remove it. One of the other script might have left that behind somehow. I will update the snapinstaller script to ignore non-snapshot when looking for the latest snapshot to install. Bill On 8/31/07, Matthew Runo [EMAIL PROTECTED] wrote: Hello! On a somewhat related note, our replication seems very much broken. I've added -v to all my cron jobs, and I think I've seen the error (below). As you can see, it's rsyncing an updated index, but then doesn't seem to know to install it. I'm not sure why though.. no errors are reported anywhere via the -v. Any help would be most appreciated, I'm sure I'm just missing something. You can see the cronjob command in the subject of the forwarded message. ++ | Matthew Runo | Zappos Development | [EMAIL PROTECTED] | 702-943-7833 ++ Begin forwarded message: From: [EMAIL PROTECTED] (Cron Daemon) Date: August 31, 2007 1:02:36 PM PDT To: [EMAIL PROTECTED] Subject: Cron [EMAIL PROTECTED] /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data -v;/ opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/ solr/data -v started by tomcat5 command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/ data -S /opt/solr/logs -d /opt/solr/data -v pulling snapshot snapshot.20070831130005 receiving file list ... done deleting segments_19zm deleting _10bg.tis deleting _10bg.tii deleting _10bg.prx deleting _10bg.nrm deleting _10bg.frq deleting _10bg.fnm deleting _10bg.fdx deleting _10bg.fdt ./ _14ff.fdt _14ff.fdx _14ff.fnm _14ff.frq _14ff.nrm _14ff.prx _14ff.tii _14ff.tis _14ff_9.del _14fq.fdt _14fq.fdx _14fq.fnm _14fq.frq _14fq.nrm _14fq.prx _14fq.tii _14fq.tis _14fq_3.del _14fr.fdt _14fr.fdx _14fr.fnm _14fr.frq _14fr.nrm _14fr.prx _14fr.tii _14fr.tis _14fr_2.del _14fs.fdt _14fs.fdx _14fs.fnm _14fs.frq _14fs.nrm _14fs.prx _14fs.tii _14fs.tis _14fs_1.del segments.gen segments_1fza write.lock sent 871 bytes received 843185604 bytes 24440187.68 bytes/sec total size is 843080453 speedup is 1.00 started by tomcat5 command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs - d /opt/solr/data -v latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already installed