Re: Replication broken.. no helpful errors?

2007-09-07 Thread Bill Au
As I had pointed out in my first reply to this thread, you had a directory
named temp-snapshot.20070816120113
in your data directory on the slave.  Snapinstaller was mistakenly treating
that as the lastest snapshot and was installing that every time it was
called.  Snapinstaller didn't trigger a commit since the same snapshot had
already been installed.

 latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
 installed

I have open a bug to improve snapinstaller:

http://issues.apache.org/jira/browse/SOLR-346

Bill


On 9/6/07, Matthew Runo [EMAIL PROTECTED] wrote:

 Well, I've been playing around with it (removed all the snapshots,
 restarted tomcat) and it seems like it works now.. maybe.

 I was noticing that search2 and search3, the slaves, had searchers
 that had been opened several days ago - when we do several 100
 commits and 2 optimizes on search1, the master, every day.

 ++
   | Matthew Runo
   | Zappos Development
   | [EMAIL PROTECTED]
   | 702-943-7833
 ++


 On Sep 6, 2007, at 12:37 PM, Yonik Seeley wrote:

  On 9/6/07, Matthew Runo [EMAIL PROTECTED] wrote:
  The thing is that a new searcher is not opened if I look in the
  stats.jsp page. The index version never changes.
 
  The index version is read from the index... hence if the lucene index
  doesn't change (even if a ew snapshot was taken), the version won't
  change even if a new searcher was opened.
 
  Is the problem on the master side now since it looks like the slave is
  pulling a temp-snapshot?
 
  -Yonik
 




Re: Replication broken.. no helpful errors?

2007-09-06 Thread Bill Au
The snapinstaller script opens a new searcher by calling commit.  From the
attached debug output it looks like that actually worked:

+ /opt/solr/bin/commit
+ [[ 0 != 0 ]]
+ logExit ended 0

Try running the /opt/solr/bin/commit directly with the -V option.

Bill

On 9/5/07, Matthew Runo [EMAIL PROTECTED] wrote:

 If it helps anyone, this index is around a gig in size.

 ++
   | Matthew Runo
   | Zappos Development
   | [EMAIL PROTECTED]
   | 702-943-7833
 ++


 On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote:

  It seems that the scripts cannot open new searchers at the end of
  the process, for some reason. Here's a message from cron, but I'm
  not sure what to make of it... It looks like the files properly
  copied over, but failed the install. I removed the temp* directory,
  but still SOLR could not launch a new searcher. I don't see any
  activity in catalina.out though...
 
 
  started by tomcat5
  command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
  data -S /opt/solr/logs -d /opt/solr/data -v
  pulling snapshot temp-snapshot.20070905150504
  receiving file list ... done
  deleting segments_1ine
  deleting _164h_1.del
  deleting _164h.tis
  deleting _164h.tii
  deleting _164h.prx
  deleting _164h.nrm
  deleting _164h.frq
  deleting _164h.fnm
  deleting _164h.fdx
  deleting _164h.fdt
  deleting _164g_1.del
  deleting _164g.tis
  deleting _164g.tii
  deleting _164g.prx
  deleting _164g.nrm
  deleting _164g.frq
  deleting _164g.fnm
  deleting _164g.fdx
  deleting _164g.fdt
  deleting _164f_1.del
  deleting _164f.tis
  deleting _164f.tii
  deleting _164f.prx
  deleting _164f.nrm
  deleting _164f.frq
  deleting _164f.fnm
  deleting _164f.fdx
  deleting _164f.fdt
  deleting _164e_1.del
  deleting _164e.tis
  deleting _164e.tii
  deleting _164e.prx
  deleting _164e.nrm
  deleting _164e.frq
  deleting _164e.fnm
  deleting _164e.fdx
  deleting _164e.fdt
  deleting _164d_1.del
  deleting _164d.tis
  deleting _164d.tii
  deleting _164d.prx
  deleting _164d.nrm
  deleting _164d.frq
  deleting _164d.fnm
  deleting _164d.fdx
  deleting _164d.fdt
  deleting _164c_1.del
  deleting _164c.tis
  deleting _164c.tii
  deleting _164c.prx
  deleting _164c.nrm
  deleting _164c.frq
  deleting _164c.fnm
  deleting _164c.fdx
  deleting _164c.fdt
  deleting _164b_1.del
  deleting _164b.tis
  deleting _164b.tii
  deleting _164b.prx
  deleting _164b.nrm
  deleting _164b.frq
  deleting _164b.fnm
  deleting _164b.fdx
  deleting _164b.fdt
  deleting _164a_1.del
  deleting _164a.tis
  deleting _164a.tii
  deleting _164a.prx
  deleting _164a.nrm
  deleting _164a.frq
  deleting _164a.fnm
  deleting _164a.fdx
  deleting _164a.fdt
  deleting _163z_3.del
  deleting _163z.tis
  deleting _163z.tii
  deleting _163z.prx
  deleting _163z.nrm
  deleting _163z.frq
  deleting _163z.fnm
  deleting _163z.fdx
  deleting _163z.fdt
  deleting _163o_3.del
  deleting _163o.tis
  deleting _163o.tii
  deleting _163o.prx
  deleting _163o.nrm
  deleting _163o.frq
  deleting _163o.fnm
  deleting _163o.fdx
  deleting _163o.fdt
  deleting _163d_4.del
  deleting _163d.tis
  deleting _163d.tii
  deleting _163d.prx
  deleting _163d.nrm
  deleting _163d.frq
  deleting _163d.fnm
  deleting _163d.fdx
  deleting _163d.fdt
  deleting _1632_6.del
  deleting _1632.tis
  deleting _1632.tii
  deleting _1632.prx
  deleting _1632.nrm
  deleting _1632.frq
  deleting _1632.fnm
  deleting _1632.fdx
  deleting _1632.fdt
  deleting _162r_7.del
  deleting _162r.tis
  deleting _162r.tii
  deleting _162r.prx
  deleting _162r.nrm
  deleting _162r.frq
  deleting _162r.fnm
  deleting _162r.fdx
  deleting _162r.fdt
  deleting _162g_d.del
  deleting _162g.tis
  deleting _162g.tii
  deleting _162g.prx
  deleting _162g.nrm
  deleting _162g.frq
  deleting _162g.fnm
  deleting _162g.fdx
  deleting _162g.fdt
  deleting _1625_m.del
  deleting _1625.tis
  deleting _1625.tii
  deleting _1625.prx
  deleting _1625.nrm
  deleting _1625.frq
  deleting _1625.fnm
  deleting _1625.fdx
  deleting _1625.fdt
  deleting _161u_w.del
  deleting _161u.tis
  deleting _161u.tii
  deleting _161u.prx
  deleting _161u.nrm
  deleting _161u.frq
  deleting _161u.fnm
  deleting _161u.fdx
  deleting _161u.fdt
  deleting _161j_16.del
  ./
  _161j_17.del
  _164m.fdt
  _164m.fdx
  _164m.fnm
  _164m.frq
  _164m.nrm
  _164m.prx
  _164m.tii
  _164m.tis
  _164m_1.del
  _164x.fdt
  _164x.fdx
  _164x.fnm
  _164x.frq
  _164x.nrm
  _164x.prx
  _164x.tii
  _164x.tis
  _164x_1.del
  segments.gen
  segments_1inv
 
  sent 516 bytes  received 105864302 bytes  30247090.86 bytes/sec
  total size is 966107226  speedup is 9.13
  + [[ -z search1 ]]
  + [[ -z /opt/solr/logs ]]
  + fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V
  + [[ -z tomcat5 ]]
  ++ whoami
  + [[ tomcat5 != tomcat5 ]]
  ++ who -m
  ++ cut '-d ' -f1
  ++ sed '-es/^.*!//'
  + oldwhoami=
  + [[ '' == '' ]]
  +++ pgrep -g0 snapinstaller

Re: Replication broken.. no helpful errors?

2007-09-06 Thread Matthew Runo
The thing is that a new searcher is not opened if I look in the  
stats.jsp page. The index version never changes.


When I run..

sudo /opt/solr/bin/commit -V -u tomcat5

..I get a new searcher opened, but even though it (in theory)  
installed the new index, I see no docs in there. During the  
snapinstaller...


+ echo 2007/09/06 11:43:49 command: /opt/solr/bin/snapinstaller -M  
search1 -S /opt/solr/logs -d /opt/solr/data -V -u tomcat5

+ [[ -n '' ]]
++ ls /opt/solr/data
++ grep 'snapshot\.'
++ grep -v wip
++ sort -r
++ head -1
+ name=temp-snapshot.20070905150504
+ trap 'echo caught INT/TERM, exiting now but partial installation  
may have already occured;/bin/rm -rf ${data_dir/index.tmp$$;logExit  
aborted 13' INT TERM

+ [[ temp-snapshot.20070905150504 == '' ]]
+ name=/opt/solr/data/temp-snapshot.20070905150504
++ cat /opt/solr/logs/snapshot.current


...it would seem that snappuller might not be properly setting the  
directory name - or should it be temp-*?


I had replication working for a few weeks, and then it broke, and has  
been down since. We're going live with this project in about a week,  
and I really need to get this going before then =p


++
 | Matthew Runo
 | Zappos Development
 | [EMAIL PROTECTED]
 | 702-943-7833
++


On Sep 6, 2007, at 6:01 AM, Bill Au wrote:

The snapinstaller script opens a new searcher by calling commit.   
From the

attached debug output it looks like that actually worked:

+ /opt/solr/bin/commit
+ [[ 0 != 0 ]]
+ logExit ended 0

Try running the /opt/solr/bin/commit directly with the -V option.

Bill

On 9/5/07, Matthew Runo [EMAIL PROTECTED] wrote:


If it helps anyone, this index is around a gig in size.

++
  | Matthew Runo
  | Zappos Development
  | [EMAIL PROTECTED]
  | 702-943-7833
++


On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote:


It seems that the scripts cannot open new searchers at the end of
the process, for some reason. Here's a message from cron, but I'm
not sure what to make of it... It looks like the files properly
copied over, but failed the install. I removed the temp* directory,
but still SOLR could not launch a new searcher. I don't see any
activity in catalina.out though...


started by tomcat5
command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
data -S /opt/solr/logs -d /opt/solr/data -v
pulling snapshot temp-snapshot.20070905150504
receiving file list ... done
deleting segments_1ine
deleting _164h_1.del
deleting _164h.tis
deleting _164h.tii
deleting _164h.prx
deleting _164h.nrm
deleting _164h.frq
deleting _164h.fnm
deleting _164h.fdx
deleting _164h.fdt
deleting _164g_1.del
deleting _164g.tis
deleting _164g.tii
deleting _164g.prx
deleting _164g.nrm
deleting _164g.frq
deleting _164g.fnm
deleting _164g.fdx
deleting _164g.fdt
deleting _164f_1.del
deleting _164f.tis
deleting _164f.tii
deleting _164f.prx
deleting _164f.nrm
deleting _164f.frq
deleting _164f.fnm
deleting _164f.fdx
deleting _164f.fdt
deleting _164e_1.del
deleting _164e.tis
deleting _164e.tii
deleting _164e.prx
deleting _164e.nrm
deleting _164e.frq
deleting _164e.fnm
deleting _164e.fdx
deleting _164e.fdt
deleting _164d_1.del
deleting _164d.tis
deleting _164d.tii
deleting _164d.prx
deleting _164d.nrm
deleting _164d.frq
deleting _164d.fnm
deleting _164d.fdx
deleting _164d.fdt
deleting _164c_1.del
deleting _164c.tis
deleting _164c.tii
deleting _164c.prx
deleting _164c.nrm
deleting _164c.frq
deleting _164c.fnm
deleting _164c.fdx
deleting _164c.fdt
deleting _164b_1.del
deleting _164b.tis
deleting _164b.tii
deleting _164b.prx
deleting _164b.nrm
deleting _164b.frq
deleting _164b.fnm
deleting _164b.fdx
deleting _164b.fdt
deleting _164a_1.del
deleting _164a.tis
deleting _164a.tii
deleting _164a.prx
deleting _164a.nrm
deleting _164a.frq
deleting _164a.fnm
deleting _164a.fdx
deleting _164a.fdt
deleting _163z_3.del
deleting _163z.tis
deleting _163z.tii
deleting _163z.prx
deleting _163z.nrm
deleting _163z.frq
deleting _163z.fnm
deleting _163z.fdx
deleting _163z.fdt
deleting _163o_3.del
deleting _163o.tis
deleting _163o.tii
deleting _163o.prx
deleting _163o.nrm
deleting _163o.frq
deleting _163o.fnm
deleting _163o.fdx
deleting _163o.fdt
deleting _163d_4.del
deleting _163d.tis
deleting _163d.tii
deleting _163d.prx
deleting _163d.nrm
deleting _163d.frq
deleting _163d.fnm
deleting _163d.fdx
deleting _163d.fdt
deleting _1632_6.del
deleting _1632.tis
deleting _1632.tii
deleting _1632.prx
deleting _1632.nrm
deleting _1632.frq
deleting _1632.fnm
deleting _1632.fdx
deleting _1632.fdt
deleting _162r_7.del
deleting _162r.tis
deleting _162r.tii
deleting _162r.prx
deleting _162r.nrm
deleting _162r.frq
deleting _162r.fnm
deleting _162r.fdx
deleting _162r.fdt
deleting _162g_d.del
deleting _162g.tis
deleting _162g.tii
deleting _162g.prx
deleting _162g.nrm
deleting 

Re: Replication broken.. no helpful errors?

2007-09-06 Thread Yonik Seeley
On 9/6/07, Matthew Runo [EMAIL PROTECTED] wrote:
 The thing is that a new searcher is not opened if I look in the
 stats.jsp page. The index version never changes.

The index version is read from the index... hence if the lucene index
doesn't change (even if a ew snapshot was taken), the version won't
change even if a new searcher was opened.

Is the problem on the master side now since it looks like the slave is
pulling a temp-snapshot?

-Yonik


Re: Replication broken.. no helpful errors?

2007-09-05 Thread Matthew Runo
It seems that the scripts cannot open new searchers at the end of the  
process, for some reason. Here's a message from cron, but I'm not  
sure what to make of it... It looks like the files properly copied  
over, but failed the install. I removed the temp* directory, but  
still SOLR could not launch a new searcher. I don't see any activity  
in catalina.out though...



started by tomcat5
command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/ 
data -S /opt/solr/logs -d /opt/solr/data -v

pulling snapshot temp-snapshot.20070905150504
receiving file list ... done
deleting segments_1ine
deleting _164h_1.del
deleting _164h.tis
deleting _164h.tii
deleting _164h.prx
deleting _164h.nrm
deleting _164h.frq
deleting _164h.fnm
deleting _164h.fdx
deleting _164h.fdt
deleting _164g_1.del
deleting _164g.tis
deleting _164g.tii
deleting _164g.prx
deleting _164g.nrm
deleting _164g.frq
deleting _164g.fnm
deleting _164g.fdx
deleting _164g.fdt
deleting _164f_1.del
deleting _164f.tis
deleting _164f.tii
deleting _164f.prx
deleting _164f.nrm
deleting _164f.frq
deleting _164f.fnm
deleting _164f.fdx
deleting _164f.fdt
deleting _164e_1.del
deleting _164e.tis
deleting _164e.tii
deleting _164e.prx
deleting _164e.nrm
deleting _164e.frq
deleting _164e.fnm
deleting _164e.fdx
deleting _164e.fdt
deleting _164d_1.del
deleting _164d.tis
deleting _164d.tii
deleting _164d.prx
deleting _164d.nrm
deleting _164d.frq
deleting _164d.fnm
deleting _164d.fdx
deleting _164d.fdt
deleting _164c_1.del
deleting _164c.tis
deleting _164c.tii
deleting _164c.prx
deleting _164c.nrm
deleting _164c.frq
deleting _164c.fnm
deleting _164c.fdx
deleting _164c.fdt
deleting _164b_1.del
deleting _164b.tis
deleting _164b.tii
deleting _164b.prx
deleting _164b.nrm
deleting _164b.frq
deleting _164b.fnm
deleting _164b.fdx
deleting _164b.fdt
deleting _164a_1.del
deleting _164a.tis
deleting _164a.tii
deleting _164a.prx
deleting _164a.nrm
deleting _164a.frq
deleting _164a.fnm
deleting _164a.fdx
deleting _164a.fdt
deleting _163z_3.del
deleting _163z.tis
deleting _163z.tii
deleting _163z.prx
deleting _163z.nrm
deleting _163z.frq
deleting _163z.fnm
deleting _163z.fdx
deleting _163z.fdt
deleting _163o_3.del
deleting _163o.tis
deleting _163o.tii
deleting _163o.prx
deleting _163o.nrm
deleting _163o.frq
deleting _163o.fnm
deleting _163o.fdx
deleting _163o.fdt
deleting _163d_4.del
deleting _163d.tis
deleting _163d.tii
deleting _163d.prx
deleting _163d.nrm
deleting _163d.frq
deleting _163d.fnm
deleting _163d.fdx
deleting _163d.fdt
deleting _1632_6.del
deleting _1632.tis
deleting _1632.tii
deleting _1632.prx
deleting _1632.nrm
deleting _1632.frq
deleting _1632.fnm
deleting _1632.fdx
deleting _1632.fdt
deleting _162r_7.del
deleting _162r.tis
deleting _162r.tii
deleting _162r.prx
deleting _162r.nrm
deleting _162r.frq
deleting _162r.fnm
deleting _162r.fdx
deleting _162r.fdt
deleting _162g_d.del
deleting _162g.tis
deleting _162g.tii
deleting _162g.prx
deleting _162g.nrm
deleting _162g.frq
deleting _162g.fnm
deleting _162g.fdx
deleting _162g.fdt
deleting _1625_m.del
deleting _1625.tis
deleting _1625.tii
deleting _1625.prx
deleting _1625.nrm
deleting _1625.frq
deleting _1625.fnm
deleting _1625.fdx
deleting _1625.fdt
deleting _161u_w.del
deleting _161u.tis
deleting _161u.tii
deleting _161u.prx
deleting _161u.nrm
deleting _161u.frq
deleting _161u.fnm
deleting _161u.fdx
deleting _161u.fdt
deleting _161j_16.del
./
_161j_17.del
_164m.fdt
_164m.fdx
_164m.fnm
_164m.frq
_164m.nrm
_164m.prx
_164m.tii
_164m.tis
_164m_1.del
_164x.fdt
_164x.fdx
_164x.fnm
_164x.frq
_164x.nrm
_164x.prx
_164x.tii
_164x.tis
_164x_1.del
segments.gen
segments_1inv

sent 516 bytes  received 105864302 bytes  30247090.86 bytes/sec
total size is 966107226  speedup is 9.13
+ [[ -z search1 ]]
+ [[ -z /opt/solr/logs ]]
+ fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V
+ [[ -z tomcat5 ]]
++ whoami
+ [[ tomcat5 != tomcat5 ]]
++ who -m
++ cut '-d ' -f1
++ sed '-es/^.*!//'
+ oldwhoami=
+ [[ '' == '' ]]
+++ pgrep -g0 snapinstaller
++ tail -1
++ cut -f1 '-d '
++ ps h -Hfp 3621 3629 3630 3631
+ oldwhoami=tomcat5
+ [[ -z /opt/solr/data ]]
++ echo /opt/solr/data
++ cut -c1
+ [[ / != \/ ]]
++ echo /opt/solr/logs
++ cut -c1
+ [[ / != \/ ]]
++ date +%s
+ start=1189030205
+ logMessage started by tomcat5
++ timeStamp
++ date '+%Y/%m/%d %H:%M:%S'
+ echo 2007/09/05 15:10:05 started by tomcat5
+ [[ -n '' ]]
+ logMessage command: /opt/solr/bin/snapinstaller -M search1 -S /opt/ 
solr/logs -d /opt/solr/data -V

++ timeStamp
++ date '+%Y/%m/%d %H:%M:%S'
+ echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M  
search1 -S /opt/solr/logs -d /opt/solr/data -V

+ [[ -n '' ]]
++ ls /opt/solr/data
++ grep 'snapshot\.'
++ grep -v wip
++ sort -r
++ head -1
+ name=temp-snapshot.20070905150504
+ trap 'echo caught INT/TERM, exiting now but partial installation  
may have already occured;/bin/rm -rf ${data_dir/index.tmp$$;logExit  
aborted 13' INT TERM

+ [[ temp-snapshot.20070905150504 == '' ]]
+ 

Re: Replication broken.. no helpful errors?

2007-09-05 Thread Matthew Runo

If it helps anyone, this index is around a gig in size.

++
 | Matthew Runo
 | Zappos Development
 | [EMAIL PROTECTED]
 | 702-943-7833
++


On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote:

It seems that the scripts cannot open new searchers at the end of  
the process, for some reason. Here's a message from cron, but I'm  
not sure what to make of it... It looks like the files properly  
copied over, but failed the install. I removed the temp* directory,  
but still SOLR could not launch a new searcher. I don't see any  
activity in catalina.out though...



started by tomcat5
command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/ 
data -S /opt/solr/logs -d /opt/solr/data -v

pulling snapshot temp-snapshot.20070905150504
receiving file list ... done
deleting segments_1ine
deleting _164h_1.del
deleting _164h.tis
deleting _164h.tii
deleting _164h.prx
deleting _164h.nrm
deleting _164h.frq
deleting _164h.fnm
deleting _164h.fdx
deleting _164h.fdt
deleting _164g_1.del
deleting _164g.tis
deleting _164g.tii
deleting _164g.prx
deleting _164g.nrm
deleting _164g.frq
deleting _164g.fnm
deleting _164g.fdx
deleting _164g.fdt
deleting _164f_1.del
deleting _164f.tis
deleting _164f.tii
deleting _164f.prx
deleting _164f.nrm
deleting _164f.frq
deleting _164f.fnm
deleting _164f.fdx
deleting _164f.fdt
deleting _164e_1.del
deleting _164e.tis
deleting _164e.tii
deleting _164e.prx
deleting _164e.nrm
deleting _164e.frq
deleting _164e.fnm
deleting _164e.fdx
deleting _164e.fdt
deleting _164d_1.del
deleting _164d.tis
deleting _164d.tii
deleting _164d.prx
deleting _164d.nrm
deleting _164d.frq
deleting _164d.fnm
deleting _164d.fdx
deleting _164d.fdt
deleting _164c_1.del
deleting _164c.tis
deleting _164c.tii
deleting _164c.prx
deleting _164c.nrm
deleting _164c.frq
deleting _164c.fnm
deleting _164c.fdx
deleting _164c.fdt
deleting _164b_1.del
deleting _164b.tis
deleting _164b.tii
deleting _164b.prx
deleting _164b.nrm
deleting _164b.frq
deleting _164b.fnm
deleting _164b.fdx
deleting _164b.fdt
deleting _164a_1.del
deleting _164a.tis
deleting _164a.tii
deleting _164a.prx
deleting _164a.nrm
deleting _164a.frq
deleting _164a.fnm
deleting _164a.fdx
deleting _164a.fdt
deleting _163z_3.del
deleting _163z.tis
deleting _163z.tii
deleting _163z.prx
deleting _163z.nrm
deleting _163z.frq
deleting _163z.fnm
deleting _163z.fdx
deleting _163z.fdt
deleting _163o_3.del
deleting _163o.tis
deleting _163o.tii
deleting _163o.prx
deleting _163o.nrm
deleting _163o.frq
deleting _163o.fnm
deleting _163o.fdx
deleting _163o.fdt
deleting _163d_4.del
deleting _163d.tis
deleting _163d.tii
deleting _163d.prx
deleting _163d.nrm
deleting _163d.frq
deleting _163d.fnm
deleting _163d.fdx
deleting _163d.fdt
deleting _1632_6.del
deleting _1632.tis
deleting _1632.tii
deleting _1632.prx
deleting _1632.nrm
deleting _1632.frq
deleting _1632.fnm
deleting _1632.fdx
deleting _1632.fdt
deleting _162r_7.del
deleting _162r.tis
deleting _162r.tii
deleting _162r.prx
deleting _162r.nrm
deleting _162r.frq
deleting _162r.fnm
deleting _162r.fdx
deleting _162r.fdt
deleting _162g_d.del
deleting _162g.tis
deleting _162g.tii
deleting _162g.prx
deleting _162g.nrm
deleting _162g.frq
deleting _162g.fnm
deleting _162g.fdx
deleting _162g.fdt
deleting _1625_m.del
deleting _1625.tis
deleting _1625.tii
deleting _1625.prx
deleting _1625.nrm
deleting _1625.frq
deleting _1625.fnm
deleting _1625.fdx
deleting _1625.fdt
deleting _161u_w.del
deleting _161u.tis
deleting _161u.tii
deleting _161u.prx
deleting _161u.nrm
deleting _161u.frq
deleting _161u.fnm
deleting _161u.fdx
deleting _161u.fdt
deleting _161j_16.del
./
_161j_17.del
_164m.fdt
_164m.fdx
_164m.fnm
_164m.frq
_164m.nrm
_164m.prx
_164m.tii
_164m.tis
_164m_1.del
_164x.fdt
_164x.fdx
_164x.fnm
_164x.frq
_164x.nrm
_164x.prx
_164x.tii
_164x.tis
_164x_1.del
segments.gen
segments_1inv

sent 516 bytes  received 105864302 bytes  30247090.86 bytes/sec
total size is 966107226  speedup is 9.13
+ [[ -z search1 ]]
+ [[ -z /opt/solr/logs ]]
+ fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V
+ [[ -z tomcat5 ]]
++ whoami
+ [[ tomcat5 != tomcat5 ]]
++ who -m
++ cut '-d ' -f1
++ sed '-es/^.*!//'
+ oldwhoami=
+ [[ '' == '' ]]
+++ pgrep -g0 snapinstaller
++ tail -1
++ cut -f1 '-d '
++ ps h -Hfp 3621 3629 3630 3631
+ oldwhoami=tomcat5
+ [[ -z /opt/solr/data ]]
++ echo /opt/solr/data
++ cut -c1
+ [[ / != \/ ]]
++ echo /opt/solr/logs
++ cut -c1
+ [[ / != \/ ]]
++ date +%s
+ start=1189030205
+ logMessage started by tomcat5
++ timeStamp
++ date '+%Y/%m/%d %H:%M:%S'
+ echo 2007/09/05 15:10:05 started by tomcat5
+ [[ -n '' ]]
+ logMessage command: /opt/solr/bin/snapinstaller -M search1 -S / 
opt/solr/logs -d /opt/solr/data -V

++ timeStamp
++ date '+%Y/%m/%d %H:%M:%S'
+ echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M  
search1 -S /opt/solr/logs -d /opt/solr/data -V

+ [[ -n '' ]]
++ ls /opt/solr/data
++ grep 'snapshot\.'
++ grep -v 

Replication broken.. no helpful errors?

2007-08-31 Thread Matthew Runo

Hello!

On a somewhat related note, our replication seems very much broken.  
I've added -v to all my cron jobs, and I think I've seen the error  
(below).


As you can see, it's rsyncing an updated index, but then doesn't seem  
to know to install it. I'm not sure why though.. no errors are  
reported anywhere via the -v. Any help would be most appreciated, I'm  
sure I'm just missing something.   You can see the cronjob command in  
the subject of the forwarded message.


++
 | Matthew Runo
 | Zappos Development
 | [EMAIL PROTECTED]
 | 702-943-7833
++


Begin forwarded message:


From: [EMAIL PROTECTED] (Cron Daemon)
Date: August 31, 2007 1:02:36 PM PDT
To: [EMAIL PROTECTED]
Subject: Cron [EMAIL PROTECTED] /opt/solr/bin/snappuller -M search1  
-P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data -v;/ 
opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/ 
solr/data -v


started by tomcat5
command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/ 
data -S /opt/solr/logs -d /opt/solr/data -v

pulling snapshot snapshot.20070831130005
receiving file list ... done
deleting segments_19zm
deleting _10bg.tis
deleting _10bg.tii
deleting _10bg.prx
deleting _10bg.nrm
deleting _10bg.frq
deleting _10bg.fnm
deleting _10bg.fdx
deleting _10bg.fdt
./
_14ff.fdt
_14ff.fdx
_14ff.fnm
_14ff.frq
_14ff.nrm
_14ff.prx
_14ff.tii
_14ff.tis
_14ff_9.del
_14fq.fdt
_14fq.fdx
_14fq.fnm
_14fq.frq
_14fq.nrm
_14fq.prx
_14fq.tii
_14fq.tis
_14fq_3.del
_14fr.fdt
_14fr.fdx
_14fr.fnm
_14fr.frq
_14fr.nrm
_14fr.prx
_14fr.tii
_14fr.tis
_14fr_2.del
_14fs.fdt
_14fs.fdx
_14fs.fnm
_14fs.frq
_14fs.nrm
_14fs.prx
_14fs.tii
_14fs.tis
_14fs_1.del
segments.gen
segments_1fza
write.lock

sent 871 bytes  received 843185604 bytes  24440187.68 bytes/sec
total size is 843080453  speedup is 1.00
started by tomcat5
command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs - 
d /opt/solr/data -v
latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already  
installed






Re: Replication broken.. no helpful errors?

2007-08-31 Thread Bill Au
 latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
 installed

It looks like you have a directory named temp-snapshot.20070816120113
in your data directory.  You should remove it.  One of the other
script might have left that behind somehow.

I will update the snapinstaller script to ignore non-snapshot when
looking for the latest snapshot to install.

Bill

On 8/31/07, Matthew Runo [EMAIL PROTECTED] wrote:
 Hello!

 On a somewhat related note, our replication seems very much broken.
 I've added -v to all my cron jobs, and I think I've seen the error
 (below).

 As you can see, it's rsyncing an updated index, but then doesn't seem
 to know to install it. I'm not sure why though.. no errors are
 reported anywhere via the -v. Any help would be most appreciated, I'm
 sure I'm just missing something.   You can see the cronjob command in
 the subject of the forwarded message.

 ++
   | Matthew Runo
   | Zappos Development
   | [EMAIL PROTECTED]
   | 702-943-7833
 ++


 Begin forwarded message:

  From: [EMAIL PROTECTED] (Cron Daemon)
  Date: August 31, 2007 1:02:36 PM PDT
  To: [EMAIL PROTECTED]
  Subject: Cron [EMAIL PROTECTED] /opt/solr/bin/snappuller -M search1
  -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data -v;/
  opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/
  solr/data -v
 
  started by tomcat5
  command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
  data -S /opt/solr/logs -d /opt/solr/data -v
  pulling snapshot snapshot.20070831130005
  receiving file list ... done
  deleting segments_19zm
  deleting _10bg.tis
  deleting _10bg.tii
  deleting _10bg.prx
  deleting _10bg.nrm
  deleting _10bg.frq
  deleting _10bg.fnm
  deleting _10bg.fdx
  deleting _10bg.fdt
  ./
  _14ff.fdt
  _14ff.fdx
  _14ff.fnm
  _14ff.frq
  _14ff.nrm
  _14ff.prx
  _14ff.tii
  _14ff.tis
  _14ff_9.del
  _14fq.fdt
  _14fq.fdx
  _14fq.fnm
  _14fq.frq
  _14fq.nrm
  _14fq.prx
  _14fq.tii
  _14fq.tis
  _14fq_3.del
  _14fr.fdt
  _14fr.fdx
  _14fr.fnm
  _14fr.frq
  _14fr.nrm
  _14fr.prx
  _14fr.tii
  _14fr.tis
  _14fr_2.del
  _14fs.fdt
  _14fs.fdx
  _14fs.fnm
  _14fs.frq
  _14fs.nrm
  _14fs.prx
  _14fs.tii
  _14fs.tis
  _14fs_1.del
  segments.gen
  segments_1fza
  write.lock
 
  sent 871 bytes  received 843185604 bytes  24440187.68 bytes/sec
  total size is 843080453  speedup is 1.00
  started by tomcat5
  command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -
  d /opt/solr/data -v
  latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
  installed