Yesterday Andrew and I modified the status script to check to see if the beam number is stuck. It looks like it successfully detected a problem and rebooted.
Laura On Tue, May 4, 2010 at 9:28 PM, <[email protected]> wrote: > > SERENDIP V.5 CRITICAL ERROR REPORT > > beam switcher appears to be *STUCK* > Stuck > > dr2 tail looks ok: > Tue May 4 21:19:59 2010 : DSK[0] : FrameSeq is 50814 DataSeq is 183450 > IdleCount is 132636 Corrected Diff is 0 > The BEE2 is responding to pings: > PING beecourageous (192.168.1.86) 56(84) bytes of data. > 64 bytes from beecourageous (192.168.1.86): icmp_seq=1 ttl=64 time=0.540 ms > > --- beecourageous ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.540/0.540/0.540/0.000 ms > The iBOB is responding to pings: > PING ddc (192.168.2.6) 56(84) bytes of data. > 64 bytes from ddc (192.168.2.6): icmp_seq=1 ttl=64 time=1.09 ms > > --- ddc ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 1.094/1.094/1.094/0.000 ms > The data collection process is running as of Tuesday 04th May 2010 09:20:01 PM > Attempting to restart the BEE... > > Power off command output: > > Power on command output: > Attempting to restart the IBOB... > > Power off command output: > > Power on command output: > > Current Status: 1 ON > 2 OFF > 3 ON > 4 ON > 5 ON > 6 OFF > 7 ON > 8 OFF > > Sleeping 60 seconds to wait for fpga devices to come back up... > > !!Success!! The iBOB has awoken: > PING ddc (192.168.2.6) 56(84) bytes of data. > 64 bytes from ddc (192.168.2.6): icmp_seq=1 ttl=64 time=1.10 ms > > --- ddc ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 1.107/1.107/1.107/0.000 ms > !!Success!! The BEE2 lives! > Sleeping 5 minutes to let the BEE2 boot... > Attempting to restart the whole shebang... > Check output below: > root > copying dr2_config_short into dr2_config > copying old output files for safekeeping > killing any previous setispec_dr runs > killing old disk buf collector... > killing previous ssh instances to run sendstatus on o...@beecourageous > initalizing the iBOB > > 7 beam, 2 pol > Number of beams: 7 > Number of pols: 2 > Number of cycles: 1 > Dwelltime: 1 > Max address: 14 > > > 0x0000 / 00000 -> 0x000000ED / 0b00000000000000000000000011101101 / 0000000237 > > > > 0x0001 / 00001 -> 0x000000EB / 0b00000000000000000000000011101011 / 0000000235 > > > > 0x0002 / 00002 -> 0x000000E7 / 0b00000000000000000000000011100111 / 0000000231 > > > > 0x0003 / 00003 -> 0x000000DE / 0b00000000000000000000000011011110 / 0000000222 > > > > 0x0004 / 00004 -> 0x000000DD / 0b00000000000000000000000011011101 / 0000000221 > > > > 0x0005 / 00005 -> 0x000000DB / 0b00000000000000000000000011011011 / 0000000219 > > > > 0x0006 / 00006 -> 0x000000D7 / 0b00000000000000000000000011010111 / 0000000215 > > > > 0x0007 / 00007 -> 0x000000BE / 0b00000000000000000000000010111110 / 0000000190 > > > > 0x0008 / 00008 -> 0x000000BD / 0b00000000000000000000000010111101 / 0000000189 > > > > 0x0009 / 00009 -> 0x000000BB / 0b00000000000000000000000010111011 / 0000000187 > > > > 0x000A / 00010 -> 0x000000B7 / 0b00000000000000000000000010110111 / 0000000183 > > > > 0x000B / 00011 -> 0x0000007E / 0b00000000000000000000000001111110 / 0000000126 > > > > 0x000C / 00012 -> 0x0000007D / 0b00000000000000000000000001111101 / 0000000125 > > > > 0x000D / 00013 -> 0x0000007B / 0b00000000000000000000000001111011 / 0000000123 > > > > 0x000E / 00014 -> 0x00000000 / 0b00000000000000000000000000000000 / 0000000000 > > > rebooting the bee2 > tcgetattr: Inappropriate ioctl for device > killing previous sendstatus > deleting previous nohup file > killing running bofs > loading new bofs > nohup: appending output to `nohup.out' > nohup: nohup: nohup: appending output to `nohup.out' > appending output to `nohup.out' > appending output to `nohup.out' > configuring chips > Setting the maximum number of hits to: 25 > Setting the scale threshold to 80 > done > Connection to beecourageous closed. > starting sendstatus > starting diskbuf cleaner > starting new run > imdonenow > > > Disk Usage: > Filesystem Size Used Avail Use% Mounted on > /dev/md1 226G 98G 128G 44% / > tmpfs 2.0G 0 2.0G 0% /lib/init/rw > udev 10M 108K 9.9M 2% /dev > tmpfs 2.0G 0 2.0G 0% /dev/shm > /dev/md0 236M 35M 190M 16% /boot > /dev/sdc1 1.4T 147G 1.3T 11% /mockdata > > > >
