Hi all, It looks like the datarecorder code is indeed crashing as much as is indicated by these emails, failing with something like:
Sun Jan 9 15:17:15 2011 : DSK[0] : RING BUFFER OVERRUN !! 1294600635.770000 1294600631.356000 0.209715 Sun Jan 9 15:17:17 2011 : DSK[0] : RING BUFFER OVERRUN !! 1294600637.062000 1294600635.770000 0.209715 Sun Jan 9 15:17:17 2011 : MEM[0] : Closing down EDT Sun Jan 9 15:17:22 2011 : MEM[0] : Exiting. This has happened in the past when ALFA is misconfigured somehow and we max out hits/bin on all bins. If these messages continue today, we ought to look into this more closely. - Andrew On 1/9/11 9:28 AM, "[email protected]" <[email protected]> wrote: > >SERENDIP V.5 CRITICAL ERROR REPORT > >beam switcher appears to be working... >Same file > >dr2 tail looks ok: >Sun Jan 9 12:15:33 2011 : Idle Watch Thread has rejoined >The BEE2 is responding to pings: >PING beecourageous (192.168.1.86) 56(84) bytes of data. >64 bytes from beecourageous (192.168.1.86): icmp_seq=1 ttl=64 time=0.354 >ms > >--- beecourageous ping statistics --- >1 packets transmitted, 1 received, 0% packet loss, time 0ms >rtt min/avg/max/mdev = 0.354/0.354/0.354/0.000 ms >The iBOB is responding to pings: >PING ddc (192.168.2.6) 56(84) bytes of data. >64 bytes from ddc (192.168.2.6): icmp_seq=1 ttl=64 time=7.98 ms > >--- ddc ping statistics --- >1 packets transmitted, 1 received, 0% packet loss, time 0ms >rtt min/avg/max/mdev = 7.981/7.981/7.981/0.000 ms > > >*****The data collection process is **NOT RUNNING** as of Sunday 09th >January 2011 12:20:01 PM >Attempting to restart the BEE... > > Power off command output: > > Power on command output: >Attempting to restart the IBOB... > > Power off command output: > > Power on command output: > > Current Status: 1 ON >2 OFF >3 ON >4 ON >5 ON >6 OFF >7 ON >8 OFF > >Sleeping 60 seconds to wait for fpga devices to come back up... > >!!Success!! The iBOB has awoken: >PING ddc (192.168.2.6) 56(84) bytes of data. >64 bytes from ddc (192.168.2.6): icmp_seq=1 ttl=64 time=9.15 ms > >--- ddc ping statistics --- >1 packets transmitted, 1 received, 0% packet loss, time 0ms >rtt min/avg/max/mdev = 9.158/9.158/9.158/0.000 ms >!!Success!! The BEE2 lives! >Sleeping 5 minutes to let the BEE2 boot... >Attempting to restart the whole shebang... >Check output below: >root >copying dr2_config_short into dr2_config >copying old output files for safekeeping >killing any previous setispec_dr runs >killing old disk buf collector... >killing previous ssh instances to run sendstatus on o...@beecourageous >initalizing the iBOB > >7 beam, 2 pol >Number of beams: 7 > Number of pols: 2 > Number of cycles: 1 >Dwelltime: 1 >Max address: 14 > > >0x0000 / 00000 -> 0x000000ED / 0b00000000000000000000000011101101 / >0000000237 > > > >0x0001 / 00001 -> 0x000000EB / 0b00000000000000000000000011101011 / >0000000235 > > > >0x0002 / 00002 -> 0x000000E7 / 0b00000000000000000000000011100111 / >0000000231 > > > >0x0003 / 00003 -> 0x000000DE / 0b00000000000000000000000011011110 / >0000000222 > > > >0x0004 / 00004 -> 0x000000DD / 0b00000000000000000000000011011101 / >0000000221 > > > >0x0005 / 00005 -> 0x000000DB / 0b00000000000000000000000011011011 / >0000000219 > > > >0x0006 / 00006 -> 0x000000D7 / 0b00000000000000000000000011010111 / >0000000215 > > > >0x0007 / 00007 -> 0x000000BE / 0b00000000000000000000000010111110 / >0000000190 > > > >0x0008 / 00008 -> 0x000000BD / 0b00000000000000000000000010111101 / >0000000189 > > > >0x0009 / 00009 -> 0x000000BB / 0b00000000000000000000000010111011 / >0000000187 > > > >0x000A / 00010 -> 0x000000B7 / 0b00000000000000000000000010110111 / >0000000183 > > > >0x000B / 00011 -> 0x0000007E / 0b00000000000000000000000001111110 / >0000000126 > > > >0x000C / 00012 -> 0x0000007D / 0b00000000000000000000000001111101 / >0000000125 > > > >0x000D / 00013 -> 0x0000007B / 0b00000000000000000000000001111011 / >0000000123 > > > >0x000E / 00014 -> 0x00000000 / 0b00000000000000000000000000000000 / >0000000000 > > >rebooting the bee2 >tcgetattr: Inappropriate ioctl for device >killing previous sendstatus >deleting previous nohup file >killing running bofs >loading new bofs >nohup: appending output to `nohup.out' >nohup: appending output to `nohup.out' >nohup: appending output to `nohup.out' >nohup: appending output to `nohup.out' >configuring chips >Setting the maximum number of hits to: 25 >Setting the scale threshold to 80 >done >Connection to beecourageous closed. >starting sendstatus >starting diskbuf cleaner >starting new run >imdonenow > > >Disk Usage: >Filesystem Size Used Avail Use% Mounted on >/dev/md1 226G 195G 32G 87% / >tmpfs 2.0G 0 2.0G 0% /lib/init/rw >udev 10M 104K 9.9M 2% /dev >tmpfs 2.0G 0 2.0G 0% /dev/shm >/dev/md0 236M 42M 182M 19% /boot >/dev/sdc1 1.4T 192G 1.2T 14% /mockdata > > >
