Hello! It seems my long run of good fortune has taken a turn for the worse. After years of rock-solid AstLinux reliability, I've run into some trouble. I'm hopeful that your expertise can help me get back to better times.
Sorry... This is a bit of a ramble. But I'd like to provide a detailed view of what we are seeing before soliciting your thoughts. Six hours after putting new hardware into service, I was INUNDATED with file system related error messages in the log. I've pasted a snippet below, which repeated continuously every few seconds for several hours until I shut it down. This is the second time I've encountered this symptom in just a few days. Both events were on virtually on the same hardware, a brand new NF99FL-525. The first time this symptom appeared, I was temporarily using a USB thumb drive in lieu of the recommended Emphase Industrial SATA module. Because I suspected both the USB thumb drive and my twist on the official installation procedure, I later installed the proper SATA module and asked our group for advice regarding the general idea of copying /oldroot/mnt/asturw from one platform to another. (Thanks again for the ideas and comments.) Regardless, the symptom has repeated. Here is a more detailed version of the installation procedure that I used: 1.) Physdiskwrite.exe under Windows to write the SATA image 2.) Boot from SATA on NF99FL-525 3.) Use web interface to create a new /dev/sda2, install sounds, etc. 3.) Use a Linux version of FileZilla to download /oldroot/mnt/asturw from the old Soekris net5501 4.) Use the same Filezilla to upload /oldroot/mnt/asturw to the new NF99FL-525 5.) Reboot 6.) Test 7.) Put in service The first time around (booting from USB last Friday), the unit worked fine for ~36 hours and then starting throwing I/O errors. When I realized this was happening, I remotely requested a reboot. The machine shut down but didn't come back up until I arrived on-site and cycled the power. This time (booting from SATA flash yesterday evening), it ran only ~6 hours before errors appeared. Before rebooting, I downloaded the messages file and noticed these symptoms: When I called into my voice mail, app_voicemail generated this WARNING: "Unable to stream password file". When dialing into our conference bridge, the dial plan tried but failed to Playback sound file "conf-thereare". App_playback WARNED, "ast_streamfile failed for conf-thereare." Upon a power cycle and reboot this morning, I noticed "fsck detected errors on sta2" which it apparently corrected. (It went by in a blur...) After taking the unit out of service, it's been running without I/O errors for a couple of hours. However, it's really not really doing anything either - except bitterly complaining about not having a network route to phones or providers. Here's the log snippet. Imagine many MB of this, repeated over and over again. Roughly 95 percent of the lines that specify a sector number, such as "I/O error, dev sda, sector 524160" show either sector 524160 or 3145616. Every now and again, a few other sector numbers also appear. Nov 29 06:31:13 sip user.info kernel: sd 1:0:0:0: [sda] Unhandled error code Nov 29 06:31:13 sip user.info kernel: sd 1:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Nov 29 06:31:13 sip user.info kernel: sd 1:0:0:0: [sda] CDB: Read(10): 28 00 00 2f ff 90 00 00 08 00 Nov 29 06:31:13 sip user.err kernel: end_request: I/O error, dev sda, sector 3145616 Nov 29 06:31:13 sip user.err kernel: EXT2-fs (sda2): previous I/O error to superblock detected Nov 29 06:31:13 sip user.info kernel: sd 1:0:0:0: [sda] Unhandled error code Nov 29 06:31:13 sip user.info kernel: sd 1:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Nov 29 06:31:13 sip user.info kernel: sd 1:0:0:0: [sda] CDB: Write(10): 2a 00 00 07 ff 80 00 00 08 00 Nov 29 06:31:13 sip user.err kernel: end_request: I/O error, dev sda, sector 524160 Nov 29 06:31:13 sip user.err kernel: Buffer I/O error on device sda2, logical block 0 Nov 29 06:31:13 sip user.warn kernel: lost page write due to I/O error on sda2 Nov 29 06:31:13 sip user.crit kernel: EXT2-fs (sda2): error: ext2_get_inode: unable to read inode block - inode=81936, block=327682 Nov 29 06:31:13 sip user.err kernel: EXT2-fs (sda2): previous I/O error to superblock detected Nov 29 06:31:13 sip user.info kernel: sd 1:0:0:0: [sda] Unhandled error code Nov 29 06:31:13 sip user.info kernel: sd 1:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Nov 29 06:31:13 sip user.info kernel: sd 1:0:0:0: [sda] CDB: Write(10): 2a 00 00 07 ff 80 00 00 08 00 Nov 29 06:31:13 sip user.err kernel: end_request: I/O error, dev sda, sector 524160 Nov 29 06:31:13 sip user.err kernel: Buffer I/O error on device sda2, logical block 0 Nov 29 06:31:13 sip user.warn kernel: lost page write due to I/O error on sda2 Nov 29 06:31:13 sip user.crit kernel: EXT2-fs (sda2): error: ext2_fsync: detected IO error when writing metadata buffers While this really smells like a hardware problem, it seems odd that I could have both a bad USB and bad SATA flash module. Where might I be going wrong? Other than trying the serial version (which I'll probably do this afternoon), what might you do in a situation like this? Thanks for any insight, Dan -----Original Message----- From: Lonnie Abelbeck [mailto:li...@lonnie.abelbeck.com] Sent: Wednesday, November 28, 2012 3:31 PM To: AstLinux Users Mailing List Subject: Re: [Astlinux-users] Backup => Restore to Different Platform? Hi Dan, In addition to the "70-persistent-net.rules" file as Michael pointed out, you may (or many not) have a "/etc/rc.modules" file, you will need the "e1000e" line uncommented for your NF99FL-525. Personally I prefer the geni586-serial version, but the geni586 (VGA) will work fine if you prefer a video console. Lonnie On Nov 28, 2012, at 8:55 AM, Dan Ryson wrote: > All, > > Following David's excellent comment about backups by rsync (pasted below), I've been keeping copies of /oldroot/mnt/asturw. It's a simple, painless backup procedure. > > I'm now replacing a production Soekris net5501 with a shiny new Jetway NF99FL-525. Presuming I start by installing the Generic i586 (VGA) image on the Jetway, is it permissible to simply use ssh to copy the Soekris' asturw file tree to the new machine? If so, this would be a brain-dead simple way to migrate to the different hardware. Or are there differences between the two asturws that will cause pain and agony? > > Thanks for any thoughts, recommendations, or advice. > > Dan > > From: David Kerr [mailto:da...@kerr.net] > Sent: Tuesday, November 20, 2012 12:13 PM > To: AstLinux Users Mailing List > Subject: Re: [Astlinux-users] Mixmonitor storage options > > I actually do a pull from a server (actually a ReadyNAS box) for my AstLinux backup... rsync over ssh with certificates so no userid/password. The remote server has an hourly cron job that runs the rsync.... > > #!/bin/bash > rsync -avz -e "ssh -p<your-ssh-port>" root@<your-pbx-url>:/oldroot/mnt/asturw /c/home/david/PBXbackup > > This way there is no dependency on anything running on the Astlinux box. Something similar would work to pull off call recordings for safe storage. > > David ------------------------------------------------------------------------------ Keep yourself connected to Go Parallel: VERIFY Test and improve your parallel project with help from experts and peers. http://goparallel.sourceforge.net _______________________________________________ Astlinux-users mailing list Astlinux-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/astlinux-users Donations to support AstLinux are graciously accepted via PayPal to pay...@krisk.org.
