CentOS (RH lite) 5 with TSM 7.1.4.4 client User called about an archive that keeps aborting/failing with no real details. Sometimes the archive will run for a while and sometimes fail pretty quickly.
04/28/2016 17:39:50 ANS4023E Error processing '/home1': file input/output error 04/28/2016 17:41:30 ANS1999E Archive processing of '/home1/ngandhi/*' stopped. 04/28/2016 17:41:30 ANS4023E Error processing '/home1': file input/output error 04/28/2016 17:56:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down. 04/28/2016 17:56:44 ANS0361I DIAG: linux/psunxthr.cpp ( 207): unknown thread, fatal error, signal 11 04/28/2016 17:57:22 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down. 04/28/2016 17:57:30 ANS0361I DIAG: linux/psunxthr.cpp ( 207): Unknown thread, fatal error, signal 11 04/28/2016 17:58:36 ANS1999E Archive processing of '/home1/ngandhi/*' stopped. 04/28/2016 17:58:36 ANS4023E Error processing '/home1': file input/output error Turned on tracing and after lots of different Googling, I guessed it has something to do with symbolic links (this users has lots of them) and the filesystem is NFS. The user confirmed this. They also did all kinds of verification that there aren't any problems with the symbolic links or the filesystem. They can't use archsyml=no since these are end-of-semester archiving and if needed, the users would have to have it back the way they left it. Problem is, all hits that describe this issue are from 6.x days and say they are resolved. 04/29/2016 11:34:40.318 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /home1/ngandhi/AT/at_tetra.qsu b.69111/all_tetrasacc_m203/gold_all_tetrasacc_m203.mol2.gz 04/29/2016 11:34:40.318 : unxfilio.cpp (2796): fioGetAttrib: type: 1111 size: 3148 for /home1/ngandhi/AT/ at_tetra.qsub.69111/all_tetrasacc_m203/gold_all_tetrasacc_m203.mol2.gz 04/29/2016 11:34:40.318 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /home1/ngandhi/AT/at_tetra.qsu b.69111/all_tetrasacc_m203/ranked_all_tetrasacc_m203_1.mol2 04/29/2016 11:34:40.453 : unxfilio.cpp (2348): fioGetAttrib: stat error for /home1/ngandhi/AT/at_tetra.qs ub.69111/all_tetrasacc_m203/ranked_all_tetrasacc_m203_1.mol2, errno = 5 04/29/2016 11:34:40.453 : linux/pserrno.cpp ( 309): TransErrno: Received error from stat, errno = 5, rc = 164 04/29/2016 11:34:40.453 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path / 04/29/2016 11:34:40.453 : virtstat.cpp ( 622): dsmNFSsyscall stat for path / 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /proc 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /proc 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /sys 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /sys 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /tmp 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /tmp 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /usr/global 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /usr/global 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /home 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /home 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /export/global 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /export/global 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /export/home 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /export/home 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /proc/sys/fs/binfmt_misc 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /proc/sys/fs/binfmt_misc 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /ipathfs 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /ipathfs 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /var/lib/nfs/rpc_pipefs 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /var/lib/nfs/rpc_pipefs 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /proc/fs/nfsd 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /proc/fs/nfsd 04/29/2016 11:34:40.454 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /home1 04/29/2016 11:34:40.456 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /home1 04/29/2016 11:34:40.456 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /home2 04/29/2016 11:34:40.456 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /home2 04/29/2016 11:34:40.457 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /home3 04/29/2016 11:34:40.457 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /home3 04/29/2016 11:34:40.457 : virtstat.cpp ( 622): dsmNFSsyscall statfs for path /home1 04/29/2016 11:34:40.458 : virtstat.cpp ( 622): dsmNFSsyscall stat for path /home1 Another odd thing in the trace (above) is when the failure occurs, the client seems to randomly start scanning all kinds of filesystems, not just the ones it is archiving. Thoughts or suggestions ? -- *Zoltan Forray* TSM Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator (in training) Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://infosecurity.vcu.edu/phishing.html