RE: unexplained system hangs - possible smbfs issue ??
--- The information transmitted in this e-mail is for the exclusive use of the intended addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. E-mails may not be secure, may contain computer viruses and may be corrupted in transmission. Please carefully check this e-mail (and any attachment) accordingly. No warranties are given and no liability is accepted for any loss or damage caused by such matters. --- ***This Email has been scanned for Viruses by MailMarshal.*** -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Garrett Cooper Sent: Monday, October 10, 2005 3:35 PM To: freebsd-questions@freebsd.org Subject: Re: unexplained system hangs - possible smbfs issue ?? Murray, Have you thought of looking into filing a bug report with the Samba people (http://www.samba.org/)? This may be an issue with either your client program, or the SMB implementation in Win2k3, which can be solved by getting the ball rolling with SMB and/or possibly MS. Either way, that is quite a few files to have to parse through, and although it may seem somewhat ludicrous, adding an additional script to presort out your minute reports would greatly reduce the amount of open-file records you need, and while that may not be a permanent solution it can serve as a better base for sorting your data. You could just create proper directories on the Win2k3 server, like %BASE_DIR%\Year\Day\Hour, if you get a large volume of files, or just strictly put them in a daily directory since it sounds like your volume is manageable. Plus, it's probably easier for humans to manage as opposed to 2000+ flat files in the same directory ;). Any SQL would handle this issue nicely as well since one of databases' best selling points is this type of application. -Garrett ___ Garrett, Thanks for your input, as it all ties things up with the observed problem which we have been slowly closing in on by a process of isolating processes onto a 'sacrifical' host. BTW it still seems to be a bit time dependant, unless the comments regarding the extra files of zero length mentioned in other replies and PR's apply here. Our test bed _never_ crashed under high load testing (20K+ files grown incrementally). Maybe the test bed always had 'appropriate' files and/or file structures. (sigh) (really silly thought - I wonder if it is as 'simple' as needing an even / odd file count when the count gets high?? ) A 'move_files to dated directory' process is being built within the main process as a final operation for cleanup. This will mean that the smbfs will never have more than 10 files or so in any given 1 minute cycle. cheers mjt ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: unexplained system hangs - possible smbfs issue ??
On Oct 9, 2005, at 5:14 PM, Murray Taylor wrote: Hi all, I have a system using smbfs that locks up every few hours. The system is running a cron job every minute, that opens a directory that is actually a smbfs mounted directory on a Windows Server 2003 machine. The program reads the directory looking for a particular file pattern, opens and processes the file then closes and renames the file. What this means is that the smnfs mounted directory file count constantly increases. The hang mentioned is just that... the cron job doesnt run, you cannot login on the console (well you can type root and enter and thats it, no password prompt appears). If there is an open console session, a single command can be issued then the console freezes. The only recovery is a power cycle. We have run the programs under a test framework to do a month load of files in an hour, and cant hang things, yet the production box (which is technically a better machine, but runs the same OS versions as the testbed) will hang under 'normal' use every few hours. Its almost clock regular, except that it is slowly reducing the time between hangs. We have just purged the Windows directory, and it seems so far to have extended the hang window. Are there any known issues with smbfs and large directories??? NB We have other processes that use smbfs without the large number of files on the Win systems and these have run ok for years. It will hang on both these production versions. FreeBSD xxxhostnamexxx 4.10-RELEASE FreeBSD 4.10-RELEASE #0: Tue May 25 22:47:12 GMT 2004 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC i386 FreeBSD xxxhostnamexxx 4.10-RELEASE-p7 FreeBSD 4.10-RELEASE-p7 #3: Thu Apr 14 15:34:37 EST 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SVMYSQL3 i386 mjt -- A: Because it fouls the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? Murray Taylor Bytecraft Systems P: +61 3 8710 2555 F: +61 3 8710 2599 D: +61 3 9238 4275 E: [EMAIL PROTECTED] Those are a lot of files =\... a temporary solution might be to compress old log files past a particular date, or use an actual database system, but yeah... I could see something odd occurring with the FreeBSD machine. However, you didn't provide a lot of information about the FBSD machine. What are the specs for it, hardware-wise? Just curious. Also, what's the approximate amount of files in the directory in question? -Garrett ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: unexplained system hangs - possible smbfs issue ??
--- The information transmitted in this e-mail is for the exclusive use of the intended addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. E-mails may not be secure, may contain computer viruses and may be corrupted in transmission. Please carefully check this e-mail (and any attachment) accordingly. No warranties are given and no liability is accepted for any loss or damage caused by such matters. --- ***This Email has been scanned for Viruses by MailMarshal.*** -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Garrett Cooper Sent: Monday, October 10, 2005 11:45 AM To: FreeBSD Questions Subject: Re: unexplained system hangs - possible smbfs issue ?? On Oct 9, 2005, at 5:14 PM, Murray Taylor wrote: Hi all, I have a system using smbfs that locks up every few hours. The system is running a cron job every minute, that opens a directory that is actually a smbfs mounted directory on a Windows Server 2003 machine. The program reads the directory looking for a particular file pattern, opens and processes the file then closes and renames the file. What this means is that the smnfs mounted directory file count constantly increases. The hang mentioned is just that... the cron job doesnt run, you cannot login on the console (well you can type root and enter and thats it, no password prompt appears). If there is an open console session, a single command can be issued then the console freezes. The only recovery is a power cycle. We have run the programs under a test framework to do a month load of files in an hour, and cant hang things, yet the production box (which is technically a better machine, but runs the same OS versions as the testbed) will hang under 'normal' use every few hours. Its almost clock regular, except that it is slowly reducing the time between hangs. We have just purged the Windows directory, and it seems so far to have extended the hang window. Are there any known issues with smbfs and large directories??? NB We have other processes that use smbfs without the large number of files on the Win systems and these have run ok for years. It will hang on both these production versions. FreeBSD xxxhostnamexxx 4.10-RELEASE FreeBSD 4.10-RELEASE #0: Tue May 25 22:47:12 GMT 2004 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC i386 FreeBSD xxxhostnamexxx 4.10-RELEASE-p7 FreeBSD 4.10-RELEASE-p7 #3: Thu Apr 14 15:34:37 EST 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SVMYSQL3 i386 mjt -- A: Because it fouls the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? Murray Taylor Bytecraft Systems P: +61 3 8710 2555 F: +61 3 8710 2599 D: +61 3 9238 4275 E: [EMAIL PROTECTED] Those are a lot of files =\... a temporary solution might be to compress old log files past a particular date, or use an actual database system, but yeah... I could see something odd occurring with the FreeBSD machine. However, you didn't provide a lot of information about the FBSD machine. What are the specs for it, hardware-wise? Just curious. Also, what's the approximate amount of files in the directory in question? -Garrett Hi Garrett, File count is on the order of 2500 and growing... They are actually message files about 2 k long, which have to be kept for 20 days and we get 300 ish per day. I am going to mod the processes to move files through the smbfs link to other holding (rotated) directories on the FBSD box, but am trying to nail down if this file count / access method is the root cause of the problem. The machines are CPU: Intel(R) Pentium(R) 4 CPU 2.00GHz (2000.04-MHz 686-class CPU) Origin = GenuineIntel Id = 0xf24 Stepping = 4 Features=0x3febfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SS E2,SS,HTT,TM real memory = 266801152 (260548K bytes) avail memory = 254156800 (248200K bytes) we have had the same problem on this one CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3200.13-MHz 686-class CPU) Origin = GenuineIntel Id = 0xf41 Stepping = 1 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SS E2,SS,HTT,TM,PBE Hyperthreading: 2 logical CPUs real memory = 2147274752 (2096948K bytes) avail memory = 2088042496 (2039104K bytes) This one was also running the database at the time which got ugly, so
Re: unexplained system hangs - possible smbfs issue ??
Murray, Have you thought of looking into filing a bug report with the Samba people (http://www.samba.org/)? This may be an issue with either your client program, or the SMB implementation in Win2k3, which can be solved by getting the ball rolling with SMB and/or possibly MS. Either way, that is quite a few files to have to parse through, and although it may seem somewhat ludicrous, adding an additional script to presort out your minute reports would greatly reduce the amount of open-file records you need, and while that may not be a permanent solution it can serve as a better base for sorting your data. You could just create proper directories on the Win2k3 server, like %BASE_DIR%\Year\Day\Hour, if you get a large volume of files, or just strictly put them in a daily directory since it sounds like your volume is manageable. Plus, it's probably easier for humans to manage as opposed to 2000+ flat files in the same directory ;). Any SQL would handle this issue nicely as well since one of databases' best selling points is this type of application. -Garrett ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
