Re: Out: 452 Insufficient system storage
On Mar 3, 2010, at 5:24 PM, Wietse Venema wrote: donovan jeffrey j: Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: resource Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: software Mar 3 09:49:59 mx1 postfix/qmgr[603]: 0529299C4604: removed Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 220 [127.0.0.1] ESMTP amavisd-new service ready You need to instrument the smtpd process after the content filter, Wietse This turned out to be an Autowhitelist growing out of control in /var/ clamav/.spamassassin 68816850944 Mar 6 16:39 auto-whitelist I suspect something is up with this heap. -j
Re: Out: 452 Insufficient system storage
On Mar 3, 2010, at 12:09 AM, Wietse Venema wrote: donovan jeffrey j: SNIP 5468961666% 13235578 6836202 66% / And that's 26 GBytes as well. It would be interesting to see what Postfix smtpd logs. You can turn it on selectively postconf -e debug_peer_list=127.0.0.1 postfix reload Then do telnet 127.0.0.1 25, and grep smtpd_check_queue /the/ maillog/file. You can kill the logging with postconf -e debug_peer_list= Wietse i just tried this, i wasn't sure i was reading logs correctly. maillog is also being used by amavis, but i greped out postfix messages to a file, then tailed that file. in the short time I ran this I did not see the smtpd_check_queue . I am out of my maintenance window so I will try it later and let it run longer. Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: resource Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: software Mar 3 09:49:59 mx1 postfix/qmgr[603]: 0529299C4604: removed Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 220 [127.0.0.1] ESMTP amavisd-new service ready Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: EHLO mx1.beth.k12.pa.us Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250- [127.0.0.1] Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250- PIPELINING Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250-SIZE Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250-8BITMIME Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250- ENHANCEDSTATUSCODES Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250 XFORWARD NAME ADDR PROTO HELO Mar 3 09:49:59 mx1 postfix/smtp[1054]: server features: 0x78f size 0 Mar 3 09:49:59 mx1 postfix/smtp[1054]: Using ESMTP PIPELINING, TCP send buffer size is 4096 Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: XFORWARD NAME=mx2.beth.k12.pa.us ADDR=192.168.1.10 Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: XFORWARD PROTO=ESMTP HELO=mx2.beth.k12.pa.us Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: MAIL FROM:owner-macsc...@listserv.dartmouth.edu SIZE=3652 BODY=8BITMIME Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: RCPT TO:jdono...@beth.k12.pa.us Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: DATA Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250 2.5.0 Ok Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250 2.5.0 Ok Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250 2.1.0 Sender owner-macsc...@listserv.dartmouth.edu OK Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 250 2.1.5 Recipient jdono...@beth.k12.pa.us OK Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 354 End data with CRLF.CRLF Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: . Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: QUIT Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: 250 2.6.0 Ok, id=01261-05, from MTA: 250 Ok: queued as 4DDF323BA248 Mar 3 09:49:59 mx1 postfix/smtp[1066]: E1AEF99C4624: to=dro...@beth.k12.pa.us , relay=127.0.0.1[127.0.0.1], delay=3, status=sent (250 2.6.0 Ok, id=01261-05, from MTA: 250 Ok: queued as 4DDF323BA248) Mar 3 09:49:59 mx1 postfix/smtp[1066]: name_mask: resource Mar 3 09:49:59 mx1 postfix/smtp[1066]: name_mask: software Mar 3 09:49:59 mx1 postfix/qmgr[603]: E1AEF99C4624: removed Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: 220 [127.0.0.1] ESMTP amavisd-new service ready Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: EHLO mx1.beth.k12.pa.us Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: 250- [127.0.0.1] Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: 250- PIPELINING Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: 250-SIZE Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: 250-8BITMIME Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: 250- ENHANCEDSTATUSCODES Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: 250 XFORWARD NAME ADDR PROTO HELO Mar 3 09:49:59 mx1 postfix/smtp[1066]: server features: 0x78f size 0 Mar 3 09:49:59 mx1 postfix/smtp[1066]: Using ESMTP PIPELINING, TCP send buffer size is 4096 Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: XFORWARD NAME=mx2.beth.k12.pa.us ADDR=192.168.1.10 Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: XFORWARD PROTO=ESMTP HELO=mail2.beth.k12.pa.us Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: MAIL FROM:gordon.m...@kidspeace.org SIZE=6185 Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: RCPT TO:jm...@bethsd.org Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: DATA Mar 3 09:49:59 mx1 postfix/smtp[1066]: 127.0.0.1[127.0.0.1]: 250 2.5.0 Ok Mar 3 09:49:59 mx1 postfix/smtp[1066]:
Re: Out: 452 Insufficient system storage
donovan jeffrey j: Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: resource Mar 3 09:49:59 mx1 postfix/smtp[1054]: name_mask: software Mar 3 09:49:59 mx1 postfix/qmgr[603]: 0529299C4604: removed Mar 3 09:49:59 mx1 postfix/smtp[1054]: 127.0.0.1[127.0.0.1]: 220 [127.0.0.1] ESMTP amavisd-new service ready You need to instrument the smtpd process after the content filter, Wietse
Re: Out: 452 Insufficient system storage
On Mar 1, 2010, at 10:56 AM, lst_ho...@kwsoft.de wrote: Zitat von donovan jeffrey j dono...@beth.k12.pa.us: Greetings I had several of these on my primary MX this weekend and one just popped up. Can someone explain where this Insufficient system storage is ? both mail queues are empty, and DF shows 20% on the reporting system. Out: 220 mx2.beth.k12.pa.us ESMTP Postfix In: EHLO mx1.beth.k12.pa.us Out: 250-mx2.beth.k12.pa.us Out: 250-PIPELINING Out: 250-SIZE 1024 Out: 250-ETRN Out: 250 8BITMIME In: MAIL FROM:bo-bu7vmghbprcxt5axe5vzqabvadp...@b.e.jcrew.com SIZE=11687 BODY=8BITMIME Out: 452 Insufficient system storage In: RCPT TO:someu...@beth.k12.pa.us Out: 503 Error: need MAIL command In: DATA Out: 503 Error: need RCPT command In: RSET Out: 250 Ok In: QUIT Out: 221 Bye Have a look for queue_minfree parameter in main.cf. The default for Postfix 2.1 and later is 1,5*message_size_limit which can be more than your 20% free if the filesystem the queues reside is rather small. Regards Andreas greetings thanks for the reply queue_minfree = 0 this is default on all my systems. MX1 /dev/disk1s3 77G51G26G66%/ MX2 /dev/disk0s3 234G46G 187G20%/ here is another example; Transcript of session follows. Out: 220 mx1.beth.k12.pa.us ESMTP Postfix In: EHLO MX2.beth.k12.pa.us Out: 250-mx1.beth.k12.pa.us Out: 250-PIPELINING Out: 250-SIZE 1024 Out: 250-ETRN Out: 250 8BITMIME In: MAIL FROM:nore...@somesite.com SIZE=2878 BODY=8BITMIME Out: 452 Insufficient system storage In: RCPT TO:u...@bethsd.org Out: 503 Error: need MAIL command In: DATA Out: 503 Error: need RCPT command In: RSET Out: 250 Ok In: QUIT Out: 221 Bye mx2 picks up and sends it to mx1 for AV scan. i can see MX1 has a smaller drive, but it's just amavis scanning and relay. No real big space required. System seems to be fine, but these errors are eluding me. any insight would be helpful. postconf available if need be,. -j
Re: Out: 452 Insufficient system storage
this is default on all my systems. MX1 /dev/disk1s3 77G51G26G66%/ MX2 /dev/disk0s3 234G46G 187G20%/ Can you show the partitioning of these systems? Thanks Daniel Reinhardt Website: www.cryptodan.com Email: crypto...@yahoo.com
Re: Out: 452 Insufficient system storage
On Mar 2, 2010, at 7:31 PM, Daniel V. Reinhardt wrote: this is default on all my systems. MX1 /dev/disk1s3 77G51G26G66%/ MX2 /dev/disk0s3 234G46G 187G20%/ Can you show the partitioning of these systems? Thanks thats all i have,.. no partitions. MX1 FilesystemSize Used Avail Capacity Mounted on /dev/disk1s3 77G51G26G66%/ devfs 97K97K 0B 100%/dev fdesc 1.0K 1.0K 0B 100%/dev volfs 512K 512K 0B 100%/.vol automount -nsl [227]0B 0B 0B 100%/Network automount -fstab [233] 0B 0B 0B 100%/automount/ Servers automount -static [233] 0B 0B 0B 100%/automount/static /dev/disk0s3 234G22G 211G10%/Volumes/SSHD2 MX2 FilesystemSize Used Avail Capacity Mounted on /dev/disk0s3 234G46G 187G20%/ devfs 104K 104K 0B 100%/dev fdesc 1.0K 1.0K 0B 100%/dev volfs 512K 512K 0B 100%/.vol /dev/disk1s3 172G 431M 172G 0%/Volumes/mailHD /dev/disk2s10 172G46G 127G26%/Volumes/backup2 automount -nsl [213]0B 0B 0B 100%/Network automount -fstab [226] 0B 0B 0B 100%/automount/ Servers automount -static [226] 0B 0B 0B 100%/automount/static
Re: Out: 452 Insufficient system storage
donovan jeffrey j: On Mar 2, 2010, at 7:31 PM, Daniel V. Reinhardt wrote: this is default on all my systems. MX1 /dev/disk1s3 77G51G26G66%/ MX2 /dev/disk0s3 234G46G 187G20%/ It may be worthwhile to run the Postfix fsspace test program. - Download any Postfix source code that compiles on your system. - cd into the source tree, then execute the following commands: make makefiles cd src/util make fsspace ./fsspace /var/spool/postfix and report if the numbers look wrong. Postfix uses the fsspace routine to determine the amount of free space in the queue file system. Wietse
Re: Out: 452 Insufficient system storage
donovan jeffrey j put forth on 3/1/2010 8:06 AM: Greetings I had several of these on my primary MX this weekend and one just popped up. Can someone explain where this Insufficient system storage is ? What filesystem are you using? Are you running out of inodes? /$ df -i -- Stan
Re: Out: 452 Insufficient system storage
Wietse Venema: donovan jeffrey j: On Mar 2, 2010, at 7:31 PM, Daniel V. Reinhardt wrote: this is default on all my systems. MX1 /dev/disk1s3 77G51G26G66%/ MX2 /dev/disk0s3 234G46G 187G20%/ It may be worthwhile to run the Postfix fsspace test program. Not needed. The Postfix SMTP server logs this with verbose mode turned on: msg_info(%s: blocks %lu avail %lu min_free %lu msg_size_limit %lu, myname, (unsigned long) fsbuf.block_size, (unsigned long) fsbuf.block_free, (unsigned long) var_queue_minfree, (unsigned long) var_message_limit); That is, it logs the file system blocksize, the number of free blocks, and the queue_minfree and message_size_limit Postfix parameter values. Based on these, it decides if the number of free blocks is more than queue_minfree, and if the message size limit in blocks is less than the free blocks/1.5. Wietse - Download any Postfix source code that compiles on your system. - cd into the source tree, then execute the following commands: make makefiles cd src/util make fsspace ./fsspace /var/spool/postfix and report if the numbers look wrong. Postfix uses the fsspace routine to determine the amount of free space in the queue file system. Wietse
Re: Out: 452 Insufficient system storage
On Mar 2, 2010, at 9:03 PM, Wietse Venema wrote: It may be worthwhile to run the Postfix fsspace test program. - Download any Postfix source code that compiles on your system. - cd into the source tree, then execute the following commands: make makefiles cd src/util make fsspace ./fsspace /var/spool/postfix and report if the numbers look wrong. Postfix uses the fsspace routine to determine the amount of free space in the queue file system. Wietse mx1:/usr/local/postfix-2.7.0/src/util root# ./fsspace /var/spool/postfix /var/spool/postfix: block size 4096, blocks free 6836216 mx1:/usr/local/postfix-2.7.0/src/util root# df -i Filesystem 512-blocks Used Avail Capacity iusedifree %iused Mounted on /dev/disk0s3 160574256 105372640 5468961666% 13235578 6836202 66% / devfs 198 198 0 100% 5900 100% /dev fdesc2 2 0 100% 4 2532% /dev volfs 1024 1024 0 100% 00 100% /.vol /dev/disk1s3 489972528 46944240 44302828810% 5868028 55378536 10% /Volumes/SSHD2 automount -nsl [212] 0 0 0 100% 00 100% /Network automount -fstab [223] 0 0 0 100% 00 100% /automount/Servers automount -static [223] 0 0 0 100% 00 100% /automount/static /dev/disk2s3 489972528 105210640 38476188821% 13151328 48095236 21% /Volumes/MX1 I was sitting on a memory upgrade, and I through some in , possibly some Virtual memory foo in the OS. i also did a block level clone to another larger drive /dev/disk2s3 as a backup. did a reboot. nothing yet. I'lll have to wait and see if maybe the reboot makes this vanish. still running on original drive. maybe the drive isn't so S.M.A.R.T stay tuned. :) -j
Re: Out: 452 Insufficient system storage
donovan jeffrey j: On Mar 2, 2010, at 9:03 PM, Wietse Venema wrote: It may be worthwhile to run the Postfix fsspace test program. - Download any Postfix source code that compiles on your system. - cd into the source tree, then execute the following commands: make makefiles cd src/util make fsspace ./fsspace /var/spool/postfix and report if the numbers look wrong. Postfix uses the fsspace routine to determine the amount of free space in the queue file system. Wietse mx1:/usr/local/postfix-2.7.0/src/util root# ./fsspace /var/spool/postfix /var/spool/postfix: block size 4096, blocks free 6836216 That would be 26 GBytes. Filesystem 512-blocks Used Avail Capacity iusedifree %iused Mounted on /dev/disk0s3 160574256 105372640 5468961666% 13235578 6836202 66% / And that's 26 GBytes as well. It would be interesting to see what Postfix smtpd logs. You can turn it on selectively postconf -e debug_peer_list=127.0.0.1 postfix reload Then do telnet 127.0.0.1 25, and grep smtpd_check_queue /the/maillog/file. You can kill the logging with postconf -e debug_peer_list= Wietse
Re: Out: 452 Insufficient system storage
Zitat von donovan jeffrey j dono...@beth.k12.pa.us: Greetings I had several of these on my primary MX this weekend and one just popped up. Can someone explain where this Insufficient system storage is ? both mail queues are empty, and DF shows 20% on the reporting system. Out: 220 mx2.beth.k12.pa.us ESMTP Postfix In: EHLO mx1.beth.k12.pa.us Out: 250-mx2.beth.k12.pa.us Out: 250-PIPELINING Out: 250-SIZE 1024 Out: 250-ETRN Out: 250 8BITMIME In: MAIL FROM:bo-bu7vmghbprcxt5axe5vzqabvadp...@b.e.jcrew.com SIZE=11687 BODY=8BITMIME Out: 452 Insufficient system storage In: RCPT TO:someu...@beth.k12.pa.us Out: 503 Error: need MAIL command In: DATA Out: 503 Error: need RCPT command In: RSET Out: 250 Ok In: QUIT Out: 221 Bye Have a look for queue_minfree parameter in main.cf. The default for Postfix 2.1 and later is 1,5*message_size_limit which can be more than your 20% free if the filesystem the queues reside is rather small. Regards Andreas smime.p7s Description: S/MIME Signatur
