Re: 4KB of disk space burning on a single email
On Friday 20 November 2009 13:42:58 Arora, Sumit wrote: Hi folks, I'm just testing my postfix server for load and disk usage. I'm using content filter on some another server, and I don't feed the email back to postfix. Email data is getting deleted from my postfix server, but I don't know where 4KB of my disk space gone on every email my postfix server receives. If anybody have some idea, please tell me. -Sumit Arora My guess is that the filesystem has a block-size of 4KB. A file is stored over a set of blocks of this size. Which means that if a file is smaller then 4KB, it will still use a 4KB block. As far as I know, only reiserfs has the notailoption which will try to stick multiple smaller files into a single block. -- Joost
Re: 4KB of disk space burning on a single email
J. Roeleveld: On Friday 20 November 2009 13:42:58 Arora, Sumit wrote: Hi folks, I'm just testing my postfix server for load and disk usage. I'm using content filter on some another server, and I don't feed the email back to postfix. Email data is getting deleted from my postfix server, but I don't know where 4KB of my disk space gone on every email my postfix server receives. If anybody have some idea, please tell me. -Sumit Arora My guess is that the filesystem has a block-size of 4KB. A file is stored over a set of blocks of this size. Which means that if a file is smaller then 4KB, it will still use a 4KB block. As far as I know, only reiserfs has the notailoption which will try to stick multiple smaller files into a single block. 25 Years ago, the UFS file system came with large blocks (typ 8k) and small fragments (typ 1k). The tail fragments from different files could then share the same large block. In the mean time, the 8k blocksize has been replaced by larger blocksizes like 32k but the idea is the same: low fragmentation without wasting space. Wietse Marshall Kirk McKusick, William N. Joy, Samuel J. Leffler and Robert S. Fabry. A Fast File System for UNIX. Technical Report Computer Systems Research Group, Computer Science Division, Department of Electrical Engineering and Computer Science, University of California, Berkeley, Berkeley, CA 94720. http://www.cs.berkeley.edu/~brewer/cs262/FFS.pdf
RE: 4KB of disk space burning on a single email
You are right Joost, 4KB will be the block size... But my question is who is taking this disk space, because email is deleted already. Is this any log or what I just have to stop consumption of any kind of disk space except mail logs. Please help -Sumit Arora -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent: Friday, November 20, 2009 6:26 PM To: postfix-users@postfix.org Subject: Re: 4KB of disk space burning on a single email On Friday 20 November 2009 13:42:58 Arora, Sumit wrote: Hi folks, I'm just testing my postfix server for load and disk usage. I'm using content filter on some another server, and I don't feed the email back to postfix. Email data is getting deleted from my postfix server, but I don't know where 4KB of my disk space gone on every email my postfix server receives. If anybody have some idea, please tell me. -Sumit Arora My guess is that the filesystem has a block-size of 4KB. A file is stored over a set of blocks of this size. Which means that if a file is smaller then 4KB, it will still use a 4KB block. As far as I know, only reiserfs has the notailoption which will try to stick multiple smaller files into a single block. -- Joost
Re: 4KB of disk space burning on a single email
Arora, Sumit wrote: You are right Joost, 4KB will be the block size... But my question is who is taking this disk space, because email is deleted already. Is this any log or what I just have to stop consumption of any kind of disk space except mail logs. Please help -Sumit Arora -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent: Friday, November 20, 2009 6:26 PM To: postfix-users@postfix.org Subject: Re: 4KB of disk space burning on a single email On Friday 20 November 2009 13:42:58 Arora, Sumit wrote: Hi folks, I'm just testing my postfix server for load and disk usage. I'm using content filter on some another server, and I don't feed the email back to postfix. Email data is getting deleted from my postfix server, but I don't know where 4KB of my disk space gone on every email my postfix server receives. If anybody have some idea, please tell me. -Sumit Arora My guess is that the filesystem has a block-size of 4KB. A file is stored over a set of blocks of this size. Which means that if a file is smaller then 4KB, it will still use a 4KB block. As far as I know, only reiserfs has the notailoption which will try to stick multiple smaller files into a single block. -- Joost Perhaps tell your mail client to compact the trash. That's what I have to do in Thunderbird, or else the files stay on the server. Dan Schaefer Web Developer/Systems Analyst Performance Administration Corp.
Re: 4KB of disk space burning on a single email
On Friday 20 November 2009 14:14:03 Arora, Sumit wrote: You are right Joost, 4KB will be the block size... But my question is who is taking this disk space, because email is deleted already. Is this any log or what I just have to stop consumption of any kind of disk space except mail logs. Please help -Sumit Arora 4K on logs per email are quite a lot. Unless you have really verbose logging. Unless your system logger is duplicating messages over several logfiles? Possible places to look: /var/spool/. /tmp/ /var/tmp/ Maybe something leave temporary files, or maybe your postfix is keeping files in a spool-directory. Maybe try the following to see where additional files are placed: du / PRE-newemail.txt have postfix parse a few emails du / POST-newemail.txt Then do a diff on these 2 text-files to see which directories had an increase. Hopefully, this will reduce the possible locations to check for possible locations. -- Joost -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent: Friday, November 20, 2009 6:26 PM To: postfix-users@postfix.org Subject: Re: 4KB of disk space burning on a single email On Friday 20 November 2009 13:42:58 Arora, Sumit wrote: Hi folks, I'm just testing my postfix server for load and disk usage. I'm using content filter on some another server, and I don't feed the email back to postfix. Email data is getting deleted from my postfix server, but I don't know where 4KB of my disk space gone on every email my postfix server receives. If anybody have some idea, please tell me. -Sumit Arora My guess is that the filesystem has a block-size of 4KB. A file is stored over a set of blocks of this size. Which means that if a file is smaller then 4KB, it will still use a 4KB block. As far as I know, only reiserfs has the notailoption which will try to stick multiple smaller files into a single block. -- Joost
RE: 4KB of disk space burning on a single email
I'm really amazed... its maillog that is consuming 4KB on a simple 5-7 line logging for each email -Sumit Arora -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent: Friday, November 20, 2009 6:51 PM To: postfix-users@postfix.org Subject: Re: 4KB of disk space burning on a single email On Friday 20 November 2009 14:14:03 Arora, Sumit wrote: You are right Joost, 4KB will be the block size... But my question is who is taking this disk space, because email is deleted already. Is this any log or what I just have to stop consumption of any kind of disk space except mail logs. Please help -Sumit Arora 4K on logs per email are quite a lot. Unless you have really verbose logging. Unless your system logger is duplicating messages over several logfiles? Possible places to look: /var/spool/. /tmp/ /var/tmp/ Maybe something leave temporary files, or maybe your postfix is keeping files in a spool-directory. Maybe try the following to see where additional files are placed: du / PRE-newemail.txt have postfix parse a few emails du / POST-newemail.txt Then do a diff on these 2 text-files to see which directories had an increase. Hopefully, this will reduce the possible locations to check for possible locations. -- Joost -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent: Friday, November 20, 2009 6:26 PM To: postfix-users@postfix.org Subject: Re: 4KB of disk space burning on a single email On Friday 20 November 2009 13:42:58 Arora, Sumit wrote: Hi folks, I'm just testing my postfix server for load and disk usage. I'm using content filter on some another server, and I don't feed the email back to postfix. Email data is getting deleted from my postfix server, but I don't know where 4KB of my disk space gone on every email my postfix server receives. If anybody have some idea, please tell me. -Sumit Arora My guess is that the filesystem has a block-size of 4KB. A file is stored over a set of blocks of this size. Which means that if a file is smaller then 4KB, it will still use a 4KB block. As far as I know, only reiserfs has the notailoption which will try to stick multiple smaller files into a single block. -- Joost
Re: 4KB of disk space burning on a single email
* Arora, Sumit sumit.ar...@hp.com: You are right Joost, 4KB will be the block size... But my question is who is taking this disk space, because email is deleted already. How EXACTLY is the email being deleted ? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: 4KB of disk space burning on a single email
On Friday 20 November 2009 14:38:50 Arora, Sumit wrote: I'm really amazed... its maillog that is consuming 4KB on a simple 5-7 line logging for each email -Sumit Arora How long are those lines? -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent: Friday, November 20, 2009 6:51 PM To: postfix-users@postfix.org Subject: Re: 4KB of disk space burning on a single email On Friday 20 November 2009 14:14:03 Arora, Sumit wrote: You are right Joost, 4KB will be the block size... But my question is who is taking this disk space, because email is deleted already. Is this any log or what I just have to stop consumption of any kind of disk space except mail logs. Please help -Sumit Arora 4K on logs per email are quite a lot. Unless you have really verbose logging. Unless your system logger is duplicating messages over several logfiles? Possible places to look: /var/spool/. /tmp/ /var/tmp/ Maybe something leave temporary files, or maybe your postfix is keeping files in a spool-directory. Maybe try the following to see where additional files are placed: du / PRE-newemail.txt have postfix parse a few emails du / POST-newemail.txt Then do a diff on these 2 text-files to see which directories had an increase. Hopefully, this will reduce the possible locations to check for possible locations. -- Joost -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of J. Roeleveld Sent: Friday, November 20, 2009 6:26 PM To: postfix-users@postfix.org Subject: Re: 4KB of disk space burning on a single email On Friday 20 November 2009 13:42:58 Arora, Sumit wrote: Hi folks, I'm just testing my postfix server for load and disk usage. I'm using content filter on some another server, and I don't feed the email back to postfix. Email data is getting deleted from my postfix server, but I don't know where 4KB of my disk space gone on every email my postfix server receives. If anybody have some idea, please tell me. -Sumit Arora My guess is that the filesystem has a block-size of 4KB. A file is stored over a set of blocks of this size. Which means that if a file is smaller then 4KB, it will still use a 4KB block. As far as I know, only reiserfs has the notailoption which will try to stick multiple smaller files into a single block. -- Joost
RE: 4KB of disk space burning on a single email
It's being passed for content filtering and email is not sent back to postfix. For postfix it's just mail forwarding -Sumit Arora -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of Ralf Hildebrandt Sent: Friday, November 20, 2009 7:11 PM To: postfix-users@postfix.org Subject: Re: 4KB of disk space burning on a single email * Arora, Sumit sumit.ar...@hp.com: You are right Joost, 4KB will be the block size... But my question is who is taking this disk space, because email is deleted already. How EXACTLY is the email being deleted ? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: 4KB of disk space burning on a single email
On 11/20/2009 7:44 AM, Arora, Sumit wrote: It's being passed for content filtering and email is not sent back to postfix. For postfix it's just mail forwarding Amavisd-new (and probably other content filters too) keeps temporary files between messages to prevent creating new files every time a mail comes in. Sounds as if you're chasing a problem that doesn't exist. -- Noel Jones
RE: 4KB of disk space burning on a single email
I've checked the size of maillog It's size is getting increased by 4KB on receiving each email. Noel Jones, please read the other emails of this thread. -Sumit Arora -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of Noel Jones Sent: Friday, November 20, 2009 7:20 PM To: postfix-users@postfix.org Subject: Re: 4KB of disk space burning on a single email On 11/20/2009 7:44 AM, Arora, Sumit wrote: It's being passed for content filtering and email is not sent back to postfix. For postfix it's just mail forwarding Amavisd-new (and probably other content filters too) keeps temporary files between messages to prevent creating new files every time a mail comes in. Sounds as if you're chasing a problem that doesn't exist. -- Noel Jones
Re: 4KB of disk space burning on a single email
Arora, Sumit: I've checked the size of maillog It's size is getting increased by 4KB on receiving each email. We know what you believe. Now we would like to see some actual concrete evidence. Wietse
Re: 4KB of disk space burning on a single email
On 11/20/2009 7:53 AM, Arora, Sumit wrote: I've checked the size of maillog It's size is getting increased by 4KB on receiving each email. Noel Jones, please read the other emails of this thread. -Sumit Arora I did. All I see are unsupported top-posted claims. -- Noel Jones
Re: 4KB of disk space burning on a single email
On Friday 20 November 2009 14:53:58 Arora, Sumit wrote: I've checked the size of maillog It's size is getting increased by 4KB on receiving each email. Noel Jones, please read the other emails of this thread. -Sumit Arora The log-entries you sent me are nowhere near 4KB in size. How do you check that the maillog increases by 4KB per message? Did you use the suggested method to check where the increases all occur? It might easily be a collection of different logfiles and/or temp-files that are being kept. If it's a collection of different logfiles adding up to 4KB in size per email, then you might want to recheck your syslog-configuration. -- Joost
Re: 4KB of disk space burning on a single email
On Friday 20 November 2009 15:52:13 J. Roeleveld wrote: If it's a collection of different logfiles adding up to 4KB in size per email, then you might want to recheck your syslog-configuration. Probably pays to know what file system and mount options are applicable here as well. As I find it all too believable one could make a filesystem eat a block per distinct write these days. Now me, I get 24Kb of log file for every message delivered, but that includes messages about rejected spam :(
Re: 4KB of disk space burning on a single email
On Fri, Nov 20, 2009 at 04:01:56PM +, Simon Waters wrote: On Friday 20 November 2009 15:52:13 J. Roeleveld wrote: If it's a collection of different logfiles adding up to 4KB in size per email, then you might want to recheck your syslog-configuration. Probably pays to know what file system and mount options are applicable here as well. As I find it all too believable one could make a filesystem eat a block per distinct write these days. If the OP is measuring I/O rates, not disk storage, then the simplest explanation is failure to turn off synchronous syslog writes. FAQ (Linux syslog.conf for mail): mail.info -/var/log/maillog the - is not optional on systems that handle non-trivial volumes of email. -- Viktor. Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. To unsubscribe from the postfix-users list, visit http://www.postfix.org/lists.html or click the link below: mailto:majord...@postfix.org?body=unsubscribe%20postfix-users If my response solves your problem, the best way to thank me is to not send an it worked, thanks follow-up. If you must respond, please put It worked, thanks in the Subject so I can delete these quickly.
Re: 4KB of disk space burning on a single email
On Friday 20 November 2009 17:01:56 Simon Waters wrote: On Friday 20 November 2009 15:52:13 J. Roeleveld wrote: If it's a collection of different logfiles adding up to 4KB in size per email, then you might want to recheck your syslog-configuration. Probably pays to know what file system and mount options are applicable here as well. As I find it all too believable one could make a filesystem eat a block per distinct write these days. Now me, I get 24Kb of log file for every message delivered, but that includes messages about rejected spam :( I also get quite a bit of logfile entries for every message, but that does include the filtering/redirecting/sieve/. stuff that's in there as well. If people want, I can repost the logentries the OP sent me, but they are pretty standard as far as I can tell. -- Joost
Re: 4KB of disk space burning on a single email
* J. Roeleveld jo...@antarean.org: The log-entries you sent me are nowhere near 4KB in size. How do you check that the maillog increases by 4KB per message? My guess is the mails are marked as deleted. Nothing more. They never get deleted, thus the space never gets free. Furthermore it's OT! -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: 4KB of disk space burning on a single email
Arora, Sumit wrote: You are right Joost, 4KB will be the block size... But my question is who is taking this disk space, because email is deleted already. Please show Postfix log entries showing the message being delivered to the content filter and successfully accepted and deleted as: Nov 20 13:41:10 wormhole postfix/lmtp[17080]: 81D9430101: to=te...@cnysupport.com, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.19/0/0/0.88, dsn=2.0.0 Nov 20 13:41:10 wormhole postfix/qmgr[5151]: 81D9430101: removed then try postcat -q queue id If it says: [r...@wormhole ~]# postcat -q 81D9430101 postcat: fatal: open queue file 81D9430101: No such file or directory then it's not a postfix problem. Terry
