Re: What's the idea behind not enough disk space temporarily rejecting messages
Sounds like a good solution :D On 06/10/14 10:20, Gilles Chehade wrote: a different approach was used for this, we reserve 100 blocks at startup which is what Sendmail does on OpenBSD, if it has worked for years, it should keep working. On Tue, May 20, 2014 at 02:29:41PM -0700, Barbier, Jason wrote: restart away good man it is a topic that should be discussed because we honestly should beable to set when it decides to reject since as some one pointed out 5% of a 1tb disk is like 50gigs, and no where near full. On Tue, May 20, 2014 at 12:20 PM, Gilles Chehade gil...@poolp.org wrote: This has not been forgotten, the conversation with fellow hackers has led to nowhere but I'm about to restart it with a diff :-) On Tue, May 13, 2014 at 03:35:56AM -0700, Barbier, Jason wrote: Hmm, I see. at worst you now locked yourself out. I've already got the ext4 reserved blocks covering for that. Of course, not every file system has something like that. Even with reserve blocks it is still quite possible to fill the disk up past the reserve blocks. Never rely on a feature like that to save you from yourself ;), Ive had a few systems that were a fun disaster because the disk hit 105% full. But also every MTA does this also and you will be hard pressed to find one that doesn't. Indeed. postfix, for example, has a limit of 1.5 ?? message_size_limit. Way more sane, imho. Im not advising the limit is sane, it should/could be a knob, which is probably a discussion to have now. To be fair I as the admin should be able to decide how much I want to fill my queue folder, since in all fairness my queue folder could not be in /var/spool I could have moved it to a san where it shares space with other mission critical applications and has 50 terabytes of storage, and I really don't care if the disk fills past 5% because really it wont effect anything for the disk to get to near zero and I will always be able to recover from it filling. but that means your script or app needs to learn to be more resilient Any standard solutions for cron/at to do that? Standard no, many ideas, yes. If you just bebop around the internet there are many great examples of scripts that are resiliant. Since Im not sure what your scripts are written in I can only give you generic advice which is if you use the sendmail sender as the wrapper it should return a success/fail result and you could case or if/else for that result so it becomes in essence: Check for unsent queue and try to send Try to send new mail if Success exit if fail count=1 try again if fail count=2 write mail to disk (maybe pickup directory so it will get sent at some point when the mailer recognizes it) -- Jason Barbier | jab...@serversave.us Pro Patria Vigilans -- Gilles Chehade https://www.poolp.org @poolpOrg -- Jason Barbier | jab...@serversave.us Pro Patria Vigilans -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org
Re: What's the idea behind not enough disk space temporarily rejecting messages
This has not been forgotten, the conversation with fellow hackers has led to nowhere but I'm about to restart it with a diff :-) On Tue, May 13, 2014 at 03:35:56AM -0700, Barbier, Jason wrote: Hmm, I see. at worst you now locked yourself out. I've already got the ext4 reserved blocks covering for that. Of course, not every file system has something like that. Even with reserve blocks it is still quite possible to fill the disk up past the reserve blocks. Never rely on a feature like that to save you from yourself ;), Ive had a few systems that were a fun disaster because the disk hit 105% full. But also every MTA does this also and you will be hard pressed to find one that doesn't. Indeed. postfix, for example, has a limit of 1.5 ?? message_size_limit. Way more sane, imho. Im not advising the limit is sane, it should/could be a knob, which is probably a discussion to have now. To be fair I as the admin should be able to decide how much I want to fill my queue folder, since in all fairness my queue folder could not be in /var/spool I could have moved it to a san where it shares space with other mission critical applications and has 50 terabytes of storage, and I really don't care if the disk fills past 5% because really it wont effect anything for the disk to get to near zero and I will always be able to recover from it filling. but that means your script or app needs to learn to be more resilient Any standard solutions for cron/at to do that? Standard no, many ideas, yes. If you just bebop around the internet there are many great examples of scripts that are resiliant. Since Im not sure what your scripts are written in I can only give you generic advice which is if you use the sendmail sender as the wrapper it should return a success/fail result and you could case or if/else for that result so it becomes in essence: Check for unsent queue and try to send Try to send new mail if Success exit if fail count=1 try again if fail count=2 write mail to disk (maybe pickup directory so it will get sent at some point when the mailer recognizes it) -- Jason Barbier | jab...@serversave.us Pro Patria Vigilans -- Gilles Chehade https://www.poolp.org @poolpOrg -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org
Re: What's the idea behind not enough disk space temporarily rejecting messages
On Mon, May 12, 2014 at 11:24:09PM -0300, Hugo Osvaldo Barrera wrote: On 2014-05-10 11:43, Julius wrote: Hello opensmtpd-misc! If there is less than 5% of disk space left, OpenSMTP rejects messages. That is a compile-time/in-source constant. The restriction is simple to remove from smtpd/queue_fs.c but I want to know what the big idea behind this is. Mail is very precious, even the output from cronjobs, which will be directly lost in such a situation. More Info: I'm running opensmtpd on my NAS where the var-partition is close to full. All mail is relayed, so I find the restriction especially annoying. Also, at 5%, I still have 200GB of space left. I could store 100 000 000 of the average mail that machine handles. Mai 10 11:15:58 micpi1 smtpd[559]: smtp-in: New session 5fc97cda452fbcee from host 0@localhost [local] Mai 10 11:15:59 micpi1 smtpd[551]: warn: not enough disk space: 4% left Mai 10 11:15:59 micpi1 smtpd[551]: warn: temporarily rejecting messages Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Failed command on session 5fc97cda452fbcee: MAIL FROM:b Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Closing session 5fc97cda452fbcee Julius -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org Indeed, 5% seems a bit arbitrary. It can be 50MiB, or 50GiB. Maybe something like min(5%, 200MiB) would be a bit more flexible. (200MiB is completely arbitrary; something else can be chosen). Also, maybe email incoming from the mail command (that's what cron uses, right?), can have a *slightly* lower limit, so as not to reject it's messages as quickly. I'll never notice I'm under 5% if I don't get cron's daily output. I agree with you guys. Eric and I have been against this limit but there's been requests that a limit of some sort be introduced to prevent /var filling from OpenSMTPD. I had an idea that is more sensible than an arbitrary limit, I'll have a chat with a few people and will provide a diff. -- Gilles Chehade https://www.poolp.org @poolpOrg -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org
Re: What's the idea behind not enough disk space temporarily rejecting messages
Hmm, I see. at worst you now locked yourself out. I've already got the ext4 reserved blocks covering for that. Of course, not every file system has something like that. Even with reserve blocks it is still quite possible to fill the disk up past the reserve blocks. Never rely on a feature like that to save you from yourself ;), Ive had a few systems that were a fun disaster because the disk hit 105% full. But also every MTA does this also and you will be hard pressed to find one that doesn't. Indeed. postfix, for example, has a limit of 1.5 × message_size_limit. Way more sane, imho. Im not advising the limit is sane, it should/could be a knob, which is probably a discussion to have now. To be fair I as the admin should be able to decide how much I want to fill my queue folder, since in all fairness my queue folder could not be in /var/spool I could have moved it to a san where it shares space with other mission critical applications and has 50 terabytes of storage, and I really don't care if the disk fills past 5% because really it wont effect anything for the disk to get to near zero and I will always be able to recover from it filling. but that means your script or app needs to learn to be more resilient Any standard solutions for cron/at to do that? Standard no, many ideas, yes. If you just bebop around the internet there are many great examples of scripts that are resiliant. Since Im not sure what your scripts are written in I can only give you generic advice which is if you use the sendmail sender as the wrapper it should return a success/fail result and you could case or if/else for that result so it becomes in essence: Check for unsent queue and try to send Try to send new mail if Success exit if fail count=1 try again if fail count=2 write mail to disk (maybe pickup directory so it will get sent at some point when the mailer recognizes it) -- Jason Barbier | jab...@serversave.us Pro Patria Vigilans
Re: What's the idea behind not enough disk space temporarily rejecting messages
On 2014-05-10 11:43, Julius wrote: Hello opensmtpd-misc! If there is less than 5% of disk space left, OpenSMTP rejects messages. That is a compile-time/in-source constant. The restriction is simple to remove from smtpd/queue_fs.c but I want to know what the big idea behind this is. Mail is very precious, even the output from cronjobs, which will be directly lost in such a situation. More Info: I'm running opensmtpd on my NAS where the var-partition is close to full. All mail is relayed, so I find the restriction especially annoying. Also, at 5%, I still have 200GB of space left. I could store 100 000 000 of the average mail that machine handles. Mai 10 11:15:58 micpi1 smtpd[559]: smtp-in: New session 5fc97cda452fbcee from host 0@localhost [local] Mai 10 11:15:59 micpi1 smtpd[551]: warn: not enough disk space: 4% left Mai 10 11:15:59 micpi1 smtpd[551]: warn: temporarily rejecting messages Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Failed command on session 5fc97cda452fbcee: MAIL FROM:b Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Closing session 5fc97cda452fbcee Julius -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org Indeed, 5% seems a bit arbitrary. It can be 50MiB, or 50GiB. Maybe something like min(5%, 200MiB) would be a bit more flexible. (200MiB is completely arbitrary; something else can be chosen). Also, maybe email incoming from the mail command (that's what cron uses, right?), can have a *slightly* lower limit, so as not to reject it's messages as quickly. I'll never notice I'm under 5% if I don't get cron's daily output. -- Hugo Osvaldo Barrera A: No, it doesn't make sense. Q: Should I include quotations *after* my reply? pgpZcUBnal42j.pgp Description: PGP signature
Re: What's the idea behind not enough disk space temporarily rejecting messages
Hmm, I see. at worst you now locked yourself out. I've already got the ext4 reserved blocks covering for that. Of course, not every file system has something like that. But also every MTA does this also and you will be hard pressed to find one that doesn't. Indeed. postfix, for example, has a limit of 1.5 × message_size_limit. Way more sane, imho. but that means your script or app needs to learn to be more resilient Any standard solutions for cron/at to do that? On 10 May 06:50, Barbier, Jason wrote: Well the idea is much like what the RFC says to do, if your server is resource constrained you send a temp fail and let the other side keep trying until the receiving end is good or you hit the TTL on the sending side. In practice this is a good idea since in you fill the queue disk you will have problems deleting files from it at best, at worst you now locked yourself out. There maybe cases where this behavior may feel undesirable but that means your script or app needs to learn to be more resilient not that your MTA needs to clobber the system. But also every MTA does this also and you will be hard pressed to find one that doesn't. On May 10, 2014 2:43 AM, Julius opensm...@liftm.de wrote: Hello opensmtpd-misc! If there is less than 5% of disk space left, OpenSMTP rejects messages. That is a compile-time/in-source constant. The restriction is simple to remove from smtpd/queue_fs.c but I want to know what the big idea behind this is. Mail is very precious, even the output from cronjobs, which will be directly lost in such a situation. More Info: I'm running opensmtpd on my NAS where the var-partition is close to full. All mail is relayed, so I find the restriction especially annoying. Also, at 5%, I still have 200GB of space left. I could store 100 000 000 of the average mail that machine handles. Mai 10 11:15:58 micpi1 smtpd[559]: smtp-in: New session 5fc97cda452fbcee from host 0@localhost [local] Mai 10 11:15:59 micpi1 smtpd[551]: warn: not enough disk space: 4% left Mai 10 11:15:59 micpi1 smtpd[551]: warn: temporarily rejecting messages Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Failed command on session 5fc97cda452fbcee: MAIL FROM:b Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Closing session 5fc97cda452fbcee Julius -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org
What's the idea behind not enough disk space temporarily rejecting messages
Hello opensmtpd-misc! If there is less than 5% of disk space left, OpenSMTP rejects messages. That is a compile-time/in-source constant. The restriction is simple to remove from smtpd/queue_fs.c but I want to know what the big idea behind this is. Mail is very precious, even the output from cronjobs, which will be directly lost in such a situation. More Info: I'm running opensmtpd on my NAS where the var-partition is close to full. All mail is relayed, so I find the restriction especially annoying. Also, at 5%, I still have 200GB of space left. I could store 100 000 000 of the average mail that machine handles. Mai 10 11:15:58 micpi1 smtpd[559]: smtp-in: New session 5fc97cda452fbcee from host 0@localhost [local] Mai 10 11:15:59 micpi1 smtpd[551]: warn: not enough disk space: 4% left Mai 10 11:15:59 micpi1 smtpd[551]: warn: temporarily rejecting messages Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Failed command on session 5fc97cda452fbcee: MAIL FROM:b Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Closing session 5fc97cda452fbcee Julius -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org
Re: What's the idea behind not enough disk space temporarily rejecting messages
Well the idea is much like what the RFC says to do, if your server is resource constrained you send a temp fail and let the other side keep trying until the receiving end is good or you hit the TTL on the sending side. In practice this is a good idea since in you fill the queue disk you will have problems deleting files from it at best, at worst you now locked yourself out. There maybe cases where this behavior may feel undesirable but that means your script or app needs to learn to be more resilient not that your MTA needs to clobber the system. But also every MTA does this also and you will be hard pressed to find one that doesn't. On May 10, 2014 2:43 AM, Julius opensm...@liftm.de wrote: Hello opensmtpd-misc! If there is less than 5% of disk space left, OpenSMTP rejects messages. That is a compile-time/in-source constant. The restriction is simple to remove from smtpd/queue_fs.c but I want to know what the big idea behind this is. Mail is very precious, even the output from cronjobs, which will be directly lost in such a situation. More Info: I'm running opensmtpd on my NAS where the var-partition is close to full. All mail is relayed, so I find the restriction especially annoying. Also, at 5%, I still have 200GB of space left. I could store 100 000 000 of the average mail that machine handles. Mai 10 11:15:58 micpi1 smtpd[559]: smtp-in: New session 5fc97cda452fbcee from host 0@localhost [local] Mai 10 11:15:59 micpi1 smtpd[551]: warn: not enough disk space: 4% left Mai 10 11:15:59 micpi1 smtpd[551]: warn: temporarily rejecting messages Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Failed command on session 5fc97cda452fbcee: MAIL FROM:b Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Closing session 5fc97cda452fbcee Julius -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org
Re: What's the idea behind not enough disk space temporarily rejecting messages
Rejecting messages when disk space is getting scarce is a good thing, but a hardcoded percentage is not really optimal. Having 200GB left doesn't seem like a reason to reject messages to me. On 10 May 2014 15:50, Barbier, Jason jab...@serversave.us wrote: Well the idea is much like what the RFC says to do, if your server is resource constrained you send a temp fail and let the other side keep trying until the receiving end is good or you hit the TTL on the sending side. In practice this is a good idea since in you fill the queue disk you will have problems deleting files from it at best, at worst you now locked yourself out. There maybe cases where this behavior may feel undesirable but that means your script or app needs to learn to be more resilient not that your MTA needs to clobber the system. But also every MTA does this also and you will be hard pressed to find one that doesn't. On May 10, 2014 2:43 AM, Julius opensm...@liftm.de wrote: Hello opensmtpd-misc! If there is less than 5% of disk space left, OpenSMTP rejects messages. That is a compile-time/in-source constant. The restriction is simple to remove from smtpd/queue_fs.c but I want to know what the big idea behind this is. Mail is very precious, even the output from cronjobs, which will be directly lost in such a situation. More Info: I'm running opensmtpd on my NAS where the var-partition is close to full. All mail is relayed, so I find the restriction especially annoying. Also, at 5%, I still have 200GB of space left. I could store 100 000 000 of the average mail that machine handles. Mai 10 11:15:58 micpi1 smtpd[559]: smtp-in: New session 5fc97cda452fbcee from host 0@localhost [local] Mai 10 11:15:59 micpi1 smtpd[551]: warn: not enough disk space: 4% left Mai 10 11:15:59 micpi1 smtpd[551]: warn: temporarily rejecting messages Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Failed command on session 5fc97cda452fbcee: MAIL FROM:b Mai 10 11:15:59 micpi1 smtpd[559]: smtp-in: Closing session 5fc97cda452fbcee Julius -- You received this mail because you are subscribed to misc@opensmtpd.org To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org
