Re: Invalid cross-device link
The patch seems interesting. I do have a few questions: (1) Have you tried enabling cache files? That eliminates processing the spool if no new mail has arrived since the prior mail check, which cuts way down on I/O. (2) A logical extension for cache files is to not throw them out just because new mail has arrived, but instead process only the new mail and append to the cache. That should be a big win with users who have large spools. I haven't had the time to make this change, but if someone else wanted to, that would be terrific. (3) Qpopper does have some rudimentary hooks for telling clients the minimum mail check interval, but it doesn't enforce it. This could be extended to enforce the limit, perhaps in conjunction with the keep-temp-drop option (which keeps the temp drop around so that the user's last mail check can be easily determined at any time). (4) Keep in mind that returning an [AUTH] error should only be done if the user's credentials are bad, since it tells the clients to throw out the saved password and ask the user to retype it. (5) I'm puzzled by the desire to return a 'no new messages' error. Many clients authenticate and then issue LIST and UIDL. Is the intent to cache these responses and return them without actually checking? Or is the intent to return empty values for each, thus telling the client that all mail on the server has been deleted, only to have it reappear later? I'm probably missing something very obvious here, so please forgive me. (6) If the happymail patch were to be intended to be incorporated into the standard distribution, I'd like to see it make much less use of #ifdef, and use boolean or other option checks instead. I think it makes the code much more maintainable, especially where lots of options can get independently selected. It would also need to run on all OS flavors, of course. -- Randall Gellens Opinions are personal;facts are suspect;I speak for myself only -- Randomly-selected tag: --- Finagle's Second Law: No matter what the anticipated result, there will always be someone eager to (a) misinterpret it, (b) fake it, or (c) believe it happened according to his own pet theory.
Re: Invalid cross-device link
The patch seems interesting. I do have a few questions: (1) Have you tried enabling cache files? That eliminates processing the spool if no new mail has arrived since the prior mail check, which cuts way down on I/O. (2) A logical extension for cache files is to not throw them out just because new mail has arrived, but instead process only the new mail and append to the cache. That should be a big win with users who have large spools. I haven't had the time to make this change, but if someone else wanted to, that would be terrific. (3) Qpopper does have some rudimentary hooks for telling clients the minimum mail check interval, but it doesn't enforce it. This could be extended to enforce the limit, perhaps in conjunction with the keep-temp-drop option (which keeps the temp drop around so that the user's last mail check can be easily determined at any time). (4) Keep in mind that returning an [AUTH] error should only be done if the user's credentials are bad, since it tells the clients to throw out the saved password and ask the user to retype it. (5) I'm puzzled by the desire to return a 'no new messages' error. Many clients authenticate and then issue LIST and UIDL. Is the intent to cache these responses and return them without actually checking? Or is the intent to return empty values for each, thus telling the client that all mail on the server has been deleted, only to have it reappear later? I'm probably missing something very obvious here, so please forgive me. (6) If the happymail patch were to be intended to be incorporated into the standard distribution, I'd like to see it make much less use of #ifdef, and use boolean or other option checks instead. I think it makes the code much more maintainable, especially where lots of options can get independently selected. It would also need to run on all OS flavors, of course. -- Randall Gellens Opinions are personal;facts are suspect;I speak for myself only -- Randomly-selected tag: --- Finagle's Second Law: No matter what the anticipated result, there will always be someone eager to (a) misinterpret it, (b) fake it, or (c) believe it happened according to his own pet theory.
Re: Invalid cross-device link
<[EMAIL PROTECTED]> In-Reply-To: <[EMAIL PROTECTED]> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Ken A wrote: > > There seem to be some issues with this patch. It stalls the pop3 session > after getting the password for about 15 seconds, then says 'no new > messages in the MUA (tested with thunderbird & command line). > > Thanks, > Ken A > > New version posted. http://www.jmaimon.com/qpopper/patches/happymail-sleep-seconds.pl5b.408.patch http://www.jmaimon.com/qpopper/ Thanks for the report.
Re: Invalid cross-device link
<[EMAIL PROTECTED]> <[EMAIL PROTECTED]> In-Reply-To: <[EMAIL PROTECTED]> X-Enigmail-Version: 0.90.0.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Joe Maimon wrote: > > > Ken A wrote: > > >> >> Joe Maimon wrote: >> >>> >>> Daniel Senie wrote: >>> >>> At 05:18 PM 5/26/2005, Clifton Royston wrote: > On Thu, May 26, 2005 at 06:26:42AM -0400, Joe Maimon wrote: > > > > > > >>> >>> >> >> >> I would also like to see some way to limit users (per user) to a >> number of pop3 checks per minute, but without generating support calls >> because of an error message. It would be better to simply return "no >> new messages on server" for x minutes if possible (still with no i/o). >> I'm not at all sure how difficult that change would be to implement. >> >> Short of that, I'd definitely like to see the HappyMail patch put into >> the main codebase. >> >> Ken Anderson >> Pacific.Net >> >> > I have posted a minimally tested patch to the happymail patch which does > exactly that. > > http://www.jmaimon.com/qpopper/#happy-pl8-jmpl5 > > Basically the patch causes popper to return early from pop_dropcopy() > and from pop_updt() before any IO is done. > > There seem to be some issues with this patch. It stalls the pop3 session after getting the password for about 15 seconds, then says 'no new messages in the MUA (tested with thunderbird & command line). Thanks, Ken A >
Re: Invalid cross-device link
<[EMAIL PROTECTED]> In-Reply-To: <[EMAIL PROTECTED]> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Ken A wrote: > > Joe Maimon wrote: > >> >>Daniel Senie wrote: >> >> >>>At 05:18 PM 5/26/2005, Clifton Royston wrote: >>> >>> On Thu, May 26, 2005 at 06:26:42AM -0400, Joe Maimon wrote: >> >> > > > I would also like to see some way to limit users (per user) to a number > of pop3 checks per minute, but without generating support calls because > of an error message. It would be better to simply return "no new > messages on server" for x minutes if possible (still with no i/o). I'm > not at all sure how difficult that change would be to implement. > > Short of that, I'd definitely like to see the HappyMail patch put into > the main codebase. > > Ken Anderson > Pacific.Net > > I have posted a minimally tested patch to the happymail patch which does exactly that. http://www.jmaimon.com/qpopper/#happy-pl8-jmpl5 Basically the patch causes popper to return early from pop_dropcopy() and from pop_updt() before any IO is done.
Re: Invalid cross-device link
<[EMAIL PROTECTED]> In-Reply-To: <[EMAIL PROTECTED]> X-Enigmail-Version: 0.90.0.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit David Champion wrote: > ive.org> <[EMAIL PROTECTED]> > Mime-Version: 1.0 > Content-Type: text/plain; charset=us-ascii > Content-Disposition: inline > In-Reply-To: <[EMAIL PROTECTED]> > X-Comment: no > User-Agent: Mutt/1.5.8i > X-Decanonizer: current version 0.2005.05.10 > for j4RMOhww016410 from localhost [127.0.0.1] > at 1117232684: Fri May 27 22:24:44 2005 [0.172s] > > * On 2005.05.27, in <[EMAIL PROTECTED]>, > * "Daniel Senie" <[EMAIL PROTECTED]> wrote: > >>yes, it'd have to happen without actually reading the mailbox. Guess a >>"last checked time" database would be needed. > > > That's actually the point of the patch. It adjusts a user's allowable > check frequency based on his or her individual usage, and remembers the > last check time for each user separately. > > The only thing in this that's not done is allowing the response message > to be configured. And that's necessary; I won't change it evenly to "No > new messages" across the board. The original proof of concept patch did > that, and our support center objected that they'd get calls from people > who got "no new messages" and then a big flood of new messages on the > next check. They *wanted* an error response. > > I'll make it a runtime option. > Thank you! I think the 'no new messages' is probably preferable if you intend to set happymail-base to a low value. There should be no big flood of mail in this case. Where rate limiting could result in larger delays for mail, then the error message is probably preferable. Making it a runtime option is perfect way to allow for both configurations of HappyMail. Ken Anderson Pacific.Net
Re: Invalid cross-device link
ive.org> <[EMAIL PROTECTED]> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <[EMAIL PROTECTED]> X-Comment: no User-Agent: Mutt/1.5.8i X-Decanonizer: current version 0.2005.05.10 for j4RMOhww016410 from localhost [127.0.0.1] at 1117232684: Fri May 27 22:24:44 2005 [0.172s] * On 2005.05.27, in <[EMAIL PROTECTED]>, * "Daniel Senie" <[EMAIL PROTECTED]> wrote: > > yes, it'd have to happen without actually reading the mailbox. Guess a > "last checked time" database would be needed. That's actually the point of the patch. It adjusts a user's allowable check frequency based on his or her individual usage, and remembers the last check time for each user separately. The only thing in this that's not done is allowing the response message to be configured. And that's necessary; I won't change it evenly to "No new messages" across the board. The original proof of concept patch did that, and our support center objected that they'd get calls from people who got "no new messages" and then a big flood of new messages on the next check. They *wanted* an error response. I'll make it a runtime option. -- -D.[EMAIL PROTECTED]NSITUniversity of Chicago
Re: Invalid cross-device link
At 11:06 AM 5/27/2005, Ken A wrote: ive.org> In-Reply-To: <[EMAIL PROTECTED]> X-Enigmail-Version: 0.90.0.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Joe Maimon wrote: > > > Daniel Senie wrote: > >> At 05:18 PM 5/26/2005, Clifton Royston wrote: >> >>> On Thu, May 26, 2005 at 06:26:42AM -0400, Joe Maimon wrote: >>> > David Champion wrote: >>> > >* On 2005.05.25, in <[EMAIL PROTECTED]>, >>> > >*"Ken A" <[EMAIL PROTECTED]> wrote: >>> > >>> > > >>> > >Disclaimer: we don't really use this anymore, as we've been hauled >>> > >kicking and screaming by upper management into carefree turnkey mail >>> > >appliance blissful all day long land. So I don't know how well this >>> > >works with current qpoppers -- it was developed on 4.0.5, and I don't >>> > >really maintain it anymore, since I can't defend that to mgmt and >>> have >>> > >enough other projects for my free time. (If anyone who uses it is >>> > >interested in taking over, please let me know.) >>> > >>> > I use it (patched) and I host patches for it. >>> > Works EXCELLENT for quite some time. >>> >>> This is the "Happymail" patch we're talking about here, right? >>> >>> If so, it pretty much solved all our mailserver load headaches. The >>> bigger the mailbox, the less often they are allowed to check it; this >>> makes the load on the server virtually flat. >>> >>> I also enthusiastically endorse it. >>> >> >> Are these features that would make sense to consider integrating into >> the qpopper code base and configuring with options? >> >> > > I would vote for rate limiting by user,ip relevant to mailbox size to be > a standard feature, whether it is by use of this aproach or any other. > > I would also like to see some way to limit users (per user) to a number of pop3 checks per minute, but without generating support calls because of an error message. It would be better to simply return "no new messages on server" for x minutes if possible (still with no i/o). I'm not at all sure how difficult that change would be to implement. Yes, this would be my preference too. We get enough support calls from people who reboot their computer during a download and then can't get back in while the server waits to time out their TCP session. Don't need more from this case. Returning a "thanks, no new messages" would be perfect. And yes, it'd have to happen without actually reading the mailbox. Guess a "last checked time" database would be needed. Short of that, I'd definitely like to see the HappyMail patch put into the main codebase. Ken Anderson Pacific.Net
Re: Invalid cross-device link
ive.org> <[EMAIL PROTECTED]> In-Reply-To: <[EMAIL PROTECTED]> X-Enigmail-Version: 0.90.0.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit David Champion wrote: > [Daniel Senie:] > >>>Are these features that would make sense to consider integrating into >>>the qpopper code base and configuring with options? > > > I think so (perhaps obviously). We used the patch for about three years > between developing it and being switched over to a mail appliance for > most of our users. It's still in production here for a small subset of > users, though. > > The happymail features are all configurable with config file and/or > command-line options, and they are completely inactive if you don't set > those options. It might be worth a compile-time option in configure.in, > but only (to my thinking) to detect whether the system supports System V > shared memory. > > > * On 2005.05.27, in <[EMAIL PROTECTED]>, > * "Ken A" <[EMAIL PROTECTED]> wrote: > >>I would also like to see some way to limit users (per user) to a number >>of pop3 checks per minute, but without generating support calls because >>of an error message. It would be better to simply return "no new >>messages on server" for x minutes if possible (still with no i/o). I'm >>not at all sure how difficult that change would be to implement. > > > This can be a runtime option. > That would be ideal. We frequently have hundreds of 50-60MB mailboxes that are checked every 1 minute. Simply reducing that to every 5 minutes, without generating support calls would make a huge difference. Thanks, Ken Anderson Pacific.Net > >>Short of that, I'd definitely like to see the HappyMail patch put into >>the main codebase. > > > I know I've said this before, but I'll look soon at the pending requests > on happymail and at Joe's patches, and put together an updated, > integrated patch suitable for inclusion in the core code base. If > nothing else, it'll be a better basis for a new maintainer, and a more > respectable handoff on my part. :) >
Re: Invalid cross-device link
ive.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <[EMAIL PROTECTED]> X-Comment: no User-Agent: Mutt/1.5.8i X-Decanonizer: decanon-milter version 0.2005.05.10 for j4RFQS1W022264 from localhost [127.0.0.1] at 1117207588: Fri May 27 15:26:28 2005 [0.165s] [Daniel Senie:] >> Are these features that would make sense to consider integrating into >> the qpopper code base and configuring with options? I think so (perhaps obviously). We used the patch for about three years between developing it and being switched over to a mail appliance for most of our users. It's still in production here for a small subset of users, though. The happymail features are all configurable with config file and/or command-line options, and they are completely inactive if you don't set those options. It might be worth a compile-time option in configure.in, but only (to my thinking) to detect whether the system supports System V shared memory. * On 2005.05.27, in <[EMAIL PROTECTED]>, * "Ken A" <[EMAIL PROTECTED]> wrote: > > I would also like to see some way to limit users (per user) to a number > of pop3 checks per minute, but without generating support calls because > of an error message. It would be better to simply return "no new > messages on server" for x minutes if possible (still with no i/o). I'm > not at all sure how difficult that change would be to implement. This can be a runtime option. > Short of that, I'd definitely like to see the HappyMail patch put into > the main codebase. I know I've said this before, but I'll look soon at the pending requests on happymail and at Joe's patches, and put together an updated, integrated patch suitable for inclusion in the core code base. If nothing else, it'll be a better basis for a new maintainer, and a more respectable handoff on my part. :) -- -D.[EMAIL PROTECTED]NSITUniversity of Chicago
Re: Invalid cross-device link
<[EMAIL PROTECTED]> In-Reply-To: <[EMAIL PROTECTED]> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Ken A wrote: > > of an error message. It would be better to simply return "no new > messages on server" for x minutes if possible (still with no i/o). I'm > not at all sure how difficult that change would be to implement. This is an interesting idea. > > Short of that, I'd definitely like to see the HappyMail patch put into > the main codebase. > > Ken Anderson > Pacific.Net > >
Re: Invalid cross-device link
ive.org> In-Reply-To: <[EMAIL PROTECTED]> X-Enigmail-Version: 0.90.0.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Joe Maimon wrote: > > > Daniel Senie wrote: > >> At 05:18 PM 5/26/2005, Clifton Royston wrote: >> >>> On Thu, May 26, 2005 at 06:26:42AM -0400, Joe Maimon wrote: >>> > David Champion wrote: >>> > >* On 2005.05.25, in <[EMAIL PROTECTED]>, >>> > >*"Ken A" <[EMAIL PROTECTED]> wrote: >>> > >>> > > >>> > >Disclaimer: we don't really use this anymore, as we've been hauled >>> > >kicking and screaming by upper management into carefree turnkey mail >>> > >appliance blissful all day long land. So I don't know how well this >>> > >works with current qpoppers -- it was developed on 4.0.5, and I don't >>> > >really maintain it anymore, since I can't defend that to mgmt and >>> have >>> > >enough other projects for my free time. (If anyone who uses it is >>> > >interested in taking over, please let me know.) >>> > >>> > I use it (patched) and I host patches for it. >>> > Works EXCELLENT for quite some time. >>> >>> This is the "Happymail" patch we're talking about here, right? >>> >>> If so, it pretty much solved all our mailserver load headaches. The >>> bigger the mailbox, the less often they are allowed to check it; this >>> makes the load on the server virtually flat. >>> >>> I also enthusiastically endorse it. >>> >> >> Are these features that would make sense to consider integrating into >> the qpopper code base and configuring with options? >> >> > > I would vote for rate limiting by user,ip relevant to mailbox size to be > a standard feature, whether it is by use of this aproach or any other. > > I would also like to see some way to limit users (per user) to a number of pop3 checks per minute, but without generating support calls because of an error message. It would be better to simply return "no new messages on server" for x minutes if possible (still with no i/o). I'm not at all sure how difficult that change would be to implement. Short of that, I'd definitely like to see the HappyMail patch put into the main codebase. Ken Anderson Pacific.Net
Re: Invalid cross-device link
Joe Maimon wrote: David Champion wrote: * On 2005.05.25, in <[EMAIL PROTECTED]>, *"Ken A" <[EMAIL PROTECTED]> wrote: Disclaimer: we don't really use this anymore, as we've been hauled kicking and screaming by upper management into carefree turnkey mail appliance blissful all day long land. So I don't know how well this works with current qpoppers -- it was developed on 4.0.5, and I don't really maintain it anymore, since I can't defend that to mgmt and have enough other projects for my free time. (If anyone who uses it is interested in taking over, please let me know.) I use it (patched) and I host patches for it. Works EXCELLENT for quite some time. I havent tested the patches with latest version yet http://www.jmaimon.com/qpopper ( the above was supposed to read like this: I use a patched happymail patch ) There are now 4.0.8 versions of the patches at http://www.jmaimon.com/qpopper http://www.jmaimon.com/qpopper/patches Any and all feedback welcome.
Re: Invalid cross-device link
Daniel Senie wrote: At 05:18 PM 5/26/2005, Clifton Royston wrote: On Thu, May 26, 2005 at 06:26:42AM -0400, Joe Maimon wrote: > David Champion wrote: > >* On 2005.05.25, in <[EMAIL PROTECTED]>, > >*"Ken A" <[EMAIL PROTECTED]> wrote: > > > > >Disclaimer: we don't really use this anymore, as we've been hauled > >kicking and screaming by upper management into carefree turnkey mail > >appliance blissful all day long land. So I don't know how well this > >works with current qpoppers -- it was developed on 4.0.5, and I don't > >really maintain it anymore, since I can't defend that to mgmt and have > >enough other projects for my free time. (If anyone who uses it is > >interested in taking over, please let me know.) > > I use it (patched) and I host patches for it. > Works EXCELLENT for quite some time. This is the "Happymail" patch we're talking about here, right? If so, it pretty much solved all our mailserver load headaches. The bigger the mailbox, the less often they are allowed to check it; this makes the load on the server virtually flat. I also enthusiastically endorse it. Are these features that would make sense to consider integrating into the qpopper code base and configuring with options? I would vote for rate limiting by user,ip relevant to mailbox size to be a standard feature, whether it is by use of this aproach or any other.
Re: Invalid cross-device link
At 05:18 PM 5/26/2005, Clifton Royston wrote: On Thu, May 26, 2005 at 06:26:42AM -0400, Joe Maimon wrote: > David Champion wrote: > >* On 2005.05.25, in <[EMAIL PROTECTED]>, > >*"Ken A" <[EMAIL PROTECTED]> wrote: > > > > >Disclaimer: we don't really use this anymore, as we've been hauled > >kicking and screaming by upper management into carefree turnkey mail > >appliance blissful all day long land. So I don't know how well this > >works with current qpoppers -- it was developed on 4.0.5, and I don't > >really maintain it anymore, since I can't defend that to mgmt and have > >enough other projects for my free time. (If anyone who uses it is > >interested in taking over, please let me know.) > > I use it (patched) and I host patches for it. > Works EXCELLENT for quite some time. This is the "Happymail" patch we're talking about here, right? If so, it pretty much solved all our mailserver load headaches. The bigger the mailbox, the less often they are allowed to check it; this makes the load on the server virtually flat. I also enthusiastically endorse it. Are these features that would make sense to consider integrating into the qpopper code base and configuring with options?
Re: Invalid cross-device link
On Thu, May 26, 2005 at 06:26:42AM -0400, Joe Maimon wrote: > David Champion wrote: > >* On 2005.05.25, in <[EMAIL PROTECTED]>, > >*"Ken A" <[EMAIL PROTECTED]> wrote: > > > > >Disclaimer: we don't really use this anymore, as we've been hauled > >kicking and screaming by upper management into carefree turnkey mail > >appliance blissful all day long land. So I don't know how well this > >works with current qpoppers -- it was developed on 4.0.5, and I don't > >really maintain it anymore, since I can't defend that to mgmt and have > >enough other projects for my free time. (If anyone who uses it is > >interested in taking over, please let me know.) > > I use it (patched) and I host patches for it. > Works EXCELLENT for quite some time. This is the "Happymail" patch we're talking about here, right? If so, it pretty much solved all our mailserver load headaches. The bigger the mailbox, the less often they are allowed to check it; this makes the load on the server virtually flat. I also enthusiastically endorse it. -- Clifton -- Clifton Royston -- [EMAIL PROTECTED] Tiki Technologies Lead Programmer/Software Architect "I'm gonna tell my son to grow up pretty as the grass is green And whip-smart as the English Channel's wide..." -- 'Whip-Smart', Liz Phair
Re: Invalid cross-device link
On Thu, May 26, 2005 at 10:39:11AM -0700, Ken A wrote:
> Clifton Royston wrote:
> >On Wed, May 25, 2005 at 02:47:39PM -0700, Ken A wrote:
...
> >>I think found a way to work around this, at least with linux kernel 2.4
> >>and up. The fix is to put the temp drop on a different partition so the
> >>quota system doesn't see it, and then use the vfs feature "mount --bind"
> >>to remount the temp drop filesystem as part of the /var/spool
> >>filesystem. ie: mount --bind /var/tempdrop /var/spool/tempdrop
> >>
> >>This allows qpopper to rename files using fast-update, and since the
> >>quota system is looking at the underlying filesystem, it ignores files
> >>in the temp drop directory.
...
> > I just talked to a more Linux-oriented friend who thinks this might
> >simply result in the same file copy happening, but hidden from you
> >because it's been pushed down into the kernel level where vfs resides.
> >He's not sure, though. Before concluding that this really had the
> >effect you want, I would try turning on detailed timing info; you might
> >find that the "relink" is now taking approximately as much time as the
> >file copy used to, because it's now actually doing a file copy.
> > I'll be interested to hear the results.
>
> Qpopper's --enable-timing only resolves to seconds afaik, so I did some
> testing copying a large file with perl's Time::HiRes, and system("mv").
> I'm assuming mv uses relink() just like qpopper. (not much of a C
> programmer here!)
>
> Apparently, the mount --bind option doesn't really give any advantage
> over leaving the temp-drop on a different partition. :-(
>
> Time to copy on same partition: 0.147061
> Time to move on same partition: 0.001959
> Time to move to with bound partition: 0.150901
> Time to move across partitions: 0.151158
Oh well, TANSTAAFL, I guess. I was really hoping you were on to
something here, but logic was telling me it couldn't be that good. :-/
Thanks for the followup!
-- Clifton
--
Clifton Royston -- [EMAIL PROTECTED]
Tiki Technologies Lead Programmer/Software Architect
"I'm gonna tell my son to grow up pretty as the grass is green
And whip-smart as the English Channel's wide..."
-- 'Whip-Smart', Liz Phair
Re: Invalid cross-device link
Clifton Royston wrote:
On Wed, May 25, 2005 at 02:47:39PM -0700, Ken A wrote:
Clifton Royston wrote:
On Wed, May 25, 2005 at 10:51:58AM -0700, Ken A wrote:
I'm running the latest qpopper on FC3 with user quotas turned on.
With fast-update on, I see "Invalid cross-device link (18)
[pop_updt.c:770]" in the logs. I know this is normal for fast-update
when the spool and temp drop are on different partitions because of the
way rename works.
But, if I put the temp drop on the same filesystem as the mail spool,
users can't check mail when they hit their hard quota. :-(
I'd like to have the temp drop on a different partition so quotas work,
but still use fast-update, since I'd like to keep disk i/o down as much
as possible. I'd appreciate any ideas.
This is inherently impossible due to issues with the Unix quota model
and filesystems. "fast-update" is a simple relink, i.e. rename, to
replace the spool when the ideal case is hit; that is how it can do
essentially no I/O at this step. Obviously this can only happen when
the two files exist on the same filesystem (indeed, in any OS I know
of, not just UNIX.)
However, it's not possible in the UNIX model to have a user quota
that applies only to a portion of a filesystem, or to "all files except
...". That sharply limits the usefulness of hard quotas if you have
the spool and the temp drop on the same filesystem, as you say.
...
I think found a way to work around this, at least with linux kernel 2.4
and up. The fix is to put the temp drop on a different partition so the
quota system doesn't see it, and then use the vfs feature "mount --bind"
to remount the temp drop filesystem as part of the /var/spool
filesystem. ie: mount --bind /var/tempdrop /var/spool/tempdrop
This allows qpopper to rename files using fast-update, and since the
quota system is looking at the underlying filesystem, it ignores files
in the temp drop directory.
Wow. I guess I never thought of trying something like that; I've got
to think about how that would work. Other OSes may have something
similar available via unionfs mounts or the like, so it might be
feasible to do in other OSes.
I'm not sure if this is intended behavior for quotas or not, so it may
change if the quota system changes, but it certainly makes the using the
quota system more 'flexible'. Please let me know if you see any problems
with this approach.
I just talked to a more Linux-oriented friend who thinks this might
simply result in the same file copy happening, but hidden from you
because it's been pushed down into the kernel level where vfs resides.
He's not sure, though. Before concluding that this really had the
effect you want, I would try turning on detailed timing info; you might
find that the "relink" is now taking approximately as much time as the
file copy used to, because it's now actually doing a file copy.
--
I'll be interested to hear the results.
Qpopper's --enable-timing only resolves to seconds afaik, so I did some
testing copying a large file with perl's Time::HiRes, and system("mv").
I'm assuming mv uses relink() just like qpopper. (not much of a C
programmer here!)
Apparently, the mount --bind option doesn't really give any advantage
over leaving the temp-drop on a different partition. :-(
Time to copy on same partition: 0.147061
Time to move on same partition: 0.001959
Time to move to with bound partition: 0.150901
Time to move across partitions: 0.151158
Thanks,
Ken Anderson
Pacific.net
-- Clifton
Re: Invalid cross-device link
David Champion wrote: * On 2005.05.25, in <[EMAIL PROTECTED]>, * "Ken A" <[EMAIL PROTECTED]> wrote: Disclaimer: we don't really use this anymore, as we've been hauled kicking and screaming by upper management into carefree turnkey mail appliance blissful all day long land. So I don't know how well this works with current qpoppers -- it was developed on 4.0.5, and I don't really maintain it anymore, since I can't defend that to mgmt and have enough other projects for my free time. (If anyone who uses it is interested in taking over, please let me know.) I use it (patched) and I host patches for it. Works EXCELLENT for quite some time. I havent tested the patches with latest version yet http://www.jmaimon.com/qpopper
Re: Invalid cross-device link
On Wed, May 25, 2005 at 02:47:39PM -0700, Ken A wrote: > Clifton Royston wrote: > >On Wed, May 25, 2005 at 10:51:58AM -0700, Ken A wrote: > >>I'm running the latest qpopper on FC3 with user quotas turned on. > >> > >>With fast-update on, I see "Invalid cross-device link (18) > >>[pop_updt.c:770]" in the logs. I know this is normal for fast-update > >>when the spool and temp drop are on different partitions because of the > >>way rename works. > >> > >>But, if I put the temp drop on the same filesystem as the mail spool, > >>users can't check mail when they hit their hard quota. :-( > >> > >>I'd like to have the temp drop on a different partition so quotas work, > >>but still use fast-update, since I'd like to keep disk i/o down as much > >>as possible. I'd appreciate any ideas. > > > > This is inherently impossible due to issues with the Unix quota model > >and filesystems. "fast-update" is a simple relink, i.e. rename, to > >replace the spool when the ideal case is hit; that is how it can do > >essentially no I/O at this step. Obviously this can only happen when > >the two files exist on the same filesystem (indeed, in any OS I know > >of, not just UNIX.) > > > > However, it's not possible in the UNIX model to have a user quota > >that applies only to a portion of a filesystem, or to "all files except > >...". That sharply limits the usefulness of hard quotas if you have > >the spool and the temp drop on the same filesystem, as you say. > > ... > I think found a way to work around this, at least with linux kernel 2.4 > and up. The fix is to put the temp drop on a different partition so the > quota system doesn't see it, and then use the vfs feature "mount --bind" > to remount the temp drop filesystem as part of the /var/spool > filesystem. ie: mount --bind /var/tempdrop /var/spool/tempdrop > > This allows qpopper to rename files using fast-update, and since the > quota system is looking at the underlying filesystem, it ignores files > in the temp drop directory. Wow. I guess I never thought of trying something like that; I've got to think about how that would work. Other OSes may have something similar available via unionfs mounts or the like, so it might be feasible to do in other OSes. > I'm not sure if this is intended behavior for quotas or not, so it may > change if the quota system changes, but it certainly makes the using the > quota system more 'flexible'. Please let me know if you see any problems > with this approach. I just talked to a more Linux-oriented friend who thinks this might simply result in the same file copy happening, but hidden from you because it's been pushed down into the kernel level where vfs resides. He's not sure, though. Before concluding that this really had the effect you want, I would try turning on detailed timing info; you might find that the "relink" is now taking approximately as much time as the file copy used to, because it's now actually doing a file copy. I'll be interested to hear the results. -- Clifton -- Clifton Royston -- [EMAIL PROTECTED] Tiki Technologies Lead Programmer/Software Architect "I'm gonna tell my son to grow up pretty as the grass is green And whip-smart as the English Channel's wide..." -- 'Whip-Smart', Liz Phair
Re: Invalid cross-device link
Clifton Royston wrote: On Wed, May 25, 2005 at 10:51:58AM -0700, Ken A wrote: I'm running the latest qpopper on FC3 with user quotas turned on. With fast-update on, I see "Invalid cross-device link (18) [pop_updt.c:770]" in the logs. I know this is normal for fast-update when the spool and temp drop are on different partitions because of the way rename works. But, if I put the temp drop on the same filesystem as the mail spool, users can't check mail when they hit their hard quota. :-( I'd like to have the temp drop on a different partition so quotas work, but still use fast-update, since I'd like to keep disk i/o down as much as possible. I'd appreciate any ideas. This is inherently impossible due to issues with the Unix quota model and filesystems. "fast-update" is a simple relink, i.e. rename, to replace the spool when the ideal case is hit; that is how it can do essentially no I/O at this step. Obviously this can only happen when the two files exist on the same filesystem (indeed, in any OS I know of, not just UNIX.) However, it's not possible in the UNIX model to have a user quota that applies only to a portion of a filesystem, or to "all files except ...". That sharply limits the usefulness of hard quotas if you have the spool and the temp drop on the same filesystem, as you say. Finally, it would be a severe security risk if qpopper were to hold superuser privileges until the end of the session, when they would be needed if one were to juggle file ownership at that point (e.g. by making the tempdrop file qpopper-owned and group writable until it were swapped into place with the user's spool.) AFAIK, nobody's come up with a better way to juggle files that would reconcile these issues. -- Clifton I think found a way to work around this, at least with linux kernel 2.4 and up. The fix is to put the temp drop on a different partition so the quota system doesn't see it, and then use the vfs feature "mount --bind" to remount the temp drop filesystem as part of the /var/spool filesystem. ie: mount --bind /var/tempdrop /var/spool/tempdrop This allows qpopper to rename files using fast-update, and since the quota system is looking at the underlying filesystem, it ignores files in the temp drop directory. I'm not sure if this is intended behavior for quotas or not, so it may change if the quota system changes, but it certainly makes the using the quota system more 'flexible'. Please let me know if you see any problems with this approach. Thanks, Ken Anderson Pacific.Net
Re: Invalid cross-device link
David Champion wrote: * On 2005.05.25, in <[EMAIL PROTECTED]>, * "Ken A" <[EMAIL PROTECTED]> wrote: But, if I put the temp drop on the same filesystem as the mail spool, users can't check mail when they hit their hard quota. :-( I'd like to have the temp drop on a different partition so quotas work, but still use fast-update, since I'd like to keep disk i/o down as much as possible. I'd appreciate any ideas. Quotas have always been a problem for us. There have been enough issues that we basically couldn't't use filesystem-based quotas, and that's one of the inspirations for the "happymail" extensions we developed. It's not a direct answer to you situation, but it obliquely provides a workaround for I/O issues, if not to storage constraints. http://home.uchicago.edu/~dgc/sw/qpopper/index.html Disclaimer: we don't really use this anymore, as we've been hauled kicking and screaming by upper management into carefree turnkey mail appliance blissful all day long land. So I don't know how well this works with current qpoppers -- it was developed on 4.0.5, and I don't really maintain it anymore, since I can't defend that to mgmt and have enough other projects for my free time. (If anyone who uses it is interested in taking over, please let me know.) Wow. That's very cool. I've been looking for a way to stop those 'check every 1 minute' pop users. I think our support dept would be overwhelmed with calls if users got "Please wait at least x minutes between checks". I'd rather it just returned a fake "no new messages" response for x minutes. Thanks, Ken Anderson Pacific.Net
Re: Invalid cross-device link
On Wed, May 25, 2005 at 10:51:58AM -0700, Ken A wrote: > I'm running the latest qpopper on FC3 with user quotas turned on. > > With fast-update on, I see "Invalid cross-device link (18) > [pop_updt.c:770]" in the logs. I know this is normal for fast-update > when the spool and temp drop are on different partitions because of the > way rename works. > > But, if I put the temp drop on the same filesystem as the mail spool, > users can't check mail when they hit their hard quota. :-( > > I'd like to have the temp drop on a different partition so quotas work, > but still use fast-update, since I'd like to keep disk i/o down as much > as possible. I'd appreciate any ideas. This is inherently impossible due to issues with the Unix quota model and filesystems. "fast-update" is a simple relink, i.e. rename, to replace the spool when the ideal case is hit; that is how it can do essentially no I/O at this step. Obviously this can only happen when the two files exist on the same filesystem (indeed, in any OS I know of, not just UNIX.) However, it's not possible in the UNIX model to have a user quota that applies only to a portion of a filesystem, or to "all files except ...". That sharply limits the usefulness of hard quotas if you have the spool and the temp drop on the same filesystem, as you say. Finally, it would be a severe security risk if qpopper were to hold superuser privileges until the end of the session, when they would be needed if one were to juggle file ownership at that point (e.g. by making the tempdrop file qpopper-owned and group writable until it were swapped into place with the user's spool.) AFAIK, nobody's come up with a better way to juggle files that would reconcile these issues. -- Clifton -- Clifton Royston -- [EMAIL PROTECTED] Tiki Technologies Lead Programmer/Software Architect "I'm gonna tell my son to grow up pretty as the grass is green And whip-smart as the English Channel's wide..." -- 'Whip-Smart', Liz Phair
Re: Invalid cross-device link
* On 2005.05.25, in <[EMAIL PROTECTED]>, * "Ken A" <[EMAIL PROTECTED]> wrote: > > But, if I put the temp drop on the same filesystem as the mail spool, > users can't check mail when they hit their hard quota. :-( > > I'd like to have the temp drop on a different partition so quotas work, > but still use fast-update, since I'd like to keep disk i/o down as much > as possible. I'd appreciate any ideas. Quotas have always been a problem for us. There have been enough issues that we basically couldn't't use filesystem-based quotas, and that's one of the inspirations for the "happymail" extensions we developed. It's not a direct answer to you situation, but it obliquely provides a workaround for I/O issues, if not to storage constraints. http://home.uchicago.edu/~dgc/sw/qpopper/index.html Disclaimer: we don't really use this anymore, as we've been hauled kicking and screaming by upper management into carefree turnkey mail appliance blissful all day long land. So I don't know how well this works with current qpoppers -- it was developed on 4.0.5, and I don't really maintain it anymore, since I can't defend that to mgmt and have enough other projects for my free time. (If anyone who uses it is interested in taking over, please let me know.) -- -D.[EMAIL PROTECTED]NSITUniversity of Chicago
Re: Invalid cross-device link
i don't believe you can do fast update with seperate partitions/drives. You'd have to use Maildir to solve this, in which case you don't need a temp drop or would have a quota problem. -Tony -.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.- Anthony J. BiaccoSystems/Network Administrator [EMAIL PROTECTED] http://www.asteroid-b612.org "as I always say, why go Merlot, when you can call a Cab?" .-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.- On Wed, 25 May 2005, Ken A wrote: > Hello, > > I'm running the latest qpopper on FC3 with user quotas turned on. > > With fast-update on, I see "Invalid cross-device link (18) > [pop_updt.c:770]" in the logs. I know this is normal for fast-update > when the spool and temp drop are on different partitions because of the > way rename works. > > But, if I put the temp drop on the same filesystem as the mail spool, > users can't check mail when they hit their hard quota. :-( > > I'd like to have the temp drop on a different partition so quotas work, > but still use fast-update, since I'd like to keep disk i/o down as much > as possible. I'd appreciate any ideas. > > Thanks, > Ken Anderson > Pacific.Net > >
Re: Invalid cross-device link
At 7:44 AM -0500 4/30/01, Ted George wrote: > we are running a redhat 6.2 box as a pop mail server with qpopper in > server mode. we have been using qpopper 3.1.2 and several versions > before that. recently tried 4.0 and everything looked ok except these > errors started showing up in the logs. > > Apr 29 11:45:54 mail popper[18125]: Unable to move > /var/spool/poptemp/.username.pop > to /var/spool/mail/username: Invalid cross-device link (18) > > i built the binary with > > --enable-servermode > --enable-temp-drop-dir=/var/spool/poptemp > --enable-cache-dir=/var/spool/popcache > > did not use the enable-cache before with 3.1.2. all other options the > same as before. the only other thing we did different with 4.0 is using > the -F option from the command line, popper -sF. also we run a quota on > /var/spoo/mail, and no quota on the other 2 dirs (the other dirs have > plenty of available space). does 4.0 now need more space > /var/spool/mail even though it was successfully using the poptemp and > popcache dirs, we verified files were being created there. > > does anyone have any information on this error, we have downgraded back > to 3.1.2 until we can find out what caused this > > thanks in advance There is no need to fall back to 3.1.2. All that is happening is that you told Qpopper 4.0 to move instead of copy the spool during the end-of-session update; when Qpopper tried to do so it got an error, so it let you know (and of course fell back on copying). So you can continue using Qpopper 4, and either (a) ignore the errors, (b) turn off the '-F' option, or (c) make sure that your temp spool directory is on the same file system as your spool directory.
