RE: Header parsing
Additional information: Capture.PNG is a screen shot of the email that software-lab.de sent me, which contained my own message. So I wondered if my own email that I sent to software-lab.de contained the extra data! To check this, I recreated my original message, step by step.. but changed the TO: address to another address instead of picolisp@software-lab.de... By that method, here is what I originally sent to the list: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 PiBJZiBwZW9wbGUgYXJlIGN1cnJlbnRseSB3b3JraW5nIG9uIGltcHJvdmluZyB0aGUgbWFpbGlu ZyBsaXN0Lg0KDQpJIHdvdWxkIGxpa2UgdG8gc3RvcCByZWNlaXZpbmcgYmluYXJ5IGp1bmsgYXQg dGhlIGVuZCBvZiBwbGFpbi10ZXh0IGVtYWlscy4NCg0K I hope that helps! Cheers, --Dave -Original Message- From: picolisp@software-lab.de [mailto:picolisp@software-lab.de] On Behalf Of Loyall, David Sent: Thursday, April 20, 2017 1:11 PM To: picolisp@software-lab.de Subject: RE: Header parsing See attached PNG. [snip]
RE: Header parsing
See attached PNG. I wasn't able to view the message source in Outlook, so I forwarded it to my personal account and got the following, which may or may not be exactly what I originally received... Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 DQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBwaWNvbGlzcEBzb2Z0d2FyZS1s YWIuZGUgW21haWx0bzpwaWNvbGlzcEBzb2Z0d2FyZS1sYWIuZGVdIE9uIEJlaGFsZiBPZiBMb3lh bGwsIERhdmlkDQpTZW50OiBXZWRuZXNkYXksIEFwcmlsIDE5LCAyMDE3IDM6NDAgUE0NClRvOiBw aWNvbGlzcEBzb2Z0d2FyZS1sYWIuZGUNClN1YmplY3Q6IFJFOiBIZWFkZXIgcGFyc2luZw0KDQo+ IElmIHBlb3BsZSBhcmUgY3VycmVudGx5IHdvcmtpbmcgb24gaW1wcm92aW5nIHRoZSBtYWlsaW5n IGxpc3QgWy4uLl0gDQoNCkkgd291bGQgbGlrZSB0byBzdG9wIHJlY2VpdmluZyBiaW5hcnkganVu ayBhdCB0aGUgZW5kIG9mIHBsYWluLXRleHQgZW1haWxzLg0KBQ0KSUBSCRIBEmbvv73vv73vv70p 77+977+9Je+/ve+/vWzvv73vv71wau+/ve+/vWnvv71e77+977+977+9ee+/vVTvv73Lm++/ve+/ ve+/vW0NCg== Cheers, --Dave -Original Message- From: picolisp@software-lab.de [mailto:picolisp@software-lab.de] On Behalf Of Alexander Burger Sent: Thursday, April 20, 2017 4:54 AM To: picolisp@software-lab.de Subject: Re: Header parsing Hi David, > I would like to stop receiving binary junk at the end of plain-text emails. Strange, what kind of junk might that be? I'm not aware of any ... ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Header parsing
Hi David, > I would like to stop receiving binary junk at the end of plain-text emails. Strange, what kind of junk might that be? I'm not aware of any ... ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
RE: Header parsing
> If people are currently working on improving the mailing list [...] I would like to stop receiving binary junk at the end of plain-text emails.
Re: Header parsing
If people are currently working on improving the mailing list. I would be very pleased not receive automated emails about thanking people for joining or leaving. Has anybody else strong opinions about that? 2017-04-19 22:16 GMT+02:00 Henrik Sarvell : > Hi Rowan, > > If it makes you feel better, I get all your emails because I've created a > Gmail filter that makes sure picolisp mail list mails always go through the > filters. > > > > > > > On Wed, Apr 19, 2017 at 11:57 AM, Rowan Thorpe > wrote: > >> BTW: I receive DMARC reports about bounced/quarantined mails spoofing >> my address, and just received one from fastmail which matches the >> quarantined deliveries vukini reported (they failed validation due to >> the accidentally forwarded SPF and DKIM headers, as I predicted). >> >> .so it seems my posts to the list over the years have mostly reached >> an audience of one :-D >> >> -- >> Rowan Thorpe >> http://twitter.com/rowanthorpe >> "There is a great difference between worry and concern. A >> worried person sees a problem, and a concerned person solves a >> problem." - Harold Stephens >> >> -- >> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe >> > >
Re: Header parsing
Hi Rowan, If it makes you feel better, I get all your emails because I've created a Gmail filter that makes sure picolisp mail list mails always go through the filters. On Wed, Apr 19, 2017 at 11:57 AM, Rowan Thorpe wrote: > BTW: I receive DMARC reports about bounced/quarantined mails spoofing > my address, and just received one from fastmail which matches the > quarantined deliveries vukini reported (they failed validation due to > the accidentally forwarded SPF and DKIM headers, as I predicted). > > ..so it seems my posts to the list over the years have mostly reached > an audience of one :-D > > -- > Rowan Thorpe > http://twitter.com/rowanthorpe > "There is a great difference between worry and concern. A > worried person sees a problem, and a concerned person solves a > problem." - Harold Stephens > > -- > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe >
Re: Header parsing
> > * when the script is processing a file at the same time the > > mail-server is delivering to it (causing processing of an unfinished > > message at the end of the file) > > * rewinding/truncating the file at the end of processing at the same > > time the mail-server is delivering to it (not sure what that would > > cause, but I doubt it would be good) Sorry everybody! I'm talking nonsense, did probably too many things at the same time today. And I wrote the script 8 years ago :( Locking is not an issue at all! Nobody is accessing the spool file at unpredictable moments, because it is the script itself which triggers 'fetchmail' when done with processing. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Header parsing
What about using Maildir on the email server? Then you don't have any issues with parsing and locking the spool file. -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Header parsing
On Wed, Apr 19, 2017 at 05:05:31PM +0300, Rowan Thorpe wrote: > > On my server this is currently no problem, as I kill the process only at > > times > > when it is not yet starting the next fetch. Still it would be better of > > course. > > The race-conditions I mention are not about the script being stopped Correct. Sorry my confusion! Killing the process is handled by 'protect' anyway, as you noted. > * when the script is processing a file at the same time the > mail-server is delivering to it (causing processing of an unfinished > message at the end of the file) > * rewinding/truncating the file at the end of processing at the same > time the mail-server is delivering to it (not sure what that would > cause, but I doubt it would be good) True! > The chances of those things happening are low, but not zero. > > > "postfix -l" doesn't work here. > > Oops, I meant "postconf -l". It says flock fcntl dotlock But what does it mean? PicoLisp's 'ctl' uses fcntl(), and sets a exclusive lock (or shared lock if desired) on the whole file. So will this be obeyed by exim? If so, it would be easy. Another option that comes to mind is calling 'mv' on the file before opening it. 'mv' is atomic as long it is on the same device. If the file is still being written it will all go to the same (old) file descriptor, and new messages should re-create the file. If we wait after the 'mv' for one or two minutes, things should be safe too. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Header parsing
On 19 April 2017 at 15:43, Alexander Burger wrote: > ..[snip].. > > PPPS: I see you use (protect) to ensure spool-processing is > > uninterruptible by signals, but don't see file-locking of the > > spool-file, to avoid race-conditions with the mail-server during > > Yes, this would be nice, but it is not under my control. The pil way would be > with 'ctl', but that is not obeyed by exim. > > On my server this is currently no problem, as I kill the process only at times > when it is not yet starting the next fetch. Still it would be better of > course. The race-conditions I mention are not about the script being stopped part-way through its own processing, but rather: * when the script is processing a file at the same time the mail-server is delivering to it (causing processing of an unfinished message at the end of the file) * rewinding/truncating the file at the end of processing at the same time the mail-server is delivering to it (not sure what that would cause, but I doubt it would be good) The chances of those things happening are low, but not zero. > "postfix -l" doesn't work here. Oops, I meant "postconf -l". -- Rowan Thorpe http://twitter.com/rowanthorpe "There is a great difference between worry and concern. A worried person sees a problem, and a concerned person solves a problem." - Harold Stephens -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Header parsing
On Wed, Apr 19, 2017 at 12:41:41PM +0300, Rowan Thorpe wrote: > PS: Even when using (peek) I think [A] the second patch (for not > chomping the final "^J") may still be applicable (at least for the > last email that appears in the spool file), and also *perhaps* [B] the > first line of the main patch - for skipping trailing timestamp on the > "From " line (but maybe that was just needed to deal with a Yes, I kept these of your changes. I uploaded a new release this morning immediately, so you might check when you have time. > PPPS: I see you use (protect) to ensure spool-processing is > uninterruptible by signals, but don't see file-locking of the > spool-file, to avoid race-conditions with the mail-server during Yes, this would be nice, but it is not under my control. The pil way would be with 'ctl', but that is not obeyed by exim. On my server this is currently no problem, as I kill the process only at times when it is not yet starting the next fetch. Still it would be better of course. "postfix -l" doesn't work here. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Header parsing
BTW: I receive DMARC reports about bounced/quarantined mails spoofing my address, and just received one from fastmail which matches the quarantined deliveries vukini reported (they failed validation due to the accidentally forwarded SPF and DKIM headers, as I predicted). ..so it seems my posts to the list over the years have mostly reached an audience of one :-D -- Rowan Thorpe http://twitter.com/rowanthorpe "There is a great difference between worry and concern. A worried person sees a problem, and a concerned person solves a problem." - Harold Stephens -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Header parsing
On 19 April 2017 at 10:45, Alexander Burger wrote: > Hi Rowan, > > > On Wed, Apr 19, 2017 at 08:12:11AM +0200, a...@software-lab.de wrote: > > Rowan, are you sure? I tried your fix, but now it seems the mail > > body is lost? Any idea? > > OK, so I tried a different approach. According to "a CRLF may be inserted > before > any WSP", I simply use (peek) to see if the next line starts with white space. > Shouldn't this be enough? Yes, that sounds more like "the right way" :-) ...as long as at least one byte of push-back is always guaranteed - like for POSIX ungetc() - and if it doesn't ever cause noticeable performance issues (apparently not - (till) seems to use the same lookahead anyway). My patch was a late-night quick-hack adapted from a sed-script of mine, to the subset of picolisp vocabulary I could easily recall, just to get the ball rolling (hence my mention that there were surely better ways to do it). As for my patch eating the message-body during your test, I can't imagine there is a difference between the spool-file (mbox) format on your mail-server compared to mine, so perhaps I fumble-fingered something while preparing the patch? otherwise the mailing-list might have messed with the attached patches when inlining them? I won't look into that though because your (peek) suggestion makes more sense (less invasive) anyway. PS: Even when using (peek) I think [A] the second patch (for not chomping the final "^J") may still be applicable (at least for the last email that appears in the spool file), and also *perhaps* [B] the first line of the main patch - for skipping trailing timestamp on the "From " line (but maybe that was just needed to deal with a side-effect of the sliding-window technique). They were both needed when testing on my internal Exim server, I'm not sure about Postfix. The second patch could be tested against your mail-server by sending it a mail with no trailing blank-line then triggering a run of the script immediately (so that the mail appears last in the spoolfile), and looking for something like this: > ... > abcdefg: this is the penultimate line > -- defg: this is the ultimate line > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe .having said that, I see from http://www.postfix.org/local.8.html > In the case of UNIX-style mailbox delivery, the local(8) daemon > prepends a "From sender time_stamp" envelope header to each message, > ..[snip].. > and appends an empty line. so maybe the worst that would happen on postfix is having a non-conformant empty line with two trailing "\r" characters... PPS: While tweaking to use the sliding-window technique, I used the following: * (chop) first line of each header * get (index) of the first " " in the first line * split the first line using (head) and (tail) based on that index * for each continuation line just (chop) then (conc) them to the tail without extra splitting-and-gluing-on-space and I think I saw a speedup (anecdotal, not officially tested). If so, then perhaps it is worth seeing if that part is useful, even when the 'peek' technique is more elegant than the sliding window technique. PPPS: I see you use (protect) to ensure spool-processing is uninterruptible by signals, but don't see file-locking of the spool-file, to avoid race-conditions with the mail-server during concurrent runs. Looking at: http://www.postfix.org/postconf.5.html#mailbox_delivery_lock and based on the output of "postfix -l" it should be trivial to add handling for its configured locking mechanism - just to avoid the "once in a blue moon" garbling/disappearance of an email being delivered during processing. -- Rowan Thorpe http://twitter.com/rowanthorpe "There is a great difference between worry and concern. A worried person sees a problem, and a concerned person solves a problem." - Harold Stephens -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe