RE: Header parsing

[Loyall, David]

Additional information:

Capture.PNG is a screen shot of the email that software-lab.de sent me, which 
contained my own message.

So I wondered if my own email that I sent to software-lab.de contained the 
extra data!

To check this, I recreated my original message, step by step.. but changed the 
TO: address to another address instead of picolisp@software-lab.de...

By that method, here is what I originally sent to the list:

Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0

PiBJZiBwZW9wbGUgYXJlIGN1cnJlbnRseSB3b3JraW5nIG9uIGltcHJvdmluZyB0aGUgbWFpbGlu
ZyBsaXN0Lg0KDQpJIHdvdWxkIGxpa2UgdG8gc3RvcCByZWNlaXZpbmcgYmluYXJ5IGp1bmsgYXQg
dGhlIGVuZCBvZiBwbGFpbi10ZXh0IGVtYWlscy4NCg0K

I hope that helps!

Cheers,
--Dave

-Original Message-
From: picolisp@software-lab.de [mailto:picolisp@software-lab.de] On Behalf Of 
Loyall, David
Sent: Thursday, April 20, 2017 1:11 PM
To: picolisp@software-lab.de
Subject: RE: Header parsing

See attached PNG.

[snip]

RE: Header parsing

[Loyall, David]

See attached PNG.

I wasn't able to view the message source in Outlook, so I forwarded it to my 
personal account and got the following, which may or may not be exactly what I 
originally received...

Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0

DQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBwaWNvbGlzcEBzb2Z0d2FyZS1s
YWIuZGUgW21haWx0bzpwaWNvbGlzcEBzb2Z0d2FyZS1sYWIuZGVdIE9uIEJlaGFsZiBPZiBMb3lh
bGwsIERhdmlkDQpTZW50OiBXZWRuZXNkYXksIEFwcmlsIDE5LCAyMDE3IDM6NDAgUE0NClRvOiBw
aWNvbGlzcEBzb2Z0d2FyZS1sYWIuZGUNClN1YmplY3Q6IFJFOiBIZWFkZXIgcGFyc2luZw0KDQo+
IElmIHBlb3BsZSBhcmUgY3VycmVudGx5IHdvcmtpbmcgb24gaW1wcm92aW5nIHRoZSBtYWlsaW5n
IGxpc3QgWy4uLl0gDQoNCkkgd291bGQgbGlrZSB0byBzdG9wIHJlY2VpdmluZyBiaW5hcnkganVu
ayBhdCB0aGUgZW5kIG9mIHBsYWluLXRleHQgZW1haWxzLg0KBQ0KSUBSCRIBEmbvv73vv73vv70p
77+977+9Je+/ve+/vWzvv73vv71wau+/ve+/vWnvv71e77+977+977+9ee+/vVTvv73Lm++/ve+/
ve+/vW0NCg==

Cheers,
--Dave

-Original Message-
From: picolisp@software-lab.de [mailto:picolisp@software-lab.de] On Behalf Of 
Alexander Burger
Sent: Thursday, April 20, 2017 4:54 AM
To: picolisp@software-lab.de
Subject: Re: Header parsing

Hi David,

> I would like to stop receiving binary junk at the end of plain-text emails.

Strange, what kind of junk might that be? I'm not aware of any ...
♪♫ Alex

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Header parsing

[Alexander Burger]

Hi David,

> I would like to stop receiving binary junk at the end of plain-text emails.

Strange, what kind of junk might that be? I'm not aware of any ...
♪♫ Alex

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

RE: Header parsing

[Loyall, David]

> If people are currently working on improving the mailing list [...] 

I would like to stop receiving binary junk at the end of plain-text emails.

Re: Header parsing

[Joh-Tob Schäg]

If people are currently working on improving the mailing list. I would be
very pleased not receive automated emails about thanking people for joining
or leaving.
Has anybody else strong opinions about that?

2017-04-19 22:16 GMT+02:00 Henrik Sarvell :

> Hi Rowan,
>
> If it makes you feel better, I get all your emails because I've created a
> Gmail filter that makes sure picolisp mail list mails always go through the
> filters.
>
>
>
>
>
>
> On Wed, Apr 19, 2017 at 11:57 AM, Rowan Thorpe 
> wrote:
>
>> BTW: I receive DMARC reports about bounced/quarantined mails spoofing
>> my address, and just received one from fastmail which matches the
>> quarantined deliveries vukini reported (they failed validation due to
>> the accidentally forwarded SPF and DKIM headers, as I predicted).
>>
>> .so it seems my posts to the list over the years have mostly reached
>> an audience of one  :-D
>>
>> --
>> Rowan Thorpe
>> http://twitter.com/rowanthorpe
>> "There is a great difference between worry and concern. A
>> worried person sees a problem, and a concerned person solves a
>> problem." - Harold Stephens
>>
>> --
>> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
>>
>
>

Re: Header parsing

[Henrik Sarvell]

Hi Rowan,

If it makes you feel better, I get all your emails because I've created a
Gmail filter that makes sure picolisp mail list mails always go through the
filters.






On Wed, Apr 19, 2017 at 11:57 AM, Rowan Thorpe 
wrote:

> BTW: I receive DMARC reports about bounced/quarantined mails spoofing
> my address, and just received one from fastmail which matches the
> quarantined deliveries vukini reported (they failed validation due to
> the accidentally forwarded SPF and DKIM headers, as I predicted).
>
> ..so it seems my posts to the list over the years have mostly reached
> an audience of one  :-D
>
> --
> Rowan Thorpe
> http://twitter.com/rowanthorpe
> "There is a great difference between worry and concern. A
> worried person sees a problem, and a concerned person solves a
> problem." - Harold Stephens
>
> --
> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
>

Re: Header parsing

[Alexander Burger]

> >  * when the script is processing a file at the same time the
> > mail-server is delivering to it (causing processing of an unfinished
> > message at the end of the file)
> >  * rewinding/truncating the file at the end of processing at the same
> > time the mail-server is delivering to it (not sure what that would
> > cause, but I doubt it would be good)

Sorry everybody! I'm talking nonsense, did probably too many things at the same
time today. And I wrote the script 8 years ago :(

Locking is not an issue at all! Nobody is accessing the spool file at
unpredictable moments, because it is the script itself which triggers
'fetchmail' when done with processing.

♪♫ Alex

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Header parsing

[Tomas Hlavaty]

What about using Maildir on the email server?  Then you don't have any
issues with parsing and locking the spool file.

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Header parsing

[Alexander Burger]

On Wed, Apr 19, 2017 at 05:05:31PM +0300, Rowan Thorpe wrote:
> > On my server this is currently no problem, as I kill the process only at 
> > times
> > when it is not yet starting the next fetch. Still it would be better of 
> > course.
> 
> The race-conditions I mention are not about the script being stopped

Correct. Sorry my confusion! Killing the process is handled by 'protect' anyway,
as you noted.


>  * when the script is processing a file at the same time the
> mail-server is delivering to it (causing processing of an unfinished
> message at the end of the file)
>  * rewinding/truncating the file at the end of processing at the same
> time the mail-server is delivering to it (not sure what that would
> cause, but I doubt it would be good)

True!

> The chances of those things happening are low, but not zero.
> 
> > "postfix -l" doesn't work here.
> 
> Oops, I meant "postconf -l".

It says

   flock
   fcntl
   dotlock

But what does it mean? PicoLisp's 'ctl' uses fcntl(), and sets a exclusive lock
(or shared lock if desired) on the whole file. So will this be obeyed by exim?
If so, it would be easy.


Another option that comes to mind is calling 'mv' on the file before opening it.
'mv' is atomic as long it is on the same device. If the file is still being
written it will all go to the same (old) file descriptor, and new messages
should re-create the file. If we wait after the 'mv' for one or two minutes,
things should be safe too.

♪♫ Alex

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Header parsing

[Alexander Burger]

On Wed, Apr 19, 2017 at 12:41:41PM +0300, Rowan Thorpe wrote:
> PS: Even when using (peek) I think [A] the second patch (for not
> chomping the final "^J") may still be applicable (at least for the
> last email that appears in the spool file), and also *perhaps* [B] the
> first line of the main patch - for skipping trailing timestamp on the
> "From " line (but maybe that was just needed to deal with a

Yes, I kept these of your changes. I uploaded a new release this morning
immediately, so you might check when you have time.


> PPPS: I see you use (protect) to ensure spool-processing is
> uninterruptible by signals, but don't see file-locking of the
> spool-file, to avoid race-conditions with the mail-server during

Yes, this would be nice, but it is not under my control. The pil way would be
with 'ctl', but that is not obeyed by exim.

On my server this is currently no problem, as I kill the process only at times
when it is not yet starting the next fetch. Still it would be better of course.
"postfix -l" doesn't work here.

♪♫ Alex

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Header parsing

[Rowan Thorpe]

BTW: I receive DMARC reports about bounced/quarantined mails spoofing
my address, and just received one from fastmail which matches the
quarantined deliveries vukini reported (they failed validation due to
the accidentally forwarded SPF and DKIM headers, as I predicted).

..so it seems my posts to the list over the years have mostly reached
an audience of one  :-D

-- 
Rowan Thorpe
http://twitter.com/rowanthorpe
"There is a great difference between worry and concern. A
worried person sees a problem, and a concerned person solves a
problem." - Harold Stephens

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Header parsing

[Rowan Thorpe]

On 19 April 2017 at 10:45, Alexander Burger  wrote:
> Hi Rowan,
>
> > On Wed, Apr 19, 2017 at 08:12:11AM +0200, a...@software-lab.de wrote:
> > Rowan, are you sure? I tried your fix, but now it seems the mail
> > body is lost? Any idea?
>
> OK, so I tried a different approach. According to "a CRLF may be inserted 
> before
> any WSP", I simply use (peek) to see if the next line starts with white space.
> Shouldn't this be enough?

Yes, that sounds more like "the right way" :-) ...as long as at least
one byte of push-back is always guaranteed - like for POSIX ungetc() -
and if it doesn't ever cause noticeable performance issues (apparently
not - (till) seems to use the same lookahead anyway).

My patch was a late-night quick-hack adapted from a sed-script of
mine, to the subset of picolisp vocabulary I could easily recall, just
to get the ball rolling (hence my mention that there were surely
better ways to do it). As for my patch eating the message-body during
your test, I can't imagine there is a difference between the
spool-file (mbox) format on your mail-server compared to mine, so
perhaps I fumble-fingered something while preparing the patch?
otherwise the mailing-list might have messed with the attached patches
when inlining them? I won't look into that though because your (peek)
suggestion makes more sense (less invasive) anyway.

PS: Even when using (peek) I think [A] the second patch (for not
chomping the final "^J") may still be applicable (at least for the
last email that appears in the spool file), and also *perhaps* [B] the
first line of the main patch - for skipping trailing timestamp on the
"From " line (but maybe that was just needed to deal with a
side-effect of the sliding-window technique). They were both needed
when testing on my internal Exim server, I'm not sure about Postfix.
The second patch could be tested against your mail-server by sending
it a mail with no trailing blank-line then triggering a run of the
script immediately (so that the mail appears last in the spoolfile),
and looking for something like this:

> ...
> abcdefg: this is the penultimate line
> -- defg: this is the ultimate line
> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

.having said that, I see from http://www.postfix.org/local.8.html

> In the case of UNIX-style mailbox delivery, the local(8) daemon
> prepends a "From sender time_stamp" envelope header  to  each  message,
> ..[snip]..
> and appends an empty line.

so maybe the worst that would happen on postfix is having a
non-conformant empty line with two trailing "\r" characters...

PPS: While tweaking to use the sliding-window technique, I used the following:

 * (chop) first line of each header
 * get (index) of the first " " in the first line
 * split the first line using (head) and (tail) based on that index
 * for each continuation line just (chop) then (conc) them to the tail
without extra splitting-and-gluing-on-space

and I think I saw a speedup (anecdotal, not officially tested). If so,
then perhaps it is worth seeing if that part is useful, even when the
'peek' technique is more elegant than the sliding window technique.

PPPS: I see you use (protect) to ensure spool-processing is
uninterruptible by signals, but don't see file-locking of the
spool-file, to avoid race-conditions with the mail-server during
concurrent runs. Looking at:

 http://www.postfix.org/postconf.5.html#mailbox_delivery_lock

and based on the output of "postfix -l" it should be trivial to add
handling for its configured locking mechanism - just to avoid the
"once in a blue moon" garbling/disappearance of an email being
delivered during processing.

-- 
Rowan Thorpe
http://twitter.com/rowanthorpe
"There is a great difference between worry and concern. A
worried person sees a problem, and a concerned person solves a
problem." - Harold Stephens

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe