Re: Customize log messages?

[Wietse Venema]

@ lbutlr:
> On 12/4/16 8:17 AM, Wietse Venema wrote:
> > @ lbutlr:
> >> On 12/3/16 2:57 PM, Wietse Venema wrote:
> >>> Proof of concept:
> >>>
> >>>   MAIL FROM<" >>> type='text/javascript'>alert('xss');"@example.com>
> >>
> >> That result in "501 5.5.4 Syntax: MAIL FROM:"
> >
> > OK, so insert a the missing ':'
> >
> > MAIL FROM:" > type='text/javascript'>alert('xss');"@example.com>
> > 250 2.1.0 Ok
> 
> Fair enough. But the script strips out < and > (and [] and ,), so I'm 
> still not seeing an issue.
> 
> bzgrep "$MATCHPAT" $LOGF | grep -i reject | egrep 'from=<[^>]+>' | grep 
> -v "Protocol error" | grep -v "$EXCLUDE" | sort -u | sed 's/from= tr -d '>,[]:' | grep -v rejected
> 
> I guess the sed only strips the enclosing <, so spurious opening 
> brakcets could be left behind, but the tr -d will take out all the 
> closing >'s. I've added '<' to the tr list just in case, so no <> from 
> the log file will remain.

Good. I think that we have now agreement that some logfile content
is under control by untrusted users.

Wietse

Re: Customize log messages?

On 12/4/16 8:17 AM, Wietse Venema wrote:

@ lbutlr:

On 12/3/16 2:57 PM, Wietse Venema wrote:

Proof of concept:

  MAIL FROM<"alert('xss');"@example.com>


That result in "501 5.5.4 Syntax: MAIL FROM:"


OK, so insert a the missing ':'

MAIL FROM:"alert('xss');"@example.com>
250 2.1.0 Ok


Fair enough. But the script strips out < and > (and [] and ,), so I'm 
still not seeing an issue.


bzgrep "$MATCHPAT" $LOGF | grep -i reject | egrep 'from=<[^>]+>' | grep 
-v "Protocol error" | grep -v "$EXCLUDE" | sort -u | sed 's/from=tr -d '>,[]:' | grep -v rejected


I guess the sed only strips the enclosing <, so spurious opening 
brakcets could be left behind, but the tr -d will take out all the 
closing >'s. I've added '<' to the tr list just in case, so no <> from 
the log file will remain.

Re: Customize log messages?

[Wietse Venema]

@ lbutlr:
> On 12/3/16 2:57 PM, Wietse Venema wrote:
> > Proof of concept:
> >
> >   MAIL FROM<" > type='text/javascript'>alert('xss');"@example.com>
> 
> That result in "501 5.5.4 Syntax: MAIL FROM:"

OK, so insert a the missing ':'

MAIL FROM:"alert('xss');"@example.com>
250 2.1.0 Ok

Instead of an alert, a real attacker would provide more nefarious
code.  This code runs without the user even having to click a link.

Wietse

Re: Customize log messages?

[Benning, Markus]

On 2016-12-02 15:10, Michael Munger wrote:

This is a great idea. This is a spam filter that is integrated into a
CRM system, so I needed to parse and dump the information so it could 
be

sucked up later.

Here's what I ultimately created. It still needs some work (mainly
because it re-reads the whole file every time, and I should use
timestamps and a half-interval search algorithm to find the
last-processed time stamp. I am relying on log rotate to make it
not-too-terribly-big).

https://github.com/mjmunger/postfix-log-parser


Maybe you want to take a look at my saftpresse project:

https://github.com/benningm/saftpresse

Its a event pipe/log analyzer.
Its modular and the Postfix plugin is based on refactured
code of the pflogsum script:

https://github.com/benningm/saftpresse/blob/master/lib/Log/Saftpresse/Plugin/Postfix.pm

It has a syslog and systemd-journald input and could output to 
elasticsearch.

It may be easier to query an elasticsearch index than parsing logs.
Or you just click together some reports with kibana.
I remeber that somewhere there was a plugin or PDF generator for it.

 Markus
--
https://markusbenning.de/

Re: Customize log messages?

[Viktor Dukhovni]

> On Dec 4, 2016, at 12:58 AM, @lbutlr  wrote:
> 
>> MAIL FROM<"> type='text/javascript'>alert('xss');"@example.com>
> 
> That result in "501 5.5.4 Syntax: MAIL FROM:"

There's a missing ":" after FROM.  In any case, even if a particular
exploit mechanism fails, or even all attacks happen to fail, what
you're doing is still unwise.

-- 
Viktor.

Re: Customize log messages?

On 12/3/16 2:57 PM, Wietse Venema wrote:

Proof of concept:

  MAIL FROM<"alert('xss');"@example.com>


That result in "501 5.5.4 Syntax: MAIL FROM:"

Re: Customize log messages?

[Wietse Venema]

Wietse Venema:
> @ lbutlr:
> > > Careful with that.  To easy to create a script injection vector.  Bash is 
> > > not
> > > a good language in which to construct safely quoted remote content for 
> > > injection
> > > into a suitable HTML skeleton.
> > 
> > Injection from where? the script is only accessible to the root user on 
> > the mail server and only checks /var/log/maillog (or the log specified 
> > at the top of the script). There's no remote content involved.
> 
> Injection from the SMTP port.

SMTP session:

  220 mail.example.com
  EHLO client.example
  ...
  MAIL FROM<"some HTML code inside double quotes"@example.com>

Proof of concept:

  MAIL FROM<"alert('xss');"@example.com>

If you read this with a web browser, the following may be more readable:

  MAIL FROM"script 
type='text/javascript'alert('xss');/script"@example.com

Wietse

Re: Customize log messages?

[Wietse Venema]

@ lbutlr:
> > Careful with that.  To easy to create a script injection vector.  Bash is 
> > not
> > a good language in which to construct safely quoted remote content for 
> > injection
> > into a suitable HTML skeleton.
> 
> Injection from where? the script is only accessible to the root user on 
> the mail server and only checks /var/log/maillog (or the log specified 
> at the top of the script). There's no remote content involved.

Injection from the SMTP port.

Wietse

Re: Customize log messages?

On 12/3/16 9:53 AM, Bastian Blank wrote:

On Sat, Dec 03, 2016 at 09:44:03AM -0700, @lbutlr wrote:

Injection from where? the script is only accessible to the root user
on the mail server and only checks /var/log/maillog (or the log
specified at the top of the script). There's no remote content
involved.


The contents of the log are from remote sources.


The contents of the logs are from postfix.

Re: Customize log messages?

[Bastian Blank]

On Sat, Dec 03, 2016 at 09:44:03AM -0700, @lbutlr wrote:
> Injection from where? the script is only accessible to the root user
> on the mail server and only checks /var/log/maillog (or the log
> specified at the top of the script). There's no remote content
> involved.

The contents of the log are from remote sources.

Bastian

-- 
I have never understood the female capacity to avoid a direct answer to
any question.
-- Spock, "This Side of Paradise", stardate 3417.3

Re: Customize log messages?

On 12/3/16 1:48 AM, Viktor Dukhovni wrote:

On Dec 2, 2016, at 1:30 AM, @lbutlr  wrote:

I have a bash script that does it, and when a user wants this, I simply set up 
a crontab for them. Usually after a week or so they want it turned off. The 
script sends them a lightly styled HTML table in the email.

The heart of the script is:

if [ "$REJECT" = 1 ]; then
  echo 'IP addressClaimed address'
bzgrep "$MATCHPAT" $LOGF | grep -i reject | egrep 'from=<[^>]+>' | grep -v 
"Protocol error" | \
 grep -v "$EXCLUDE" | sort -u | sed 's/from=,[]:' | grep -v 
rejected | \
 awk '{print "REJECTED"$16""$20""}'
  fi

Careful with that.  To easy to create a script injection vector.  Bash is not
a good language in which to construct safely quoted remote content for injection
into a suitable HTML skeleton.


Injection from where? the script is only accessible to the root user on 
the mail server and only checks /var/log/maillog (or the log specified 
at the top of the script). There's no remote content involved.

Re: Customize log messages?

[Wietse Venema]

Viktor Dukhovni:
> 
> > On Dec 2, 2016, at 1:30 AM, @lbutlr  wrote:
> > 
> > I have a bash script that does it, and when a user wants this, I simply set 
> > up a crontab for them. Usually after a week or so they want it turned off. 
> > The script sends them a lightly styled HTML table in the email.
> > 
> > The heart of the script is:
> > 
> > if [ "$REJECT" = 1 ]; then
> >  echo 'IP addressClaimed address'
> >bzgrep "$MATCHPAT" $LOGF | grep -i reject | egrep 'from=<[^>]+>' | grep 
> > -v "Protocol error" | \
> > grep -v "$EXCLUDE" | sort -u | sed 's/from=,[]:' | grep 
> > -v rejected | \
> > awk '{print "REJECTED > class=\"right\">"$16""$20""}'
> >  fi
> 
> Careful with that.  To easy to create a script injection vector.
> Bash is not a good language in which to construct safely quoted
> remote content for injection into a suitable HTML skeleton.

In the AWK script, ``gsub(/[<>"]/, "_"); print...'' might do the job.

Wietse

Re: Customize log messages?

[Viktor Dukhovni]

> On Dec 2, 2016, at 1:30 AM, @lbutlr  wrote:
> 
> I have a bash script that does it, and when a user wants this, I simply set 
> up a crontab for them. Usually after a week or so they want it turned off. 
> The script sends them a lightly styled HTML table in the email.
> 
> The heart of the script is:
> 
> if [ "$REJECT" = 1 ]; then
>  echo 'IP addressClaimed address'
>bzgrep "$MATCHPAT" $LOGF | grep -i reject | egrep 'from=<[^>]+>' | grep -v 
> "Protocol error" | \
> grep -v "$EXCLUDE" | sort -u | sed 's/from=,[]:' | grep -v 
> rejected | \
> awk '{print "REJECTED class=\"right\">"$16""$20""}'
>  fi

Careful with that.  To easy to create a script injection vector.  Bash is not
a good language in which to construct safely quoted remote content for injection
into a suitable HTML skeleton.

-- 
Viktor.

Re: Customize log messages?

[Michael Munger]

This is a great idea. This is a spam filter that is integrated into a
CRM system, so I needed to parse and dump the information so it could be
sucked up later.

Here's what I ultimately created. It still needs some work (mainly
because it re-reads the whole file every time, and I should use
timestamps and a half-interval search algorithm to find the
last-processed time stamp. I am relying on log rotate to make it
not-too-terribly-big).

https://github.com/mjmunger/postfix-log-parser


Michael Munger, dCAP, MCPS, MCNPS, MBSS
High Powered Help, Inc.
Microsoft Certified Professional
Microsoft Certified Small Business Specialist
Digium Certified Asterisk Professional
mich...@highpoweredhelp.com
On 12/02/2016 01:30 AM, @ wrote:
> On 11/30/16 2:35 PM, Michael Munger wrote:
>
>> I am writing a log parser so that when users complain "so and so sent me
>> an email and I didn't get it" I can query the logs and find this with
>> ease. Ultimately, I want ot make this self service through a web page.
> I went a different way. Users can chose to receive a "DMR" (Daily Mail
> Report) and that report can contain either all the rejected email
> addresses that were not accepted for their account (or domain), all
> the accepted emails they got, or both.
>
> I have a bash script that does it, and when a user wants this, I
> simply set up a crontab for them. Usually after a week or so they want
> it turned off. The script sends them a lightly styled HTML table in
> the email.
>
> The heart of the script is:
>
>  if [ "$REJECT" = 1 ]; then
>   echo 'IP addressClaimed
> address'
> bzgrep "$MATCHPAT" $LOGF | grep -i reject | egrep 'from=<[^>]+>' |
> grep -v "Protocol error" | \
>  grep -v "$EXCLUDE" | sort -u | sed 's/from=,[]:' |
> grep -v rejected | \
>  awk '{print "REJECTED class=\"right\">"$16""$20""}'
>   fi
>
>   if [ "$ACCEPT" = 1 ]; then
> echo 'Accepted ID style="width:6em;">TimeFrom'
>  bzgrep -E 'DATA|\"from=\"' $LOGF | grep -v "<>"| \
> awk '{print $6"\t"$3"\t"$17"\t"$16}' | grep -v ESMTP | \
> grep -v "to= "to=<.*$MATCHPAT" | \
> grep -v "$EXCLUDE" | sed 's/from//g' | sed 's/://' | tr -d
> '=><' |
> awk '{print ""$1" class=\"right\">"$2""$4""}'
>fi
>
> For this to work
>
> smtpd_log_access_permit_actions = static:all
>
> must be set in main.cf. This makes your logs chattier, but provides me
> with the line in the logs that I need to get this working.
>
> One user, in particular, was calling several times a week looking for
> an email and now never calls.
>
>
>

Re: Customize log messages?

On 11/30/16 2:35 PM, Michael Munger wrote:


I am writing a log parser so that when users complain "so and so sent me
an email and I didn't get it" I can query the logs and find this with
ease. Ultimately, I want ot make this self service through a web page.
I went a different way. Users can chose to receive a "DMR" (Daily Mail 
Report) and that report can contain either all the rejected email 
addresses that were not accepted for their account (or domain), all the 
accepted emails they got, or both.


I have a bash script that does it, and when a user wants this, I simply 
set up a crontab for them. Usually after a week or so they want it 
turned off. The script sends them a lightly styled HTML table in the email.


The heart of the script is:

 if [ "$REJECT" = 1 ]; then
  echo 'IP addressClaimed 
address'
bzgrep "$MATCHPAT" $LOGF | grep -i reject | egrep 'from=<[^>]+>' | 
grep -v "Protocol error" | \
 grep -v "$EXCLUDE" | sort -u | sed 's/from=,[]:' | 
grep -v rejected | \
 awk '{print "REJECTEDclass=\"right\">"$16""$20""}'

  fi

  if [ "$ACCEPT" = 1 ]; then
echo 'Accepted IDstyle="width:6em;">TimeFrom'

 bzgrep -E 'DATA|\"from=\"' $LOGF | grep -v "<>"| \
awk '{print $6"\t"$3"\t"$17"\t"$16}' | grep -v ESMTP | \
grep -v "to=<' |
awk '{print ""$1"class=\"right\">"$2""$4""}'

   fi

For this to work

smtpd_log_access_permit_actions = static:all

must be set in main.cf. This makes your logs chattier, but provides me 
with the line in the logs that I need to get this working.


One user, in particular, was calling several times a week looking for an 
email and now never calls.

Re: Customize log messages?

[Michael Munger]

On 12/01/2016 09:37 AM, Wietse Venema wrote:

And I have made a note to log the sender when rejecting the (MAIL
FROM) SIZE parameter.

Wow. Wasn't expecting that! Thank you, sir.

Re: Customize log messages?

[Wietse Venema]

Michael Munger:
> Bill:
> 
> Thank you for both items. I shall pour over them.

And I have made a note to log the sender when rejecting the (MAIL
FROM) SIZE parameter.

Wietse

Re: Customize log messages?

[Michael Munger]

Bill:

Thank you for both items. I shall pour over them.

On 11/30/2016 11:49 PM, Bill Cole wrote:

On 30 Nov 2016, at 20:20, Michael Munger wrote:


First, there can be no TO address before the client sends MAIL FROM.
Second, the size check is done before checking the sender address,
presumably because it is more efficient that way. But I guess some
code could be swapped around.


My mistake. I thought:

552 5.3.4 Message size exceeds fixed limit;

I did not know that the message size was declared prior to the MAIL FROM
and RCPT TO commands.

I figured it would come either after the DATA command or after the
message was received so that it could calculate sizes. I'll rethink my
strategy here.


See https://tools.ietf.org/html/rfc1870

Maximum message size is advertised in the EHLO response and senders 
can announce message size as an extra argument to MAIL. Many SMTP 
clients honor the advertised maximum and simply QUIT the session if it 
is too small, others proceed to MAIL with the SIZE argument and get 
rejected explicitly there (that 552 reply.)



I have been scouring the docs to determine what checks are perform when
and in what order. I assume they follow the SMTP prtocol (HELO checks
first, MAIL FROM checks next, RCPT TO checks next, and so on...).

I found the architecture readme, which is excellent. Is there a list of
which programs do which checks? I could assemble a list from there.
Unless you (or someone) already had such a list?


You can get most of this in the postconf(5) man page, and whatever 
isn't detailed there will be in the man pages for the various Postfix 
components. The Postfix README files are great for a high-level 
instructional view of how it all works, but for the full technical 
details you really need to look at man pages.

Re: Customize log messages?

[Bill Cole]

On 30 Nov 2016, at 20:20, Michael Munger wrote:


First, there can be no TO address before the client sends MAIL FROM.
Second, the size check is done before checking the sender address,
presumably because it is more efficient that way. But I guess some
code could be swapped around.


My mistake. I thought:

552 5.3.4 Message size exceeds fixed limit;

I did not know that the message size was declared prior to the MAIL 
FROM

and RCPT TO commands.

I figured it would come either after the DATA command or after the
message was received so that it could calculate sizes. I'll rethink my
strategy here.


See https://tools.ietf.org/html/rfc1870

Maximum message size is advertised in the EHLO response and senders can 
announce message size as an extra argument to MAIL. Many SMTP clients 
honor the advertised maximum and simply QUIT the session if it is too 
small, others proceed to MAIL with the SIZE argument and get rejected 
explicitly there (that 552 reply.)


I have been scouring the docs to determine what checks are perform 
when

and in what order. I assume they follow the SMTP prtocol (HELO checks
first, MAIL FROM checks next, RCPT TO checks next, and so on...).

I found the architecture readme, which is excellent. Is there a list 
of

which programs do which checks? I could assemble a list from there.
Unless you (or someone) already had such a list?


You can get most of this in the postconf(5) man page, and whatever isn't 
detailed there will be in the man pages for the various Postfix 
components. The Postfix README files are great for a high-level 
instructional view of how it all works, but for the full technical 
details you really need to look at man pages.

Re: Customize log messages?

[Michael Munger]

> First, there can be no TO address before the client sends MAIL FROM.
> Second, the size check is done before checking the sender address,
> presumably because it is more efficient that way. But I guess some
> code could be swapped around.

My mistake. I thought:

552 5.3.4 Message size exceeds fixed limit;

I did not know that the message size was declared prior to the MAIL FROM
and RCPT TO commands.

I figured it would come either after the DATA command or after the
message was received so that it could calculate sizes. I'll rethink my
strategy here.

I have been scouring the docs to determine what checks are perform when
and in what order. I assume they follow the SMTP prtocol (HELO checks
first, MAIL FROM checks next, RCPT TO checks next, and so on...).

I found the architecture readme, which is excellent. Is there a list of
which programs do which checks? I could assemble a list from there.
Unless you (or someone) already had such a list?


Michael Munger, dCAP, MCPS, MCNPS, MBSS
High Powered Help, Inc.
Microsoft Certified Professional
Microsoft Certified Small Business Specialist
Digium Certified Asterisk Professional
mich...@highpoweredhelp.com

Re: Customize log messages?

[Wietse Venema]

Michael Munger:
> I am writing a log parser so that when users complain "so and so sent me
> an email and I didn't get it" I can query the logs and find this with
> ease. Ultimately, I want ot make this self service through a web page.
> 
> In a transaction like this:
> 
> 119970-Nov 29 13:56:12 mcdb2 postfix/smtpd[12371]: disconnect from
> unknown[118.201.69.1]
> 119971-Nov 29 13:56:33 mcdb2 postfix/smtpd[12587]: connect from
> mail-lf0-f51.google.com[209.85.215.51]
> 119972:Nov 29 13:56:34 mcdb2 postfix/smtpd[12587]: NOQUEUE: reject: MAIL
> from mail-lf0-f51.google.com[209.85.215.51]: 552 5.3.4 Message size
> exceeds fixed limit; proto=ESMTP helo=
> 119973-Nov 29 13:56:35 mcdb2 postfix/smtpd[12587]: disconnect from
> mail-lf0-f51.google.com[209.85.215.51]
> 
> The email is rejected without making reference to the 'from' and or 'to'
> addresss?. Is there a way I can configure this to include the from here?

First, there can be no TO address before the client sends MAIL FROM.
Second, the size check is done before checking the sender address,
presumably because it is more efficient that way. But I guess some
code could be swapped around.

However, Postfix logging is intended for system adminstrators, not
users. Asking to make it suitable for users is really outside the
scope of this project. Users can enter the wrong information in the
wrong place, and other users should not be exposed to that.

Wietse