Re: connection notes
https://github.com/smtpd/qpsmtpd/pull/18 Matt On Jun 1, 2012, at 8:34 PM, Ask Bjørn Hansen wrote: > On Jun 1, 2012, at 20:25, Matt Simerson wrote: > >> Which is preferred within a plugin? >> >> $self->qp->connection->notes >> >> or >> >> $self->connection->notes > > Assuming the second method is just a proxy for the first, then use the > shorter version. > > > Ask
Re: connection notes
On Jun 1, 2012, at 20:25, Matt Simerson wrote: > Which is preferred within a plugin? > > $self->qp->connection->notes > > or > > $self->connection->notes Assuming the second method is just a proxy for the first, then use the shorter version. Ask
Re: Connection notes and TLS
On 26-May-08, at 5:24 PM, John Peacock wrote: Those are *connection* attributes, not *transaction* attributes. If a plugin is storing them in the transaction object, that is a mistake. We can and probably should maintain the connection notes from before the TLS reset, but I maintain that we should not keep the transaction notes. OK, I'm fine with that.
Re: Connection notes and TLS
Matt Sergeant wrote: OK, I'm fine with that. Hopefully to put this thread to rest, I committed my one line change to preserve the connection notes. If anyone's favorite core plugin stashes connection-related stuff (like dnsbl) in the transaction notes, let's start individual threads (with patches please ;-) to deal with those individually... John
Re: Connection notes and TLS
Chris Lewis wrote: SMTP state and qpsmtpd/filtering state are not the same thing. The RFCs talk about destroying SMTP state. It already does that without having to destroy any notes() AFAICT. But it shouldn't blithely throwing away qpsmtpd/filtering state - it's none of the RFC's business. The current behavior throws away the entire transaction object and resets the connection object to include only the fields at the top of Connection.pm: # All of these parameters depend only on the physical connection, # i.e. not on anything sent from the remote machine. Hence, they # are an appropriate set to use for either start() or clone(). Do # not add parameters here unless they also meet that criteria. my @parameters = qw( remote_host remote_ip remote_info remote_port local_ip local_port relay_client ); All we are talking about at this point, really, is whether we should preserve any connection notes as well [during clone], which I suggest is perfectly fine from both a pragmatic and RFC standpoint. It makes sense, for example, for a source-IP reputation checker to launch its queries at connect time, and query the results at a later phase (eg: rcpt/data) to make the decisions then, independent of whether TLS was established or not. Those are *connection* attributes, not *transaction* attributes. If a plugin is storing them in the transaction object, that is a mistake. We can and probably should maintain the connection notes from before the TLS reset, but I maintain that we should not keep the transaction notes. John
Re: Connection notes and TLS
John Peacock wrote: On a more pragmatic note: do we have any evidence whatsoever that spammers are using TLS at all? This may be a completely theoretical exercise, since AFAIK, all standard MTA's will switch to TLS as soon as the received the EHLO prompt, in which case there is no transaction information at all (that is if they are configured to use TLS at all). Yes. Lots of spammers use it, mostly perhaps without knowing it. I don't know of any BOT-contained TLS, but my traps are _full_ of TLS connections, and it's not just blowback. Anybody spamming thru, for example, an auth-hacked TLS-enabled MTA will be doing TLS to whomever supports it. Or a fixed-position spammer using standard TLS-enabled MTAs. Etc.
Re: Connection notes and TLS
Matt Sergeant wrote: On 22-May-08, at 10:45 PM, John Peacock wrote: STARTTLS is not required to happen immediately after EHLO (not HELO, which doesn't support ESMTP extensions). And yes, you must completely discard every portion of the SMTP state that has occurred up to that point (just like with RSET). SMTP state and qpsmtpd/filtering state are not the same thing. The RFCs talk about destroying SMTP state. It already does that without having to destroy any notes() AFAICT. But it shouldn't blithely throwing away qpsmtpd/filtering state - it's none of the RFC's business. The RFC is extremely plain on this point: after STARTTLS has been sent and negotiated, the MTA must behave as if a completely new transaction has started (as indeed, it has). The transaction *must* be reset and all information contained therein must be thrown away. In practice, there isn't anything there to begin with, since all of the well-formed MTA's always sent STARTTLS as soon as practical (i.e. as soon as they see the initial EHLO banner), if they are going to send it at all. I don't think we should care so much about the RFCs. If there are bits in connection notes that might help determining if this is spam (or some other thing we're trying to detect) before STARTTLS we need to allow qpsmtpd to keep that information. Indeed. It makes sense, for example, for a source-IP reputation checker to launch its queries at connect time, and query the results at a later phase (eg: rcpt/data) to make the decisions then, independent of whether TLS was established or not. My hack to the async DNSBL code does that.
Re: Connection notes and TLS
Matt Sergeant wrote: I don't think we should care so much about the RFCs. If there are bits in connection notes that might help determining if this is spam (or some other thing we're trying to detect) before STARTTLS we need to allow qpsmtpd to keep that information. I'm talking about keeping the *connection* notes, but not the *transaction* notes (which is what my one-line change would do). On a more pragmatic note: do we have any evidence whatsoever that spammers are using TLS at all? This may be a completely theoretical exercise, since AFAIK, all standard MTA's will switch to TLS as soon as the received the EHLO prompt, in which case there is no transaction information at all (that is if they are configured to use TLS at all). John
Re: Connection notes and TLS
On 22-May-08, at 10:45 PM, John Peacock wrote: STARTTLS is not required to happen immediately after EHLO (not HELO, which doesn't support ESMTP extensions). And yes, you must completely discard every portion of the SMTP state that has occurred up to that point (just like with RSET). The RFC is extremely plain on this point: after STARTTLS has been sent and negotiated, the MTA must behave as if a completely new transaction has started (as indeed, it has). The transaction *must* be reset and all information contained therein must be thrown away. In practice, there isn't anything there to begin with, since all of the well-formed MTA's always sent STARTTLS as soon as practical (i.e. as soon as they see the initial EHLO banner), if they are going to send it at all. I don't think we should care so much about the RFCs. If there are bits in connection notes that might help determining if this is spam (or some other thing we're trying to detect) before STARTTLS we need to allow qpsmtpd to keep that information. Matt.
Re: Connection notes and TLS
On Thu, 22 May 2008, Chris Babcock wrote: Chris Lewis wrote: ... Killing transaction/connection notes() is a bit of a kludge, because there's lots of other plugins who may be relying on the data being persistent. Without killing connection notes, I don't see any sane way to determine what notes were generated before vs after TLS was enabled. My point is that notes relating to the transaction should be in connection->transaction, and those relating to the connection shouldn't be. On TLS negotiation, the transaction notes must be cleared, and the connection notes shouldn't be. Unfortunately, for some plugins that determination may be critical to the behavior of the plugin. It seems to me retaining both the pre-TLS and post-TLS states would be the closest to ideal, but I'm sure there's a way to do that sanely without breaking how the current plugins are working. If any plugins are currently doing the wrong thing, they should be identified and fixed. Do we know that there are any currently "broken" plugins?
Re: Connection notes and TLS
On 22 May 2008 10:34:46 -0700 "Chris Babcock" <[EMAIL PROTECTED]> wrote: > Chris Lewis wrote: > Without killing connection notes, I don't see any sane way to determine > what notes were generated before vs after TLS was enabled. > Unfortunately, for some plugins that determination may be critical to > the behavior of the plugin. It seems to me retaining both the pre-TLS > and post-TLS states would be the closest to ideal, but I'm sure there's > a way to do that sanely without breaking how the current plugins are > working. As said before: if there's a real need for a "post-starttls" hook, we can add it. But I think it should be enough to run a helo/ehlo hook and see if the connection->note tls_enabled is true and probably check if this is the second run of this hook with tls_enabled. Hanno
Re: Connection notes and TLS
On Thu, 22 May 2008 22:45:20 -0400 John Peacock <[EMAIL PROTECTED]> wrote: > The argument can be made that only the *transaction* should be reset, and the > connection notes can be kept around. I specifically chose to tie > Connection::start and Connection::clone together with the minimal list of > properties required for a connection (see Connection.pm lines 8-15) when I > implemented clone(). > > That being said, keeping the connection notes is a one line change to > Connection::clone(): I'd take this in. Maybe some plugins like cound_unrecognized_commands need to reset themself, but that should be a choice of the admin IMO. Hanno
Re: Connection notes and TLS
Chris Lewis wrote:
John Peacock says:
It's a required part of the RFC, see RFC-2487 Section 5.2:
Upon completion of the TLS handshake, ... [t]he server MUST discard any
knowledge obtained from the client...
Do you even have to purge envelope addresses per-se? It doesn't say it,
but doesn't STARTTLS have to occur immediately after HELO? (or the
reset equivalent). It's all going to get filled in again before the
client can send anything anyway.
STARTTLS is not required to happen immediately after EHLO (not HELO, which
doesn't support ESMTP extensions). And yes, you must completely discard every
portion of the SMTP state that has occurred up to that point (just like with
RSET).
The RFC is extremely plain on this point: after STARTTLS has been sent and
negotiated, the MTA must behave as if a completely new transaction has started
(as indeed, it has). The transaction *must* be reset and all information
contained therein must be thrown away. In practice, there isn't anything there
to begin with, since all of the well-formed MTA's always sent STARTTLS as soon
as practical (i.e. as soon as they see the initial EHLO banner), if they are
going to send it at all.
The argument can be made that only the *transaction* should be reset, and the
connection notes can be kept around. I specifically chose to tie
Connection::start and Connection::clone together with the minimal list of
properties required for a connection (see Connection.pm lines 8-15) when I
implemented clone().
That being said, keeping the connection notes is a one line change to
Connection::clone():
Index: lib/Qpsmtpd/Connection.pm
===
--- lib/Qpsmtpd/Connection.pm (revision 916)
+++ lib/Qpsmtpd/Connection.pm (working copy)
@@ -43,6 +43,7 @@
foreach my $f ( @parameters ) {
$new->$f($self->$f()) if $self->$f();
}
+ $new->{_notes} = $self->{_notes} if defined $self->{_notes};
# reset the old connection object like it's done at the end of a connection
# to prevent leaks (like prefork/tls problem with the old SSL file handle
# still around)
Killing transaction/connection notes() is a bit of a kludge, because
there's lots of other plugins who may be relying on the data being
persistent.
Then those plugins are not well designed. The old transaction is over when
STARTTLS has been completed.
John
Re: Connection notes and TLS
Chris Lewis wrote: Charlie Brady wrote: On Wed, 21 May 2008, Hanno Hecker wrote: On Wed, 21 May 2008 10:39:37 -0400 (EDT) Charlie Brady <[EMAIL PROTECTED]> wrote: On Wed, 21 May 2008, Hanno Hecker wrote: - if you use the tls plugin, all connection notes are lost after the TLS setup is complete Why? Isn't that a bug in the tls plugin? When TLS is negotiated, the transaction notes should be cleared (then TLS status added), but the connection notes should still be valid - it's the same connection still. Yes and no :) IMO at least some of the notes are still valid, but see this thread: http://www.nntp.perl.org/group/perl.qpsmtpd/2006/08/msg5371.html I don't see anything in the thread which identifies which notes might and might not be valid post-starttls. John Peacock says: I think that sounds appropriate, as TLS essentially starts a new connection. It's a required part of the RFC, see RFC-2487 Section 5.2: Upon completion of the TLS handshake, ... [t]he server MUST discard any knowledge obtained from the client... I don't dispute what the RFC says, but I don't see that all connection notes are "knowledge obtained from the client". Some is knowledge obtained from the TCP connection. I think we can even take "any knowledge obtained from the client must be discarded", with a grain of salt in our case. If the spammer says something pre-tls which makes it very obvious to us that he is a spammer, do we forget that information once tls is negotiated? Ditto for "reset" - do we allow a spammer to try again with a clear slate by sending "reset"? I don't think so. No, we just need to clear our SMTP transaction state machine, helo hostname and envelope address lists when TLS is negotiated. That should be sufficient for RFC protocol compliance. Do you even have to purge envelope addresses per-se? It doesn't say it, but doesn't STARTTLS have to occur immediately after HELO? (or the reset equivalent). It's all going to get filled in again before the client can send anything anyway. Killing transaction/connection notes() is a bit of a kludge, because there's lots of other plugins who may be relying on the data being persistent. Without killing connection notes, I don't see any sane way to determine what notes were generated before vs after TLS was enabled. Unfortunately, for some plugins that determination may be critical to the behavior of the plugin. It seems to me retaining both the pre-TLS and post-TLS states would be the closest to ideal, but I'm sure there's a way to do that sanely without breaking how the current plugins are working. -Chris
Re: Connection notes and TLS
Charlie Brady wrote: On Wed, 21 May 2008, Hanno Hecker wrote: On Wed, 21 May 2008 10:39:37 -0400 (EDT) Charlie Brady <[EMAIL PROTECTED]> wrote: On Wed, 21 May 2008, Hanno Hecker wrote: - if you use the tls plugin, all connection notes are lost after the TLS setup is complete Why? Isn't that a bug in the tls plugin? When TLS is negotiated, the transaction notes should be cleared (then TLS status added), but the connection notes should still be valid - it's the same connection still. Yes and no :) IMO at least some of the notes are still valid, but see this thread: http://www.nntp.perl.org/group/perl.qpsmtpd/2006/08/msg5371.html I don't see anything in the thread which identifies which notes might and might not be valid post-starttls. John Peacock says: I think that sounds appropriate, as TLS essentially starts a new connection. It's a required part of the RFC, see RFC-2487 Section 5.2: Upon completion of the TLS handshake, ... [t]he server MUST discard any knowledge obtained from the client... I don't dispute what the RFC says, but I don't see that all connection notes are "knowledge obtained from the client". Some is knowledge obtained from the TCP connection. I think we can even take "any knowledge obtained from the client must be discarded", with a grain of salt in our case. If the spammer says something pre-tls which makes it very obvious to us that he is a spammer, do we forget that information once tls is negotiated? Ditto for "reset" - do we allow a spammer to try again with a clear slate by sending "reset"? I don't think so. No, we just need to clear our SMTP transaction state machine, helo hostname and envelope address lists when TLS is negotiated. That should be sufficient for RFC protocol compliance. Do you even have to purge envelope addresses per-se? It doesn't say it, but doesn't STARTTLS have to occur immediately after HELO? (or the reset equivalent). It's all going to get filled in again before the client can send anything anyway. Killing transaction/connection notes() is a bit of a kludge, because there's lots of other plugins who may be relying on the data being persistent.
Re: Connection notes and TLS (was: Re: Creating global DB connection)
On Wed, 21 May 2008, Hanno Hecker wrote: On Wed, 21 May 2008 10:39:37 -0400 (EDT) Charlie Brady <[EMAIL PROTECTED]> wrote: On Wed, 21 May 2008, Hanno Hecker wrote: - if you use the tls plugin, all connection notes are lost after the TLS setup is complete Why? Isn't that a bug in the tls plugin? When TLS is negotiated, the transaction notes should be cleared (then TLS status added), but the connection notes should still be valid - it's the same connection still. Yes and no :) IMO at least some of the notes are still valid, but see this thread: http://www.nntp.perl.org/group/perl.qpsmtpd/2006/08/msg5371.html I don't see anything in the thread which identifies which notes might and might not be valid post-starttls. John Peacock says: I think that sounds appropriate, as TLS essentially starts a new connection. It's a required part of the RFC, see RFC-2487 Section 5.2: Upon completion of the TLS handshake, ... [t]he server MUST discard any knowledge obtained from the client... I don't dispute what the RFC says, but I don't see that all connection notes are "knowledge obtained from the client". Some is knowledge obtained from the TCP connection. I think we can even take "any knowledge obtained from the client must be discarded", with a grain of salt in our case. If the spammer says something pre-tls which makes it very obvious to us that he is a spammer, do we forget that information once tls is negotiated? Ditto for "reset" - do we allow a spammer to try again with a clear slate by sending "reset"? I don't think so. No, we just need to clear our SMTP transaction state machine, helo hostname and envelope address lists when TLS is negotiated. That should be sufficient for RFC protocol compliance.
Re: Connection notes and TLS (was: Re: Creating global DB connection)
On Wed, 21 May 2008, Hanno Hecker wrote: On Wed, 21 May 2008 10:39:37 -0400 (EDT) Charlie Brady <[EMAIL PROTECTED]> wrote: On Wed, 21 May 2008, Hanno Hecker wrote: - if you use the tls plugin, all connection notes are lost after the TLS setup is complete Why? Isn't that a bug in the tls plugin? When TLS is negotiated, the transaction notes should be cleared (then TLS status added), but the connection notes should still be valid - it's the same connection still. Yes and no :) IMO at least some of the notes are still valid, but see this thread: http://www.nntp.perl.org/group/perl.qpsmtpd/2006/08/msg5371.html Keeping notes across the TLS setup is a simple patch, but I won't commit it until all agree that we want to keep all notes What connection notes might exist at starttls which shouldn't be kept? I'd say that if there are any, they should be transaction notes and not connection notes (e.g. helo host is SMTP transaction metadata, not TCP connection metadata). (but: why do we need connection->clone() then? ;-)) I didn't say that we did need it :-) Another approach would be to add a "starttl-post" hook and let all plugins set the connection notes they need/want again in there (think of dnsbl, which will not block clients if the start with TLS, currently). I don't see any need for another approach. It seems straightforward to me that the connection exists from when it is created to when it is broken, and any connection notes should apply to the connection, and not to the data stream which passes over the connection. Plugins should be able to add and query notes about the connection, and the connection notes should not be discarded until the connection goes away. ... and to add some more new hooks: http://ankh-morp.org/~vetinari/tmp/new_command.diff is what I'm currently working on (The hardest part of this patch is still to do: the AUTH* commands): any plugin can add new hooks / commands. Adding a starttls-post hook in there would be easy. Hanno
Re: connection notes across TLS?
On 9-Aug-06, at 12:47 PM, Michael Holzt wrote: I consider TLS beeing implemented by unrecognized_command unclean anyway. Ironically I consider the way AUTH was implemented to be unclean. It's all a matter of perspective. Oh, i'm sure we can agree that both ways are not too clean. I think the new proposal for registering extensions/commands is a good solution to cleanup both. Yeah agreed - it does seem like a good plan (though I guess there are a finite number of useful extensions - but at work we're talking about ways in which we could use a custom version of SMTP as a sort of milter interface).
Re: connection notes across TLS?
> >I consider TLS beeing implemented by unrecognized_command unclean > >anyway. > Ironically I consider the way AUTH was implemented to be unclean. > It's all a matter of perspective. Oh, i'm sure we can agree that both ways are not too clean. I think the new proposal for registering extensions/commands is a good solution to cleanup both. Regards Michael -- It's an insane world, but i'm proud to be a part of it. -- Bill Hicks
Re: connection notes across TLS?
On 9-Aug-06, at 7:17 AM, Michael Holzt wrote: I consider TLS beeing implemented by unrecognized_command unclean anyway. Ironically I consider the way AUTH was implemented to be unclean. It's all a matter of perspective.
Re: connection notes across TLS?
Michael Holzt wrote: But if the switch to TLS has basically the same effect as opening a new connection, we should probably run the connect-hook again at that time. Or we could have an additional 'connect_tls' (or 'connect_restart'?) hook (which could also be used on port 465 for the deprecated ssmtp). It is and it isn't the same thing as opening a new connection. I think of connect() being at the TCP level and everything after EHLO/HELO to be at the SMTP level. I don't think that it would be appropriate to generically rerun the connect hook after TLSSTART (but see below). I consider TLS beeing implemented by unrecognized_command unclean anyway. We should probably have a special TLS-Hook and TLS is only offered by the core if a plugin is installed into the TLS-Hook (like AUTH is implemented). This would also allow to have more than one TLS-Plugin, for example a generic plugin first, starting a encrypted connection, and a second plugin next, checking e.g. a client certificate. Would be more flexible imho. At this point, I think TLS is stable enough that this would be my preferred route as well. I wonder if it is time to provide a register_extension() sub that could be used to generically support extended SMTP commands without having to hack the core every time one comes around. AUTH could get refitted to use this method then. In any case, once we have a tls() hook, we can then have a post_tls() hook which would be the appropriate place to redo any connect()-like hooks (like the OP wanted). John
Re: connection notes across TLS?
> No, the connect hook has already fired by the time that the TLS connection is > established. But if the switch to TLS has basically the same effect as opening a new connection, we should probably run the connect-hook again at that time. Or we could have an additional 'connect_tls' (or 'connect_restart'?) hook (which could also be used on port 465 for the deprecated ssmtp). > That happens later at STARTTLS (by hooking into unrecognized_command), > which is typically immediately after EHLO has completed. So maybe the unrecognized_command hook should have a possible return value of 'RESTART' which signals the core to restart the connection. I consider TLS beeing implemented by unrecognized_command unclean anyway. We should probably have a special TLS-Hook and TLS is only offered by the core if a plugin is installed into the TLS-Hook (like AUTH is implemented). This would also allow to have more than one TLS-Plugin, for example a generic plugin first, starting a encrypted connection, and a second plugin next, checking e.g. a client certificate. Would be more flexible imho. As a side note and preliminary information: I'm proud to tell you that there will be a printed article about qpsmtpd in one of the next issues of the german magazine 'iX' written by me :-) Regards Michael -- It's an insane world, but i'm proud to be a part of it. -- Bill Hicks
Re: connection notes across TLS?
On Tue, 8 Aug 2006, John Peacock wrote:
> [EMAIL PROTECTED] wrote:
> > Can anyone please suggest a simple way to pass a note from the
> > connection plugin to everything after TLS?
>
> You can't. What information are you trying to pass? Can you redo whatever
> analysis you performed before, say in a rcpt hook?
I set a note based on the results of an IP lookup on a local
database. I should be able to get the IP again with
$self->qp->connection->remote_ip and rerun the lookup - just need to
figure the best hook to put it into, and whether or not to create a
separate plugin for it.
Probably a separate plugin that checks for
$transaction->notes('tls_enabled'). Which hook...
(Thanks.)
Re: connection notes across TLS?
[EMAIL PROTECTED] wrote: > I think that sounds appropriate, as TLS essentially starts a new > connection. It's a required part of the RFC, see RFC-2487 Section 5.2: > Upon completion of the TLS handshake, ... [t]he server MUST discard any > knowledge obtained from the client... > Can anyone confirm if the connect hook will fire after TLS (it's > listed after TLS in the config/plugins file)? No, the connect hook has already fired by the time that the TLS connection is established. That happens later at STARTTLS (by hooking into unrecognized_command), which is typically immediately after EHLO has completed. The order of lines in config/plugins only determines what order *within* *each* *hook* the plugins will fire. > Can anyone please suggest a simple way to pass a note from the > connection plugin to everything after TLS? You can't. What information are you trying to pass? Can you redo whatever analysis you performed before, say in a rcpt hook? John
