1. Generate a bounce message to the envelope sender of the message, and
2. During the SMTP session, refuse to accept mail from the client, by returning a 500-series SMTP error code.
Option 1 is almost always a terrible idea, unless perhaps the sender has published an SPF record and the result of an SPF check at the server is pass (but definitely not anything else, including neutral or none). Anyway, option 1 is strictly inferior to option 2, as it will always require more resources at the mail server.
Option 2 is actually in my opinion a good idea, because in the case of false positives, it lets the sender know that the mail might not be read. In the case of actual spam, assuming you refuse the mail at the outermost mail relay at your organization, very often the mail is coming from a spambot that will never generate the bounce message. In other cases, the client may be an open mail relay, but such machines are very often blacklisted anyway, so I don't view causing them to send bounces as a terrible thing. Others may disagree on that point, but at any rate you are not risking getting your own server blacklisted--only the open mail relay is in danger of being blacklisted (which it should be anyway).
I've been quite interested in this issue of bouncing/refusing spam messages, and so built an SMTP server that makes it easy for individual users to refuse spam at the SMTP level. (See www.mailavenger.org for details.)
Recently, I have set up my account to reject with a 554 SMTP error code anything that spamassassin flags as spam, using the default threshold of 5.0, which is more aggressive than other people have been suggesting here. However, I also keep a copy of the messages I bounce, both so as to monitor how this is working out, and to build a corpus with which to train the Bayesian filter. At least anecdotally, this seems to be working well for me. When I spot check spams, I don't think I'm causing a lot of innocent people to get bounce messages.
Well I've reached the point with those that bounce spam using Option 1 I block with the following bounce:
554 Tell your admin to quit bouncing spam as that type of thing does nothing but DoS innocent domains.
Bouncing spam is IMHO just as big a problem as the spam itself.
It seems a certain appliance named after a fish likes to bounce spam by default which has caused our server to receive over 30,000 false bounces to legitimate email addresses in less an hour yesterday.
What I've do now is:
1) Spam over a certain score goes to /dev/null
2) Spam under a certain score, and over a certain score go to spamtrap incase someone's looking for something.
3) Low scoring spam gets delivered the user with **SPAM** in the subject which the users have a client side rules to move those to a spam folder.
Viruses 1) Identifiable viruses go to /dev/null 2) Executeable's get quarantined in a filetrap
