Alex,
> Do we have an idea of a timeline for the next release and/or
> production release currently?
Not a fixed date yet, but we are getting there, the light at the
end of a tunnel is getting bright. No problems with stability,
it just would be nice to finish some remaining details,
and The Great Scoring computation still needs to be done.
> How about dependencies?
> What modules will need to be updated?
- minimum required version of ExtUtils::MakeMaker is 6.17
- now required modules: Time::HiRes, NetAddr::IP, Archive::Tar
- minimal version of Mail::DKIM is 0.31 (preferred 0.36_5 or later)
(did I miss anything?)
> Will perl-5.8 work okay?
- preferred versions of perl are 5.8.8, 5.8.9, and 5.10.0 or later
(of these three the 5.8.9 appears to be the most buggy)
- support for versions of perl 5.6.* is being gradually revoked
(may still work, but no promises and no support)
> How about for use with amavis? Will I need to upgrade that?
Some of the new features are only available with recent
versions of amavisd. Specifically:
- TIMING-SA reports in the log (at log level 2) are available
since amavisd-new-2.6.0;
- passing of truncated large mail to SpamAssassin without
breaking DKIM signature results is available since 2.6.3
- with version amavisd-new-2.6.2 and older
the following change is necessary:
- my($data_representation) = 'GLOB'; # pass data to SA as ARRAY or a GLOB
+ my($data_representation) = 'ARRAY';
> A list of the top five best new features would also be great! *salivates*
Here is my personal choice (amavis and DKIM -centric, sorry):
- improved error detection and reporting
- support for ADSP (with manual overrides) in a DKIM plugin
(deals with phishing on domains like eBay, PayPal, amazon.com, ...)
- ability to check a truncated long message without breaking DKIM signatures
- timing breakdown reports
- much improved IPv6 support
- avoid Perl 5.8.9 and 5.10.0 compiler crashing when compiling many rules
on FreeBSD (smaller threads stack)
(ok, that's six)
> I'm trying to anticipate what I can do ahead of time to get it into
> place as soon as possible.
With amavisd-new 2.6.3 it should be fine.
Here is my attempt at compiling release notes for SpamAssassin 3.3
from a SVN change log. I left out changes to infrastructure, trivial changes
and details on rule changes. Please say so if something important
was left out.
SpamAssassin 3.3.0 RELEASE NOTES --- DRAFT / UNOFFICIAL
COMPATIBILITY WITH 3.2.5
- rules are no longer distributed with the package, but installed by
sa-update - either automatically fetched from the network (preferably),
or from a tar archive, which is available for downloading separately
- minimum required version of ExtUtils::MakeMaker is 6.17
- now required modules: Time::HiRes, NetAddr::IP, Archive::Tar
- minimal version of Mail::DKIM is 0.31 (preferred 0.36_5 or later)
- no longer used modules: Mail::DomainKeys, Mail::SPF::Query
- support for versions of perl 5.6.* is being gradually revoked
(may still work, but no promises and no support)
- preferred versions of perl are 5.8.8, 5.8.9, and 5.10.0 or later
(of these three the 5.8.9 appears to be the most buggy)
BUILDING AND PACKAGING
- rules are no longer distributed with the package, but installed by
sa-update
- simplify Makefile.PL and fix a bug in DESTDIR support by increasing
the minimum ExtUtils::MakeMaker version required to 6.17
- include check_whitelist and check_spamd in distribution;
now called 'sa-awl' and 'sa-check_spamd'
WORKAROUNDS TO PERL BUGS AND LIMITATIONS
- let the Check.pm plugin produce smaller chunks of source code from rules
(60 kB) to avoid Perl compiler crashing on exceeding stack size
- localize $1, $2, etc at several places, avoiding taint issue from propagating
- avoid Perl I/O bug by replacing line-by-line reading with read() where
suitable, or play down the EBADF status in other places and only report it
as dbg instead of a die - while also providing a little speedup (10..25%)
on reading a message
- new sub Message::split_into_array_of_short_lines to nicely split
a text into array of paragraph chunks of sizes between 1 kB and 2 kB,
gives less opportunity to runaway regular expressions in rules;
fixes bugs: 5717, 5644, 5795, 5486, 5801, 5041
ERROR HANDLING, ROBUSTNESS
- improved error detection and reporting: test status of all system calls
and I/O operations (or explicitly document where not), and report
unexpected failures;
- eval calls now check for eval result instead of testing the $@, which
is not always reliable;
- localized $@ and $! in DESTROY methods to prevent potential calls to eval
and calls to system routines in code executed from a DESTROY method
from clobbering global variables $@ and $!;
- Util::helper_app_pipe_open_unix: contain a failing exec with an eval
to prevent additional cases of process cloning. The exec could fail
this way when given tainted arguments;
- Util::helper_app_pipe_open_unix: flush stdout and stderr before forking,
otherwise an error reported by exec (such as 'insecure dependency')
was lost in a buffer;
- eval-protect an open($fh,'-|') to capture implied fork failures
due to lack of system resource;
- explicit untainting: combine "use re 'taint'" with untaint_var(),
avoiding implicit perl untainting and workarounds to prevent it;
- add 'use strict' where missing;
- avoid a bunch of warnings on "Use of uninitialized value"
- clearly report reasons for helper application process failures
MEMORY FOOTPRINT
- as a side-effect of compiling rules in smaller chunks (to avoid compiler
crashes) virtual memory footprint of SpamAssassin is reduced;
- save some memory by not importing Pod::Usage unless it is needed;
- save 350k+ of memory in sa-compile by replacing DynaLoader with XSLoader;
- remove unneeded index from MySQL bayes_token table;
IPv6 SUPPORT
- IPv6 support for trusted_networks, internal_networks, whitelist_from_rcvd,
msa_networks, and other stuff that uses NetSet and the Received header
parser, using NetAddr::IP;
- allow usage of a remote dccifd host through an INET or INET6 socket;
- README.awl: increase suggested awl.ip field width to 45 to be able to
hold IPv6 addresses (optionally);
- IP_PRIVATE now includes the ipv6 variants of private address space,
as well as the ipv6-mapped ipv4 addresses.
- NetSet now understands that ::ffff:192.168.1.2 and 192.168.1.2 are
the same address;
- IPv6 addresses are now recognised in Received headers;
- when reading Received header fields, the "IPv6:" prefix is stripped from
IPv6 addresses, and "::ffff:" is removed from IPv6-mapped IPv4 addresses
(so strings can match them as simply IPv4 addresses);
- ::1 is always included in the trusted_networks/internal_networks set
similar to 127.0.0.0/8;
SPAMC
- add -n / --connect-timeout switch to spamc, allowing separate
connection timeout from communication timeout;
- add --filter-retries and --filter-retry-sleep
- spamc would not time out connections to a hung spamd
- spamc client library leaks the zlib compression buffer
if compression is used
- spamc long option '--dest' was broken
SPAMD
- when spamd is started with the daemonize option do not exit the parent
until the child signals that it has logged the pid, to allow a wrapper
script to simply continue immediately after starting spamd;
- additional tempfile cleanup in kill_handler;
- add SPAMD_LOCALHOST option to make-test to allow specifying non-127.0.0.1
IP address for use in FreeBSD jail;
API
- adding one optional argument to Mail::SpamAssassin::parse allows caller
to pass additional out-of-band information to SpamAssassin (such as DKIM
verification results, information about a SMTP session, or dynamic rule
hits); this information is made available to plugins and the rest of the
code through a 'suppl_attrib' hash;
- Plugin::Check - pick up 'rule_hits' from caller via the new mechanism
and call got_hit() on them;
- simplified adding dynamic score hits and dynamic rules by plugins
(such as AWL, CRM114, FuzzyOcr, Check) by letting got_hit() accept
options tflags and description, and letting it store a supplied
dynamic score for proper reporting;
- let the timing breakdown information be accessible to a caller through
the existing get_tag mechanism (tag TIMING);
- let the generated header fields ('add_header' configuration options)
be accessible to a caller through the existing get_tag mechanism
(tag ADDEDHEADER and friends);
RULES
- rules are no longer distributed with the package;
- dropped redundant rules or rules causing too many false positives;
- added or updated rules (incomplete list in no particular order:
vbounce, money, image spam, fill_this_form, FreeMail, European Parliament,
HTML attachments, uri_obfu*, urinsrhsbl, urinsrhssub, urifullnsrhsbl,
URI_OBFU_X9_WS, rDNS=localhost, INVALID_DATE_TZ_ABSURD, KHOP_SC,
RCVD_IN_PSBL, FRT_VALIUM*, BOUNCE_MESSAGE, VBOUNCE_MESSAGE,
__BOUNCE_UNDELIVERABLE, HELO_STATIC_HOST, FILL_THIS_FORM_FRAUD_PHISH,
CHALLENGE_RESPONSE, DKIM_VALID, DKIM_VALID_AU, DKIM_ADSP_*,
NML_ADSP_CUSTOM_{LOW,MED,HIGH}, __VIA_ML, MIME_BASE64_TEXT,
FORGED_MUA_THEBAT_BOUN, FORGED_MUA_THEBAT_CS, UNRESOLVED_TEMPLATE,
__THEBAT_MUA, __ANY_OUTLOOK_MUA, RP_MATCHES_RCVD, one-word X-Mailer, ...
- rule for plain text attachments with octet-stream MIME type;
- avoid false positives on ISO-2022-JP messages in several rules;
- updated various default whitelists, uridnsbl_skip_domain, adsp_override, ...
PLUGINS
- new plugins: FreeMail, PhishTag, Reuse
- now enabled by default: DKIM
- now disabled by default: AWL
- retired plugin DomainKeys
AWL PLUGIN
- plugin AWL is now disabled by default;
- README.awl: increase suggested awl.ip field width to 45 to support IPv6
addresses or DKIM signer domains;
- AutoWhitelist.pm: allow storing an IPv6 address (previously causing
SQL server errors: value too long)
- let AWL keep separate records for DKIM-signed and unsigned mail
(when auto_whitelist_distinguish_signed configuration option is true,
and SQL field awl.ip field size is increased or made dynamic);
- gracefully handle NaN from corrupted database or a broken emulator;
DCC PLUGIN
- added support for DCC reputations, added setting dcc_rep_percent,
new test check_dcc_reputation_range(), new tag DCCREP
(DCC servers supply reputation data only to licensed clients);
- allow usage of a remote dccifd host through an INET or INET6 socket;
DKIM PLUGIN
- plugin is now enabled by default;
- absolute minimal version of Mail::DKIM is 0.31;
support for ADSP requires Mail::DKIM 0.34;
a DNS test (and rule) for NXDOMAIN is operational since Mail::DKIM 0.36_5
- supports multiple signatures (useful for whitelisting);
- distinguishes author domain signatures from third party signatures
(useful for whitelisting);
- provides a tag DKIMIDENTITY (in addition to DKIMDOMAIN);
- DKIM now supports Author Domain Signing Practices - ADSP (RFC 5617);
- use the Mail::DKIM::AuthorDomainPolicy instead of Mail::DKIM::DkimPolicy,
when available (since Mail::DKIM 0.34);
- implements an 'adsp_override' configuration directive and adds
an eval:check_dkim_adsp check, which is used by new DKIM_ADSP_* rules;
- rules contain an initial set of 'adsp_override' directives, listing
some of the more popular target domains for phishing (applicable only to
domains which sign all their direct mail with a DKIM or DK signature);
- this plugin can now re-use Mail::DKIM verification results if made
available by a caller, which saves resources and makes it possible
for SpamAssassin to work on a truncated large mail without breaking
DKIM signatures;
BUG FIXES
- fixed Rule2XSBody segfaults;
- do not treat user data as perl booleans (a string "0" is a false);
- avoid data from the wild be interpreted as perl regular expressions;
- ArchiveIterator: prevent _scan_directory from passing directories
to _scan_file (on NFS it would fail with EISDIR on read(2);
- fixed vpopmail support;
- the 'exists:' evaluator in HEADER rules now works as documented
and tests for existence of a header field, instead of testing for
a header field body being nonempty; internally, the pms->get can
also now distinguish between empty and nonexistent header fields
- fixes to header fields parsing in several places: header field names
are case-insensitive, whitespace is not required after a colon,
obsolete rfc822 syntax allowed whitespace before a colon;
VBounce: match "Received:" only at the beginning of a line;
- Exporter never really worked in SA, was not enclosed in BEGIN{}
OTHER
- more expensive code sections are now instrumented with timing measurements;
timing report is logged as a debug message by the end of processing,
and made available to a caller and to 'add_header' directives through
a TIMING tag;
- total rewrite of URI detection in plain text body;
- pseudoheader "ALL:raw" returns a pristine header section,
and pseudoheader "ALL" returns a cleaned header section
- many updates to the list of top level domains;
- add 'util_rb_3tld', allowing 3-level TLDs to be listed in URIBLs and
allowing new 3TLDs to be added from rule updates;
- avoid trusted_networks bog down due to O(n^2) loop with millions of entries;
- preserve order of declared 'add_header' haeder fields;
- allow debug areas to be excluded from debugging,
e.g.: -D all,norules,noconfig,nodcc
- fixes to Plugin/VBounce.pm, updated VBounce ruleset;
- new module Mail::SpamAssassin::BayesStore::BDB, but is not yet ready for use
- numerous additional and updated self-tests;
- updated documentation, fixed numerous typos and mistakes
in documentation text and in log messages;
- extensive improvements to development process:
automated testing through Hudson, improvements to mass-check and rules
Mark
