I removed IO::SigGuard dependency from my diff and I will work on deleting Email::MIME as well. Giovanni
On 10/17/21 16:58, Henrik K wrote: > > Atleast these seem completely unneeded module dependencies. > > IO::SigGuard (not even found in Ubuntu packages) > Email::MIME > > So the code should be refactored to use SA methods as necessary. > > > On Sat, Oct 16, 2021 at 11:06:07PM -0400, Kevin A. McGrail wrote: >> No worries there that I know of. >> >> cPanel has the paperwork for CCLA on file and several people with ICLA's as >> well. They've given us permission to commit the code too. >> >> I think it will be better than any dependency on external binaries. >> >> Regards, >> >> KAM >> >> On 10/14/2021 10:37 AM, Henrik K wrote: >>> If that's the case, I probably wouldn't have any objections. Not sure if it >>> requires some Contributor License Agreement from cPanels part (maybe they >>> already have one), and I guess atleast a bug to make it official.. Sidney >>> or KAM can probably chime in on the admin side.. >>> >>> >>> On Thu, Oct 14, 2021 at 04:32:53PM +0200, Giovanni Bechis wrote: >>>> Once committed, code will be no more developed by cPanel on CPAN >>>> and original code will be removed. >>>> >>>> I can work to integrate old and new Pyzor versions. >>>> >>>> Giovanni >>>> >>>> On Thu, Oct 14, 2021 at 05:27:16PM +0300, Henrik K wrote: >>>>> If it's developed by cPanel in CPAN, then it should not be committed to >>>>> SA, >>>>> unless it's clearly donated to SpamAssassin and removed from CPAN. >>>>> Assuming >>>>> we have developer resources and will to take it aboard. >>>>> >>>>> As it is, Plugin/Pyzor.pm should have an option to choose which one to >>>>> use, >>>>> as it makes no sense to ditch support for the widely installed original >>>>> Pyzor. >>>>> >>>>> >>>>> On Thu, Oct 14, 2021 at 04:15:13PM +0200, Giovanni Bechis wrote: >>>>>> Hi, >>>>>> cPanel has developed a native Perl Pyzor implementation for SpamAssassin >>>>>> and a diff against SpamAssassin 4.0 follows. >>>>>> Atm I am using it in production on a small server, more tests and >>>>>> opinions are welcome. >>>>>> >>>>>> Original cPanel code is at https://metacpan.org/pod/Mail::Pyzor. >>>>>> >>>>>> Cheers >>>>>> Giovanni >>>>>> >>>>>> diff --git a/MANIFEST b/MANIFEST >>>>>> index 25d0192..2d9588c 100644 >>>>>> --- a/MANIFEST >>>>>> +++ b/MANIFEST >>>>>> @@ -126,6 +126,11 @@ lib/Mail/SpamAssassin/Plugin/WLBLEval.pm >>>>>> lib/Mail/SpamAssassin/Plugin/WhiteListSubject.pm >>>>>> lib/Mail/SpamAssassin/PluginHandler.pm >>>>>> lib/Mail/SpamAssassin/Plugin/URILocalBL.pm >>>>>> +lib/Mail/SpamAssassin/Pyzor/Client.pm >>>>>> +lib/Mail/SpamAssassin/Pyzor/Digest/Pieces.pm >>>>>> +lib/Mail/SpamAssassin/Pyzor/Digest/StripHtml.pm >>>>>> +lib/Mail/SpamAssassin/Pyzor/Digest.pm >>>>>> +lib/Mail/SpamAssassin/Pyzor.pm >>>>>> lib/Mail/SpamAssassin/RegistryBoundaries.pm >>>>>> lib/Mail/SpamAssassin/Reporter.pm >>>>>> lib/Mail/SpamAssassin/SQLBasedAddrList.pm >>>>>> diff --git a/lib/Mail/SpamAssassin/Plugin/Pyzor.pm >>>>>> b/lib/Mail/SpamAssassin/Plugin/Pyzor.pm >>>>>> index 3efd4b4..e4c9c05 100644 >>>>>> --- a/lib/Mail/SpamAssassin/Plugin/Pyzor.pm >>>>>> +++ b/lib/Mail/SpamAssassin/Plugin/Pyzor.pm >>>>>> @@ -36,17 +36,13 @@ package Mail::SpamAssassin::Plugin::Pyzor; >>>>>> use Mail::SpamAssassin::Plugin; >>>>>> use Mail::SpamAssassin::Logger; >>>>>> -use Mail::SpamAssassin::Timeout; >>>>>> -use Mail::SpamAssassin::Util qw(untaint_var untaint_file_path >>>>>> - proc_status_ok exit_status_str); >>>>>> +use Mail::SpamAssassin::Util qw(untaint_var); >>>>>> + >>>>>> use strict; >>>>>> use warnings; >>>>>> # use bytes; >>>>>> use re 'taint'; >>>>>> -use Storable; >>>>>> -use POSIX qw(PIPE_BUF WNOHANG _exit); >>>>>> - >>>>>> our @ISA = qw(Mail::SpamAssassin::Plugin); >>>>>> sub new { >>>>>> @@ -78,7 +74,7 @@ sub set_config { >>>>>> my ($self, $conf) = @_; >>>>>> my @cmds; >>>>>> -=head1 USER OPTIONS >>>>>> +=head1 ADMINISTRATOR OPTIONS >>>>>> =over 4 >>>>>> @@ -95,22 +91,7 @@ Whether to use Pyzor, if it is available. >>>>>> type => $Mail::SpamAssassin::Conf::CONF_TYPE_BOOL >>>>>> }); >>>>>> -=item pyzor_fork (0|1) (default: 0) >>>>>> - >>>>>> -Instead of running Pyzor synchronously, fork separate process for it and >>>>>> -read the results in later (similar to async DNS lookups). Increases >>>>>> -throughput. Experimental. >>>>>> - >>>>>> -=cut >>>>>> - >>>>>> - push(@cmds, { >>>>>> - setting => 'pyzor_fork', >>>>>> - is_admin => 1, >>>>>> - default => 0, >>>>>> - type => $Mail::SpamAssassin::Conf::CONF_TYPE_NUMERIC, >>>>>> - }); >>>>>> - >>>>>> -=item pyzor_count_min NUMBER (default: 5) >>>>>> +=item pyzor_count_min NUMBER (default: 5) >>>>>> This option sets how often a message's body checksum must have been >>>>>> reported to the Pyzor server before SpamAssassin will consider the >>>>>> Pyzor >>>>>> @@ -128,54 +109,8 @@ set this to a relatively low value, e.g. C<5>. >>>>>> type => $Mail::SpamAssassin::Conf::CONF_TYPE_NUMERIC >>>>>> }); >>>>>> - # Deprecated setting, the name makes no sense! >>>>>> - push (@cmds, { >>>>>> - setting => 'pyzor_max', >>>>>> - is_admin => 1, >>>>>> - type => $Mail::SpamAssassin::Conf::CONF_TYPE_NUMERIC, >>>>>> - code => sub { >>>>>> - my ($self, $key, $value, $line) = @_; >>>>>> - warn("deprecated setting used, change pyzor_max to >>>>>> pyzor_count_min\n"); >>>>>> - if ($value !~ /^\d+$/) { >>>>>> - return $Mail::SpamAssassin::Conf::INVALID_VALUE; >>>>>> - } >>>>>> - $self->{pyzor_count_min} = $value; >>>>>> - } >>>>>> - }); >>>>>> - >>>>>> -=item pyzor_whitelist_min NUMBER (default: 10) >>>>>> - >>>>>> -This option sets how often a message's body checksum must have been >>>>>> -whitelisted to the Pyzor server for SpamAssassin to consider ignoring >>>>>> the >>>>>> -result. Final decision is made by pyzor_whitelist_factor. >>>>>> - >>>>>> -=cut >>>>>> - >>>>>> - push (@cmds, { >>>>>> - setting => 'pyzor_whitelist_min', >>>>>> - is_admin => 1, >>>>>> - default => 10, >>>>>> - type => $Mail::SpamAssassin::Conf::CONF_TYPE_NUMERIC >>>>>> - }); >>>>>> - >>>>>> -=item pyzor_whitelist_factor NUMBER (default: 0.2) >>>>>> - >>>>>> -Ignore Pyzor result if REPORTCOUNT x NUMBER >= pyzor_whitelist_min. >>>>>> -For default setting this means: 50 reports requires 10 whitelistings. >>>>>> - >>>>>> -=cut >>>>>> - >>>>>> - push (@cmds, { >>>>>> - setting => 'pyzor_whitelist_factor', >>>>>> - is_admin => 1, >>>>>> - default => 0.2, >>>>>> - type => $Mail::SpamAssassin::Conf::CONF_TYPE_NUMERIC >>>>>> - }); >>>>>> - >>>>>> =back >>>>>> -=head1 ADMINISTRATOR OPTIONS >>>>>> - >>>>>> =over 4 >>>>>> =item pyzor_timeout n (default: 5) >>>>>> @@ -210,478 +145,182 @@ removing one of them. >>>>>> type => $Mail::SpamAssassin::Conf::CONF_TYPE_DURATION >>>>>> }); >>>>>> -=item pyzor_options options >>>>>> +=item pyzor_whitelist_min NUMBER (default: 10) >>>>>> -Specify additional options to the pyzor(1) command. Please note that >>>>>> only >>>>>> -characters in the range [0-9A-Za-z =,._/-] are allowed for security >>>>>> reasons. >>>>>> +This option sets how often a message's body checksum must have been >>>>>> +whitelisted to the Pyzor server for SpamAssassin to consider ignoring >>>>>> the >>>>>> +result. Final decision is made by pyzor_whitelist_factor. >>>>>> =cut >>>>>> push (@cmds, { >>>>>> - setting => 'pyzor_options', >>>>>> + setting => 'pyzor_whitelist_min', >>>>>> is_admin => 1, >>>>>> - default => '', >>>>>> - type => $Mail::SpamAssassin::Conf::CONF_TYPE_STRING, >>>>>> - code => sub { >>>>>> - my ($self, $key, $value, $line) = @_; >>>>>> - if ($value !~ m{^([0-9A-Za-z =,._/-]+)$}) { >>>>>> - return $Mail::SpamAssassin::Conf::INVALID_VALUE; >>>>>> - } >>>>>> - $self->{pyzor_options} = $1; >>>>>> - } >>>>>> + default => 10, >>>>>> + type => $Mail::SpamAssassin::Conf::CONF_TYPE_NUMERIC >>>>>> }); >>>>>> -=item pyzor_path STRING >>>>>> +=item pyzor_whitelist_factor NUMBER (default: 0.2) >>>>>> -This option tells SpamAssassin specifically where to find the C<pyzor> >>>>>> -client instead of relying on SpamAssassin to find it in the current >>>>>> -PATH. Note that if I<taint mode> is enabled in the Perl interpreter, >>>>>> -you should use this, as the current PATH will have been cleared. >>>>>> +Ignore Pyzor result if REPORTCOUNT x NUMBER >= pyzor_whitelist_min. >>>>>> +For default setting this means: 50 reports requires 10 whitelistings. >>>>>> =cut >>>>>> push (@cmds, { >>>>>> - setting => 'pyzor_path', >>>>>> + setting => 'pyzor_whitelist_factor', >>>>>> is_admin => 1, >>>>>> - default => undef, >>>>>> - type => $Mail::SpamAssassin::Conf::CONF_TYPE_STRING, >>>>>> - code => sub { >>>>>> - my ($self, $key, $value, $line) = @_; >>>>>> - if (!defined $value || !length $value) { >>>>>> - return $Mail::SpamAssassin::Conf::MISSING_REQUIRED_VALUE; >>>>>> - } >>>>>> - $value = untaint_file_path($value); >>>>>> - if (!-x $value) { >>>>>> - info("config: pyzor_path \"$value\" isn't an executable"); >>>>>> - return $Mail::SpamAssassin::Conf::INVALID_VALUE; >>>>>> - } >>>>>> - >>>>>> - $self->{pyzor_path} = $value; >>>>>> - } >>>>>> + default => 0.2, >>>>>> + type => $Mail::SpamAssassin::Conf::CONF_TYPE_NUMERIC >>>>>> }); >>>>>> $conf->{parser}->register_commands(\@cmds); >>>>>> } >>>>>> sub is_pyzor_available { >>>>>> - my ($self) = @_; >>>>>> + my ($self) = @_; >>>>>> - my $pyzor = $self->{main}->{conf}->{pyzor_path} || >>>>>> - Mail::SpamAssassin::Util::find_executable_in_env_path('pyzor'); >>>>>> - >>>>>> - unless ($pyzor && -x $pyzor) { >>>>>> - dbg("pyzor: no pyzor executable found"); >>>>>> - $self->{pyzor_available} = 0; >>>>>> - return 0; >>>>>> - } >>>>>> - >>>>>> - # remember any found pyzor >>>>>> - $self->{main}->{conf}->{pyzor_path} = $pyzor; >>>>>> - >>>>>> - dbg("pyzor: pyzor is available: $pyzor"); >>>>>> - return 1; >>>>>> + local $@; >>>>>> + eval { >>>>>> + require Mail::SpamAssassin::Pyzor::Digest; >>>>>> + require Mail::SpamAssassin::Pyzor::Client; >>>>>> + }; >>>>>> + return $@ ? 0 : 1; >>>>>> } >>>>>> -sub finish_parsing_start { >>>>>> - my ($self, $opts) = @_; >>>>>> +sub get_pyzor_interface { >>>>>> + my ($self) = @_; >>>>>> - # If forking, hard adjust priority -100 to launch early >>>>>> - # Find rulenames from eval_to_rule mappings >>>>>> - if ($opts->{conf}->{pyzor_fork}) { >>>>>> - foreach (@{$opts->{conf}->{eval_to_rule}->{check_pyzor}}) { >>>>>> - dbg("pyzor: adjusting rule $_ priority to -100"); >>>>>> - $opts->{conf}->{priority}->{$_} = -100; >>>>>> - } >>>>>> + if (!$self->{main}->{conf}->{use_pyzor}) { >>>>>> + dbg("pyzor: use_pyzor option not enabled, disabling Pyzor"); >>>>>> + $self->{pyzor_interface} = "disabled"; >>>>>> + $self->{pyzor_available} = 0; >>>>>> + } >>>>>> + elsif ($self->is_pyzor_available()) { >>>>>> + $self->{pyzor_interface} = "pyzor"; >>>>>> + $self->{pyzor_available} = 1; >>>>>> + } >>>>>> + else { >>>>>> + dbg("pyzor: no pyzor found, disabling Pyzor"); >>>>>> + $self->{pyzor_available} = 0; >>>>>> } >>>>>> } >>>>>> sub check_pyzor { >>>>>> - my ($self, $pms, $full) = @_; >>>>>> - >>>>>> - return 0 if !$self->{pyzor_available}; >>>>>> - return 0 if !$self->{main}->{conf}->{use_pyzor}; >>>>>> - >>>>>> - return 0 if $pms->{pyzor_running}; >>>>>> - $pms->{pyzor_running} = 1; >>>>>> - >>>>>> - return 0 if !$self->is_pyzor_available(); >>>>>> - >>>>>> - my $timer = $self->{main}->time_method("check_pyzor"); >>>>>> + my ($self, $permsgstatus, $full) = @_; >>>>>> # initialize valid tags >>>>>> - $pms->{tag_data}->{PYZOR} = ''; >>>>>> - >>>>>> - # create fulltext tmpfile now (before possible forking) >>>>>> - $pms->{pyzor_tmpfile} = $pms->create_fulltext_tmpfile(); >>>>>> - >>>>>> - ## non-forking method >>>>>> - >>>>>> - if (!$self->{main}->{conf}->{pyzor_fork}) { >>>>>> - my @results = $self->pyzor_lookup($pms); >>>>>> - return $self->_check_result($pms, \@results); >>>>>> - } >>>>>> - >>>>>> - ## forking method >>>>>> - >>>>>> - $pms->{pyzor_rulename} = $pms->get_current_eval_rule_name(); >>>>>> - $pms->rule_pending($pms->{pyzor_rulename}); # mark async >>>>>> - >>>>>> - # create socketpair for communication >>>>>> - $pms->{pyzor_backchannel} = >>>>>> Mail::SpamAssassin::SubProcBackChannel->new(); >>>>>> - my $back_selector = ''; >>>>>> - $pms->{pyzor_backchannel}->set_selector(\$back_selector); >>>>>> - eval { >>>>>> - $pms->{pyzor_backchannel}->setup_backchannel_parent_pre_fork(); >>>>>> - } or do { >>>>>> - dbg("pyzor: backchannel pre-setup failed: $@"); >>>>>> - delete $pms->{pyzor_backchannel}; >>>>>> - return 0; >>>>>> - }; >>>>>> + $permsgstatus->{tag_data}->{PYZOR} = ""; >>>>>> - my $pid = fork(); >>>>>> - if (!defined $pid) { >>>>>> - info("pyzor: child fork failed: $!"); >>>>>> - delete $pms->{pyzor_backchannel}; >>>>>> - return 0; >>>>>> - } >>>>>> - if (!$pid) { >>>>>> - $0 = "$0 (pyzor)"; >>>>>> - $SIG{CHLD} = 'DEFAULT'; >>>>>> - $SIG{PIPE} = 'IGNORE'; >>>>>> - $SIG{$_} = sub { >>>>>> - eval { dbg("pyzor: child process $$ caught signal $_[0]"); }; >>>>>> - _exit(6); # avoid END and destructor processing >>>>>> - kill('KILL',$$); # still kicking? die! >>>>>> - } foreach qw(INT HUP TERM TSTP QUIT USR1 USR2); >>>>>> - dbg("pyzor: child process $$ forked"); >>>>>> - $pms->{pyzor_backchannel}->setup_backchannel_child_post_fork(); >>>>>> - my @results = $self->pyzor_lookup($pms); >>>>>> - my $backmsg; >>>>>> - eval { >>>>>> - $backmsg = Storable::freeze(\@results); >>>>>> - }; >>>>>> - if ($@) { >>>>>> - dbg("pyzor: child return value freeze failed: $@"); >>>>>> - _exit(0); # avoid END and destructor processing >>>>>> - } >>>>>> - if (!syswrite($pms->{pyzor_backchannel}->{parent}, $backmsg)) { >>>>>> - dbg("pyzor: child backchannel write failed: $!"); >>>>>> - } >>>>>> - _exit(0); # avoid END and destructor processing >>>>>> - } >>>>>> - >>>>>> - $pms->{pyzor_pid} = $pid; >>>>>> + my $timer = $self->{main}->time_method("check_pyzor"); >>>>>> - eval { >>>>>> - $pms->{pyzor_backchannel}->setup_backchannel_parent_post_fork($pid); >>>>>> - } or do { >>>>>> - dbg("pyzor: backchannel post-setup failed: $@"); >>>>>> - delete $pms->{pyzor_backchannel}; >>>>>> - return 0; >>>>>> - }; >>>>>> + $self->get_pyzor_interface(); >>>>>> + return 0 unless $self->{pyzor_available}; >>>>>> - return 0; >>>>>> + return $self->pyzor_lookup($permsgstatus, $full); >>>>>> } >>>>>> sub pyzor_lookup { >>>>>> - my ($self, $pms) = @_; >>>>>> - >>>>>> - my $conf = $self->{main}->{conf}; >>>>>> - my $timeout = $conf->{pyzor_timeout}; >>>>>> - >>>>>> - # note: not really tainted, this came from system configuration file >>>>>> - my $path = untaint_file_path($conf->{pyzor_path}); >>>>>> - my $opts = untaint_var($conf->{pyzor_options}) || ''; >>>>>> - >>>>>> - $pms->enter_helper_run_mode(); >>>>>> - >>>>>> - my $pid; >>>>>> - my @resp; >>>>>> - my $timer = Mail::SpamAssassin::Timeout->new( >>>>>> - { secs => $timeout, deadline => $pms->{master_deadline} }); >>>>>> - my $err = $timer->run_and_catch(sub { >>>>>> - local $SIG{PIPE} = sub { die "__brokenpipe__ignore__\n" }; >>>>>> - >>>>>> - dbg("pyzor: opening pipe: ". >>>>>> - join(' ', $path, $opts, "check", "<".$pms->{pyzor_tmpfile})); >>>>>> - >>>>>> - $pid = Mail::SpamAssassin::Util::helper_app_pipe_open(*PYZOR, >>>>>> - $pms->{pyzor_tmpfile}, 1, $path, split(' ', $opts), "check"); >>>>>> - $pid or die "$!\n"; >>>>>> - >>>>>> - # read+split avoids a Perl I/O bug (Bug 5985) >>>>>> - my($inbuf, $nread); >>>>>> - my $resp = ''; >>>>>> - while ($nread = read(PYZOR, $inbuf, 8192)) { $resp .= $inbuf } >>>>>> - defined $nread or die "error reading from pipe: $!"; >>>>>> - @resp = split(/^/m, $resp, -1); >>>>>> - >>>>>> - my $errno = 0; >>>>>> - close PYZOR or $errno = $!; >>>>>> - if (proc_status_ok($?, $errno)) { >>>>>> - dbg("pyzor: [%s] finished successfully", $pid); >>>>>> - } elsif (proc_status_ok($?, $errno, 0, 1)) { # sometimes it exits >>>>>> with 1 >>>>>> - dbg("pyzor: [%s] finished: %s", $pid, exit_status_str($?, >>>>>> $errno)); >>>>>> - } else { >>>>>> - info("pyzor: [%s] error: %s", $pid, exit_status_str($?, $errno)); >>>>>> - } >>>>>> - >>>>>> - }); >>>>>> - >>>>>> - if (defined(fileno(*PYZOR))) { # still open >>>>>> - if ($pid) { >>>>>> - if (kill('TERM', $pid)) { >>>>>> - dbg("pyzor: killed stale helper [$pid]"); >>>>>> - } else { >>>>>> - dbg("pyzor: killing helper application [$pid] failed: $!"); >>>>>> - } >>>>>> - } >>>>>> - my $errno = 0; >>>>>> - close PYZOR or $errno = $!; >>>>>> - proc_status_ok($?, $errno) >>>>>> - or info("pyzor: [%s] error: %s", $pid, exit_status_str($?, >>>>>> $errno)); >>>>>> - } >>>>>> - >>>>>> - $pms->leave_helper_run_mode(); >>>>>> - >>>>>> - if ($timer->timed_out()) { >>>>>> - dbg("pyzor: check timed out after $timeout seconds"); >>>>>> - return (); >>>>>> - } elsif ($err) { >>>>>> - chomp $err; >>>>>> - info("pyzor: check failed: $err"); >>>>>> - return (); >>>>>> - } >>>>>> - >>>>>> - return @resp; >>>>>> -} >>>>>> - >>>>>> -sub check_tick { >>>>>> - my ($self, $opts) = @_; >>>>>> - $self->_check_forked_result($opts->{permsgstatus}, 0); >>>>>> -} >>>>>> - >>>>>> -sub check_cleanup { >>>>>> - my ($self, $opts) = @_; >>>>>> - $self->_check_forked_result($opts->{permsgstatus}, 1); >>>>>> -} >>>>>> - >>>>>> -sub _check_forked_result { >>>>>> - my ($self, $pms, $finish) = @_; >>>>>> - >>>>>> - return 0 if !$pms->{pyzor_backchannel}; >>>>>> - return 0 if !$pms->{pyzor_pid}; >>>>>> + my ( $self, $permsgstatus, $fulltext ) = @_; >>>>>> + my $conf = $self->{main}->{conf}; >>>>>> + my $timeout = $conf->{pyzor_timeout}; >>>>>> + >>>>>> + my $client = ( $self->{'_pyzor_client'} ||= >>>>>> Mail::SpamAssassin::Pyzor::Client->new( 'timeout' => $timeout ) ); >>>>>> + my $digest = Mail::SpamAssassin::Pyzor::Digest::get( $fulltext ); >>>>>> + >>>>>> + local $@; >>>>>> + my $ref = eval { $client->check($digest); }; >>>>>> + dbg("pyzor: got response: $client->{'_server_host'}"); >>>>>> + # $client reply must be an hash >>>>>> + return 0 if (not (ref $ref eq ref {})); >>>>>> + if ($@) { >>>>>> + my $err = $@; >>>>>> - my $timer = $self->{main}->time_method("check_pyzor"); >>>>>> + $err = eval { $err->get_message() } || $err; >>>>>> - $pms->{pyzor_abort} = $pms->{deadline_exceeded} || >>>>>> $pms->{shortcircuited}; >>>>>> - >>>>>> - my $kid_pid = $pms->{pyzor_pid}; >>>>>> - # if $finish, force waiting for the child >>>>>> - my $pid = waitpid($kid_pid, $finish && !$pms->{pyzor_abort} ? 0 : >>>>>> WNOHANG); >>>>>> - if ($pid == 0) { >>>>>> - #dbg("pyzor: child process $kid_pid not finished yet, trying >>>>>> later"); >>>>>> - if ($pms->{pyzor_abort}) { >>>>>> - dbg("pyzor: bailing out due to deadline/shortcircuit"); >>>>>> - kill('TERM', $kid_pid); >>>>>> - if (waitpid($kid_pid, WNOHANG) == 0) { >>>>>> - sleep(1); >>>>>> - if (waitpid($kid_pid, WNOHANG) == 0) { >>>>>> - dbg("pyzor: child process $kid_pid still alive, KILL"); >>>>>> - kill('KILL', $kid_pid); >>>>>> - waitpid($kid_pid, 0); >>>>>> + warn("pyzor: check failed: $err\n"); >>>>>> + return 0; >>>>>> + } elsif ( defined $ref->{'Code'} and $ref->{'Code'} ne 200 ) { >>>>>> + if(defined $ref->{'Code'} and defined $ref->{'Diag'}) { >>>>>> + dbg("pyzor: check failed with invalid code: $ref->{'Code'}: >>>>>> $ref->{'Diag'}"); >>>>>> + } else { >>>>>> + dbg("pyzor: check failed with undefined code"); >>>>>> } >>>>>> - } >>>>>> - delete $pms->{pyzor_pid}; >>>>>> - delete $pms->{pyzor_backchannel}; >>>>>> + return 0; >>>>>> } >>>>>> - return 0; >>>>>> - } elsif ($pid == -1) { >>>>>> - # child does not exist? >>>>>> - dbg("pyzor: child process $kid_pid already handled?"); >>>>>> - delete $pms->{pyzor_backchannel}; >>>>>> - return 0; >>>>>> - } >>>>>> - $pms->rule_ready($pms->{pyzor_rulename}); # mark rule ready for metas >>>>>> + my $pyzor_count = untaint_var($ref->{'Count'}) + 0; >>>>>> + my $pyzor_whitelisted = untaint_var($ref->{'WL-Count'}) + 0; >>>>>> + my $count_min = $conf->{pyzor_count_min}; >>>>>> + my $wl_min = $conf->{pyzor_whitelist_min}; >>>>>> - dbg("pyzor: child process $kid_pid finished, reading results"); >>>>>> + my $wl_limit = $pyzor_whitelisted >= $wl_min ? >>>>>> + $pyzor_count * $conf->{pyzor_whitelist_factor} : 0; >>>>>> - my $backmsg; >>>>>> - my $ret = sysread($pms->{pyzor_backchannel}->{latest_kid_fh}, >>>>>> $backmsg, PIPE_BUF); >>>>>> - if (!defined $ret || $ret == 0) { >>>>>> - dbg("pyzor: could not read result from child: ".($ret == 0 ? 0 : >>>>>> $!)); >>>>>> - delete $pms->{pyzor_backchannel}; >>>>>> - return 0; >>>>>> - } >>>>>> - >>>>>> - delete $pms->{pyzor_backchannel}; >>>>>> + $permsgstatus->set_tag('PYZOR', "Reported $pyzor_count times, >>>>>> whitelisted $pyzor_whitelisted times."); >>>>>> - my $results; >>>>>> - eval { >>>>>> - $results = Storable::thaw($backmsg); >>>>>> - }; >>>>>> - if ($@) { >>>>>> - dbg("pyzor: child return value thaw failed: $@"); >>>>>> - return; >>>>>> - } >>>>>> - >>>>>> - $self->_check_result($pms, $results); >>>>>> -} >>>>>> + dbg("pyzor: result: COUNT=$pyzor_count/$count_min >>>>>> WHITELIST=$pyzor_whitelisted/$wl_min/%.1f", >>>>>> + $wl_limit); >>>>>> -sub _check_result { >>>>>> - my ($self, $pms, $results) = @_; >>>>>> - >>>>>> - if (!@$results) { >>>>>> - dbg("pyzor: no response from server"); >>>>>> - return 0; >>>>>> - } >>>>>> - >>>>>> - my $count = 0; >>>>>> - my $count_wl = 0; >>>>>> - foreach my $res (@$results) { >>>>>> - chomp($res); >>>>>> - if ($res =~ /^Traceback/) { >>>>>> - info("pyzor: internal error, python traceback seen in response: >>>>>> $res"); >>>>>> + # Empty body etc results in same hash, we should skip very large >>>>>> numbers.. >>>>>> + if ($pyzor_count >= 1000000 || $pyzor_whitelisted >= 10000) { >>>>>> + dbg("pyzor: result exceeded hardcoded limits, ignoring: count/wl >>>>>> 1000000/10000"); >>>>>> return 0; >>>>>> } >>>>>> - dbg("pyzor: got response: $res"); >>>>>> - # this regexp is intended to be a little bit forgiving >>>>>> - if ($res =~ /^\S+\t.*?\t(\d+)\t(\d+)\s*$/) { >>>>>> - # until pyzor servers can sync their DBs, >>>>>> - # sum counts obtained from all servers >>>>>> - $count += untaint_var($1)+0; # crazy but needs untainting >>>>>> - $count_wl += untaint_var($2)+0; >>>>>> - } else { >>>>>> - # warn on failures to parse >>>>>> - info("pyzor: failure to parse response \"$res\""); >>>>>> - } >>>>>> - } >>>>>> - >>>>>> - my $conf = $self->{main}->{conf}; >>>>>> - >>>>>> - my $count_min = $conf->{pyzor_count_min}; >>>>>> - my $wl_min = $conf->{pyzor_whitelist_min}; >>>>>> - my $wl_limit = $count_wl >= $wl_min ? >>>>>> - $count * $conf->{pyzor_whitelist_factor} : 0; >>>>>> - >>>>>> - dbg("pyzor: result: COUNT=$count/$count_min >>>>>> WHITELIST=$count_wl/$wl_min/%.1f", >>>>>> - $wl_limit); >>>>>> - $pms->set_tag('PYZOR', "Reported $count times, whitelisted $count_wl >>>>>> times."); >>>>>> - >>>>>> - # Empty body etc results in same hash, we should skip very large >>>>>> numbers.. >>>>>> - if ($count >= 1000000 || $count_wl >= 10000) { >>>>>> - dbg("pyzor: result exceeded hardcoded limits, ignoring: count/wl >>>>>> 1000000/10000"); >>>>>> - return 0; >>>>>> - } >>>>>> - >>>>>> - # Whitelisted? >>>>>> - if ($wl_limit && $count_wl >= $wl_limit) { >>>>>> - dbg("pyzor: message whitelisted"); >>>>>> - return 0; >>>>>> - } >>>>>> + # Whitelisted? >>>>>> + if ($wl_limit && $pyzor_whitelisted >= $wl_limit) { >>>>>> + dbg("pyzor: message whitelisted"); >>>>>> + return 0; >>>>>> + } >>>>>> - if ($count >= $count_min) { >>>>>> - if ($conf->{pyzor_fork}) { >>>>>> - # forked needs to run got_hit() >>>>>> - $pms->got_hit($pms->{pyzor_rulename}, "", ruletype => 'eval'); >>>>>> + if ( $pyzor_count >= $count_min ) { >>>>>> + return 1; >>>>>> } >>>>>> - return 1; >>>>>> - } >>>>>> - return 0; >>>>>> + return 0; >>>>>> } >>>>>> sub plugin_report { >>>>>> my ($self, $options) = @_; >>>>>> - return if !$self->{pyzor_available}; >>>>>> - return if !$self->{main}->{conf}->{use_pyzor}; >>>>>> - return if $options->{report}->{options}->{dont_report_to_pyzor}; >>>>>> - return if !$self->is_pyzor_available(); >>>>>> - >>>>>> - # use temporary file: open2() is unreliable due to buffering under >>>>>> spamd >>>>>> - my $tmpf = >>>>>> $options->{report}->create_fulltext_tmpfile($options->{text}); >>>>>> - if ($self->pyzor_report($options, $tmpf)) { >>>>>> - $options->{report}->{report_available} = 1; >>>>>> - info("reporter: spam reported to Pyzor"); >>>>>> - $options->{report}->{report_return} = 1; >>>>>> - } >>>>>> - else { >>>>>> - info("reporter: could not report spam to Pyzor"); >>>>>> - } >>>>>> - $options->{report}->delete_fulltext_tmpfile($tmpf); >>>>>> + return unless $self->{pyzor_available}; >>>>>> + return unless $self->{main}->{conf}->{use_pyzor}; >>>>>> - return 1; >>>>>> + if (!$options->{report}->{options}->{dont_report_to_pyzor} && >>>>>> $self->is_pyzor_available()) >>>>>> + { >>>>>> + if ($self->pyzor_report($options)) { >>>>>> + $options->{report}->{report_available} = 1; >>>>>> + info("reporter: spam reported to Pyzor"); >>>>>> + $options->{report}->{report_return} = 1; >>>>>> + } >>>>>> + else { >>>>>> + info("reporter: could not report spam to Pyzor"); >>>>>> + } >>>>>> + } >>>>>> } >>>>>> sub pyzor_report { >>>>>> - my ($self, $options, $tmpf) = @_; >>>>>> - >>>>>> - # note: not really tainted, this came from system configuration file >>>>>> - my $path = >>>>>> untaint_file_path($options->{report}->{conf}->{pyzor_path}); >>>>>> - my $opts = untaint_var($options->{report}->{conf}->{pyzor_options}) >>>>>> || ''; >>>>>> + my ( $self, $options ) = @_; >>>>>> - my $timeout = $self->{main}->{conf}->{pyzor_timeout}; >>>>>> + my $timeout = $self->{main}->{conf}->{pyzor_timeout}; >>>>>> - $options->{report}->enter_helper_run_mode(); >>>>>> + my $client = ( $self->{'_pyzor_client'} ||= >>>>>> Mail::SpamAssassin::Pyzor::Client->new( 'timeout' => $timeout ) ); >>>>>> - my $timer = Mail::SpamAssassin::Timeout->new({ secs => $timeout }); >>>>>> - my $err = $timer->run_and_catch(sub { >>>>>> + my $digest = Mail::SpamAssassin::Pyzor::Digest::get( >>>>>> $options->{'text'} ); >>>>>> - local $SIG{PIPE} = sub { die "__brokenpipe__ignore__\n" }; >>>>>> - >>>>>> - dbg("pyzor: opening pipe: " . join(' ', $path, $opts, "report", "< >>>>>> $tmpf")); >>>>>> - >>>>>> - my $pid = Mail::SpamAssassin::Util::helper_app_pipe_open(*PYZOR, >>>>>> - $tmpf, 1, $path, split(' ', $opts), "report"); >>>>>> - $pid or die "$!\n"; >>>>>> - >>>>>> - my($inbuf,$nread,$nread_all); $nread_all = 0; >>>>>> - # response is ignored, just check its existence >>>>>> - while ( $nread=read(PYZOR,$inbuf,8192) ) { $nread_all += $nread } >>>>>> - defined $nread or die "error reading from pipe: $!"; >>>>>> - >>>>>> - dbg("pyzor: empty response") if $nread_all < 1; >>>>>> - >>>>>> - my $errno = 0; close PYZOR or $errno = $!; >>>>>> - # closing a pipe also waits for the process executing on the pipe to >>>>>> - # complete, no need to explicitly call waitpid >>>>>> - # my $child_stat = waitpid($pid,0) > 0 ? $? : undef; >>>>>> - if (proc_status_ok($?,$errno, 0)) { >>>>>> - dbg("pyzor: [%s] reporter finished successfully", $pid); >>>>>> - } else { >>>>>> - info("pyzor: [%s] reporter error: %s", $pid, >>>>>> exit_status_str($?,$errno)); >>>>>> + local $@; >>>>>> + my $ref = eval { $client->report($digest); }; >>>>>> + if ($@) { >>>>>> + warn("pyzor: report failed: $@"); >>>>>> + return 0; >>>>>> } >>>>>> - >>>>>> - }); >>>>>> - >>>>>> - $options->{report}->leave_helper_run_mode(); >>>>>> - >>>>>> - if ($timer->timed_out()) { >>>>>> - dbg("reporter: pyzor report timed out after $timeout seconds"); >>>>>> - return 0; >>>>>> - } >>>>>> - >>>>>> - if ($err) { >>>>>> - chomp $err; >>>>>> - if ($err eq '__brokenpipe__ignore__') { >>>>>> - dbg("reporter: pyzor report failed: broken pipe"); >>>>>> - } else { >>>>>> - warn("reporter: pyzor report failed: $err\n"); >>>>>> + elsif ( $ref->{'Code'} ne 200 ) { >>>>>> + dbg("pyzor: report failed with invalid code: $ref->{'Code'}: >>>>>> $ref->{'Diag'}"); >>>>>> + return 0; >>>>>> } >>>>>> - return 0; >>>>>> - } >>>>>> - return 1; >>>>>> + return 1; >>>>>> } >>>>>> -# Version features >>>>>> -sub has_fork { 1 } >>>>>> - >>>>>> 1; >>>>>> - >>>>>> -=back >>>>>> - >>>>>> -=cut >>>>>> diff --git a/lib/Mail/SpamAssassin/Pyzor.pm >>>>>> b/lib/Mail/SpamAssassin/Pyzor.pm >>>>>> new file mode 100644 >>>>>> index 0000000..8ac27f4 >>>>>> --- /dev/null >>>>>> +++ b/lib/Mail/SpamAssassin/Pyzor.pm >>>>>> @@ -0,0 +1,56 @@ >>>>>> +package Mail::SpamAssassin::Pyzor; >>>>>> + >>>>>> +# Copyright 2018 cPanel, LLC. >>>>>> +# All rights reserved. >>>>>> +# http://cpanel.net >>>>>> +# >>>>>> +# <@LICENSE> >>>>>> +# Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> +# contributor license agreements. See the NOTICE file distributed with >>>>>> +# this work for additional information regarding copyright ownership. >>>>>> +# The ASF licenses this file to you under the Apache License, Version >>>>>> 2.0 >>>>>> +# (the "License"); you may not use this file except in compliance with >>>>>> +# the License. You may obtain a copy of the License at: >>>>>> +# >>>>>> +# http://www.apache.org/licenses/LICENSE-2.0 >>>>>> +# >>>>>> +# Unless required by applicable law or agreed to in writing, software >>>>>> +# distributed under the License is distributed on an "AS IS" BASIS, >>>>>> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>>> implied. >>>>>> +# See the License for the specific language governing permissions and >>>>>> +# limitations under the License. >>>>>> +# </@LICENSE> >>>>>> +# >>>>>> + >>>>>> +use strict; >>>>>> +use warnings; >>>>>> + >>>>>> +our $VERSION = '0.06_01'; >>>>>> + >>>>>> +=encoding utf-8 >>>>>> + >>>>>> +=head1 NAME >>>>>> + >>>>>> +Mail::SpamAssassin::Pyzor - Pyzor spam filtering in Perl >>>>>> + >>>>>> +=head1 DESCRIPTION >>>>>> + >>>>>> +This distribution contains Perl implementations of parts of >>>>>> +L<Pyzor|http://pyzor.org>, a tool for use in spam email filtering. >>>>>> +It is intended for use with L<Mail::SpamAssassin> but may be useful >>>>>> +in other contexts. >>>>>> + >>>>>> +See the following modules for information on specific tools that >>>>>> +the distribution includes: >>>>>> + >>>>>> +=over >>>>>> + >>>>>> +=item * L<Mail::SpamAssassin::Pyzor::Client> >>>>>> + >>>>>> +=item * L<Mail::SpamAssassin::Pyzor::Digest> >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +1; >>>>>> diff --git a/lib/Mail/SpamAssassin/Pyzor/Client.pm >>>>>> b/lib/Mail/SpamAssassin/Pyzor/Client.pm >>>>>> new file mode 100644 >>>>>> index 0000000..ccff868 >>>>>> --- /dev/null >>>>>> +++ b/lib/Mail/SpamAssassin/Pyzor/Client.pm >>>>>> @@ -0,0 +1,415 @@ >>>>>> +package Mail::SpamAssassin::Pyzor::Client; >>>>>> + >>>>>> +# Copyright 2018 cPanel, LLC. >>>>>> +# All rights reserved. >>>>>> +# http://cpanel.net >>>>>> +# >>>>>> +# <@LICENSE> >>>>>> +# Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> +# contributor license agreements. See the NOTICE file distributed with >>>>>> +# this work for additional information regarding copyright ownership. >>>>>> +# The ASF licenses this file to you under the Apache License, Version >>>>>> 2.0 >>>>>> +# (the "License"); you may not use this file except in compliance with >>>>>> +# the License. You may obtain a copy of the License at: >>>>>> +# >>>>>> +# http://www.apache.org/licenses/LICENSE-2.0 >>>>>> +# >>>>>> +# Unless required by applicable law or agreed to in writing, software >>>>>> +# distributed under the License is distributed on an "AS IS" BASIS, >>>>>> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>>> implied. >>>>>> +# See the License for the specific language governing permissions and >>>>>> +# limitations under the License. >>>>>> +# </@LICENSE> >>>>>> +# >>>>>> + >>>>>> +use strict; >>>>>> +use warnings; >>>>>> + >>>>>> +=encoding utf-8 >>>>>> + >>>>>> +=head1 NAME >>>>>> + >>>>>> +Mail::SpamAssassin::Pyzor::Client - Pyzor client logic >>>>>> + >>>>>> +=head1 SYNOPSIS >>>>>> + >>>>>> + use Mail::SpamAssassin::Pyzor::Client (); >>>>>> + use Mail::SpamAssassin::Pyzor::Digest (); >>>>>> + >>>>>> + my $client = Mail::SpamAssassin::Pyzor::Client->new(); >>>>>> + >>>>>> + my $digest = Mail::SpamAssassin::Pyzor::Digest::get( $msg ); >>>>>> + >>>>>> + my $check_ref = $client->check($digest); >>>>>> + die $check_ref->{'Diag'} if $check_ref->{'Code'} ne '200'; >>>>>> + >>>>>> + my $report_ref = $client->report($digest); >>>>>> + die $report_ref->{'Diag'} if $report_ref->{'Code'} ne '200'; >>>>>> + >>>>>> +=head1 DESCRIPTION >>>>>> + >>>>>> +A bare-bones L<Pyzor|http://pyzor.org> client that currently only >>>>>> +implements the functionality needed for L<Mail::SpamAssassin>. >>>>>> + >>>>>> +=head1 PROTOCOL DETAILS >>>>>> + >>>>>> +The Pyzor protocol is not a published standard, and there appears to be >>>>>> +no meaningful public documentation. What follows is enough information, >>>>>> +largely gleaned through forum posts and reverse engineering, to >>>>>> facilitate >>>>>> +effective use of this module: >>>>>> + >>>>>> +Pyzor is an RPC-oriented, message-based protocol. Each message >>>>>> +is a simple dictionary of 7-bit ASCII keys and values. Server responses >>>>>> +always include at least the following: >>>>>> + >>>>>> +=over >>>>>> + >>>>>> +=item * C<Code> - Similar to HTTP status codes; anything besides C<200> >>>>>> +is an error. >>>>>> + >>>>>> +=item * C<Diag> - Similar to HTTP status reasons: a text description >>>>>> +of the status. >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +(NB: There are additional standard response headers that are useful >>>>>> only for >>>>>> +the protocol itself and thus are not part of this module???s returns.) >>>>>> + >>>>>> +=head2 Reliability >>>>>> + >>>>>> +Pyzor uses UDP rather than TCP, so no message is guaranteed to reach its >>>>>> +destination. A transmission failure can happen in either the request or >>>>>> +the response; in either case, a timeout error will result. Such errors >>>>>> +are represented as thrown instances of L<Mail::Pyzor::X::Timeout>. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +our $VERSION = '0.04'; >>>>>> + >>>>>> +our $DEFAULT_SERVER_HOST = 'public.pyzor.org'; >>>>>> +our $DEFAULT_SERVER_PORT = 24441; >>>>>> +our $DEFAULT_USERNAME = 'anonymous'; >>>>>> +our $DEFAULT_PASSWORD = ''; >>>>>> +our $DEFAULT_OP_SPEC = '20,3,60,3'; >>>>>> +our $PYZOR_PROTOCOL_VERSION = 2.1; >>>>>> +our $DEFAULT_TIMEOUT = 3.5; >>>>>> +our $READ_SIZE = 8192; >>>>>> + >>>>>> +use IO::Socket::INET (); >>>>>> +use Digest::SHA qw(sha1 sha1_hex); >>>>>> + >>>>>> +my @hash_order = ( 'Op', 'Op-Digest', 'Op-Spec', 'Thread', 'PV', >>>>>> 'User', 'Time', 'Sig' ); >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head1 CONSTRUCTOR >>>>>> + >>>>>> +=head2 new(%OPTS) >>>>>> + >>>>>> +Create a new pyzor client. >>>>>> + >>>>>> +=over 2 >>>>>> + >>>>>> +=item Input >>>>>> + >>>>>> +%OPTS are (all optional): >>>>>> + >>>>>> +=over 3 >>>>>> + >>>>>> +=item * C<server_host> - The pyzor server host to connect to (default is >>>>>> +C<public.pyzor.org>) >>>>>> + >>>>>> +=item * C<server_port> - The pyzor server port to connect to (default is >>>>>> +24441) >>>>>> + >>>>>> +=item * C<username> - The username to present to the pyzor server >>>>>> (default >>>>>> +is C<anonymous>) >>>>>> + >>>>>> +=item * C<password> - The password to present to the pyzor server >>>>>> (default >>>>>> +is empty) >>>>>> + >>>>>> +=item * C<timeout> - The maximum time, in seconds, to wait for a >>>>>> response >>>>>> +from the pyzor server (defeault is 3.5) >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=item Output >>>>>> + >>>>>> +=over 3 >>>>>> + >>>>>> +Returns a L<Mail::SpamAssassin::Pyzor::Client> object. >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub new { >>>>>> + my ( $class, %OPTS ) = @_; >>>>>> + >>>>>> + return bless { >>>>>> + '_server_host' => $OPTS{'server_host'} || $DEFAULT_SERVER_HOST, >>>>>> + '_server_port' => $OPTS{'server_port'} || $DEFAULT_SERVER_PORT, >>>>>> + '_username' => $OPTS{'username'} || $DEFAULT_USERNAME, >>>>>> + '_password' => $OPTS{'password'} || $DEFAULT_PASSWORD, >>>>>> + '_op_spec' => $DEFAULT_OP_SPEC, >>>>>> + '_timeout' => $OPTS{'timeout'} || $DEFAULT_TIMEOUT, >>>>>> + }, $class; >>>>>> +} >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head1 REQUEST METHODS >>>>>> + >>>>>> +=head2 report($digest) >>>>>> + >>>>>> +Report the digest of a spam message to the pyzor server. This function >>>>>> +will throw if a messaging failure or timeout happens. >>>>>> + >>>>>> +=over 2 >>>>>> + >>>>>> +=item Input >>>>>> + >>>>>> +=over 3 >>>>>> + >>>>>> +=item $digest C<SCALAR> >>>>>> + >>>>>> +The message digest to report, as given by >>>>>> +C<Mail::SpamAssassin::Pyzor::Digest::get()>. >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=item Output >>>>>> + >>>>>> +=over 3 >>>>>> + >>>>>> +=item C<HASHREF> >>>>>> + >>>>>> +Returns a hashref of the standard attributes noted above. >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub report { >>>>>> + my ( $self, $digest ) = @_; >>>>>> + >>>>>> + my $msg_ref = $self->_get_base_msg( 'report', $digest ); >>>>>> + >>>>>> + $msg_ref->{'Op-Spec'} = $self->{'_op_spec'}; >>>>>> + >>>>>> + return $self->_send_receive_msg($msg_ref); >>>>>> +} >>>>>> + >>>>>> +=head2 check($digest) >>>>>> + >>>>>> +Check the digest of a message to see if >>>>>> +the pyzor server has a report for it. This function >>>>>> +will throw if a messaging failure or timeout happens. >>>>>> + >>>>>> +=over 2 >>>>>> + >>>>>> +=item Input >>>>>> + >>>>>> +=over 3 >>>>>> + >>>>>> +=item $digest C<SCALAR> >>>>>> + >>>>>> +The message digest to check, as given by >>>>>> +C<Mail::SpamAssassin::Pyzor::Digest::get()>. >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=item Output >>>>>> + >>>>>> +=over 3 >>>>>> + >>>>>> +=item C<HASHREF> >>>>>> + >>>>>> +Returns a hashref of the standard attributes noted above >>>>>> +as well as the following: >>>>>> + >>>>>> +=over >>>>>> + >>>>>> +=item * C<Count> - The number of reports the server has received >>>>>> +for the given digest. >>>>>> + >>>>>> +=item * C<WL-Count> - The number of whitelist requests the server has >>>>>> received >>>>>> +for the given digest. >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=back >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub check { >>>>>> + my ( $self, $digest ) = @_; >>>>>> + >>>>>> + return $self->_send_receive_msg( $self->_get_base_msg( 'check', >>>>>> $digest ) ); >>>>>> +} >>>>>> + >>>>>> +# ---------------------------------------- >>>>>> + >>>>>> +sub _send_receive_msg { >>>>>> + my ( $self, $msg_ref ) = @_; >>>>>> + >>>>>> + my $thread_id = $msg_ref->{'Thread'} or warn 'No thread ID?'; >>>>>> + >>>>>> + $self->_sign_msg($msg_ref); >>>>>> + >>>>>> + return $self->_do_send_receive( >>>>>> + $self->_generate_packet_from_message($msg_ref) . "\n\n", >>>>>> + $thread_id, >>>>>> + ); >>>>>> +} >>>>>> + >>>>>> +sub _get_base_msg { >>>>>> + my ( $self, $op, $digest ) = @_; >>>>>> + >>>>>> + die "Implementor error: op is required" if !$op; >>>>>> + die "error: digest is required" if !$digest; >>>>>> + >>>>>> + return { >>>>>> + 'User' => $self->{'_username'}, >>>>>> + 'PV' => $PYZOR_PROTOCOL_VERSION, >>>>>> + 'Time' => time(), >>>>>> + 'Op' => $op, >>>>>> + 'Op-Digest' => $digest, >>>>>> + 'Thread' => $self->_generate_thread_id() >>>>>> + }; >>>>>> +} >>>>>> + >>>>>> +sub _do_send_receive { >>>>>> + my ( $self, $packet, $thread_id ) = @_; >>>>>> + >>>>>> + my $sock = $self->_get_connection_or_die(); >>>>>> + >>>>>> + $self->_send_packet( $sock, $packet ); >>>>>> + my $response = $self->_receive_packet( $sock, $thread_id ); >>>>>> + >>>>>> + return 0 if not defined $response; >>>>>> + >>>>>> + my $resp_hr = { map { ( split(m{: }) )[ 0, 1 ] } split( m{\n}, >>>>>> $response ) }; >>>>>> + >>>>>> + delete $resp_hr->{'Thread'}; >>>>>> + >>>>>> + my $response_pv = delete $resp_hr->{'PV'}; >>>>>> + >>>>>> + if ( $PYZOR_PROTOCOL_VERSION ne $response_pv ) { >>>>>> + warn "Unexpected protocol version ($response_pv) in Pyzor >>>>>> response!"; >>>>>> + } >>>>>> + >>>>>> + return $resp_hr; >>>>>> +} >>>>>> + >>>>>> +sub _receive_packet { >>>>>> + my ( $self, $sock, $thread_id ) = @_; >>>>>> + >>>>>> + my $timeout = $self->{'_timeout'} * 1000; >>>>>> + >>>>>> + my $end_time = time + $self->{'_timeout'}; >>>>>> + >>>>>> + $sock->blocking(0); >>>>>> + my $response = ''; >>>>>> + my $rout = ''; >>>>>> + my $rin = ''; >>>>>> + vec( $rin, fileno($sock), 1 ) = 1; >>>>>> + >>>>>> + while (1) { >>>>>> + my $time_left = $end_time - time; >>>>>> + >>>>>> + if ( $time_left <= 0 ) { >>>>>> + warn("Did not receive a response from the pyzor server >>>>>> $self->{'_server_host'}:$self->{'_server_port'} for $self->{'_timeout'} >>>>>> seconds!"); >>>>>> + return; >>>>>> + } >>>>>> + >>>>>> + my $bytes = sysread( $sock, $response, $READ_SIZE, length >>>>>> $response ); >>>>>> + if ( !defined($bytes) && !$!{'EAGAIN'} && !$!{'EWOULDBLOCK'} ) { >>>>>> + warn "read from socket: $!"; >>>>>> + } >>>>>> + >>>>>> + if ( index( $response, "\n\n" ) > -1 ) { >>>>>> + >>>>>> + # Reject the response unless its thread ID matches what we >>>>>> sent. >>>>>> + # This prevents confusion among concurrent Pyzor reqeusts. >>>>>> + if ( index( $response, "\nThread: $thread_id\n" ) != -1 ) { >>>>>> + last; >>>>>> + } >>>>>> + else { >>>>>> + $response = ''; >>>>>> + } >>>>>> + } >>>>>> + >>>>>> + my $found = select( $rout = $rin, undef, undef, $time_left ); >>>>>> + warn "select(): $!" if $found == -1; >>>>>> + } >>>>>> + >>>>>> + return $response; >>>>>> +} >>>>>> + >>>>>> +sub _send_packet { >>>>>> + my ( $self, $sock, $packet ) = @_; >>>>>> + >>>>>> + $sock->blocking(1); >>>>>> + syswrite( $sock, $packet ) or warn "write to socket: $!"; >>>>>> + >>>>>> + return; >>>>>> +} >>>>>> + >>>>>> +sub _get_connection_or_die { >>>>>> + my ($self) = @_; >>>>>> + >>>>>> + # clear the socket if the PID changes >>>>>> + if ( defined $self->{'_sock_pid'} && $self->{'_sock_pid'} != $$ ) { >>>>>> + undef $self->{'_sock_pid'}; >>>>>> + undef $self->{'_sock'}; >>>>>> + } >>>>>> + >>>>>> + $self->{'_sock_pid'} ||= $$; >>>>>> + $self->{'_sock'} ||= IO::Socket::INET->new( >>>>>> + 'PeerHost' => $self->{'_server_host'}, >>>>>> + 'PeerPort' => $self->{'_server_port'}, >>>>>> + 'Proto' => 'udp' >>>>>> + ) or die "Cannot connect to >>>>>> $self->{'_server_host'}:$self->{'_server_port'}: $@ $!"; >>>>>> + >>>>>> + return $self->{'_sock'}; >>>>>> +} >>>>>> + >>>>>> +sub _sign_msg { >>>>>> + my ( $self, $msg_ref ) = @_; >>>>>> + >>>>>> + $msg_ref->{'Sig'} = lc Digest::SHA::sha1_hex( >>>>>> + Digest::SHA::sha1( >>>>>> $self->_generate_packet_from_message($msg_ref) ) >>>>>> + ); >>>>>> + >>>>>> + return 1; >>>>>> +} >>>>>> + >>>>>> +sub _generate_packet_from_message { >>>>>> + my ( $self, $msg_ref ) = @_; >>>>>> + >>>>>> + return join( "\n", map { "$_: $msg_ref->{$_}" } grep { length >>>>>> $msg_ref->{$_} } @hash_order ); >>>>>> +} >>>>>> + >>>>>> +sub _generate_thread_id { >>>>>> + my $RAND_MAX = 2**16; >>>>>> + my $val = 0; >>>>>> + $val = int rand($RAND_MAX) while $val < 1024; >>>>>> + return $val; >>>>>> +} >>>>>> + >>>>>> +sub _get_user_pass_hash_key { >>>>>> + my ($self) = @_; >>>>>> + >>>>>> + return lc Digest::SHA::sha1_hex( $self->{'_username'} . ':' . >>>>>> $self->{'_password'} ); >>>>>> +} >>>>>> + >>>>>> +1; >>>>>> diff --git a/lib/Mail/SpamAssassin/Pyzor/Digest.pm >>>>>> b/lib/Mail/SpamAssassin/Pyzor/Digest.pm >>>>>> new file mode 100644 >>>>>> index 0000000..0e8a5ae >>>>>> --- /dev/null >>>>>> +++ b/lib/Mail/SpamAssassin/Pyzor/Digest.pm >>>>>> @@ -0,0 +1,103 @@ >>>>>> +package Mail::SpamAssassin::Pyzor::Digest; >>>>>> + >>>>>> +# Copyright 2018 cPanel, LLC. >>>>>> +# All rights reserved. >>>>>> +# http://cpanel.net >>>>>> +# >>>>>> +# <@LICENSE> >>>>>> +# Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> +# contributor license agreements. See the NOTICE file distributed with >>>>>> +# this work for additional information regarding copyright ownership. >>>>>> +# The ASF licenses this file to you under the Apache License, Version >>>>>> 2.0 >>>>>> +# (the "License"); you may not use this file except in compliance with >>>>>> +# the License. You may obtain a copy of the License at: >>>>>> +# >>>>>> +# http://www.apache.org/licenses/LICENSE-2.0 >>>>>> +# >>>>>> +# Unless required by applicable law or agreed to in writing, software >>>>>> +# distributed under the License is distributed on an "AS IS" BASIS, >>>>>> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>>> implied. >>>>>> +# See the License for the specific language governing permissions and >>>>>> +# limitations under the License. >>>>>> +# </@LICENSE> >>>>>> +# >>>>>> + >>>>>> +use strict; >>>>>> +use warnings; >>>>>> + >>>>>> +=encoding utf-8 >>>>>> + >>>>>> +=head1 NAME >>>>>> + >>>>>> +Mail::SpamAssassin::Pyzor::Digest >>>>>> + >>>>>> +=head1 SYNOPSIS >>>>>> + >>>>>> + my $digest = Mail::SpamAssassin::Pyzor::Digest::get( $mime_text ); >>>>>> + >>>>>> +=head1 DESCRIPTION >>>>>> + >>>>>> +A reimplementation of >>>>>> L<https://github.com/SpamExperts/pyzor/blob/master/pyzor/digest.py>. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +use Email::MIME (); >>>>>> + >>>>>> +use Mail::SpamAssassin::Pyzor::Digest::Pieces (); >>>>>> +use Digest::SHA qw(sha1_hex); >>>>>> + >>>>>> +our $VERSION = '0.03'; >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head1 FUNCTIONS >>>>>> + >>>>>> +=head2 $hex = get( $MSG ) >>>>>> + >>>>>> +This takes an email message in raw MIME text format (i.e., as saved in >>>>>> the >>>>>> +standard mbox format) and returns the message???s Pyzor digest in >>>>>> lower-case >>>>>> +hexadecimal. >>>>>> + >>>>>> +The output from this function should normally be identical to that of >>>>>> +the C<pyzor> script???s C<digest> command. It is suitable for use in >>>>>> +L<Mail::SpamAssassin::Pyzor::Client>???s request methods. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub get { >>>>>> + my ($text) = @_; >>>>>> + return Digest::SHA::sha1_hex( ${ _get_predigest( $text ) } ); >>>>>> +} >>>>>> + >>>>>> +# NB: This is called from the test. >>>>>> +sub _get_predigest { ## no critic qw(RequireArgUnpacking) >>>>>> + my ($msg_text_sr) = @_; >>>>>> + >>>>>> + my $parsed = Email::MIME->new($$msg_text_sr); >>>>>> + >>>>>> + my @lines; >>>>>> + >>>>>> + my $payloads_ar = >>>>>> Mail::SpamAssassin::Pyzor::Digest::Pieces::digest_payloads($parsed); >>>>>> + >>>>>> + for my $payload (@$payloads_ar) { >>>>>> + my @p_lines = >>>>>> Mail::SpamAssassin::Pyzor::Digest::Pieces::splitlines($payload); >>>>>> + for my $line (@p_lines) { >>>>>> + Mail::SpamAssassin::Pyzor::Digest::Pieces::normalize($line); >>>>>> + >>>>>> + next if >>>>>> !Mail::SpamAssassin::Pyzor::Digest::Pieces::should_handle_line($line); >>>>>> + >>>>>> + # Make sure we have an octet string. >>>>>> + utf8::encode($line) if utf8::is_utf8($line); >>>>>> + >>>>>> + push @lines, $line; >>>>>> + } >>>>>> + } >>>>>> + >>>>>> + my $digest_sr = >>>>>> Mail::SpamAssassin::Pyzor::Digest::Pieces::assemble_lines( \@lines ); >>>>>> + >>>>>> + return $digest_sr; >>>>>> +} >>>>>> + >>>>>> +1; >>>>>> diff --git a/lib/Mail/SpamAssassin/Pyzor/Digest/Pieces.pm >>>>>> b/lib/Mail/SpamAssassin/Pyzor/Digest/Pieces.pm >>>>>> new file mode 100644 >>>>>> index 0000000..522accd >>>>>> --- /dev/null >>>>>> +++ b/lib/Mail/SpamAssassin/Pyzor/Digest/Pieces.pm >>>>>> @@ -0,0 +1,301 @@ >>>>>> +package Mail::SpamAssassin::Pyzor::Digest::Pieces; >>>>>> + >>>>>> +# Copyright 2018 cPanel, LLC. >>>>>> +# All rights reserved. >>>>>> +# http://cpanel.net >>>>>> +# >>>>>> +# <@LICENSE> >>>>>> +# Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> +# contributor license agreements. See the NOTICE file distributed with >>>>>> +# this work for additional information regarding copyright ownership. >>>>>> +# The ASF licenses this file to you under the Apache License, Version >>>>>> 2.0 >>>>>> +# (the "License"); you may not use this file except in compliance with >>>>>> +# the License. You may obtain a copy of the License at: >>>>>> +# >>>>>> +# http://www.apache.org/licenses/LICENSE-2.0 >>>>>> +# >>>>>> +# Unless required by applicable law or agreed to in writing, software >>>>>> +# distributed under the License is distributed on an "AS IS" BASIS, >>>>>> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>>> implied. >>>>>> +# See the License for the specific language governing permissions and >>>>>> +# limitations under the License. >>>>>> +# </@LICENSE> >>>>>> +# >>>>>> + >>>>>> +use strict; >>>>>> +use warnings; >>>>>> + >>>>>> +=encoding utf-8 >>>>>> + >>>>>> +=head1 NAME >>>>>> + >>>>>> +Mail::SpamAssassin::Pyzor::Digest::Pieces >>>>>> + >>>>>> +=head1 DESCRIPTION >>>>>> + >>>>>> +This module houses backend logic for >>>>>> L<Mail::SpamAssassin::Pyzor::Digest>. >>>>>> + >>>>>> +It reimplements logic found in pyzor???s F<digest.py> module >>>>>> +(L<https://github.com/SpamExperts/pyzor/blob/master/pyzor/digest.py>). >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +use Email::MIME::ContentType (); >>>>>> +use Encode (); >>>>>> + >>>>>> +our $VERSION = '0.03'; >>>>>> + >>>>>> +# each tuple is [ offset, length ] >>>>>> +use constant _HASH_SPEC => ( [ 20, 3 ], [ 60, 3 ] ); >>>>>> + >>>>>> +use constant { >>>>>> + _MIN_LINE_LENGTH => 8, >>>>>> + >>>>>> + _ATOMIC_NUM_LINES => 4, >>>>>> +}; >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head1 FUNCTIONS >>>>>> + >>>>>> +=head2 $strings_ar = digest_payloads( $EMAIL_MIME ) >>>>>> + >>>>>> +This imitates the corresponding object method in F<digest.py>. >>>>>> +It returns a reference to an array of strings. Each string can be either >>>>>> +a byte string or a character string (e.g., UTF-8 decoded). >>>>>> + >>>>>> +NB: RFC 2822 stipulates that message bodies should use CRLF >>>>>> +line breaks, not plain LF (nor plain CR). L<Email::MIME::Encodings> >>>>>> +will thus convert any plain CRs in a quoted-printable message >>>>>> +body into CRLF. Python, though, doesn???t do this, so the output of >>>>>> +our implementation of C<digest_payloads()> diverges from that of the >>>>>> Python >>>>>> +original. It doesn???t ultimately make a difference since the >>>>>> line-ending >>>>>> +whitespace gets trimmed regardless, but it???s necessary to factor in >>>>>> when >>>>>> +comparing the output of our implementation with the Python output. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub digest_payloads { >>>>>> + my ($parsed) = @_; >>>>>> + >>>>>> + my @subparts = $parsed->subparts(); >>>>>> + >>>>>> + my @payloads; >>>>>> + >>>>>> + if (@subparts) { >>>>>> + @payloads = map { @{ digest_payloads($_) } } >>>>>> $parsed->subparts(); >>>>>> + } >>>>>> + else { >>>>>> + my ( $main_type, $subtype, $encoding, $encode_check ) = >>>>>> parse_content_type( $parsed->content_type() ); >>>>>> + >>>>>> + my $payload; >>>>>> + >>>>>> + if ( $main_type eq 'text' ) { >>>>>> + >>>>>> + # Decode transfer encoding, but leave us as a byte string. >>>>>> + # Note that this is where Email::MIME converts plain LF to >>>>>> CRLF. >>>>>> + $payload = $parsed->body(); >>>>>> + >>>>>> + # This does the actual character decoding (i.e., >>>>>> ???charset???). >>>>>> + $payload = Encode::decode( $encoding, $payload, >>>>>> $encode_check ); >>>>>> + >>>>>> + if ( $subtype eq 'html' ) { >>>>>> + require Mail::SpamAssassin::Pyzor::Digest::StripHtml; >>>>>> + $payload = >>>>>> Mail::SpamAssassin::Pyzor::Digest::StripHtml::strip($payload); >>>>>> + } >>>>>> + } >>>>>> + else { >>>>>> + >>>>>> + # This does no decoding, even of, e.g., quoted-printable or >>>>>> base64. >>>>>> + $payload = $parsed->body_raw(); >>>>>> + } >>>>>> + >>>>>> + push @payloads, $payload; >>>>>> + } >>>>>> + >>>>>> + return \@payloads; >>>>>> +} >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head2 normalize( $STRING ) >>>>>> + >>>>>> +This imitates the corresponding object method in F<digest.py>. >>>>>> +It modifies C<$STRING> in-place. >>>>>> + >>>>>> +As with the original implementation, if C<$STRING> contains (decoded) >>>>>> +Unicode characters, those characters will be parsed accordingly. So: >>>>>> + >>>>>> + $str = "123\xc2\xa0"; # [ c2 a0 ] == \u00a0, non-breaking space >>>>>> + >>>>>> + normalize($str); >>>>>> + >>>>>> +The above will leave C<$str> alone, but this: >>>>>> + >>>>>> + utf8::decode($str); >>>>>> + >>>>>> + normalize($str); >>>>>> + >>>>>> +??? will trim off the last two bytes from C<$str>. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub normalize { ## no critic qw( Subroutines::RequireArgUnpacking ) >>>>>> + >>>>>> + # NULs are bad, mm-kay? >>>>>> + $_[0] =~ tr<\0><>d; >>>>>> + >>>>>> + # NB: Python???s \s without re.UNICODE is the same as Perl???s \s >>>>>> + # with the /a modifier. >>>>>> + # >>>>>> + # https://docs.python.org/2/library/re.html >>>>>> + # https://perldoc.perl.org/perlrecharclass.html#Backslash-sequences >>>>>> + >>>>>> + # Python: re.compile(r'\S{10,}') >>>>>> + $_[0] =~ s<\S{10,}><>ag; >>>>>> + >>>>>> + # Python: re.compile(r'\S+@\S+') >>>>>> + $_[0] =~ s<\S+ @ \S+><>agx; >>>>>> + >>>>>> + # Python: re.compile(r'[a-z]+:\S+', re.IGNORECASE) >>>>>> + $_[0] =~ s<[a-zA-Z]+ : \S+><>agx; >>>>>> + >>>>>> + # (from digest.py ???) >>>>>> + # Make sure we do the whitespace last because some of the previous >>>>>> + # patterns rely on whitespace. >>>>>> + $_[0] =~ tr< \x09-\x0d><>d; >>>>>> + >>>>>> + # This is fun. digest.py???s normalize() does a non-UNICODE >>>>>> whitespace >>>>>> + # strip, then calls strip() on the string, which *will* strip >>>>>> Unicode >>>>>> + # whitespace from the ends. >>>>>> + $_[0] =~ s<\A\s+><>; >>>>>> + $_[0] =~ s<\s+\z><>; >>>>>> + >>>>>> + return; >>>>>> +} >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head2 $yn = should_handle_line( $STRING ) >>>>>> + >>>>>> +This imitates the corresponding object method in F<digest.py>. >>>>>> +It returns a boolean. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub should_handle_line { >>>>>> + return $_[0] && length( $_[0] ) >= _MIN_LINE_LENGTH(); >>>>>> +} >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head2 $sr = assemble_lines( \@LINES ) >>>>>> + >>>>>> +This assembles a string buffer out of @LINES. The string is the buffer >>>>>> +of octets that will be hashed to produce the message digest. >>>>>> + >>>>>> +Each member of @LINES is expected to be an B<octet string>, not a >>>>>> +character string. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub assemble_lines { >>>>>> + my ($lines_ar) = @_; >>>>>> + >>>>>> + if ( @$lines_ar <= _ATOMIC_NUM_LINES() ) { >>>>>> + >>>>>> + # cf. handle_atomic() in digest.py >>>>>> + return \join( q<>, @$lines_ar ); >>>>>> + } >>>>>> + >>>>>> + >>>>>> #---------------------------------------------------------------------- >>>>>> + # cf. handle_atomic() in digest.py >>>>>> + >>>>>> + my $str = q<>; >>>>>> + >>>>>> + for my $ofs_len ( _HASH_SPEC() ) { >>>>>> + my ( $offset, $length ) = @$ofs_len; >>>>>> + >>>>>> + for my $i ( 0 .. ( $length - 1 ) ) { >>>>>> + my $idx = int( $offset * @$lines_ar / 100 ) + $i; >>>>>> + >>>>>> + next if !defined $lines_ar->[$idx]; >>>>>> + >>>>>> + $str .= $lines_ar->[$idx]; >>>>>> + } >>>>>> + } >>>>>> + >>>>>> + return \$str; >>>>>> +} >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head2 ($main, $sub, $encoding, $checkval) = parse_content_type( >>>>>> $CONTENT_TYPE ) >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +use constant _QUOTED_PRINTABLE_NAMES => ( >>>>>> + "quopri-codec", >>>>>> + "quopri", >>>>>> + "quoted-printable", >>>>>> + "quotedprintable", >>>>>> +); >>>>>> + >>>>>> +# Make Encode::decode() ignore anything that doesn???t fit the >>>>>> +# given encoding. >>>>>> +use constant _encode_check_ignore => q<>; >>>>>> + >>>>>> +sub parse_content_type { >>>>>> + my ($content_type) = @_; >>>>>> + >>>>>> + $Email::MIME::ContentType::STRICT_PARAMS = 0; >>>>>> + my $ct_parse = Email::MIME::ContentType::parse_content_type( >>>>>> + $content_type, >>>>>> + ); >>>>>> + >>>>>> + my $main = $ct_parse->{'type'} || q<>; >>>>>> + my $sub = $ct_parse->{'subtype'} || q<>; >>>>>> + >>>>>> + my $encoding = $ct_parse->{'attributes'}{'charset'}; >>>>>> + >>>>>> + my $checkval; >>>>>> + >>>>>> + if ($encoding) { >>>>>> + >>>>>> + # Lower-case everything, convert underscore to dash, and remove >>>>>> NUL. >>>>>> + $encoding =~ tr<A-Z_\0><a-z->d; >>>>>> + >>>>>> + # Apparently pyzor accommodates messages that put the transfer >>>>>> + # encoding in the Content-Type. >>>>>> + if ( grep { $_ eq $encoding } _QUOTED_PRINTABLE_NAMES() ) { >>>>>> + $checkval = Encode::FB_CROAK(); >>>>>> + } >>>>>> + } >>>>>> + else { >>>>>> + $encoding = 'ascii'; >>>>>> + } >>>>>> + >>>>>> + # Match Python .decode()???s 'ignore' behavior >>>>>> + $checkval ||= \&_encode_check_ignore; >>>>>> + >>>>>> + return ( $main, $sub, $encoding, $checkval ); >>>>>> +} >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head2 @lines = splitlines( $TEXT ) >>>>>> + >>>>>> +Imitates C<str.splitlines()>. (cf. C<pydoc str>) >>>>>> + >>>>>> +Returns a plain list in list context. Returns the number of >>>>>> +items to be returned in scalar context. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub splitlines { >>>>>> + return split m<\r\n?|\n>, $_[0]; >>>>>> +} >>>>>> + >>>>>> +1; >>>>>> diff --git a/lib/Mail/SpamAssassin/Pyzor/Digest/StripHtml.pm >>>>>> b/lib/Mail/SpamAssassin/Pyzor/Digest/StripHtml.pm >>>>>> new file mode 100644 >>>>>> index 0000000..2617b4a >>>>>> --- /dev/null >>>>>> +++ b/lib/Mail/SpamAssassin/Pyzor/Digest/StripHtml.pm >>>>>> @@ -0,0 +1,177 @@ >>>>>> +package Mail::SpamAssassin::Pyzor::Digest::StripHtml; >>>>>> + >>>>>> +# Copyright 2018 cPanel, LLC. >>>>>> +# All rights reserved. >>>>>> +# http://cpanel.net >>>>>> +# >>>>>> +# <@LICENSE> >>>>>> +# Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> +# contributor license agreements. See the NOTICE file distributed with >>>>>> +# this work for additional information regarding copyright ownership. >>>>>> +# The ASF licenses this file to you under the Apache License, Version >>>>>> 2.0 >>>>>> +# (the "License"); you may not use this file except in compliance with >>>>>> +# the License. You may obtain a copy of the License at: >>>>>> +# >>>>>> +# http://www.apache.org/licenses/LICENSE-2.0 >>>>>> +# >>>>>> +# Unless required by applicable law or agreed to in writing, software >>>>>> +# distributed under the License is distributed on an "AS IS" BASIS, >>>>>> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>>> implied. >>>>>> +# See the License for the specific language governing permissions and >>>>>> +# limitations under the License. >>>>>> +# </@LICENSE> >>>>>> +# >>>>>> + >>>>>> +use strict; >>>>>> +use warnings; >>>>>> + >>>>>> +=encoding utf-8 >>>>>> + >>>>>> +=head1 NAME >>>>>> + >>>>>> +Mail::SpamAssassin::Pyzor::Digest::StripHtml >>>>>> + >>>>>> +=head1 SYNOPSIS >>>>>> + >>>>>> + my $stripped = >>>>>> Mail::SpamAssassin::Pyzor::Digest::StripHtml::strip($html); >>>>>> + >>>>>> +=head1 DESCRIPTION >>>>>> + >>>>>> +This module attempts to duplicate pyzor???s HTML-stripping logic. >>>>>> + >>>>>> +=head1 ACCURACY >>>>>> + >>>>>> +This library cannot achieve 100%, bug-for-bug parity with pyzor >>>>>> +because to do so would require duplicating Python???s own HTML parsing >>>>>> +library. Since that library???s output has changed over time, and those >>>>>> +changes in turn affect pyzor, it???s literally impossible to arrive at >>>>>> +a single, fully-compatible reimplementation. >>>>>> + >>>>>> +That said, all known divergences between pyzor and this library involve >>>>>> +invalid HTML as input. >>>>>> + >>>>>> +Please open bug reports for any divergences you identify, particularly >>>>>> +if the input is valid HTML. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +use HTML::Parser (); >>>>>> + >>>>>> +our $VERSION = '0.03'; >>>>>> + >>>>>> +#---------------------------------------------------------------------- >>>>>> + >>>>>> +=head1 FUNCTIONS >>>>>> + >>>>>> +=head2 $stripped = strip( $HTML ) >>>>>> + >>>>>> +Give it some HTML, and it???ll give back the stripped text. >>>>>> + >>>>>> +In B<general>, the stripping consists of removing tags as well as >>>>>> +C<E<lt>scriptE<gt>> and C<E<lt>styleE<gt>> elements; however, it also >>>>>> +removes HTML entities. >>>>>> + >>>>>> +This tries very hard to duplicate pyzor???s behavior with invalid HTML. >>>>>> + >>>>>> +=cut >>>>>> + >>>>>> +sub strip { >>>>>> + my ($html) = @_; >>>>>> + >>>>>> + $html =~ s<\A\s+><>; >>>>>> + $html =~ s<\s+\z><>; >>>>>> + >>>>>> + my $p = HTML::Parser->new( api_version => 3 ); >>>>>> + >>>>>> + my @pieces; >>>>>> + >>>>>> + my $accumulate = 1; >>>>>> + >>>>>> + $p->handler( >>>>>> + start => sub { >>>>>> + my ($tagname) = @_; >>>>>> + >>>>>> + $accumulate = 0 if $tagname eq 'script'; >>>>>> + $accumulate = 0 if $tagname eq 'style'; >>>>>> + >>>>>> + return; >>>>>> + }, >>>>>> + 'tagname', >>>>>> + ); >>>>>> + >>>>>> + $p->handler( >>>>>> + end => sub { >>>>>> + $accumulate = 1; >>>>>> + return; >>>>>> + } >>>>>> + ); >>>>>> + >>>>>> + $p->handler( >>>>>> + text => sub { >>>>>> + my ($copy) = @_; >>>>>> + >>>>>> + return if !$accumulate; >>>>>> + >>>>>> + # pyzor???s HTML parser discards HTML entities. On top of >>>>>> that, >>>>>> + # we need to match, as closely as possible, pyzor???s >>>>>> handling of >>>>>> + # invalid HTML entities ??? which is a function of >>>>>> Python???s >>>>>> + # standard HTML parsing library. This will probably never be >>>>>> + # fully compatible with the pyzor, but we can get it close. >>>>>> + >>>>>> + # The original is: >>>>>> + # >>>>>> + # re.compile('&#(?:[0-9]+|[xX][0-9a-fA-F]+)[^0-9a-fA-F]') >>>>>> + # >>>>>> + # The parsing loop then ???backs up??? one byte if the last >>>>>> + # character isn???t a ???;???. We use a look-ahead >>>>>> assertion to >>>>>> + # mimic that behavior. >>>>>> + $copy =~ s<\&\# (?:[0-9]+ | [xX][0-9a-fA-F]+) (?: ; | \z | >>>>>> (?=[^0-9a-fA-F]) )>< >gx; >>>>>> + >>>>>> + # The original is: >>>>>> + # >>>>>> + # re.compile('&([a-zA-Z][-.a-zA-Z0-9]*)[^a-zA-Z0-9]') >>>>>> + # >>>>>> + # We again use a look-ahead assertion to mimic Python. >>>>>> + $copy =~ s<\& [a-zA-Z] [-.a-zA-Z0-9]* (?: ; | \z | >>>>>> (?=[^a-zA-Z0-9]) )>< >gx; >>>>>> + >>>>>> + # Python???s HTMLParser aborts its parsing loop when it >>>>>> encounters >>>>>> + # an invalid numeric reference. >>>>>> + $copy =~ s<\&\# >>>>>> + (?: >>>>>> + [^0-9xX] # anything but the expected first >>>>>> char >>>>>> + | >>>>>> + [0-9]+[a-fA-F] # hex within decimal >>>>>> + | >>>>>> + [xX][^0-9a-fA-F] >>>>>> + ) >>>>>> + (.*) >>>>>> + >< >>>>>> + ( -1 == index($1, ';') ) ? q<> : '&#' >>>>>> + >exs; >>>>>> + >>>>>> + # Python???s HTMLParser treats invalid entities as >>>>>> incomplete >>>>>> + $copy =~ s<(\&\#?)><$1 >gx; >>>>>> + >>>>>> + $copy =~ s<\A\s+><>; >>>>>> + $copy =~ s<\s+\z><>; >>>>>> + >>>>>> + push @pieces, \$copy if length $copy; >>>>>> + }, >>>>>> + 'text,tagname', >>>>>> + ); >>>>>> + >>>>>> + $p->parse($html); >>>>>> + $p->eof(); >>>>>> + >>>>>> + my $payload = join( q< >, map { $$_ } @pieces ); >>>>>> + >>>>>> + # Convert all sequences of whitespace OTHER THAN non-breaking >>>>>> spaces to >>>>>> + # plain spaces. >>>>>> + $payload =~ s<[^\S\x{a0}]+>< >g; >>>>>> + >>>>>> + return $payload; >>>>>> +} >>>>>> + >>>>>> +1; >>>>>> diff --git a/t/pyzor.t b/t/pyzor.t >>>>>> index 891f38d..e4ef83f 100755 >>>>>> --- a/t/pyzor.t >>>>>> +++ b/t/pyzor.t >>>>>> @@ -3,12 +3,9 @@ >>>>>> use lib '.'; use lib 't'; >>>>>> use SATest; sa_t_init("pyzor"); >>>>>> -use constant HAS_PYZOR => eval { $_ = untaint_cmd("which pyzor"); >>>>>> chomp; -x }; >>>>>> - >>>>>> use Test::More; >>>>>> plan skip_all => "Net tests disabled" unless >>>>>> conf_bool('run_net_tests'); >>>>>> -plan skip_all => "Pyzor executable not found in path" unless HAS_PYZOR; >>>>>> -plan tests => 8; >>>>>> +plan tests => 5; >>>>>> diag('Note: Failures may not be an SpamAssassin bug, as Pyzor tests >>>>>> can fail due to problems with the Pyzor servers.'); >>>>>> @@ -30,7 +27,7 @@ tstprefs (" >>>>>> sarun ("-t < data/spam/pyzor", \&patterns_run_cb); >>>>>> ok_all_patterns(); >>>>>> # Same with fork >>>>>> -sarun ("--cf='pyzor_fork 1' -t < data/spam/pyzor", \&patterns_run_cb); >>>>>> +sarun ("-t < data/spam/pyzor", \&patterns_run_cb); >>>>>> ok_all_patterns(); >>>>>> #TESTING FOR HAM >>>>>> @@ -44,7 +41,3 @@ ok_all_patterns(); >>>>>> sarun ("-D pyzor -t < data/nice/001 2>&1", \&patterns_run_cb); >>>>>> ok_all_patterns(); >>>>>> -# same with fork >>>>>> -sarun ("-D pyzor --cf='pyzor_fork 1' -t < data/nice/001 2>&1", >>>>>> \&patterns_run_cb); >>>>>> -ok_all_patterns(); >>>>>> - >>>>> >>> >> -- >> Kevin A. McGrail >> kmcgr...@apache.org >> >> Member, Apache Software Foundation >> Chair Emeritus Apache SpamAssassin Project >> https://www.linkedin.com/in/kmcgrail - 703.798.0171
OpenPGP_signature
Description: OpenPGP digital signature
