Send inn-committers mailing list submissions to
inn-committers@lists.isc.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/inn-committers
or, via email, send a message with subject or body 'help' to
inn-committers-requ...@lists.isc.org
You can reach the person managing the list at
inn-committers-ow...@lists.isc.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of inn-committers digest..."
Today's Topics:
1. INN commit: trunk (3 files) (INN Commit)
2. INN commit: trunk/frontends (pullnews.in) (INN Commit)
3. INN commit: trunk (doc/pod/news.pod frontends/pullnews.in)
(INN Commit)
----------------------------------------------------------------------
Message: 1
Date: Fri, 22 Aug 2014 14:38:40 -0700 (PDT)
From: INN Commit <r...@isc.org>
To: inn-committ...@isc.org
Subject: INN commit: trunk (3 files)
Message-ID: <20140822213840.e534467...@hope.eyrie.org>
Date: Friday, August 22, 2014 @ 14:38:40
Author: iulius
Revision: 9652
pullnews: remove headers matching (or not) a given regexp
Enable the -m flag to remove headers matching (or not) a given regexp.
Thanks to Geraint Edwards for the patch.
Modified:
trunk/doc/pod/news.pod
trunk/doc/pod/pullnews.pod
trunk/frontends/pullnews.in
-----------------------+
doc/pod/news.pod | 6 ++++++
doc/pod/pullnews.pod | 5 ++++-
frontends/pullnews.in | 34 ++++++++++++++++++++++++----------
3 files changed, 34 insertions(+), 11 deletions(-)
Modified: doc/pod/news.pod
===================================================================
--- doc/pod/news.pod 2014-08-09 10:32:07 UTC (rev 9651)
+++ doc/pod/news.pod 2014-08-22 21:38:40 UTC (rev 9652)
@@ -183,6 +183,12 @@
not all processes die within timeout). Thanks to Lauri Tirkkonen for
the patch.
+=item *
+
+Several improvements have been contributed to B<pullnews> by Geraint
+Edwards: the B<-m> flag now permits to remove headers matching (or not)
+a given regexp.
+
=back
=head1 Changes in 2.5.4
Modified: doc/pod/pullnews.pod
===================================================================
--- doc/pod/pullnews.pod 2014-08-09 10:32:07 UTC (rev 9651)
+++ doc/pod/pullnews.pod 2014-08-22 21:38:40 UTC (rev 9652)
@@ -115,10 +115,13 @@
whitespace-separated tuples (each tuple being a colon-separated header and
regular expression). For instance:
- -m "Hdr1:regexp1 !Hdr2:regexp2"
+ -m "Hdr1:regexp1 !Hdr2:regexp2 #Hdr3:regexp3 !#Hdr4:regexp4"
specifies that the article will be passed only if the C<Hdr1:> header
matches C<regexp1> and the C<Hdr2:> header does not match C<regexp2>.
+Besides, if the C<Hdr3:> header matches C<regexp3>, that header is
+removed; and if the C<Hdr4:> header does not match C<regexp4>, that
+header is removed.
=item B<-M> I<num>
Modified: frontends/pullnews.in
===================================================================
--- frontends/pullnews.in 2014-08-09 10:32:07 UTC (rev 9651)
+++ frontends/pullnews.in 2014-08-22 21:38:40 UTC (rev 9652)
@@ -160,10 +160,13 @@
-l logfile log progress/stats to logfile (default is stdout).
- -m 'Hdr1:regexp1 !Hdr2:regexp2 ...'
+ -m 'Hdr1:regexp1 !Hdr2:regexp2 #Hdr3:regexp3 !#Hdr4:regexp4 ...'
feed article only if:
- the Hdr1: header matches regexp1
- and the Hdr2: header does not match regexp2.
+ the Hdr1: header matches regexp1;
+ and the Hdr2: header does not match regexp2;
+ also, process the message thus:
+ if the Hdr3: header matches regexp3, remove that header;
+ if the Hdr4: header does not match regexp4, remove it.
-M num maximum number of articles (per group) to process before
bailing out.
@@ -730,22 +733,33 @@
my ($hdr_un, $val_un) = split(':', $unfolded_art_hdr, 2);
$val_un = '' if not defined $val_un;
$val_un =~ s/^\s*//;
+ my $remove_hdr = 0;
for my $tuple_match (@hdr_to_match) {
my ($hdr_m, $val_m) = split(':', $tuple_match, 2);
my $negate_h = ($hdr_m =~ s/^!//);
+ my $remove_h = ($hdr_m =~ s/^#//);
next if lc($hdr_un) ne lc($hdr_m);
- $m_found_hdrs{lc($hdr_m)} = 1;
+ $m_found_hdrs{lc($hdr_m)} = 1 if not $remove_h;
if ($negate_h) {
if ($val_un =~ /$val_m/i) {
print LOG "\tDEBUGGING $i\t-- $hdr_un
[$val_un]\n" if $debug >= 2;
- $match_all_hdrs = 0;
+ if (not $remove_h) {
+ $match_all_hdrs = 0;
+ }
+ } elsif ($remove_h) {
+ $remove_hdr = 1;
}
} elsif (not $val_un =~ /$val_m/i) {
print LOG "\tDEBUGGING $i\t++ $hdr_un [$val_un]\n"
if $debug >= 2;
- $match_all_hdrs = 0;
+ if (not $remove_h) {
+ $match_all_hdrs = 0;
+ }
+ } elsif ($remove_h) {
+ $remove_hdr = 1;
}
last if not $match_all_hdrs;
}
+ push @header_nums_to_go, $idx if $remove_hdr;
}
if (grep { $curr_hdr eq $_ } split(':', $skip_headers)) {
@@ -781,12 +795,12 @@
$headers = 0 if $article->[$idx] eq "\n";
}
- if (@hdr_to_match and (not $match_all_hdrs or @hdr_to_match !=
scalar(keys %m_found_hdrs))) {
- print LOG "\tDEBUGGING $i\thdr_skip_art $i\n" if $debug >= 2;
- $skip_due_to_hdrs = 1;
+ if (@hdr_to_match and (not $match_all_hdrs
+ or scalar(grep { ! /^!?#/ } @hdr_to_match) != keys
%m_found_hdrs)) {
+ $skip_due_to_hdrs = 2;
}
while (@header_nums_to_go) {
- my $idx = pop @header_nums_to_go; # Start from last.
+ my $idx = pop @header_nums_to_go; # Start from last, so
numbers are not affected.
my $cut = join("\n\t", splice(@{$article}, $idx, 1));
$tx_len -= length($cut);
print LOG "\tDEBUGGING $i\tcut1 $cut" if $debug >= 2;
------------------------------
Message: 2
Date: Fri, 22 Aug 2014 15:18:28 -0700 (PDT)
From: INN Commit <r...@isc.org>
To: inn-committ...@isc.org
Subject: INN commit: trunk/frontends (pullnews.in)
Message-ID: <20140822221828.dfccd67...@hope.eyrie.org>
Date: Friday, August 22, 2014 @ 15:18:28
Author: iulius
Revision: 9653
pullnews: improve wording
* When pullnews runs for the first time against a newsgroup, say "never"
instead of January, 1st 1970 as the last run date.
* Improve spaces, uppercase characters and singular forms when 1 article
is retrieved.
* Update the config file even when the group is empty.
Modified:
trunk/frontends/pullnews.in
-------------+
pullnews.in | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
Modified: pullnews.in
===================================================================
--- pullnews.in 2014-08-22 21:38:40 UTC (rev 9652)
+++ pullnews.in 2014-08-22 22:18:28 UTC (rev 9653)
@@ -625,9 +625,10 @@
if (not $quiet) {
printf LOG "\n%s:\n", $name;
- printf LOG "\tlast checked: %s\n", scalar(localtime($prevDate));
- printf LOG "\t%d articles available. First %d Last %d\n",
- $narticles, $first, $last;
+ printf LOG "\tLast checked: %s\n",
+ $prevDate ? scalar(localtime($prevDate)) : "never";
+ printf LOG "\t%d article%s available (first %d, last %d)\n",
+ $narticles, $narticles != 1 ? "s" : "", $first, $last;
}
if (defined $watermark) {
printf LOG "\tOur previous highest: %d\n", $prevHigh if not $quiet;
@@ -641,6 +642,8 @@
return 0 if ! $name;
if ($narticles == 0) {
print LOG " (nothing to get)\n" unless $quiet;
+ # Just update the time; keep the last known high watermark.
+ $shash->{$group} = [ time, $high ];
return 1;
}
------------------------------
Message: 3
Date: Sat, 23 Aug 2014 01:25:48 -0700 (PDT)
From: INN Commit <r...@isc.org>
To: inn-committ...@isc.org
Subject: INN commit: trunk (doc/pod/news.pod frontends/pullnews.in)
Message-ID: <20140823082548.6926067...@hope.eyrie.org>
Date: Saturday, August 23, 2014 @ 01:25:48
Author: iulius
Revision: 9654
pullnews: bug fix to rnews when -O; improved rnews reporting
Thanks to Geraint Edwards for the patch.
Modified:
trunk/doc/pod/news.pod
trunk/frontends/pullnews.in
-----------------------+
doc/pod/news.pod | 2 +-
frontends/pullnews.in | 28 +++++++++++++++++++++-------
2 files changed, 22 insertions(+), 8 deletions(-)
Modified: doc/pod/news.pod
===================================================================
--- doc/pod/news.pod 2014-08-22 22:18:28 UTC (rev 9653)
+++ doc/pod/news.pod 2014-08-23 08:25:48 UTC (rev 9654)
@@ -187,7 +187,7 @@
Several improvements have been contributed to B<pullnews> by Geraint
Edwards: the B<-m> flag now permits to remove headers matching (or not)
-a given regexp.
+a given regexp, and B<rnews> reporting is improved.
=back
Modified: frontends/pullnews.in
===================================================================
--- frontends/pullnews.in 2014-08-22 22:18:28 UTC (rev 9653)
+++ frontends/pullnews.in 2014-08-23 08:25:48 UTC (rev 9654)
@@ -9,6 +9,13 @@
# RCSId: $Id$
#
# History:
+# Full changelog can be found in the Subversion repository of the
+# INN project. Major changes are:
+#
+# January 2010: Geraint A. Edwards
+# enable -m to remove headers matching (or not) a given regexp;
+# minor bug fix to rnews when -O; improved rnews reporting.
+#
# December 2008: Matija Nalis added -O (optimized mode, checking
# whether the downstream server already has the article to
download).
# Bug fixes too.
@@ -114,9 +121,9 @@
}
$usage =~ s!.*/!!;
-$usage .= " [ -hnqRx -b fraction -c config -C width -d level
+$usage .= " [ -hnOqRx -b fraction -c config -C width -d level
-f fraction -F fakehop -g groups -G newsgroups -H headers
- -k checkpt -l logfile -m header_pats -M num -N num -O
+ -k checkpt -l logfile -m header_pats -M num -N num
-p port -P hop_limit -Q level -r file -s host[:port] -S num
-t retries -T seconds -w num -z num -Z num ]
[ upstream_host ... ]
@@ -306,6 +313,8 @@
open(RNEWS, ">$rnews") ||
die "can't open rnews-format output: $rnews: $!\n";
}
+ $info{'rnews'}->{bytes} = 0;
+ $info{'rnews'}->{fed} = 0;
}
open(LOG, $logFile) || die "can't open logfile ($logFile)!: $!\n";
@@ -449,11 +458,11 @@
}}
if (!$upstream) {
- print LOG "failed.\n" unless $quiet;
+ print LOG " failed.\n" unless $quiet;
warn "can't connect to upstream server $server: $!\n";
next;
} else {
- print LOG "done.\n" unless $quiet;
+ print LOG " done.\n" unless $quiet;
}
if ($username && !$upstream->authinfo($username, $passwd)) {
@@ -515,6 +524,9 @@
if ($quiet) {
printf LOG localtime() . " [$$] %d article%s to $localServer\n",
$sum, ($sum != 1 ? "s" : "");
+ } elsif ($rnews) {
+ printf LOG "\n%d article%s written to $rnews\n",
+ $sum, ($sum != 1 ? "s were" : " was");
} else {
printf LOG "\n%d article%s offered to server on $localServer\n",
$sum, ($sum != 1 ? "s were" : " was");
@@ -524,7 +536,7 @@
if ($quiet) {
print LOG localtime() . " [$$] $ltotal ok, $reftotal ref, $rejtotal
rej\n";
- } else {
+ } elsif (not $rnews) {
printf LOG "%d article%s accepted\n",
$ltotal, ($ltotal != 1 ? "s were" : " was")
if ($ltotal != 0);
@@ -682,7 +694,7 @@
my $org_code = $fromServer->code();
# Continue if the article exists on the upstream server.
- if ($org_code == 223) {
+ if ($org_code == 223 and not $rnews) {
my $new_msgid = $toServer->nntpstat($org_msgid);
my $new_code = $toServer->code();
print LOG "\tDEBUGGING $i\t$org_msgid ($org_code) =>
$new_code\n" if $debug >= 3;
@@ -839,10 +851,12 @@
printf RNEWS "#! rnews %d\n", $tx_len;
map { print RNEWS $_ } @{$article};
print LOG "+" unless $quiet;
+ $fed{$group}++;
+ $info{'rnews'}->{fed}++;
+ $info{fed}++;
} else {
if ($no_op) {
print "Would offer $msgid\n";
-
} elsif ($reader and not $toServer->post($article)) {
# 240 article posted ok
# 340 send article to be posted. End with
<CR-LF>.<CR-LF>
------------------------------
_______________________________________________
inn-committers mailing list
inn-committers@lists.isc.org
https://lists.isc.org/mailman/listinfo/inn-committers
End of inn-committers Digest, Vol 66, Issue 2
*********************************************
