Send inn-workers mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/inn-workers
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of inn-workers digest..."
Today's Topics:
1. Re: tradspool tooken weirdness (Christoph Biedl)
2. [PATCH 0/4] Improve delayer (Christoph Biedl)
3. [PATCH 1/4] delayer: Code cleanup (Christoph Biedl)
4. [PATCH 2/4] delayer: Add documentation (Christoph Biedl)
5. [PATCH 3/4] delayer: Introduce options for buffering and
persistence (Christoph Biedl)
----------------------------------------------------------------------
Message: 1
Date: Mon, 15 Jan 2024 21:05:50 +0100
From: Christoph Biedl <[email protected]>
To: [email protected]
Subject: Re: tradspool tooken weirdness
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Julien ?LIE wrote...
> Hi Christoph,
>
> All right, it would indeed be better to have a homogenized syntax for
> tradspool tokens between 32 and 64-bit archs.
>
> The only drawback I see if we enforce uint32_t instead of unsigned long, is
> that it is not in the path towards supporting 64-bit article numbers (but
> well, that's another story and amount of work globally in INN). At least it
> would make consistent the generation of 32-bit article numbers in tradspool
> tokens, but while we're at doing a migration, why not use uint64_t instead
> for article numbers? (group numbers may be kept as uint32_t)
Heh, was thinking about 64bit article numbers as well ...
Here, I am slightly leaning towards doing this, or at least to open the
path for it.
Major concern is "Is this really needed?". The highest article number I
have here has seven digits (in control.cancel, no surprise), and the
last number reset was at least 25 years ago. So hitting the 32 bit limit
is not going to happen any time soon.
On the other hand, there's the main lesson from y2k and upcoming y2038:
Software will be used for way longer times and at way bigger scale than
initially anticipated - therefore adapt early. Also, today's systems are
really powerful enough to handle the additional computational overhead.
As a little bonus and if I assess the situation correctly, that would
move the job of token conversion from 64bit to 32bit systems, and no
doubt their number is small, and shrinking.
Related, has anybody already tested whether clients can handle >32bit
article numbers? I fear the worst ...
> > * Extra code that deals with these invalid tokens by picking the article
> > number from the place where it is. That the transition I was talking
> > about.
> >
> > * Additionally, the next news.daily could repair these invalid tokens as
> > well.
> > Pro: The handling of invalid tokens could be removed again faily soon,
> > say after the next major release.
>
> Seems like the better way. When news.daily regenerates the history file,
> these tokens can indeed be fixed.
Now I feel the temptation to implement that but I even haven't sent out
the patches I had promised last month ...
Christoph
------------------------------
Message: 2
Date: Mon, 15 Jan 2024 18:00:00 +0100
From: Christoph Biedl <[email protected]>
To: [email protected]
Subject: [PATCH 0/4] Improve delayer
Message-ID: <[email protected]>
Hello,
a while ago I re-discovered the delayer as a way to increase efficiency
of NoCeM notice handling: By deliberately delaying articles from the
well-known spam site, the NoCeMs will arrive first, blocking the spam's
Message-IDs. As a result, users will not see the spam at all.
However, I found this little program could need some love, so there's
some code cleanup, documentation, and a bugfix. And new features, most
notably persistence for the lines not sent yet.
Aside, and always: Feel free to apply code formatter. Also, as I'm not
a native English speaker, my wording might be weird or misleading. In
that case, just replace it with something that seems more appropriate.
Christoph Biedl (4):
delayer: Code cleanup
delayer: Add documentation
delayer: Introduce options for buffering and persistence
delayer: Handle drained queue properly
contrib/README | 2 +-
contrib/delayer.in | 211 ++++++++++++++++++++++++++++++++++++++-------
2 files changed, 179 insertions(+), 34 deletions(-)
--
2.39.2
------------------------------
Message: 3
Date: Mon, 15 Jan 2024 18:00:00 +0100
From: Christoph Biedl <[email protected]>
To: [email protected]
Subject: [PATCH 1/4] delayer: Code cleanup
Message-ID: <[email protected]>
* Enable strict and warnings, and declare variables.
* Use safe pipe open.
* Drop $eof, it's not used.
---
contrib/delayer.in | 36 ++++++++++++++++++------------------
1 file changed, 18 insertions(+), 18 deletions(-)
diff --git a/contrib/delayer.in b/contrib/delayer.in
index abef0d4e2..5febfc04d 100644
--- a/contrib/delayer.in
+++ b/contrib/delayer.in
@@ -15,31 +15,33 @@
# done by christian mock <[email protected]> sometime in july 1998,
# and put into the public domain.
#
-$delay = shift || die "usage: $0 delay prog-n-args\n";
-$timeout = $delay;
-$eof = 0;
+use strict;
+use warnings;
-open(OUT, "|" . join(" ", @ARGV)) || die "open |prog-n-args: $!\n";
+my $delay = shift || die "usage: $0 delay prog-n-args\n";
+
+my $timeout = $delay;
+
+open(OUT, "|-", @ARGV) || die "open |prog-n-args: $!\n";
#select(OUT);
#$| = 1;
#select(STDOUT);
-$rin = '';
+my $rin = '';
+my $rout;
vec($rin, fileno(STDIN), 1) = 1;
-while (!$eof || $#queue >= 0) {
- if (!$eof) {
- ($nfound, $timeleft) = select($rout = $rin, undef, undef, $timeout);
- } else {
- sleep($timeout);
- }
- $now = time();
- $exp = $now + $delay;
+my @queue;
- if (!$eof && vec($rout, fileno(STDIN), 1)) {
- $line = <STDIN>;
+while (1) {
+ my ($nfound, $timeleft) = select($rout = $rin, undef, undef, $timeout);
+ my $now = time();
+ my $exp = $now + $delay;
+
+ if (vec($rout, fileno(STDIN), 1)) {
+ my $line = <STDIN>;
if (!defined $line) { # exit NOW!
foreach (@queue) {
s/^[^:]+://g;
@@ -57,13 +59,11 @@ while (!$eof || $#queue >= 0) {
next;
}
- ($first, $line) = split(/:/, $queue[0], 2);
+ my ($first, $line) = split(/:/, $queue[0], 2);
while ($#queue >= 0 && $first <= $now) {
print OUT $line;
shift(@queue);
($first, $line) = split(/:/, $queue[0], 2);
}
$timeout = $first - $now;
-
}
-
--
2.39.2
------------------------------
Message: 4
Date: Mon, 15 Jan 2024 18:00:00 +0100
From: Christoph Biedl <[email protected]>
To: [email protected]
Subject: [PATCH 2/4] delayer: Add documentation
Message-ID: <[email protected]>
---
contrib/README | 2 +-
contrib/delayer.in | 68 ++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 69 insertions(+), 1 deletion(-)
diff --git a/contrib/README b/contrib/README
index 944963908..fd6063103 100644
--- a/contrib/README
+++ b/contrib/README
@@ -63,7 +63,7 @@ delayer
Sits in a data stream and delays it by some constant period of time.
Mostly useful for delaying innfeed feeds to allow cancels a chance to
remove articles before innfeed sends them to your peers. See the
- beginning of the file for an example of how to use it.
+ embedded documentation for an example of how to use it.
expirectl
diff --git a/contrib/delayer.in b/contrib/delayer.in
index 5febfc04d..974efcd2a 100644
--- a/contrib/delayer.in
+++ b/contrib/delayer.in
@@ -67,3 +67,71 @@ while (1) {
}
$timeout = $first - $now;
}
+
+__END__
+
+=head1 NAME
+
+delayer - A pipe to delay line-based input by a given time
+
+=head1 SYNOPSIS
+
+ <some program> | delayer <seconds> | <some other program>
+
+=head1 DESCRIPTION
+
+This program implements a delaying pipe. Lines sent to the process are
+spooled and printed after the given delay time has passed.
+
+The main use case is for a newsfeed that deliberately should not
+distribute articles as soon as possible. One reason is giving
+cybercancels and NoCeMs time to arrive and cancel articles before they
+are spread. Another one is using a link only as a backup.
+
+=head2 Configuration
+
+=over
+
+=item * Choose a name for that feed, e.g. C<delayfeed>.
+
+=item * Copy C<pathetc>/innfeed.conf to C<pathetc>/delayfeed.conf
+
+=item * Edit C<pathetc>/delayfeed.conf and change C<newsfeed> to
+C<delayfeed>, typically C<pid-file>, C<log-file>, C<backlog-directory>,
+and C<status-file>.
+
+=item * Possibly limit C<max-connections> to C<1>.
+
+=item * Add a new entry to B<pathetc>/newsfeeds like
+
+ innfeed-delayed!\
+ :!*\
+ :Tc,Wnm*,S16384:/path/to/delayer 120 \
+ <newsbin>/innfeed -c innfeed-delayed.conf
+
+This will delay articles via that feed for 120 seconds.
+
+=item * Enable the feed:
+
+ ctlinnd begin innfeed-delayed
+
+=back
+
+=head1 BUGS
+
+If the feed is closed, all lines in the store are printed immediately,
+breaking the contract of delaying them.
+
+If the number of articles in that feed is rather low - just a few
+articles per delay time or less - some effects of buffering will delay
+the transmission even further.
+
+=head1 HISTORY
+
+July 1998 christian mock <[email protected]>
+
+=head1 SEE ALSO
+
+delay(1), newsfeeds(5)
+
+=cut
--
2.39.2
------------------------------
Message: 5
Date: Mon, 15 Jan 2024 18:00:00 +0100
From: Christoph Biedl <[email protected]>
To: [email protected]
Subject: [PATCH 3/4] delayer: Introduce options for buffering and
persistence
Message-ID: <[email protected]>
---
contrib/delayer.in | 107 +++++++++++++++++++++++++++++++++++++++------
1 file changed, 93 insertions(+), 14 deletions(-)
diff --git a/contrib/delayer.in b/contrib/delayer.in
index 974efcd2a..8b66c3b48 100644
--- a/contrib/delayer.in
+++ b/contrib/delayer.in
@@ -19,22 +19,45 @@
use strict;
use warnings;
-my $delay = shift || die "usage: $0 delay prog-n-args\n";
+use Getopt::Long;
+use Storable;
+
+(@ARGV >= 2) || die "usage: $0 delay prog-n-args\n";
+
+my $delay = 60;
+my $buffered = 1;
+my $store;
+
+if ($ARGV[0] =~ /^[0-9]+$/) {
+ # Legacy calling convention
+ $delay = shift;
+} else {
+ GetOptions (
+ 'delay=i' => \$delay,
+ 'buffered!' => \$buffered,
+ 'store=s' => \$store,
+ ) or die ('Usage!');
+}
my $timeout = $delay;
open(OUT, "|-", @ARGV) || die "open |prog-n-args: $!\n";
+if (!$buffered) {
+ select(OUT);
+ $| = 1;
+ select(STDOUT);
+}
-#select(OUT);
-#$| = 1;
-#select(STDOUT);
+my @queue;
+if ($store) {
+ eval { @queue = @{retrieve ($store)} };
+ unlink ($store);
+}
my $rin = '';
my $rout;
vec($rin, fileno(STDIN), 1) = 1;
-my @queue;
-
while (1) {
my ($nfound, $timeleft) = select($rout = $rin, undef, undef, $timeout);
my $now = time();
@@ -43,12 +66,16 @@ while (1) {
if (vec($rout, fileno(STDIN), 1)) {
my $line = <STDIN>;
if (!defined $line) { # exit NOW!
- foreach (@queue) {
- s/^[^:]+://g;
- print OUT;
+ if ($store) {
+ eval { store (\@queue, $store); };
+ } else {
+ foreach (@queue) {
+ s/^[^:]+://g;
+ print OUT;
+ }
+ close(OUT);
+ sleep(1);
}
- close(OUT);
- sleep(1);
exit;
}
push(@queue, "$exp:$line");
@@ -76,6 +103,10 @@ delayer - A pipe to delay line-based input by a given time
=head1 SYNOPSIS
+ <some program> | delayer [options] | <some other program>
+
+Legacy calling convention:
+
<some program> | delayer <seconds> | <some other program>
=head1 DESCRIPTION
@@ -106,7 +137,9 @@ and C<status-file>.
innfeed-delayed!\
:!*\
- :Tc,Wnm*,S16384:/path/to/delayer 120 \
+ :Tc,Wnm*,S16384:/path/to/delayer \
+ --delay 120 \
+ -- \
<newsbin>/innfeed -c innfeed-delayed.conf
This will delay articles via that feed for 120 seconds.
@@ -117,14 +150,60 @@ This will delay articles via that feed for 120 seconds.
=back
+=head1 OPTIONS
+
+=head2 C<--delay> <seconds>
+
+Delay articles by the given amount of seconds. Default is C<60>.
+
+=head2 C<--buffered>, C<--no-buffered>
+
+Per default, output is buffered. This increases the time until an
+article is actually sent if the number of articles is small. Disable
+buffering to have a more accurate delay, at a price of a (possibly
+neglectable) performance overhead.
+
+=head2 C<--store> F<file>
+
+By default, all buffered lines are written out if the input is closed,
+even if the configured delay has not been reached yet.
+
+With the given store, these lines will be written to the given file
+instead, to be used upon next startup.
+
+=head2 C<-->
+
+This separates the options to the delay from the process the output
+is written to.
+
+=head2 C<The rest>
+
+This should be the program the output is written to, optionally with
+some parameters.
+
+=head1 OPTIONS, LEGACY
+
+This interface is considered legacy and will be removed some day.
+
+=head2 C<delay>
+
+Delay articles by the given amount of seconds.
+
+=head2 C<The rest>
+
+This should be the program the output is written to, optionally with
+some parameters.
+
=head1 BUGS
If the feed is closed, all lines in the store are printed immediately,
-breaking the contract of delaying them.
+breaking the contract of delaying them. See C<--store> to avoid this.
If the number of articles in that feed is rather low - just a few
articles per delay time or less - some effects of buffering will delay
-the transmission even further.
+the transmission even further. See C<--no-buffering> to alleviate this.
+Still a line is not sent to output until another line is read from the
+input.
=head1 HISTORY
--
2.39.2
------------------------------
Subject: Digest Footer
_______________________________________________
inn-workers mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/inn-workers
------------------------------
End of inn-workers Digest, Vol 157, Issue 7
*******************************************
