Package: www.debian.org
User: www.debian@packages.debian.org
Usertag: scripts
Severity: normal
Dear all
Since several days, the script wnpp.pl fails intermittently when trying to
connect to bugs.debian.org, and when it fails, we receive the following
messages in the webwml build log:
[...]
make -C wnpp install
make[3]: Entering directory '/srv/www.debian.org/webwml/english/devel/wnpp'
wml -q -D CUR_YEAR=2022 -o UNDEFuEN:../../../english/devel/wnpp/wnpp.data@g+w
-W7,-Squotes ../../../english/devel/wnpp/wnpp.pl
ePerl:Error: Perl runtime error (interpreter rc=2)
Contents of STDERR channel: -
500 Can't connect to bugs.debian.org:443 (Connection refused) at
/tmp/wml.12011.tmp1 line 147.
--
** WML:Break: Error in Pass 3 (rc=1).
Died at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 403.
TheWML::Frontends::Wml::Runner::_run_pass(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458),
3, SCALAR(0x55f984f718d8), SCALAR(0x55f984f718a8), SCALAR(0x55f984f718c0))
called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 441
TheWML::Frontends::Wml::Runner::_passes_loop(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458))
called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 726
TheWML::Frontends::Wml::Runner::_output_and_cleanup(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458))
called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 930
TheWML::Frontends::Wml::Runner::run_with_ARGV(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458),
HASH(0x55f984b252b8)) called at /usr/bin/wml line 47
make[3]: *** [Makefile:34: ../../../english/devel/wnpp/wnpp.data] Error 2
make[3]: Leaving directory '/srv/www.debian.org/webwml/english/devel/wnpp'
make[2]: [../../Makefile.common:84: wnpp-install] Error 2 (ignored)
make[2]: Leaving directory '/srv/www.debian.org/webwml/english/devel'
[...]
Since the line refers to a temp file, I'm not 100% sure about which is the
command that causes the failure, but the code in wnpp.pl that uses
bugs.debian.org I think is restricted to these lines (initial connection, and
retrieval of bug info to build the wnpp.data local database):
https://salsa.debian.org/webmaster-team/webwml/-/blob/master/english/devel/wnpp/wnpp.pl#L41
(L41 to 48)
my $soap =
SOAP::Lite->uri('Debbugs/SOAP')->proxy('https://bugs.debian.org/cgi-bin/soap.cgi')
or die "Couldn't make connection to SOAP interface: $@";
my $bugs = $soap->get_bugs(package=>'wnpp')->result;
my $status = {};
while (my @slice = splice(@$bugs, 0, 500)) {
my $tmp = $soap->get_status(@slice)->result() or die;
%$status = (%$status, %$tmp);
}
Some people commented the issue in the #debian-admin IRC channel, for the case
there was some issue in the bugs.d.o servers or some configuration change, but
it seems the failures are related to the connections being blocked by the http
rate-limiting. It's not clear, however, why these issues are appearing now
(nothing in the script changed in the recent times, and the amount of bugs is
not drastically bigger than weeks before). In addition to this, some other days
the script is tolerated well (for example, we had failures on 2022/02/18 and
2022/02/22, but today 2022/02/23 the builds are finishing successfully).
Looking at the code, I see we're processing bugs in groups of 500 bugs. I'm not sure if
we could try to reduce the number of the bugs in the group and introduce some
"sleep" time between queries, or it's better to do the opposite, increase the
number of bugs in a group to reduce the total number of queries.
Other approach could be to rewrite the wnpp.pl script so it uses queries
against UDD instead of the SOAP CGI interface of bugs.debian.org
I don't know which approach is better, and also my Perl skills are not enough
to propose a patch myself for none of this two options. If anybody can and
wants to work on this, it's very appreciated.
I'll point the Debbugs team to this bug one it is created for the case they
changed any configuration in their deployment that caused the connection
failures, or for the case they have any advice about the better way to connect
or retrieve the information we need to create the WNPP-related web pages.
Kind regards,
Laura Arjona Reina
https://wiki.debian.org/LauraArjona
