Bug#1057431: fakeupstream.cgi: add support for static.rust-lang.org
Hi Zixing, The newest stable upstream release is advertised in more than one way allowing to write regular debian/watch files. Here below I give two working examples. | version=4 | https://forge.rust-lang.org/infra/other-installation-methods.html#source-code \ | https://static.rust-lang.org/dist/rustc-(\d+(?:\.\d+)*)-src\.tar\.gz | | $ uscan --safe --package rustc --upstream-version 1.70.0+dfsg1 --watchfile watch1 | Newest version of rustc on remote site is 1.74.1, local version is 1.70.0+dfsg1 | => Newer package available from: | => https://static.rust-lang.org/dist/rustc-1.74.1-src.tar.gz | | version=4 | https://github.com/rust-lang/rust/tags \ | .*/rust-lang/rust/archive/refs/tags/(\d+(?:\.\d+)*)\.tar\.gz | | Newest version of rustc on remote site is 1.74.1, local version is 1.70.0+dfsg1 | => Newer package available from: | => https://github.com/rust-lang/rust/archive/refs/tags/1.74.1.tar.gz And the watch file currently in Debian works fine. The version on the package tracker is correct. https://tracker.debian.org/pkg/rustc | A new upstream version is available: 1.74.1 high | A new upstream version 1.74.1 is available, you should consider packaging it. Fakeupstream is for when the above would be impossible. In my understanding of the described special use cases, they can be covered separately. For example, some scraping script can collect the info 4x/day and produce a html page suitable for debian/watch. The loops in your patch fit better in such script than in a cgi script. Or maybe upstream is willing to put the info you need on their website. Have you asked them? Cheers, Bart On Tue, Dec 12, 2023 at 03:06:53PM -0700, Zixing Liu wrote: > Yes, while that is true, the page is updated frequently with older > versions replaced with the new version. > In Debian, usually, we can't package the latest version as quickly as > possible due to bootstrapping requirements. > Hence, a fakeupstream service is needed to retrieve a non-latest > version from the upstream. > > Also, another use-case might be, you want a specific nightly or beta > to test a certain regression. Since that page > does not list the previous releases, it would be difficult to rely on > it to retrieve a previous beta using uscan by > specifying `--download-version`.
Bug#1057431: fakeupstream.cgi: add support for static.rust-lang.org
Hi Bart, > That should be suitable for a watch file. Right? Yes, while that is true, the page is updated frequently with older versions replaced with the new version. In Debian, usually, we can't package the latest version as quickly as possible due to bootstrapping requirements. Hence, a fakeupstream service is needed to retrieve a non-latest version from the upstream. Also, another use-case might be, you want a specific nightly or beta to test a certain regression. Since that page does not list the previous releases, it would be difficult to rely on it to retrieve a previous beta using uscan by specifying `--download-version`. Thanks, Zixing On Tue, Dec 12, 2023 at 2:58 PM Bart Martens wrote: > > Hi Zixing, the source code and signature are also advertised here: > https://forge.rust-lang.org/infra/other-installation-methods.html#source-code > That should be suitable for a watch file. Right? > > > On Tue, Dec 05, 2023 at 02:44:06PM -0700, Zixing Liu wrote: > > Hi Bart, > > > > > The link https://static.rust-lang.org/dist displays an error in xml > > > format. > > > > This is because this website is hosted on AWS S3 and is machine-readable > > only. > > To list the files, you need to either use an S3-compatible tool or visit > > https://static.rust-lang.org/?prefix=dist/ to see the file listing in > > XML format. > > > > > Can I view an example of a package in Debian that can be watched with > > > such extension of fakeupstream? > > > > This fakeupstream service is intended for > > https://tracker.debian.org/pkg/rustc. > > Because it needs to download the signed source tarball from > > https://static.rust-lang.org. > > > > Thanks, > > Zixing > > > > On Tue, Dec 5, 2023 at 1:46 PM Bart Martens wrote: > > > > > > Hello Zixing Liu, > > > > > > Can I view an example of a package in Debian that can be watched with such > > > extension of fakeupstream? The link https://static.rust-lang.org/dist > > > displays > > > an error in xml format. A real example would help me understand the > > > context. > > > > > > Ciao, > > > > > > Bart > > > > > > > > > On Mon, Dec 04, 2023 at 03:32:39PM -0700, Zixing Liu wrote: > > > > Package: qa.debian.org > > > > Severity: wishlist > > > > Tags: patch > > > > X-Debbugs-Cc: zixing@canonical.com > > > > > > > > Hi Debian QA Team, > > > > > > > > I would like to see a fakeupstream service implemented for > > > > https://static.rust-lang.org/dist so that > > > > > --
Bug#1057431: Fwd: Bug#1057431: fakeupstream.cgi: add support for static.rust-lang.org
Hi Bart, > The link https://static.rust-lang.org/dist displays an error in xml format. This is because this website is hosted on AWS S3 and is machine-readable only. To list the files, you need to either use an S3-compatible tool or visit https://static.rust-lang.org/?prefix=dist/ to see the file listing in XML format. > Can I view an example of a package in Debian that can be watched with such > extension of fakeupstream? This fakeupstream service is intended for https://tracker.debian.org/pkg/rustc. Because it needs to download the signed source tarball from https://static.rust-lang.org. Thanks, Zixing On Tue, Dec 5, 2023 at 1:46 PM Bart Martens wrote: > > Hello Zixing Liu, > > Can I view an example of a package in Debian that can be watched with such > extension of fakeupstream? The link https://static.rust-lang.org/dist displays > an error in xml format. A real example would help me understand the context. > > Ciao, > > Bart > > > On Mon, Dec 04, 2023 at 03:32:39PM -0700, Zixing Liu wrote: > > Package: qa.debian.org > > Severity: wishlist > > Tags: patch > > X-Debbugs-Cc: zixing@canonical.com > > > > Hi Debian QA Team, > > > > I would like to see a fakeupstream service implemented for > > https://static.rust-lang.org/dist so that >
Bug#1057431: fakeupstream.cgi: add support for static.rust-lang.org
Hello Zixing Liu, Can I view an example of a package in Debian that can be watched with such extension of fakeupstream? The link https://static.rust-lang.org/dist displays an error in xml format. A real example would help me understand the context. Ciao, Bart On Mon, Dec 04, 2023 at 03:32:39PM -0700, Zixing Liu wrote: > Package: qa.debian.org > Severity: wishlist > Tags: patch > X-Debbugs-Cc: zixing@canonical.com > > Hi Debian QA Team, > > I would like to see a fakeupstream service implemented for > https://static.rust-lang.org/dist so that
Bug#1057431: fakeupstream.cgi: add support for static.rust-lang.org
Package: qa.debian.org
Severity: wishlist
Tags: patch
X-Debbugs-Cc: zixing@canonical.com
Hi Debian QA Team,
I would like to see a fakeupstream service implemented for
https://static.rust-lang.org/dist so that the debian/watch file inside the
rustc package could be simplified and actually useful for checking new
releases.
Attached below is a patch that I proposed to add the said functionality.
You can find the same patch at:
https://salsa.debian.org/qa/qa/-/merge_requests/48
if you prefer using Debian Salsa to merge the patch/merge request.
Thanks,
Zixing
>From 9bbd2ecffe31822cf2a77ed0ce9ed69f78931aee Mon Sep 17 00:00:00 2001
From: liushuyu
Date: Mon, 23 Oct 2023 22:27:29 -0600
Subject: [PATCH] cgi-bin/fakeupstream.cgi: add support ...
... for monitoring Rust releases
---
cgi-bin/fakeupstream.cgi | 77
1 file changed, 77 insertions(+)
diff --git a/cgi-bin/fakeupstream.cgi b/cgi-bin/fakeupstream.cgi
index c13538e8..718b2540 100755
--- a/cgi-bin/fakeupstream.cgi
+++ b/cgi-bin/fakeupstream.cgi
@@ -26,6 +26,8 @@ use JSON;
use File::Temp qw/ tempdir /;
use Date::Format;
use Date::Parse;
+use Time::Piece;
+use Time::Seconds qw(ONE_DAY);
use Dpkg::Version;
use threads;
use constant MAX_THREADS => 3;
@@ -1110,6 +1112,81 @@ elsif( $upstream_param =~
m%^google-fonts/($project_char_re+)/($project_char_re+
exit 0;
}
+elsif( $upstream_param eq 'rustc' )
+{
+ my $ua = LWP::UserAgent->new;
+
+ sub fetch_rust_github_tags($) {
+ my $ua = shift;
+ my @versions;
+ my $page_number = 1;
+ my $per_page = 100;
+ my $url =
"https://api.github.com/repos/rust-lang/rust/tags?per_page=$per_page;;
+ while (1) {
+ my $response = $ua->get( "$url=$page_number" );
+ return_error( "failed to read $url :
$response->status_line" ) if( not $response->is_success );
+ my $json = JSON::decode_json(
$response->decoded_content );
+ foreach my $version (@{$json}) {
+ my $version_name = $version->{"name"};
+ if ( $version_name =~ /^\d+\.\d+\.\d+$/ ) {
+ push @versions, $version_name;
+ }
+ }
+ last if (scalar @versions) < $per_page;
+ }
+ $page_number++;
+ return @versions;
+ }
+
+ sub fetch_rust_last_unstable_date($$) {
+ my $ua = shift;
+ my $channel = shift;
+ my $url =
"https://static.rust-lang.org/dist/channel-rust-$channel-date.txt;;
+ my $response = $ua->get ( $url );
+ return_error( "failed to read $url : $response->status_line" )
unless( $response->is_success );
+ return $response->decoded_content;
+ }
+
+ sub make_header($) {
+ my $channel = shift;
+ my $title = "Rust toolchain version listing ($channel)";
+ print $q->header;
+ print $q->start_html({ -title => $title });
+ print $q->h1($title) . "\n";
+ print $q->start_ul;
+ }
+
+ $type_param = 'stable' unless $type_param;
+ my $channel = $type_param;
+
+ if ($channel eq 'stable') {
+ my @tags = fetch_rust_github_tags( $ua );
+ make_header($channel);
+ foreach my $version (@tags) {
+ print $q->li( $q->a( { -href =>
"https://static.rust-lang.org/dist/rustc-$version-src.tar.xz; }, $version ) );
+ }
+ } else {
+ my $last_date = fetch_rust_last_unstable_date ( $ua, $channel );
+ my $limit = 10;
+ make_header($channel);
+ print $q->p( "For unstable channels, only releases within the
last $limit days are listed." );
+ while ($limit > 0) {
+ my $response = $ua->get(
"https://static.rust-lang.org/dist/$last_date/channel-rust-$channel.toml; );
+ if ($response->is_success()) {
+ if ($response->decoded_content =~
/(1\.\d+\.\d+-(?:nightly|beta)(?:\.\d+)?\s+\([0-9a-f]{9}\s+\d+-\d+-\d+\))/) {
+ print $q->li( $q->a( { -href =>
"https://static.rust-lang.org/dist/$last_date/rustc-$channel-src.tar.xz; }, $1
) );
+ }
+ }
+ my $next_last_date = Time::Piece->strptime($last_date,
"%Y-%m-%d") - ONE_DAY;
+ $last_date = $next_last_date->ymd;
+ $limit--;
+ }
+ }
+
+ print $q->end_ul;
+ print $q->end_html;
+ exit 0;
+}
my %upstream_info_per_package =
(
--
GitLab
