Bug#818761: packages.debian.org doesn't list experimental packages anymore

[Frank Lichtenheld]

Package: www.debian.org
Followup-For: Bug #818761

I've prepared a patch for this. Please find it attached.

I don't just want to push it though, since it would be good to clean
up the archive/ directory on packages.debian.org. So it would be
preferable that someone with access to the deployment merges it and
then takes care of the deployment immediately.

Regards,
  Frank


-- System Information:
Debian Release: 8.3
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
>From 63322b969c931107c3d26d3e9c8b1b24a41b7da5 Mon Sep 17 00:00:00 2001
From: Frank Lichtenheld 
Date: Mon, 28 Mar 2016 20:21:12 +0200
Subject: [PATCH] Teach packages.debian.org to handle .xz archive files

Currently only for main archive, not backports or
debports, yet.
Crude mix of autodetection and configuration but
since at least experimental is completely broken
right now my priority was to get it working.

Closes: #818761
---
 bin/parse-packages| 38 ++
 bin/parse-sources | 35 ---
 config.sh.sed.in  |  4 
 cron.d/100syncarchive | 33 +
 4 files changed, 91 insertions(+), 19 deletions(-)

diff --git a/bin/parse-packages b/bin/parse-packages
index a7403a9..280adaf 100755
--- a/bin/parse-packages
+++ b/bin/parse-packages
@@ -63,6 +63,37 @@ mkpath( "$DBDIR/xapian.new" );
 my %descriptions_english_db;
 tie %descriptions_english_db, "DB_File", "files/db/descriptions_translated_english_only.db", O_RDONLY, 0666, $DB_BTREE;
 
+my %ext_to_prog = (
+xz => 'xzcat',
+gz => 'zcat',
+);
+sub open_packages_files {
+my( $suite_dir, $component) = @_;
+
+my (@files, $prog);
+for my $ext (qw(xz gz)){
+	my $packages_match = "binary-*/Packages.$ext";
+	@files = ();
+	push @files, <"$suite_dir/$component/$packages_match">;
+	push @files, <"$suite_dir/updates/$component/$packages_match">;
+	push @files, <"$suite_dir/$component/debian-installer/$packages_match">;
+
+	if( @files ){
+	$prog = $ext_to_prog{$ext};
+	last;
+	}
+}
+
+if( @files && $prog ){
+	print "\tprog=$prog\n";
+	print "\tfiles=@files\n";
+	open my $fh, '-|', $prog, @files;
+	return $fh;
+}
+print "\tno files found, skipping...\n";
+return;
+}
+
 for my $suite (@SUITES) {
 my %package_names_suite = ();
 my %packages_all_db;
@@ -76,10 +107,9 @@ for my $suite (@SUITES) {
 print "\tseems not to exist, skipping...\n";
 next;
 }
-	open PKG, "zcat $TOPDIR/archive/$archive/$suite/$what/binary-*/Packages.gz"
-	. " $TOPDIR/archive/$archive/$suite/updates/$what/binary-*/Packages.gz"
-	. " $TOPDIR/archive/$archive/$suite/$what/debian-installer/binary-*/Packages.gz|";
-	while () {
+
+	my $fh = open_packages_files("$TOPDIR/archive/$archive/$suite/", $what) || next;
+	while (<$fh>) {
 		next if /^\s*$/;
 		my $data = "";
 		my %data = ();
diff --git a/bin/parse-sources b/bin/parse-sources
index 1f73bcc..a4e6d06 100755
--- a/bin/parse-sources
+++ b/bin/parse-sources
@@ -38,6 +38,36 @@ $/ = "";
 
 -d $DBDIR || mkpath( $DBDIR );
 
+my %ext_to_prog = (
+xz => 'xzcat',
+gz => 'zcat',
+);
+sub open_sources_files {
+my( $suite_dir, $component) = @_;
+
+my (@files, $prog);
+for my $ext (qw(xz gz)){
+	my $sources_match = "source/Sources.$ext";
+	@files = ();
+	push @files, <"$suite_dir/$component/$sources_match">;
+	push @files, <"$suite_dir/updates/$component/$sources_match">;
+
+	if( @files ){
+	$prog = $ext_to_prog{$ext};
+	last;
+	}
+}
+
+if( @files && $prog ){
+	print "\tprog=$prog\n";
+	print "\tfiles=@files\n";
+	open my $fh, '-|', $prog, @files;
+	return $fh;
+}
+print "\tno files found, skipping...\n";
+return;
+}
+
 for my $archive (@ARCHIVES) {
 for my $suite (@SUITES) {
 
@@ -51,9 +81,8 @@ for my $archive (@ARCHIVES) {
 		print "\tseems not to exist, skipping...\n";
 		next;
 	}
-	open PKG, "zcat $TOPDIR/archive/$archive/$suite/$what/source/Sources.gz"
-	. " $TOPDIR/archive/$archive/$suite/updates/$what/source/Sources.gz|";
-	while () {
+	my $fh = open_sources_files("$TOPDIR/archive/$archive/$suite/", $what) || next;
+	while (<$fh>) {
 		next if /^\s*$/;
 		my $data = "";
 		my %data = ();
diff --git a/config.sh.sed.in b/config.sh.sed.in
index 10970d2..03ffc31 100644
--- a/config.sh.sed.in
+++ b/config.sh.sed.in
@@ -62,6 +62,10 @@ arch_sid="${arch_stretch} alpha avr32 hppa hurd-i386 m68k powerpcspe ppc64el sh4
 arch_experimental="${arch_sid}"
 arch_wheezy_proposed_updates="${arch_wheezy}"
 
+ext_wheezy=gz
+#for security Sources
+ext_jessie=gz
+
 # from http://ftp.de.debian.org/debian-ports/dists/{sid,experimental}/Release
 arch_debports="alpha hppa m68k powerpcspe ppc64 

Bug#818761: packages.debian.org doesn't list experimental packages anymore

[Laura Arjona Reina]

Control: retitle -1 packages.debian.org doesn't list experimental packages 
anymore

Ah, true!
It shows the packages in experimental only when in debports, not in other 
architectures.
Retitling back to the original title. Sorry for the noise.


El 20/03/16 a las 17:03, Didier 'OdyX' Raboud escribió:
> Le dimanche, 20 mars 2016, 16.23:57 Laura Arjona Reina a écrit :
>> Control: retitle -1 packages.debian.org doesn't list the source of
>> experimental packages anymore
>>
>> Hi all
>> Using a web browser, I can see the package in experimental:
>>
>> https://packages.debian.org/experimental/libgcrypt20
> 
> Note that this page only lists Debian Ports' packages, so I stand by my 
> initial bug title :-)
> 

-- 
Laura Arjona Reina
https://wiki.debian.org/LauraArjona

Bug#818761: packages.debian.org doesn't list experimental packages anymore

[Didier 'OdyX' Raboud]

Le dimanche, 20 mars 2016, 16.23:57 Laura Arjona Reina a écrit :
> Control: retitle -1 packages.debian.org doesn't list the source of
> experimental packages anymore
> 
> Hi all
> Using a web browser, I can see the package in experimental:
> 
> https://packages.debian.org/experimental/libgcrypt20

Note that this page only lists Debian Ports' packages, so I stand by my 
initial bug title :-)

-- 
Cheers,
OdyX

Bug#818761: packages.debian.org doesn't list experimental packages anymore

[Laura Arjona Reina]

Control: retitle -1 packages.debian.org doesn't list the source of experimental 
packages anymore

Bug#818761: packages.debian.org doesn't list experimental packages anymore

[Laura Arjona Reina]

Control: retitle -1 packages.debian.org doesn't list the source of
experimental packages anymore

Hi all
Using a web browser, I can see the package in experimental:

https://packages.debian.org/experimental/libgcrypt20

returns a page with the package details.

However, in the right sidebar:
Download Source Package :
Not found

Another example:

https://packages.debian.org/experimental/database/

shows a list of 5 packages, but

https://packages.debian.org/source/experimental/database/
shows an empty list.

Regards
-- 
Laura Arjona Reina
https://wiki.debian.org/LauraArjona

Bug#818761: packages.debian.org doesn't list experimental packages anymore

[Didier 'OdyX' Raboud]

Package: www.debian.org
Severity: important

Apparently since the archive changes listed in #818463, packages.debian.org
doesn't list experimental packages anymore:

$ apt-cache policy libgcrypt20 | grep -B1 experimental
 1.6.5+git20160220+839d12c-2 1
  1 http://httpredir.debian.org/debian experimental/main amd64 Packages
$ GET 'https://packages.debian.org/source/experimental/libgcrypt20' | grep 'not 
available'
 Package not available in this suite.

Cheers,
OdyX