Bug#859122: about 500 DLAs missing from the website
On Mon, Feb 11, 2019 at 03:56:41PM -0500, Antoine Beaupré wrote: > It's true there's a lot of junk in there... I suspect most of the `.pl` > scripts in there could actually be symlink to the main secteam scripts, > because they are basically the same. > > I also suspect most of the stuff is unused, even from the secteam's > point of view. For example, `check-cve-refs.pl` assumes there's a > `security/data` directory in the website, which is not the case > (anymore?). I'll also leave that to the security/www teams considerations ;) > I would suggest removing those from at least the LTS > section and have done so in the following MR: > https://salsa.debian.org/webmaster-team/webwml/merge_requests/55 I've reviewed, merged and pushed this now. Thank you! > > * This new /lts section of the website is not referenced yet in other > > places of the Debian website. I'm not sure if it should be referenced in > > /security, in /releases/, or in both. There is also the temptation > > of creating a link in the homepage but there is also the suggestion of > > reducing the links in the homepage, so... For now, I'll try to add it to > > the sitemap and see how many references to the LTS wiki page we have > > currently, to see if any of them can be replaced with link to this > > section in the website. But I'll wait some days to do it because it's > > not clear for me if you want to populate the section to cover all the > > aspects of LTS, or keep it only/mainly for security stuff. > I would avoid putting the LTS work too proeminently on the website at > this point, to be honest. The goal of publishing those advisories there, > for me, is coherence: they were already partly present and I wanted to > have them *all* available *somewhere* with a predictable URL and RSS > feeds (as opposed to, say the mailing list). agreed. > We shouldn't get into the slippery debate of how much we want LTS > content on the website, in my opinion. at least for here and now! :) -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Bug#859122: about 500 DLAs missing from the website
Hi, On Mon, Feb 11, 2019 at 04:26:38PM -0500, Antoine Beaupré wrote: > > I think we want the other DLAs linked from the indexes as well. > > shall we file a bug to not forget this? > I looked into this, and couldn't figure it out. > Please do file a bug for now, I have no idea how to fix this... ok, will do. > >> that sets the redirect from > >> https://www.debian.org/security/any_year/dla-whatever to > >> https://www.debian.org/security/lts/any_year/dla-whatever > > right. shall we file a bug to not forget this? > Filed the patch here: > https://salsa.debian.org/anarcat/dsa-puppet/merge_requests/1 cool, thank you. > Reviews welcome. I'm particularly doubtful of the dla-map thing - it's > not in the source repo, but can I assume it's present on the website > deployment? I cannot comment on that dla-map, the rest looks good to me. (And simpler than I expected.) > >> * Adaptation in the security tracker so the new URL paths are used from > >> now on is also needed. > > right. shall we file a bug to not forget this? ok, will do. -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Bug#859122: about 500 DLAs missing from the website
On 2019-02-12 08:13:18, Salvatore Bonaccorso wrote:
> Hi,
>
> On Sat, Feb 09, 2019 at 03:55:44AM +0100, Laura Arjona Reina wrote:
>> * We still need the Apache redirects, so the people that try the old
>> URLs (wether directly because they knew, or via the security tracker),
>> find the files they need. What we need to do is send a patch to
>>
>> https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/roles/templates/apache-www.debian.org.erb
>>
>> that sets the redirect from
>> https://www.debian.org/security/any_year/dla-whatever to
>> https://www.debian.org/security/lts/any_year/dla-whatever
>>
>> * Adaptation in the security tracker so the new URL paths are used from
>> now on is also needed.
>
> I have the attached patch commited in a local branch, but want first
> to confirm is this the final intended URL to reach the DLAs?
>
> Regards,
> Salvatore
> From ceda9e3d1fc38f505462bce8c0aa4cdd2b165d87 Mon Sep 17 00:00:00 2001
> From: Salvatore Bonaccorso
> Date: Tue, 12 Feb 2019 08:10:16 +0100
> Subject: [PATCH] Adapt URL to DLA advisories in a
> https://www.debian.org/security/lts/
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
>
> As discussed in https://bugs.debian.org/859122 DLAs and DSAs will be
> separated in different supages. This needs adaption for the URL
> referenced in the source fields of the security-tracker for DLAs.
>
> Thanks: Laura Arjona Reina, Holger Levsen and Antoine Beaupré
> ---
> bin/tracker_service.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/bin/tracker_service.py b/bin/tracker_service.py
> index 971f4b4e38eb..a2ea755d8f39 100755
> --- a/bin/tracker_service.py
> +++ b/bin/tracker_service.py
> @@ -1574,7 +1574,7 @@ Debian bug number.'''),
> for (date,) in self.db.cursor().execute(
> "SELECT release_date FROM bugs WHERE name = ?", (dla,)):
> (y, m, d) = date.split('-')
> -return
> url.absolute("https://www.debian.org/security/%d/dla-%d;
> +return
> url.absolute("https://www.debian.org/security/lts/%d/dla-%d;
> % (int(y), int(number)))
> return None
I believe this is backwards, you want /lts/security, not /security/lts.
For example:
https://www.debian.org/lts/security/2019/dla-1659
I was also hoping to see the "errata number" in there, but it seems I
was mistaken.
--
L'ennui avec la grande famille humaine, c'est que tout le monde veut
en être le père.
- Mafalda
Bug#859122: about 500 DLAs missing from the website
Hi,
On Sat, Feb 09, 2019 at 03:55:44AM +0100, Laura Arjona Reina wrote:
> * We still need the Apache redirects, so the people that try the old
> URLs (wether directly because they knew, or via the security tracker),
> find the files they need. What we need to do is send a patch to
>
> https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/roles/templates/apache-www.debian.org.erb
>
> that sets the redirect from
> https://www.debian.org/security/any_year/dla-whatever to
> https://www.debian.org/security/lts/any_year/dla-whatever
>
> * Adaptation in the security tracker so the new URL paths are used from
> now on is also needed.
I have the attached patch commited in a local branch, but want first
to confirm is this the final intended URL to reach the DLAs?
Regards,
Salvatore
>From ceda9e3d1fc38f505462bce8c0aa4cdd2b165d87 Mon Sep 17 00:00:00 2001
From: Salvatore Bonaccorso
Date: Tue, 12 Feb 2019 08:10:16 +0100
Subject: [PATCH] Adapt URL to DLA advisories in a
https://www.debian.org/security/lts/
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
As discussed in https://bugs.debian.org/859122 DLAs and DSAs will be
separated in different supages. This needs adaption for the URL
referenced in the source fields of the security-tracker for DLAs.
Thanks: Laura Arjona Reina, Holger Levsen and Antoine Beaupré
---
bin/tracker_service.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/tracker_service.py b/bin/tracker_service.py
index 971f4b4e38eb..a2ea755d8f39 100755
--- a/bin/tracker_service.py
+++ b/bin/tracker_service.py
@@ -1574,7 +1574,7 @@ Debian bug number.'''),
for (date,) in self.db.cursor().execute(
"SELECT release_date FROM bugs WHERE name = ?", (dla,)):
(y, m, d) = date.split('-')
-return url.absolute("https://www.debian.org/security/%d/dla-%d;
+return url.absolute("https://www.debian.org/security/lts/%d/dla-%d;
% (int(y), int(number)))
return None
--
2.20.1
Bug#859122: about 500 DLAs missing from the website
On 2019-02-09 14:39:50, Holger Levsen wrote: > Hi Laura, > > many many thanks for your work on this, including and especially this > writeup! > > some comments below, where I dont say anything I mean 'yay"! :) > > On Sat, Feb 09, 2019 at 03:55:44AM +0100, Laura Arjona Reina wrote: >> * The /lts/security//index.*.html files show the last advisory for >> the cases where there are several files with the same beginning (e.g. >> for DSA- and DSA--2, both html files are generated, but the >> index only points to the -2 file). If this is not the intended >> behaviour, changes in index.wml and Makefiles are needed. > > I think we want the other DLAs linked from the indexes as well. > > shall we file a bug to not forget this? I looked into this, and couldn't figure it out. Please do file a bug for now, I have no idea how to fix this... [...] >> * We still need the Apache redirects, so the people that try the old >> URLs (wether directly because they knew, or via the security tracker), >> find the files they need. What we need to do is send a patch to >> >> https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/roles/templates/apache-www.debian.org.erb >> >> that sets the redirect from >> https://www.debian.org/security/any_year/dla-whatever to >> https://www.debian.org/security/lts/any_year/dla-whatever > > right. shall we file a bug to not forget this? Filed the patch here: https://salsa.debian.org/anarcat/dsa-puppet/merge_requests/1 Reviews welcome. I'm particularly doubtful of the dla-map thing - it's not in the source repo, but can I assume it's present on the website deployment? >> * Adaptation in the security tracker so the new URL paths are used from >> now on is also needed. > > right. shall we file a bug to not forget this? Sure, please do. A. -- People arbitrarily, or as a matter of taste, assigning numerical values to non-numerical things. And then they pretend that they haven't just made the numbers up, which they have. Economics is like astrology in that sense, except that economics serves to justify the current power structure, and so it has a lot of fervent believers among the powerful. - Kim Stanley Robinson, Red Mars
Bug#859122: about 500 DLAs missing from the website
On 2019-02-09 03:55:44, Laura Arjona Reina wrote: > Hello all > > Holger Levsen merged the generated DLAs and I've worked to create the > /lts tree to show them separated from the DSA. I have moved to this new > /lts folder the DLAs from years 2014, 2015 and 2016 that we had already, > and remove them from the /security tree and removed references to DLAs > in the Makefiles/indexes in /security. > > I think it's mostly done, I've closed all the related MR except one, but > there are some small tasks left, that I hope we can solve together: > > * I have initially copied the content of /security/ to /lts/security, > removed subfolders that I think are not needed (audit, key-rollover, > oval, undated) and some other files that I think they were not needed > too. Then I did a search and replace DSA -> DLA, dsa- -> dla- in the > scripts, makefiles and indexes, and fixed the paths, and built locally > (with "make) and I couldn't spot errors, but I don't trust every file > that is currently in /lts/security is needed or has been used with my > "make" command, so a review of the folder (comparing it with /security) > done by an LTS or security team member, is welcome. It's true there's a lot of junk in there... I suspect most of the `.pl` scripts in there could actually be symlink to the main secteam scripts, because they are basically the same. I also suspect most of the stuff is unused, even from the secteam's point of view. For example, `check-cve-refs.pl` assumes there's a `security/data` directory in the website, which is not the case (anymore?). I would suggest removing those from at least the LTS section and have done so in the following MR: https://salsa.debian.org/webmaster-team/webwml/merge_requests/55 > * The README needs to be reviewed and adapted (I just did the search and > replace dsa -> dla and DSA -> DLA). Done as well in the same MR. > * I guess that parse-advisory.pl (and maybe others) can be removed, but > I was not confident to do it without advice. Done as well in the same MR. > * I didn't check the results of the generated RSS feeds. If anybody uses > RSS readers, a review is welcome too. It looks good to me here. > * The /lts/security//index.*.html files show the last advisory for > the cases where there are several files with the same beginning (e.g. > for DSA- and DSA--2, both html files are generated, but the > index only points to the -2 file). If this is not the intended > behaviour, changes in index.wml and Makefiles are needed. Ideally, we'd show both, is that possible? > * Please review the content (text, links) of these files: > > /lts/index.wml > /lts/security/index.wml > > I've tried to be short (for the case translators are fast and then you > decide to heavy rewrite, to not to loose much work). That makes sense to me. I wonder if we should link to the crossreferences.wml content, which is also relevant here. > * Translations have been handled, but I've left the *title* of these > files unchanged: > > french/lts/security/*/dla*.wml > russian/lts/security/*/dla*.wml > danish/lts/security/*/dla*.wml > japanese/lts/security/*/dla*.wml > > All those files have title "LTS Security Advisories from " (being > the year: 2014, or 2015, or 2016). I guess translators can do a > quick search and replace with the correct sentence and they don't need > to update the commit hash, that's already done. I'll contact translators > and point them to this message. Fair enough. > * This new /lts section of the website is not referenced yet in other > places of the Debian website. I'm not sure if it should be referenced in > /security, in /releases/, or in both. There is also the temptation > of creating a link in the homepage but there is also the suggestion of > reducing the links in the homepage, so... For now, I'll try to add it to > the sitemap and see how many references to the LTS wiki page we have > currently, to see if any of them can be replaced with link to this > section in the website. But I'll wait some days to do it because it's > not clear for me if you want to populate the section to cover all the > aspects of LTS, or keep it only/mainly for security stuff. I would avoid putting the LTS work too proeminently on the website at this point, to be honest. The goal of publishing those advisories there, for me, is coherence: they were already partly present and I wanted to have them *all* available *somewhere* with a predictable URL and RSS feeds (as opposed to, say the mailing list). We shouldn't get into the slippery debate of how much we want LTS content on the website, in my opinion. > * We still need the Apache redirects, so the people that try the old > URLs (wether directly because they knew, or via the security tracker), > find the files they need. What we need to do is send a patch to > > https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/roles/templates/apache-www.debian.org.erb > > that sets the redirect from >
Bug#859122: about 500 DLAs missing from the website
Hi Laura, many many thanks for your work on this, including and especially this writeup! some comments below, where I dont say anything I mean 'yay"! :) On Sat, Feb 09, 2019 at 03:55:44AM +0100, Laura Arjona Reina wrote: > * The /lts/security//index.*.html files show the last advisory for > the cases where there are several files with the same beginning (e.g. > for DSA- and DSA--2, both html files are generated, but the > index only points to the -2 file). If this is not the intended > behaviour, changes in index.wml and Makefiles are needed. I think we want the other DLAs linked from the indexes as well. shall we file a bug to not forget this? > * Please review the content (text, links) of these files: > /lts/index.wml > /lts/security/index.wml the former seems a bit bare to me. Also, isnt the 2nd enough, so that we can just drop/not have the former? > * This new /lts section of the website is not referenced yet in other > places of the Debian website. I'm not sure if it should be referenced in > /security, in /releases/, or in both. I think there is no hurry for this, rather I would suggest to not reference for now and then look again in 2-4 weeks, so that we get a better idea where we want it. > * We still need the Apache redirects, so the people that try the old > URLs (wether directly because they knew, or via the security tracker), > find the files they need. What we need to do is send a patch to > > https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/roles/templates/apache-www.debian.org.erb > > that sets the redirect from > https://www.debian.org/security/any_year/dla-whatever to > https://www.debian.org/security/lts/any_year/dla-whatever right. shall we file a bug to not forget this? > * Adaptation in the security tracker so the new URL paths are used from > now on is also needed. right. shall we file a bug to not forget this? > Thanks for reading so long! Thank you for getting us here! -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Bug#859122: about 500 DLAs missing from the website
Hello all Holger Levsen merged the generated DLAs and I've worked to create the /lts tree to show them separated from the DSA. I have moved to this new /lts folder the DLAs from years 2014, 2015 and 2016 that we had already, and remove them from the /security tree and removed references to DLAs in the Makefiles/indexes in /security. I think it's mostly done, I've closed all the related MR except one, but there are some small tasks left, that I hope we can solve together: * I have initially copied the content of /security/ to /lts/security, removed subfolders that I think are not needed (audit, key-rollover, oval, undated) and some other files that I think they were not needed too. Then I did a search and replace DSA -> DLA, dsa- -> dla- in the scripts, makefiles and indexes, and fixed the paths, and built locally (with "make) and I couldn't spot errors, but I don't trust every file that is currently in /lts/security is needed or has been used with my "make" command, so a review of the folder (comparing it with /security) done by an LTS or security team member, is welcome. * The README needs to be reviewed and adapted (I just did the search and replace dsa -> dla and DSA -> DLA). * I guess that parse-advisory.pl (and maybe others) can be removed, but I was not confident to do it without advice. * I didn't check the results of the generated RSS feeds. If anybody uses RSS readers, a review is welcome too. * The /lts/security//index.*.html files show the last advisory for the cases where there are several files with the same beginning (e.g. for DSA- and DSA--2, both html files are generated, but the index only points to the -2 file). If this is not the intended behaviour, changes in index.wml and Makefiles are needed. * Please review the content (text, links) of these files: /lts/index.wml /lts/security/index.wml I've tried to be short (for the case translators are fast and then you decide to heavy rewrite, to not to loose much work). * Translations have been handled, but I've left the *title* of these files unchanged: french/lts/security/*/dla*.wml russian/lts/security/*/dla*.wml danish/lts/security/*/dla*.wml japanese/lts/security/*/dla*.wml All those files have title "LTS Security Advisories from " (being the year: 2014, or 2015, or 2016). I guess translators can do a quick search and replace with the correct sentence and they don't need to update the commit hash, that's already done. I'll contact translators and point them to this message. * This new /lts section of the website is not referenced yet in other places of the Debian website. I'm not sure if it should be referenced in /security, in /releases/, or in both. There is also the temptation of creating a link in the homepage but there is also the suggestion of reducing the links in the homepage, so... For now, I'll try to add it to the sitemap and see how many references to the LTS wiki page we have currently, to see if any of them can be replaced with link to this section in the website. But I'll wait some days to do it because it's not clear for me if you want to populate the section to cover all the aspects of LTS, or keep it only/mainly for security stuff. * We still need the Apache redirects, so the people that try the old URLs (wether directly because they knew, or via the security tracker), find the files they need. What we need to do is send a patch to https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/roles/templates/apache-www.debian.org.erb that sets the redirect from https://www.debian.org/security/any_year/dla-whatever to https://www.debian.org/security/lts/any_year/dla-whatever * Adaptation in the security tracker so the new URL paths are used from now on is also needed. Thanks for reading so long! Kind regards El 8/2/19 a las 15:52, Holger Levsen escribió: > Hi Antoine, > > On Sun, Feb 03, 2019 at 02:08:06PM +0100, Salvatore Bonaccorso wrote: >> Thanks for working on this. > > indeed! > >> On Fri, Feb 01, 2019 at 01:44:10PM -0500, Antoine Beaupré wrote: >>> On 2018-12-19 18:05:36, Antoine Beaupré wrote: The DLAs are visible here: https://www-staging.debian.org/security/2018/dla-1580 > > that one is also visible on > https://www.debian.org/security/2018/dla-1580 now \o/ > One thing that's unclear is how the entries get added to the main list in: https://www-staging.debian.org/security/2018/ >> IMHO they should not be mixed into the same namespace as the DSAs. >> https://www.debian.org/security/ is very specific to the >> debian-security-announce list and contains items for e.g. contacting >> the Debian security team or referecing the respective FAQ. > > I agree. > > (Thus I think > https://salsa.debian.org/webmaster-team/webwml/merge_requests/50 should > be cloded and not merged.) > > OTOH I plan to review > https://salsa.debian.org/webmaster-team/webwml/merge_requests/53 once > more and then merge it.) > >> I think having a dedicated
Bug#859122: about 500 DLAs missing from the website
On 2018-12-19 18:05:36, Antoine Beaupré wrote: > The DLAs are visible here: > > https://www-staging.debian.org/security/2018/dla-1580 > > One thing that's unclear is how the entries get added to the main list > in: > > https://www-staging.debian.org/security/2018/ > > That still needs to be cleared up. That's actually in the webwml code, I opened a MR to add those: https://salsa.debian.org/webmaster-team/webwml/merge_requests/50 > In the meantime, I did do a mass > import here: > > https://salsa.debian.org/webmaster-team/webwml/merge_requests/47 ... and I just updated that with the latest, up until DLA-1657-1. A. -- It will be a great day when our schools get all the money they need and the air force has to hold a bake sale to buy a bomber. - Unknown
Bug#859122: about 500 DLAs missing from the website
On 2017-03-30 11:22:05, Antoine Beaupre wrote: > Is there any reason why new DLAs have not been imported? > > Is there anything we can do to help in completing that import? So after further research, I can answer my own questions. It's unclear why the process has broken down, but it's clear that the current webmaster team is not in a position to do that work. For DLAs, they do not have the templates they normally use for DSA. I looked at the parse-dsa.pl script and it looks like it might just be possible to batch-import the missing advisories. I started looking into that into the following MRs: https://salsa.debian.org/webmaster-team/webwml/merge_requests/41 https://salsa.debian.org/webmaster-team/webwml/merge_requests/42 https://salsa.debian.org/webmaster-team/webwml/merge_requests/43 And will eventually batch-import everything in one monstrous merge request. Then we need to figure out workflow, which I'll do in that other bug report. A. -- Blind respect for authority is the greatest enemy of truth. - Albert Einstein
Bug#859122: about 500 DLAs missing from the website
Package: www.debian.org
Severity: normal
Hi!
First, thanks for doing the work of importing DLAs and DSAs in the
website, it is greatly appreciated.
However, during a discussion on the debian-lts@ mailing list, we have
noticed that DLAs since squeeze LTS support was terminated have not
been imported:
https://lists.debian.org/debian-lts/2017/03/msg00205.html
An excerpt from the discussion:
> Here's the bits that are missing:
>
> * the last DLA on the website is DLA-445-2, which is basically the last
>DLA before squeeze support ended and wheezy was handed over
>
> * among those 445 DLAs, there are actually 31 missing:
>
>webwml$ cd english/security/; find -name 'dla-*.wml' | wc -l
>424
>
> * even worse, it seems there are at least 20 advisories missing from
>the website because regression uploads hide advisories, because our
>naming convention differs from DSA ("DLA-XXX-N", where XXX is the
>original advisory and N are regression updates)
>
>$ grep DLA- data/DLA/list | sed 's/.* DLA-//;s/ .*//' | sort -n | sed
> '/445-2/,$d' | wc -l
>465
>
> * the canonical list has 928 advisories:
>
>secure-testing$ grep DLA- data/DLA/list | wc -l
>928
Is there any reason why new DLAs have not been imported?
Is there anything we can do to help in completing that import?
I will open a separate ticket regarding possible automation shortly.
Thanks!
-- System Information:
Debian Release: 9.0
APT prefers testing
APT policy: (500, 'testing'), (1, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: armhf
Kernel: Linux 4.9.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_CA.UTF-8, LC_CTYPE=fr_CA.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
