Re: Bug#761945: fixing links for DLAs in the security tracker
On 2017-03-30 08:38:07, Salvatore Bonaccorso wrote:
> Hi Antoine,
>
> On Wed, Mar 29, 2017 at 03:49:31PM -0400, Antoine Beaupré wrote:
>> On 2017-03-29 17:02:44, Salvatore Bonaccorso wrote:
>> > Hi Antoine,
>>
>> Hi!
>>
>> > If you want to look at this part: There is a ./parse-dla.pl script in
>> > the webwml CVS, which is used to import the DLAs (this is an
>> > analogeous script to parse-advisory.pl which is used to import the
>> > DSAs).
>>
>> I see... The scripts are in /english/security for anyone looking. And if
>> people are (like me) thinking "... wat.. CVS?" then yes, we are still
>> using this:
>>
>> https://www.debian.org/devel/website/using_cvs
>>
>> My cvs commandline finger memory is *definitely* still there though, so
>> that works for me. :)
>>
>> > The "manual" steps one would perform are roughly:
>> >
>> > ./parse-dla.pl $message
>> > cvs add $year/dla-$nr.{wml,data}
>> > cvs commit -m '[DLA $nr] $source security update'
>>
>> Is this something the security team performs as part of the DSA release
>> process? Or is this something the debian-www people do? I guess you need
>> write access to the repository and I see that *you* do, but is this
>> expected from everyone working on releasing public advisories, the same
>> way we need access to the security tracker?
>
> No it's not something we do as part of a regular DSA releasing
> process, and as well not expected to do so, as the websites are under
> debian-www "domain" (and btw, they do a great job!). But often, when I
> have time I do as well the import (but as you will see from cvs log,
> not always). For the security team the current process is: preparing
> the DSA (packages, tracker work, text, releasing packages), send out
> the advisory (at this stage our work is basically done).
Okay, so this is just something the www team needs to catchup with then...
>> And to import older entries, we'll need the original templates, which we
>> deliberately did *not* commit anywhere, so they are basically available
>> only as mailing list archives, and thus hard to find automatically.
>
> But given the debian-lts-announce is archived, shouldn't it be
> relatively easy to frist grab all announces from
> https://lists.debian.org/debian-lts-announce/ then check which one
> need to still be imported, extract the mail and do the import? Or do I
> missunderstand you?
I was assuming I was just web-browsing, but it's true I can probably
access the mailboxes and grep through this directly. It's still a pain
in the butt. ;)
>> I foresee difficulties in importing the missing data...
>>
>> Here's the bits that are missing:
>>
>> * the last DLA on the website is DLA-445-2, which is basically the last
>>DLA before squeeze support ended and wheezy was handed over
>>
>> * among those 445 DLAs, there are actually 31 missing:
>>
>>webwml$ cd english/security/; find -name 'dla-*.wml' | wc -l
>>424
>>
>> * even worse, it seems there are at least 20 advisories missing from
>>the website because regression uploads hide advisories, because our
>>naming convention differs from DSA ("DLA-XXX-N", where XXX is the
>>original advisory and N are regression updates)
>
> I do not understand this point. What do you mean by hinding? For DSA's
> as well only one https://www.debian.org/security/$year/dsa-$nr is ever
> visible as well (and it depends if the text has been then updated
> according to a regression update or not, and in DLAs cases I guess
> just only the last iteration might has been imported, not the initial
> -1 one).
Here's an example. DLA-445-1 was a Squid upload to squeeze, announced
here:
https://lists.debian.org/debian-lts-announce/2016/02/msg00037.html
it caused a regression, which was fixed in DLA-445-2, announced here:
https://lists.debian.org/debian-lts-announce/2016/03/msg1.html
You can see both DLAs in the sectracker:
https://security-tracker.debian.org/tracker/DLA-445-1
https://security-tracker.debian.org/tracker/DLA-445-2
(Note, BTW, that the regression update doesn't refer to the previous DLA
or the CVE, you just need to know the convention to figure that out.)
On the website, you only see the regression update:
https://www.debian.org/security/2016/dla-445
That is the equivalent of DLA-445-2. I am not sure that DLA-445-1 is
anywhere.
I guess that's another bug to report as well?
[...]
>> > having something on the debian- side which does this
>> > automatically, once a DSA or DLA arrives would help surely the
>> > debian-www team who then "only" have to do the translations and fix
>> > obvious mistakes. OTOH keep in mind: When the debian- team imports
>> > a DSA or DLA they may need to do some adjustments so, I'm not sure if
>> > it's liked to have the automatism, since sometimes before cvs commit
>> > some changes need to be done on the .wml file.
>>
>> It looks like this is something that should be discussed with the www
>> people... Maybe a bug against www.debian.org?
>
> I think
Re: Bug#761945: fixing links for DLAs in the security tracker
Hi Antoine,
On Wed, Mar 29, 2017 at 03:49:31PM -0400, Antoine Beaupré wrote:
> On 2017-03-29 17:02:44, Salvatore Bonaccorso wrote:
> > Hi Antoine,
>
> Hi!
>
> > If you want to look at this part: There is a ./parse-dla.pl script in
> > the webwml CVS, which is used to import the DLAs (this is an
> > analogeous script to parse-advisory.pl which is used to import the
> > DSAs).
>
> I see... The scripts are in /english/security for anyone looking. And if
> people are (like me) thinking "... wat.. CVS?" then yes, we are still
> using this:
>
> https://www.debian.org/devel/website/using_cvs
>
> My cvs commandline finger memory is *definitely* still there though, so
> that works for me. :)
>
> > The "manual" steps one would perform are roughly:
> >
> > ./parse-dla.pl $message
> > cvs add $year/dla-$nr.{wml,data}
> > cvs commit -m '[DLA $nr] $source security update'
>
> Is this something the security team performs as part of the DSA release
> process? Or is this something the debian-www people do? I guess you need
> write access to the repository and I see that *you* do, but is this
> expected from everyone working on releasing public advisories, the same
> way we need access to the security tracker?
No it's not something we do as part of a regular DSA releasing
process, and as well not expected to do so, as the websites are under
debian-www "domain" (and btw, they do a great job!). But often, when I
have time I do as well the import (but as you will see from cvs log,
not always). For the security team the current process is: preparing
the DSA (packages, tracker work, text, releasing packages), send out
the advisory (at this stage our work is basically done).
> And to import older entries, we'll need the original templates, which we
> deliberately did *not* commit anywhere, so they are basically available
> only as mailing list archives, and thus hard to find automatically.
But given the debian-lts-announce is archived, shouldn't it be
relatively easy to frist grab all announces from
https://lists.debian.org/debian-lts-announce/ then check which one
need to still be imported, extract the mail and do the import? Or do I
missunderstand you?
> I foresee difficulties in importing the missing data...
>
> Here's the bits that are missing:
>
> * the last DLA on the website is DLA-445-2, which is basically the last
>DLA before squeeze support ended and wheezy was handed over
>
> * among those 445 DLAs, there are actually 31 missing:
>
>webwml$ cd english/security/; find -name 'dla-*.wml' | wc -l
>424
>
> * even worse, it seems there are at least 20 advisories missing from
>the website because regression uploads hide advisories, because our
>naming convention differs from DSA ("DLA-XXX-N", where XXX is the
>original advisory and N are regression updates)
I do not understand this point. What do you mean by hinding? For DSA's
as well only one https://www.debian.org/security/$year/dsa-$nr is ever
visible as well (and it depends if the text has been then updated
according to a regression update or not, and in DLAs cases I guess
just only the last iteration might has been imported, not the initial
-1 one).
>$ grep DLA- data/DLA/list | sed 's/.* DLA-//;s/ .*//' | sort -n | sed
> '/445-2/,$d' | wc -l
>465
>
> * the canonical list has 928 advisories:
>
>secure-testing$ grep DLA- data/DLA/list | wc -l
>928
>
> So, lots of work there.
>
> > The background work leading to that was done by Frank Lichtenheld in
> > #762255.
>
> Great to see that! It does seem problematic to import regression updates
> however.
>
> > having something on the debian- side which does this
> > automatically, once a DSA or DLA arrives would help surely the
> > debian-www team who then "only" have to do the translations and fix
> > obvious mistakes. OTOH keep in mind: When the debian- team imports
> > a DSA or DLA they may need to do some adjustments so, I'm not sure if
> > it's liked to have the automatism, since sometimes before cvs commit
> > some changes need to be done on the .wml file.
>
> It looks like this is something that should be discussed with the www
> people... Maybe a bug against www.debian.org?
I think yes on looking further at this with a bug against
www.debian.org and the debian www team. In particular to find out why
DLA imports ended, and if someone is willing to help doing the
remaining task. And the other aspect is if DSA and DLA imports should
be automated (and problems in the wml fixed later on manually, which
will be detected since they might cause cron errors mails to the
debian-www team).
> This begs the question, however - wouldn't it be simpler to import those
> advisories in the security tracker directly?
Feel free to, say for example data/DLA/advisories/ (or some other
directory, but below the data/DLA "namespace"); As the
testing-security team did for a while, for historical view look at
data/DTSA/advs. This is my opinion: But for displaying
Re: Bug#761945: fixing links for DLAs in the security tracker
On 2017-03-29 17:02:44, Salvatore Bonaccorso wrote:
> Hi Antoine,
Hi!
> If you want to look at this part: There is a ./parse-dla.pl script in
> the webwml CVS, which is used to import the DLAs (this is an
> analogeous script to parse-advisory.pl which is used to import the
> DSAs).
I see... The scripts are in /english/security for anyone looking. And if
people are (like me) thinking "... wat.. CVS?" then yes, we are still
using this:
https://www.debian.org/devel/website/using_cvs
My cvs commandline finger memory is *definitely* still there though, so
that works for me. :)
> The "manual" steps one would perform are roughly:
>
> ./parse-dla.pl $message
> cvs add $year/dla-$nr.{wml,data}
> cvs commit -m '[DLA $nr] $source security update'
Is this something the security team performs as part of the DSA release
process? Or is this something the debian-www people do? I guess you need
write access to the repository and I see that *you* do, but is this
expected from everyone working on releasing public advisories, the same
way we need access to the security tracker?
And to import older entries, we'll need the original templates, which we
deliberately did *not* commit anywhere, so they are basically available
only as mailing list archives, and thus hard to find automatically.
I foresee difficulties in importing the missing data...
Here's the bits that are missing:
* the last DLA on the website is DLA-445-2, which is basically the last
DLA before squeeze support ended and wheezy was handed over
* among those 445 DLAs, there are actually 31 missing:
webwml$ cd english/security/; find -name 'dla-*.wml' | wc -l
424
* even worse, it seems there are at least 20 advisories missing from
the website because regression uploads hide advisories, because our
naming convention differs from DSA ("DLA-XXX-N", where XXX is the
original advisory and N are regression updates)
$ grep DLA- data/DLA/list | sed 's/.* DLA-//;s/ .*//' | sort -n | sed
'/445-2/,$d' | wc -l
465
* the canonical list has 928 advisories:
secure-testing$ grep DLA- data/DLA/list | wc -l
928
So, lots of work there.
> The background work leading to that was done by Frank Lichtenheld in
> #762255.
Great to see that! It does seem problematic to import regression updates
however.
> having something on the debian- side which does this
> automatically, once a DSA or DLA arrives would help surely the
> debian-www team who then "only" have to do the translations and fix
> obvious mistakes. OTOH keep in mind: When the debian- team imports
> a DSA or DLA they may need to do some adjustments so, I'm not sure if
> it's liked to have the automatism, since sometimes before cvs commit
> some changes need to be done on the .wml file.
It looks like this is something that should be discussed with the www
people... Maybe a bug against www.debian.org?
This begs the question, however - wouldn't it be simpler to import those
advisories in the security tracker directly?
At least, we should figure out why the imports have ceased after
wheezy-LTS started...
> Writing the above a bit in a hurry let me know if unclear what I
> meant.
Thanks for the response!
A.
--
What this country needs is more unemployed politicians.
- Angela Davis
Re: Bug#761945: fixing links for DLAs in the security tracker
Hi Antoine,
On Wed, Mar 29, 2017 at 10:33:34AM -0400, Antoine Beaupré wrote:
> On 2017-03-29 07:29:06, Salvatore Bonaccorso wrote:
> > Hi,
> >
> > On Wed, Mar 29, 2017 at 06:28:49AM +0200, Salvatore Bonaccorso wrote:
> >> Hi,
> >>
> >> On Tue, Mar 28, 2017 at 10:16:52PM +, Holger Levsen wrote:
> >> > On Tue, Mar 28, 2017 at 10:35:34PM +0200, Moritz Muehlenhoff wrote:
> >> > > Well, you don't have a web site comparable to
> >> > > https://www.debian.org/security/2017/dsa-3796, so where should
> >> > > it possibly link to?
> >> >
> >> > I guess it's time to create this "web site" then :)
> >>
> >> See as well https://bugs.debian.org/761945 (and respective clones for
> >> debian-).
> >
> > The security-tracker side of this has been implemented now, Paul Wise
> > did the corresponding work. But around 400 DLA's are not yet imported
> > so many links will sow a page not found.
> >
> > A working example:
> > https://security-tracker.debian.org/tracker/DLA-55-1 or
> > https://security-tracker.debian.org/tracker/DLA-400-1
>
> So I guess the next steps are for the LTS team:
>
> 1. update the documentation so that updating the website is part of the
> workflow
>
> 2. import old DLA advisories into the websites
>
> I can try and complete this by the end of the week.
If you want to look at this part: There is a ./parse-dla.pl script in
the webwml CVS, which is used to import the DLAs (this is an
analogeous script to parse-advisory.pl which is used to import the
DSAs).
The "manual" steps one would perform are roughly:
./parse-dla.pl $message
cvs add $year/dla-$nr.{wml,data}
cvs commit -m '[DLA $nr] $source security update'
The background work leading to that was done by Frank Lichtenheld in
#762255.
having something on the debian- side which does this
automatically, once a DSA or DLA arrives would help surely the
debian-www team who then "only" have to do the translations and fix
obvious mistakes. OTOH keep in mind: When the debian- team imports
a DSA or DLA they may need to do some adjustments so, I'm not sure if
it's liked to have the automatism, since sometimes before cvs commit
some changes need to be done on the .wml file.
Writing the above a bit in a hurry let me know if unclear what I
meant.
Regards,
Salvatore
