Re: [dns-operations] DNSViz Service Restoration

[Jim Popovitch via dns-operations]

--- Begin Message ---
On Thu, 2020-03-12 at 08:41 -0400, Matthew Pounsett wrote:
> > On Mar 12, 2020, at 07:04, Jim Popovitch via dns-operations 
> >  wrote:
> > 
> > 
> > From: Jim Popovitch 
> > Subject: Re: [dns-operations] DNSViz Service Restoration
> > Date: March 12, 2020 at 07:04:23 EDT
> > To: dns-operations@lists.dns-oarc.net
> > 
> > 
> > On March 12, 2020 5:04:23 AM UTC, Casey Deccio  wrote:
> > > Thanks for the perspective.  I believe there is value in being able to 
> > > answer the question: "what did foo.example.net look like at time X?"
> > 
> > Sounds great.  I think the most important feature of dnsvis was the ability 
> > to link to a report to show a recent problem to others.  People haven't had 
> > that capability, in over a year, because someone else saw greater value in 
> > being able to show very very very old data.
> 
> While the snark may have sounded witty in your head, the decision-
> making was a actually a lot less obvious than that.

I apologize. There was no snark intended.  Dnsviz was a valuable
resource for many years, it suddenly went off line and reappeared with a
significant loss of functionality.  People were promised over and over,
over the course of a year, that the historical aspect and data were
going to re-appear. In the end, the data never appeared, but the part
that most people loved has.  Thank you for finally delivering it.


> Had we known it was going to be a year of hacking at a broken
> database, of course we’d have taken this route in the first place. 
> But, when we first found that some corruption had been introduced it
> wasn’t obvious that would take very long to fix.  At all decision
> points along the way, it appeared as if we were no more than a month
> from having a functioning historical database.
> 
> At the OARC workshop in October, we thought we were hours away from
> announcing that it was back up and running with all of its historical
> data, but the import script running at that time was interrupted by
> the DB running up against its transaction limit, and we had to start a
> vacuum of the db.  That ran for another six weeks before failing on a
> full disk.
> 
> About six months in we started to consider the possibility of
> resetting the database and merging old data later, but that’s a much
> more complicated procedure as it involves both restructuring the
> corruption that broken the import in the first place AND massaging
> that data on import to avoid collisions with newly created rows that
> have unique constraints on them, all on top of the increased time it
> would take to do such an import while the service is active.  There’s
> also the risk that certain tests could never be imported as-is because
> of the potential of a new test’s reference name (the unique 6
> characters in a specific test’s URL) colliding with an old test’s
> name, causing any stored URLs out there to show the wrong test data.
> 
> And Casey isn’t the only one who looks at—or links to—old tests; there
> are web sites out there with links to old tests used as a historical
> record or as case studies of the ways DNS can be broken, so it still
> seems useful to get those tests back online somehow.


Thank you for those details, they make for an interesting postmortem. 

-Jim P.

--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] DNSViz Service Restoration

[Matthew Pounsett]

> On Mar 12, 2020, at 07:04, Jim Popovitch via dns-operations 
>  wrote:
> 
> 
> From: Jim Popovitch 
> Subject: Re: [dns-operations] DNSViz Service Restoration
> Date: March 12, 2020 at 07:04:23 EDT
> To: dns-operations@lists.dns-oarc.net
> 
> 
> On March 12, 2020 5:04:23 AM UTC, Casey Deccio  wrote:
>> 
>> Thanks for the perspective.  I believe there is value in being able to 
>> answer the question: "what did foo.example.net look like at time X?"
> 
> Sounds great.  I think the most important feature of dnsvis was the ability 
> to link to a report to show a recent problem to others.  People haven't had 
> that capability, in over a year, because someone else saw greater value in 
> being able to show very very very old data.

While the snark may have sounded witty in your head, the decision-making was a 
actually a lot less obvious than that.

Had we known it was going to be a year of hacking at a broken database, of 
course we’d have taken this route in the first place.  But, when we first found 
that some corruption had been introduced it wasn’t obvious that would take very 
long to fix.  At all decision points along the way, it appeared as if we were 
no more than a month from having a functioning historical database.

At the OARC workshop in October, we thought we were hours away from announcing 
that it was back up and running with all of its historical data, but the import 
script running at that time was interrupted by the DB running up against its 
transaction limit, and we had to start a vacuum of the db.  That ran for 
another six weeks before failing on a full disk.

About six months in we started to consider the possibility of resetting the 
database and merging old data later, but that’s a much more complicated 
procedure as it involves both restructuring the corruption that broken the 
import in the first place AND massaging that data on import to avoid collisions 
with newly created rows that have unique constraints on them, all on top of the 
increased time it would take to do such an import while the service is active.  
There’s also the risk that certain tests could never be imported as-is because 
of the potential of a new test’s reference name (the unique 6 characters in a 
specific test’s URL) colliding with an old test’s name, causing any stored URLs 
out there to show the wrong test data.

And Casey isn’t the only one who looks at—or links to—old tests; there are web 
sites out there with links to old tests used as a historical record or as case 
studies of the ways DNS can be broken, so it still seems useful to get those 
tests back online somehow.

Matt Pounsett
DNS-OARC Systems Engineering





signature.asc
Description: Message signed with OpenPGP
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] DNSViz Service Restoration

[Jim Popovitch via dns-operations]

--- Begin Message ---
On March 12, 2020 5:04:23 AM UTC, Casey Deccio  wrote:
>
>Thanks for the perspective.  I believe there is value in being able to answer 
>the question: "what did foo.example.net look like at time X?"

Sounds great.  I think the most important feature of dnsvis was the ability to 
link to a report to show a recent problem to others.  People haven't had that 
capability, in over a year, because someone else saw greater value in being 
able to show very very very old data.

-Jim P.

--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] DNSViz Service Restoration

[Landi Ahmed]

Good stuff.

Keep with the good job you are doing.




*__**___**_**___**___**_**___**__**_**___**__**_**_*
Mit freundlichen Grüßen / Kind regards.

Landi, Ahmed
Email - a hmedland...@gmail.com
Phone - +254 704 316 945


On Wed, Mar 11, 2020 at 10:40 PM Matthew Pounsett  wrote:

> Hi all!
>
> OARC is happy… no, ecstatic… to announce that the DNSViz historical
> functions have been restored!  Users will now be seeing full functionality
> from the site at .
>
> A few weeks ago we made the decision to temporarily put aside the attempt
> to completely restore the old historical data and instead stand up a new,
> empty database so that we could get the full featureset online.  So, for
> now there is no access to old tests run prior to the service’s move to
> OARC, however new tests will be available for review.
>
> We’re continuing to work to restore the full historical database; we hope
> that with the pressure off, and the temptation to cut corners in order to
> speed up the process removed, we can restart the import from scratch with a
> slower—but more reliable—approach to recovering those data.
>
> I’d like to thank everyone again for your patience with this whole saga.
>
> Matt Pounsett
> DNS-OARC Systems Engineering
>
>
>
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] DNSViz Service Restoration

[Casey Deccio]

> On Mar 11, 2020, at 2:03 PM, Warren Kumari  wrote:
> 
> On Wed, Mar 11, 2020 at 3:44 PM Matthew Pounsett  wrote:
>> A few weeks ago we made the decision to temporarily put aside the attempt to 
>> completely restore the old historical data and instead stand up a new, empty 
>> database so that we could get the full featureset online.  So, for now there 
>> is no access to old tests run prior to the service’s move to OARC, however 
>> new tests will be available for review.
>> 
>> We’re continuing to work to restore the full historical database; we hope 
>> that with the pressure off, and the temptation to cut corners in order to 
>> speed up the process removed, we can restart the import from scratch with a 
>> slower—but more reliable—approach to recovering those data.
> 
> 
> Excuse my not remembering, but have y'all confirmed that this is
> really worth the faff? What *I* care about is being able to use the
> service *from now on*- going back and seeing the breakage of
> foo.example.net in 2018 is only mildly interesting, but certainly not
> (to me!) worth your time and effort...

Thanks for the perspective.  I believe there is value in being able to answer 
the question: "what did foo.example.net look like at time X?", though it is 
arguably less important to most users, and the value diminishes over time.  
Much more important, in my opinion, is understanding trends over time.  For 
example, can we understand what the current pitfalls are and how to improve 
them, so we can get off NTAs and be more strict in DNSSEC validation adherence? 
 In other words, looking at the aggregate offered by the historical data is 
much more valuable than looking at individual setups.  An understanding of 
DNS(SEC) deployment behavior can be used to improve tools, processes, and 
standards.

Casey
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] DNSViz Service Restoration

[Georg Kahest]

Great news!

Thank you for your effort.

On 11.03.20 21:31, Matthew Pounsett wrote:
> Hi all!
>
> OARC is happy… no, ecstatic… to announce that the DNSViz historical functions 
> have been restored!  Users will now be seeing full functionality from the 
> site at .
>
> A few weeks ago we made the decision to temporarily put aside the attempt to 
> completely restore the old historical data and instead stand up a new, empty 
> database so that we could get the full featureset online.  So, for now there 
> is no access to old tests run prior to the service’s move to OARC, however 
> new tests will be available for review.
>
> We’re continuing to work to restore the full historical database; we hope 
> that with the pressure off, and the temptation to cut corners in order to 
> speed up the process removed, we can restart the import from scratch with a 
> slower—but more reliable—approach to recovering those data.
>
> I’d like to thank everyone again for your patience with this whole saga.
>
> Matt Pounsett
> DNS-OARC Systems Engineering
>
>
>
>
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] DNSViz Service Restoration

[Marco Davids (SIDN) via dns-operations]

--- Begin Message ---
Awesome!

Thanks.

--
Marco

Op 11-03-20 om 20:31 schreef Matthew Pounsett:

> OARC is happy… no, ecstatic… to announce that the DNSViz historical functions 
> have been restored!
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] DNSViz Service Restoration

[Warren Kumari]

On Wed, Mar 11, 2020 at 3:44 PM Matthew Pounsett  wrote:
>
> Hi all!
>
> OARC is happy… no, ecstatic… to announce that the DNSViz historical functions 
> have been restored!  Users will now be seeing full functionality from the 
> site at .
>

Awesome, thank you all.

> A few weeks ago we made the decision to temporarily put aside the attempt to 
> completely restore the old historical data and instead stand up a new, empty 
> database so that we could get the full featureset online.  So, for now there 
> is no access to old tests run prior to the service’s move to OARC, however 
> new tests will be available for review.
>
> We’re continuing to work to restore the full historical database; we hope 
> that with the pressure off, and the temptation to cut corners in order to 
> speed up the process removed, we can restart the import from scratch with a 
> slower—but more reliable—approach to recovering those data.


Excuse my not remembering, but have y'all confirmed that this is
really worth the faff? What *I* care about is being able to use the
service *from now on*- going back and seeing the breakage of
foo.example.net in 2018 is only mildly interesting, but certainly not
(to me!) worth your time and effort...

W

>
> I’d like to thank everyone again for your patience with this whole saga.
>
> Matt Pounsett
> DNS-OARC Systems Engineering
>
>
>
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations



-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] DNSViz Service Restoration

[negativeindex]

That is excellent news indeed!  Thank you for all your hard work on this.

On Wed, Mar 11, 2020 at 12:40 PM Matthew Pounsett  wrote:

> Hi all!
>
> OARC is happy… no, ecstatic… to announce that the DNSViz historical
> functions have been restored!  Users will now be seeing full functionality
> from the site at .
>
> A few weeks ago we made the decision to temporarily put aside the attempt
> to completely restore the old historical data and instead stand up a new,
> empty database so that we could get the full featureset online.  So, for
> now there is no access to old tests run prior to the service’s move to
> OARC, however new tests will be available for review.
>
> We’re continuing to work to restore the full historical database; we hope
> that with the pressure off, and the temptation to cut corners in order to
> speed up the process removed, we can restart the import from scratch with a
> slower—but more reliable—approach to recovering those data.
>
> I’d like to thank everyone again for your patience with this whole saga.
>
> Matt Pounsett
> DNS-OARC Systems Engineering
>
>
>
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] DNSViz Service Restoration

[Scott Morizot]

Fantastic news! Thanks. Already checked and confirmed. Access to the
responses is especially appreciated.

On Wed, Mar 11, 2020 at 2:40 PM Matthew Pounsett  wrote:

> Hi all!
>
> OARC is happy… no, ecstatic… to announce that the DNSViz historical
> functions have been restored!  Users will now be seeing full functionality
> from the site at .
>
> A few weeks ago we made the decision to temporarily put aside the attempt
> to completely restore the old historical data and instead stand up a new,
> empty database so that we could get the full featureset online.  So, for
> now there is no access to old tests run prior to the service’s move to
> OARC, however new tests will be available for review.
>
> We’re continuing to work to restore the full historical database; we hope
> that with the pressure off, and the temptation to cut corners in order to
> speed up the process removed, we can restart the import from scratch with a
> slower—but more reliable—approach to recovering those data.
>
> I’d like to thank everyone again for your patience with this whole saga.
>
> Matt Pounsett
> DNS-OARC Systems Engineering
>
>
>
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

[dns-operations] DNSViz Service Restoration

[Matthew Pounsett]

Hi all!

OARC is happy… no, ecstatic… to announce that the DNSViz historical functions 
have been restored!  Users will now be seeing full functionality from the site 
at .

A few weeks ago we made the decision to temporarily put aside the attempt to 
completely restore the old historical data and instead stand up a new, empty 
database so that we could get the full featureset online.  So, for now there is 
no access to old tests run prior to the service’s move to OARC, however new 
tests will be available for review.

We’re continuing to work to restore the full historical database; we hope that 
with the pressure off, and the temptation to cut corners in order to speed up 
the process removed, we can restart the import from scratch with a slower—but 
more reliable—approach to recovering those data.

I’d like to thank everyone again for your patience with this whole saga.

Matt Pounsett
DNS-OARC Systems Engineering





signature.asc
Description: Message signed with OpenPGP
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations