Re: KASP Key Rollover: ZSK Disappears Immediately

[Matthijs Mekking]

Hi Nick,

The timings are based on what is configured in the dnssec-policy: It is 
too costly to observe the zone every time to see if there is still a 
signature of the predecessor key. So yes: it takes the maximum possible 
time to determine when all signatures have been replaced.


This time is Iret (based on RFC 7583) and the main portion of that is 
Dsgn, the delay needed to ensure that all existing RRsets have been 
re-signed with the new key. The maximum sign delay is:


Dsgn = (signatures-validity - signatures-refresh)

In the default policy this is indeed 9 days.

I still suspect that the original issue from Eddie was because the KSK 
had its DS state in hidden.


Best regards,

Matthijs

On 11/13/23 09:55, Nick Tait via bind-users wrote:

On 03/10/2023 09:59, Eddie Rowe wrote:

I appreciate the feedback.  I did make sure the ZSK is omnipresent and the
issue still happens so it might be that my attempt to take the default 
policy
and bring it down to 1 day to hurry along testing.  I will see if I 
can find
any test policies in the list archives and failing that use the 
default one

with a greater amount of patience.

Hi Eddie.

Not sure if you're still interested in this topic, but a couple of weeks 
ago I did a manual ZSK roll-over, to see if I observed results similar 
to what you described in your original email...


Before starting the rollover /everything/ was showing omnipresent.

After initiating the rollover things mostly happened in the expected 
timeframes, but there was one thing that surprised me: The old ZSK 
removal date was set 9-and-a-bit days(!) after the roll-over was 
initiated, and the old ZSK and new ZSK state files showed "ZRRSIGState: 
unretentive" and "ZRRSIGState: rumoured" (respectively) right up until 
the old ZSK removal date, before eventually transitioning to the 
expected end states of "ZRRSIGState: hidden" and "ZRRSIGState: 
omnipresent" (respectively). This was in spite of the fact that all 
RRSIG records were replaced with the new ZSK more than a week prior. I 
can only assume that the 9 days somehow relates to how long BIND wanted 
to allow itself to generate RRSIGs for all the records in a really, 
really large zone file?


Anyway, I remembered seeing "ZRRSIGState: rumoured" in your ZSK state 
file before you initiated your ZSK roll-over, and so I suspect that all 
your issues stem from the fact that not everything was omnipresent 
before you initiated the roll-over?


Nick.


--
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: KASP Key Rollover: ZSK Disappears Immediately

[Nick Tait via bind-users]

On 03/10/2023 09:59, Eddie Rowe wrote:

I appreciate the feedback.  I did make sure the ZSK is omnipresent and the
issue still happens so it might be that my attempt to take the default 
policy
and bring it down to 1 day to hurry along testing.  I will see if I 
can find
any test policies in the list archives and failing that use the 
default one

with a greater amount of patience.

Hi Eddie.

Not sure if you're still interested in this topic, but a couple of weeks 
ago I did a manual ZSK roll-over, to see if I observed results similar 
to what you described in your original email...


Before starting the rollover /everything/ was showing omnipresent.

After initiating the rollover things mostly happened in the expected 
timeframes, but there was one thing that surprised me: The old ZSK 
removal date was set 9-and-a-bit days(!) after the roll-over was 
initiated, and the old ZSK and new ZSK state files showed "ZRRSIGState: 
unretentive" and "ZRRSIGState: rumoured" (respectively) right up until 
the old ZSK removal date, before eventually transitioning to the 
expected end states of "ZRRSIGState: hidden" and "ZRRSIGState: 
omnipresent" (respectively). This was in spite of the fact that all 
RRSIG records were replaced with the new ZSK more than a week prior. I 
can only assume that the 9 days somehow relates to how long BIND wanted 
to allow itself to generate RRSIGs for all the records in a really, 
really large zone file?


Anyway, I remembered seeing "ZRRSIGState: rumoured" in your ZSK state 
file before you initiated your ZSK roll-over, and so I suspect that all 
your issues stem from the fact that not everything was omnipresent 
before you initiated the roll-over?


Nick.-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: KASP Key Rollover: ZSK Disappears Immediately

[Eddie Rowe]

I appreciate the feedback.  I did make sure the ZSK is omnipresent and the 
issue still happens so it might be that my attempt to take the default policy 
and bring it down to 1 day to hurry along testing.  I will see if I can find 
any test policies in the list archives and failing that use the default one 
with a greater amount of patience.

From: bind-users  on behalf of Nick Tait via 
bind-users 
Sent: Friday, September 29, 2023 5:37 PM
To: bind-users@lists.isc.org 
Subject: Re: KASP Key Rollover: ZSK Disappears Immediately


Sorry I just realised that all that waffle about DS records is only relevant 
for KSKs (and CSKs), not ZSKs. So please disregard that. :-P


But I think the "rumoured" vs. "omnipresent" thing is still relevant and is the 
most likely explanation for why the old ZSK doesn't stick around. I can only 
assume that the reason you have rumoured state is because you are trying to 
roll your ZSK to soon after the previous ZSK rollover? Have you checked the 
various timing settings in the KASP definition?


Nick.



On 30/09/23 11:32, Nick Tait via bind-users wrote:
On 29/09/23 12:05, Eddie Rowe wrote:
When I perform a ZSK key rollover the existing ZSK disappears immediately so 
not sure what I am missing when using the KASP to manage key rollover.  The 
state for the keys looks good and for this test I have TTL set to 1 hour..  But 
why does dig not show me both DNSKEY records for the ZSK after I initiate the 
rollover when there should be overlap as described in Automatic DNSSEC Zone 
Signing Key rollover explained (isc.org)<https://kb.isc.org/docs/aa-00822>?

Bind 9.16.23 which seems to be the newest release provided by my distribution.  
I reviewed the ARM for notes for newer releases in the 9.16 branch and did not 
see mention of any rollover bugs or for dig.

  1.   Here is the key info from dig for ZSK key 15465 at 17:17.

# dig @localhost myexample.com DNSKEY +multi

; <<>> DiG 9.16.23-RH <<>> @localhost myexample.com DNSKEY +multi
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41895
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 7c2a0e61926d2d3a01006515fb68eef12b631ca40c20 (good)
;; QUESTION SECTION:
;myexample.com. IN DNSKEY

;; ANSWER SECTION:
myexample.com.  3600 IN DNSKEY 257 3 13 (
20agIXl9sQCo00yiHHviYWZG8TvVmDoVxPJwO3mlcwxB
le7UNrzNQaeukC6teT4XrqYflqDxcM6d9L/mtREIKA==
) ; KSK; alg = ECDSAP256SHA256 ; key id = 31296
myexample.com.  3600 IN DNSKEY 256 3 13 (
AlKXH5aebvboC4laAovc6wfg6uGK1uTbTqYYnhKadSq6
78nSI3DyM+1t91jqQ81tlBy+e3hJyKtlX/OiOhuZcA==
) ; ZSK; alg = ECDSAP256SHA256 ; key id = 15465

;; Query time: 3 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 28 17:17:12 CDT 2023
;; MSG SIZE  rcvd: 230


  1.   Here is the info from the key as far as state goes.

# more Kmyexample.com.+013+15465.key
; This is a zone-signing key, keyid 15465, for myexample.com.
; Created: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Publish: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Activate: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Inactive: 20231127221438 (Mon Nov 27 16:14:38 2023)
; Delete: 20231207231938 (Thu Dec  7 17:19:38 2023)
myexample.com. 3600 IN DNSKEY 256 3 13 
AlKXH5aebvboC4laAovc6wfg6uGK1uTbTqYYnhKadSq678nSI3DyM+1t 
91jqQ81tlBy+e3hJyKtlX/OiOhuZcA==

# more Kmyexample.com.+013+15465.state
; This is the state of key 15465, for myexample.com.
Algorithm: 13
Length: 256
Lifetime: 5184000
KSK: no
ZSK: yes
Generated: 20230928221438 (Thu Sep 28 17:14:38 2023)
Published: 20230928221438 (Thu Sep 28 17:14:38 2023)
Active: 20230928221438 (Thu Sep 28 17:14:38 2023)
Retired: 20231127221438 (Mon Nov 27 16:14:38 2023)
Removed: 20231207231938 (Thu Dec  7 17:19:38 2023)
DNSKEYChange: 20230928221438 (Thu Sep 28 17:14:38 2023)
ZRRSIGChange: 20230928221438 (Thu Sep 28 17:14:38 2023)
DNSKEYState: rumoured
ZRRSIGState: rumoured
GoalState: omnipresent

I suspect that the crucial detail above is "DNSKEYState: rumoured". This 
suggests that the last ZSK rollover hasn't been fully completed.

Before starting a rollover it is a good idea to make sure the ZSK that you are 
retiring is in an "omnipresent" state.


The usual reason that the key isn't in omnipresent state is because BIND is 
still waiting for the corresponding DS record to be published and available in 
the parent zone. BIND 9.16 only knows if the DS record is published if you've 
set up parental-agents, or if you've explicitly told it using rndc. (BTW BIND 
9.19 introduces new default behaviour which means you

Re: KASP Key Rollover: ZSK Disappears Immediately

[Nick Tait via bind-users]

Sorry I just realised that all that waffle about DS records is only 
relevant for KSKs (and CSKs), not ZSKs. So please disregard that. :-P



But I think the "rumoured" vs. "omnipresent" thing is still relevant and 
is the most likely explanation for why the old ZSK doesn't stick around. 
I can only assume that the reason you have rumoured state is because you 
are trying to roll your ZSK to soon after the previous ZSK rollover? 
Have you checked the various timing settings in the KASP definition?



Nick.



On 30/09/23 11:32, Nick Tait via bind-users wrote:

On 29/09/23 12:05, Eddie Rowe wrote:
When I perform a ZSK key rollover the existing ZSK disappears 
*immediately* so not sure what I am missing when using the KASP to 
manage key rollover.  The state for the keys looks good and for this 
test I have TTL set to 1 hour..  But why does dig not show me both 
DNSKEY records for the ZSK after I initiate the rollover when there 
should be overlap as described in Automatic DNSSEC Zone Signing Key 
rollover explained (isc.org) ?


Bind 9.16.23 which seems to be the newest release provided by my 
distribution.  I reviewed the ARM for notes for newer releases in the 
9.16 branch and did not see mention of any rollover bugs or for dig.


 1.  Here is the key info from dig for ZSK key 15465 at 17:17.

# dig @localhost myexample.com DNSKEY +multi

; <<>> DiG 9.16.23-RH <<>> @localhost myexample.com DNSKEY +multi
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41895
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0,
ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 7c2a0e61926d2d3a01006515fb68eef12b631ca40c20 (good)
;; QUESTION SECTION:
;myexample.com.         IN DNSKEY

;; ANSWER SECTION:
myexample.com.  3600 IN DNSKEY 257 3 13 (
      20agIXl9sQCo00yiHHviYWZG8TvVmDoVxPJwO3mlcwxB
      le7UNrzNQaeukC6teT4XrqYflqDxcM6d9L/mtREIKA==
      ) ; KSK; alg = ECDSAP256SHA256 ; key id = 31296
myexample.com.  3600 IN DNSKEY 256 3 13 (
      AlKXH5aebvboC4laAovc6wfg6uGK1uTbTqYYnhKadSq6
      78nSI3DyM+1t91jqQ81tlBy+e3hJyKtlX/OiOhuZcA==
                                ) ; ZSK; alg = ECDSAP256SHA256 ;
key id = 15465

;; Query time: 3 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 28 17:17:12 CDT 2023
;; MSG SIZE  rcvd: 230

 3.  Here is the info from the key as far as state goes.

# more Kmyexample.com.+013+15465.key
; This is a zone-signing key, keyid 15465, for myexample.com.
; Created: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Publish: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Activate: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Inactive: 20231127221438 (Mon Nov 27 16:14:38 2023)
; Delete: 20231207231938 (Thu Dec  7 17:19:38 2023)
myexample.com. 3600 IN DNSKEY 256 3 13
AlKXH5aebvboC4laAovc6wfg6uGK1uTbTqYYnhKadSq678nSI3DyM+1t
91jqQ81tlBy+e3hJyKtlX/OiOhuZcA==

# more Kmyexample.com.+013+15465.state
; This is the state of key 15465, for myexample.com.
Algorithm: 13
Length: 256
Lifetime: 5184000
KSK: no
ZSK: yes
Generated: 20230928221438 (Thu Sep 28 17:14:38 2023)
Published: 20230928221438 (Thu Sep 28 17:14:38 2023)
Active: 20230928221438 (Thu Sep 28 17:14:38 2023)
Retired: 20231127221438 (Mon Nov 27 16:14:38 2023)
Removed: 20231207231938 (Thu Dec  7 17:19:38 2023)
DNSKEYChange: 20230928221438 (Thu Sep 28 17:14:38 2023)
ZRRSIGChange: 20230928221438 (Thu Sep 28 17:14:38 2023)
DNSKEYState: rumoured
ZRRSIGState: rumoured
GoalState: omnipresent

I suspect that the crucial detail above is "DNSKEYState: rumoured". 
This suggests that the last ZSK rollover hasn't been fully completed.



Before starting a rollover it is a good idea to make sure the ZSK that 
you are retiring is in an "omnipresent" state.



The usual reason that the key isn't in omnipresent state is because 
BIND is still waiting for the corresponding DS record to be published 
and available in the parent zone. BIND 9.16 only knows if the DS 
record is published if you've set up parental-agents, or if you've 
explicitly told it using rndc. (BTW BIND 9.19 introduces new default 
behaviour which means you don't need to set parental-agents in order 
for it to figure this out.)



Have a look here: 
https://bind9.readthedocs.io/en/latest/chapter5.html#key-rollover



Specifically:

/If setting up a parental agent is undesirable, it is also
possible to tell BIND that the DS is published in the parent with:
/|rndc dnssec -checkds -key 12345 published dnssec.example.|

/.
and the DS for the predecessor key has been removed with: /|rndc
dnssec -checkds -key 54321 withdrawn dnssec.example.|

Re: KASP Key Rollover: ZSK Disappears Immediately

[Nick Tait via bind-users]

On 29/09/23 12:05, Eddie Rowe wrote:
When I perform a ZSK key rollover the existing ZSK disappears 
*immediately* so not sure what I am missing when using the KASP to 
manage key rollover.  The state for the keys looks good and for this 
test I have TTL set to 1 hour..  But why does dig not show me both 
DNSKEY records for the ZSK after I initiate the rollover when there 
should be overlap as described in Automatic DNSSEC Zone Signing Key 
rollover explained (isc.org) ?


Bind 9.16.23 which seems to be the newest release provided by my 
distribution.  I reviewed the ARM for notes for newer releases in the 
9.16 branch and did not see mention of any rollover bugs or for dig.


 1.  Here is the key info from dig for ZSK key 15465 at 17:17.

# dig @localhost myexample.com DNSKEY +multi

; <<>> DiG 9.16.23-RH <<>> @localhost myexample.com DNSKEY +multi
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41895
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0,
ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 7c2a0e61926d2d3a01006515fb68eef12b631ca40c20 (good)
;; QUESTION SECTION:
;myexample.com.         IN DNSKEY

;; ANSWER SECTION:
myexample.com.          3600 IN DNSKEY 257 3 13 (
    20agIXl9sQCo00yiHHviYWZG8TvVmDoVxPJwO3mlcwxB
    le7UNrzNQaeukC6teT4XrqYflqDxcM6d9L/mtREIKA==
    ) ; KSK; alg = ECDSAP256SHA256 ; key id = 31296
myexample.com.          3600 IN DNSKEY 256 3 13 (
    AlKXH5aebvboC4laAovc6wfg6uGK1uTbTqYYnhKadSq6
    78nSI3DyM+1t91jqQ81tlBy+e3hJyKtlX/OiOhuZcA==
                                ) ; ZSK; alg = ECDSAP256SHA256 ;
key id = 15465

;; Query time: 3 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 28 17:17:12 CDT 2023
;; MSG SIZE  rcvd: 230

 3.  Here is the info from the key as far as state goes.

# more Kmyexample.com.+013+15465.key
; This is a zone-signing key, keyid 15465, for myexample.com.
; Created: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Publish: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Activate: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Inactive: 20231127221438 (Mon Nov 27 16:14:38 2023)
; Delete: 20231207231938 (Thu Dec  7 17:19:38 2023)
myexample.com. 3600 IN DNSKEY 256 3 13
AlKXH5aebvboC4laAovc6wfg6uGK1uTbTqYYnhKadSq678nSI3DyM+1t
91jqQ81tlBy+e3hJyKtlX/OiOhuZcA==

# more Kmyexample.com.+013+15465.state
; This is the state of key 15465, for myexample.com.
Algorithm: 13
Length: 256
Lifetime: 5184000
KSK: no
ZSK: yes
Generated: 20230928221438 (Thu Sep 28 17:14:38 2023)
Published: 20230928221438 (Thu Sep 28 17:14:38 2023)
Active: 20230928221438 (Thu Sep 28 17:14:38 2023)
Retired: 20231127221438 (Mon Nov 27 16:14:38 2023)
Removed: 20231207231938 (Thu Dec  7 17:19:38 2023)
DNSKEYChange: 20230928221438 (Thu Sep 28 17:14:38 2023)
ZRRSIGChange: 20230928221438 (Thu Sep 28 17:14:38 2023)
DNSKEYState: rumoured
ZRRSIGState: rumoured
GoalState: omnipresent

I suspect that the crucial detail above is "DNSKEYState: rumoured". This 
suggests that the last ZSK rollover hasn't been fully completed.



Before starting a rollover it is a good idea to make sure the ZSK that 
you are retiring is in an "omnipresent" state.



The usual reason that the key isn't in omnipresent state is because BIND 
is still waiting for the corresponding DS record to be published and 
available in the parent zone. BIND 9.16 only knows if the DS record is 
published if you've set up parental-agents, or if you've explicitly told 
it using rndc. (BTW BIND 9.19 introduces new default behaviour which 
means you don't need to set parental-agents in order for it to figure 
this out.)



Have a look here: 
https://bind9.readthedocs.io/en/latest/chapter5.html#key-rollover



Specifically:

   /If setting up a parental agent is undesirable, it is also possible
   to tell BIND that the DS is published in the parent with: /|rndc
   dnssec -checkds -key 12345 published dnssec.example.|
   
/.
   and the DS for the predecessor key has been removed with: /|rndc
   dnssec -checkds -key 54321 withdrawn dnssec.example.|
   
/.
   where 12345 and 54321 are the key tags of the successor and
   predecessor key, respectively./

Nick.
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

KASP Key Rollover: ZSK Disappears Immediately

[Eddie Rowe]

When I perform a ZSK key rollover the existing ZSK disappears immediately so 
not sure what I am missing when using the KASP to manage key rollover.  The 
state for the keys looks good and for this test I have TTL set to 1 hour..  But 
why does dig not show me both DNSKEY records for the ZSK after I initiate the 
rollover when there should be overlap as described in Automatic DNSSEC Zone 
Signing Key rollover explained (isc.org)?

Bind 9.16.23 which seems to be the newest release provided by my distribution.  
I reviewed the ARM for notes for newer releases in the 9.16 branch and did not 
see mention of any rollover bugs or for dig.

  1.   Here is the key info from dig for ZSK key 15465 at 17:17.

# dig @localhost myexample.com DNSKEY +multi

; <<>> DiG 9.16.23-RH <<>> @localhost myexample.com DNSKEY +multi
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41895
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 7c2a0e61926d2d3a01006515fb68eef12b631ca40c20 (good)
;; QUESTION SECTION:
;myexample.com. IN DNSKEY

;; ANSWER SECTION:
myexample.com.  3600 IN DNSKEY 257 3 13 (
20agIXl9sQCo00yiHHviYWZG8TvVmDoVxPJwO3mlcwxB
le7UNrzNQaeukC6teT4XrqYflqDxcM6d9L/mtREIKA==
) ; KSK; alg = ECDSAP256SHA256 ; key id = 31296
myexample.com.  3600 IN DNSKEY 256 3 13 (
AlKXH5aebvboC4laAovc6wfg6uGK1uTbTqYYnhKadSq6
78nSI3DyM+1t91jqQ81tlBy+e3hJyKtlX/OiOhuZcA==
) ; ZSK; alg = ECDSAP256SHA256 ; key id = 15465

;; Query time: 3 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 28 17:17:12 CDT 2023
;; MSG SIZE  rcvd: 230


  1.   Here is the info from the key as far as state goes.

# more Kmyexample.com.+013+15465.key
; This is a zone-signing key, keyid 15465, for myexample.com.
; Created: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Publish: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Activate: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Inactive: 20231127221438 (Mon Nov 27 16:14:38 2023)
; Delete: 20231207231938 (Thu Dec  7 17:19:38 2023)
myexample.com. 3600 IN DNSKEY 256 3 13 
AlKXH5aebvboC4laAovc6wfg6uGK1uTbTqYYnhKadSq678nSI3DyM+1t 
91jqQ81tlBy+e3hJyKtlX/OiOhuZcA==

# more Kmyexample.com.+013+15465.state
; This is the state of key 15465, for myexample.com.
Algorithm: 13
Length: 256
Lifetime: 5184000
KSK: no
ZSK: yes
Generated: 20230928221438 (Thu Sep 28 17:14:38 2023)
Published: 20230928221438 (Thu Sep 28 17:14:38 2023)
Active: 20230928221438 (Thu Sep 28 17:14:38 2023)
Retired: 20231127221438 (Mon Nov 27 16:14:38 2023)
Removed: 20231207231938 (Thu Dec  7 17:19:38 2023)
DNSKEYChange: 20230928221438 (Thu Sep 28 17:14:38 2023)
ZRRSIGChange: 20230928221438 (Thu Sep 28 17:14:38 2023)
DNSKEYState: rumoured
ZRRSIGState: rumoured
GoalState: omnipresent

  1.   Ran rollover at 17:22.

# rndc dnssec -rollover -key 15465 myexample.com
Key 15465: Rollover scheduled on 28-Sep-2023 17:22:35.000

5. Ran dig at 17:22 and it now just shows the new ZSK 3913!  Where did the 
prior ZSK go (15465)?

# dig @localhost myexample.com DNSKEY +multi

; <<>> DiG 9.16.23-RH <<>> @localhost myexample.com DNSKEY +multi
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21486
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 419494d5cbae21fd01006515fcb383972461377bf329 (good)
;; QUESTION SECTION:
;myexample.com. IN DNSKEY

;; ANSWER SECTION:
myexample.com.  3600 IN DNSKEY 257 3 13 (
20agIXl9sQCo00yiHHviYWZG8TvVmDoVxPJwO3mlcwxB
le7UNrzNQaeukC6teT4XrqYflqDxcM6d9L/mtREIKA==
) ; KSK; alg = ECDSAP256SHA256 ; key id = 31296
myexample.com.  3600 IN DNSKEY 256 3 13 (
FdOW/okezOBOscx4/E4UfaSBkgK9tsnUZ8dvV5AZKMeH
jdH/jtYfASaeyrVYfclPsFuW5dLO7CU86vIplYKpEg==
) ; ZSK; alg = ECDSAP256SHA256 ; key id = 3913

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 28 17:22:43 CDT 2023
;; MSG SIZE  rcvd: 230

6.  The key state for the original ZSK (15465) has been updated and looks like 
it is set to become inactive at 19:27 which is what I expect based on my TTL of 
60 minutes.  The new key (03913) shows that it will be active at 19:27 which 
looks correct.

# more Kmyexample.com.+013+15465.key
; This is a zone-signing key, keyid 15465, for myexample.com.
; Created: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Publish: 20230928221438 (Thu Sep 28 17:14:38 2023)
; Activate: 20230928221438 (T