Send kea-dev mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/kea-dev
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of kea-dev digest..."
Today's Topics:
1. Re: Lease File Cleanup in Kea - Design Document (Stephen Morris)
2. Re: Lease File Cleanup in Kea - Design Document
(Marcin Siodelski)
3. Re: Lease File Cleanup in Kea - Design Document (Tomek Mrugalski)
4. Re: Lease File Cleanup in Kea - Design Document
(Marcin Siodelski)
----------------------------------------------------------------------
Message: 1
Date: Fri, 09 Jan 2015 14:33:37 +0000
From: Stephen Morris <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] Lease File Cleanup in Kea - Design Document
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
My comments are mainly concerned with the recovery from crashes.
Terminology
---
I think that "Lease File Copy" is a misnomer, because the file is not a
copy of the current lease file: it is the lease file as it was before a
new one was started.
I would also suggest that numberic suffixes are clearer to indicate
the ordering of lease files. So:
lease-file ... is the current lease file
lease-file.1 ... is the what the current lease file is renamed to
lease-file.2 ... is the result of the previous LFC run
Reading the lease files in descending order of version (2, 1, <blank>)
reads the lease information in the correct order.
We do need two more filenames, and I suggest:
lease-file.output (currently named LFC Output File)
lease-file.completed (LFC Output File when LFC has completed, see below)
I'm not hung up on names though, so choose whatever seems best. In
the remaining comments I've used the names given in the design.
State Diagram
---
The bit that concerned me was the comment about doubled information if
the LFC crashes between renaming the LFC Output File and deleting the
Lease File Copy. (Yes, it is unlikely, but I have a pessimistic view
of the Universe.)
The described situation is due to the fact that after the LFC Output
File has been renamed to the Previous Lease File, the state is
ambiguous. Either:
a) The last LFC completed successfully: the Previous Lease File holds
the result of the last LFC run and the Lease File Copy is the result
of the server renaming the current lease file since then.
b) The last LFC completed successfully: the Previous Lease File holds
the result of the last LFC run and the Lease File Copy is one of the
input files to the last LFC run.
To address this and remove the (admittedly small) risk of duplicate
information, once the cleanup process has completed, the LFC Output
File should be renamed before the file deletion starts. So the
sequence would be:
Rename LFC Output File to LFC Completed File
Delete Previous Lease File
Delete Lease File Copy
Rename LFC Completed File to Previous Lease File
Now the state is unambiguous when the LFC starts up:
If (LFC Completed File exists) {
// Previous LFC processing completed apart from file
// deletion. Finish the deletion & exit.
Delete Previous Lease File
Delete Lease File Copy
Rename LFC Completed File to Previous Lease File
exit
} else if (LFC Output File exists) {
// Previous LFC processing did not complete. Restart
// the processing from the beginning.
Delete LFC Output File
Do LFC Processing
} else {
// Previous LFC processing completed successfully. Start
// the next set of processing.
Do LFC Processing
}
Server Startup
---
When the server starts up, it needs to read the lease files. This
operation needs to take account of the possibility of a "LFC Completed
File". If that is present, it should read that instead of the "Lease
File Copy" and "Previous Lease" files.
A second point about server startup is that it should ensure that no
LFC process is running when it reads the lease files. The scenario
exists whereby LFC is running when the server crashes; as the server
starts up again and reads the lease files, the LFC process completes
and starts deleting those files.
Stephen
------------------------------
Message: 2
Date: Fri, 09 Jan 2015 15:51:14 +0100
From: Marcin Siodelski <[email protected]>
To: Stephen Morris <[email protected]>, [email protected]
Subject: Re: [kea-dev] Lease File Cleanup in Kea - Design Document
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
Stephen,
Thanks for the comments. I will go through them some time next week.
Note that Shawn had a reviewed the design and the relevant ticket was
closed. Also, the implementation is already in progress. However, it is
probably still easy to include some changes.
Marcin
On 01/09/15 15:33, Stephen Morris wrote:
> My comments are mainly concerned with the recovery from crashes.
>
> Terminology
> ---
> I think that "Lease File Copy" is a misnomer, because the file is not a
> copy of the current lease file: it is the lease file as it was before a
> new one was started.
>
> I would also suggest that numberic suffixes are clearer to indicate
> the ordering of lease files. So:
>
> lease-file ... is the current lease file
> lease-file.1 ... is the what the current lease file is renamed to
> lease-file.2 ... is the result of the previous LFC run
>
> Reading the lease files in descending order of version (2, 1, <blank>)
> reads the lease information in the correct order.
>
> We do need two more filenames, and I suggest:
>
> lease-file.output (currently named LFC Output File)
> lease-file.completed (LFC Output File when LFC has completed, see below)
>
> I'm not hung up on names though, so choose whatever seems best. In
> the remaining comments I've used the names given in the design.
>
>
> State Diagram
> ---
> The bit that concerned me was the comment about doubled information if
> the LFC crashes between renaming the LFC Output File and deleting the
> Lease File Copy. (Yes, it is unlikely, but I have a pessimistic view
> of the Universe.)
>
> The described situation is due to the fact that after the LFC Output
> File has been renamed to the Previous Lease File, the state is
> ambiguous. Either:
>
> a) The last LFC completed successfully: the Previous Lease File holds
> the result of the last LFC run and the Lease File Copy is the result
> of the server renaming the current lease file since then.
>
> b) The last LFC completed successfully: the Previous Lease File holds
> the result of the last LFC run and the Lease File Copy is one of the
> input files to the last LFC run.
>
> To address this and remove the (admittedly small) risk of duplicate
> information, once the cleanup process has completed, the LFC Output
> File should be renamed before the file deletion starts. So the
> sequence would be:
>
> Rename LFC Output File to LFC Completed File
> Delete Previous Lease File
> Delete Lease File Copy
> Rename LFC Completed File to Previous Lease File
>
> Now the state is unambiguous when the LFC starts up:
>
> If (LFC Completed File exists) {
> // Previous LFC processing completed apart from file
> // deletion. Finish the deletion & exit.
> Delete Previous Lease File
> Delete Lease File Copy
> Rename LFC Completed File to Previous Lease File
> exit
>
> } else if (LFC Output File exists) {
> // Previous LFC processing did not complete. Restart
> // the processing from the beginning.
> Delete LFC Output File
> Do LFC Processing
>
> } else {
> // Previous LFC processing completed successfully. Start
> // the next set of processing.
> Do LFC Processing
> }
>
>
> Server Startup
> ---
> When the server starts up, it needs to read the lease files. This
> operation needs to take account of the possibility of a "LFC Completed
> File". If that is present, it should read that instead of the "Lease
> File Copy" and "Previous Lease" files.
>
> A second point about server startup is that it should ensure that no
> LFC process is running when it reads the lease files. The scenario
> exists whereby LFC is running when the server crashes; as the server
> starts up again and reads the lease files, the LFC process completes
> and starts deleting those files.
>
>
> Stephen
> _______________________________________________
> kea-dev mailing list
> [email protected]
> https://lists.isc.org/mailman/listinfo/kea-dev
>
------------------------------
Message: 3
Date: Fri, 09 Jan 2015 16:56:34 +0100
From: Tomek Mrugalski <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] Lease File Cleanup in Kea - Design Document
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
On 05.01.2015 16:31, Marcin Siodelski wrote:
> Following the thread regarding the requirements for Lease File Cleanup
> in Kea
> (https://lists.isc.org/pipermail/kea-dev/2014-December/000211.html), I
> have created a short design document for this feature:
>
> http://kea.isc.org/wiki/LFCDesign
Apologies for replying after 4 days, but there was a holiday in between
and I'm a bit under the weather.
Anyway, I read that document and think it's in a good shape. I have two
comments, though.
1. The clean-up algorithm will work, but will be very inefficient. Let
me give you specific example. You have T1 set to 10 minutes and lifetime
to 1 hour. A client comes in, gets and address, then keeps renewing
every 10 minutes. After an hour, LFC process is triggered. How many
entries for his lease you will have after LFC finishes. I think the
answer is 5. You can tweak the parameters and get this algorithm as
inefficient as you want, just lower the T1 to lifetime ratio. I think we
should do something different: kea-lfc should instantiate
MemfileLeaseMgr, tell it to read previous lease file and lease file copy
(just as planned now). The code there is already able to handle
sequential entries for the same lease. Once it's loaded, the code has to
write down the whole content.
The huge advantage of this approach is that for X leases you'll have
exactly X entries in the output file. Another advantage is that exactly
the same code is used to clean up the lease file and by the actual DHCP
server, so it's much easier to maintain and the chances for and
discrepancies are minimal. Finally, that approach would the
implementation easier, because you don't need any extra columns, like
the released. (That column can be useful for address affinity, but
that's a separate matter and is a different feature, that has nothing to
do with lease cleanup). So you could remove #3674 completely and make
implementation of #3665 and #3666 simpler.
The only real disavantage I see of this approach is that there won't be
a way to retain unparseable entries. I think that's not a big issue,
though. First, the unparseable entry will be ignored by the server (or
worse, will cause the server to refuse to load the lease file). Second,
I'm not aware of any good use cases for this. Two were given: one is a
partially written lease when server experienced power outage and another
was a sysadmin that added comments to the lease file. Both are weak.
Partially written lease is unusable and often is followed by a random
garbage. This cannot be recovered. The second one is really bizarre -
lease file is an autogenerated file intended for automatic processing.
Someone shutting down the server, injecting comments there and then
restarting the server should not be surprised for them to be removed.
Having said that, there's an easy workaround if you think that there's a
value in keeping the unparsable elements. When the MemfileLeaseMgr code
loads lease files and discovers something unparsable, it should log is
(so sysadmin becomes aware of this) and copy the whole file aside
(lease-file-unparsed-YYYY-MM-DD-HH-MM), so the sysadmin could intervene
(and possibly manually recover the lease).
2. My second comment is trivial. I think we should not use 'lfc' in the
kea-lfc name. Nobody would know off hand or even have a clue what that
means. I would propose kea-cleanup, kea-db-maintenance,
kea-memfile-maint or something similar. It's probably useful to signify
that the cleanup is specific to memfile backend and there's no
equivalent for MySQL or Postgres.
I understand that my comments were sent a bit late, so I won't insist on
implementing them now. In any case, if you think it's too late to
consider comment #1 now, this is something that will have to be
implemented at some later stage. I can imagine that there may be other
clean up strategies that are better suited for specific deployments.
Hence it seems reasonable to assume that sooner or later other cleanup
algorithms will become available.
Tomek
------------------------------
Message: 4
Date: Fri, 09 Jan 2015 17:25:56 +0100
From: Marcin Siodelski <[email protected]>
To: Tomek Mrugalski <[email protected]>
Cc: [email protected]
Subject: Re: [kea-dev] Lease File Cleanup in Kea - Design Document
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Tomek,
As we discussed this over the jabber, I will propose the change to the
design to bulk load leases from the input file and remove the
duplicates. I agree there are networks where that would perform much
better. I was thinking about the example of the cable network with a
lease lifetime being a week or so. Switching on and off the client would
cause a lot of new entries in the lease file that will not be removed
for a long period of time. On the other hand, if this is your ISPs
network you'd probably have a router connected to it which would be
powered-on for the whole time and the renewals would occur every half of
the lease duration or so. But, that's obviously depends on other things
like T1/T2 and it hard to make a good assumption for all networks.
So, overall I agree that it is better to do what you propose and relax
on the requirement that the unparsable parts of the lease file are not
rewritten. We can probably add support for this over time, if we feel it
is important.
Regarding the second request. I think it is a judgement call and there
will always be people who don't like a specific name for the
application. It is just a file name after all, so I don't want to spend
to much time on pushing one way or another. Note that this application
is not meant to be executed directly but will rather be executed by the
server process. Also, the purpose of this application is to perform
"Lease File Cleanup" and it is already abbreviated "LFC" in the existing
documentation so I think the name precisely describes the purpose.
Things like database-maintanance do not defend themselves in my opinion
because they are ambiguous. It is not the database, but a file. It is
not maintanance but cleanup, unless you precisely describe what
maintanance means in this context. If we wanted to call it like this we
should have started from naming the whole feature like this in the
design, requirements and other discussions. So, I object to changes to
the name and I prefer keeping names as short as possible.
Marcin
On 01/09/15 16:56, Tomek Mrugalski wrote:
> On 05.01.2015 16:31, Marcin Siodelski wrote:
> > Following the thread regarding the requirements for Lease File Cleanup
> > in Kea
> > (https://lists.isc.org/pipermail/kea-dev/2014-December/000211.html), I
> > have created a short design document for this feature:
> >
> > http://kea.isc.org/wiki/LFCDesign
> Apologies for replying after 4 days, but there was a holiday in between
> and I'm a bit under the weather.
>
> Anyway, I read that document and think it's in a good shape. I have two
> comments, though.
>
> 1. The clean-up algorithm will work, but will be very inefficient. Let
> me give you specific example. You have T1 set to 10 minutes and lifetime
> to 1 hour. A client comes in, gets and address, then keeps renewing
> every 10 minutes. After an hour, LFC process is triggered. How many
> entries for his lease you will have after LFC finishes. I think the
> answer is 5. You can tweak the parameters and get this algorithm as
> inefficient as you want, just lower the T1 to lifetime ratio. I think we
> should do something different: kea-lfc should instantiate
> MemfileLeaseMgr, tell it to read previous lease file and lease file copy
> (just as planned now). The code there is already able to handle
> sequential entries for the same lease. Once it's loaded, the code has to
> write down the whole content.
>
> The huge advantage of this approach is that for X leases you'll have
> exactly X entries in the output file. Another advantage is that exactly
> the same code is used to clean up the lease file and by the actual DHCP
> server, so it's much easier to maintain and the chances for and
> discrepancies are minimal. Finally, that approach would the
> implementation easier, because you don't need any extra columns, like
> the released. (That column can be useful for address affinity, but
> that's a separate matter and is a different feature, that has nothing to
> do with lease cleanup). So you could remove #3674 completely and make
> implementation of #3665 and #3666 simpler.
>
> The only real disavantage I see of this approach is that there won't be
> a way to retain unparseable entries. I think that's not a big issue,
> though. First, the unparseable entry will be ignored by the server (or
> worse, will cause the server to refuse to load the lease file). Second,
> I'm not aware of any good use cases for this. Two were given: one is a
> partially written lease when server experienced power outage and another
> was a sysadmin that added comments to the lease file. Both are weak.
> Partially written lease is unusable and often is followed by a random
> garbage. This cannot be recovered. The second one is really bizarre -
> lease file is an autogenerated file intended for automatic processing.
> Someone shutting down the server, injecting comments there and then
> restarting the server should not be surprised for them to be removed.
>
> Having said that, there's an easy workaround if you think that there's a
> value in keeping the unparsable elements. When the MemfileLeaseMgr code
> loads lease files and discovers something unparsable, it should log is
> (so sysadmin becomes aware of this) and copy the whole file aside
> (lease-file-unparsed-YYYY-MM-DD-HH-MM), so the sysadmin could intervene
> (and possibly manually recover the lease).
>
> 2. My second comment is trivial. I think we should not use 'lfc' in the
> kea-lfc name. Nobody would know off hand or even have a clue what that
> means. I would propose kea-cleanup, kea-db-maintenance,
> kea-memfile-maint or something similar. It's probably useful to signify
> that the cleanup is specific to memfile backend and there's no
> equivalent for MySQL or Postgres.
>
> I understand that my comments were sent a bit late, so I won't insist on
> implementing them now. In any case, if you think it's too late to
> consider comment #1 now, this is something that will have to be
> implemented at some later stage. I can imagine that there may be other
> clean up strategies that are better suited for specific deployments.
> Hence it seems reasonable to assume that sooner or later other cleanup
> algorithms will become available.
>
> Tomek
>
> _______________________________________________
> kea-dev mailing list
> [email protected]
> https://lists.isc.org/mailman/listinfo/kea-dev
>
------------------------------
_______________________________________________
kea-dev mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/kea-dev
End of kea-dev Digest, Vol 10, Issue 3
**************************************
