Bug#311188: debconf as a registry
On Thursday 16 October 2014 22:34:15 Bas Wijnen wrote: Oh yes, and I have some code ready for feedback. I haven't written the script libraries yet (and I want others to write some of them), but I have written the debhelper module for using them. For what it's worth, lcdproc package [1] uses cme (aka Config::Model) to merge configuration file and upstream changes. On the other hand, debconf is not involved because lcdproc does not need to store a debconf value. cme can be adapted to use a debconf value as some kind of default values (probably a preset value in Config::Model terminology) This may provide a way to solve your problem while minimizing the amount of duplicated code between packages. Hope this helps [1] http://anonscm.debian.org/cgit/collab-maint/lcdproc.git/ -- https://github.com/dod38fr/ -o- http://search.cpan.org/~ddumont/ http://ddumont.wordpress.com/ -o- irc: dod at irc.debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#311188: debconf as a registry
On 10/17/2014 01:41 PM, Bas Wijnen wrote:
Getting random packages from apt-cache rdepends debconf shows:
- several packages that use debconf for questions that are only about
actions that don't need to be (and aren't) stored in config files.
I previously thought that it was the case. Then thinking about it, it's
often possible create configuration files for the sole purpose of being
policy compliant and store values out of debconf. The /etc/default
folder is a good place for that.
I see only one case where the output of questions from Debconf should
not be stored: when it makes sense to *always* prompt (eg: when the
maintainer really wants the question to be asked each time the package
is upgraded). I do maintain packages with such a case.
- cxref uses ucf in postinst. This doesn't violate policy, but does (in
case of local changes) give the wrong default in the debconf question,
and an annoying do you want to overwrite local changes? after
answering the question.
Without looking, this sounds like a postinst modifies a CONFFILE,
which indeed is a policy violation.
- esmtp starts by asking if you want to overwrite your config and
refuses to be configured by debconf if you don't. Also
policy-compliant
This is debatable. To me, it doesn't make it policy compliant just
because the frist debconf prompt makes it possible to not do anything.
- dvi2ps does something I don't understand... It asks questions but
never runs db_get. Presumably it pre-seeds some other package?
Its looking like it doesn't need to do db_get. The only debconf
templates which it has are of type note, so it's not taking any decision.
- ibam depends on debconf but doesn't seem to ask any questions; it
doesn't even have a config or postinst script.
From the debian/changelog, it's looking like a leftover from previous
versions, and that it shouldn't depend on debconf (anymore). I'd say:
feel free to open a bug (of the lower severity).
- gpm does it right, in surprisingly few lines.
- grr does it wrong.
Yup, it should read the configuration file first.
This tiny investigation shows that most of the packages that handle
configuration files are either doing it in a way that is not
user-friendly, or that uses significant code in the config script. Both
of those cases would benefit from taking that code out of the source (if
it was there) and replacing it with an include statement.
As I wrote previously, this may only happen if we decide to have such a
library as essential, otherwise this forces to use pre-depends, which
isn't good.
I don't think investigating only 6 packages grants you the rights to
write most packages. Please be careful. This doesn't mean that I do
not agree, in fact, I do agree with you that we would benefit from
having this kind of library in Debian.
Maybe we could even have what you're talking about directly in Debconf
itself? I think it would make a lot of sense. If you want to work this
out, please investigate the possibility to enhance Debconf directly,
without needing to ship any supplementary lib.
Then such a library *must* be marked as essential, and installed by
default, otherwise it would break the install workflow.
No; it's a _static_ library. It is included in the config script at
package build time.
That's what I'm currently doing in all the OpenStack packages, and I'm
not satisfied with this approach. I very much would prefer an
enhancement of debconf itself.
This means every binary package using it will have
a copy of the code in its maintainerscript.
Which isn't nice.
A change in the library requires a rebuild of all the packages for
the change to take effect.
Which doesn't scale archive wide if you find a bug.
That's not ideal, but better than marking it as essential, or making it
part of debconf (which would also work, but requires Joey Hess to accept
it, and so far he refuses to even acknowledge that there is a problem;
even if he would, I find making it a separate package a more elegant
solution).
I don't agree with what you wrote above. Making it essential is a much
better approach. As for Joey Hess, he is a very reasonable person. If
you come with a patch which is well written, and does enough to be
helpful, I'm sure he'll accept it. If it's just bad, then probably he
will refuse. I've seen this type of pattern multiple times with him.
Ok, you have a repository. But which package should we look into?
Oops, sorry and thanks for the reminder. It's dh-parseconfig:
http://wijnen.dtdns.net/archive/unstable/{all,source}/dh-parseconfig*
As for parsing files, I don't think using perl is a great idea.
The perl script only pastes the file into the config script.
I've done this with a few lines of shell script. If that is what you
want me to look into dh-parseconfig, then I don't think it's worth
looking at.
The actual
parsing is done by a script in the language that config is written in.
That means there must be an
Bug#311188: debconf as a registry
Really, really cool analysis and wor, Bas!
Quoting Bas Wijnen (2014-10-17 07:41:21)
It's dh-parseconfig:
http://wijnen.dtdns.net/archive/unstable/{all,source}/dh-parseconfig*
[...]
I don't have a bug tracker yet, but I can upload this to unstable if
people don't complain too much about the code. ;-) Then the bts can be
used for feature requests (and bugs of course).
Please do release it to Debian. If you feel it is not yet in a usable
state for unstable then release it to experimental. I think having it
in Debian - even if not yet targeting a stable release, helps encourage
collaboration and (experimentation for future) adoption.
- Jonas
--
* Jonas Smedegaard - idealist Internet-arkitekt
* Tlf.: +45 40843136 Website: http://dr.jones.dk/
[x] quote me freely [ ] ask before reusing [ ] keep private
signature.asc
Description: signature
Bug#311188: debconf as a registry
Quoting Thomas Goirand (2014-10-17 09:51:04) On 10/17/2014 01:41 PM, Bas Wijnen wrote: I don't have a bug tracker yet, but I can upload this to unstable if people don't complain too much about the code. ;-) Then the bts can be used for feature requests (and bugs of course). Please don't upload this type of experimental software to Sid just right before the freeze. Please use experimental. A new package has no ties to the freeze - nothing depends on it and no older versions of itself is in testing - and therefore is fine to release to unstable, as it does not disrupt the freeze process. - Jonas -- * Jonas Smedegaard - idealist Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#311188: debconf as a registry
On 10/17/2014 04:51 PM, Jonas Smedegaard wrote: Quoting Thomas Goirand (2014-10-17 09:51:04) On 10/17/2014 01:41 PM, Bas Wijnen wrote: I don't have a bug tracker yet, but I can upload this to unstable if people don't complain too much about the code. ;-) Then the bts can be used for feature requests (and bugs of course). Please don't upload this type of experimental software to Sid just right before the freeze. Please use experimental. A new package has no ties to the freeze - nothing depends on it and no older versions of itself is in testing - and therefore is fine to release to unstable, as it does not disrupt the freeze process. - Jonas Probably, however if we don't need the software in Stable for the next 3 years, because it's not production ready or even useful (yet), then there's no point to have it in Jessie. Cheers, Thomas -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#311188: debconf as a registry
Quoting Thomas Goirand (2014-10-17 17:47:27) On 10/17/2014 04:51 PM, Jonas Smedegaard wrote: Quoting Thomas Goirand (2014-10-17 09:51:04) On 10/17/2014 01:41 PM, Bas Wijnen wrote: I don't have a bug tracker yet, but I can upload this to unstable if people don't complain too much about the code. ;-) Then the bts can be used for feature requests (and bugs of course). Please don't upload this type of experimental software to Sid just right before the freeze. Please use experimental. A new package has no ties to the freeze - nothing depends on it and no older versions of itself is in testing - and therefore is fine to release to unstable, as it does not disrupt the freeze process. Probably, however if we don't need the software in Stable for the next 3 years, because it's not production ready or even useful (yet), then there's no point to have it in Jessie. Jessie != unstable. If package is suitable for unstable, please upload to unstable. If package is suitable for unstable but not for testing, please upload to unstable and file severe bugreport to keep it from entering testing. - Jonas -- * Jonas Smedegaard - idealist Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#311188: debconf as a registry
As I wrote in the blend thread, reading through bug #311188 raised some new questions for me about this one. I will start by explaining the original problem again; it seemed to me that it wasn't understood by everyone. Then I'll add some new thoughts based on that bug. Finally, I present some code to solve the problem, on which I welcome feedback. = Explanation of the problem = Policy says (10.7.3): local changes must be preserved during a package upgrade However, configuration is stored in two places: in the configuration files in /etc, and in debconf's configuration space (which I'll call debconf's cache). The expected way that things work is: - On package install, a debconf question is asked and the answer is stored in debconf's cache. - Later during the same install, the cached value is copied into the configuration file by postinst. The problem appears on upgrade: debconf will again be called, and the question is whether or not to ask a question, and if so, what the default answer should be. And if not, what the value in the cache should be after not asking the question. That latter question is easily answered: it must have the same value that would have been the default if the question was asked. For the default answer there are three options: - The default that was supplied by the package. This is obviously wrong, it must only be used if there is no other data. - The value from debconf's cache. - The value from the configuration file. Since debconf's cache is a cache, not a registry, the latter is the correct answer. Debconf's cache MUST BE IGNORED if there is a configuration file. In practice it often doesn't matter, because the values are identical. But when they aren't, the admin has made local changes and they must be preserved. So debconf needs to read configuration files, but it doesn't know how to parse them. So it does the only thing it can: it uses its cache. Which means that each and every package that uses debconf must make sure that they read the configuration files and update debconf's cache _before_ running debconf. And most of them don't do that. (It also involves a significant amount of nontrivial code, which we really shouldn't want to see duplicated in every debconf-using package.) In fact, the debconf specification[0] even suggests that using debconf's cache is perfectly fine. The paragraph on The configuration space explains how two packages that share a configuration value can use the same value in the cache. But the important thing is that they should share a configuration file (as described in policy 10.7.4). While sharing a cache key is a good idea, because it avoids a duplicate question if both packages are installed simultaneously (both config scripts are run before both postinst scripts, so the shared configuration file is not updated when the second config script runs), that is really a minor issue. In the quote below my text, Russ agrees that all the packages are buggy, but he doesn't address the issue of how those bugs should be fixed. IMO telling all those maintainers to copy a significant piece of code into their config script is a bad idea. [0] https://www.debian.org/doc/packaging-manuals/debconf_specification.html which is linked from policy 3.9.1. = New thoughts inspired by #311188 = This bug is about debian-edu which needs to configure the system for its users. Because it wants to be a Pure Blend, it uses packages from the official archive. That's very nice. However, those packages all have their own ways of storing their configuration. So debian-edu has a similar problem as debconf: it cannot change the configuration of the packages in a policy-compliant way. Debconf doesn't know how to edit the files; debian-edu does know (because it uses code that is targeting specific files), but isn't allowed to do it. The solution would be for every package that debian-edu wants to configure to either add debconf questions which can be preseeded, or to add update-* scripts to allow those configuration files to be shared. If debconf is used properly, as described above, preseeding the cache should only work while the configuration file doesn't exist yet (otherwise the cache is overwritten before it is used). For this purpose, I consider that a feature, not a bug (because it means the admin can trust that the system doesn't decide to change things seemingly at random). However, regardless of which option is chosen, the packages require significant code to make them work. On the bright side, the result of that work is that they can be configured with debconf, which means it improves the quality of Debian. = Conclusion = The above problems are solved by my plan to create a static script library, which is included in config scripts at package build time. This library would define functions for parsing common config file formats, and can be included in config scripts with a
Bug#311188: debconf as a registry
On 10/17/2014 04:34 AM, Bas Wijnen wrote: So debconf needs to read configuration files, but it doesn't know how to parse them. So it does the only thing it can: it uses its cache. Which means that each and every package that uses debconf must make sure that they read the configuration files and update debconf's cache _before_ running debconf. And most of them don't do that. Could you name a few package for which this is the case? Has bugs been opened for them? The above problems are solved by my plan to create a static script library, which is included in config scripts at package build time. This library would define functions for parsing common config file formats, and can be included in config scripts with a line similar to ##DEBHELPER##. This has the benefit of solving the problem, without causing the problem of adding lots of duplicate code to the soures. Then such a library *must* be marked as essential, and installed by default, otherwise it would break the install workflow. Oh yes, and I have some code ready for feedback. I haven't written the script libraries yet (and I want others to write some of them), but I have written the debhelper module for using them. I have set up a mini-dinstall repository where you can get the binary and source packages: deb [arch=all,amd64] http://wijnen.dtdns.net/archive unstable main deb-src http://wijnen.dtdns.net/archive unstable main (Please ignore problems with the keys; I'm still getting it to work right.) The code is derived from dh-autoreconf and it is the first ever perl program I wrote. So please don't insult me, but also don't hold back when you see things that need to be improved. :-) Thanks, Bas Ok, you have a repository. But which package should we look into? As for parsing files, I don't think using perl is a great idea. Such configuration files sometimes may be huge. But anyway. I have a list of features that such a library would need to do for .ini files. This would include not only reading and writing to .ini files, but also allow maintenance of them, like for example moving a directive from one section to another (when this happens upstream). Cheers, Thomas Goirand (zigo) -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#311188: debconf as a registry
On Fri, Oct 17, 2014 at 12:37:27PM +0800, Thomas Goirand wrote:
On 10/17/2014 04:34 AM, Bas Wijnen wrote:
So debconf needs to read configuration files, but it doesn't know how to
parse them. So it does the only thing it can: it uses its cache. Which
means that each and every package that uses debconf must make sure that
they read the configuration files and update debconf's cache _before_
running debconf. And most of them don't do that.
Could you name a few package for which this is the case? Has bugs been
opened for them?
I have not reported any bugs, because there is no solution that I
consider acceptable yet. Any package with a config script that does not
include db_set and that writes the result of db_get to a config file (in
postinst) is broken. But, as I found, strictly speaking they are not
always violating policy.
Getting random packages from apt-cache rdepends debconf shows:
- several packages that use debconf for questions that are only about
actions that don't need to be (and aren't) stored in config files.
- cxref uses ucf in postinst. This doesn't violate policy, but does (in
case of local changes) give the wrong default in the debconf question,
and an annoying do you want to overwrite local changes? after
answering the question. If the default would have been correct, there
would have been no need for that.
- esmtp starts by asking if you want to overwrite your config and
refuses to be configured by debconf if you don't. Also
policy-compliant, but very unfriendly to users.
- dict does it right. This costs it more than 20 lines of code in the
config script.
- dvi2ps does something I don't understand... It asks questions but
never runs db_get. Presumably it pre-seeds some other package?
- ibam depends on debconf but doesn't seem to ask any questions; it
doesn't even have a config or postinst script.
- gpm does it right, in surprisingly few lines.
- grr does it wrong.
This tiny investigation shows that most of the packages that handle
configuration files are either doing it in a way that is not
user-friendly, or that uses significant code in the config script. Both
of those cases would benefit from taking that code out of the source (if
it was there) and replacing it with an include statement.
The above problems are solved by my plan to create a static script
library, which is included in config scripts at package build time.
This library would define functions for parsing common config file
formats, and can be included in config scripts with a line similar to
##DEBHELPER##. This has the benefit of solving the problem, without
causing the problem of adding lots of duplicate code to the soures.
Then such a library *must* be marked as essential, and installed by
default, otherwise it would break the install workflow.
No; it's a _static_ library. It is included in the config script at
package build time. This means every binary package using it will have
a copy of the code in its maintainerscript. But the source packages do
not. A change in the library requires a rebuild of all the packages for
the change to take effect. That's not ideal, but better than marking it
as essential, or making it part of debconf (which would also work, but
requires Joey Hess to accept it, and so far he refuses to even
acknowledge that there is a problem; even if he would, I find making it
a separate package a more elegant solution).
Oh yes, and I have some code ready for feedback. I haven't written the
script libraries yet (and I want others to write some of them), but I
have written the debhelper module for using them. I have set up a
mini-dinstall repository where you can get the binary and source
packages:
deb [arch=all,amd64] http://wijnen.dtdns.net/archive unstable main
deb-src http://wijnen.dtdns.net/archive unstable main
(Please ignore problems with the keys; I'm still getting it to work
right.)
The code is derived from dh-autoreconf and it is the first ever perl
program I wrote. So please don't insult me, but also don't hold back
when you see things that need to be improved. :-)
Ok, you have a repository. But which package should we look into?
Oops, sorry and thanks for the reminder. It's dh-parseconfig:
http://wijnen.dtdns.net/archive/unstable/{all,source}/dh-parseconfig*
As for parsing files, I don't think using perl is a great idea.
The perl script only pastes the file into the config script. The actual
parsing is done by a script in the language that config is written in.
That means there must be an implementation for every language (/bin/sh
being the most important) and every common file type (ini and csv
probably being the most important).
Note that none of those parsing libraries have been written yet. I'll
probably take some code from pioneers as a starting point.
I intend most libraries to be included in the dh-parseconfig source, but
it is set up in a way that allows other packages to
