Re: Easier %config management?

2015-12-21 Thread Christopher 
That blog seems to describe supporting a stateless use case, not for
advocating that all systems should be stateless. In any case, the factory
reset use case they describe is similar to some of my own.

On Mon, Dec 21, 2015, 23:32 Nico Kadel-Garcia  wrote:

> On Thu, Dec 17, 2015 at 3:55 AM, Harald Hoyer  wrote:
> > On 16.12.2015 03:32, Colin Walters wrote:
> >> On Tue, Dec 15, 2015, at 06:43 PM, Japheth Cleaver wrote:
> >>>
> >>> Perhaps RPM (or yum/dnf, via plugin) could write a duplicate copy of
> all
> >>> config files into a tree somewhere? (E.g., /usr/lib/config/ or
> >>> /usr/share/config/?)
> >>
> >> I mentioned this above, but might as well repeat since it was missed;
> OSTree
> >> (as used by the existing Fedora Atomic Host) does this by default today
> in
> >> /usr/etc, so if one was adapting this change to the client-side system
> assembly
> >> tools like yum/dnf, I'd say it would make sense to follow the precedent.
> >>
> >>
> >
> > I already experimented with /usr/share/factory/{etc,var} , but /usr/etc
> sounds
> > fine to me, too.
>
> /usr/etc is expressly forbidden in the verison 3 of the File Hierarchy
> System published at
> http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf. "/etc" files
> go in "/etc", and *maybe* /usr/local/etc depending on whether the
> package is local to the speicific host.
>
> Also pay attention to systemd encroaching on /etc/ The long term plan
> of at least one of the core systemd deveopers is to replace locally
> modifiable "/etc" with systemd program management, in order to make
> Linux "stateless". There's a description of the approach at
> http://0pointer.net/blog/projects/stateless.html.
>
> > Additionally I would like to have that in the rpm package itsself, not
> with
> > some plugin on installation, because "rpm -qf" should output to which
> package
> > the file in /usr/etc belongs. Also the %config(noreplace) attribute has
> to be
> > removed from the pristine config files.
> >
> > Attached is a quick hack to rpm I have done for experimenting with that
> feature.
> >
> > --
> > devel mailing list
> > devel@lists.fedoraproject.org
> > http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
> --
> devel mailing list
> devel@lists.fedoraproject.org
> http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
>
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-21 Thread Nico Kadel-Garcia 
On Thu, Dec 17, 2015 at 3:55 AM, Harald Hoyer  wrote:
> On 16.12.2015 03:32, Colin Walters wrote:
>> On Tue, Dec 15, 2015, at 06:43 PM, Japheth Cleaver wrote:
>>>
>>> Perhaps RPM (or yum/dnf, via plugin) could write a duplicate copy of all
>>> config files into a tree somewhere? (E.g., /usr/lib/config/ or
>>> /usr/share/config/?)
>>
>> I mentioned this above, but might as well repeat since it was missed; OSTree
>> (as used by the existing Fedora Atomic Host) does this by default today in
>> /usr/etc, so if one was adapting this change to the client-side system 
>> assembly
>> tools like yum/dnf, I'd say it would make sense to follow the precedent.
>>
>>
>
> I already experimented with /usr/share/factory/{etc,var} , but /usr/etc sounds
> fine to me, too.

/usr/etc is expressly forbidden in the verison 3 of the File Hierarchy
System published at
http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf. "/etc" files
go in "/etc", and *maybe* /usr/local/etc depending on whether the
package is local to the speicific host.

Also pay attention to systemd encroaching on /etc/ The long term plan
of at least one of the core systemd deveopers is to replace locally
modifiable "/etc" with systemd program management, in order to make
Linux "stateless". There's a description of the approach at
http://0pointer.net/blog/projects/stateless.html.

> Additionally I would like to have that in the rpm package itsself, not with
> some plugin on installation, because "rpm -qf" should output to which package
> the file in /usr/etc belongs. Also the %config(noreplace) attribute has to be
> removed from the pristine config files.
>
> Attached is a quick hack to rpm I have done for experimenting with that 
> feature.
>
> --
> devel mailing list
> devel@lists.fedoraproject.org
> http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-21 Thread Miroslav Suchý 
Dne 13.12.2015 v 05:58 Christopher napsal(a):
> Which components/packages are best candidates for adding a feature which 
> would make it easier for users to track
> changes from the default %config %files on the system?
... [SNIP] ...
> rpmconf is nice, because it helps me easily compare configuration files whose 
> user-changes and maintainer-changes
> conflict... but that's not quite the same thing.

If you send me the patch with new option for rpmconf I will be happy to review 
it and merge it.

> rpm could track more than hashes of config files, and instead track the full 
> file. 

I really do not believe it should belong directly to rpm (and even to DNF).

> rpmconf might also need modification to support tracking configuration 
> management more fully, rather than just for updates.

As other pointed out, etckeeper do that better.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-17 Thread Harald Hoyer 
On 16.12.2015 03:32, Colin Walters wrote:
> On Tue, Dec 15, 2015, at 06:43 PM, Japheth Cleaver wrote:
>>  
>> Perhaps RPM (or yum/dnf, via plugin) could write a duplicate copy of all
>> config files into a tree somewhere? (E.g., /usr/lib/config/ or
>> /usr/share/config/?)
>  
> I mentioned this above, but might as well repeat since it was missed; OSTree
> (as used by the existing Fedora Atomic Host) does this by default today in
> /usr/etc, so if one was adapting this change to the client-side system 
> assembly
> tools like yum/dnf, I'd say it would make sense to follow the precedent.
>  
>  

I already experimented with /usr/share/factory/{etc,var} , but /usr/etc sounds
fine to me, too.

Additionally I would like to have that in the rpm package itsself, not with
some plugin on installation, because "rpm -qf" should output to which package
the file in /usr/etc belongs. Also the %config(noreplace) attribute has to be
removed from the pristine config files.

Attached is a quick hack to rpm I have done for experimenting with that feature.
From 1eb125dd814a08b0a8bfd5e4ab95d38525caf5d3 Mon Sep 17 00:00:00 2001
From: Harald Hoyer 
Date: Thu, 12 Jun 2014 18:31:16 +0200
Subject: [PATCH] Quick hack to place all /etc files also in
 /usr/share/factory/etc

---
 build/Makefile.am |   2 +-
 build/files.c | 112 +++---
 build/rpmfc.c |   6 ++-
 3 files changed, 77 insertions(+), 43 deletions(-)

diff --git a/build/Makefile.am b/build/Makefile.am
index 1a540bc..5d75ef2 100644
--- a/build/Makefile.am
+++ b/build/Makefile.am
@@ -16,7 +16,7 @@ librpmbuild_la_SOURCES = \
 	parsePolicies.c policies.c \
 	rpmbuild_internal.h rpmbuild_misc.h
 
-librpmbuild_la_LDFLAGS = -version-info 4:0:1
+librpmbuild_la_LDFLAGS = -version-info 6:0:3
 librpmbuild_la_LIBADD = \
 	$(top_builddir)/lib/librpm.la \
 	$(top_builddir)/rpmio/librpmio.la \
diff --git a/build/files.c b/build/files.c
index 07a6847..f823f79 100644
--- a/build/files.c
+++ b/build/files.c
@@ -1188,6 +1188,69 @@ static struct stat * fakeStat(FileEntry cur, struct stat * statp)
 return statp;
 }
 
+static rpmRC add_to_file_list(FileList fl,
+   const char* diskPath,
+   const char *cpioPath,
+   mode_t fileMode,
+   const char *fileUname,
+   const char *fileGname,
+   uid_t fileUid,
+   gid_t fileGid,
+   struct stat* statp,
+   int clone
+   )
+{
+if (fl->files.used == fl->files.alloced) {
+fl->files.alloced += 128;
+fl->files.recs = xrealloc(fl->files.recs,
+fl->files.alloced * sizeof(*(fl->files.recs)));
+}
+
+{	FileListRec flp = >files.recs[fl->files.used];
+
+flp->fl_st = *statp;	/* structure assignment */
+flp->fl_mode = fileMode;
+flp->fl_uid = fileUid;
+flp->fl_gid = fileGid;
+
+flp->cpioPath = xstrdup(cpioPath);
+flp->diskPath = xstrdup(diskPath);
+flp->uname = rpmstrPoolId(fl->pool, fileUname, 1);
+flp->gname = rpmstrPoolId(fl->pool, fileGname, 1);
+
+if (fl->cur.langs) {
+flp->langs = argvJoin(fl->cur.langs, "|");
+} else {
+flp->langs = xstrdup("");
+}
+
+if (fl->cur.caps) {
+flp->caps = xstrdup(fl->cur.caps);
+} else {
+flp->caps = xstrdup("");
+}
+
+flp->flags = fl->cur.attrFlags;
+flp->specdFlags = fl->cur.specdFlags;
+flp->verifyFlags = fl->cur.verifyFlags;
+
+if (clone) {
+flp->flags &= ~(RPMFILE_MISSINGOK|RPMFILE_CONFIG|RPMFILE_NOREPLACE);
+flp->specdFlags |= SPECD_VERIFY;
+flp->verifyFlags = RPMVERIFY_ALL;
+}
+
+if (!(flp->flags & RPMFILE_EXCLUDE) && S_ISREG(flp->fl_mode)) {
+if (flp->fl_size >= UINT32_MAX) {
+fl->largeFiles = 1;
+}
+}
+}
+
+fl->files.used++;
+return RPMRC_OK;
+}
+
 /**
  * Add a file to the package manifest.
  * @param fl		package file tree walk data
@@ -1333,49 +1396,18 @@ static rpmRC addFile(FileList fl, const char * diskPath,
 }
 
 /* Add to the file list */
-if (fl->files.used == fl->files.alloced) {
-	fl->files.alloced += 128;
-	fl->files.recs = xrealloc(fl->files.recs,
-			fl->files.alloced * sizeof(*(fl->files.recs)));
-}
-	
-{	FileListRec flp = >files.recs[fl->files.used];
-
-	flp->fl_st = *statp;	/* structure assignment */
-	flp->fl_mode = fileMode;
-	flp->fl_uid = fileUid;
-	flp->fl_gid = fileGid;
+rc = add_to_file_list(fl, diskPath, cpioPath, fileMode, fileUname, fileGname,
+

Re: Easier %config management?

2015-12-16 Thread Daniel J Walsh 


On 12/15/2015 09:32 PM, Colin Walters wrote:
> On Tue, Dec 15, 2015, at 06:43 PM, Japheth Cleaver wrote:
>>  
>> Perhaps RPM (or yum/dnf, via plugin) could write a duplicate copy of
>> all config files into a tree somewhere? (E.g., /usr/lib/config/ or
>> /usr/share/config/?)
>  
> I mentioned this above, but might as well repeat since it was missed;
> OSTree (as used by the existing Fedora Atomic Host) does this by
> default today in /usr/etc, so if one was adapting this change to the
> client-side system assembly tools like yum/dnf, I'd say it would make
> sense to follow the precedent.
>  
>  
>  
>
>
> --
> devel mailing list
> devel@lists.fedoraproject.org
> http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
I think this makes sense as a first step to getting to the point where
you could reset your machine by removing /etc/ and /var.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-16 Thread Panu Matilainen 

On 12/16/2015 01:43 AM, Japheth Cleaver wrote:
[...]

It seems like there's already a little bit of precedent for "alternate
trees" representing the existing file system with the debuginfo rpms and
/usr/lib/debug/ hierarchy. Although the easiest solution might be to
have RPM deposit a duplicate copy of *all* config files (not just
%config(noreplace)) adjacent to the config file actually read by the
programs (e.g., /etc/httpd/conf.d/ssl.conf and
/etc/httpd/conf.d/ssl.conf.rpmdist), that leaves lots of ugly, mostly
unused, files everywhere.

Perhaps RPM (or yum/dnf, via plugin) could write a duplicate copy of all
config files into a tree somewhere? (E.g., /usr/lib/config/ or
/usr/share/config/?) RPM/yum/dnf could then do the equivalent of an `rpm
-qlca` and compare each /etc/httpd/conf.d/ssl.conf to
/usr/lib/config/etc/httpd/conf.d/ssl.conf with no real magic or complex
tech needed. Merging left up to the sysadmin.


There are no yum/dnf-level hooks/callbacks that execute per file, you'll 
need an rpm plugin for that.


- Panu -

--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-15 Thread Christopher 
On Tue, Dec 15, 2015 at 12:37 AM J. Randall Owens <
jrowens.fed...@ghiapet.net> wrote:

> On 12/14/2015 02:47 PM, Christopher wrote:
> > On Mon, Dec 14, 2015 at 11:22 AM Reindl Harald  > > wrote:
> >
> > i modified my "httpd.conf" based on Apache 2.2 years ago, as Fedora
> > swicthed to Apache 2.4 your approach would have compared a customized
> > Apache 2.2 version with a Fedora 2.4 version
> >
> >
> > But, that's precisely what I want to do I want to know which fields
> > in the current configuration could possibly have introduced a breakage I
> > see... or which changes I need to make from a clean installation to get
> > a user back to the state they wanted.
> >
>
> But in order to do the former, you'd best compare the new default to the
> old default, to see what had changed in the default. (E.g. distro 2.2 to
> distro 2.4) To do the latter, you'd compare the old default (distro 2.2)
> to your modified version (altered 2.2), to see what you'd changed
> locally. And yet, you seem to be proposing comparing the new default to
> the old default (distro 2.4 to altered 2.2), which gets you neither? (Or
> both, really, but uselessly mashed together so you don't know which is
> which.) Am I missing something?
>
>
I'm less interested in what the user intended to change in an older version
than the full difference between the current default (presumably working)
configs and the current files (which would include any user modifications
AND any old defaults carried forward). The current system breakage could be
the result of user edits, or a carry-forward of an older config. I don't
care which is the case... I just want to see the differences to
troubleshoot the problem... I just want to fix the system as it exists
today... not question user edit decisions which occurred years ago.

Certainly, a full history, tracking all user edits, and all system updates,
would be useful to some. It's just more than what I personally need, and
seems a bit outside the scope of the existing installer tools (as
previously stated with the suggestion to use etckeeper).

For now, I think I'm going to write a script to parse the output of `rpm
-V` and then `dnf download` all the packages corresponding to changed
configs, then extract each rpm's %config files, to compare with those on
the system. It's messy, but it'll work, and in the absence of rpm storing
the full configs or being able to instruct rpm/dnf to reset configs back to
the defaults (after backing up), there doesn't seem to be an existing
solution that will give me the same result.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-15 Thread Japheth Cleaver 

On 12/15/2015 10:54 AM, Christopher wrote:
On Tue, Dec 15, 2015 at 12:37 AM J. Randall Owens 
> wrote:


On 12/14/2015 02:47 PM, Christopher wrote:
> On Mon, Dec 14, 2015 at 11:22 AM Reindl Harald

> >>
wrote:
>
> i modified my "httpd.conf" based on Apache 2.2 years ago, as
Fedora
> swicthed to Apache 2.4 your approach would have compared a
customized
> Apache 2.2 version with a Fedora 2.4 version
>
>
> But, that's precisely what I want to do I want to know which
fields
> in the current configuration could possibly have introduced a
breakage I
> see... or which changes I need to make from a clean installation
to get
> a user back to the state they wanted.
>

But in order to do the former, you'd best compare the new default
to the
old default, to see what had changed in the default. (E.g. distro
2.2 to
distro 2.4) To do the latter, you'd compare the old default
(distro 2.2)
to your modified version (altered 2.2), to see what you'd changed
locally. And yet, you seem to be proposing comparing the new
default to
the old default (distro 2.4 to altered 2.2), which gets you
neither? (Or
both, really, but uselessly mashed together so you don't know which is
which.) Am I missing something?


I'm less interested in what the user intended to change in an older 
version than the full difference between the current default 
(presumably working) configs and the current files (which would 
include any user modifications AND any old defaults carried forward). 
The current system breakage could be the result of user edits, or a 
carry-forward of an older config. I don't care which is the case... I 
just want to see the differences to troubleshoot the problem... I just 
want to fix the system as it exists today... not question user edit 
decisions which occurred years ago.


Certainly, a full history, tracking all user edits, and all system 
updates, would be useful to some. It's just more than what I 
personally need, and seems a bit outside the scope of the existing 
installer tools (as previously stated with the suggestion to use 
etckeeper).




Full tracking of user edits and system edits, seems like a very 
development-focused perspective on config management. Sometimes, a 
sysadmin just wants to see what's different *now* without too much more 
detail.


For now, I think I'm going to write a script to parse the output of 
`rpm -V` and then `dnf download` all the packages corresponding to 
changed configs, then extract each rpm's %config files, to compare 
with those on the system. It's messy, but it'll work, and in the 
absence of rpm storing the full configs or being able to instruct 
rpm/dnf to reset configs back to the defaults (after backing up), 
there doesn't seem to be an existing solution that will give me the 
same result.


It seems like there's already a little bit of precedent for "alternate 
trees" representing the existing file system with the debuginfo rpms and 
/usr/lib/debug/ hierarchy. Although the easiest solution might be to 
have RPM deposit a duplicate copy of *all* config files (not just 
%config(noreplace)) adjacent to the config file actually read by the 
programs (e.g., /etc/httpd/conf.d/ssl.conf and 
/etc/httpd/conf.d/ssl.conf.rpmdist), that leaves lots of ugly, mostly 
unused, files everywhere.


Perhaps RPM (or yum/dnf, via plugin) could write a duplicate copy of all 
config files into a tree somewhere? (E.g., /usr/lib/config/ or 
/usr/share/config/?) RPM/yum/dnf could then do the equivalent of an `rpm 
-qlca` and compare each /etc/httpd/conf.d/ssl.conf to 
/usr/lib/config/etc/httpd/conf.d/ssl.conf with no real magic or complex 
tech needed. Merging left up to the sysadmin.


Having everything in one location might also give some future options 
for automated compression, refreshing scripts, and the like.


Regards,
-jc
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-15 Thread Colin Walters 
On Tue, Dec 15, 2015, at 06:43 PM, Japheth Cleaver wrote:
>
>
Perhaps RPM (or yum/dnf, via plugin) could write a duplicate copy of
all config files into a tree somewhere? (E.g., /usr/lib/config/ or
/usr/share/config/?)

I mentioned this above, but might as well repeat since it was missed;
OSTree (as used by the existing Fedora Atomic Host) does this by default
today in /usr/etc, so if one was adapting this change to the client-side
system assembly tools like yum/dnf, I'd say it would make sense to
follow the precedent.


--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-14 Thread Fabian Deutsch 
On Sun, Dec 13, 2015 at 5:58 AM, Christopher  wrote:
> Which components/packages are best candidates for adding a feature which
> would make it easier for users to track changes from the default %config
> %files on the system?
>
> I've found this to be a deficiency, requiring users to do configuration
> management independently of the installer tools on a system, and I think the
> situation could be improved by adding functionality to the installation
> tooling to track these changes better.
>
> For example, I can see which %config files have changed with `rpm -V`, but I
> can't see what the changes actually are unless I do `dnf download $myrpm`,
> extract it, and diff them. Alternatively, I could rename the configuration
> file, and do a `dnf reinstall $myrpm` to replace the original, and then diff
> them. Both of these methods are clunky, wasteful, and not without
> side-effects.

The naive approaches of diff and rebase work for some or even most config files.
But a more sustainable approach is probably to make applications aware
of this paradigm shift ([partial] configs in multiple locations), and
allow them to look in multiple locations and also allow overrides with
partial configurations like we know it from systemd [0].

This could potentially even solve the issue which we - oVirt Node -
and also Atomic see with /etc/passwd - where the naive approach does
not work.

- fabian

--
[0] http://www.freedesktop.org/software/systemd/man/systemd.unit.html
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-14 Thread Christopher 
On Sun, Dec 13, 2015 at 2:50 PM Reindl Harald 
wrote:

>
> Am 13.12.2015 um 05:58 schrieb Christopher:
> > rpm could track more than hashes of config files, and instead track the
> > full file. This could be optional, as it uses more disk space, but disk
> > space is cheap these days, and config files are relatively small. This
> > would avoid having to re-download the rpm, and would make it easier to
> > see what has been modified on their system. So, some users may find that
> > a worthwhile trade-off
>
> first:  disk space is *not* cheap these days *
> second: etckeeper exists
> third:  no need to re-invent the wheel
>
>
I meant that it's cheaper than it ever has been, and that the marginal
costs of adding a 100MB or so is very small compared to the overall size of
the disk. Certainly, it wouldn't be appropriate for all cases, but I feel
confident saying that the average user isn't going to notice, in terms of
disk utilization or costs, if the size of /etc doubles. Honestly, I didn't
think that would be a contentious premise, and it's not exactly the main
point of what I was trying to say. I did point out that it would be
worthwhile only for "some users".

Didn't know about etckeeper. Suggestions like that are exactly why I raised
the question. I'm not suggesting the wheel be reinvented. I'm trying to
figure out whether there exists an appropriate wheel to use, and if not,
what materials should we build one out of.

After looking into it, I think etckeeper might be a bit overkill (for my
use case, at least). While etckeeper hooks the update system to track
changes between updates, I'm more interested in knowing the answer to the
question "what is the difference between the current package maintainer's
version, and the current version on my system?", at any point in time. In
other words, "what has the user modified?". Currently, the only time that
is easily answered is when an *.rpmnew file exists, because that's when I
have a copy of the package maintainer's unmodified version to compare with.
However, I'd like to be able to ask this of any system, at any time. I'm
not sure a VCS really helps all that much.

99% of all users don't care and the yone who cares using a VCS for /etc
> like "etckeeper" while i never ever would use that directly on
> production servers but on a admin-server pull from the infrastrcuture
> and fire etckepper there - works like a charme for many year
>
>
You're right, and I agree that most users don't care. This makes it hard to
support/troubleshoot user's systems, when they've changed from the working
defaults. I'm sure there's other use cases (I've seen some interesting ones
in this thread), but my primary interest is having baked-in configuration
management, when users don't do it themselves.

If users were good at configuration management, they'd keep a record of
everything they change, and this wouldn't be an issue. I'm more concerned
about asking the system the question about what the user has edited,
precisely because I want to support users who aren't good at configuration
management, and don't keep track of what they've changed from the defaults.
That's hard to do in Fedora... there isn't even really a way to easily
"reset Fedora to defaults" using yum/dnf/rpm, because there isn't an easy
way to retrieve the defaults. That's essentially the core of this problem:
how to easily retrieve the (current) defaults.



> *
> disk space maybe cheap for some fire-and-forget machines, but not on
> enterprise storage hosting hundrets of instances
> 
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-14 Thread Christopher 
On Sun, Dec 13, 2015 at 3:39 PM Reindl Harald 
wrote:

>
>
> Am 13.12.2015 um 21:22 schrieb Andrew Lutomirski:
> > For the few cases that can't or won't comply, then having rpm
> > (optionally?) make the originals available would be fantastic for
> > system management
>
> how many copies would you like to store in the rpm database and how to
> access them in a useful manner
>
>
For my purposes, exactly 1 copy would be needed: the one contained in the
currently installed packages, before any user modifications.


> define "originals"
>
>
For me, I'd want the up-to-date one from the current version of the
installed packages, not the initial state... just the up-to-date state as
determined by the package maintainer. My main interest is finding what the
user changed, to make it different from the packager's defaults.

when the system was installed versus current state?
> worthless - most of my Fedora setups are from 2008
>
> the current and the previous version?
> well, you need to handle cases where a config file is unchanged but the
> package is updated, that's the majority of all updates
>
>

> honestly:
> that all violates the unix principles one tool for one task and there
> are tools available for config file management many years, people
> interested just need to use them, "etckeeper" is integrated in dnf/yum
> and makes a versioned "snapshot" of /etc before and after updates are
> applied
>
>
For me, I just want to know what the user changed. (like how `rpm -V` shows
that a file has been modified from the package version with an md5 check...
but I want to know the content of those modifications). I'm not sure what
the best tool for this task is, but I'm pretty sure etckeeper isn't it. rpm
is already doing similar functionality, but not tracking the full files,
only the md5 hashes. yum/dnf is also doing similar functionality when it
creates *.rpmnew files, but only when the packager's version is updated...
essentially I'd want *.rpmnew to be created every time (and left on the
system, never being deleted) so it can be compared with the user version.

I'm not sure I agree that UNIX principles are to have one tool for one
task. It is common for there to be many tools capable of achieving the same
task. I think the principle is to have one task for each tool and to do
that task well (in other words, tools shouldn't be bloated, but they can
certainly have alternatives). But, that's not even what we're talking about
here. At least, it's not what I'm talking about. What I'm talking about is:
1) trying to find the best tool for the task if it exists, 2) determining
if a tool must be created, and 2) figuring out if features must be added to
the underlying package management system to support those tools.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-14 Thread Reindl Harald 



Am 14.12.2015 um 17:01 schrieb Christopher:

On Sun, Dec 13, 2015 at 3:39 PM Reindl Harald

Re: Easier %config management?

2015-12-14 Thread J. Randall Owens 
On 12/14/2015 02:47 PM, Christopher wrote:
> On Mon, Dec 14, 2015 at 11:22 AM Reindl Harald  > wrote:
> 
> i modified my "httpd.conf" based on Apache 2.2 years ago, as Fedora
> swicthed to Apache 2.4 your approach would have compared a customized
> Apache 2.2 version with a Fedora 2.4 version
> 
> 
> But, that's precisely what I want to do I want to know which fields
> in the current configuration could possibly have introduced a breakage I
> see... or which changes I need to make from a clean installation to get
> a user back to the state they wanted.
>  

But in order to do the former, you'd best compare the new default to the
old default, to see what had changed in the default. (E.g. distro 2.2 to
distro 2.4) To do the latter, you'd compare the old default (distro 2.2)
to your modified version (altered 2.2), to see what you'd changed
locally. And yet, you seem to be proposing comparing the new default to
the old default (distro 2.4 to altered 2.2), which gets you neither? (Or
both, really, but uselessly mashed together so you don't know which is
which.) Am I missing something?

-- 
J. Randall Owens | http://www.ghiapet.net/



signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-14 Thread Christopher 
On Mon, Dec 14, 2015 at 11:22 AM Reindl Harald 
wrote:

>
>
> Am 14.12.2015 um 17:01 schrieb Christopher:
> > On Sun, Dec 13, 2015 at 3:39 PM Reindl Harald  > For me, I'd want the up-to-date one from the current version of the
> > installed packages, not the initial state... just the up-to-date state
> > as determined by the package maintainer. My main interest is finding
> > what the user changed, to make it different from the packager's defaults.
> >
> > when the system was installed versus current state?
> > worthless - most of my Fedora setups are from 2008
> >
> > the current and the previous version?
> > well, you need to handle cases where a config file is unchanged but
> the
> > package is updated, that's the majority of all updates
> >
> > For me, I just want to know what the user changed
>
> what you propose can't work for that because you only compare the
> *current users* version with the *current package* version
>
>
Yes, that's all I want.


> that is a naive approach
>
>
Yes, but the naive approach satisfies my use case. Others may have more
needs, but this is all I need. As an SA, I don't need to track the full
history of the packages (at least, not on every user's machine), and
there's easier ways to get that anyway (look in cgit and/or upstream VCS).
I just need to periodically check in on what the user has tampered with
(for example, when they come to me for assistance when something is
broken), or to periodically back up or audit their changes.


> i modified my "httpd.conf" based on Apache 2.2 years ago, as Fedora
> swicthed to Apache 2.4 your approach would have compared a customized
> Apache 2.2 version with a Fedora 2.4 version
>
>
But, that's precisely what I want to do I want to know which fields in
the current configuration could possibly have introduced a breakage I
see... or which changes I need to make from a clean installation to get a
user back to the state they wanted.


> the same happens to anything which has large changes over time
>
> the moment when config formats are changing is the one where it starts
> to become interesting and at that moment is completly wortless to
> compare different generations of a config file without the full history
>
>
I would disagree that it's completely worthless, but I agree that it's more
difficult. But, this is where I go to the docs anyway: "K1 did F in version
V1, but K2 does F in version V2". I would never migrate configs based on
history; I do it based on the docs.


> what you *really* would need to compare at *that moment* is your changes
> years ago based on the dist-version of the config file at the same moment
>
>
No, I don't... not for my use case. I want to know what differences are to
either fix a problem a user is having (when a default config works), or to
back up those differences so I can reinstall a clean system and get a user
back to the state they had before.


> that's nothing you can or should try to cover with rpm - try to solve
> this with 1 or 2 limited copies would fail after dist-upgrades as i have
> already said
>
>
I agree... I don't think rpm should do all that. But, again, that's not
what I need, nor what I'm suggesting rpm be able to do.

compare two versions is worthless, you need at least three to *get a
> context*
>
> * previous dist version - CAUTION
> * your current version
> * now to install/installed dist-version
>
> which previous dist-version owul dbe helpful depends, the most
> interesting one is that from where user changes derived and to answer
> that question you need more data than one copy because form the first
> change on the *users version* will always differ, but from which distro
> state did that happen
>
> 
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-14 Thread Panu Matilainen 

On 12/13/2015 10:50 PM, Andrew Lutomirski wrote:

On Dec 13, 2015 12:38 PM, "Reindl Harald"  wrote:




Am 13.12.2015 um 21:22 schrieb Andrew Lutomirski:


For the few cases that can't or won't comply, then having rpm
(optionally?) make the originals available would be fantastic for
system management



how many copies would you like to store in the rpm database and how to

access them in a useful manner

Exactly one, for the current installed version.



honestly:
that all violates the unix principles one tool for one task and there are

tools available for config file management many years, people interested
just need to use them, "etckeeper" is integrated in dnf/yum and makes a
versioned "snapshot" of /etc before and after updates are applied

Since when is there a UNIX principle that, just because a mediocre solution
exists, a better solution shouldn't be added.

In any case, most existing tools really struggle with "what has changed on
my configuration relative to what I'd have if I reinstalled?".


With rpm >= 4.12 its possible to create a plugin which grabs the 
original config files from packages as they're unpacked, and see the 
.rpmnew/.rpmsave/.rpmorig decisions in "real-time" etc.


So you could stuff the configs into a VCS, perform intelligent merges 
and such. In theory at least ;)


- Panu -
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-13 Thread Jan Kratochvil 
On Sun, 13 Dec 2015 05:58:47 +0100, Christopher wrote:
> For example, I can see which %config files have changed with `rpm -V`, but
> I can't see what the changes actually are unless I do `dnf download
> $myrpm`, extract it, and diff them.

Using this script of mine.  It keeps original unchanged configuration files
from rpm in ~/rpmmerge/** so that you can diff local changes:
http://git.jankratochvil.net/?p=nethome.git;a=blob;f=bin/rpmmerge
One needs to run the command after OS install and best both before+after each
'yum/dnf upgrade'.


> rpmconf is nice, because it helps me easily compare configuration files
> whose user-changes and maintainer-changes conflict... but that's not quite
> the same thing.

Neither rpmconf nor anything else in Fedora keeps the original configuration
file of the installed NVRA.  Therefore after 'yum/dnf upgrade' you have:
  * old-NVRA modified config file
  * new-NVRA original config file
And there is no way to _automatically_ merge them to get the needed:
  * new-NVRA modified config file
Because for that 3-way merge you need also
  * old-NVRA original config file
which is kept (and merged) by my 'rpmmerge' tool above.


> rpmconf might also need modification to support tracking configuration
> management more fully, rather than just for updates.

It should be primarily a default behavior of the default package management
tools.


Jan
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-13 Thread J. Randall Owens 
On 12/13/2015 01:07 AM, Jan Kratochvil wrote:
> On Sun, 13 Dec 2015 05:58:47 +0100, Christopher wrote:
>> For example, I can see which %config files have changed with `rpm -V`, but
>> I can't see what the changes actually are unless I do `dnf download
>> $myrpm`, extract it, and diff them.
> 
> Using this script of mine.  It keeps original unchanged configuration files
> from rpm in ~/rpmmerge/** so that you can diff local changes:
> http://git.jankratochvil.net/?p=nethome.git;a=blob;f=bin/rpmmerge
> One needs to run the command after OS install and best both before+after each
> 'yum/dnf upgrade'.
> 
> 
>> rpmconf is nice, because it helps me easily compare configuration files
>> whose user-changes and maintainer-changes conflict... but that's not quite
>> the same thing.
> 
> Neither rpmconf nor anything else in Fedora keeps the original configuration
> file of the installed NVRA.  Therefore after 'yum/dnf upgrade' you have:
>   * old-NVRA modified config file
>   * new-NVRA original config file
> And there is no way to _automatically_ merge them to get the needed:
>   * new-NVRA modified config file
> Because for that 3-way merge you need also
>   * old-NVRA original config file
> which is kept (and merged) by my 'rpmmerge' tool above.
> 
> 
>> rpmconf might also need modification to support tracking configuration
>> management more fully, rather than just for updates.
> 
> It should be primarily a default behavior of the default package management
> tools.
> 
> 
> Jan
> 

etckeeper basically makes a git tree of the entire /etc directory, and
automagically commits before & after each yum or dnf transaction (been a
while since I did a raw rpm install or erase, I don't think it kicks in
then though). It's a bit disk-hungry compared to the /etc dir, e.g.
right now I have 89 MiB of actual /etc, plus 142 MiB in the /etc/.git,
but /etc is modest enough that it isn't much of a problem. It does slow
down the yum/dnf transactions a bit, certainly, while git checks for
changes. But yeah, it stores the new, the modified, and the old, and
while I'm not too handy with git yet, I'm pretty sure someone better
with it than I could do that kind of merging you're talking about.

Actually, I take a little of that back; it ignores *.rpm* files, so it
doesn't see the contents of the *.rpmnew. But you'll have that right
there, and can easily work with that.

-- 
J. Randall Owens | http://www.ghiapet.net/



signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-13 Thread Reindl Harald 



Am 13.12.2015 um 21:50 schrieb Andrew Lutomirski:

On Dec 13, 2015 12:38 PM, "Reindl Harald" > wrote:
 >
 > Am 13.12.2015 um 21:22 schrieb Andrew Lutomirski:
 >>
 >> For the few cases that can't or won't comply, then having rpm
 >> (optionally?) make the originals available would be fantastic for
 >> system management
 >
 >
 > how many copies would you like to store in the rpm database and how
to access them in a useful manner

Exactly one, for the current installed version.


not terrible helpful


 > honestly:
 > that all violates the unix principles one tool for one task and there
are tools available for config file management many years, people
interested just need to use them, "etckeeper" is integrated in dnf/yum
and makes a versioned "snapshot" of /etc before and after updates are
applied

Since when is there a UNIX principle that, just because a mediocre
solution exists, a better solution shouldn't be added.


how can bloating the rpm-database in a very limited way be a better 
solution than having any changes below /etc versioned over years



In any case, most existing tools really struggle with "what has changed
on my configuration relative to what I'd have if I reinstalled?"


how is rpm here a solution?

how should it know which version is "if I reinstalled" after several 
dist-upgrades - keep the very first version of a config file makes no 
sense - how would a httpd.conf from 2008 help on a Fedora 23 with 
httpd-2.4 and the same for most other package over the time?




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-13 Thread Andrew Lutomirski 
On Dec 13, 2015 12:38 PM, "Reindl Harald"  wrote:
>
>
>
> Am 13.12.2015 um 21:22 schrieb Andrew Lutomirski:
>>
>> For the few cases that can't or won't comply, then having rpm
>> (optionally?) make the originals available would be fantastic for
>> system management
>
>
> how many copies would you like to store in the rpm database and how to
access them in a useful manner

Exactly one, for the current installed version.

>
> honestly:
> that all violates the unix principles one tool for one task and there are
tools available for config file management many years, people interested
just need to use them, "etckeeper" is integrated in dnf/yum and makes a
versioned "snapshot" of /etc before and after updates are applied

Since when is there a UNIX principle that, just because a mediocre solution
exists, a better solution shouldn't be added.

In any case, most existing tools really struggle with "what has changed on
my configuration relative to what I'd have if I reinstalled?".

>
>
> --
> devel mailing list
> devel@lists.fedoraproject.org
> http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-13 Thread Richard W.M. Jones 
On Sun, Dec 13, 2015 at 04:58:47AM +, Christopher wrote:
> For example, I can see which %config files have changed with `rpm -V`, but
> I can't see what the changes actually are unless I do `dnf download
> $myrpm`, extract it, and diff them. Alternatively, I could rename the
> configuration file, and do a `dnf reinstall $myrpm` to replace the
> original, and then diff them. Both of these methods are clunky, wasteful,
> and not without side-effects.
[...]
> rpm could track more than hashes of config files, and instead track the
> full file. This could be optional, as it uses more disk space, but disk
> space is cheap these days, and config files are relatively small. This
> would avoid having to re-download the rpm, and would make it easier to see
> what has been modified on their system. So, some users may find that a
> worthwhile trade-off.

Supermin has this problem too.  It solves it by downloading the
original RPMs ('dnf download', not 'dnf reinstall'), unpacking them
('rpm2cpio'), etc. but it's a pain in the rear.

  http://libguestfs.org/supermin.1.html

I'm guessing also this is something that Fedora Atomic has had to
solve.

Also, it would be better if packages by default never needed
configuration files.  You would only add a configuration file when you
need to change the default configuration.  A default installation of
Fedora would have an empty /etc (solving your problem ... in a
slightly different way).  Some packages have been heading in this
direction.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-13 Thread Reindl Harald 


Am 13.12.2015 um 05:58 schrieb Christopher:

rpm could track more than hashes of config files, and instead track the
full file. This could be optional, as it uses more disk space, but disk
space is cheap these days, and config files are relatively small. This
would avoid having to re-download the rpm, and would make it easier to
see what has been modified on their system. So, some users may find that
a worthwhile trade-off


first:  disk space is *not* cheap these days *
second: etckeeper exists
third:  no need to re-invent the wheel

99% of all users don't care and the yone who cares using a VCS for /etc 
like "etckeeper" while i never ever would use that directly on 
production servers but on a admin-server pull from the infrastrcuture 
and fire etckepper there - works like a charme for many year


*
disk space maybe cheap for some fire-and-forget machines, but not on 
enterprise storage hosting hundrets of instances




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-13 Thread Reindl Harald 



Am 13.12.2015 um 21:22 schrieb Andrew Lutomirski:

For the few cases that can't or won't comply, then having rpm
(optionally?) make the originals available would be fantastic for
system management


how many copies would you like to store in the rpm database and how to 
access them in a useful manner


define "originals"

when the system was installed versus current state?
worthless - most of my Fedora setups are from 2008

the current and the previous version?
well, you need to handle cases where a config file is unchanged but the 
package is updated, that's the majority of all updates


honestly:
that all violates the unix principles one tool for one task and there 
are tools available for config file management many years, people 
interested just need to use them, "etckeeper" is integrated in dnf/yum 
and makes a versioned "snapshot" of /etc before and after updates are 
applied




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-13 Thread Andrew Lutomirski 
On Sat, Dec 12, 2015 at 8:58 PM, Christopher  wrote:
> rpm could track more than hashes of config files, and instead track the full
> file. This could be optional, as it uses more disk space, but disk space is
> cheap these days, and config files are relatively small. This would avoid
> having to re-download the rpm, and would make it easier to see what has been
> modified on their system. So, some users may find that a worthwhile
> trade-off.
>

I rather like this idea.

My laptop has 35MB of /etc contents.  My laptop has 240M of conffiles.
I suspect it has rather less that that in stock conffiles (the
majority is the RPM DB itself, and most of that is presumably shipped
empty).

Doing this might add a considerable incentive for packages to fix the
fact that they have large conffiles.  For example:

106M/usr/lib/locale/locale-archive

excuse me?

80K/usr/share/ghostscript/9.16/Resource/Init/gs_init.ps

not that big, but that has to be a bug.

Buggy things aside, we really should move to a model in which pretty
much everything lives in /usr/share and /etc is just overrides.  For
example:

56K/etc/mime.types
58K/etc/mail/sendmail.cf
60K/etc/mono/2.0/DefaultWsdlHelpGenerator.aspx
60K/etc/mono/4.0/DefaultWsdlHelpGenerator.aspx
60K/etc/mono/4.5/DefaultWsdlHelpGenerator.aspx
64K/etc/pki/nssdb/cert8.db
70K/etc/jwhois.conf
81K/etc/lvm/lvm.conf

For the few cases that can't or won't comply, then having rpm
(optionally?) make the originals available would be fantastic for
system management.

--Andy
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Easier %config management?

2015-12-13 Thread Colin Walters 


On Sun, Dec 13, 2015, at 02:41 PM, Richard W.M. Jones wrote:

> > rpm could track more than hashes of config files, and instead track the
> > full file. This could be optional, as it uses more disk space, but disk
> > space is cheap these days, and config files are relatively small. This
> > would avoid having to re-download the rpm, and would make it easier to see
> > what has been modified on their system. So, some users may find that a
> > worthwhile trade-off.

See below.

> I'm guessing also this is something that Fedora Atomic has had to
> solve.

Yes, OSTree mandates that the config defaults live in /usr/etc and are
immutable, then when creating a "deployment" a 3-way merge is
performed.  Just ls -al /usr/etc on a Fedora Atomic host.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Easier %config management?

2015-12-12 Thread Christopher 
Which components/packages are best candidates for adding a feature which
would make it easier for users to track changes from the default %config
%files on the system?

I've found this to be a deficiency, requiring users to do configuration
management independently of the installer tools on a system, and I think
the situation could be improved by adding functionality to the installation
tooling to track these changes better.

For example, I can see which %config files have changed with `rpm -V`, but
I can't see what the changes actually are unless I do `dnf download
$myrpm`, extract it, and diff them. Alternatively, I could rename the
configuration file, and do a `dnf reinstall $myrpm` to replace the
original, and then diff them. Both of these methods are clunky, wasteful,
and not without side-effects.

rpmconf is nice, because it helps me easily compare configuration files
whose user-changes and maintainer-changes conflict... but that's not quite
the same thing.

Here's some things I was thinking:

rpm could track more than hashes of config files, and instead track the
full file. This could be optional, as it uses more disk space, but disk
space is cheap these days, and config files are relatively small. This
would avoid having to re-download the rpm, and would make it easier to see
what has been modified on their system. So, some users may find that a
worthwhile trade-off.

Alternatively, dnf could also add an '--update-configs' option to the
'reinstall' command, to force the creation of rpmorig/rpmnew files for
reinstall (currently these only get created during updates, as far as I
understand it). This won't avoid the re-download, but might be an easier
solution, and wouldn't require the extra disk space (after the rpm is
reinstalled). This option would enable the existing rpmconf to do its job.

rpmconf might also need modification to support tracking configuration
management more fully, rather than just for updates.

I'm sure there's other components/solutions (in general, I find `git init
/etc/` to be quite convenient, but it's a messy hack).

Any thoughts? Is this an area which can be improved easily? Are there
existing solutions within Fedora? Something we could bring in? Specific
upstream packages we can target for feature requests?
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org