Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Chris Murphy]

On Tue, Jan 11, 2022 at 9:42 AM Lukáš Hrázký  wrote:
>
> Chris, apologies for the late reply.
>
> On Mon, 2021-12-13 at 13:02 -0500, Chris Murphy wrote:
> > On Mon, Dec 13, 2021 at 10:07 AM Lukáš Hrázký  wrote:
> > >
> > > On Fri, 2021-12-10 at 17:08 +0100, Daniel Mach wrote:
> >
> > > > I understand your point about auditing, but does DNF have to handle
> > > > everything?
> > > > There should be better options of tracking filesystem rollbacks than
> > > > DNF's history database.
> > > >
> > >
> > > Maybe, but the history DB still contains a log of events as they
> > > happened over time, I don't think that belongs to /usr.
> >
> > How about:
> >
> > For read-write /usr store any database describing /usr state in
> > /usr/lib/sysimage
> > For read-only /usr store any database describing /usr state in
> > /var/lib/dnf with a /usr/lib/sysimage symlink pointing to it
> > /var is always read-write so any database describing /var state goes
> > in /var/lib/dnf
>
> That seems unnecessarily complex, and also:
> - if there's no symlink on read-only /usr/lib/sysimage, we wouldn't be
> able to create one
>
> - if you consider /usr and /var loosely coupled, linking from
> /usr/lib/sysimage to /var means the link could break any time or the
> content just not be what you expect anyway
>
> > This suggests that anything that can describe state, should have
> > separate databases for /usr and /var. I don't know to what degree dnf
> > touches various top level FHS directories: /var /usr /etc /home and so
> > on. But I wonder if the history databases should separately track the
> > things being touched?
>
> Note dnf only deals with installing RPMs, we have no means to track
> changes to particular files and thus track changes across various
> directories under root.
>
> My tentative conclusion from the disucssion so far is that we'll put
> the system state files to /usr/lib/sysimage in dnf 5 (I dislike the
> "sysimage" part of the path but we're not going to break the
> established convention I don't think). And the history db will likely
> stay in /var/lib. This adheres to KISS and anything more involved seems
> to not solve issues, only move them around while adding complexity.

The Fedora 36 change proposal thread also now a sub-thread proposing a
new top-level directory for this purpose, /state. Easiest way to find
it it go to the main thread then search for /state
https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/G6LFNGOXZXFWZM3IWT6MLZWGMNSIZCWM/#XG4AT7NLMYZD2UHPG35JLPYWDSR5UAZT

If dnf history remains in /var/lib/dnf it needs to be snapshot and
rolled back with /usr in order for things like dnf history, and its
subcommands. If it were to go in /state along with rpmdb, it might
work out better? Although that suggests /state needs to be snapshot
and rolled back along with /usr as well.


-- 
Chris Murphy
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Lukáš Hrázký]

Chris, apologies for the late reply.

On Mon, 2021-12-13 at 13:02 -0500, Chris Murphy wrote:
> On Mon, Dec 13, 2021 at 10:07 AM Lukáš Hrázký  wrote:
> > 
> > On Fri, 2021-12-10 at 17:08 +0100, Daniel Mach wrote:
> 
> > > I understand your point about auditing, but does DNF have to handle
> > > everything?
> > > There should be better options of tracking filesystem rollbacks than
> > > DNF's history database.
> > > 
> > 
> > Maybe, but the history DB still contains a log of events as they
> > happened over time, I don't think that belongs to /usr.
> 
> How about:
> 
> For read-write /usr store any database describing /usr state in
> /usr/lib/sysimage
> For read-only /usr store any database describing /usr state in
> /var/lib/dnf with a /usr/lib/sysimage symlink pointing to it
> /var is always read-write so any database describing /var state goes
> in /var/lib/dnf

That seems unnecessarily complex, and also:
- if there's no symlink on read-only /usr/lib/sysimage, we wouldn't be
able to create one

- if you consider /usr and /var loosely coupled, linking from
/usr/lib/sysimage to /var means the link could break any time or the
content just not be what you expect anyway

> This suggests that anything that can describe state, should have
> separate databases for /usr and /var. I don't know to what degree dnf
> touches various top level FHS directories: /var /usr /etc /home and so
> on. But I wonder if the history databases should separately track the
> things being touched?

Note dnf only deals with installing RPMs, we have no means to track
changes to particular files and thus track changes across various
directories under root.

My tentative conclusion from the disucssion so far is that we'll put
the system state files to /usr/lib/sysimage in dnf 5 (I dislike the
"sysimage" part of the path but we're not going to break the
established convention I don't think). And the history db will likely
stay in /var/lib. This adheres to KISS and anything more involved seems
to not solve issues, only move them around while adding complexity.

Cheers,
Lukas

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Ludwig Nussel]

Florian Weimer wrote:
> * Colin Walters:
> [...]
>> What I *did* wake up one day and say I'm going to do is make upgrades
>> transactional and offline by default and hence safe.  No one should
>> ever fear starting an operating system upgrade while their laptop is
>> at 30% battery.  Administrators running important servers must be able
>> to easily roll back when the kernel *or* systemd (or something) else
>> regresses, because it's software, it regresses all the time despite
>> our best efforts.
> 
> I appreciate these efforts.
> 
> Although transactional-update (as documented below) seems have one major
> regression: transactional RPM updates now require reboots. 8-(

Yes, that's a PITA. However, I'm sure if we wanted to invest some brains
the mechanism could be made smarter to figure out what is actually safe
to apply live somehow.

>> So yes again, this does matter.  And it matters because whether you're
>> doing "image based upgrades" like ostree or just "client side offline
>> updates" like the
>> https://kubic.opensuse.org/documentation/man-pages/transactional-update.8.html
>> thing - it's very important *what data specifically* is
>> versioned/snapshotted and what isn't.  On an ostree system for
>> example, it's completely normal that there are *two* rpm databases
>> (one you're running, one that's pending in the new root).
>>
>> All the data in the rpmdb is about content that's in `/usr`.
> 
> That totally ignores Software Collections and packages from ISVs.  If
> the expected future that RPM is going to be an OS-internal software
> deployment mechanism (and not be used for third-party software), it
> should not be a side effect of this change, but an explicit decision.

AFAIK at least for SLE Micro which uses the same tech as openSUSE
MicroOS ie transactional updates, customers are expected to install
extra software as container. The OS is not meant to install rpms from
sources other than it's own repo. So for that product it was an explicit
decision that rpm is only for OS stuff.
However, that policy only applies to the host. Inside containers rpm can
be used in whatever creative way as usual.

Anyway, what's happening is that in order to make OS updates more
robust, a stricter separation of what is OS and what isn't gets
implemented. The read-only bits of the OS have converged into /usr
nowadays¹. It's expected to be kind of self contained. Systems can now
even boot with empty /etc and /var as everything needed in there gets
bootstrapped from content in /usr.
If rpm is used to manage /usr content, all required information for that
needs to be in /usr too and rpms have no business outside /usr anymore.
Since rpm keeps package information in the rpmdb, the obvious next step
is to demand moving the rpmdb to /usr.

I think a reason why that doesn't feel right is because the rpmdb is a
single binary blob that mixes different kinds of data. Maybe it could be
split up.
Package headers could be considered original data, strictly bound to the
files the header refers to. So for OS files the headers would belong
into /usr. Name indexes etc however are derivative data that can be
reconstructed from headers. As such could be stored in /var and
generated when needed², eg when actually installing packages or after
rollback.

I've suggested storing headers separately here:
https://github.com/rpm-software-management/rpm/issues/1151
I acknowledge that the title is misleading though as that doesn't mean
no database, just different format :-)

If the headers are stored where the files are that would open the
possibility to define additional prefixes such as /opt or /usr/local
with their own headers. The database in /var could merge the information
to have a unified view. That would allow to have /usr read-only while
still being able to install 3rd party add-ons in separate locations.

cu
Ludwig

[1] https://lnussel.github.io/2020/12/16/fslayout/
[2] File flags (eg skipped files due to --excludedocs) are computed and
stored on installation time. Would have to think how and where to store
that. Or maybe not support them in certain scenarios.

-- 
 (o_   Ludwig Nussel
 //\
 V_/_  http://www.suse.com/
SUSE Software Solutions Germany GmbH, GF: Ivo Totev
HRB 36809 (AG Nürnberg)
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Florian Weimer]

* Colin Walters:

> On Wed, Dec 15, 2021, at 5:34 PM, Florian Weimer wrote:
>> * Chris Murphy:
>>
>>> Fedora 36 seems like a good time to do this. What do you think?
>>
>> It's a bit odd to locate a database under /usr that isn't pre-built and
>> installed.  
>
> Why is that odd?

Mentally, I still associate /usr with something that can be mounted
read-only from shared storage.

>> I guess in theory there could be systems with a read-only
>> /usr out there that still allow installation of packages into /opt.
>>
>> Does it really help to improve the snapshot situation? 
>
> Yes.  
>
> I didn't wake up one day and say "hey you know what, today I'm going
> to move the rpm database just for fun".  Neither, for that matter did
> the OpenSUSE folks.  We haven't had this conversation over and over
> throughout the years just because it was some minor thing.
>
> What I *did* wake up one day and say I'm going to do is make upgrades
> transactional and offline by default and hence safe.  No one should
> ever fear starting an operating system upgrade while their laptop is
> at 30% battery.  Administrators running important servers must be able
> to easily roll back when the kernel *or* systemd (or something) else
> regresses, because it's software, it regresses all the time despite
> our best efforts.

I appreciate these efforts.

Although transactional-update (as documented below) seems have one major
regression: transactional RPM updates now require reboots. 8-(

> So yes again, this does matter.  And it matters because whether you're
> doing "image based upgrades" like ostree or just "client side offline
> updates" like the
> https://kubic.opensuse.org/documentation/man-pages/transactional-update.8.html
> thing - it's very important *what data specifically* is
> versioned/snapshotted and what isn't.  On an ostree system for
> example, it's completely normal that there are *two* rpm databases
> (one you're running, one that's pending in the new root).
>
> All the data in the rpmdb is about content that's in `/usr`.

That totally ignores Software Collections and packages from ISVs.  If
the expected future that RPM is going to be an OS-internal software
deployment mechanism (and not be used for third-party software), it
should not be a side effect of this change, but an explicit decision.

Thanks,
Florian

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Colin Walters]

On Wed, Dec 15, 2021, at 5:34 PM, Florian Weimer wrote:
> * Chris Murphy:
>
>> Fedora 36 seems like a good time to do this. What do you think?
>
> It's a bit odd to locate a database under /usr that isn't pre-built and
> installed.  

Why is that odd?  

> I guess in theory there could be systems with a read-only
> /usr out there that still allow installation of packages into /opt.
>
> Does it really help to improve the snapshot situation? 

Yes.  

I didn't wake up one day and say "hey you know what, today I'm going to move 
the rpm database just for fun".  Neither, for that matter did the OpenSUSE 
folks.  We haven't had this conversation over and over throughout the years 
just because it was some minor thing.

What I *did* wake up one day and say I'm going to do is make upgrades 
transactional and offline by default and hence safe.  No one should ever fear 
starting an operating system upgrade while their laptop is at 30% battery.  
Administrators running important servers must be able to easily roll back when 
the kernel *or* systemd (or something) else regresses, because it's software, 
it regresses all the time despite our best efforts.

So yes again, this does matter.  And it matters because whether you're doing 
"image based upgrades" like ostree or just "client side offline updates" like 
the 
https://kubic.opensuse.org/documentation/man-pages/transactional-update.8.html 
thing - it's very important *what data specifically* is versioned/snapshotted 
and what isn't.  On an ostree system for example, it's completely normal that 
there are *two* rpm databases (one you're running, one that's pending in the 
new root).  

All the data in the rpmdb is about content that's in `/usr`.


> What about
> software under /opt? 

https://github.com/coreos/rpm-ostree/issues/233
Today indeed, rpm-ostree explicitly denies that.  Which, note we can do because 
we basically take over chunks of what librpm is doing on non-ostree systems.  
But, it would be nice to drive some of this support into librpm too so it 
applies to non-ostree-but-snapshottable systems.


>  Maybe this needs multiple RPM databases
> eventually, roughly aligned along file system boundaries.

Yeah, that's one approach, though I would scope it really to just "/usr rpmdb" 
and "/usr/local" rpmdb - i.e. `/opt` and `/usr/local` are basically the same 
thing with different names.
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Chris Murphy]

On Mon, Dec 13, 2021 at 10:09 AM Lukáš Hrázký  wrote:
>
> On Fri, 2021-12-10 at 13:53 -0500, Chris Murphy wrote:

> > For my non-emphemeral systems:
> >
> > * /usr /var /etc are directories contained in btrfs subvolume
> > "root-x86-64:fedora.35"
> > * read-only snapshots have an ISO 8601 datetime format tacked onto the end
> >
> > * /var/log is a directory contained in subvolume "varlog-x86-64:fedora.35"
> > * /var/lib/libvirt/images is a directory contained in subvolume
> > "varlibvirtimages-x86-64:fedora.35"
> > * no snapshots are ever made of these; by virtue of themselves being
> > subvolumes, any snapshot of "root-x86-64:fedora.35" doesn't contain
> > /var/log or /var/lib/libvirt/images so just like /home, they only ever
> > carry forward in time.
> >
> > Therefore in my case, snapshot and rollback doesn't affect any logs,
> > including the systemd journal. They only ever carry forward in time.
> > Same for VM images and for ~/ since that lives in /home and is a
> > separate subvolume too by default in Fedora. But both RPM database and
> > DNF history do get rolled back in this case. Which works out OK
> > because usr/ var/ etc/ are all in one root. A rollback means DNF
> > history still accurately describes that root's history and state.
> >
> > As soon as there's mix and match any of usr/ var/ and etc/ it gets trickier.
>
> So, is this basically the summary of your motivation for this change?
> That rollbacks of your /usr is always in sync with the dnf-stored
> system state?

Tentative answer is yes. But does dnf-stored system state mean /usr
state or /var state or both? And is anything else included?


> FWIW we could do with a practical evaluation of impact and issues this
> is causing now, are we talking desktops/servers/containers, etc.?

Desktops, servers, virtual machines. My thought about containers is
that they are self-contained, i.e. /usr and /var are paired together
for the life of a container, and there isn't a roll for mixing and
matching /usr and /var for a given container. There are technologies
that can make this trivially possible so will it happen? *shrug* no
idea. But right now if you had to do a rollback, I'd expect you start
a new container predicated on a previous state of a container image,
i.e. manage container versioning+snapshot+rollback within the
container realm, not with the tools inside of it. The idea of using
snapper to maintain many container variants inside a container just
seems a bit mind boggling to me but I guess it could happen.



> > > There are also issues with sqlite creating some lock files even if just
> > > reading, which wouldn't be possible on a RO FS (IIRC how it was
> > > explained to me), so there are other reasons why the history db will
> > > likely not move to /usr.
> >
> > Detect it and use overlayfs on a volatile /run directory for this use
> > case? I'm not sure the use case really calls for a persistent file
> > that survives a crash for reading read-only files?
>
> AFAICT it doesn't require a persistent file. But if you're suggesting
> for dnf itself to set up that overlayfs, I may be misjudging, but it
> seems way out of bounds of what dnf should be doing.

Neither sqlite nor dnf must have this ability, I'm suggesting they
could do it. As in, it's not improper. If dnf must create lock files
in order to read files on a read-only system, then the choice is (I'm
guessing) to either fail or to setup an overlay or even use /tmp in
order to not fail. And i'm not suggesting which choice is right or
wrong, just that no matter what there's some consequence.

Ideally, I'd expect a system should anticipate the read-only /usr
requirements, and that its initramfs would have the ability to detect
read-only /usr and setup a volatile overlay on /run using overlayfs,
similar to a LiveOS boot. In such a case, it's transparent to
software. It'll look like a read-write system. It's just that any
changes will be lost at reboot because all the changes are stored on a
volatile storage. And the UI is either "user expressly chose a rescue
boot which implies this kind of volatile environment" or "the
environment notifies the user that the environment is on volatile
storage".


> But the situation may be solvable in other ways, e.g. storing those
> files in a writable temporary location. I haven't investigated.

Sure.

> Tbh. given the ugliness and complexity added by history DB and how it
> stores the system state (you seem to agree the transaction log
> shouldn't be on /usr, but those two are stored together now) I doubt
> we'll resolve this before dnf 5, unless there's a contribution which
> neatly ties the loose ends (again, dnf 4 is ugly in a lot of ways...)

I'm definitely reluctant to consider conflating logs and state in the
same file. Those seem like really different things. A log can't
reliably infer state by replaying the log unless that log contains all
things that can affect state, including events that dnf doesn't
control. That's a tall order.

If dnf 

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Chris Murphy]

On Mon, Dec 13, 2021 at 10:07 AM Lukáš Hrázký  wrote:
>
> On Fri, 2021-12-10 at 17:08 +0100, Daniel Mach wrote:

> > I understand your point about auditing, but does DNF have to handle
> > everything?
> > There should be better options of tracking filesystem rollbacks than
> > DNF's history database.
> >
>
> Maybe, but the history DB still contains a log of events as they
> happened over time, I don't think that belongs to /usr.

How about:

For read-write /usr store any database describing /usr state in
/usr/lib/sysimage
For read-only /usr store any database describing /usr state in
/var/lib/dnf with a /usr/lib/sysimage symlink pointing to it
/var is always read-write so any database describing /var state goes
in /var/lib/dnf

This suggests that anything that can describe state, should have
separate databases for /usr and /var. I don't know to what degree dnf
touches various top level FHS directories: /var /usr /etc /home and so
on. But I wonder if the history databases should separately track the
things being touched?



-- 
Chris Murphy
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Lukáš Hrázký]

On Fri, 2021-12-10 at 13:53 -0500, Chris Murphy wrote:
> {
> Apologies for not sending to rpm-ecosystem@ first off [1], then b.
> since I wasn't subscribed to rpm-ecosystem@ my reply [2] was dropped,
> messing up the threading.
> [1] http://lists.rpm.org/pipermail/rpm-maint/2021-December/018662.html
> [2] http://lists.rpm.org/pipermail/rpm-maint/2021-December/018687.html
> }
> 
> On Fri, Dec 10, 2021 at 7:39 AM Lukáš Hrázký  wrote:
> > 
> > On Fri, 2021-12-10 at 06:16 -0500, Neal Gompa wrote:
> > > (Adding the correct Daniel Mach email address, he moved from Red Hat
> > > to SUSE last month...)
> > > 
> > > On Thu, Dec 9, 2021 at 10:11 AM Chris Murphy  
> > > wrote:
> > > > 
> > > > I'm pretty sure I'm generally opposed to the idea of logs being rolled
> > > > back (in a snapshot and rollback regime). Logs should always carry
> > > > forward. Ideally, indicate relevant things like rollbacks in the log,
> > > > because rolling back a log is a form of data loss. The history
> > > > database sounds more like a log than not. So I'm getting squeamish
> > > > about locating that somewhere else. Whereas system state information,
> > > > if it's primarily describing /usr, sounds like it needs to get
> > > > somewhere in /usr.
> > > > 
> > > 
> > > The history database is basically a transaction database like the
> > > rpmdb is. It maps rpmdb transactions to dnf actions.
> > 
> > I'm not sure this is true at all. I haven't examined the rpmdb ever,
> > but AFAIK it contains the list of currently installed rpms (and their
> > data), but not the history as they came and went (i.e. a log).
> > 
> > OTOH dnf history database is exactly that. The current system state is
> > in dnf 4 also computed as the aggregation of the transactions in the
> > history database, which is IMO very wrong and we're fixing it in dnf 5.
> 
> What's the timeframe for dnf 5 as it relates to the Fedora release cycle?
> 

The aim is to release dnf 5 in Fedora 38. Long before that we should
have a sort-of tech preview (as soon as it's somewhat consistently
working) and transition it into alpha/beta releases for testing, we'll
still be figuring out how to do that exactly.

> > > So if you're
> > > rolling back the rpm-managed content, it still makes sense to roll
> > > back the history database, since that's the state of the system. It's
> > > not terribly useful if the matching rpmdb transactions don't exist in
> > > the rpmdb.
> > 
> > Conceptually I agree with Chris on this: We shouldn't roll back the log
> > of transactions if we'd be rolling back the system itself. That way you
> > in effect erase that the system was running in a different state for an
> > amount of time, imagine an auditing situation.
> 
> Yeah, the auditing case is important to consider. The FHS can be
> aggravating in the context of snapshot and rollback regimes, in
> particular deciding whether a /usr /var and /etc are tied to each
> other for all time or if they can be mix and match. And that brings up
> how to name the snapshots (I've varied between simple version control
> by tacking on a .1 .2 .3 counter; to using ISO 8601 datetime stamps)
> and whether and how to do system assembly at startup [3].
> 
> [3] need for a discoverable subvolumes specification (thread crosses two 
> months)
> https://lists.freedesktop.org/archives/systemd-devel/2021-November/047059.html
> https://lists.freedesktop.org/archives/systemd-devel/2021-December/047176.html
> 
> For my non-emphemeral systems:
> 
> * /usr /var /etc are directories contained in btrfs subvolume
> "root-x86-64:fedora.35"
> * read-only snapshots have an ISO 8601 datetime format tacked onto the end
> 
> * /var/log is a directory contained in subvolume "varlog-x86-64:fedora.35"
> * /var/lib/libvirt/images is a directory contained in subvolume
> "varlibvirtimages-x86-64:fedora.35"
> * no snapshots are ever made of these; by virtue of themselves being
> subvolumes, any snapshot of "root-x86-64:fedora.35" doesn't contain
> /var/log or /var/lib/libvirt/images so just like /home, they only ever
> carry forward in time.
> 
> Therefore in my case, snapshot and rollback doesn't affect any logs,
> including the systemd journal. They only ever carry forward in time.
> Same for VM images and for ~/ since that lives in /home and is a
> separate subvolume too by default in Fedora. But both RPM database and
> DNF history do get rolled back in this case. Which works out OK
> because usr/ var/ etc/ are all in one root. A rollback means DNF
> history still accurately describes that root's history and state.
> 
> As soon as there's mix and match any of usr/ var/ and etc/ it gets trickier.

So, is this basically the summary of your motivation for this change?
That rollbacks of your /usr is always in sync with the dnf-stored
system state?

FWIW we could do with a practical evaluation of impact and issues this
is causing now, are we talking desktops/servers/containers, etc.?

So far this looks like it can wait for dnf 5.

> > There are also 

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Lukáš Hrázký]

On Fri, 2021-12-10 at 17:08 +0100, Daniel Mach wrote:
> On 10. 12. 21 13:39, Lukáš Hrázký wrote:
> > On Fri, 2021-12-10 at 06:16 -0500, Neal Gompa wrote:
> > > (Adding the correct Daniel Mach email address, he moved from Red Hat
> > > to SUSE last month...)
> > > 
> > > On Thu, Dec 9, 2021 at 10:11 AM Chris Murphy  
> > > wrote:
> > > > I'm pretty sure I'm generally opposed to the idea of logs being rolled
> > > > back (in a snapshot and rollback regime). Logs should always carry
> > > > forward. Ideally, indicate relevant things like rollbacks in the log,
> > > > because rolling back a log is a form of data loss. The history
> > > > database sounds more like a log than not. So I'm getting squeamish
> > > > about locating that somewhere else. Whereas system state information,
> > > > if it's primarily describing /usr, sounds like it needs to get
> > > > somewhere in /usr.
> > > > 
> > > The history database is basically a transaction database like the
> > > rpmdb is. It maps rpmdb transactions to dnf actions.
> > I'm not sure this is true at all. I haven't examined the rpmdb ever,
> > but AFAIK it contains the list of currently installed rpms (and their
> > data), but not the history as they came and went (i.e. a log).
> > 
> > OTOH dnf history database is exactly that. The current system state is
> > in dnf 4 also computed as the aggregation of the transactions in the
> > history database, which is IMO very wrong and we're fixing it in dnf 5.
> > 
> > > So if you're
> > > rolling back the rpm-managed content, it still makes sense to roll
> > > back the history database, since that's the state of the system. It's
> > > not terribly useful if the matching rpmdb transactions don't exist in
> > > the rpmdb.
> > Conceptually I agree with Chris on this: We shouldn't roll back the log
> > of transactions if we'd be rolling back the system itself. That way you
> > in effect erase that the system was running in a different state for an
> > amount of time, imagine an auditing situation.
> 
> As a user, I wouldn't expect history db to contain transactions that are 
> not relevant to my system.
> If I do filesystem rollback, I'd probably want to rollback even the 
> history db.
> Also, if we don't rollback history db too, it's going to break `dnf 
> history rollback` functionality.
> 
> I understand your point about auditing, but does DNF have to handle 
> everything?
> There should be better options of tracking filesystem rollbacks than 
> DNF's history database.
> 

Maybe, but the history DB still contains a log of events as they
happened over time, I don't think that belongs to /usr.

> > There are also issues with sqlite creating some lock files even if just
> > reading, which wouldn't be possible on a RO FS (IIRC how it was
> > explained to me), so there are other reasons why the history db will
> > likely not move to /usr.
> If it's the same location where rpmdb is supposed to live, it's probably ok.
> AFAIK its rw when you change the image and ro when you use it.
> > 
> > > That said, the dnfdb is only *important* to move because it's required
> > > for modularity stuff to work properly. Otherwise, it's not a huge deal
> > > one way or another.
> > Please don't introduce dnfdb as a term, AFAIK it doesn't exist :P
> > 
> > If there's anything in history db that is required for modularity to be
> > functional on a system, it should be moved to the dnf system state (in
> > dnf 5), history db should just be a log and nothing else (at most we
> > could use it to reconstruct the system state if it gets damaged while
> > the history db remains intact).
> It's mainly comps that lives nowhere else than in history db - and this 
> must be fixed by introducing "@System" (== installed) data for comps.
> Modularity mainly relies on data in /etc, but it is similar problem to 
> comps above.
> 

Even more importantly, history DB now contains the list of
userinstalled packages (which also lives nowhere else).

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Chris Murphy]

{
Apologies for not sending to rpm-ecosystem@ first off [1], then b.
since I wasn't subscribed to rpm-ecosystem@ my reply [2] was dropped,
messing up the threading.
[1] http://lists.rpm.org/pipermail/rpm-maint/2021-December/018662.html
[2] http://lists.rpm.org/pipermail/rpm-maint/2021-December/018687.html
}

On Fri, Dec 10, 2021 at 7:39 AM Lukáš Hrázký  wrote:
>
> On Fri, 2021-12-10 at 06:16 -0500, Neal Gompa wrote:
> > (Adding the correct Daniel Mach email address, he moved from Red Hat
> > to SUSE last month...)
> >
> > On Thu, Dec 9, 2021 at 10:11 AM Chris Murphy  
> > wrote:
> > >
> > > I'm pretty sure I'm generally opposed to the idea of logs being rolled
> > > back (in a snapshot and rollback regime). Logs should always carry
> > > forward. Ideally, indicate relevant things like rollbacks in the log,
> > > because rolling back a log is a form of data loss. The history
> > > database sounds more like a log than not. So I'm getting squeamish
> > > about locating that somewhere else. Whereas system state information,
> > > if it's primarily describing /usr, sounds like it needs to get
> > > somewhere in /usr.
> > >
> >
> > The history database is basically a transaction database like the
> > rpmdb is. It maps rpmdb transactions to dnf actions.
>
> I'm not sure this is true at all. I haven't examined the rpmdb ever,
> but AFAIK it contains the list of currently installed rpms (and their
> data), but not the history as they came and went (i.e. a log).
>
> OTOH dnf history database is exactly that. The current system state is
> in dnf 4 also computed as the aggregation of the transactions in the
> history database, which is IMO very wrong and we're fixing it in dnf 5.

What's the timeframe for dnf 5 as it relates to the Fedora release cycle?

>
> > So if you're
> > rolling back the rpm-managed content, it still makes sense to roll
> > back the history database, since that's the state of the system. It's
> > not terribly useful if the matching rpmdb transactions don't exist in
> > the rpmdb.
>
> Conceptually I agree with Chris on this: We shouldn't roll back the log
> of transactions if we'd be rolling back the system itself. That way you
> in effect erase that the system was running in a different state for an
> amount of time, imagine an auditing situation.

Yeah, the auditing case is important to consider. The FHS can be
aggravating in the context of snapshot and rollback regimes, in
particular deciding whether a /usr /var and /etc are tied to each
other for all time or if they can be mix and match. And that brings up
how to name the snapshots (I've varied between simple version control
by tacking on a .1 .2 .3 counter; to using ISO 8601 datetime stamps)
and whether and how to do system assembly at startup [3].

[3] need for a discoverable subvolumes specification (thread crosses two months)
https://lists.freedesktop.org/archives/systemd-devel/2021-November/047059.html
https://lists.freedesktop.org/archives/systemd-devel/2021-December/047176.html

For my non-emphemeral systems:

* /usr /var /etc are directories contained in btrfs subvolume
"root-x86-64:fedora.35"
* read-only snapshots have an ISO 8601 datetime format tacked onto the end

* /var/log is a directory contained in subvolume "varlog-x86-64:fedora.35"
* /var/lib/libvirt/images is a directory contained in subvolume
"varlibvirtimages-x86-64:fedora.35"
* no snapshots are ever made of these; by virtue of themselves being
subvolumes, any snapshot of "root-x86-64:fedora.35" doesn't contain
/var/log or /var/lib/libvirt/images so just like /home, they only ever
carry forward in time.

Therefore in my case, snapshot and rollback doesn't affect any logs,
including the systemd journal. They only ever carry forward in time.
Same for VM images and for ~/ since that lives in /home and is a
separate subvolume too by default in Fedora. But both RPM database and
DNF history do get rolled back in this case. Which works out OK
because usr/ var/ etc/ are all in one root. A rollback means DNF
history still accurately describes that root's history and state.

As soon as there's mix and match any of usr/ var/ and etc/ it gets trickier.

> There are also issues with sqlite creating some lock files even if just
> reading, which wouldn't be possible on a RO FS (IIRC how it was
> explained to me), so there are other reasons why the history db will
> likely not move to /usr.

Detect it and use overlayfs on a volatile /run directory for this use
case? I'm not sure the use case really calls for a persistent file
that survives a crash for reading read-only files?

> If there's anything in history db that is required for modularity to be
> functional on a system, it should be moved to the dnf system state (in
> dnf 5), history db should just be a log and nothing else (at most we
> could use it to reconstruct the system state if it gets damaged while
> the history db remains intact).


I wonder if the dnf history ought to go in /var/log and be exempt like
the systemd journal? 

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Lukáš Hrázký]

On Fri, 2021-12-10 at 06:16 -0500, Neal Gompa wrote:
> (Adding the correct Daniel Mach email address, he moved from Red Hat
> to SUSE last month...)
> 
> On Thu, Dec 9, 2021 at 10:11 AM Chris Murphy  wrote:
> > 
> > I'm pretty sure I'm generally opposed to the idea of logs being rolled
> > back (in a snapshot and rollback regime). Logs should always carry
> > forward. Ideally, indicate relevant things like rollbacks in the log,
> > because rolling back a log is a form of data loss. The history
> > database sounds more like a log than not. So I'm getting squeamish
> > about locating that somewhere else. Whereas system state information,
> > if it's primarily describing /usr, sounds like it needs to get
> > somewhere in /usr.
> > 
> 
> The history database is basically a transaction database like the
> rpmdb is. It maps rpmdb transactions to dnf actions. 

I'm not sure this is true at all. I haven't examined the rpmdb ever,
but AFAIK it contains the list of currently installed rpms (and their
data), but not the history as they came and went (i.e. a log).

OTOH dnf history database is exactly that. The current system state is
in dnf 4 also computed as the aggregation of the transactions in the
history database, which is IMO very wrong and we're fixing it in dnf 5.

> So if you're
> rolling back the rpm-managed content, it still makes sense to roll
> back the history database, since that's the state of the system. It's
> not terribly useful if the matching rpmdb transactions don't exist in
> the rpmdb.

Conceptually I agree with Chris on this: We shouldn't roll back the log
of transactions if we'd be rolling back the system itself. That way you
in effect erase that the system was running in a different state for an
amount of time, imagine an auditing situation.

There are also issues with sqlite creating some lock files even if just
reading, which wouldn't be possible on a RO FS (IIRC how it was
explained to me), so there are other reasons why the history db will
likely not move to /usr.

> That said, the dnfdb is only *important* to move because it's required
> for modularity stuff to work properly. Otherwise, it's not a huge deal
> one way or another.

Please don't introduce dnfdb as a term, AFAIK it doesn't exist :P

If there's anything in history db that is required for modularity to be
functional on a system, it should be moved to the dnf system state (in
dnf 5), history db should just be a log and nothing else (at most we
could use it to reconstruct the system state if it gets damaged while
the history db remains intact).

Cheers,
Lukas

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Neal Gompa]

(Adding the correct Daniel Mach email address, he moved from Red Hat
to SUSE last month...)

On Thu, Dec 9, 2021 at 10:11 AM Chris Murphy  wrote:
>
> On Thu, Dec 9, 2021 at 5:01 AM Jaroslav Mracek  wrote:
> >
> > Hello Chris,
> >
> > On Tue, Dec 7, 2021 at 5:01 AM Chris Murphy  wrote:
> >>
> >> Hi RPM and DNF folks,
> >>
> >> I have a draft change proposal for review and comment, i.e. it's not yet 
> >> set to be published to Fedora devel@. It's a bit thin, but I expect to 
> >> fill in more detail following discussion in this thread.
> >>
> >> https://fedoraproject.org/wiki/Changes/RelocateRPMDNFToUsr
> >
> >
> > The change is not so simple. It is not only the movement of files from one 
> > location to another one. We store more types of data in that location - 
> > history database (sqlite), module failsafe data (yamls). In future we will 
> > store system state data (toml). Data is not only modified after RPM 
> > transactions but also by module and mark commands. What I want to say is 
> > that the change will be painful but in the proposal there are limited 
> > benefits.
>
> I'm pretty sure I'm generally opposed to the idea of logs being rolled
> back (in a snapshot and rollback regime). Logs should always carry
> forward. Ideally, indicate relevant things like rollbacks in the log,
> because rolling back a log is a form of data loss. The history
> database sounds more like a log than not. So I'm getting squeamish
> about locating that somewhere else. Whereas system state information,
> if it's primarily describing /usr, sounds like it needs to get
> somewhere in /usr.
>

The history database is basically a transaction database like the
rpmdb is. It maps rpmdb transactions to dnf actions. So if you're
rolling back the rpm-managed content, it still makes sense to roll
back the history database, since that's the state of the system. It's
not terribly useful if the matching rpmdb transactions don't exist in
the rpmdb.

That said, the dnfdb is only *important* to move because it's required
for modularity stuff to work properly. Otherwise, it's not a huge deal
one way or another.

> Similar to openSUSE, design efforts around a default snapshot and
> rollback regime in Fedora are complicated when anything in /var
> depends on the state of /usr, and vice versa. As independent and
> interchangeable as they can be, I think the better.
>
> > There is also a question in which location DNF can move data. proposed 
> > `/usr/lib/sysimage/dnf` is maybe not the best one.
>
> What are some alternative suggestions?
>
>
> >> Fedora 36 seems like a good time to do this. What do you think?
> >
> >
> > I don't think it is a good time to perform such a change from a DNF 
> > perspective. We have a plan to introduce a major update to Fedora 38, 
> > therefore it is a better time frame for such a change.
>
> Is it better, worse, or indifferent if the RPM database location were
> changed in Fedora prior to any DNF change?
>

It won't matter. That's how it happened in openSUSE anyway. The rpmdb
path change happened several years earlier, and I changed the
dnfdb path in openSUSE at the top of the year as part of making the
DNF stack work properly on MicroOS.

https://code.opensuse.org/package/libdnf/c/37cdf99c506a7a2189ba26527d43665437c1db86

https://code.opensuse.org/package/dnf/c/24888ad21af0081e7432468d1aadfafd46e23526






--
真実はいつも一つ！/ Always, there's only one truth!
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr

[Neal Gompa]

On Tue, Dec 7, 2021 at 9:05 AM Panu Matilainen  wrote:
>
> On 12/7/21 06:00, Chris Murphy wrote:
> > Hi RPM and DNF folks,
> >
> > I have a draft change proposal for review and comment, i.e. it's not yet
> > set to be published to Fedora devel@. It's a bit thin, but I expect to
> > fill in more detail following discussion in this thread.
> >
> > https://fedoraproject.org/wiki/Changes/RelocateRPMDNFToUsr
> > 
> >
> >
> > The prior discussions for RPM have happened here:
> > http://lists.rpm.org/pipermail/rpm-maint/2017-October/006681.html
> > 
> > http://lists.rpm.org/pipermail/rpm-maint/2017-October/006723.html
> > 
> >
> > Fedora 36 seems like a good time to do this. What do you think?
>
> rpm-maint is about upstream rpm changes, and for an upstream change, the
> earliest something this drastic could go in would be 4.18 next year,
> presumably aligning with Fedora 37.
>

The more appropriate list would have been rpm-ecosystem@, but many of
the same people are on this list anyway...

> > Is it a given that the relocation should (or must) happen for upgrades?
> > What concerns do you have about relocating both databases during
> > offline-upgrade and ensuring its crash safe? When should the relocation
> > happen; as first or last order of business, or other?
>
> It's not clear to me what exactly you're planning here. Are you talking
> about just following what openSUSE did or something more elaborate?
>
> Technically, pointing rpm to a different database is a matter of
> changing exactly one macro in the configuration. Upgrade business
> generally needs to be left to the distro as rpm will have no way of
> knowing when it's appropriate to migrate and when it's an old chroot
> you're only wanting to peek at. Querying those old chroots and other
> images is something people do quite a lot, and chasing the correct path
> without creating new ones on the way certainly requires more than just a
> macro change.

I think the idea is to pull off the same migration openSUSE did.


-- 
真実はいつも一つ！/ Always, there's only one truth!
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem