Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr
{
Apologies for not sending to rpm-ecosystem@ first off [1], then b.
since I wasn't subscribed to rpm-ecosystem@ my reply [2] was dropped,
messing up the threading.
[1] http://lists.rpm.org/pipermail/rpm-maint/2021-December/018662.html
[2] http://lists.rpm.org/pipermail/rpm-maint/2021-December/018687.html
}
On Fri, Dec 10, 2021 at 7:39 AM Lukáš Hrázký wrote:
>
> On Fri, 2021-12-10 at 06:16 -0500, Neal Gompa wrote:
> > (Adding the correct Daniel Mach email address, he moved from Red Hat
> > to SUSE last month...)
> >
> > On Thu, Dec 9, 2021 at 10:11 AM Chris Murphy
> > wrote:
> > >
> > > I'm pretty sure I'm generally opposed to the idea of logs being rolled
> > > back (in a snapshot and rollback regime). Logs should always carry
> > > forward. Ideally, indicate relevant things like rollbacks in the log,
> > > because rolling back a log is a form of data loss. The history
> > > database sounds more like a log than not. So I'm getting squeamish
> > > about locating that somewhere else. Whereas system state information,
> > > if it's primarily describing /usr, sounds like it needs to get
> > > somewhere in /usr.
> > >
> >
> > The history database is basically a transaction database like the
> > rpmdb is. It maps rpmdb transactions to dnf actions.
>
> I'm not sure this is true at all. I haven't examined the rpmdb ever,
> but AFAIK it contains the list of currently installed rpms (and their
> data), but not the history as they came and went (i.e. a log).
>
> OTOH dnf history database is exactly that. The current system state is
> in dnf 4 also computed as the aggregation of the transactions in the
> history database, which is IMO very wrong and we're fixing it in dnf 5.
What's the timeframe for dnf 5 as it relates to the Fedora release cycle?
>
> > So if you're
> > rolling back the rpm-managed content, it still makes sense to roll
> > back the history database, since that's the state of the system. It's
> > not terribly useful if the matching rpmdb transactions don't exist in
> > the rpmdb.
>
> Conceptually I agree with Chris on this: We shouldn't roll back the log
> of transactions if we'd be rolling back the system itself. That way you
> in effect erase that the system was running in a different state for an
> amount of time, imagine an auditing situation.
Yeah, the auditing case is important to consider. The FHS can be
aggravating in the context of snapshot and rollback regimes, in
particular deciding whether a /usr /var and /etc are tied to each
other for all time or if they can be mix and match. And that brings up
how to name the snapshots (I've varied between simple version control
by tacking on a .1 .2 .3 counter; to using ISO 8601 datetime stamps)
and whether and how to do system assembly at startup [3].
[3] need for a discoverable subvolumes specification (thread crosses two months)
https://lists.freedesktop.org/archives/systemd-devel/2021-November/047059.html
https://lists.freedesktop.org/archives/systemd-devel/2021-December/047176.html
For my non-emphemeral systems:
* /usr /var /etc are directories contained in btrfs subvolume
"root-x86-64:fedora.35"
* read-only snapshots have an ISO 8601 datetime format tacked onto the end
* /var/log is a directory contained in subvolume "varlog-x86-64:fedora.35"
* /var/lib/libvirt/images is a directory contained in subvolume
"varlibvirtimages-x86-64:fedora.35"
* no snapshots are ever made of these; by virtue of themselves being
subvolumes, any snapshot of "root-x86-64:fedora.35" doesn't contain
/var/log or /var/lib/libvirt/images so just like /home, they only ever
carry forward in time.
Therefore in my case, snapshot and rollback doesn't affect any logs,
including the systemd journal. They only ever carry forward in time.
Same for VM images and for ~/ since that lives in /home and is a
separate subvolume too by default in Fedora. But both RPM database and
DNF history do get rolled back in this case. Which works out OK
because usr/ var/ etc/ are all in one root. A rollback means DNF
history still accurately describes that root's history and state.
As soon as there's mix and match any of usr/ var/ and etc/ it gets trickier.
> There are also issues with sqlite creating some lock files even if just
> reading, which wouldn't be possible on a RO FS (IIRC how it was
> explained to me), so there are other reasons why the history db will
> likely not move to /usr.
Detect it and use overlayfs on a volatile /run directory for this use
case? I'm not sure the use case really calls for a persistent file
that survives a crash for reading read-only files?
> If there's anything in history db that is required for modularity to be
> functional on a system, it should be moved to the dnf system state (in
> dnf 5), history db should just be a log and nothing else (at most we
> could use it to reconstruct the system state if it gets damaged while
> the history db remains intact).
I wonder if the dnf history ought to go in /var/log and be exempt like
the systemd journal?
Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr
On Fri, 2021-12-10 at 06:16 -0500, Neal Gompa wrote: > (Adding the correct Daniel Mach email address, he moved from Red Hat > to SUSE last month...) > > On Thu, Dec 9, 2021 at 10:11 AM Chris Murphy wrote: > > > > I'm pretty sure I'm generally opposed to the idea of logs being rolled > > back (in a snapshot and rollback regime). Logs should always carry > > forward. Ideally, indicate relevant things like rollbacks in the log, > > because rolling back a log is a form of data loss. The history > > database sounds more like a log than not. So I'm getting squeamish > > about locating that somewhere else. Whereas system state information, > > if it's primarily describing /usr, sounds like it needs to get > > somewhere in /usr. > > > > The history database is basically a transaction database like the > rpmdb is. It maps rpmdb transactions to dnf actions. I'm not sure this is true at all. I haven't examined the rpmdb ever, but AFAIK it contains the list of currently installed rpms (and their data), but not the history as they came and went (i.e. a log). OTOH dnf history database is exactly that. The current system state is in dnf 4 also computed as the aggregation of the transactions in the history database, which is IMO very wrong and we're fixing it in dnf 5. > So if you're > rolling back the rpm-managed content, it still makes sense to roll > back the history database, since that's the state of the system. It's > not terribly useful if the matching rpmdb transactions don't exist in > the rpmdb. Conceptually I agree with Chris on this: We shouldn't roll back the log of transactions if we'd be rolling back the system itself. That way you in effect erase that the system was running in a different state for an amount of time, imagine an auditing situation. There are also issues with sqlite creating some lock files even if just reading, which wouldn't be possible on a RO FS (IIRC how it was explained to me), so there are other reasons why the history db will likely not move to /usr. > That said, the dnfdb is only *important* to move because it's required > for modularity stuff to work properly. Otherwise, it's not a huge deal > one way or another. Please don't introduce dnfdb as a term, AFAIK it doesn't exist :P If there's anything in history db that is required for modularity to be functional on a system, it should be moved to the dnf system state (in dnf 5), history db should just be a log and nothing else (at most we could use it to reconstruct the system state if it gets damaged while the history db remains intact). Cheers, Lukas ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr
(Adding the correct Daniel Mach email address, he moved from Red Hat to SUSE last month...) On Thu, Dec 9, 2021 at 10:11 AM Chris Murphy wrote: > > On Thu, Dec 9, 2021 at 5:01 AM Jaroslav Mracek wrote: > > > > Hello Chris, > > > > On Tue, Dec 7, 2021 at 5:01 AM Chris Murphy wrote: > >> > >> Hi RPM and DNF folks, > >> > >> I have a draft change proposal for review and comment, i.e. it's not yet > >> set to be published to Fedora devel@. It's a bit thin, but I expect to > >> fill in more detail following discussion in this thread. > >> > >> https://fedoraproject.org/wiki/Changes/RelocateRPMDNFToUsr > > > > > > The change is not so simple. It is not only the movement of files from one > > location to another one. We store more types of data in that location - > > history database (sqlite), module failsafe data (yamls). In future we will > > store system state data (toml). Data is not only modified after RPM > > transactions but also by module and mark commands. What I want to say is > > that the change will be painful but in the proposal there are limited > > benefits. > > I'm pretty sure I'm generally opposed to the idea of logs being rolled > back (in a snapshot and rollback regime). Logs should always carry > forward. Ideally, indicate relevant things like rollbacks in the log, > because rolling back a log is a form of data loss. The history > database sounds more like a log than not. So I'm getting squeamish > about locating that somewhere else. Whereas system state information, > if it's primarily describing /usr, sounds like it needs to get > somewhere in /usr. > The history database is basically a transaction database like the rpmdb is. It maps rpmdb transactions to dnf actions. So if you're rolling back the rpm-managed content, it still makes sense to roll back the history database, since that's the state of the system. It's not terribly useful if the matching rpmdb transactions don't exist in the rpmdb. That said, the dnfdb is only *important* to move because it's required for modularity stuff to work properly. Otherwise, it's not a huge deal one way or another. > Similar to openSUSE, design efforts around a default snapshot and > rollback regime in Fedora are complicated when anything in /var > depends on the state of /usr, and vice versa. As independent and > interchangeable as they can be, I think the better. > > > There is also a question in which location DNF can move data. proposed > > `/usr/lib/sysimage/dnf` is maybe not the best one. > > What are some alternative suggestions? > > > >> Fedora 36 seems like a good time to do this. What do you think? > > > > > > I don't think it is a good time to perform such a change from a DNF > > perspective. We have a plan to introduce a major update to Fedora 38, > > therefore it is a better time frame for such a change. > > Is it better, worse, or indifferent if the RPM database location were > changed in Fedora prior to any DNF change? > It won't matter. That's how it happened in openSUSE anyway. The rpmdb path change happened several years earlier, and I changed the dnfdb path in openSUSE at the top of the year as part of making the DNF stack work properly on MicroOS. https://code.opensuse.org/package/libdnf/c/37cdf99c506a7a2189ba26527d43665437c1db86 https://code.opensuse.org/package/dnf/c/24888ad21af0081e7432468d1aadfafd46e23526 -- 真実はいつも一つ！/ Always, there's only one truth! ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
