Re: Failed RPM database migrations
On 29-10-2022 17:37, Neal Gompa wrote: On Sat, Oct 29, 2022 at 10:49 AM Orion Poplawski wrote: On 10/28/22 22:53, Todd Zullinger wrote: It seems that it's not so much a failed migration as a migration which is never attempted. :/ Likely in certain situations. What I don't understand is why all the trickery around enabling the service - why not just enable it unconditionally? It already has a condition: I was being careful because I wasn't initially sure what approach I'd use. It changed a couple of times. It can probably change now to be unconditional activation during install/upgrade of rpm, though. Now may be a good time to get this fixed. Once F37 is released and again once F35 goes EOL a large number of users will upgrade. -- Sandro ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
On Sat, Oct 29, 2022 at 10:49 AM Orion Poplawski wrote: > > On 10/28/22 22:53, Todd Zullinger wrote: > > Hi, > > > > Richard W.M. Jones wrote: > >> https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2 > >> https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr > >> https://bugzilla.redhat.com/2042099 > >> > >> The RPM database is supposed to move from /var/lib/rpm to > >> /usr/lib/sysimage/rpm. This was supposed to happen automatically when > >> you upgraded the rpm package from an earlier version to 4.17.0-10.fc36 > >> (Feb-Mar 2022). > >> > >> On several machines it is reported that the migration was only half- > >> completed. The symptoms are that the RPM database is still in > >> /var/lib/rpm (/usr contains symlinks to it). See typical output from > >> failed & successful migrations at the end of the email. > >> > >> So _if_ you have rpm >= 4.17.0-10.fc36 installed: > >> > >> - Do you see the symptom of a failed migration? Or does it appear > >> to be successful? (Or neither case??) > >> > >> - Did you: > >> * Install F37 or Rawhide from scratch? > >> * Upgrade using ordinary dnf update or similar? > >> * Upgrade using dnf system-upgrade? > >> * Some other install/upgrade method? > > > > I upgraded two systems from f35 to f36 in the past week via > > dnf system-upgrade. Neither of them completed the rpm > > migration. > > > > I _think_ the issue is due to the version-release in the > > %triggerun which should enable the rpmdb-migrate service. > > That is: > > > > %triggerun -- rpm < 4.17.0-7 > > # Handle rpmdb migrate service on erasure of old to avoid ordering > > issues > > if [ -x /usr/bin/systemctl ]; then > > systemctl --no-reload preset rpmdb-migrate ||: > > fi > > > > That was accurate when it was added in 0b9f813 (Migrate > > rpmdb to /usr/lib/sysimage/rpm (#2042099), 2022-01-26). > > > > However, when rpm was updated in f35 in e9927df (Rebase to > > rpm 4.17.1 (http://rpm.org/wiki/Releases/4.17.1), > > 2022-07-01), which never had the migration code, it > > prevented any up-to-date f35 system from triggering the > > migration. > > > > Anyone upgrading from f35 after rpm-4.17.1 was pushed to > > f35 won't have the rpmdb-migrate service enabled and the > > database will remain in /var/lib/rpm (with .migratedb) and > > symlinks in /usr/lib/sysimage/rpm. > > > > It seems that it's not so much a failed migration as a > > migration which is never attempted. :/ > > Likely in certain situations. > > What I don't understand is why all the trickery around enabling the > service - why not just enable it unconditionally? It already has a > condition: > I was being careful because I wasn't initially sure what approach I'd use. It changed a couple of times. It can probably change now to be unconditional activation during install/upgrade of rpm, though. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
On 10/28/22 22:53, Todd Zullinger wrote: Hi, Richard W.M. Jones wrote: https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2 https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr https://bugzilla.redhat.com/2042099 The RPM database is supposed to move from /var/lib/rpm to /usr/lib/sysimage/rpm. This was supposed to happen automatically when you upgraded the rpm package from an earlier version to 4.17.0-10.fc36 (Feb-Mar 2022). On several machines it is reported that the migration was only half- completed. The symptoms are that the RPM database is still in /var/lib/rpm (/usr contains symlinks to it). See typical output from failed & successful migrations at the end of the email. So _if_ you have rpm >= 4.17.0-10.fc36 installed: - Do you see the symptom of a failed migration? Or does it appear to be successful? (Or neither case??) - Did you: * Install F37 or Rawhide from scratch? * Upgrade using ordinary dnf update or similar? * Upgrade using dnf system-upgrade? * Some other install/upgrade method? I upgraded two systems from f35 to f36 in the past week via dnf system-upgrade. Neither of them completed the rpm migration. I _think_ the issue is due to the version-release in the %triggerun which should enable the rpmdb-migrate service. That is: %triggerun -- rpm < 4.17.0-7 # Handle rpmdb migrate service on erasure of old to avoid ordering issues if [ -x /usr/bin/systemctl ]; then systemctl --no-reload preset rpmdb-migrate ||: fi That was accurate when it was added in 0b9f813 (Migrate rpmdb to /usr/lib/sysimage/rpm (#2042099), 2022-01-26). However, when rpm was updated in f35 in e9927df (Rebase to rpm 4.17.1 (http://rpm.org/wiki/Releases/4.17.1), 2022-07-01), which never had the migration code, it prevented any up-to-date f35 system from triggering the migration. Anyone upgrading from f35 after rpm-4.17.1 was pushed to f35 won't have the rpmdb-migrate service enabled and the database will remain in /var/lib/rpm (with .migratedb) and symlinks in /usr/lib/sysimage/rpm. It seems that it's not so much a failed migration as a migration which is never attempted. :/ Likely in certain situations. What I don't understand is why all the trickery around enabling the service - why not just enable it unconditionally? It already has a condition: ConditionPathExists=/var/lib/rpm/.migratedb and so won't run multiple times. In my case though it seems to be a product of a bad conditional. I'm running a continually updating rawhide VM and: Jan 06 10:16:09 vmrawhide-rufous.cora.nwra.com systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of a failed condition check (ConditionPathExists=/var/lib/rpm/.rebuilddb). ... Feb 06 19:35:14 vmrawhide-rufous.cora.nwra.com systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of a failed condition check (ConditionPathExists=/var/lib/rpm/.rebuilddb). Feb 08 19:56:31 vmrawhide-rufous.cora.nwra.com systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of a failed condition check (ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb). ... Oct 16 09:47:01 vmrawhide-rufous.cora.nwra.com systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of a failed condition check (ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb). Oct 27 17:32:57 vmrawhide-rufous.cora.nwra.com systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of an unmet condition check (ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb). # systemctl status rpmdb-rebuild.service ○ rpmdb-rebuild.service - RPM database rebuild Loaded: loaded (/usr/lib/systemd/system/rpmdb-rebuild.service; enabled; preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-10-27 17:32:57 MDT; 1 day 15h ago Oct 27 17:32:57 vmrawhide-rufous.cora.nwra.com systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of an unmet condition check (ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb). # ls -la /usr/lib/sysimage/rpm/ total 8 drwxr-xr-x. 2 root root 4096 Oct 5 05:05 . drwxr-xr-x. 3 root root 4096 Aug 9 07:27 .. lrwxrwxrwx. 1 root root 34 Oct 5 20:32 .migratedb -> ../../../../var/lib/rpm/.migratedb lrwxrwxrwx. 1 root root 36 Oct 5 20:32 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite lrwxrwxrwx. 1 root root 40 Oct 5 20:32 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm lrwxrwxrwx. 1 root root 40 Oct 5 20:32 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal lrwxrwxrwx. 1 root root 33 Oct 5 20:32 .rpm.lock -> ../../../../var/lib/rpm/.rpm.lock # ls -la /var/lib/rpm total 444064 drwxr-xr-x. 2 root root 4096 Feb 6 2022 . drwxr-xr-x. 68 root root 4096 Aug 9 07:27 .. -rw-r--r--. 1 root root 0 Oct 5 20:35 .migratedb -rw-r--r--. 1 root root 454676480 Oct 28 20:27 rpmdb.sqlite -rw-r--r--.
Re: Failed RPM database migrations
Hi, Richard W.M. Jones wrote: > https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2 > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr > https://bugzilla.redhat.com/2042099 > > The RPM database is supposed to move from /var/lib/rpm to > /usr/lib/sysimage/rpm. This was supposed to happen automatically when > you upgraded the rpm package from an earlier version to 4.17.0-10.fc36 > (Feb-Mar 2022). > > On several machines it is reported that the migration was only half- > completed. The symptoms are that the RPM database is still in > /var/lib/rpm (/usr contains symlinks to it). See typical output from > failed & successful migrations at the end of the email. > > So _if_ you have rpm >= 4.17.0-10.fc36 installed: > > - Do you see the symptom of a failed migration? Or does it appear >to be successful? (Or neither case??) > > - Did you: > * Install F37 or Rawhide from scratch? > * Upgrade using ordinary dnf update or similar? > * Upgrade using dnf system-upgrade? > * Some other install/upgrade method? I upgraded two systems from f35 to f36 in the past week via dnf system-upgrade. Neither of them completed the rpm migration. I _think_ the issue is due to the version-release in the %triggerun which should enable the rpmdb-migrate service. That is: %triggerun -- rpm < 4.17.0-7 # Handle rpmdb migrate service on erasure of old to avoid ordering issues if [ -x /usr/bin/systemctl ]; then systemctl --no-reload preset rpmdb-migrate ||: fi That was accurate when it was added in 0b9f813 (Migrate rpmdb to /usr/lib/sysimage/rpm (#2042099), 2022-01-26). However, when rpm was updated in f35 in e9927df (Rebase to rpm 4.17.1 (http://rpm.org/wiki/Releases/4.17.1), 2022-07-01), which never had the migration code, it prevented any up-to-date f35 system from triggering the migration. Anyone upgrading from f35 after rpm-4.17.1 was pushed to f35 won't have the rpmdb-migrate service enabled and the database will remain in /var/lib/rpm (with .migratedb) and symlinks in /usr/lib/sysimage/rpm. It seems that it's not so much a failed migration as a migration which is never attempted. :/ -- Todd signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
On Fri, 28 Oct 2022 14:59:54 - "Sergey Mende" wrote: > thank you, I got the recipe from the previous conversations. I just > asked if I could help with logs analysis or somehow else before I fix > the issue. Sorry for the misunderstanding. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
On Fri, Oct 28, 2022 at 13:49:21 +0100, Richard W.M. Jones wrote: > On Fri, Oct 28, 2022 at 01:45:32PM +0100, Tom Hughes wrote: > > On 28/10/2022 13:31, Richard W.M. Jones wrote: > > >On Fri, Oct 28, 2022 at 12:29:30PM +0100, Tom Hughes wrote: > > >>The reason it hadn't completed is that rpmdb-migrate.service > > >>was enabled on that machine. > > >[was not, I guess?] > > > > Yes ;-) > > > > >>Enabling (and starting) that service made it complete. > > > > > >Interesting. The current state of that service is: > > > > > >○ rpmdb-migrate.service - RPM database migration to /usr > > > Loaded: loaded (/usr/lib/systemd/system/rpmdb-migrate.service; > disabled; vendor preset: enabled) > > > Active: inactive (dead) > > > > > >There are no log entries for this service, but my logs only go back to > > >around April which is probably too late to see anything. > > > > > >After starting the service: > > > > > >Oct 28 13:29:33 systemd[1]: Starting rpmdb-migrate.service - RPM > database migration to /usr... > > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed > '/var/lib/rpm/.migratedb' > > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed > '/var/lib/rpm/rpmdb.sqlite-shm' > > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed > '/var/lib/rpm/rpmdb.sqlite' > > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed > '/var/lib/rpm/rpmdb.sqlite-wal' > > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.rpm.lock' > > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed directory '/var/lib/rpm' > > >Oct 28 13:29:35 rpmdb_migrate[1722468]: '/var/lib/rpm' -> > '../../usr/lib/sysimage/rpm' > > >Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Deactivated > successfully. > > >Oct 28 13:29:35 systemd[1]: Finished rpmdb-migrate.service - RPM > database migration to /usr. > > >Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Consumed 2.433s CPU > time. > > > > > >... and the migration has been successful. So at least we know how to > > >fix this. Although it's quite curious that the service was installed > > >and supposed to run but didn't. > > > > The idea is that the service is enabled (not sure off hand what is > > supposed to do that) but not started, so that it runs when you reboot > > and completes the migration as part of the boot. > > > > When it runs it removes /var/lib/rpm/.migratedb which was created > > by the rpm scripts and hence prevents the service running on future > > boots as it's no longer needed. > > It's hard to tell what happened without logs going back all the way, > but the machine was rebooted 65 days ago and several times before > that. Within the available logs there is no mention of the service so > it may already have tried to run on a boot prior to the logs and > failed for some reason. > > We do have reports of this happening from 3 independent people now so > it's not a one-off. I wonder if folks rebooted or shut down the system after upgrading, and after /var/lib/rpm/.migratedb is removed but before the migration actually finished. The example above shows the file being removed before the cleanup completed. That run only took two seconds, so it seems to be an unlikely cause, but is that typically how long the migration takes? -- Steve ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
Rich, stan, thank you, I got the recipe from the previous conversations. I just asked if I could help with logs analysis or somehow else before I fix the issue. Regards, Sergey ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
On Fri, Oct 28, 2022 at 07:46:03AM -0700, stan via devel wrote: > On Fri, 28 Oct 2022 13:08:19 - > "Sergey Mende" wrote: > > > I upgraded from 35 to 36 in the beginning of Sept this year. The > > migration failed. May I help somehow? > > I used the advice from Tom Hughes in his earlier message to complete my > failed migration. > > # systemctl enable rpmdb-migrate > # systemctl start rpmdb-migrate > > When that completed, the migration was successfully done. > > I am running F37, started as F35 rawhide, updated using dnf, > usually daily, with reboots also usually daily. I am currently running > rpm-4.18.0-1.fc37.x86_64. Correct -- while we're trying to find out how widespread the problem is, the way to fix it on individual machines is as outlined above. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
On Fri, 28 Oct 2022 13:08:19 - "Sergey Mende" wrote: > I upgraded from 35 to 36 in the beginning of Sept this year. The > migration failed. May I help somehow? I used the advice from Tom Hughes in his earlier message to complete my failed migration. # systemctl enable rpmdb-migrate # systemctl start rpmdb-migrate When that completed, the migration was successfully done. I am running F37, started as F35 rawhide, updated using dnf, usually daily, with reboots also usually daily. I am currently running rpm-4.18.0-1.fc37.x86_64. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
Hi, I upgraded from 35 to 36 in the beginning of Sept this year. The migration failed. May I help somehow? Regards, Sergey. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
On Fri, Oct 28, 2022 at 01:45:32PM +0100, Tom Hughes wrote: > On 28/10/2022 13:31, Richard W.M. Jones wrote: > >On Fri, Oct 28, 2022 at 12:29:30PM +0100, Tom Hughes wrote: > >>The reason it hadn't completed is that rpmdb-migrate.service > >>was enabled on that machine. > >[was not, I guess?] > > Yes ;-) > > >>Enabling (and starting) that service made it complete. > > > >Interesting. The current state of that service is: > > > >○ rpmdb-migrate.service - RPM database migration to /usr > > Loaded: loaded (/usr/lib/systemd/system/rpmdb-migrate.service; > > disabled; vendor preset: enabled) > > Active: inactive (dead) > > > >There are no log entries for this service, but my logs only go back to > >around April which is probably too late to see anything. > > > >After starting the service: > > > >Oct 28 13:29:33 systemd[1]: Starting rpmdb-migrate.service - RPM database > >migration to /usr... > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.migratedb' > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed > >'/var/lib/rpm/rpmdb.sqlite-shm' > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite' > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed > >'/var/lib/rpm/rpmdb.sqlite-wal' > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.rpm.lock' > >Oct 28 13:29:35 rpmdb_migrate[1722092]: removed directory '/var/lib/rpm' > >Oct 28 13:29:35 rpmdb_migrate[1722468]: '/var/lib/rpm' -> > >'../../usr/lib/sysimage/rpm' > >Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Deactivated successfully. > >Oct 28 13:29:35 systemd[1]: Finished rpmdb-migrate.service - RPM database > >migration to /usr. > >Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Consumed 2.433s CPU time. > > > >... and the migration has been successful. So at least we know how to > >fix this. Although it's quite curious that the service was installed > >and supposed to run but didn't. > > The idea is that the service is enabled (not sure off hand what is > supposed to do that) but not started, so that it runs when you reboot > and completes the migration as part of the boot. > > When it runs it removes /var/lib/rpm/.migratedb which was created > by the rpm scripts and hence prevents the service running on future > boots as it's no longer needed. It's hard to tell what happened without logs going back all the way, but the machine was rebooted 65 days ago and several times before that. Within the available logs there is no mention of the service so it may already have tried to run on a boot prior to the logs and failed for some reason. We do have reports of this happening from 3 independent people now so it's not a one-off. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
On 28/10/2022 13:31, Richard W.M. Jones wrote: On Fri, Oct 28, 2022 at 12:29:30PM +0100, Tom Hughes wrote: The reason it hadn't completed is that rpmdb-migrate.service was enabled on that machine. [was not, I guess?] Yes ;-) Enabling (and starting) that service made it complete. Interesting. The current state of that service is: ○ rpmdb-migrate.service - RPM database migration to /usr Loaded: loaded (/usr/lib/systemd/system/rpmdb-migrate.service; disabled; vendor preset: enabled) Active: inactive (dead) There are no log entries for this service, but my logs only go back to around April which is probably too late to see anything. After starting the service: Oct 28 13:29:33 systemd[1]: Starting rpmdb-migrate.service - RPM database migration to /usr... Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.migratedb' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite-shm' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite-wal' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.rpm.lock' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed directory '/var/lib/rpm' Oct 28 13:29:35 rpmdb_migrate[1722468]: '/var/lib/rpm' -> '../../usr/lib/sysimage/rpm' Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Deactivated successfully. Oct 28 13:29:35 systemd[1]: Finished rpmdb-migrate.service - RPM database migration to /usr. Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Consumed 2.433s CPU time. ... and the migration has been successful. So at least we know how to fix this. Although it's quite curious that the service was installed and supposed to run but didn't. The idea is that the service is enabled (not sure off hand what is supposed to do that) but not started, so that it runs when you reboot and completes the migration as part of the boot. When it runs it removes /var/lib/rpm/.migratedb which was created by the rpm scripts and hence prevents the service running on future boots as it's no longer needed. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
On Fri, Oct 28, 2022 at 12:29:30PM +0100, Tom Hughes wrote: > The reason it hadn't completed is that rpmdb-migrate.service > was enabled on that machine. [was not, I guess?] > Enabling (and starting) that service made it complete. Interesting. The current state of that service is: ○ rpmdb-migrate.service - RPM database migration to /usr Loaded: loaded (/usr/lib/systemd/system/rpmdb-migrate.service; disabled; vendor preset: enabled) Active: inactive (dead) There are no log entries for this service, but my logs only go back to around April which is probably too late to see anything. After starting the service: Oct 28 13:29:33 systemd[1]: Starting rpmdb-migrate.service - RPM database migration to /usr... Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.migratedb' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite-shm' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite-wal' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.rpm.lock' Oct 28 13:29:35 rpmdb_migrate[1722092]: removed directory '/var/lib/rpm' Oct 28 13:29:35 rpmdb_migrate[1722468]: '/var/lib/rpm' -> '../../usr/lib/sysimage/rpm' Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Deactivated successfully. Oct 28 13:29:35 systemd[1]: Finished rpmdb-migrate.service - RPM database migration to /usr. Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Consumed 2.433s CPU time. ... and the migration has been successful. So at least we know how to fix this. Although it's quite curious that the service was installed and supposed to run but didn't. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
I've just checked 4 systems (2x Workstation, MATE and Cloud), and the migration was successful on all of them. They were all upgraded to the latest versions in testing at the time with dnf system-upgrade. Regular package updates were performed weekly at the latest. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
The reason it hadn't completed is that rpmdb-migrate.service was enabled on that machine. Enabling (and starting) that service made it complete. Tom On 28/10/2022 12:24, Tom Hughes via devel wrote: I have one machine that has failed. It was an upgrade from 35 to 36 done using dnf distro-sync but I have plenty of others done the same way that worked. One difference is that it's a machine that wasn't upgraded until August while other ones were done back in May as a result of which the upgrade was: from rpm-4.17.1-3.fc35.x86_64 to rpm-4.17.1-3.fc36.x86_64 while other machines were more like: from rpm-4.17.0-4.fc35.x86_64 to rpm-4.17.0-10.fc36.x86_64 Tom On 28/10/2022 11:49, Richard W.M. Jones wrote: https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2 https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr https://bugzilla.redhat.com/2042099 The RPM database is supposed to move from /var/lib/rpm to /usr/lib/sysimage/rpm. This was supposed to happen automatically when you upgraded the rpm package from an earlier version to 4.17.0-10.fc36 (Feb-Mar 2022). On several machines it is reported that the migration was only half- completed. The symptoms are that the RPM database is still in /var/lib/rpm (/usr contains symlinks to it). See typical output from failed & successful migrations at the end of the email. So _if_ you have rpm >= 4.17.0-10.fc36 installed: - Do you see the symptom of a failed migration? Or does it appear to be successful? (Or neither case??) - Did you: * Install F37 or Rawhide from scratch? * Upgrade using ordinary dnf update or similar? * Upgrade using dnf system-upgrade? * Some other install/upgrade method? Rich. ** After a failed migration: ** $ ls -la /var/lib/rpm total 339004 drwxr-xr-x. 2 root root 4096 Feb 16 2022 . drwxr-xr-x. 76 root root 4096 Aug 23 13:28 .. -rw-r--r--. 1 root root 0 Oct 18 14:28 .migratedb -rw-r--r--. 1 root root 347095040 Oct 26 16:50 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Oct 27 15:28 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Oct 26 16:50 rpmdb.sqlite-wal -rw-r--r--. 1 root root 0 Feb 16 2022 .rpm.lock $ ls -la /usr/lib/sysimage/rpm total 8 drwxr-xr-x. 2 root root 4096 Oct 5 12:05 . drwxr-xr-x. 3 root root 4096 Feb 9 2022 .. lrwxrwxrwx. 1 root root 34 Oct 18 14:28 .migratedb -> ../../../../var/lib/rpm/.migratedb lrwxrwxrwx. 1 root root 36 Oct 18 14:28 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal lrwxrwxrwx. 1 root root 33 Oct 18 14:28 .rpm.lock -> ../../../../var/lib/rpm/.rpm.lock ** After a successful migration: ** $ ls -la /var/lib/rpm lrwxrwxrwx. 1 root root 26 Sep 5 13:35 /var/lib/rpm -> ../../usr/lib/sysimage/rpm $ ls -la /usr/lib/sysimage/rpm total 316324 drwxr-xr-x. 2 root root 91 Sep 17 23:03 . drwxr-xr-x. 3 root root 17 Aug 9 14:27 .. -rw-r--r--. 1 root root 323878912 Oct 27 16:12 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Oct 28 10:45 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Oct 27 16:12 rpmdb.sqlite-wal -rw-r--r--. 1 root root 0 Sep 5 13:53 .rpm.lock -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Failed RPM database migrations
I have one machine that has failed. It was an upgrade from 35 to 36 done using dnf distro-sync but I have plenty of others done the same way that worked. One difference is that it's a machine that wasn't upgraded until August while other ones were done back in May as a result of which the upgrade was: from rpm-4.17.1-3.fc35.x86_64 to rpm-4.17.1-3.fc36.x86_64 while other machines were more like: from rpm-4.17.0-4.fc35.x86_64 to rpm-4.17.0-10.fc36.x86_64 Tom On 28/10/2022 11:49, Richard W.M. Jones wrote: https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2 https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr https://bugzilla.redhat.com/2042099 The RPM database is supposed to move from /var/lib/rpm to /usr/lib/sysimage/rpm. This was supposed to happen automatically when you upgraded the rpm package from an earlier version to 4.17.0-10.fc36 (Feb-Mar 2022). On several machines it is reported that the migration was only half- completed. The symptoms are that the RPM database is still in /var/lib/rpm (/usr contains symlinks to it). See typical output from failed & successful migrations at the end of the email. So _if_ you have rpm >= 4.17.0-10.fc36 installed: - Do you see the symptom of a failed migration? Or does it appear to be successful? (Or neither case??) - Did you: * Install F37 or Rawhide from scratch? * Upgrade using ordinary dnf update or similar? * Upgrade using dnf system-upgrade? * Some other install/upgrade method? Rich. ** After a failed migration: ** $ ls -la /var/lib/rpm total 339004 drwxr-xr-x. 2 root root 4096 Feb 16 2022 . drwxr-xr-x. 76 root root 4096 Aug 23 13:28 .. -rw-r--r--. 1 root root 0 Oct 18 14:28 .migratedb -rw-r--r--. 1 root root 347095040 Oct 26 16:50 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Oct 27 15:28 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Oct 26 16:50 rpmdb.sqlite-wal -rw-r--r--. 1 root root 0 Feb 16 2022 .rpm.lock $ ls -la /usr/lib/sysimage/rpm total 8 drwxr-xr-x. 2 root root 4096 Oct 5 12:05 . drwxr-xr-x. 3 root root 4096 Feb 9 2022 .. lrwxrwxrwx. 1 root root 34 Oct 18 14:28 .migratedb -> ../../../../var/lib/rpm/.migratedb lrwxrwxrwx. 1 root root 36 Oct 18 14:28 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal lrwxrwxrwx. 1 root root 33 Oct 18 14:28 .rpm.lock -> ../../../../var/lib/rpm/.rpm.lock ** After a successful migration: ** $ ls -la /var/lib/rpm lrwxrwxrwx. 1 root root 26 Sep 5 13:35 /var/lib/rpm -> ../../usr/lib/sysimage/rpm $ ls -la /usr/lib/sysimage/rpm total 316324 drwxr-xr-x. 2 root root91 Sep 17 23:03 . drwxr-xr-x. 3 root root17 Aug 9 14:27 .. -rw-r--r--. 1 root root 323878912 Oct 27 16:12 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Oct 28 10:45 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Oct 27 16:12 rpmdb.sqlite-wal -rw-r--r--. 1 root root 0 Sep 5 13:53 .rpm.lock -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Failed RPM database migrations
https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2 https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr https://bugzilla.redhat.com/2042099 The RPM database is supposed to move from /var/lib/rpm to /usr/lib/sysimage/rpm. This was supposed to happen automatically when you upgraded the rpm package from an earlier version to 4.17.0-10.fc36 (Feb-Mar 2022). On several machines it is reported that the migration was only half- completed. The symptoms are that the RPM database is still in /var/lib/rpm (/usr contains symlinks to it). See typical output from failed & successful migrations at the end of the email. So _if_ you have rpm >= 4.17.0-10.fc36 installed: - Do you see the symptom of a failed migration? Or does it appear to be successful? (Or neither case??) - Did you: * Install F37 or Rawhide from scratch? * Upgrade using ordinary dnf update or similar? * Upgrade using dnf system-upgrade? * Some other install/upgrade method? Rich. ** After a failed migration: ** $ ls -la /var/lib/rpm total 339004 drwxr-xr-x. 2 root root 4096 Feb 16 2022 . drwxr-xr-x. 76 root root 4096 Aug 23 13:28 .. -rw-r--r--. 1 root root 0 Oct 18 14:28 .migratedb -rw-r--r--. 1 root root 347095040 Oct 26 16:50 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Oct 27 15:28 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Oct 26 16:50 rpmdb.sqlite-wal -rw-r--r--. 1 root root 0 Feb 16 2022 .rpm.lock $ ls -la /usr/lib/sysimage/rpm total 8 drwxr-xr-x. 2 root root 4096 Oct 5 12:05 . drwxr-xr-x. 3 root root 4096 Feb 9 2022 .. lrwxrwxrwx. 1 root root 34 Oct 18 14:28 .migratedb -> ../../../../var/lib/rpm/.migratedb lrwxrwxrwx. 1 root root 36 Oct 18 14:28 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal lrwxrwxrwx. 1 root root 33 Oct 18 14:28 .rpm.lock -> ../../../../var/lib/rpm/.rpm.lock ** After a successful migration: ** $ ls -la /var/lib/rpm lrwxrwxrwx. 1 root root 26 Sep 5 13:35 /var/lib/rpm -> ../../usr/lib/sysimage/rpm $ ls -la /usr/lib/sysimage/rpm total 316324 drwxr-xr-x. 2 root root91 Sep 17 23:03 . drwxr-xr-x. 3 root root17 Aug 9 14:27 .. -rw-r--r--. 1 root root 323878912 Oct 27 16:12 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Oct 28 10:45 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Oct 27 16:12 rpmdb.sqlite-wal -rw-r--r--. 1 root root 0 Sep 5 13:53 .rpm.lock -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
