Re: How to migrate database format during package update?
Dne 01. 02. 23 v 13:56 Milan Crha napsal(a): Hi, this is a query for an opinion and a best-practice experience for a case when a package needs to change its internal database format between versions, in an environment, which does not allow real That being said, any such change surely deserves release notes, with the commands what to do to export and then back import the wordlist. There should be filled a corresponding Change too, I guess. Yes. Likely both. If you will go the release notes only way then open the issue here https://pagure.io/fedora-docs/release-notes But I strongly encourage you to create Change proposal as it gives you a lot of advertisment and therefore awareness for free. Still, how does other packages migrate their data, or at least help the users with the migration? Is there any way? Put aside all your expectation. The reality will always surprise you no matter how big is your imagination. I worked in past on project where the DB was sometimes huge. And upgrades of the schema took 20+ hours even on beefy machines. The stories... The best practice is to document it. Announce in advance. And when package is upgraded flip some variable in config (/etc/sysconfig/$package or ~/.config/$package) and until user flip the variable back, do not start the service/application and if user attempts to start it, exit with message like: The DB schema of $package changed. Please migrate to new format using a tool `$migration-tool`. And then change variable FOO in /etc/sysconfig/$package For more information see $URL Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: How to migrate database format during package update?
On Wed, 2023-02-01 at 10:12 -0800, Kevin Fenzi wrote: > Is there any way to pull the functionaly into the process itself? > ie, the first time it's called, it converts the db? Hi, the idea is to not depend on the libdb at all, neither in the build time. I reworked the proposed change to conditionally (in the .spec file) build the migration tool using static libdb, thus no pull in of the libdb package itself. Something similar is used in the cyrus-sasl Alexander pointed me to. It's much cleaner solution and it also lefts everything under the user's control, which is for good. I do not think there can be done the conversion on-the-fly, the bogofilter is build with one or other DB engine, not with multiple. Having wrapper scripts to convert the format is something I'd like to avoid. Thanks and bye, Milan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: How to migrate database format during package update?
On Wed, Feb 01, 2023 at 01:56:20PM +0100, Milan Crha wrote: > Hi, > this is a query for an opinion and a best-practice experience for a > case when a package needs to change its internal database format > between versions, in an environment, which does not allow real > migration, aka the app cannot read both formats, it can use one or the > other. ...snip... > > I'm sure you might find more disadvantages, these are just the top > four. > > That being said, any such change surely deserves release notes, with > the commands what to do to export and then back import the wordlist. > There should be filled a corresponding Change too, I guess. > > Still, how does other packages migrate their data, or at least help the > users with the migration? Is there any way? Modifying users home directories is explicitly against packagin guidelines. ;( https://docs.fedoraproject.org/en-US/packaging-guidelines/#_no_files_or_directories_under_srv_usrlocal_or_homeuser Is there any way to pull the functionaly into the process itself? ie, the first time it's called, it converts the db? (But that might make it really slow on the first call). Or perhaps it just warns there and points to the script the user can run to do the conversion? I suppose there's the way postgresql does it... including a copy of the old server binaries that can read the old format in the new rpm so it can use that to convert to the new format? kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: How to migrate database format during package update?
On Wed, 2023-02-01 at 14:15 +0100, Alexander Sosedkin wrote: > cyrus-sasl ships a migration tool for some transition period > and suggests the user to manually invoke it: Hi, aha, I see, that's much saner idea than what I came up with. I'll try to cook something similar, making the libdb(-static) dependency optional (no migration tool without it). Thanks and bye, Milan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: How to migrate database format during package update?
On Wed, Feb 1, 2023 at 1:57 PM Milan Crha wrote: > this is a query for an opinion and a best-practice experience for a > case when a package needs to change its internal database format > between versions, in an environment, which does not allow real > migration, aka the app cannot read both formats, it can use one or the > other. > > To be specific, the libdb package is going to be removed [1] sooner or > later, and one of the packages which still use it is Bogofilter [2]. > It's easy to just switch from libdb to SQLite in the .spec file, the > problem is that the years of user's training of the Bogofilter spam/ham > data would be lost by such change. The SQLite version cannot read the > libdb data and vice versa (obviously), thus there needs to be done some > manual migration by the users. The worst, the users may need to export > their wordlist data before the update/upgrade and import it back after > the update/upgrade. Also, when the bogofilter is run with the other > format of the wordlist database, it simply errors out and expects > manual intervention. It's also good, the old data is not completely > lost, after all, and the user is aware something changed. > > My idea was to help the users with the migration in a way that the > export of the libdb data could be done during the package update, in > the %pre stage. A very nasty way I came with is to traverse the /home/ > directories and if there exists the old database, then export it and > rename the old file, thus the new bogofilter can run, even with an > empty database. The thing is that the exported data is ready in such > case, no need to downgrade the package or install old Fedora or any > such thing just to recover the wordlist. It's the only advantage, while > there are many disadvantages: > - it's a nasty approach, the package should not touch users' home > - it works only if the user uses the default/expected setting; saving > the database into a different place voids this best-effort approach > - the exported file is owned by root/admin, which requires change of > the attributes thus the user can get rid of it > - the package cannot tell to the user what to do next, to restore > the exported data. > > I'm sure you might find more disadvantages, these are just the top > four. > > That being said, any such change surely deserves release notes, with > the commands what to do to export and then back import the wordlist. > There should be filled a corresponding Change too, I guess. > > Still, how does other packages migrate their data, or at least help the > users with the migration? Is there any way? cyrus-sasl ships a migration tool for some transition period and suggests the user to manually invoke it: https://src.fedoraproject.org/rpms/cyrus-sasl/blob/rawhide/f/cyrus-sasl-2.1.27-Migration-from-BerkeleyDB.patch Personally, I believe that even automatically converting a database on access and leaving a backup is already borderline risky, as the administrator doesn't know that about the conversion and nobody looks at any logs if things keep working. Please let's not entertain the idea of actively crawling for databases =) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
How to migrate database format during package update?
Hi, this is a query for an opinion and a best-practice experience for a case when a package needs to change its internal database format between versions, in an environment, which does not allow real migration, aka the app cannot read both formats, it can use one or the other. To be specific, the libdb package is going to be removed [1] sooner or later, and one of the packages which still use it is Bogofilter [2]. It's easy to just switch from libdb to SQLite in the .spec file, the problem is that the years of user's training of the Bogofilter spam/ham data would be lost by such change. The SQLite version cannot read the libdb data and vice versa (obviously), thus there needs to be done some manual migration by the users. The worst, the users may need to export their wordlist data before the update/upgrade and import it back after the update/upgrade. Also, when the bogofilter is run with the other format of the wordlist database, it simply errors out and expects manual intervention. It's also good, the old data is not completely lost, after all, and the user is aware something changed. My idea was to help the users with the migration in a way that the export of the libdb data could be done during the package update, in the %pre stage. A very nasty way I came with is to traverse the /home/ directories and if there exists the old database, then export it and rename the old file, thus the new bogofilter can run, even with an empty database. The thing is that the exported data is ready in such case, no need to downgrade the package or install old Fedora or any such thing just to recover the wordlist. It's the only advantage, while there are many disadvantages: - it's a nasty approach, the package should not touch users' home - it works only if the user uses the default/expected setting; saving the database into a different place voids this best-effort approach - the exported file is owned by root/admin, which requires change of the attributes thus the user can get rid of it - the package cannot tell to the user what to do next, to restore the exported data. I'm sure you might find more disadvantages, these are just the top four. That being said, any such change surely deserves release notes, with the commands what to do to export and then back import the wordlist. There should be filled a corresponding Change too, I guess. Still, how does other packages migrate their data, or at least help the users with the migration? Is there any way? Thanks and bye, Milan [1] https://bugzilla.redhat.com/show_bug.cgi?id=1778802 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1788486 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
