Re: [HEADS UP] pkgsrc default database directory changed
> Date: Sun, 13 Dec 2020 20:11:53 +0100
> From: Thomas Klausner
>
> On Sun, Dec 13, 2020 at 02:18:32PM +0100, Rhialto wrote:
> > A problem with pkg.refcount might be that files in that directory
> > contain absolute pathnames starting with /var/db/pkg. E.g.:
> >
> > $ cat /var/db/pkg.refcount/groups/mail/cyrus-sasl-2.1.27nb1
> > /var/db/pkg/cyrus-sasl-2.1.27nb1
> >
> > $ cat
> > /var/db/pkg.refcount/files/usr/pkg/lib/perl5/vendor_perl/5.32.0/XML/SAX/ParserDetails.ini/p5-XML-SAX-1.02
> >
> > /var/db/pkg/p5-XML-SAX-1.02
> >
> > $ cat /var/db/pkg.refcount/users/smmsp/sendmail-8.15.2nb9
> > /var/db/pkg/sendmail-8.15.2nb9
>
> That looks like a design flaw :(
>
> IIUC the effect will be that some empty directories are not deleted
> when the last packages using them are deleted.
>
> Is anyone willing to write up a shell script to fix them?
Before we distribute more band-aids on this that might require manual
intervention, maybe we could step back and assess the situation to
make sure the issues are being comprehensively addressed, and perhaps
find a way to do it that is safe, idempotent, and requires no manual
intervention by users?
We have a lot of different types of users who might be affected by
this, and I worry that the pkgdb migration had already gotten out of
hand before we multiplied the cases that are out there with updates to
base and pullups to -8 and -9:
- netbsd-{8,9,HEAD} users using base pkg_install exclusively who have
updated
. base and pkgsrc
. base
. pkgsrc
. neither
- netbsd-{8,9,HEAD} users using bootstrapped pkg_install exclusively
who {have,haven't} updated pkgsrc
- netbsd-{8,9,HEAD} users using base pkg_install for main packages but
bootstrapped pkg_install for /home/user/pkg or /usr/pbulk or
similar, who have updated
. base and pkgsrc
. base
. pkgsrc
. neither
All these need to be multiplied by the different versions of the
manual intervention we've publicly suggested so far. Did I miss any
important cases that we're likely to encounter?
Re: [HEADS UP] pkgsrc default database directory changed
On Sun, Dec 13, 2020 at 02:18:32PM +0100, Rhialto wrote: > On Sat 05 Dec 2020 at 15:14:52 +0100, Thomas Klausner wrote: > > So the complete instructions as of now are: > > > > cd /usr/pkgsrc/pkgtools/pkg_install > > make USE_CWRAPPERS=no install > > cd /usr/pkg/sbin && install -c pkg_add pkg_admin pkg_create pkg_delete > > pkg_info /usr/sbin/ > > mv /var/db/pkg /usr/pkg/pkgdb > > mv /var/db/pkg.refcount /usr/pkg/pkgdb.refcount > > A problem with pkg.refcount might be that files in that directory > contain absolute pathnames starting with /var/db/pkg. E.g.: > > $ cat /var/db/pkg.refcount/groups/mail/cyrus-sasl-2.1.27nb1 > /var/db/pkg/cyrus-sasl-2.1.27nb1 > > $ cat > /var/db/pkg.refcount/files/usr/pkg/lib/perl5/vendor_perl/5.32.0/XML/SAX/ParserDetails.ini/p5-XML-SAX-1.02 > > /var/db/pkg/p5-XML-SAX-1.02 > > $ cat /var/db/pkg.refcount/users/smmsp/sendmail-8.15.2nb9 > /var/db/pkg/sendmail-8.15.2nb9 That looks like a design flaw :( IIUC the effect will be that some empty directories are not deleted when the last packages using them are deleted. Is anyone willing to write up a shell script to fix them? Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On Sat 05 Dec 2020 at 15:14:52 +0100, Thomas Klausner wrote: > So the complete instructions as of now are: > > cd /usr/pkgsrc/pkgtools/pkg_install > make USE_CWRAPPERS=no install > cd /usr/pkg/sbin && install -c pkg_add pkg_admin pkg_create pkg_delete > pkg_info /usr/sbin/ > mv /var/db/pkg /usr/pkg/pkgdb > mv /var/db/pkg.refcount /usr/pkg/pkgdb.refcount A problem with pkg.refcount might be that files in that directory contain absolute pathnames starting with /var/db/pkg. E.g.: $ cat /var/db/pkg.refcount/groups/mail/cyrus-sasl-2.1.27nb1 /var/db/pkg/cyrus-sasl-2.1.27nb1 $ cat /var/db/pkg.refcount/files/usr/pkg/lib/perl5/vendor_perl/5.32.0/XML/SAX/ParserDetails.ini/p5-XML-SAX-1.02 /var/db/pkg/p5-XML-SAX-1.02 $ cat /var/db/pkg.refcount/users/smmsp/sendmail-8.15.2nb9 /var/db/pkg/sendmail-8.15.2nb9 -Olaf. -- Olaf 'Rhialto' Seibert -- rhialto at falu dot nl ___ Anyone who is capable of getting themselves made President should on \X/ no account be allowed to do the job. --Douglas Adams, "THGTTG" signature.asc Description: PGP signature
Re: [HEADS UP] pkgsrc default database directory changed
Thomas Klausner wrote: > (The problem is that pkg_install and pkgsrc/mk need to be changed at > the same time, which is why this change is so difficult in the first > place.) Can one of either pkg_install or pkgsrc/mk be changed now to cater for the change in advance, at the other changed "soon" (maybe before 2020Q1 or before next . releases of netbsd-8 and nebsd-9?) so that something can be put in place to deal with the old and new locations without as much manual hand holding? And that deals cleanly with people who have already made the change? [ Note: still on pkgsrc-2020Q3 so haven't actually had to deal with this yet. ] Cheers, Simon.
Re: [HEADS UP] pkgsrc default database directory changed
On Thu, Dec 10, 2020 at 07:29:05PM +0300, Valeriy E. Ushakov wrote:
> On Wed, Dec 02, 2020 at 11:28:41 +0100, Thomas Klausner wrote:
>
> > The new default for the pkgsrc database (which contains information
> > about all installed packages) in pkgsrc-HEAD has changed from
> > /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
>
> This seems to be broken with current (with the new pkg_install)
> vs. building from source from pkgsrc-quarterly
>
> The new tools use the new location by default, but the quarterly
> branch uses PKG_DBDIR?=/var/db/pkg, so when you build packages the
> build process looks (and writes) to the wrong place. So you need
> PKG_DBDIR=/usr/pkg/pkgdb in mk.conf
That is the best solution, yes.
(The problem is that pkg_install and pkgsrc/mk need to be changed at
the same time, which is why this change is so difficult in the first
place.)
Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, Dec 02, 2020 at 11:28:41 +0100, Thomas Klausner wrote:
> The new default for the pkgsrc database (which contains information
> about all installed packages) in pkgsrc-HEAD has changed from
> /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
This seems to be broken with current (with the new pkg_install)
vs. building from source from pkgsrc-quarterly
The new tools use the new location by default, but the quarterly
branch uses PKG_DBDIR?=/var/db/pkg, so when you build packages the
build process looks (and writes) to the wrong place. So you need
PKG_DBDIR=/usr/pkg/pkgdb in mk.conf
-uwe
Re: [HEADS UP] pkgsrc default database directory changed
On Mon, Dec 07, 2020 at 11:23:38AM +0100, Matthias Petermann wrote: > while I think the change makes sense (if I understand it correctly, it will > make it easier for me in the future to switch between different PKG > locations including the corresponding metadata just by renaming the > respective /usr/pkg directory), It doesn't do that unless you also change VARBASE. (and note that packages respecting the setting of VARBASE is hit-or-miss so it doesn't necessarily help that much to change it) -- David A. Holland dholl...@netbsd.org
Re: [HEADS UP] pkgsrc default database directory changed
On Mon, Dec 07, 2020 at 03:16:44PM +, Mike Pumford wrote: > And that requires a manual build of pkg_install (skipping the normal > cwrappers dependency) on 9-stable and 8-stable. Not sure if that will be > true after pullups have happened but I suspect it might still be as there > won't be a version number bump that pkgsrc can use to detect the newer tool. No, after installing a *-stable from after the pullup you don't have to do anything -- pkgsrc uses the internal version number of the tools (see "pkg_info -V"). Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On 07/12/2020 13:17, Thomas Klausner wrote: While that is true, pkgsrc will insist on using pkg_install 20200828 or newer, so if you want to build packages from source, you'll still have to update your installed pkg_install. And that requires a manual build of pkg_install (skipping the normal cwrappers dependency) on 9-stable and 8-stable. Not sure if that will be true after pullups have happened but I suspect it might still be as there won't be a version number bump that pkgsrc can use to detect the newer tool. Mike
Re: [HEADS UP] pkgsrc default database directory changed
On Mon, 7 Dec 2020 at 13:18, Thomas Klausner wrote: > > On Mon, Dec 07, 2020 at 06:53:57PM +0700, Robert Elz wrote: > > Date:Mon, 7 Dec 2020 11:38:14 +0100 > > From:Thomas Klausner > > Message-ID: <20201207103814.xbnkdiwpzo6xl...@yt.nih.at> > > > > | > 2) requires a more recent pkg_install than the one from NetBSD 9.1 > > | > > | That is correct. If you are on any currently supported release you > > | will have to manually update pkg_install first before using pkgsrc. > > > > Actually, no, that shouldn't be needed - something needs to manually be > > done, but it doesn't need to be that one, (it is non trivial). > > > > Either symlinking the old pgkdb to the new one, or adding > > PKG_DBDIR=/usr/pkg/pkgdb > > (or whevever it will be placed) in /etc/pkg_install.conf (which probably > > doesn't exist, so that one line will be all it contains .. the line should > > be at the left margin - might not be required, but works if it is). > > > > Either of those is much easier than replacing the old pkg_install just > > for this change. After doing either (on NetBSD) the system pkg_install > > set of tools will all work just fine with the relocated pkg database. > > While that is true, pkgsrc will insist on using pkg_install 20200828 > or newer, so if you want to build packages from source, you'll still > have to update your installed pkg_install. Agreed, but mixing pkg tools from /usr/sbin and /usr/pkg/sbin no longer triggers a disaster. Maybe we could update the migration script to put the symlinks in place? David
Re: [HEADS UP] pkgsrc default database directory changed
On Mon, Dec 07, 2020 at 06:53:57PM +0700, Robert Elz wrote: > Date:Mon, 7 Dec 2020 11:38:14 +0100 > From:Thomas Klausner > Message-ID: <20201207103814.xbnkdiwpzo6xl...@yt.nih.at> > > | > 2) requires a more recent pkg_install than the one from NetBSD 9.1 > | > | That is correct. If you are on any currently supported release you > | will have to manually update pkg_install first before using pkgsrc. > > Actually, no, that shouldn't be needed - something needs to manually be > done, but it doesn't need to be that one, (it is non trivial). > > Either symlinking the old pgkdb to the new one, or adding > PKG_DBDIR=/usr/pkg/pkgdb > (or whevever it will be placed) in /etc/pkg_install.conf (which probably > doesn't exist, so that one line will be all it contains .. the line should > be at the left margin - might not be required, but works if it is). > > Either of those is much easier than replacing the old pkg_install just > for this change. After doing either (on NetBSD) the system pkg_install > set of tools will all work just fine with the relocated pkg database. While that is true, pkgsrc will insist on using pkg_install 20200828 or newer, so if you want to build packages from source, you'll still have to update your installed pkg_install. Thomas
Re: [HEADS UP] pkgsrc default database directory changed
Date:Mon, 7 Dec 2020 11:38:14 +0100 From:Thomas Klausner Message-ID: <20201207103814.xbnkdiwpzo6xl...@yt.nih.at> | > 2) requires a more recent pkg_install than the one from NetBSD 9.1 | | That is correct. If you are on any currently supported release you | will have to manually update pkg_install first before using pkgsrc. Actually, no, that shouldn't be needed - something needs to manually be done, but it doesn't need to be that one, (it is non trivial). Either symlinking the old pgkdb to the new one, or adding PKG_DBDIR=/usr/pkg/pkgdb (or whevever it will be placed) in /etc/pkg_install.conf (which probably doesn't exist, so that one line will be all it contains .. the line should be at the left margin - might not be required, but works if it is). Either of those is much easier than replacing the old pkg_install just for this change. After doing either (on NetBSD) the system pkg_install set of tools will all work just fine with the relocated pkg database. kre
Re: [HEADS UP] pkgsrc default database directory changed
On Mon, Dec 07, 2020 at 11:23:38AM +0100, Matthias Petermann wrote: > Hello everybody, > > while I think the change makes sense (if I understand it correctly, it will > make it easier for me in the future to switch between different PKG > locations including the corresponding metadata just by renaming the > respective /usr/pkg directory), I would be very happy to read a short > statement about the chronological classification again. > > Is it correct that: > > 1) this change will be active in pkgsrc-2020Q4 It is active in pkgsrc right now, the first branch that will be affected will be pkgsrc-2020Q4. > 2) requires a more recent pkg_install than the one from NetBSD 9.1 That is correct. If you are on any currently supported release you will have to manually update pkg_install first before using pkgsrc. NetBSD-current (as of shortly before 9.99.77) comes with the new pkg_install. Pullups for the active release branches (8 and 9) have been filed. Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On 07/12/2020 10:06, Thomas Klausner wrote: On Mon, Dec 07, 2020 at 09:53:46AM +, Mike Pumford wrote: More fallout from this change: => Build dependency cwrappers>=20150314: NOT found => Verifying reinstall for ../../pkgtools/cwrappers ===> Trying to handle out-dated pkg_install... ===> Cleaning for pkg_install-20201205 ERROR: This package has set PKG_FAIL_REASON: ERROR: Circular dependency detected This is on an empty system trying to build cwrappers. You must rebuild and replace pkg_install first. What an awful user experience for a first time user. Oh pkgsrc doesn't work unless you do some special magic first. Also in this particular case doing that step is actually awkward because the builds are being done by an script (which I've not got to modify to handle this ridiculous special case). Mike
Re: [HEADS UP] pkgsrc default database directory changed
Hello everybody, while I think the change makes sense (if I understand it correctly, it will make it easier for me in the future to switch between different PKG locations including the corresponding metadata just by renaming the respective /usr/pkg directory), I would be very happy to read a short statement about the chronological classification again. Is it correct that: 1) this change will be active in pkgsrc-2020Q4 2) requires a more recent pkg_install than the one from NetBSD 9.1 ? Thanks and best regards Matthias
Re: [HEADS UP] pkgsrc default database directory changed
On Mon, Dec 07, 2020 at 09:53:46AM +, Mike Pumford wrote: > More fallout from this change: > > => Build dependency cwrappers>=20150314: NOT found > => Verifying reinstall for ../../pkgtools/cwrappers > ===> Trying to handle out-dated pkg_install... > ===> Cleaning for pkg_install-20201205 > ERROR: This package has set PKG_FAIL_REASON: > ERROR: Circular dependency detected > > This is on an empty system trying to build cwrappers. You must rebuild and replace pkg_install first. > Also one other question. What happens to pkgin users on an update? After replacing the pkg_install tools, I suggest reinstalling pkgin as well. Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On 07/12/2020 02:13, Thomas Mueller wrote: from Thomas Klausner : I've collected specific advice outside of the basic ones here: http://pkgsrc.org/news/pkgdb-change/ More fallout from this change: => Build dependency cwrappers>=20150314: NOT found => Verifying reinstall for ../../pkgtools/cwrappers ===> Trying to handle out-dated pkg_install... ===> Cleaning for pkg_install-20201205 ERROR: This package has set PKG_FAIL_REASON: ERROR: Circular dependency detected This is on an empty system trying to build cwrappers. Also one other question. What happens to pkgin users on an update? It doesn't treat pkg_install specally from memory. If it doesn't he transition from old db to new db will happen in the middle of an upgrade. Which will blow up spectactularly. Mike
Re: [HEADS UP] pkgsrc default database directory changed
from Thomas Klausner : > I've collected specific advice outside of the basic ones here: > http://pkgsrc.org/news/pkgdb-change/ My installed packages are in a mess, and I probably need to rebuild from the beginning; also I intend to switch from modular pkgsrc Xorg to native Xorg Packages mess was due to pkg_chk gone awry, and pkg_rolling replace not being able to handle Xorg upgrade, as well as the transition from Python 2.x to 3.x, which played havoc with other package systems besides pkgsrc. Besides, I will be upgrading from 8.99.51 to current. I was going to mv /usr/pkg to /usr/pkg-old and /var/db/pkg to /var/db/pkg-old, but I guess the latter part no longer matters. I guess /var/db/pkg and /var/db/pkg.refcount will just become deadwood. Is the advice on http://pkgsrc.org/news/pkgdb-change for installing from binary or building from source? I suppose I would need to update /etc/mk.conf , and build cwrappers and/or pkg_install with USE_CWRAPPERS=no to avoid the circular dependency. Tom
Re: [HEADS UP] pkgsrc default database directory changed
On Sat, Dec 5, 2020 at 9:19 PM Thomas Klausner wrote: > http://pkgsrc.org/news/pkgdb-change/ Thanks, that's helpful. I need some more advice. Apparently, I have gotten into a split-brain situation where /usr/sbin/pkg_info and /usr/pkg/sbin/pkg_info show me disjoint sets of packages. I would like to merge the two databases without reinstalling everything, if possible. For pkgdb, I imagine that moving all directories from /var/db/pkg to /usr/pkg/pkgdb should work. But how do I reconcile pkgdb.refcount? -- Benny
Re: [HEADS UP] pkgsrc default database directory changed
On Sat, Dec 05, 2020 at 10:32:57AM -0800, John Nemeth wrote:
> On Dec 5, 15:14, Thomas Klausner wrote:
> } On Wed, Dec 02, 2020 at 11:28:41AM +0100, Thomas Klausner wrote:
> } > The new default for the pkgsrc database (which contains information
> } > about all installed packages) in pkgsrc-HEAD has changed from
> } > /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
> }
> } martin@ just pointed out to me that I had overlooked
> } /var/db/pkg.refcount (to path to which defaults to
> } ${PKG_DBDIR}.refcount, so it also moved).
> }
> } So the complete instructions as of now are:
> }
> } cd /usr/pkgsrc/pkgtools/pkg_install
> } make USE_CWRAPPERS=no install
> } cd /usr/pkg/sbin && install -c pkg_add pkg_admin pkg_create pkg_delete
> pkg_info /usr/sbin/
> } mv /var/db/pkg /usr/pkg/pkgdb
> } mv /var/db/pkg.refcount /usr/pkg/pkgdb.refcount
>
> Are there instructions on changes needed to pbulk setups?
I've collected specific advice outside of the basic ones here:
http://pkgsrc.org/news/pkgdb-change/
For pbulk in particular:
* make sure the pkg_install in your sandbox is up-to-date
* update the pbulk.conf file for the new pkgdb location (pkgdb=/usr/pkg/pkgdb)
Cheers,
Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On Dec 5, 15:14, Thomas Klausner wrote:
} On Wed, Dec 02, 2020 at 11:28:41AM +0100, Thomas Klausner wrote:
} > The new default for the pkgsrc database (which contains information
} > about all installed packages) in pkgsrc-HEAD has changed from
} > /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
}
} martin@ just pointed out to me that I had overlooked
} /var/db/pkg.refcount (to path to which defaults to
} ${PKG_DBDIR}.refcount, so it also moved).
}
} So the complete instructions as of now are:
}
} cd /usr/pkgsrc/pkgtools/pkg_install
} make USE_CWRAPPERS=no install
} cd /usr/pkg/sbin && install -c pkg_add pkg_admin pkg_create pkg_delete
pkg_info /usr/sbin/
} mv /var/db/pkg /usr/pkg/pkgdb
} mv /var/db/pkg.refcount /usr/pkg/pkgdb.refcount
Are there instructions on changes needed to pbulk setups?
}-- End of excerpt from Thomas Klausner
Re: [HEADS UP] pkgsrc default database directory changed
On Sat, 5 Dec 2020 at 14:15, Thomas Klausner wrote:
>
> On Wed, Dec 02, 2020 at 11:28:41AM +0100, Thomas Klausner wrote:
> > The new default for the pkgsrc database (which contains information
> > about all installed packages) in pkgsrc-HEAD has changed from
> > /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
>
> martin@ just pointed out to me that I had overlooked
> /var/db/pkg.refcount (to path to which defaults to
> ${PKG_DBDIR}.refcount, so it also moved).
>
> So the complete instructions as of now are:
>
> cd /usr/pkgsrc/pkgtools/pkg_install
> make USE_CWRAPPERS=no install
> cd /usr/pkg/sbin && install -c pkg_add pkg_admin pkg_create pkg_delete
> pkg_info /usr/sbin/
> mv /var/db/pkg /usr/pkg/pkgdb
> mv /var/db/pkg.refcount /usr/pkg/pkgdb.refcount
Can we add a shell script to pkgsrc which would do this (potentially
with any necessary sanity checks), particularly as if the procedure
ends up being tweaked again the script will just do the right thing
rather than people having to hope they have the latest version of the
list of commands.
Thanks
David
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, Dec 02, 2020 at 11:28:41AM +0100, Thomas Klausner wrote:
> The new default for the pkgsrc database (which contains information
> about all installed packages) in pkgsrc-HEAD has changed from
> /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
martin@ just pointed out to me that I had overlooked
/var/db/pkg.refcount (to path to which defaults to
${PKG_DBDIR}.refcount, so it also moved).
So the complete instructions as of now are:
cd /usr/pkgsrc/pkgtools/pkg_install
make USE_CWRAPPERS=no install
cd /usr/pkg/sbin && install -c pkg_add pkg_admin pkg_create pkg_delete pkg_info
/usr/sbin/
mv /var/db/pkg /usr/pkg/pkgdb
mv /var/db/pkg.refcount /usr/pkg/pkgdb.refcount
Cheers,
Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On Dec 4, 11:07, Mike Pumford wrote: } On 02/12/2020 23:41, Thomas Klausner wrote: } > On Wed, Dec 02, 2020 at 03:07:55PM -0800, Paul Goyette wrote: } >> This is just getting too complicated. Too many manual steps, and too } >> many changes to too many long-established procedures. } > } > Yeah, I'm sorry that you spent too much time on this. } > } > Actually, the easiest way is to just: } > } > cd /usr/pkgsrc/pkgtools/pkg_install } > make USE_CWRAPPERS=no install } > install -c /usr/pkg/sbin/pkg_* /usr/sbin } } Err. This works great until the next system update. At which point the } system pkg_* binaries will be re-instated and break everything. Unless } you are going to ensure that /usr/sbin/pkg_* are going to be updated not } just in current but also in the 9, 8 and 7 release branches which are } all still supported. Minor correction: 7 is no longer supported. Generally only two release branches are supported with the oldest being desupported about a month after a new major release comes out. 7 got extra time due to Covid-19, but was completely desupported some time ago. } Any solution that overwrites core binaries with a binary from pkgsrc } won't survive an OS upgrade. True enough. } As others have said this is starting to look like the consequences of } the change have not been thought through. } } Mike }-- End of excerpt from Mike Pumford
Re: [HEADS UP] pkgsrc default database directory changed
On Fri, Dec 4, 2020 at 1:52 PM David Brownlee wrote: > > On Fri, 4 Dec 2020 at 11:07, Mike Pumford wrote: > > On 02/12/2020 23:41, Thomas Klausner wrote: > > > On Wed, Dec 02, 2020 at 03:07:55PM -0800, Paul Goyette wrote: > > >> This is just getting too complicated. Too many manual steps, and too > > >> many changes to too many long-established procedures. > > > > > > Yeah, I'm sorry that you spent too much time on this. > > > > > > Actually, the easiest way is to just: > > > > > > cd /usr/pkgsrc/pkgtools/pkg_install > > > make USE_CWRAPPERS=no install > > > install -c /usr/pkg/sbin/pkg_* /usr/sbin > > > > > Err. This works great until the next system update. At which point the > > system pkg_* binaries will be re-instated and break everything. Unless > > you are going to ensure that /usr/sbin/pkg_* are going to be updated not > > just in current but also in the 9, 8 and 7 release branches which are > > all still supported. > > > > Any solution that overwrites core binaries with a binary from pkgsrc > > won't survive an OS upgrade. > > > > As others have said this is starting to look like the consequences of > > the change have not been thought through. > > Plan for an immediate workaround, and a later fix. Possibly: > > - Immediately: Update pkg_install in pkgsrc for NetBSD: if /var/db/pkg > is present and $PREFIX/pkgdb is not, use /var/db/pkg with a warning. > If both are present, also show a warning and pick on (probably > $PREFIX/pkgdb) > - Next: Coordinate pkg_install pullups into -7 -8 & -9, possibly of > the workaround version first > - Eventually: Drop the workaround, though possibly keep "if both are > present show error and stop" logic > > May need to tweak details, but something of this form > > David So all the corner cases and caveats that this change introduces make it seem much more feasible to just boostrap any pkgsrc installation even on any netbsd version from now on to avoid all the problems one could possibly have... Regards, Connor
Re: [HEADS UP] pkgsrc default database directory changed
On Fri, 4 Dec 2020 at 11:07, Mike Pumford wrote: > On 02/12/2020 23:41, Thomas Klausner wrote: > > On Wed, Dec 02, 2020 at 03:07:55PM -0800, Paul Goyette wrote: > >> This is just getting too complicated. Too many manual steps, and too > >> many changes to too many long-established procedures. > > > > Yeah, I'm sorry that you spent too much time on this. > > > > Actually, the easiest way is to just: > > > > cd /usr/pkgsrc/pkgtools/pkg_install > > make USE_CWRAPPERS=no install > > install -c /usr/pkg/sbin/pkg_* /usr/sbin > > > Err. This works great until the next system update. At which point the > system pkg_* binaries will be re-instated and break everything. Unless > you are going to ensure that /usr/sbin/pkg_* are going to be updated not > just in current but also in the 9, 8 and 7 release branches which are > all still supported. > > Any solution that overwrites core binaries with a binary from pkgsrc > won't survive an OS upgrade. > > As others have said this is starting to look like the consequences of > the change have not been thought through. Plan for an immediate workaround, and a later fix. Possibly: - Immediately: Update pkg_install in pkgsrc for NetBSD: if /var/db/pkg is present and $PREFIX/pkgdb is not, use /var/db/pkg with a warning. If both are present, also show a warning and pick on (probably $PREFIX/pkgdb) - Next: Coordinate pkg_install pullups into -7 -8 & -9, possibly of the workaround version first - Eventually: Drop the workaround, though possibly keep "if both are present show error and stop" logic May need to tweak details, but something of this form David
Re: [HEADS UP] pkgsrc default database directory changed
On 02/12/2020 23:41, Thomas Klausner wrote: On Wed, Dec 02, 2020 at 03:07:55PM -0800, Paul Goyette wrote: This is just getting too complicated. Too many manual steps, and too many changes to too many long-established procedures. Yeah, I'm sorry that you spent too much time on this. Actually, the easiest way is to just: cd /usr/pkgsrc/pkgtools/pkg_install make USE_CWRAPPERS=no install install -c /usr/pkg/sbin/pkg_* /usr/sbin Err. This works great until the next system update. At which point the system pkg_* binaries will be re-instated and break everything. Unless you are going to ensure that /usr/sbin/pkg_* are going to be updated not just in current but also in the 9, 8 and 7 release branches which are all still supported. Any solution that overwrites core binaries with a binary from pkgsrc won't survive an OS upgrade. As others have said this is starting to look like the consequences of the change have not been thought through. Mike
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, Dec 02, 2020 at 03:07:55PM -0800, Paul Goyette wrote: > This is just getting too complicated. Too many manual steps, and too > many changes to too many long-established procedures. Yeah, I'm sorry that you spent too much time on this. Actually, the easiest way is to just: cd /usr/pkgsrc/pkgtools/pkg_install make USE_CWRAPPERS=no install install -c /usr/pkg/sbin/pkg_* /usr/sbin Then you don't have to change anything else. The rest of the thread contains just options for power users (that want to stay back on the old location, or don't want to replace /usr/sbin/pkg_*). Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, 2 Dec 2020, Thomas Klausner wrote:
On Wed, Dec 02, 2020 at 02:57:01PM -0800, Paul Goyette wrote:
On Wed, 2 Dec 2020, Thomas Klausner wrote:
On Wed, Dec 02, 2020 at 11:28:41AM +0100, Thomas Klausner wrote:
The new default for the pkgsrc database (which contains information
about all installed packages) in pkgsrc-HEAD has changed from
/var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
Since some people have trouble installing a new pkg_install from
pkgsrc (because it depends on cwrappers, which depends on pkg_install)
- the easiest way to get a new pkg_install from pkgsrc is this:
cd /usr/pkgsrc/pkgtools/pkg_install
make USE_CWRAPPERS=no install
That will build pkg_install without cwrappers.
I suggest replacing /usr/sbin/pkg_* with /usr/pkg/sbin/pkg_* after
this succeeds to avoid further problems.
(I'm one of the "somple haqve trouble installing..." mentioned above!)
Well, that's a bit of a problem for me. I'm building everything in
a chroot to a freshly-created sandbox (from sysutils/mksandbox) and
/usr is a read-only null-mount file-system. So I cannot copy the
pkgsrc images to the "real" system. (And I'm not sure I would want
to be able to affect the "real" system's /usr/bin/ from within the
chroot sandbox.)
You can cp it outside the sandbox... and use pkg_install.conf if you
want to keep using the old database directory.
This is just getting too complicated. Too many manual steps, and too
many changes to too many long-established procedures.
It sounds to me like this was not very well thought out and not very
well tested (in sufficient environments) before the change was made.
FWIW, I also don't remember ever seeing any discussion of this change
being made on any non-pkgsrc lists. (If it was discusssed, and I
simply missed it, skip this paragraph!) Since you've changed defaults
withing programs in the base system, a discussion on tech-userlevel
would probably have been appropriate (in addition to any relevant
pkgsrc lists).
++--+---+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | p...@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoye...@netbsd.org |
++--+---+
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, Dec 02, 2020 at 02:57:01PM -0800, Paul Goyette wrote:
> On Wed, 2 Dec 2020, Thomas Klausner wrote:
>
> > On Wed, Dec 02, 2020 at 11:28:41AM +0100, Thomas Klausner wrote:
> > > The new default for the pkgsrc database (which contains information
> > > about all installed packages) in pkgsrc-HEAD has changed from
> > > /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
> >
> > Since some people have trouble installing a new pkg_install from
> > pkgsrc (because it depends on cwrappers, which depends on pkg_install)
> > - the easiest way to get a new pkg_install from pkgsrc is this:
> >
> > cd /usr/pkgsrc/pkgtools/pkg_install
> > make USE_CWRAPPERS=no install
> >
> > That will build pkg_install without cwrappers.
> >
> > I suggest replacing /usr/sbin/pkg_* with /usr/pkg/sbin/pkg_* after
> > this succeeds to avoid further problems.
>
> (I'm one of the "somple haqve trouble installing..." mentioned above!)
>
> Well, that's a bit of a problem for me. I'm building everything in
> a chroot to a freshly-created sandbox (from sysutils/mksandbox) and
> /usr is a read-only null-mount file-system. So I cannot copy the
> pkgsrc images to the "real" system. (And I'm not sure I would want
> to be able to affect the "real" system's /usr/bin/ from within the
> chroot sandbox.)
You can cp it outside the sandbox... and use pkg_install.conf if you
want to keep using the old database directory.
Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, 2 Dec 2020, Thomas Klausner wrote:
On Wed, Dec 02, 2020 at 11:28:41AM +0100, Thomas Klausner wrote:
The new default for the pkgsrc database (which contains information
about all installed packages) in pkgsrc-HEAD has changed from
/var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
Since some people have trouble installing a new pkg_install from
pkgsrc (because it depends on cwrappers, which depends on pkg_install)
- the easiest way to get a new pkg_install from pkgsrc is this:
cd /usr/pkgsrc/pkgtools/pkg_install
make USE_CWRAPPERS=no install
That will build pkg_install without cwrappers.
I suggest replacing /usr/sbin/pkg_* with /usr/pkg/sbin/pkg_* after
this succeeds to avoid further problems.
(I'm one of the "somple haqve trouble installing..." mentioned above!)
Well, that's a bit of a problem for me. I'm building everything in
a chroot to a freshly-created sandbox (from sysutils/mksandbox) and
/usr is a read-only null-mount file-system. So I cannot copy the
pkgsrc images to the "real" system. (And I'm not sure I would want
to be able to affect the "real" system's /usr/bin/ from within the
chroot sandbox.)
++--+---+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | p...@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoye...@netbsd.org |
++--+---+
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, Dec 02, 2020 at 11:28:41AM +0100, Thomas Klausner wrote:
> The new default for the pkgsrc database (which contains information
> about all installed packages) in pkgsrc-HEAD has changed from
> /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
Since some people have trouble installing a new pkg_install from
pkgsrc (because it depends on cwrappers, which depends on pkg_install)
- the easiest way to get a new pkg_install from pkgsrc is this:
cd /usr/pkgsrc/pkgtools/pkg_install
make USE_CWRAPPERS=no install
That will build pkg_install without cwrappers.
I suggest replacing /usr/sbin/pkg_* with /usr/pkg/sbin/pkg_* after
this succeeds to avoid further problems.
I also found out that pbulk.conf includes the path to the database, so
please update that as well (pkgdb variable).
Cheers,
Thomas
Re: [HEADS UP] pkgsrc default database directory changed
Date:Wed, 2 Dec 2020 07:25:48 -0800 (PST) From:Paul Goyette Message-ID: | If one adds tnn's suggested entry to /etc/mk.conf, is it also needed | to apply uwe's update to /etc/pkgpath.conf? Depends what state you leave the system, mk.conf is used when building stuff (and only then), pkgpath.conf is used at run-time (including by /etc/daily). pkg_install.conf is used internally by pkg_install. | One additional series of questions: | | * should anything be changed in postinstall(8) to at least check for |the "correct" location? | * should postinstall(8) attempt to fix it? (probably no) | * should postinstall(8) "obey" the suggested changes to mk.conf and/or |pkgpath.conf? Those I'll leave for someone else to consider. | Finally, I looked, and my system doesn't have any file, anywhere | (other than in source trees) with the name ``pkg_install''. Am I | missing something? Only that pkg_install is just the generic name for the set of pkg_* utilities, none of which is actually called pkg_install. It includes pkg_admin pkg_add pkg_delete pkg_info and pkg_create. For your setup, the versions in /usr/sbin should work fine with an entry in /etc/pkg_install.conf to specify where the database is to live. You probably also need /etc/pkgpath.conf for /etc/daily and mk.conf for the next time you build everything (that or an up to date checkout of HEAD, assuming that you want to use the new default path). kre
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, Dec 02, 2020 at 07:25:48AM -0800, Paul Goyette wrote: > If one adds tnn's suggested entry to /etc/mk.conf, is it also needed > to apply uwe's update to /etc/pkgpath.conf? The advise helps in different use cases: If you follow tnn's advise, you'll keep using the old database location and don't need to change anything else, since the old and new tools will operate on the same location. uwe's advice is needed if you do switch to the new location, and use the pkg_install tools from pkgsrc so that the daily cronjob gets the correct output. If you use the new pkg_install tools from -current (updated a few hours ago) you need neither. > One additional series of questions: > > * should anything be changed in postinstall(8) to at least check for > the "correct" location? > * should postinstall(8) attempt to fix it? (probably no) > * should postinstall(8) "obey" the suggested changes to mk.conf and/or > pkgpath.conf? IMHO postinstall should not do anything. > Finally, I looked, and my system doesn't have any file, anywhere > (other than in source trees) with the name ``pkg_install''. Am I > missing something? I've been running pkgsrc for ~forever without > any (apparent) issues. /etc/pkg_install.conf (documented in pkg_install.conf(5)) is an optional file. There is no default file installed in NetBSD. Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, 2 Dec 2020 at 10:28, Thomas Klausner wrote:
>
> The new default for the pkgsrc database (which contains information
> about all installed packages) in pkgsrc-HEAD has changed from
> /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
>
> This matches the default the bootstrap script had for a long time.
>
> pkgsrc-HEAD will insist on using pkg_install 20200828, which will
> print an error and ask the user to move the existing database to the
> new location. This move is not done automatically by purpose so that
> the administrators will take note.
>
> There is one potential pitfall: you'll have to make sure
> ${PREFIX}/sbin/pkg_* is used and not mixed with /usr/sbin/pkg_* (which
> will default to the old location until -current and the branches are
> updated).
>
> So please make sure your admin user's PATH has ${PREFIX}/sbin before
> /usr/sbin, or copy the new pkg_install tools from ${PREFIX}/sbin to
> /usr/sbin.
>
> There is no change for the existing binary packages, since they do not
> care where the information about the installed packages is stored.
Not wanting to dump more onto releng, but would this be a good reason
to try to get a new release on netbsd-8 and netbsd-9?
(on a related note - thanks to everyone who worked to get this change
and is going to be fielding fallout for a while :)
David
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, 2 Dec 2020, Valery Ushakov wrote:
On Wed, Dec 02, 2020 at 11:28:41 +0100, Thomas Klausner wrote:
There is one potential pitfall: you'll have to make sure
${PREFIX}/sbin/pkg_* is used and not mixed with /usr/sbin/pkg_* (which
will default to the old location until -current and the branches are
updated).
So please make sure your admin user's PATH has ${PREFIX}/sbin before
/usr/sbin, or copy the new pkg_install tools from ${PREFIX}/sbin to
/usr/sbin.
NetBSD daily cron job uses full paths. They can be changed by adding
pkg_admin=/usr/pkg/sbin/pkg_admin
pkg_info=/usr/pkg/sbin/pkg_info
to /etc/pkgpath.conf
And Tobias Nygren replied:
> The new default for the pkgsrc database (which contains information
> about all installed packages) in pkgsrc-HEAD has changed from
> /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
Since this is about changing a default, it should be mentioned in the
heads-up how to stay on the old path if migration is currently not
convenient. To do this one can add PKG_DBDIR=/var/db/pkg to mk.conf
before the new pkg_install is built.
If one adds tnn's suggested entry to /etc/mk.conf, is it also needed
to apply uwe's update to /etc/pkgpath.conf?
One additional series of questions:
* should anything be changed in postinstall(8) to at least check for
the "correct" location?
* should postinstall(8) attempt to fix it? (probably no)
* should postinstall(8) "obey" the suggested changes to mk.conf and/or
pkgpath.conf?
Finally, I looked, and my system doesn't have any file, anywhere
(other than in source trees) with the name ``pkg_install''. Am I
missing something? I've been running pkgsrc for ~forever without
any (apparent) issues.
++--+---+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | p...@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoye...@netbsd.org |
++--+---+
Re: [HEADS UP] pkgsrc default database directory changed
Date:Wed, 2 Dec 2020 12:01:08 +0100
From:Thomas Klausner
Message-ID: <20201202110108.7ml5gw3jospli...@yt.nih.at>
| The change is documented here:
|
| http://pkgsrc.org/news/pkgdb-change/
I have been using ${PREFIX}/db now for years, on regular NetBSD.
(It seemed, still seems, kind of silly to have /usr/pkg/pkgdb
as the location - with "pkg" in there twice ... but that is not
important).
To make that work (aside from moving from /var/db) all that was
needed was to put
PKG_DBDIR=/usr/pkg/db
in /etc/pkg_install.conf (that's all I have in that file).
With that, the standard tools, of any reasonable vintage, simply work,
and so there's no need to force the use of the pkgsrc version of the
pkg_* tools. This is a much simpler upgrade path for most
users than waiting upon an upgrade for their release (I don't
see one coming any time soon for NetBSD 6!)
kre
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, Dec 02, 2020 at 11:54:31AM +0100, Tobias Nygren wrote:
> On Wed, 2 Dec 2020 11:28:41 +0100
> Thomas Klausner wrote:
>
> > The new default for the pkgsrc database (which contains information
> > about all installed packages) in pkgsrc-HEAD has changed from
> > /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
>
> Since this is about changing a default, it should be mentioned in the
> heads-up how to stay on the old path if migration is currently not
> convenient. To do this one can add PKG_DBDIR=/var/db/pkg to mk.conf
> before the new pkg_install is built.
Thanks for this note.
Another user already remarked privately that this only affects NetBSD
-- all operating systems using the bootstrap method were already using
${PREFIX}/pkgdb.
The change is documented here:
http://pkgsrc.org/news/pkgdb-change/
Feel free to improve this.
Thanks,
Thomas
Re: [HEADS UP] pkgsrc default database directory changed
On Wed, 2 Dec 2020 11:28:41 +0100
Thomas Klausner wrote:
> The new default for the pkgsrc database (which contains information
> about all installed packages) in pkgsrc-HEAD has changed from
> /var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
Since this is about changing a default, it should be mentioned in the
heads-up how to stay on the old path if migration is currently not
convenient. To do this one can add PKG_DBDIR=/var/db/pkg to mk.conf
before the new pkg_install is built.
-Tobias
[HEADS UP] pkgsrc default database directory changed
Hi!
The new default for the pkgsrc database (which contains information
about all installed packages) in pkgsrc-HEAD has changed from
/var/db/pkg to ${PREFIX}/pkgdb (so usually /usr/pkg/pkgdb).
This matches the default the bootstrap script had for a long time.
pkgsrc-HEAD will insist on using pkg_install 20200828, which will
print an error and ask the user to move the existing database to the
new location. This move is not done automatically by purpose so that
the administrators will take note.
There is one potential pitfall: you'll have to make sure
${PREFIX}/sbin/pkg_* is used and not mixed with /usr/sbin/pkg_* (which
will default to the old location until -current and the branches are
updated).
So please make sure your admin user's PATH has ${PREFIX}/sbin before
/usr/sbin, or copy the new pkg_install tools from ${PREFIX}/sbin to
/usr/sbin.
There is no change for the existing binary packages, since they do not
care where the information about the installed packages is stored.
Cheers,
Thomas
