Re: Add versioned lib to system perl's @INC for non-packaged modules
The patch over in ports that was blocking this has been in for a while:
https://github.com/openbsd/ports/commit/bd756bd601bfcfa9ee7fc265d6c90c3536e51727
Although I haven't committed any of the port fixes to tell the ones that
don't pick it up automatically to install into the vendor lib, the
patches are over in that thread. There will likely be other ports that
need some sort of fix:
https://marc.info/?l=openbsd-ports=162769529204194=2
In any case, I'd like to get this in to hopefully lower the number of
broken systems when folks install modules with CPAN clients in the
future.
I am considering removing the installperl patch and creating the
directory in Makefile.bsd-wraapper for fewer local patches to maintain.
Comments, test results, OK?
On Fri, Jul 30, 2021 at 05:34:40PM -0700, Andrew Hewus Fresh wrote:
> Below is the corrected patch for this, but it's not quite ready yet.
> See the email over on ports@ with some adjustments to perl.port.mk and
> other patches to fix some ports fallout that sthen@ was super helpful in
> tracking down.
>
> The main issue was that somehow I lost a quote in the patch and so
> it didn't work right. But, this should apply and let you play with it
> if you like.
>
> Once we have a path forward with the ports fallout, I'll be looking for
> more comments and maybe OKs.
>
> On Sun, May 16, 2021 at 03:30:39PM -0700, Andrew Hewus Fresh wrote:
> > This patch is should make it easier to recover from problems that happen
> > when installing perl modules outside of the package system. You might
> > recognize the "loadable library and perl binaries are mismatched" error.
> >
> > Adding a versioned subdirectory for cpan modules means that when the
> > perl version changes it will no longer find those old modules and it
> > will be easy to remove them without disturbing things from packages.
> >
> > Only slightly related, but we can avoid breaking base tools like pkg_add
> > and friend by using "no lib '/usr/local/libdata/perl5/site_perl';" to
> > avoid looking in those directories at all.
> >
> >
> > The new @INC looks like this. The first "5.32.1" sitelib is where
> > external things, like cpan(1), will install. The current "site_perl"
> > directory (now vendorlib) will continue to be where ports/packages go.
> > Finally, "/usr/libdata" privlib is for things that ship with the base
> > system.
> >
> > $ perl -V | perl -ne 'print if /INC/..eof'
> > @INC:
> > /usr/local/libdata/perl5/site_perl/5.32.1/sparc64-openbsd
> > /usr/local/libdata/perl5/site_perl/5.32.1
> > /usr/local/libdata/perl5/site_perl/sparc64-openbsd
> > /usr/local/libdata/perl5/site_perl
> > /usr/libdata/perl5/sparc64-openbsd
> > /usr/libdata/perl5
> >
> >
> > One place this might cause temporary issues in the ports tree is
> > anything that doesn't use the overrides in perl.port.mk and looks up
> > sitelib itself. It should be fairly obvious as it should fail to
> > package because the files in the PLIST will not exist (they'll be in the
> > versioned subdir). I didn't run into that in my testing, but I didn't
> > complete a full bulk build.
> >
> > There do appear to be some annoyances with still shared directories for
> > man pages, in that if you install a CPAN module and then attempt to
> > pkg_add it, it complains because "files already exist" in the man
> > directory. We could separate the cpan man pages, or I think un-setting
> > the site_lib man*dir will mean cpan won't instal them. I haven't yet
> > tested that, but I'm not sure what would be best there.
> >
> >
> > Unfortunately I did do a bunch of testing of different ways to add
> > things to @INC which meant this patch also does some reorganization of
> > the contents, I can turn it into a smaller patch if requested and do the
> > cleanup separately.
> >
> >
> > Comments, OK? Should I see if sthen@ will do a full bulk build first?
Index: gnu/usr.bin/perl/config.over
===
RCS file: /cvs/src/gnu/usr.bin/perl/config.over,v
retrieving revision 1.22
diff -u -p -r1.22 config.over
--- gnu/usr.bin/perl/config.over5 Feb 2017 00:33:38 - 1.22
+++ gnu/usr.bin/perl/config.over31 Jul 2021 00:02:14 -
@@ -9,49 +9,85 @@ archname="`arch -s`-${osname}"
myarchname="$archname"
# Use correct paths for a distribution
-prefix='/usr'
+# site is where cpan clients install files
+# vendor is where the ports tree puts packages
+# priv is where the system installs files
+
+# We set up vendor paths for ports/packages
+usevendorprefix="${define}"
+d_vendorbin="${define}"
+d_vendorlib="${define}"
+d_vendorarch="${define}"
+#d_vendorscript="${define}"
+
+# Things from base go into /usr
prefixexp='/usr'
+prefix="${prefixexp}"
-# But site binaries go in /usr/local/bin for ports
-siteprefix='/usr/local'
+# But vendor and site files go in /usr/local for ports and cpan
siteprefixexp='/usr/local'
-installsitebin='/usr/local/bin'
Re: Add versioned lib to system perl's @INC for non-packaged modules
Hi, thanks to sthen@ for providing more background! so the bottom line seems to be that, once the code changes are tested, committed, and prove stable and once people come round to it, moving the site manual page directories out of /usr/local/man/ and making them versioned in exactly the same way as the sitelib directories may be a reasonable option. Even if the main effect of versioning them might be to make it more obvious for people what went wrong in case things get mixed up. Yours, Ingo
Re: Add versioned lib to system perl's @INC for non-packaged modules
On 2021/08/04 19:45, Ingo Schwarze wrote:
> Hi Andrew,
>
> Andrew Fresh wrote on Fri, Jul 30, 2021 at 05:34:40PM -0700:
> > On Sun, May 16, 2021 at 03:30:39PM -0700, Andrew Hewus Fresh wrote:
>
> >> There do appear to be some annoyances with still shared directories for
> >> man pages, in that if you install a CPAN module and then attempt to
> >> pkg_add it, it complains because "files already exist" in the man
> >> directory. We could separate the cpan man pages, or I think un-setting
> >> the site_lib man*dir will mean cpan won't instal them. I haven't yet
> >> tested that, but I'm not sure what would be best there.
>
> If i understand correctly, this is all about making installation of
> un-ported Perl modules work better and reducing clashes between doing
> that on the one hand and between package installation with pkg_add(1)
> on the other hand.
Correct. The specific problem that prompted this was people who had
installed something locally outside of ports - I think it was an updated
version of a core perl module - and made worse by pkg_add at the time
searching /usr/local/libdata/perl5/site_perl for modules and refusing
to run until the locally built module was removed.
pkg_add now uses "no lib ('/usr/local/libdata/perl5/site_perl')" so
it won't run into the same problem any longer, but most other perl
software is likely to still have the problem.
> Who will profit most from that? I guess very experienced Perl users
> and porters who play around a lot with many different Perl modules
> and with very large Perl programs, in particular when evaluating
> whether those large programs are worth porting. Not having manual
> pages installed for dependencies while trying to get a complicated
> program to work and while trying to understand it seems like a
> disservice to me. So "un-setting the site_lib man*dir" doesn't
> strike me as particularly convincing on first sight.
>
> I guess ordinary users are less likely to use this route, they are more
> likely to just use pkg_add(1). But if they do, for whatever reason,
> not getting manual pages is not good for them, either.
I think it's likely to be the other way round, at least people with more
experience of OpenBSD are more likely to start by making a port, partly
because they realise that installing modules from CPAN can cause
problems, partly to make it easier to clean up if needed.
I think ordinary users who need something that isn't ported are probably
more likely to pick it up from CPAN instead. (then, if they're more
experienced with Perl, they'll realise the cause of the "key mismatch"
error quickly and know what to do with it).
> So i think in the long term, installing those manual pages into their
> own manpath outside /ust/local/man/ would seem desirable to me.
> I'm not saying it needs to be done in the same commit; sorting that
> out can certainly be a later step.
I agree with all this.
> You suggest to make the sitelib directory versioned. Consequently,
> i guess that you want the following process to work:
>
> 1. install base and arbitrary numbers of packages
> 2. install lots of modules and programs from CPAN for evaluation
> 3. install the next Perl version outside base to figure out what
> needs to be done to ultimately move base to it
> 4. install various modules for the new Perl from step 3
> for testing purposes
>
> If i got that right, then the new man directory needs to be versioned
> just like the new sitelib directory, or the manual pages from steps 2
> and 4 will clash just like the libraries from steps 2 and 4 would.
> The price to pay, of course, is that you need to edit man.conf(5)
> for each new Perl you install. And not just you, the base system
> Perl maintainer, but all users installing stuff from CPAN.
I think perldoc would pick them up anyway - if so, users still have
an easy way to read the docs without editing man.conf.
> On the other hand, if working with two different Perl versions in
> parallel is not a requirement (for example because people do testing
> like in steps 3 and 4 on separate machines?), then i don't see why
> either the sitelib or the new man dir needs to be versioned at all,
> making everything simpler.
I think it's mostly to mitigate some user problems (at least, make it so
they are more obvious and less likely to need somebody more experienced
to point out what the problem is). On the one hand, that is already
reduced by the pkg_add change. But on the other, if it's simple enough
to reduce problems further then, as long as it doesn't get in the
way too much, why not..
Re: Add versioned lib to system perl's @INC for non-packaged modules
Hi Andrew, Andrew Fresh wrote on Fri, Jul 30, 2021 at 05:34:40PM -0700: > On Sun, May 16, 2021 at 03:30:39PM -0700, Andrew Hewus Fresh wrote: >> There do appear to be some annoyances with still shared directories for >> man pages, in that if you install a CPAN module and then attempt to >> pkg_add it, it complains because "files already exist" in the man >> directory. We could separate the cpan man pages, or I think un-setting >> the site_lib man*dir will mean cpan won't instal them. I haven't yet >> tested that, but I'm not sure what would be best there. If i understand correctly, this is all about making installation of un-ported Perl modules work better and reducing clashes between doing that on the one hand and between package installation with pkg_add(1) on the other hand. Who will profit most from that? I guess very experienced Perl users and porters who play around a lot with many different Perl modules and with very large Perl programs, in particular when evaluating whether those large programs are worth porting. Not having manual pages installed for dependencies while trying to get a complicated program to work and while trying to understand it seems like a disservice to me. So "un-setting the site_lib man*dir" doesn't strike me as particularly convincing on first sight. I guess ordinary users are less likely to use this route, they are more likely to just use pkg_add(1). But if they do, for whatever reason, not getting manual pages is not good for them, either. So i think in the long term, installing those manual pages into their own manpath outside /ust/local/man/ would seem desirable to me. I'm not saying it needs to be done in the same commit; sorting that out can certainly be a later step. You suggest to make the sitelib directory versioned. Consequently, i guess that you want the following process to work: 1. install base and arbitrary numbers of packages 2. install lots of modules and programs from CPAN for evaluation 3. install the next Perl version outside base to figure out what needs to be done to ultimately move base to it 4. install various modules for the new Perl from step 3 for testing purposes If i got that right, then the new man directory needs to be versioned just like the new sitelib directory, or the manual pages from steps 2 and 4 will clash just like the libraries from steps 2 and 4 would. The price to pay, of course, is that you need to edit man.conf(5) for each new Perl you install. And not just you, the base system Perl maintainer, but all users installing stuff from CPAN. On the other hand, if working with two different Perl versions in parallel is not a requirement (for example because people do testing like in steps 3 and 4 on separate machines?), then i don't see why either the sitelib or the new man dir needs to be versioned at all, making everything simpler. Yours, Ingo
Re: Add versioned lib to system perl's @INC for non-packaged modules
Below is the corrected patch for this, but it's not quite ready yet.
See the email over on ports@ with some adjustments to perl.port.mk and
other patches to fix some ports fallout that sthen@ was super helpful in
tracking down.
The main issue was that somehow I lost a quote in the patch and so
it didn't work right. But, this should apply and let you play with it
if you like.
Once we have a path forward with the ports fallout, I'll be looking for
more comments and maybe OKs.
On Sun, May 16, 2021 at 03:30:39PM -0700, Andrew Hewus Fresh wrote:
> This patch is should make it easier to recover from problems that happen
> when installing perl modules outside of the package system. You might
> recognize the "loadable library and perl binaries are mismatched" error.
>
> Adding a versioned subdirectory for cpan modules means that when the
> perl version changes it will no longer find those old modules and it
> will be easy to remove them without disturbing things from packages.
>
> Only slightly related, but we can avoid breaking base tools like pkg_add
> and friend by using "no lib '/usr/local/libdata/perl5/site_perl';" to
> avoid looking in those directories at all.
>
>
> The new @INC looks like this. The first "5.32.1" sitelib is where
> external things, like cpan(1), will install. The current "site_perl"
> directory (now vendorlib) will continue to be where ports/packages go.
> Finally, "/usr/libdata" privlib is for things that ship with the base
> system.
>
> $ perl -V | perl -ne 'print if /INC/..eof'
> @INC:
> /usr/local/libdata/perl5/site_perl/5.32.1/sparc64-openbsd
> /usr/local/libdata/perl5/site_perl/5.32.1
> /usr/local/libdata/perl5/site_perl/sparc64-openbsd
> /usr/local/libdata/perl5/site_perl
> /usr/libdata/perl5/sparc64-openbsd
> /usr/libdata/perl5
>
>
> One place this might cause temporary issues in the ports tree is
> anything that doesn't use the overrides in perl.port.mk and looks up
> sitelib itself. It should be fairly obvious as it should fail to
> package because the files in the PLIST will not exist (they'll be in the
> versioned subdir). I didn't run into that in my testing, but I didn't
> complete a full bulk build.
>
> There do appear to be some annoyances with still shared directories for
> man pages, in that if you install a CPAN module and then attempt to
> pkg_add it, it complains because "files already exist" in the man
> directory. We could separate the cpan man pages, or I think un-setting
> the site_lib man*dir will mean cpan won't instal them. I haven't yet
> tested that, but I'm not sure what would be best there.
>
>
> Unfortunately I did do a bunch of testing of different ways to add
> things to @INC which meant this patch also does some reorganization of
> the contents, I can turn it into a smaller patch if requested and do the
> cleanup separately.
>
>
> Comments, OK? Should I see if sthen@ will do a full bulk build first?
Index: gnu/usr.bin/perl/config.over
===
RCS file: /cvs/src/gnu/usr.bin/perl/config.over,v
retrieving revision 1.22
diff -u -p -r1.22 config.over
--- gnu/usr.bin/perl/config.over5 Feb 2017 00:33:38 - 1.22
+++ gnu/usr.bin/perl/config.over31 Jul 2021 00:02:14 -
@@ -9,49 +9,85 @@ archname="`arch -s`-${osname}"
myarchname="$archname"
# Use correct paths for a distribution
-prefix='/usr'
+# site is where cpan clients install files
+# vendor is where the ports tree puts packages
+# priv is where the system installs files
+
+# We set up vendor paths for ports/packages
+usevendorprefix="${define}"
+d_vendorbin="${define}"
+d_vendorlib="${define}"
+d_vendorarch="${define}"
+#d_vendorscript="${define}"
+
+# Things from base go into /usr
prefixexp='/usr'
+prefix="${prefixexp}"
-# But site binaries go in /usr/local/bin for ports
-siteprefix='/usr/local'
+# But vendor and site files go in /usr/local for ports and cpan
siteprefixexp='/usr/local'
-installsitebin='/usr/local/bin'
+siteprefix="${siteprefixexp}"
+installsitebin='${siteprefix}/bin'
+
+vendorprefixexp='/usr/local'
+vendorprefix="${vendorprefixexp}"
+installvendorbin='${vendorprefix}/bin'
-installarchlib="/usr/libdata/perl5/${archname}"
+installprivlib="${prefix}/libdata/perl5"
+privlib="${installprivlib}"
+privlibexp="${privlib}"
+installarchlib="${installprivlib}/${archname}"
archlib="${installarchlib}"
archlibexp="${archlib}"
test $useshrplib = "true" && ccdlflags="-Wl,-R${installarchlib}/CORE"
-installprivlib="/usr/libdata/perl5"
-privlib="${installprivlib}"
-privlibexp="${privlib}"
-
-installsitearch="/usr/local/libdata/perl5/site_perl/${archname}"
-sitearch="${installsitearch}"
-sitearchexp="${sitearch}"
-installsitelib="/usr/local/libdata/perl5/site_perl"
+# We keep vendor in site_perl so we don't have to bump every perl port
+# otherwise this would be pkg_perl
+# Not versioned here because pkg tools handle that for us.
Add versioned lib to system perl's @INC for non-packaged modules
This patch is should make it easier to recover from problems that happen
when installing perl modules outside of the package system. You might
recognize the "loadable library and perl binaries are mismatched" error.
Adding a versioned subdirectory for cpan modules means that when the
perl version changes it will no longer find those old modules and it
will be easy to remove them without disturbing things from packages.
Only slightly related, but we can avoid breaking base tools like pkg_add
and friend by using "no lib '/usr/local/libdata/perl5/site_perl';" to
avoid looking in those directories at all.
The new @INC looks like this. The first "5.32.1" sitelib is where
external things, like cpan(1), will install. The current "site_perl"
directory (now vendorlib) will continue to be where ports/packages go.
Finally, "/usr/libdata" privlib is for things that ship with the base
system.
$ perl -V | perl -ne 'print if /INC/..eof'
@INC:
/usr/local/libdata/perl5/site_perl/5.32.1/sparc64-openbsd
/usr/local/libdata/perl5/site_perl/5.32.1
/usr/local/libdata/perl5/site_perl/sparc64-openbsd
/usr/local/libdata/perl5/site_perl
/usr/libdata/perl5/sparc64-openbsd
/usr/libdata/perl5
One place this might cause temporary issues in the ports tree is
anything that doesn't use the overrides in perl.port.mk and looks up
sitelib itself. It should be fairly obvious as it should fail to
package because the files in the PLIST will not exist (they'll be in the
versioned subdir). I didn't run into that in my testing, but I didn't
complete a full bulk build.
There do appear to be some annoyances with still shared directories for
man pages, in that if you install a CPAN module and then attempt to
pkg_add it, it complains because "files already exist" in the man
directory. We could separate the cpan man pages, or I think un-setting
the site_lib man*dir will mean cpan won't instal them. I haven't yet
tested that, but I'm not sure what would be best there.
Unfortunately I did do a bunch of testing of different ways to add
things to @INC which meant this patch also does some reorganization of
the contents, I can turn it into a smaller patch if requested and do the
cleanup separately.
Comments, OK? Should I see if sthen@ will do a full bulk build first?
Index: gnu/usr.bin/perl/config.over
===
RCS file: /cvs/src/gnu/usr.bin/perl/config.over,v
retrieving revision 1.22
diff -u -p -r1.22 config.over
--- gnu/usr.bin/perl/config.over5 Feb 2017 00:33:38 - 1.22
+++ gnu/usr.bin/perl/config.over16 May 2021 20:43:16 -
@@ -9,49 +9,85 @@ archname="`arch -s`-${osname}"
myarchname="$archname"
# Use correct paths for a distribution
-prefix='/usr'
+# site is where cpan clients install files
+# vendor is where the ports tree puts packages
+# priv is where the system installs files
+
+# We set up vendor paths for ports/packages
+usevendorprefix="${define}"
+d_vendorbin="${define}"
+d_vendorlib="${define}"
+d_vendorarch="${define}"
+#d_vendorscript="${define}"
+
+# Things from base go into /usr
prefixexp='/usr'
+prefix="${prefixexp}"
-# But site binaries go in /usr/local/bin for ports
-siteprefix='/usr/local'
+# But vendor and site files go in /usr/local for ports and cpan
siteprefixexp='/usr/local'
-installsitebin='/usr/local/bin'
+siteprefix="${siteprefixexp}"
+installsitebin='${siteprefix}/bin'
+
+vendorprefixexp='/usr/local'
+vendorprefix="${vendorprefixexp}"
+installvendorbin='${vendorprefix}/bin'
-installarchlib="/usr/libdata/perl5/${archname}"
+installprivlib="${prefix}/libdata/perl5"
+privlib="${installprivlib}"
+privlibexp="${privlib}"
+installarchlib="${installprivlib}/${archname}"
archlib="${installarchlib}"
archlibexp="${archlib}"
test $useshrplib = "true" && ccdlflags="-Wl,-R${installarchlib}/CORE"
-installprivlib="/usr/libdata/perl5"
-privlib="${installprivlib}"
-privlibexp="${privlib}"
-
-installsitearch="/usr/local/libdata/perl5/site_perl/${archname}"
-sitearch="${installsitearch}"
-sitearchexp="${sitearch}"
-installsitelib="/usr/local/libdata/perl5/site_perl"
+# We keep vendor in site_perl so we don't have to bump every perl port
+# otherwise this would be pkg_perl
+# Not versioned here because pkg tools handle that for us.
+installvendorlib="${vendorprefix}/libdata/perl5/site_perl"
+vendorlib="${installvendorlib}"
+vendorlibexp="${vendorlib}"
+installvendorarch="${installvendorlib}/${archname}"
+vendorarch="${installvendorarch}"
+vendorarchexp="${vendorarch}"
+
+# cpan(1) and similar tools install in a $version dir that's easy
+# to remove after upgrade and won't cause issues for packages when
+# perl versions change.
+installsitelib="${siteprefix}/libdata/perl5/site_perl/${version}"
sitelib="${installsitelib}"
sitelibexp="${sitelib}"
+installsitearch="${installsitelib}/${archname}"
+sitearch="${installsitearch}"
+sitearchexp="${sitearch}"
installstyle="${privlib}"
-# We
