extensions/known_extensions and insanity (was Re: VMS, configure.com and nonxs_ext)
On Tue, Sep 08, 2009 at 10:33:02AM +0100, Nicholas Clark wrote:
On Mon, Sep 07, 2009 at 05:15:51PM -0500, Craig A. Berry wrote:
There's also dynamic_ext, right? Shouldn't dynamic_ext plus
nonxs_extensions equal known_extensions?
Well, probably it should, but what actually happens is:
$ cat extensions.pl
#!perl -w
use strict;
use Config;
printf %-24s, 'extensions';
my @types = qw(static_ext dynamic_ext nonxs_ext known_extensions);
my %lookup;
my %short;
foreach (@types) {
my $short = $_;
$short =~ s/_.*//;
$short{$_} = $short;
printf %-8s, $short;
$lookup{$_} = {map {$_, 1} split/ /, $Config{$_}};
}
print \n\n;
foreach my $ext (sort split / /, $Config{extensions}) {
printf %-24s, $ext;
foreach (@types) {
printf %-8s, $lookup{$_}{$ext} ? $short{$_} : '';
}
print \n;
}
__END__
extensions is everything.
known_extensions = static_ext + dynamic_ext
extensions = nonxs_ext + known_extensions
which isn't logical, but I suspect is due to something historical.
Oops. I'm not correct.
I *think I am here:
0: DynaLoader doesn't count as an extension
1: known_extensions are all known XS extensions
2: static_ext, dynamic_ext and nonxs_ext are actually what they say they are,
assuming you were thinking about extensions we have configured here
3: extensions is the sum of the 3 immediately above
Hence if you have not built some XS extensions on your platform (which, I
think, is always true, as you can't build both VMS and Win32 extensions
anywhere), there will always be something in known_extensions that is not
in extensions.
Likewise, if you *have* build something nonxs (which you have do, as you can't
build DynaLoader without (at least) ExtUtils::ParseXS), there will always be
something in extensions that isn't in known_extensions.
Good, eh?
As an encore, the Unix Configure script teases you with -Dnoextensions
to disable extensions, but only actually honours it for XS extensions.
Right know it assumes that all nonxs extensions will work everywhere.
Nicholas Clark
Re: extensions/known_extensions and insanity (was Re: VMS, configure.com and nonxs_ext)
On Sep 27, 2009, at 1:26 PM, Nicholas Clark wrote: I *think I am here: 0: DynaLoader doesn't count as an extension 1: known_extensions are all known XS extensions 2: static_ext, dynamic_ext and nonxs_ext are actually what they say they are, assuming you were thinking about extensions we have configured here 3: extensions is the sum of the 3 immediately above I think we are now accurately imitating this on VMS except that DynaLoader is not currently listed in extensions. Hence if you have not built some XS extensions on your platform (which, I think, is always true, as you can't build both VMS and Win32 extensions anywhere), there will always be something in known_extensions that is not in extensions. Likewise, if you *have* build something nonxs (which you have do, as you can't build DynaLoader without (at least) ExtUtils::ParseXS), there will always be something in extensions that isn't in known_extensions. Good, eh? As an encore, the Unix Configure script teases you with -Dnoextensions to disable extensions, but only actually honours it for XS extensions. Right know it assumes that all nonxs extensions will work everywhere. Most likely that's because once upon a time, saying XS extension was redundant and saying non-XS extension was a non-sequitur. And while we might expect Camel 3 and Extending and Embedding Perl to be out-of- date when they exclusively use the word extension to talk about extending Perl with C code, the current discussions in perlxs and perlxstut do the same thing. perlfaq7 gets even more explicit and says, =head2 What's an extension? An extension is a way of calling compiled C code from Perl. Reading Lperlxstut is a good place to learn more about extensions. With that definition of what an extension is, choosing -Dnoextensions to skip just the XS modules makes perfect sense. So I think one of the places where we went wrong was in introducing nonxs_ext for pure-Perl modules that are not properly called extensions. Maybe it should have been built_modules or built_pm or similar, and extensions would have been modules (though further distinctions between built modules and included modules might be necessary). This may be of purely hysterical interest as changing %Config entries now would probably cause more trouble than it would avoid, but hopefully it helps explain why we have some of the oddities that we do. Craig A. Berry mailto:craigbe...@mac.com ... getting out of a sonnet is much more difficult than getting in. Brad Leithauser
Re: extensions/known_extensions and insanity (was Re: VMS, configure.com and nonxs_ext)
On Sun, Sep 27, 2009 at 03:04:38PM -0500, Craig A. Berry wrote: On Sep 27, 2009, at 1:26 PM, Nicholas Clark wrote: I *think I am here: 0: DynaLoader doesn't count as an extension 1: known_extensions are all known XS extensions 2: static_ext, dynamic_ext and nonxs_ext are actually what they say they are, assuming you were thinking about extensions we have configured here 3: extensions is the sum of the 3 immediately above I think we are now accurately imitating this on VMS except that DynaLoader is not currently listed in extensions. No, that's quite correct: $ grep Dyna config.sh $ As an encore, the Unix Configure script teases you with -Dnoextensions to disable extensions, but only actually honours it for XS extensions. Right know it assumes that all nonxs extensions will work everywhere. Most likely that's because once upon a time, saying XS extension was redundant and saying non-XS extension was a non-sequitur. And while we might expect Camel 3 and Extending and Embedding Perl to be out-of- date when they exclusively use the word extension to talk about extending Perl with C code, the current discussions in perlxs and perlxstut do the same thing. perlfaq7 gets even more explicit and says, =head2 What's an extension? An extension is a way of calling compiled C code from Perl. Reading Lperlxstut is a good place to learn more about extensions. Thanks for that analysis. I wasn't aware of that. With that definition of what an extension is, choosing -Dnoextensions to skip just the XS modules makes perfect sense. So I think one of the places where we went wrong was in introducing nonxs_ext for pure-Perl modules that are not properly called extensions. Maybe it should have been built_modules or built_pm or similar, and extensions would have been modules (though further distinctions between built modules and included modules might be necessary). This may be of purely hysterical interest as changing %Config entries now would probably cause more trouble than it would avoid, but hopefully it helps explain why we have some of the oddities that we do. Yes, that's pretty much my view now. We can also add new ones, or a whole new scheme, but really we need ot think it through carefully, and assume we have exactly this chance to get it right. Forever. Or until the special biologist word catches up with us. Nicholas Clark
Re: VMS, configure.com and nonxs_ext
On Mon, Sep 07, 2009 at 05:15:51PM -0500, Craig A. Berry wrote:
On Sep 7, 2009, at 4:46 PM, Nicholas Clark wrote:
On Mon, Sep 07, 2009 at 08:55:29PM +0100, Nicholas Clark wrote:
This might be rather a big favour request.
This might be easier than I first thought
It occurred to me we ought to revisit this after getting vms/ext into
ext. I don't think it will be huge, but it does involve grokking some
DCL that has to work rather hard to do what it's doing. I'll take a
look.
Would it be possible to augment the configure.com code that finds
extensions
to correctly partition nonxs_extensions from known_extensions?
(being the rather bonkers way that Configure partitions XS and non-XS
extensions)
There's also dynamic_ext, right? Shouldn't dynamic_ext plus
nonxs_extensions equal known_extensions?
Well, probably it should, but what actually happens is:
$ cat extensions.pl
#!perl -w
use strict;
use Config;
printf %-24s, 'extensions';
my @types = qw(static_ext dynamic_ext nonxs_ext known_extensions);
my %lookup;
my %short;
foreach (@types) {
my $short = $_;
$short =~ s/_.*//;
$short{$_} = $short;
printf %-8s, $short;
$lookup{$_} = {map {$_, 1} split/ /, $Config{$_}};
}
print \n\n;
foreach my $ext (sort split / /, $Config{extensions}) {
printf %-24s, $ext;
foreach (@types) {
printf %-8s, $lookup{$_}{$ext} ? $short{$_} : '';
}
print \n;
}
__END__
$ ./perl -Ilib extensions.pl
extensions static dynamic nonxs known
Archive/Extract nonxs
Archive/Tar nonxs
Attribute/Handlers nonxs
B dynamic known
B/Debug nonxs
B/Deparse nonxs
B/Lint nonxs
CGI nonxs
CPANPLUSnonxs
CPANPLUS/Dist/Build nonxs
Compress/Raw/Bzip2 dynamic known
Compress/Raw/Zlib dynamic known
Cwd dynamic known
DB_File dynamic known
Data/Dumper dynamic known
Devel/DProf dynamic known
Devel/PPPortdynamic known
Devel/Peek dynamic known
Devel/SelfStubber nonxs
Digest nonxs
Digest/MD5 dynamic known
Digest/SHA dynamic known
Encode dynamic known
Errno nonxs
Fcntl dynamic known
File/Fetch nonxs
File/Glob dynamic known
FileCache nonxs
Filter/Simple nonxs
Filter/Util/Calldynamic known
Hash/Util dynamic known
Hash/Util/FieldHash dynamic known
I18N/LangTags nonxs
I18N/Langinfo dynamic known
IO dynamic known
IO/Compress dynamic known
IO/Zlib nonxs
IPC/Cmd nonxs
IPC/Open2 nonxs
IPC/Open3 nonxs
IPC/SysVdynamic known
List/Util dynamic known
Log/Message nonxs
Log/Message/Simple nonxs
MIME/Base64 dynamic known
Math/BigInt/FastCalcdynamic known
Module/CoreList nonxs
Module/Load nonxs
Module/Load/Conditional nonxs
Module/Loaded nonxs
Module/Pluggablenonxs
Net/Pingnonxs
Object/Accessor nonxs
Opcode dynamic known
POSIX dynamic known
Package/Constants nonxs
Params/Checknonxs
Parse/CPAN/Meta nonxs
PerlIO/encoding dynamic known
PerlIO/scalar dynamic known
PerlIO/via dynamic
Re: VMS, configure.com and nonxs_ext
On Mon, Sep 07, 2009 at 08:55:29PM +0100, Nicholas Clark wrote: (I'm not sure why the Configure shell code contains tests for .c too - historical reasons? No, recent reasons. Building Win32CORE on Cygwin: http://perl5.git.perl.org/perl.git/commit/fefb2940d28bc57fa38371e2 Nicholas Clark
Re: VMS, configure.com and nonxs_ext
On Sep 8, 2009, at 4:33 AM, Nicholas Clark wrote: On Mon, Sep 07, 2009 at 05:15:51PM -0500, Craig A. Berry wrote: There's also dynamic_ext, right? Shouldn't dynamic_ext plus nonxs_extensions equal known_extensions? Well, probably it should, but what actually happens is: extensions is everything. known_extensions = static_ext + dynamic_ext extensions = nonxs_ext + known_extensions which isn't logical, but I suspect is due to something historical. I think that's about what I sorted out by reading Porting/Glossary, which does say that known_extensions only includes the XS kind (which is definitely not what we were doing on VMS before). As you've likely noticed, I think it's done: http://perl5.git.perl.org/perl.git/commitdiff/d1ce36d Hmm. Actually, we're not including nonxs_ext in extensions but that should be easy to fix and may not have any impact on the build. Craig A. Berry mailto:craigbe...@mac.com ... getting out of a sonnet is much more difficult than getting in. Brad Leithauser
Re: VMS, configure.com and nonxs_ext
On Sep 8, 2009, at 4:34 AM, Nicholas Clark wrote: On Mon, Sep 07, 2009 at 08:55:29PM +0100, Nicholas Clark wrote: (I'm not sure why the Configure shell code contains tests for .c too - historical reasons? No, recent reasons. Building Win32CORE on Cygwin: http://perl5.git.perl.org/perl.git/commit/fefb2940d28bc57fa38371e2 I thought I had seen something about it recently. I could make our extension hunter look for .c as well, bu what I ended up doing was pulling Win32CORE out of nonxs_ext explicitly because it won't run on VMS anyway. Craig A. Berry mailto:craigbe...@mac.com ... getting out of a sonnet is much more difficult than getting in. Brad Leithauser
Re: VMS, configure.com and nonxs_ext
On Tue, 08 Sep 2009 06:47:38 -0500, Craig A. Berry craigbe...@mac.com wrote: On Sep 8, 2009, at 4:33 AM, Nicholas Clark wrote: On Mon, Sep 07, 2009 at 05:15:51PM -0500, Craig A. Berry wrote: There's also dynamic_ext, right? Shouldn't dynamic_ext plus nonxs_extensions equal known_extensions? Well, probably it should, but what actually happens is: extensions is everything. known_extensions = static_ext + dynamic_ext extensions = nonxs_ext + known_extensions which isn't logical, but I suspect is due to something historical. I think that's about what I sorted out by reading Porting/Glossary, Porting/Glossary is generated from metaunits. If the comment is outdated (and the comments are not visible outside the metaunits), the Glossary will contain bad comments. The Glossary is not a bible (yet). which does say that known_extensions only includes the XS kind (which is definitely not what we were doing on VMS before). As you've likely noticed, I think it's done: http://perl5.git.perl.org/perl.git/commitdiff/d1ce36d Hmm. Actually, we're not including nonxs_ext in extensions but that should be easy to fix and may not have any impact on the build. -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00, 11.11, 11.23, and 11.31, OpenSuSE 10.3, 11.0, and 11.1, AIX 5.2 and 5.3. http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
Re: VMS, configure.com and nonxs_ext
On Mon, 7 Sep 2009, Nicholas Clark wrote: On Mon, Sep 07, 2009 at 08:55:29PM +0100, Nicholas Clark wrote: This might be rather a big favour request. This might be easier than I first thought Would it be possible to augment the configure.com code that finds extensions to correctly partition nonxs_extensions from known_extensions? (being the rather bonkers way that Configure partitions XS and non-XS extensions) Coming late to this discussion . . . The bonkers way that Configure works now is indeed a hodge-podge growth representing the way stuff grew organically. I've no time right now, but will try to look Tuesday afternoon (US, Eastern time) and see what's up here. Unless you've already solved the problem by then! -- Andy Dougherty dough...@lafayette.edu
Re: VMS, configure.com and nonxs_ext
On Sep 7, 2009, at 9:42 PM, Andrew Dougherty wrote: On Mon, 7 Sep 2009, Nicholas Clark wrote: Would it be possible to augment the configure.com code that finds extensions to correctly partition nonxs_extensions from known_extensions? (being the rather bonkers way that Configure partitions XS and non- XS extensions) Coming late to this discussion . . . The bonkers way that Configure works now is indeed a hodge-podge growth representing the way stuff grew organically. I've no time right now, but will try to look Tuesday afternoon (US, Eastern time) and see what's up here. Unless you've already solved the problem by then! Thanks. I'm pretty sure, if doing exactly the same thing in configure.com that's done in Configure is a solution (or at least consistently bonkers), we have solved it as of: http://perl5.git.perl.org/perl.git/commitdiff/d1ce36d and http://perl5.git.perl.org/perl.git/commitdiff/63d5291 On VMS we were not populating nonxs_ext at all, and we were treating dynamic_ext and extensions as the same thing and including everything in them. I think that was just a side effect of the organic growth you referred to plus my inattention during certain parts of it. Part of the evolution probably has to do with nomenclature. I'm quite sure I've been told at some point in the distant past that it's not proper to call a module an extension unless it has XS code in it. If that was ever a valid or consistent convention, the current mass migration from lib/ to ext/ buries it pretty deep. Using the extensions.pl Nicholas posted elsewhere in this thread, I now see: $ perl extensions.pl extensions static dynamic nonxs known Archive/Extract nonxs Archive/Tar nonxs Attribute/Handlers nonxs B dynamic known B/Debug nonxs B/Deparse nonxs B/Lint nonxs CGI nonxs CPANPLUSnonxs CPANPLUS/Dist/Build nonxs Compress/Raw/Bzip2 dynamic known Compress/Raw/Zlib dynamic known Cwd dynamic known Data/Dumper dynamic known Devel/DProf dynamic known Devel/PPPortdynamic known Devel/Peek dynamic known Devel/SelfStubber nonxs Digest nonxs Digest/MD5 dynamic known Digest/SHA dynamic known Encode dynamic known Errno nonxs Fcntl dynamic known File/Fetch nonxs File/Glob dynamic known FileCache nonxs Filter/Simple nonxs Filter/Util/Calldynamic known Hash/Util dynamic known Hash/Util/FieldHash dynamic known I18N/LangTags nonxs I18N/Langinfo dynamic known IO dynamic known IO/Compress nonxs IO/Zlib nonxs IPC/Cmd nonxs IPC/Open2 nonxs IPC/Open3 nonxs List/Util dynamic known Log/Message nonxs Log/Message/Simple nonxs MIME/Base64 dynamic known Math/BigInt/FastCalcdynamic known Module/CoreList nonxs Module/Load nonxs Module/Load/Conditional nonxs Module/Loaded nonxs Module/Pluggablenonxs Net/Pingnonxs Object/Accessor nonxs Opcode dynamic known POSIX dynamic known Package/Constants nonxs Params/Checknonxs Parse/CPAN/Meta nonxs PerlIO/encoding dynamic known PerlIO/scalar dynamic known PerlIO/via dynamic known Pod/Plainer nonxs SDBM_File dynamic known Safenonxs SelfLoader nonxs Shell nonxs Socket dynamic known Storabledynamic known Switch
Re: VMS, configure.com and nonxs_ext
On Mon, Sep 07, 2009 at 08:55:29PM +0100, Nicholas Clark wrote: This might be rather a big favour request. This might be easier than I first thought Would it be possible to augment the configure.com code that finds extensions to correctly partition nonxs_extensions from known_extensions? (being the rather bonkers way that Configure partitions XS and non-XS extensions) To properly split out dual life modules into ext, I think it's going to be necessary to add a Makefile.SH/Makefile/makefile.mk/descrip_mms.template rule to have mkppport run after the nonxs_ext are built, but in turn, it needs to run before dynamic_ext are built, hence the need for a split. I'm not sure how easy this is, because right now Configure does it this way: using file globbing in ext/, whereas configure.com does it by text processing MANIFEST: If it were possible to make the above into a function, then partitioning nonxs from XS would seem to be as simple as Feed the function the equivalent of `grep \.xs MANIFEST` to find the XS extensions I think that every XS extension has exactly one file named *.xs in it. Try running perl -nlwe 'print $1 if m!^(ext/[^\/]+/\S+\.xs\b)!' MANIFEST exclude from the output ext/DynaLoader (which is already a special case in configure.com) and lines matching 'fallback/constants.xs' and one has the list of all XS extensions. I hope that this algorithm is easy to encode in DCL. Feed the function MANIFEST to find all extensions Iterate over the list of all extensions - everything that isn't an XS extension, you put into nonxs_extensions At which point the existing code in configure.com can be changed around $ may_already_have_extension: to exclude any new extensions that are already in the list of XS extension generated above, and tweak it to replace $ known_extensions = known_extensions + ''extspec' with $ nonxs_extensions = nonxs_extensions + ''extspec' Nicholas Clark
Re: VMS, configure.com and nonxs_ext
On Sep 7, 2009, at 4:46 PM, Nicholas Clark wrote: On Mon, Sep 07, 2009 at 08:55:29PM +0100, Nicholas Clark wrote: This might be rather a big favour request. This might be easier than I first thought It occurred to me we ought to revisit this after getting vms/ext into ext. I don't think it will be huge, but it does involve grokking some DCL that has to work rather hard to do what it's doing. I'll take a look. Would it be possible to augment the configure.com code that finds extensions to correctly partition nonxs_extensions from known_extensions? (being the rather bonkers way that Configure partitions XS and non-XS extensions) There's also dynamic_ext, right? Shouldn't dynamic_ext plus nonxs_extensions equal known_extensions? I think that every XS extension has exactly one file named *.xs in it. Try running perl -nlwe 'print $1 if m!^(ext/[^\/]+/\S+\.xs\b)!' MANIFEST exclude from the output ext/DynaLoader (which is already a special case in configure.com) and lines matching 'fallback/constants.xs' and one has the list of all XS extensions. I hope that this algorithm is easy to encode in DCL. Well, I think what we would do is something like $ if f$search([.ext.''xxx']*.xs) .nes. $ then $dynamic_ext = dynamic_ext + ''xxx' $ else $nonxs_extensions = nonxs_extentions + ''xxx' $ endif If xxx is Foo-Bar, that checks the filesystem for the presence of ext/Foo-Bar/*.xs. This would have to be done before xxx has gone through the replace_dash_with_slash logic, or xxx could possibly be a saved dash-containing version that we drag around for this purpose. Craig A. Berry mailto:craigbe...@mac.com ... getting out of a sonnet is much more difficult than getting in. Brad Leithauser
