On 31/03/2012 20:06, Marvin Humphrey wrote:
As previously discussed on the mailing list, we should also provide parts of
the code in buildlib/Lucy/Build.pm as a Clownfish module, so extensions can
be built easily.
So the idea here is that we need a class to help extension authors do stuff
like install their .cfh header files into $installsitearch/Clownfish/_include
because we don't want them to have to write low-level code to do that
themselves.
Yes, and helping extension authors to build their extension in the first
place.
Perhaps it would be helpful to see a hacked-up, monotlithic extension build
script that does all the necessary work to build and install a Lucy C
extension, no matter how dirty the implementation. Then we can create a
helper class and beef it up bit by bit while whittling down the build script
until things look reasonable.
Attached is a modified version of Lucy::Build that I use for testing.
Installation is not yet supported. There are only minor changes to the
code I copied from Lucy::Build, so Lucy::Build could also subclass the
new Clownfish build class.
To handle different build configuration variables like
module name, additional C sources, etc., I would propose to create a
BuildConfig class that provides all these variables via methods. Then, every
extensions can optionally extend the class and create a BuildConfig object.
This object can be stored as a "property" of the Module::Build object.
Build configuration includes:
* Main class name
* Parcel
* Additional C sources, include dirs and flags
* Additional libraries and linker flags
* Clownfish include directories
* Other files to install in Clownfish/_include
This BuildConfig class is separate from the build class, right? And the idea
is that they won't have to subclass the build class, because specifying
properties in the BuildConfig class will suffice for all customization needs?
We could add all the Clownfish build parameters as additional
Module::Build properties (see
https://metacpan.org/module/Module::Build::API#add_property). My idea is
that it might be more maintainable to only add a single property that
points to a separate config object. Then we can add new parameters
without touching Module::Build internals, for example.
So an extension's Build.PL would look something like this:
my $cf_build_config = Clownfish::CFC::BuildConfig->new(
parcel => 'MyParcel',
c_sources => 'additional/c/sources',
install_files => {
'core/My/Header.h' => 'My/Header.h',
}
...
);
my $builder = Clownfish::CFC::Build->new(
module_name => 'LucyX::My::Extension',
dist_abstract => 'My Lucy extension',
...
clownfish_config => $cf_build_config,
);
$builder->create_build_script;
Open questions:
* How to name the build module? Clownfish::CFC::Build?
Presumably we're going to need analogous functionality for other languages --
but the implementation of e.g. our Ruby-specific extension build helper is
going to be very different from this one. Even if Clownfish::CFC::Build's
Perl-specific functionality is written entirely in Perl and it's Ruby-specific
functionality is written entirely in Ruby, it sounds undesirable to have
radically divergent classes under the same name.
How about Clownfish::CFC::Perl::Build instead?
+1
* We might need a better way to find the Binding build packages.
Can you please elaborate? I'm not grokking...
Nevermind. I misunderstood that part of the code in Lucy::Build.
Nick
use strict;
use warnings;
package LucyX::Build;
use base qw( Module::Build );
our $VERSION = 0.003000;
$VERSION = eval $VERSION;
use Clownfish::CFC::Binding::Core;
use Clownfish::CFC::Binding::Perl;
use Clownfish::CFC::Binding::Perl::Class;
use Clownfish::CFC::Model::Hierarchy;
use ExtUtils::CBuilder;
use File::Spec::Functions
qw( catdir catfile splitpath updir no_upwards abs2rel rel2abs );
use File::Path qw( mkpath rmtree );
use Config;
use Carp;
use Cwd qw( getcwd );
sub extra_ccflags {
my $self = shift;
my $gcc_flags = '-std=gnu99 -D_GNU_SOURCE ';
if ( $self->config('gccversion') ) {
return $gcc_flags;
}
elsif ( $self->config('cc') =~ /^cl\b/ ) {
# Compile as C++ under MSVC.
return '/TP -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_WARNINGS ';
}
}
my $is_distro_not_devel = -e 'core';
my $base_dir = rel2abs( $is_distro_not_devel ? getcwd() : updir() );
my $CLASS_FULL_NAME = 'LucyX::My::Extension';
my @CLASS_COMPONENTS = split('::', $CLASS_FULL_NAME);
my @CLASS_DIR = @CLASS_COMPONENTS;
my $CLASS_NAME = pop(@CLASS_DIR);
my $CORE_SOURCE_DIR = catdir( $base_dir, 'core' );
my $AUTOGEN_DIR = 'autogen';
my $XS_SOURCE_DIR = 'xs';
my $LIB_DIR = 'lib';
my $BUILDLIB_DIR = 'buildlib';
my $XS_FILEPATH = catfile( $LIB_DIR, @CLASS_DIR, "$CLASS_NAME.xs" );
# TODO: Get these from $hierarchy
my $AUTOGEN_INC_DIR = catfile( 'autogen', 'include' );
my $AUTOGEN_SOURCE_DIR = catfile( 'autogen', 'source' );
sub _clownfish_inc_dirs {
my $self = shift;
my @include_dirs;
for my $location (qw(site vendor)) {
my $install_dir = $Config{"install${location}arch"};
my $include_dir = catdir( $install_dir, 'Clownfish', '_include' );
next unless -d $include_dir;
push(@include_dirs, $include_dir);
}
return @include_dirs;
}
sub _compile_clownfish {
my $self = shift;
# Compile Clownfish.
my $hierarchy = Clownfish::CFC::Model::Hierarchy->new(
dest => $AUTOGEN_DIR,
);
$hierarchy->add_source_dir($CORE_SOURCE_DIR);
for my $include_dir ($self->_clownfish_inc_dirs) {
$hierarchy->add_include_dir($include_dir);
}
$hierarchy->build;
# Process all __BINDING__ blocks.
my $pm_filepaths = $self->rscan_dir( $BUILDLIB_DIR, qr/\.pm$/ );
for my $pm_filepath (@$pm_filepaths) {
next unless $pm_filepath =~ /Binding/;
require $pm_filepath;
my $package_name = $pm_filepath;
$package_name =~ s/buildlib\/(Lucy.*)\.pm$/$1/;
$package_name =~ s/\//::/g;
$package_name->bind_all;
}
my $binding = Clownfish::CFC::Binding::Perl->new(
parcel => $CLASS_NAME,
hierarchy => $hierarchy,
lib_dir => $LIB_DIR,
boot_class => $CLASS_FULL_NAME,
header => $self->autogen_header,
footer => '',
);
return ( $hierarchy, $binding );
}
sub ACTION_pod {
my $self = shift;
$self->dispatch("cfc");
$self->_write_pod(@_);
}
sub _write_pod {
my ( $self, $binding ) = @_;
if ( !$binding ) {
( undef, $binding ) = $self->_compile_clownfish;
}
print "Writing POD...\n";
my $pod_files = $binding->write_pod;
$self->add_to_cleanup($_) for @$pod_files;
}
sub ACTION_clownfish {
my $self = shift;
$self->add_to_cleanup($AUTOGEN_DIR);
my $buildlib_pm_filepaths = $self->rscan_dir( $BUILDLIB_DIR, qr/\.pm$/ );
my $cfh_filepaths = $self->rscan_dir( $CORE_SOURCE_DIR, qr/\.cfh$/ );
# XXX joes thinks this is dubious
# Don't bother parsing Clownfish files if everything's up to date.
return
if $self->up_to_date(
[ @$cfh_filepaths, @$buildlib_pm_filepaths ],
[ $XS_FILEPATH, $AUTOGEN_DIR, ]
);
# Write out all autogenerated files.
print "Parsing Clownfish files...\n";
my ( $hierarchy, $perl_binding ) = $self->_compile_clownfish;
my $core_binding = Clownfish::CFC::Binding::Core->new(
hierarchy => $hierarchy,
header => $self->autogen_header,
footer => '',
);
print "Writing Clownfish autogenerated files...\n";
my $modified = $core_binding->write_all_modified;
if ($modified) {
unlink('typemap');
print "Writing typemap...\n";
$self->add_to_cleanup('typemap');
$perl_binding->write_xs_typemap;
}
# Rewrite XS if either any .cfh files or relevant .pm files were modified.
$modified ||=
$self->up_to_date( \@$buildlib_pm_filepaths, $XS_FILEPATH )
? 0
: 1;
if ($modified) {
$self->add_to_cleanup($XS_FILEPATH);
$perl_binding->write_boot;
$perl_binding->write_bindings;
$self->_write_pod($perl_binding);
# Copy .cfh files to blib/arch/Clownfish/_include
my $inc_dir = catdir( $self->blib, 'arch', 'Clownfish', '_include' );
for my $file (@$cfh_filepaths) {
my $rel = abs2rel( $file, $CORE_SOURCE_DIR );
my $dest = catfile( $inc_dir, $rel );
$self->copy_if_modified( from => $file, to => $dest, );
}
}
# Touch autogenerated files in case the modifications were inconsequential
# and didn't trigger a rewrite, so that we won't have to check them again
# next pass.
if (!$self->up_to_date(
[ @$cfh_filepaths, @$buildlib_pm_filepaths ], $XS_FILEPATH
)
)
{
utime( time, time, $XS_FILEPATH ); # touch
}
if (!$self->up_to_date(
[ @$cfh_filepaths, @$buildlib_pm_filepaths ], $AUTOGEN_DIR
)
)
{
utime( time, time, $AUTOGEN_DIR ); # touch
}
}
# Write ppport.h, which supplies some XS routines not found in older Perls and
# allows us to use more up-to-date XS API while still supporting Perls back to
# 5.8.3.
#
# The Devel::PPPort docs recommend that we distribute ppport.h rather than
# require Devel::PPPort itself, but ppport.h isn't compatible with the Apache
# license.
sub ACTION_ppport {
my $self = shift;
if ( !-e 'ppport.h' ) {
require Devel::PPPort;
$self->add_to_cleanup('ppport.h');
Devel::PPPort::WriteFile();
}
}
sub ACTION_compile_custom_xs {
my $self = shift;
$self->dispatch('ppport');
require ExtUtils::ParseXS;
my $cbuilder
= ExtUtils::CBuilder->new( config => { cc => $self->config('cc') },
);
my $archdir = catdir( $self->blib, 'arch', 'auto', @CLASS_COMPONENTS );
mkpath( $archdir, 0, 0777 ) unless -d $archdir;
my @include_dirs = (
getcwd(), $CORE_SOURCE_DIR, $AUTOGEN_INC_DIR, $XS_SOURCE_DIR,
$self->_clownfish_inc_dirs,
);
my @objects;
# Compile C source files.
my $c_files = [];
push @$c_files, @{ $self->rscan_dir( $CORE_SOURCE_DIR, qr/\.c$/ ) };
push @$c_files, @{ $self->rscan_dir( $XS_SOURCE_DIR, qr/\.c$/ ) };
push @$c_files, @{ $self->rscan_dir( $AUTOGEN_SOURCE_DIR, qr/\.c$/ ) };
for my $c_file (@$c_files) {
my $o_file = $c_file;
my $ccs_file = $c_file;
$o_file =~ s/\.c$/$Config{_o}/ or die "no match";
$ccs_file =~ s/\.c$/.ccs/ or die "no match";
push @objects, $o_file;
next if $self->up_to_date( $c_file, $o_file );
$self->add_to_cleanup($o_file);
$self->add_to_cleanup($ccs_file);
$cbuilder->compile(
source => $c_file,
extra_compiler_flags => $self->extra_ccflags,
include_dirs => \@include_dirs,
object_file => $o_file,
);
}
# .xs => .c
my $perl_binding_c_file = catfile( $LIB_DIR, @CLASS_DIR, "$CLASS_NAME.c" );
$self->add_to_cleanup($perl_binding_c_file);
if ( !$self->up_to_date( $XS_FILEPATH, $perl_binding_c_file ) ) {
ExtUtils::ParseXS::process_file(
filename => $XS_FILEPATH,
prototypes => 0,
output => $perl_binding_c_file,
);
}
# .c => .o
my $pm_file = catfile( $LIB_DIR, @CLASS_DIR, "$CLASS_NAME.pm" );
open( my $pm_fh, '<', $pm_file )
or confess "Can't open '$pm_file': $!";
my $pm_contents = do { local $/; <$pm_fh> };
close $pm_fh or confess $!;
$pm_contents =~ /^\s*our \$VERSION = '([\d.]+)';/m
or confess "Can't extract version number from '$pm_file'";
my $version = $1;
my $perl_binding_o_file = catfile( $LIB_DIR, @CLASS_DIR,
"$CLASS_NAME$Config{_o}" );
unshift @objects, $perl_binding_o_file;
$self->add_to_cleanup($perl_binding_o_file);
if ( !$self->up_to_date( $perl_binding_c_file, $perl_binding_o_file ) ) {
$cbuilder->compile(
source => $perl_binding_c_file,
extra_compiler_flags => $self->extra_ccflags,
include_dirs => \@include_dirs,
object_file => $perl_binding_o_file,
# 'defines' is an undocumented parameter to compile(), so we
# should officially roll our own variant and generate compiler
# flags. However, that involves writing a bunch of
# platform-dependent code, so we'll just take the chance that this
# will break.
defines => {
VERSION => qq|"$version"|,
XS_VERSION => qq|"$version"|,
},
);
}
# Create .bs bootstrap file, needed by Dynaloader.
my $bs_file = catfile( $archdir, "$CLASS_NAME.bs" );
$self->add_to_cleanup($bs_file);
if ( !$self->up_to_date( $perl_binding_o_file, $bs_file ) ) {
require ExtUtils::Mkbootstrap;
ExtUtils::Mkbootstrap::Mkbootstrap($bs_file);
if ( !-f $bs_file ) {
# Create file in case Mkbootstrap didn't do anything.
open( my $fh, '>', $bs_file )
or confess "Can't open $bs_file: $!";
}
utime( (time) x 2, $bs_file ); # touch
}
# Clean up after CBuilder under MSVC.
$self->add_to_cleanup('compilet*');
$self->add_to_cleanup('*.ccs');
$self->add_to_cleanup( catfile( 'lib', @CLASS_DIR, "$CLASS_NAME.ccs" ) );
$self->add_to_cleanup( catfile( 'lib', @CLASS_DIR, "$CLASS_NAME.def" ) );
$self->add_to_cleanup( catfile( 'lib', @CLASS_DIR, "${CLASS_NAME}_def.old"
) );
$self->add_to_cleanup( catfile( 'lib', @CLASS_DIR, "$CLASS_NAME.exp" ) );
$self->add_to_cleanup( catfile( 'lib', @CLASS_DIR, "$CLASS_NAME.lib" ) );
$self->add_to_cleanup( catfile( 'lib', @CLASS_DIR, "$CLASS_NAME.lds" ) );
$self->add_to_cleanup( catfile( 'lib', @CLASS_DIR, "$CLASS_NAME.base" ) );
# .o => .(a|bundle)
my $lib_file = catfile( $archdir, "$CLASS_NAME.$Config{dlext}" );
if ( !$self->up_to_date( [ @objects, $AUTOGEN_DIR ], $lib_file ) ) {
# TODO: use Charmonizer to determine whether pthreads are userland.
my $link_flags = '';
if ( $Config{osname} =~ /openbsd/i && $Config{usethreads} ) {
$link_flags = '-lpthread ';
}
my $lucy_lib_file = catfile(
$self->install_sets( 'site', 'arch' ), 'auto',
'Lucy', "Lucy.$Config{dlext}",
);
$link_flags .= "$lucy_lib_file ";
$cbuilder->link(
module_name => $CLASS_FULL_NAME,
objects => \@objects,
lib_file => $lib_file,
extra_linker_flags => $link_flags,
);
}
}
sub ACTION_code {
my $self = shift;
$self->dispatch('clownfish');
$self->dispatch('compile_custom_xs');
$self->SUPER::ACTION_code;
}
sub autogen_header {
my $self = shift;
return <<"END_AUTOGEN";
/***********************************************
!!!! DO NOT EDIT !!!!
This file was auto-generated by Build.PL.
***********************************************/
END_AUTOGEN
}
1;
