Greets,
If you're not interested in Lucy's Perl bindings, you can skip this
post...
The most powerful, flexible way to glue C and Perl together is a
suite of C macros and code-generation tools collectively known as
"XS". Another popular option is Inline::C, which allows you to embed
C code directly within perl code. (Peeking underneath the hood
reveals that Inline::C is actually an XS code generator). You can
also use SWIG, but that's not common.
Ordinarily, XS resides in dedicated .xs files. There is no one place
that these files are "supposed" to go -- while Perl modules
themselves typically end up in $DIST_ROOT/lib, XS files might end up
in $DIST_ROOT, or in $DIST_ROOT/xs, or alongside their companion Perl
modules in $DIST_ROOT/lib, or somewhere else.
KinoSearch does something a little bit unconventional with regards to
its XS code: it's stored Inline::C-style, within the Perl modules
after an __END__ token, delimited by __XS__ and __POD__ tokens.
(Example at <http://xrl.us/vgri>.) At build time, the XS code is
extracted and concatenated into a single KinoSearch.xs file, which is
subsequently compiled using standard Perl building tools.
There are two main reasons for doing things this way.
First, having XS code right next to the perl code it services is a
real boon. It's easier to move subroutine implementations from one
space to the other, as there are 3 files (.c, .h and .pm) to deal
with instead of 4 (with the addition of the .xs file). The XS and
Perl are often tightly coupled; for example, the default values for
an XS subroutine might be managed via a hash created in Perl-space.
And even when there is no coupling, it is useful to have all the
subroutines available from a given perl package concentrated in one
place.
Second, all that XS code really does belong in one giant file. It
would be highly annoying if KS adopted a 1:1 pairing of Perl modules
to XS files, because XS takes a lllloooonnngggg time to compile. I
tried breaking things up into 6 packages a while back but quickly
reverted because of exploding compilation times. However, at 4400
lines and growing, a monolithic KinoSearch.xs file is cumbersome to
edit.
The downside of inlining the XS code is that the style is a bit
unusual: an XS hacker doesn't automatically know what to make of
a .pm file like that. I'm not sure that's a big deal, though,
because there are so few XS hackers and they're all savvy enough to
grok what's going on after a short explanation -- learning XS is a
daunting, difficult task and I'd guess that the number of people who
know it is in the low thousands, max. Plus, after a significant
push last year the XS code in KinoSearch is now 90% glue -- the
algorithmic stuff is all in either C or Perl. That will be the case
with Lucy as well.
Nevertheless, I figured I'd try the more conventional approach of
using a dedicated XS file for Lucy, so I set up KS to use one as an
experiment. I've been unhappy with the results. The inlining is
just more development-friendly.
So unless someone here raises an objection, I plan to duplicate that
style in Lucy's Perl bindings.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
