OK, we have a number of problems with our script language interface,
and I'd like some feedback. The overall problem is that our current
use of SWIG is broken, and I want to fix it one way or another.
This is long, but I do need some help. Otherwise, you're going to
just have to live with whatever I come up with :> Also note that I
think I have a mostly "correct"-ish solution, but it's going to be a
fairly substantial change, so I want to make sure everyone (who's
interested) understands the issue.
The first problem I'll mention (I mention it first because it's
quicker to explain) is that we can't use automatic SWIG processing of
our headers any more. Our current approach is broken because SWIG
doesn't have a real preprocessor, it just pretends to. The authors
don't recommend using it on "real" code. So by passing SWIG over our
unpreprocessed headers, SWIG isn't necessarily seeing the same code
that gcc is. This is bad.
I thought maybe we could fix it by just running gcc -E over our
headers first to create a preprocessed header that SWIG could then
look at, but that won't work either. If we do that, then the *huge*
preprocessed header has all kinds of glibc gcc extensions in it that
cause SWIG to fall over dead.
I looked at using G-wrap for the guile code instead of SWIG, but it
has the same problem. It doesn't have a real preprocessor either, and
it can't even grok our unpreprocesed headers (ahh if everyone only
used languages with real syntaxes, these problems would be so much
rarer...)
Anyway, this means that no matter what, we're going to have to have
separate specifications for our script exported functions. We can't
use SWIG or G-wrap directly on our headers. I do have a way to do
this (that I'll describe later) that's not too onerous and avoids the
C/scripting API skew problem, but I'll get to that later. One
advantage to not just processing everything is that we have much more
control over what gets exported, and can even take over the export of
some functions manually when that's beneficial (more on that later).
The other fundamental issue (that got me into this investigation in
the first place) is that our current use of SWIG leaks memory like a
sieve and is only going to get a lot worse. The basic issues is that
from C prototypes, SWIG can't know enough to know when it's OK to
release resources. For example, consider this prototype:
char *strdup(const char *x);
SWIG can't know that strdup doesn't store x in some static or global,
and it also can't know whether or not it's OK to delete the temporary
it creates to hold the return value before it converts it to the
scripting language representation. So to be conservative, it just
drops the pointers, leaking memory.
For one-shot scripts, this isn't a big deal, but for the guile
interface, which will be running within GnuCash during an entire
interactive session, or within a GnuCash server (if we ever have such
a beast), this is a critical problem. It creates memory leaks all
over the place. Eventually, the app *will* fall over dead.
Now, you can (in theory) fix this problem by telling SWIG how to deal
with function arguments and return types via it's "%typemap" or
pointers mechanism, but due to weaknesses in the implementation of the
mechanism (or weaknesses in my understanding), it only handles types,
not functions, so you can't easily tell it how the args for each
individual function should behave without losing information. For
example, lets say you have the following function
void foo(const char *bar, double *baz);
and lets presume that bar is an input parameter only (and is never
used by foo after the invocation), and that baz is an output parameter
only. You can tell SWIG about this (at least on it's perl side, I'm
not sure the guile module knows how to handle this) like this:
void foo(const char *INPUT, double *OUTPUT);
This will tell SWIG when it's OK to delete its temporary variables,
but note that you just lost the names of the parameters. Now you can
tell SWIG to use other names, but since the names are matched
globally, they'll affect all functions. This is ugly, and unless
you're going to have two prototypes for the function (the one SWIG
sees and the one gcc sees) you're going to lose useful information
unless you do something else quite ugly like:
void foo(const char *INPUT /*bar*/, double *OUTPUT /*baz*/);
You can also use SWIG's typemap function to do something similar, but
to do it on types, so if we were willing to modify all our prototypes
we could define things like InputString and OutputDouble and then do
this:
void foo(InputString *bar, OutputDouble *baz);
But this is going to require a large number of modifications to our
code, and a large number of new types. It definitely makes things
uglier (in some sense) from the C side.
Now in this whole process, I investigated G-wrap which is a
guile/RScheme wrapper generator, and I discovered that it is in some
ways a superior solution. It's much lighter weight than SWIG, and I
can easily stick the entire thing in ./lib, but it would only support
scheme. This is probably a deal breaker. I *might* be happy with
making that restriction, but I think the horse is already out of the
barn. We're already using things like ePerl which require the perl
interface...
One of the nice things about G-wrap is that it allows you to specify
what should be done with each individual function instead of just
alowing you to talk about global types (like SWIG). So you can say:
(new-function
'shutdown
'void "gnc_shutdown" '((int exit_status)) "shuts down gnucash")
And it's more or less obvious exactly what you mean. Or in the
strings case:
(new-function
'foo
'void "foo" '((const-string bar) (const-string baz)) "frotzes bar")
It also makes the function documentation availble from within the
interpreter and generates html docs.
Summary conclusions:
1) We can't run the script wrapper generator over our headers
directly anymore.
2) Hence we're going to have to have a separate specification for
the exported functions.
3) This separate specification should never get out of sync with the
C prototypes.
4) If we do have a separate specification, it will be much easier to
use whichever wrapper generator we want.
Recommendation:
After spending too much time over the past day or so trying things
out, here's what I propose:
1) We support .scrx files where for any header foo.h, you can create
a corresponding foo.scrx (script export) file. This file will
contain the exported function specifications for that header.
2) The given header foo.h will #include "foo.scrx.h" which will be
automatically generated from foo.scrx and will contain the C
protoypes for the exported functions.
3) foo.scrx.* will also be generated. These files will contain the
data that SWIG or G-wrap (whichever we're using) need to generate
proper, non-leaking wrappers.
4) The specifications in .scrx will look like this:
(exported-function
'shutdown
'void "gnc_shutdown" '((int exit_status)) "shuts down gnucash")
I've already got trivial guile programs that will generate the C
prototypes and the G-wrap specifications from these, so on the
guile side we're all set.
5) We'll also support (as we can) fancier type specifications
describing Accounts, AccountGroups, etc, so that we get a nice
interface to these from the scripting language.
6) This will also allow us to take manual control of the wrapping
of selected functions (if we like). In some cases we can make a
much nicer guile (or perl) interface for a given function by
doing it by hand while relying on the wrapper generator(s) for
the bulk work.
I think this will work, and will give us what we want without too much
suffering. Initially I'm going to try to do all the work with SWIG.
Then everyone (all the languages) benefit. Although I may have to
rely on others to get the details in languages other than guile right.
I know enough about guile's C side API to know that I can get it right
even if it gets tricky (which it may not), but I don't know about
perl/python/tcl's.
If I don't hear any screams of discontent, I'll probably just do all
this soon.
Thoughts?
--
Rob Browning <[EMAIL PROTECTED]> PGP=E80E0D04F521A094 532B97F5D64E3930
----- %< -------------------------------------------- >% ------
The GnuCash / X-Accountant Mailing List
To unsubscribe, send mail to [EMAIL PROTECTED] and
put "unsubscribe gnucash-devel [EMAIL PROTECTED]" in the body
