Static plugins are now operational although a bit clumsy at the moment.
This is roughly equivalent to Python's "frozen" concept.
First: "The Problem".
================
Currently programs like the webserver use plugins to split the code
into modules, provide separate compilation, and allow upgrading
of one part of the code without touching the rest. This is the traditional
advantage of DLLs.
However, shared libraries, whether linked at load time or loaded
dynamically by string name at run time, do have some problems:
you get delayed error notifications or even crashes if libraries
are missing or are the wrong version, and its hard to know which
DLL was loaded.
In addition, programs using dynamic linkage are hard to distribute
because they depend on an environment and you have to distribute
the program at least some DLLs, and its hard to know which ones,
and the client may have some already possibly out of date ..
and you end up with "DLL hell". Microsoft tried to fix this
with assemblies, Apple with frameworks, and Linux uses a special
kind of versioning and package managers but none of this really
gives one a sense of robustness.
Second: "Freezing"
===============
Freezing allows you to *bind* a DLL into a program statically so that
the program will use the bound in DLL instead of searching the
file system. This is not a complete solution to "The Problem" but it
goes a long way. It allows the programmer to develop a program with
plugins and selectively bind one or more of them into the program.
Note this applies ONLY to plugins, that is, to DLLs which are dynamically
loaded at run time by dlopen()/dlsym() kind of functions invoked under
program control. It doesn't apply to shared libraries linked at load time
(including loading of plugins!). Load time linkage can already be replaced
by compile time linkage with existing technology: just use static linkage.
Note static linkage mixed with dynamic loading is a recipe for total
disaster because you can end up with multiple copies of a module.
Third: "Use Cases"
===============
My key use case is the Felix webserver. This uses a lot of plugins.
It's a pain, because the way plugins are loaded at the moment uses
only the LD_LIBRARY_PATH for searching so a statically linked
webserver will always look for its plugins in /usr/local/lib on my Mac,
unless I use an DYLD_LIBRARY_PATH to coax it to switch to
the uninstalled build (and then the installed webserver will look there too!)
So I want to freeze it so the plugins are bound in. This will NOT stop
the webserver looking for Felix libraries in the installed location.
I can stop that too by simply linking it --static.
Note that plugin freezing is done under program control, and is
independent of static/dynamic linkage model.
There is a more important use case though: "flx", at present,
flx is statically linked at doesn't use plugins.
The problem with this is that "flx" binds in the compiler toolchain,
the configuration, and other stuff. Its impossible to just swap
the tool chain from say clang to gcc, or to swap from a linux
host compile to a MinGW cross compile for Windows.
The obvious solution is to use the command line switches to flx
to select plugins to do what we want, and indeed there are already
toolchain plugins in the library ready for use.
"The Problem" with this is that we want to use "flx" to build Felix,
including itself, and we need to ship a binary to do that.
Which needs to be fully frozen or it will be a nightmare.
So I have resisted using plugins in "flx", but now all that
is changed. "flx" can have the standard host config "frozen"
into it, but still dynamically load plugins for cross compilation etc.
Fourth: "How it works underneath"
==========================
The technology is actually very simply. A function:
add_symbol ("dllname","entrypoint",addrss);
is called to add the machine address "addrss" to a two
level lookup table. The first key is the module name,
and the second is the symbol name. These are just
the usual "dll library name" and "C symbol name"
you'd use. The module name is just the basename of
the DLL without the extension (to allow loading DLLs
on different platforms).
And .. THAT'S IT.
Now, when you call Felix version of dlopen and dlsym,
the symbol added to the module registry with add_symbol
is used instead of loading actually calling dlopen or dlsym.
The registry is attached to the garbage collector object so loading
entries into it *globally* hijacks dynamic linkage. That is, it applies
even if you actually dlopen an DLL which tries to open a registered
module and symbol.
Fifth: "How to do the program linkage steps"
==================================
I am using this make file as a test:
################
plugins:
echo "compiling plugins"
flx --echo --static -c --nolink -o libs/fdoc2html.o
src/lib/plugins/fdoc2html.flx
flx --echo --static -c --nolink -o libs/flx2html.o
src/lib/plugins/flx2html.flx
flx --echo --static -c --nolink -o libs/fpc2html.o
src/lib/plugins/fpc2html.flx
flx --echo --static -c --nolink -o libs/py2html.o
src/lib/plugins/py2html.flx
flx --echo --static -c --nolink -o libs/ocamlhtml.o
src/lib/plugins/ocaml2html.flx
flx --echo --static -c --nolink -o libs/cpp2html.o
src/lib/plugins/cpp2html.flx
flx --echo --static -c --nolink -o libs/fdoc_scanner.o
src/lib/plugins/fdoc_scanner.flx
flx --echo --static -c --nolink -o libs/fdoc_slideshow.o
src/lib/plugins/fdoc_slideshow.flx
flx --echo --static -c --nolink -o libs/fdoc_heading.o
src/lib/plugins/fdoc_heading.flx
flx --echo --static -c --nolink -o libs/fdoc_fileseq.o
src/lib/plugins/fdoc_fileseq.flx
flx --echo --static -c --nolink -o libs/fdoc_paragraph.o
src/lib/plugins/fdoc_paragraph.flx
flx --echo --static -c --nolink -o libs/fdoc_button.o
src/lib/plugins/fdoc_button.flx
ar:
echo "linking archive"
flx --echo --staticlib -o libs/weblib.a \
libs/fdoc2html.o \
libs/flx2html.o \
libs/fpc2html.o \
libs/py2html.o \
libs/ocamlhtml.o \
libs/cpp2html.o \
libs/fdoc_scanner.o \
libs/fdoc_slideshow.o \
libs/fdoc_heading.o \
libs/fdoc_fileseq.o \
libs/fdoc_paragraph.o \
libs/fdoc_button.o \
src/tools/webserver.flx
exe:
echo "compiling thunk"
flx --echo --static -c -o weblink libs/weblib.a weblink.flx
.PHONY : plugins ar exe
######################
Basically, we create a *.o object file for each plugin using --static -c
--nolink.
Next, we just link all the plugins AND the webserver into a static library
(archive).
This is only a convenience step to make the third step simpler. Note there's
a problem in "flx" with argument parsing that prevents "just" linking a bunch
of object files into an archive (there has to be a *.flx at the end to stop the
parser,
because the stuff after that is arguments for the program). I will fix this
when I
figure out how (without losing the ability to say "flx prog arg" and have it
"just work").
So now, we compile a linkage thunk that builds a webserver into an executable
by linking against the archive. It actually "dlopens" the webserver program!
But before it does this it hijacks the webserver and all plugins into the
registry,
which forces everything to statically link.
Sixth: Programming the Freeze
========================
The freezer thunk below is quite long and a pain to write but it works.
I will try to provide some conveniences later to simplify it.
There is a lot of boiler plate here but please just look through it
to the essentials.
The core formula is:
1. Make extern "C" declarations of the symbols. The type
doesn't matter, its just something to force linkage.
2. Make Felix "const" bindings to those symbols.
3. Add the symbols to the module registry.
4. Do this:
val linstance = Dynlink::prepare_lib("webserver");
var init: cont = Dynlink::get_init linstance;
proc chain : cont = "return $1;";
chain init;
When I first did this I eventually got everything linked fine,
and even the plugins got initialised right .. but the webserver
went into a spasm with a bad connection. I check async I/O was
linked .. yes.
I had to sleep on it. And then BANG. You cannot run fibres inside
a subroutine! Because they have to RETURN control to the scheduler.
The standard routine generated by the compiler to initialise stuff
does exactly that .. I just wasn't returning the result.
Hence the "chain" function is born. It just "jumps" to the webserver
by returning the continuation its init function creates to the scheduler.
///////////////////////////////////////////
// webserver plugin linker
// Step 1: name the symbols. They're all extern C so the type is irrelevant.
header syms = '''
// Fdoc
extern "C" void *fdoc2html_create_thread_frame;
extern "C" void *fdoc2html_flx_start;
extern "C" void *fdoc2html_setup;
extern "C" void *xlat_fdoc;
// Felix
extern "C" void *flx2html_create_thread_frame;
extern "C" void *flx2html_flx_start;
extern "C" void *flx2html_setup;
extern "C" void *xlat_felix;
// Flx_pkgconfig
extern "C" void *fpc2html_create_thread_frame;
extern "C" void *fpc2html_flx_start;
extern "C" void *fpc2html_setup;
extern "C" void *xlat_fpc;
// Python
extern "C" void *py2html_create_thread_frame;
extern "C" void *py2html_flx_start;
extern "C" void *py2html_setup;
extern "C" void *xlat_py;
// Ocaml
extern "C" void *ocaml2html_create_thread_frame;
extern "C" void *ocaml2html_flx_start;
extern "C" void *ocaml2html_setup;
extern "C" void *xlat_ocaml;
// C++
extern "C" void *cpp2html_create_thread_frame;
extern "C" void *cpp2html_flx_start;
extern "C" void *cpp2html_setup;
extern "C" void *xlat_cpp;
// fdoc scanner
extern "C" void *fdoc_scanner_create_thread_frame;
extern "C" void *fdoc_scanner_flx_start;
extern "C" void *fdoc_scanner_setup;
extern "C" void *fdoc_scanner;
// fdoc slideshow
extern "C" void *fdoc_slideshow_create_thread_frame;
extern "C" void *fdoc_slideshow_flx_start;
extern "C" void *fdoc_slideshow_setup;
extern "C" void *fdoc_slideshow;
// fdoc heading
extern "C" void *fdoc_heading_create_thread_frame;
extern "C" void *fdoc_heading_flx_start;
extern "C" void *fdoc_heading_setup;
extern "C" void *fdoc_heading;
// fdoc fileseq
extern "C" void *fdoc_fileseq_create_thread_frame;
extern "C" void *fdoc_fileseq_flx_start;
extern "C" void *fdoc_fileseq_setup;
extern "C" void *fdoc_fileseq;
// fdoc paragraph
extern "C" void *fdoc_paragraph_create_thread_frame;
extern "C" void *fdoc_paragraph_flx_start;
extern "C" void *fdoc_paragraph_setup;
extern "C" void *fdoc_paragraph;
// fdoc button
extern "C" void *fdoc_button_create_thread_frame;
extern "C" void *fdoc_button_flx_start;
extern "C" void *fdoc_button_setup;
extern "C" void *fdoc_button;
// the webserver
extern "C" void *webserver_create_thread_frame;
extern "C" void *webserver_flx_start;
''';
// Now, we make felix bindings of the symbols.
// Note we have to take the address of the symbol!
class WebserverPluginSymbols
{
requires syms; // make sure the extern decls are included
// We have to do this dummy requirements because static
// linking removes
requires package "re2";
requires package "faio";
requires package "flx_arun";
// Fdoc
const fdoc2html_create_thread_frame : address =
"&fdoc2html_create_thread_frame";
const fdoc2html_flx_start : address = "&fdoc2html_flx_start";
const fdoc2html_setup : address = "&fdoc2html_setup";
const fdoc2html_xlat_fdoc : address = "&xlat_fdoc";
// Felix
const flx2html_create_thread_frame : address =
"&flx2html_create_thread_frame";
const flx2html_flx_start : address = "&flx2html_flx_start";
const flx2html_setup : address = "&flx2html_setup";
const flx2html_xlat_flx : address = "&xlat_felix";
// Flx_pkgconfig
const fpc2html_create_thread_frame : address =
"&fpc2html_create_thread_frame";
const fpc2html_flx_start : address = "&fpc2html_flx_start";
const fpc2html_setup : address = "&fpc2html_setup";
const fpc2html_xlat_fpc : address = "&xlat_fpc";
// Python
const py2html_create_thread_frame : address = "&py2html_create_thread_frame";
const py2html_flx_start : address = "&py2html_flx_start";
const py2html_setup : address = "&py2html_setup";
const py2html_xlat_py : address = "&xlat_py";
// Ocaml
const ocaml2html_create_thread_frame : address =
"&ocaml2html_create_thread_frame";
const ocaml2html_flx_start : address = "&ocaml2html_flx_start";
const ocaml2html_setup : address = "&ocaml2html_setup";
const ocaml2html_xlat_ocaml : address = "&xlat_ocaml";
// C++
const cpp2html_create_thread_frame : address =
"&cpp2html_create_thread_frame";
const cpp2html_flx_start : address = "&cpp2html_flx_start";
const cpp2html_setup : address = "&cpp2html_setup";
const cpp2html_xlat_cpp : address = "&xlat_cpp";
// fdoc scanner
const fdoc_scanner_create_thread_frame : address =
"&fdoc_scanner_create_thread_frame";
const fdoc_scanner_flx_start : address = "&fdoc_scanner_flx_start";
const fdoc_scanner_setup : address = "&fdoc_scanner_setup";
const fdoc_scanner : address = "&fdoc_scanner";
// fdoc slideshow
const fdoc_slideshow_create_thread_frame : address =
"&fdoc_slideshow_create_thread_frame";
const fdoc_slideshow_flx_start : address = "&fdoc_slideshow_flx_start";
const fdoc_slideshow_setup : address = "&fdoc_slideshow_setup";
const fdoc_slideshow : address = "&fdoc_slideshow";
// fdoc heading
const fdoc_heading_create_thread_frame : address =
"&fdoc_heading_create_thread_frame";
const fdoc_heading_flx_start : address = "&fdoc_heading_flx_start";
const fdoc_heading_setup : address = "&fdoc_heading_setup";
const fdoc_heading : address = "&fdoc_heading";
// fdoc fileseq
const fdoc_fileseq_create_thread_frame : address =
"&fdoc_fileseq_create_thread_frame";
const fdoc_fileseq_flx_start : address = "&fdoc_fileseq_flx_start";
const fdoc_fileseq_setup : address = "&fdoc_fileseq_setup";
const fdoc_fileseq : address = "&fdoc_fileseq";
// fdoc paragraph
const fdoc_paragraph_create_thread_frame : address =
"&fdoc_paragraph_create_thread_frame";
const fdoc_paragraph_flx_start : address = "&fdoc_paragraph_flx_start";
const fdoc_paragraph_setup : address = "&fdoc_paragraph_setup";
const fdoc_paragraph : address = "&fdoc_paragraph";
// fdoc button
const fdoc_button_create_thread_frame : address =
"&fdoc_button_create_thread_frame";
const fdoc_button_flx_start : address = "&fdoc_button_flx_start";
const fdoc_button_setup : address = "&fdoc_button_setup";
const fdoc_button : address = "&fdoc_button";
// the webserver
const webserver_create_thread_frame : address =
"&webserver_create_thread_frame";
const webserver_flx_start : address = "&webserver_flx_start";
open Dynlink;
// Now add all the symbols.
proc addsymbols ()
{
// open Dynlink; // DID NOT WORK HERE!! WHY?? IT SHOULD HAVE!!
add_symbol("fdoc2html","fdoc2html_create_thread_frame",fdoc2html_create_thread_frame);
add_symbol("fdoc2html","fdoc2html_flx_start",fdoc2html_flx_start);
add_symbol("fdoc2html","fdoc2html_setup",fdoc2html_setup);
add_symbol("fdoc2html","xlat_fdoc",fdoc2html_xlat_fdoc);
// Felix
add_symbol("flx2html","flx2html_create_thread_frame",flx2html_create_thread_frame);
add_symbol("flx2html","flx2html_flx_start",flx2html_flx_start);
add_symbol("flx2html","flx2html_setup",flx2html_setup);
add_symbol("flx2html","xlat_felix",flx2html_xlat_flx);
// Flx_pkgconfig
add_symbol("fpc2html","fpc2html_create_thread_frame",fpc2html_create_thread_frame);
add_symbol("fpc2html","fpc2html_flx_start",fpc2html_flx_start);
add_symbol("fpc2html","fpc2html_setup",fpc2html_setup);
add_symbol("fpc2html","xlat_fpc",fpc2html_xlat_fpc);
// Python
add_symbol("py2html","py2html_create_thread_frame",py2html_create_thread_frame);
add_symbol("py2html","py2html_flx_start",py2html_flx_start);
add_symbol("py2html","py2html_setup",py2html_setup);
add_symbol("py2html","xlat_py",py2html_xlat_py);
// Ocaml
add_symbol("ocaml2html","ocaml2html_create_thread_frame",ocaml2html_create_thread_frame);
add_symbol("ocaml2html","ocaml2html_flx_start",ocaml2html_flx_start);
add_symbol("ocaml2html","ocaml2html_setup",ocaml2html_setup);
add_symbol("ocaml2html","xlat_ocaml",ocaml2html_xlat_ocaml);
// C++
add_symbol("cpp2html","cpp2html_create_thread_frame",cpp2html_create_thread_frame);
add_symbol("cpp2html","cpp2html_flx_start",cpp2html_flx_start);
add_symbol("cpp2html","cpp2html_setup",cpp2html_setup);
add_symbol("cpp2html","xlat_cpp",cpp2html_xlat_cpp);
// fdoc scanner
add_symbol("fdoc_scanner","fdoc_scanner_create_thread_frame",fdoc_scanner_create_thread_frame);
add_symbol("fdoc_scanner","fdoc_scanner_flx_start",fdoc_scanner_flx_start);
add_symbol("fdoc_scanner","fdoc_scanner_setup",fdoc_scanner_setup);
add_symbol("fdoc_scanner","fdoc_scanner",fdoc_scanner);
// fdoc slideshow
add_symbol("fdoc_slideshow","fdoc_slideshow_create_thread_frame",fdoc_slideshow_create_thread_frame);
add_symbol("fdoc_slideshow","fdoc_slideshow_flx_start",fdoc_slideshow_flx_start);
add_symbol("fdoc_slideshow","fdoc_slideshow_setup",fdoc_slideshow_setup);
add_symbol("fdoc_slideshow","fdoc_slideshow",fdoc_slideshow);
// fdoc heading
add_symbol("fdoc_heading","fdoc_heading_create_thread_frame",fdoc_heading_create_thread_frame);
add_symbol("fdoc_heading","fdoc_heading_flx_start",fdoc_heading_flx_start);
add_symbol("fdoc_heading","fdoc_heading_setup",fdoc_heading_setup);
add_symbol("fdoc_heading","fdoc_heading",fdoc_heading);
// fdoc fileseq
add_symbol("fdoc_fileseq","fdoc_fileseq_create_thread_frame",fdoc_fileseq_create_thread_frame);
add_symbol("fdoc_fileseq","fdoc_fileseq_flx_start",fdoc_fileseq_flx_start);
add_symbol("fdoc_fileseq","fdoc_fileseq_setup",fdoc_fileseq_setup);
add_symbol("fdoc_fileseq","fdoc_fileseq",fdoc_fileseq);
// fdoc paragraph
add_symbol("fdoc_paragraph","fdoc_paragraph_create_thread_frame",fdoc_paragraph_create_thread_frame);
add_symbol("fdoc_paragraph","fdoc_paragraph_flx_start",fdoc_paragraph_flx_start);
add_symbol("fdoc_paragraph","fdoc_paragraph_setup",fdoc_paragraph_setup);
add_symbol("fdoc_paragraph","fdoc_paragraph",fdoc_paragraph);
// fdoc button
add_symbol("fdoc_button","fdoc_button_create_thread_frame",fdoc_button_create_thread_frame);
add_symbol("fdoc_button","fdoc_button_flx_start",fdoc_button_flx_start);
add_symbol("fdoc_button","fdoc_button_setup",fdoc_button_setup);
add_symbol("fdoc_button","fdoc_button",fdoc_button);
// webserver
add_symbol("webserver","webserver_create_thread_frame",webserver_create_thread_frame);
add_symbol("webserver","webserver_flx_start",webserver_flx_start);
}
}
// Add the symbols
WebserverPluginSymbols::addsymbols;
// Now invoke the webserver!
println$ "Running webserver";
val linstance = Dynlink::prepare_lib("webserver");
println$ "Webserver prepared";
var init: cont = Dynlink::get_init linstance;
proc chain : cont = "return $1;";
chain init;
///////////////////////////////////////////
--
john skaller
skal...@users.sourceforge.net
http://felix-lang.org
------------------------------------------------------------------------------
Own the Future-Intel(R) Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest. Compete
for recognition, cash, and the chance to get your game on Steam.
$5K grand prize plus 10 genre and skill prizes. Submit your demo
by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2
_______________________________________________
Felix-language mailing list
Felix-language@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/felix-language
