Kamaelia People,
I've already sent this to Michael, but I thought I'd forward it to the group
because we're all in different time zones...
I've also included a recording of an early interactive session.
---
Well my "refactor" of my experimental D-based, flow-based programming system
turned from a several week project into a several month project.
In the process I created an OS interface layer, which almost turned into a
message passing OS unto itself.
In the end I had to force myself to stop adding features so I could push it
out the door and make it usable (at least for myself :-)
I have the current code base in a private Mercurial repository at
bitbucket.org.
I've stopped making patches and am ready for 0.9 release. It's more than a
proof of concept and a little less than a fully developed system.
But what it does have...
DENDRITE ARCHITECTURE
(I've tried to stick with a SOLID/Single Responsibility design and use
well-known patterns.)
:: Global Backplane::
- Components, mailboxes and processes are attached to a global backplane
that manages all attached mailboxes, message passing and
process synchronization. If there's app-layer kernel, this is it.
::Message Passing::
- Inboxes default to 32 messages max, but you can specify the default size
at creation time.
- Inboxes / Outboxes are typed. You can pass whatever type of object you
want, from an unsigned integer to complex frame buffer object.
- When you read from an empty inbox or write to a full inbox your process
is blocked, but you can check inbox readiness.
- When you write to an unconnected outbox your process is blocked.
+ When said outbox is linked to a ready inbox, the sending process is
unblocked and comms resume, enabling run-time reconfiguration of apps.
- If tracing is turned on, the message's "toString" method is called and
marked up to provide a human readable log of message transfers.
- If default output is enabled, unconnected outbox sends are also
serialized and routed to stdout, providing Unix pipe behavior.
- Unoptimized performance on a 2Ghz Linux VM is about 12 million messages
per second for simple integers.
:: Process Management::
- Thread scheduling / execution is managed via cooperative threads,
implemented using the "Fiber" primitive from D/Tango.
- The scheduler is a plug-in. Currently round-robin, single priority.
:: Components ::
- Component processes are objects that are attached to a global backplane
and request mailboxes. They can allocate any type of
in/outbox required.
- Components themselves are created via a Component Factory which can
create components by name, from a catalog created at run-time.
- The ComponentFactory is decoupled from the backplane itself since the
Backplane doesn't know about Components, just their underlying
processes and mailboxes. The ComponentFactory knows about all component
designs (called blueprints) at compile time.
:: Event Handling ::
- All events "bubble up" from a Reactor that interfaces to the OS and
pushes events to individual "Handlers", such as the Backplane,
Console, Command Line Interpreter, Session etc.
- The Reactor interface is generalized and currently a composite plug-in.
It's currently a wrapper around "libev", a very high performance
event loop.
:: Session ::
- Session is an interactive command handler that accepts commands from the
console, executes them and maintains component and
session state.
- The session can list the component catalog, create components, attach
them to the backplane, link their mailboxes and run them via
their assigned names.
- The session can also send commands to other handlers to pause, resume the
backplane, set I/O options etc.
- A one-shot capability is available in the Session to provide for batch
operation.
:: Suggested / Scheduled Improvements ::
- Backplane Improvements
- Implement a slightly cleaner dynamic/decoupled mailbox linking method
(more elegant than void * games)
- Backplane is just short of a "god" class -- currently moderated
somewhat by composition. Look at ways to split up its functionality.
- Improve the "wait" semantics to allow components to wait for multiple
assertions concurrently.
- Release a round-robin priority scheduler.
- Design a "Generator" patch that allows components to return objects
at yield time
- More components
- An object/type-specific "Bus" that implements basic pub/sub
semantics.
- An object/type-specific "Pool" that implements a Linda-like
distributed tuple-space.
- A wider array of components to handle async I/O, network I/O and
various app components ported from Axon/Kamaelia.
- Consider a structured serializer such as JSON or SEXP for
machine-independent message passing.
- Create an interface to integrate the Lua "object" system with a
series of inbox types.
- Improvements to Controller/Reactor/Handler Structure
- Currently the Reactor structure could use some significant
improvements:
- It's not a pure reactor and not particularly portable (even though
the controller is).
- It doesn't implement request queues and only works by direct
inter-handler calls. This prevents a more elegant I/O architecture.
- New structure will require IORequests and Completion Queues.
Portability will require a IORequest with embedded I/O delegate.
- Clean up event structure so that it could run as a re-entrant,
non-cooperative kernel.
- Consider multicore, multiprocessor and distributed issues.
- Formalize the low-level signal semantics.
- A port to GLIB/GTK event loop in the next version. (libev is
excellent, but I started this to support a graphic workbench :-)
- Improvements to Session
- Currently the Session breaks encapsulation by accepting command strings
directly from the console and interpreting them. The
session should only process commands, maintain state and provide undo
behavior. This allows separate command suppliers
such as a separate Command Line Interpreter (CLI) as well as a
concurrent GUI menu command supplier.
- The addition of a completely separate CLI.
- New Command Line Interpreter
- Currently a very primitive command line interpreter is embedded into the
Session handler, this will be removed in the next refactor.
- A Lua CLI will be embedded into the next release, enabling a much richer
scripting and command structure.
- Improvements to Console
- Full GNU readline-like command history and editing will be included.
Currently this is precluded by the limited functionality of the
I/O event system.
- Termcap-level portability will be available, possibly via ncurses.
- Various network, TTY and GUI Console implementations.
- Improvements to Documentation
- I've attempted to document most of the classes and modules via the
automated D documentation system, but there's room for
improvement.
:: Known Issues ::
- When a component exits, it's process is cleaned up but the Session
handler isn't notified, so higher level
Component dictionaries aren't updated. Simple to fix at the handler
level.
---
*** Session Log ***
draco% ./bplane_demo
Dendrite ver 0.9
For help, type 'help'
>help
Dendrite Commands
(enter the name of the command with no arguments for more help)
help: show this message
quit: exit Dendrite
make: make a component of type <CatalogName> called
<ComponentName>
show: displays information about session
run: begin execution of a named component
tron: trace on, enables message logging
troff: trace off, disable message logging
defaultout: enable default message serialization for unconnected
components
nodefaultout: disable default message serialization for unconnected
components
pause: pause the entire backplane
resume: resume the entire backplane
>make CoalSupplier coalCar
created and attached: coalCar
>make CoalBurner steamEngine
created and attached: steamEngine
>show components
Current Components
Name Type
-----------------------------------------------------------
coalCar dendrite.components.CoalSupplier.CoalSupplier
steamEngine dendrite.components.CoalBurner.CoalBurner
>show mailboxes
Outbox Name Outbox Type
-----------------------------------------------------------
coalCar:coalFeed int
steamEngine:coalRequests int
Inbox Name Inbox Type
-----------------------------------------------------------
coalCar:requests int
steamEngine:coalBox int
>link coalCar:coalFeed steamEngine:coalBox
linked
>run coalCar
>coalCar: waiting for requests...
>run steamEngine
>steamEngine: running...
steamEngine: requesting 100 pieces of coal...
>link steamEngine:coalRequests coalCar:requests
linked
>steamEngine: sent coal request, waiting for it...
coalCar: received request for 100 pieces of coal
steamEngine===99===steamEngine===98===steamEngine===97===steamEngine===96===
steamEngine===95===steamEngine===94===steamEngine===93===steamEngine===92===
steamEngine===91
===steamEngine===90===steamEngine===89===steamEngine===88===
steamEngine===87===steamEngine===86===steamEngine===85===steamEngine===84===
steamEngine===83===steamEngine===82
===steamEngine===81===steamEngine===80===steamEngine===79===
steamEngine===78===steamEngine===77===steamEngine===76===steamEngine===75===
steamEngine===74===steamEngine===73
===steamEngine===72===steamEngine===71===steamEngine===70===
steamEngine===69===steamEngine===68===steamEngine===67===steamEngine===66===
steamEngine===65===steamEngine===64
===steamEngine===63===steamEngine===62===steamEngine===61===
steamEngine===60===steamEngine===59===steamEngine===58===steamEngine===57===
steamEngine===56===steamEngine===55
===steamEngine===54===steamEngine===53===steamEngine===52===
steamEngine===51===steamEngine===50===steamEngine===49===steamEngine===48===
steamEngine===47===steamEngine===46
===steamEngine===45===steamEngine===44===steamEngine===43===
steamEngine===42===steamEngine===41===steamEngine===40===steamEngine===39===
steamEngine===38===steamEngine===37
===steamEngine===36===steamEngine===35===steamEngine===34===
steamEngine===33===steamEngine===32===steamEngine===31===steamEngine===30===
steamEngine===29===steamEngine===28
===steamEngine===27===steamEngine===26===steamEngine===25===
steamEngine===24===steamEngine===23===steamEngine===22===steamEngine===21===
steamEngine===20===steamEngine===19
===steamEngine===18===steamEngine===17===steamEngine===16===
steamEngine===15===steamEngine===14===steamEngine===13===steamEngine===12===
steamEngine===11===steamEngine===10
===steamEngine===9===steamEngine===8===steamEngine===7===steamEngine===6===
steamEngine===5===steamEngine===4===steamEngine===3===
steamEngine===2===steamEngine===1
===steamEngine===0===
steamEngine exited.
>quit
exiting...
draco%
--------------------------
Here's the source code for the CoalSupplier and CoalBurner components...
class CoalSupplier : Component
{
Inbox!(int) requests;
Outbox!(int) coalFeed;
void init() {
debug (2) Stdout("CoalSupplier: allocating I/O").newline;
requests = backplane.createInbox!(int)(name~":requests"); //
incoming requests
coalFeed = backplane.createOutbox!(int)(name~":coalFeed"); //
outgoing coal
}
void run() {
int maxCoal;
Stdout(name)(": waiting for requests...").newline;
while (active)
{
maxCoal = requests.read(); // someone asks for maxCoal pieces
Stdout(name)(": received request for ")(maxCoal)(" pieces of
coal").newline;
while (maxCoal--)
{
coalFeed.send(maxCoal); // send some coal
yield();
}
}
}
}
class CoalBurner : Component
{
Outbox!(int) coalRequests;
Inbox!(int) coalBox;
void init() {
debug (2) Stdout("CoalSupplier: allocating I/O").newline;
coalRequests = backplane.createOutbox!(int)(name~":coalRequests");
// outgoing requests for coal
coalBox = backplane.createInbox!(int)(name~":coalBox");
// incoming coal
}
void run() {
Stdout(name)(": running...").newline;
Stdout(name)(": requesting 100 pieces of coal...").newline;
// make a request for some coal
coalRequests.send(100); // 100 pieces please
Stdout(name)(": sent coal request, waiting for it...").newline;
yield();
while (coalBox.notEmpty())
{
Stdout(name)("===")(coalBox.read())("===").flush;
yield();
}
}
}
----
I'm currently finishing the README and INSTALL files.
The project manager in me is forcing me to release this based on the Eric
Raymond model:
- Release early, release often.
- Release it when you're still a little embarrassed about the code.
- Since I haven't done a wholesale code cleanup and there's at least one
significant known issue, that qualifies
as slightly embarrassing.
I'm releasing this any day now so I can move forward with the Reactor, GTK
and Lua upgrades.
Any comments, bug fixes or help is greatly appreciated.
As soon as I finish the README and INSTALL, I'll forward the source
repository URL and do a formal announcement.
Regards,
Jim Burnes
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
