Hi,
Given Pipeline and ProcessPipeline, the rather obvious question was asked at
Europython "where's ThreadedPipeline? ". Now I know there's issues in doing
this, but I think it's worth considering.
So, hypothetically, suppose we create ThreadedChassis, which is defined as
follows:
* It takes a single, non-activated component as it's sole argument.
We assume for the moment this is a standard generator component.
* It subclasses threadedadaptivecommscomponent
* It creates inboxes and outboxes mirroring the interface of the wrapped
component, forwarding data to/from it.
* Specifically, it also runs the wrapped generator inside the thread. Any
new components the generator starts should also be started in the same
thread as the wrapped generator.
* This implies that the threaded chassis is itself able to act as or owns a
scheduler. (This is doable, since a scheduler is a microprocess)
* It would require an audit of components to check that they use
.activate()
The first question that springs to mind is "does this seem feasible", to which
I think the answer is yes. We could require components to placed inside a
threaded chassis to set a flag stating that they don't change any *class*
state, unless it's protected by an STM store.
The second question that springs to mind is "where can this go horribly
wrong". To my mind, areas include:
* Accidental interception of messages to/from a subcomponent that's using a
service.
* Service discovery & usage
* Service advertising.
The other point is "is actually wrapping it appropriate" or "should we do
something more like @staticmethod or @classmethod" which actually transforms
the component? (ie in the manner of a class decorator)
Clearly such a beast then allows:
Pipeline(
ThreadedChassis( A() ),
ThreadedChassis( B() ),
ThreadedChassis( C() ),
ThreadedChassis( D() ),
)
Which would then obviously also allow:
* ThreadedPipeline( A(),B(), C(), D() )
But the first step is to get a threaded chassis working.
As noted, an optimisation is to "simply" convert the component from one type
to another, but that has slightly more risk involved, and I think starting
with something constrained like this makes the most sense. I suspect though
that for somethings though actually /converting/ the component from generator
to threaded would make the most sense.
It'll probably force a small rethink of some areas, but this is probably the
best initial solution to the problem "How do I use all my CPUs" since it
would provide a natural path for migrating from generators to threads, and on
a GIL-less implementation of python (IronPython, Jython, etc), it would allow
the use of multiple CPUs relatively cleanly.
If you can spot any problems with this I'd be interested. I don't think
there's any fundamental problems here, aside from the usual issues around
shared values, but a second set of eyes would be appreciated.
I think getting this done, and then someone wrapping py.execnet would probably
result in a rather compelling system.
Regards,
Michael
--
http://yeoldeclue.com/blog
http://twitter.com/kamaelian
http://www.kamaelia.org/Home
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en
-~----------~----~----~----~------~----~------~--~---
