Re: [kamaelia-list] Threaded component message handling
Hi Jim, I'm not too surprised to hear about the increase - it's probably akin to the boost we got when we optimised out the Postman class. In the past I've written scratch versions of Axon using threads as the default and Queue.Queues as the default with direct linkage and that all worked fine. Likewise I also did a version using multiprocessing library to default to processes and channels and that had some interesting results. Neither of these ever got merged onto the main codebase because I didn't have a need at the time - and the performance I saw was sufficiently good for the tasks in hand. Michael. On 16 October 2013 13:53, Jim Easterbrook jim.easterbr...@bbc.co.uk wrote: I've been experimenting with Axon's threaded components. Currently incoming messages are delivered to the wrapper, which forwards them to the thread at its next scheduler slot. I've written a threadedsink class to replace the realsink class used when creating an inbox. It's based on Queue.Queue - the append method puts a message on the queue and wakes up the thread directly, without the wrapper component getting involved. Similarly, the threaded component's send method can check to see if the receiving inbox's sink is a threadedsink object, and if so append the outgoing message directly. Otherwise the usual threaded component out queue is used. These two changes give me much better performance with pipelines of threaded components. Messages flow from one component to the next without the scheduler having to do anything. Note that I haven't given much thought to what happens if links are made or unmade while messages are flowing. That's something I try and avoid doing. (Turn the water off before disconnecting the hose.) -- Jim Easterbrook Senior Research Engineer BBC Research Development South Lab BBC Centre House 56 Wood Lane London W12 7SB Phone: +44 303 040 9705 Mobile: +44 7590 307002 FAX:+44 20 8811 8815 -- You received this message because you are subscribed to the Google Groups kamaelia group. To unsubscribe from this group and stop receiving emails from it, send an email to kamaelia+unsubscribe@**googlegroups.comkamaelia%2bunsubscr...@googlegroups.com . To post to this group, send email to kamaelia@googlegroups.com. Visit this group at http://groups.google.com/**group/kamaeliahttp://groups.google.com/group/kamaelia . For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out . -- You received this message because you are subscribed to the Google Groups kamaelia group. To unsubscribe from this group and stop receiving emails from it, send an email to kamaelia+unsubscr...@googlegroups.com. To post to this group, send email to kamaelia@googlegroups.com. Visit this group at http://groups.google.com/group/kamaelia. For more options, visit https://groups.google.com/groups/opt_out.
Re: [kamaelia-list] Threaded component message handling
Hi guys, On 16 October 2013 16:12, Michael spark...@gmail.com wrote: Found the code I was referring to - it's here: http://code.google.com/p/kamaelian/source/browse/Sketches/Axon/Axon2/ Never really reached the stage of usefulness for reasons irrelevant to the code previously discussed on list. Michael. On 16 October 2013 15:06, Michael spark...@gmail.com wrote: Hi Jim, I'm not too surprised to hear about the increase - it's probably akin to the boost we got when we optimised out the Postman class. In the past I've written scratch versions of Axon using threads as the default and Queue.Queues as the default with direct linkage and that all worked fine. Likewise I also did a version using multiprocessing library to default to processes and channels and that had some interesting results. Neither of these ever got merged onto the main codebase because I didn't have a need at the time - and the performance I saw was sufficiently good for the tasks in hand. Michael. On 16 October 2013 13:53, Jim Easterbrook jim.easterbr...@bbc.co.ukwrote: I've been experimenting with Axon's threaded components. Currently incoming messages are delivered to the wrapper, which forwards them to the thread at its next scheduler slot. I've written a threadedsink class to replace the realsink class used when creating an inbox. It's based on Queue.Queue - the append method puts a message on the queue and wakes up the thread directly, without the wrapper component getting involved. Similarly, the threaded component's send method can check to see if the receiving inbox's sink is a threadedsink object, and if so append the outgoing message directly. Otherwise the usual threaded component out queue is used. These two changes give me much better performance with pipelines of threaded components. Messages flow from one component to the next without the scheduler having to do anything. Note that I haven't given much thought to what happens if links are made or unmade while messages are flowing. That's something I try and avoid doing. (Turn the water off before disconnecting the hose.) -- Jim Easterbrook Senior Research Engineer BBC Research Development South Lab BBC Centre House 56 Wood Lane London W12 7SB Phone: +44 303 040 9705 Mobile: +44 7590 307002 FAX:+44 20 8811 8815 -- You received this message because you are subscribed to the Google Groups kamaelia group. To unsubscribe from this group and stop receiving emails from it, send an email to kamaelia+unsubscribe@**googlegroups.comkamaelia%2bunsubscr...@googlegroups.com . To post to this group, send email to kamaelia@googlegroups.com. Visit this group at http://groups.google.com/**group/kamaeliahttp://groups.google.com/group/kamaelia . For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out . -- You received this message because you are subscribed to the Google Groups kamaelia group. To unsubscribe from this group and stop receiving emails from it, send an email to kamaelia+unsubscr...@googlegroups.com. To post to this group, send email to kamaelia@googlegroups.com. Visit this group at http://groups.google.com/group/kamaelia. For more options, visit https://groups.google.com/groups/opt_out. Any thoughts on using gevent in that context? I'm not a massive user of greenlets but having used erlang quite a bit, I see their appeal. -- - Sylvain http://www.defuze.org http://twitter.com/lawouach -- You received this message because you are subscribed to the Google Groups kamaelia group. To unsubscribe from this group and stop receiving emails from it, send an email to kamaelia+unsubscr...@googlegroups.com. To post to this group, send email to kamaelia@googlegroups.com. Visit this group at http://groups.google.com/group/kamaelia. For more options, visit https://groups.google.com/groups/opt_out.
Re: [kamaelia-list] Threaded component message handling
Hi, gevent would be nice, on the (obvious) proviso any implementation integrated approrpriately with other parts of the gevent ecosystem. Michael On 16 October 2013 15:16, Sylvain Hellegouarch s...@defuze.org wrote: Hi guys, On 16 October 2013 16:12, Michael spark...@gmail.com wrote: Found the code I was referring to - it's here: http://code.google.com/p/kamaelian/source/browse/Sketches/Axon/Axon2/ Never really reached the stage of usefulness for reasons irrelevant to the code previously discussed on list. Michael. On 16 October 2013 15:06, Michael spark...@gmail.com wrote: Hi Jim, I'm not too surprised to hear about the increase - it's probably akin to the boost we got when we optimised out the Postman class. In the past I've written scratch versions of Axon using threads as the default and Queue.Queues as the default with direct linkage and that all worked fine. Likewise I also did a version using multiprocessing library to default to processes and channels and that had some interesting results. Neither of these ever got merged onto the main codebase because I didn't have a need at the time - and the performance I saw was sufficiently good for the tasks in hand. Michael. On 16 October 2013 13:53, Jim Easterbrook jim.easterbr...@bbc.co.ukwrote: I've been experimenting with Axon's threaded components. Currently incoming messages are delivered to the wrapper, which forwards them to the thread at its next scheduler slot. I've written a threadedsink class to replace the realsink class used when creating an inbox. It's based on Queue.Queue - the append method puts a message on the queue and wakes up the thread directly, without the wrapper component getting involved. Similarly, the threaded component's send method can check to see if the receiving inbox's sink is a threadedsink object, and if so append the outgoing message directly. Otherwise the usual threaded component out queue is used. These two changes give me much better performance with pipelines of threaded components. Messages flow from one component to the next without the scheduler having to do anything. Note that I haven't given much thought to what happens if links are made or unmade while messages are flowing. That's something I try and avoid doing. (Turn the water off before disconnecting the hose.) -- Jim Easterbrook Senior Research Engineer BBC Research Development South Lab BBC Centre House 56 Wood Lane London W12 7SB Phone: +44 303 040 9705 Mobile: +44 7590 307002 FAX:+44 20 8811 8815 -- You received this message because you are subscribed to the Google Groups kamaelia group. To unsubscribe from this group and stop receiving emails from it, send an email to kamaelia+unsubscribe@**googlegroups.comkamaelia%2bunsubscr...@googlegroups.com . To post to this group, send email to kamaelia@googlegroups.com. Visit this group at http://groups.google.com/**group/kamaeliahttp://groups.google.com/group/kamaelia . For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out . -- You received this message because you are subscribed to the Google Groups kamaelia group. To unsubscribe from this group and stop receiving emails from it, send an email to kamaelia+unsubscr...@googlegroups.com. To post to this group, send email to kamaelia@googlegroups.com. Visit this group at http://groups.google.com/group/kamaelia. For more options, visit https://groups.google.com/groups/opt_out. Any thoughts on using gevent in that context? I'm not a massive user of greenlets but having used erlang quite a bit, I see their appeal. -- - Sylvain http://www.defuze.org http://twitter.com/lawouach -- You received this message because you are subscribed to the Google Groups kamaelia group. To unsubscribe from this group and stop receiving emails from it, send an email to kamaelia+unsubscr...@googlegroups.com. To post to this group, send email to kamaelia@googlegroups.com. Visit this group at http://groups.google.com/group/kamaelia. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups kamaelia group. To unsubscribe from this group and stop receiving emails from it, send an email to kamaelia+unsubscr...@googlegroups.com. To post to this group, send email to kamaelia@googlegroups.com. Visit this group at http://groups.google.com/group/kamaelia. For more options, visit https://groups.google.com/groups/opt_out.
