Re: Persistent Distributed Objects
Does pyro work inside of stackless? On Sat, Oct 10, 2009 at 9:54 AM, Simon Forman sajmik...@gmail.com wrote: On Fri, Oct 9, 2009 at 1:11 AM, John Haggerty bouncy...@gmail.com wrote: I am interested in seeing how it would be possible in python to have persistent objects (basically be able to save objects midway through a computation, etc) and do so across multiple computers. Something that would allow for memory, disk space, processing power, etc to be distributed across the nodes not just for number crunching. So for example a warehouse program started up on 3 machines one for adding orders, one for searching for orders, one for organizing the warehouse and them running on a different machine with a single interface. I've seen evidence about this being done wrt what looks like insanely complex stuff on this list but I'm wondering if there is something to do this with any number of nodes and just farm out random classes/objects to them? Check out Pyro (Python Remote Objects): http://pyro.sourceforge.net/ or perhaps Stackless Python http://www.stackless.com/ (Tasklets can be serialized and passed around.) -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Persistent Distributed Objects
On Sun, Oct 11, 2009 at 12:46 PM, John Haggerty bouncy...@gmail.com wrote: Does pyro work inside of stackless? I have no idea, but you wouldn't need both. Only one or the other. ~Simon On Sat, Oct 10, 2009 at 9:54 AM, Simon Forman sajmik...@gmail.com wrote: On Fri, Oct 9, 2009 at 1:11 AM, John Haggerty bouncy...@gmail.com wrote: I am interested in seeing how it would be possible in python to have persistent objects (basically be able to save objects midway through a computation, etc) and do so across multiple computers. Something that would allow for memory, disk space, processing power, etc to be distributed across the nodes not just for number crunching. So for example a warehouse program started up on 3 machines one for adding orders, one for searching for orders, one for organizing the warehouse and them running on a different machine with a single interface. I've seen evidence about this being done wrt what looks like insanely complex stuff on this list but I'm wondering if there is something to do this with any number of nodes and just farm out random classes/objects to them? Check out Pyro (Python Remote Objects): http://pyro.sourceforge.net/ or perhaps Stackless Python http://www.stackless.com/ (Tasklets can be serialized and passed around.) -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Persistent Distributed Objects
On 10 oct, 05:39, bouncy...@gmail.com bouncy...@gmail.com wrote: Sorry about being interpreted as being vague. `et me try to narrow it down. program a creates objects b c d which each need to use 1 disk space 2 ram 3 processor time. I would like to create a heckpoint which would save the work of the object to be later used and then delete it from memory [which I assume from reading about them implimented elsewhere is more or less how they work]. The other part would be to assign the objects via a network depending on the number of network machines. I.e. suppose b c and d see machines e f g on a network they can send those machines objects b c and d and have any work saved on the main machine or local ones. What I was wondering is how would I do that in python. The arbitrary part could be no more complicated than a program where b is calculating a list of prime numbers from x to infinity c is just a notepad program and d is a program that prints say the size of the file as youi type it and says I like writing a poem of [bytes big] while my computer foun ou that [new prime] is the biggest on [machines #] computers. Rmi supposedly does this for the distribuited part. Object persistence does this for saving to the best of what I`ve seen nothing distributes and saved at the same time. Does that help? hi, you could imagine architect a system around Pyro and take care of persistence with simple pickle ? Olivier -- http://mail.python.org/mailman/listinfo/python-list
Re: Persistent Distributed Objects
On 10 oct, 05:39, bouncy...@gmail.com bouncy...@gmail.com wrote: Sorry about being interpreted as being vague. `et me try to narrow it down. program a creates objects b c d which each need to use 1 disk space 2 ram 3 processor time. I would like to create a heckpoint which would save the work of the object to be later used and then delete it from memory [which I assume from reading about them implimented elsewhere is more or less how they work]. The other part would be to assign the objects via a network depending on the number of network machines. I.e. suppose b c and d see machines e f g on a network they can send those machines objects b c and d and have any work saved on the main machine or local ones. What I was wondering is how would I do that in python. The arbitrary part could be no more complicated than a program where b is calculating a list of prime numbers from x to infinity c is just a notepad program and d is a program that prints say the size of the file as youi type it and says I like writing a poem of [bytes big] while my computer foun ou that [new prime] is the biggest on [machines #] computers. Rmi supposedly does this for the distribuited part. Object persistence does this for saving to the best of what I`ve seen nothing distributes and saved at the same time. Does that help? Did you have a look on Zope and ZEO ? It could be interesting. http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/ZEO.stx br, Olivier -- http://mail.python.org/mailman/listinfo/python-list
Re: Persistent Distributed Objects
Sorry about being interpreted as being vague. You wasn't vague. I'm sorry! `et me try to narrow it down. program a creates objects b c d which each need to use 1 disk space 2 ram 3 processor time. I would like to create a checkpoint which would save the work of the object to be later used and then delete it from memory [which I assume from reading about them implimented elsewhere is more or less how they work]. The other part would be to assign the objects via a network depending on the number of network machines. I.e. suppose b c and d see machines e f g on a network they can send those machines objects b c and d and have any work saved on the main machine or local ones. What I was wondering is how would I do that in python. The arbitrary part could be no more complicated than a program where b is calculating a list of prime numbers from x to infinity c is just a notepad program and d is a program that prints say the size of the file as youi type it and says I like writing a poem of [bytes big] while my computer foun ou that [new prime] is the biggest on [machines #] computers.Rmi supposedly does this for the distribuited part. Object persistence does this for saving to the best of what I`ve seen nothing distributes and saved at the same time. Does that help? The pickle module is part of the standard library and it can dump and load objects. One thing you need to worry about is that your machines must share the same code base. You cannot load a pickled object if its class is not available for the interpreter. Apart from that, it is not really problematic to send pickled objects over the network. Just choose any protocol you want. You can try already existing networking libraries for this. Twisted, for example. Or you can use systems that where specifically designed for building distributed networks. Look at Pyro. They manage things like what computers are available automatically. When it comes to distributing objects, the other thing comes in mind is security. Sending sensitive data (or even worse, sending code) over the network is a security concern. I hope I could help and sorry again. It's my bad english. L -- http://mail.python.org/mailman/listinfo/python-list
Re: Persistent Distributed Objects
On Fri, Oct 9, 2009 at 1:11 AM, John Haggerty bouncy...@gmail.com wrote: I am interested in seeing how it would be possible in python to have persistent objects (basically be able to save objects midway through a computation, etc) and do so across multiple computers. Something that would allow for memory, disk space, processing power, etc to be distributed across the nodes not just for number crunching. So for example a warehouse program started up on 3 machines one for adding orders, one for searching for orders, one for organizing the warehouse and them running on a different machine with a single interface. I've seen evidence about this being done wrt what looks like insanely complex stuff on this list but I'm wondering if there is something to do this with any number of nodes and just farm out random classes/objects to them? Check out Pyro (Python Remote Objects): http://pyro.sourceforge.net/ or perhaps Stackless Python http://www.stackless.com/ (Tasklets can be serialized and passed around.) -- http://mail.python.org/mailman/listinfo/python-list
Re: Persistent Distributed Objects
I've seen evidence about this being done wrt what looks like insanely complex stuff on this list but I'm wondering if there is something to do this with any number of nodes and just farm out random classes/objects to them? Designing and opreating distributed systems is a complex thing. Especially if you do not want to specify an exact problem domain that you needed to solve - then you need to design a distributed system that is able to solve general problems. It is very complex, and - at least to my knowledge - there is no efficient solution out of the box. This is not Python specific. It would be a hard task just to design how that system should work, regardless of the computer language it is implemented in. L -- http://mail.python.org/mailman/listinfo/python-list
Re: Persistent Distributed Objects
Sorry about being interpreted as being vague. `et me try to narrow it down. program a creates objects b c d which each need to use 1 disk space 2 ram 3 processor time. I would like to create a heckpoint which would save the work of the object to be later used and then delete it from memory [which I assume from reading about them implimented elsewhere is more or less how they work]. The other part would be to assign the objects via a network depending on the number of network machines. I.e. suppose b c and d see machines e f g on a network they can send those machines objects b c and d and have any work saved on the main machine or local ones. What I was wondering is how would I do that in python. The arbitrary part could be no more complicated than a program where b is calculating a list of prime numbers from x to infinity c is just a notepad program and d is a program that prints say the size of the file as youi type it and says I like writing a poem of [bytes big] while m y computer foun ou that [new prime] is the biggest on [machines #] computers. Rmi supposedly does this for the distribuited part. Object persistence does this for saving to the best of what I`ve seen nothing distributes and saved at the same time. Does that help? The death of one man is a tragedy. The death of ten million is a statistic--- Joseph Stalin --Original Message-- From: Laszlo Nagy gand...@shopzeus.com To: John Haggerty bouncy...@gmail.com,python-list (General) python-list@python.org Date: Fri, 9 Oct 2009 09:18:12 PM +0430 Subject: Re: Persistent Distributed Objects I've seen evidence about this being done wrt what looks like insanely complex stuff on this list but I'm wondering if there is something to do this with any number of nodes and just farm out random classes/objects to them? Designing and opreating distributed systems is a complex thing. Especially if you do not want to specify an exact problem domain that you needed to solve - then you need to design a distributed system that is able to solve general problems. It is very complex, and - at least to my knowledge - there is no efficient solution out of the box. This is not Python specific. It would be a hard task just to design how that system should work, regardless of the computer language it is implemented in. L -- http://mail.python.org/mailman/listinfo/python-list
