Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
Ole, I'm getting pretty tired of this thread, even reading it is annoying. You're whining about stuff which isn't a problem, and making out like Zope/ZODB has all these big critical flaws, casting huge assertions without having enough understanding to make them... Jan-Ole Esleben wrote: security.declareProtected(Use TestPChanged, test_internal) def test_internal(self, args=None, args2=None): Called by test via XML-RPC if args2: self.a.append(1) self._p_changed = 1 What the hell are you doing that for? If self.a is a simple python list, of course you're in trouble. Make it a PersistentList, or better yet, an OOTreeSet, and you'll be much better off... Zope obviously looks at the code, I can assure you Zope does ZERO code introspection... Seriously, your examples are overcomplicated, your use cases are bizarre and your attitude of it must be Zope's fault is more than a little wearing. No-one's forcing you to use Zope, I'd implore you to go and use another framework if you're so convinced Zope does it wrong ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
Jan-Ole Esleben wrote: 1. In the example, just setting _p_changed=1 does _not_ lead to a conflict error. I can assure you it will if any other connection to that ZODB does the same thing on the same object... executed) it _does_. So there _is_ some implicit magical stuff going on and ZOPE tries to take care that only subobjects change (but incompletely)! Rubbish. You're likely misunderstanding your own example, or not understanding the context in which it's executing... 2. You shouldn't use lists and dicts - it should say this on the front page. Why? You want us to rewrite all the docs just to cater for your weird use cases? Tell you what, how about you jost google for zope conflict error and have a read?! 3. It is especially confusing that ZOPE behaves differently when using XML-RPC calls. It doesn't. It behaves exactly the same as for any other request, w.r.t. zodb transactions... XML-RPC though! All the stuff that you claim as being obvious really isn't all that obvious. Maybe for you ;-) PersistentMapping etc., but I still don't know what _exactly_ to expect from ZOPE in terms of behaviour with mutable objects that aren't Persistent (because of the XML-RPC inconsistency There is NO xml-rpc inconsistency... and the self._p_changed inconsistency both mentioned above). _p_changed is pretty hugely documented in all the zodb docs I've ever read. I dunno, maybe you're stupid, lazy, or both, but please quite whining... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
Jan-Ole Esleben wrote: That's my problem. Can you, from the ZOPE documentation, predict that the example below will cause a ConflictError? Certainly, try and ask a simple version of these questions on zodb-dev@zope.org and let Tim or Dieter have some fun ;-) cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
On 12/18/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. This statement is incorrect. You're right. It should be during any transaction where there are potential changes to the object. No, this statement is still correct. Replace product with object and you are right. And I agree, it's actually a lot more complicated than I'd thouht at first. Or simpler. ;-) but it happens under rather magical circumstances. I have an example where it actually happens; this is code from a product that is instantiated twice, one of those instances called TPCDest, the other TestPChanged; the method called directly is TestPChanged.test(): security.declareProtected(Use TestPChanged, test_internal) def test_internal(self, args=None, args2=None): Called by test via XML-RPC if args2: self.a.append(1) self._p_changed = 1 if not args: import xmlrpclib s = xmlrpclib.ServerProxy(http://USER:[EMAIL PROTECTED]:8080/, allow_none=True) s.TestPChanged.test_internal(1) return self.a security.declareProtected(Use TestPChanged, test) def test(self, args2=None): Called directly via ZOPE if args2: self.a.append(1) self._p_changed = 1 import xmlrpclib s = xmlrpclib.ServerProxy(http://USER:[EMAIL PROTECTED]:8080/, allow_none=True) return s.TPCDest.test_internal() Zope obviously looks at the code, because if you remove the if args2 (which is _never_ true), then a ConflictError happens. Maybe I'm stupid, but could someone point out _where_ exactly I am? I don't understand the question. I would like to point out that you set self._p_changed even when you don't change anything. :-) It seems to me that you say that if you process calls test and another calls test_internal, you get a conflict error. This is correct, and as noted before, that code would create a conflict in any type of environment. If you have two processes trying to modify the same dataset, which is what your example seems to do, then you get a conflict. What you claim, is that if you have two processes in Zope change two different datasets, you get a conflict. That statement is still false. Yes. You said different sets of data. That reasonably means different objects. If it doesn't, yuo need to take a long hard look at your object hierarchy. Simply not true. What if I have a field titles (that is a hash of titles to Book objects) and a field comments (that is an array of comments on the library)? Is that so obviously not a sane example? Yes, sorry, having non-persistent aware dictionaries or arrays, and then complaining that you have problems with persitency... In ordinary systems, you would have to find a way to store the data and retrieve it, thus having a model that isn't implicit and entangled with your code. What is implicit with it? See the example for some major implicitness. What is implicit with it? It's also implicit because you have no control over what a transaction considers a tainted object. (You have no real control over the transaction). Thats still not true. I am explicitly talking about changing different sets of data within the same object. You noted that above yourself. Then these different sets should be different objects, and the object should be an object container: Problem solved. You claim that you can't control what a persistent class store. That is false. I don't. I say that if I want to avoid such problems as I describe altogether, I cannot store anything in a persistent class and thus lose most of what makes ZOPE so interesting. This is still not correct. You claim that things get locked that shouldn't be locked in Zope. It is not clear to me why you say that. See the example. I don't see how the example examplifies this. It is obvious to me that you have misunderstood something. I don't know what yet, though. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
On 12/18/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: I know. This is just example code. Just imagine that both methods change completely unrelated sets of data in addition to not changing self.a. Well, yes, but since both a and also b is non-persistence aware lists, that means that you in fact change neither a or b, but self. If however, you do change unrelated sets of data, such as the persistence aware object self.a, and the persistance aware object self.b, then you do NOT get a conflict error. Actually, I don't think we're getting anywhere with this same dataset/different dataset distinction. It wouldn't happen in a database using application because there would be no transaction for self.a. You see, nothing happens to it, so why would there be one? There isn't one in this case either, unless you set self._p_changed = 1, which you of course do... This only happens when you mix your data with your code and have implicit transactions handled by the server. There is no mixing of data and code going on, so we are definitely not going anywhere with TAHT distinction. ;-) Yes, sorry, having non-persistent aware dictionaries or arrays, and then complaining that you have problems with persitency... But that's part of my point: I need to go out of my way to circumvent Python, and I need to be really careful, because using dicts and lists might still work. Nothing is enforced, and where it breaks is hard to predict. No, it's dead easy to predict, as soon as you understand that you should not modify non-persistent aware attributes, and expect that to work optimally. You may be right that doing that should raise an error, but I also don't exactly see how to make that happen. See the example for some major implicitness. What is implicit with it? I explained this above. Transaction handling in Zope (someone else pointed that out in this thread), Zope looking at the code to determine that self.a has changed (which isn't really documented anywhere obvious). I'm sorry, I still don't understand what implicitness you are talking about. It is obvious to me that you have misunderstood something. I don't know what yet, though. I think we might be misunderstanding each other because we both place different value on implicitness and explicit design of data inside code. I am mostly talking about what is, pragmatically, good programming and a supportive environment. No, I think the misunderstanding is that you are overcomplicating something that is really quite simple. But I'm not sure. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
What you say is all perfectly true except: 1. In the example, just setting _p_changed=1 does _not_ lead to a conflict error. With the ineffectual code above it (that never gets executed) it _does_. So there _is_ some implicit magical stuff going on and ZOPE tries to take care that only subobjects change (but incompletely)! 2. You shouldn't use lists and dicts - it should say this on the front page. It is never really mentioned in any way that intuitively leads to such problems as we are now talking about. It isn't very obvious that things work like this when you look at the documentation, and 3. It is especially confusing that ZOPE behaves differently when using XML-RPC calls. From what you say, it should be the same within the ZOPE system as when using XML-RPC. It gets more complicated with XML-RPC though! All the stuff that you claim as being obvious really isn't all that obvious. And of course, once you've been burned by something like this and the confusion it engenders, you will only use PersistentMapping etc., but I still don't know what _exactly_ to expect from ZOPE in terms of behaviour with mutable objects that aren't Persistent (because of the XML-RPC inconsistency and the self._p_changed inconsistency both mentioned above). Ole 2005/12/18, Lennart Regebro [EMAIL PROTECTED]: On 12/18/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: I know. This is just example code. Just imagine that both methods change completely unrelated sets of data in addition to not changing self.a. Well, yes, but since both a and also b is non-persistence aware lists, that means that you in fact change neither a or b, but self. If however, you do change unrelated sets of data, such as the persistence aware object self.a, and the persistance aware object self.b, then you do NOT get a conflict error. Actually, I don't think we're getting anywhere with this same dataset/different dataset distinction. It wouldn't happen in a database using application because there would be no transaction for self.a. You see, nothing happens to it, so why would there be one? There isn't one in this case either, unless you set self._p_changed = 1, which you of course do... This only happens when you mix your data with your code and have implicit transactions handled by the server. There is no mixing of data and code going on, so we are definitely not going anywhere with TAHT distinction. ;-) Yes, sorry, having non-persistent aware dictionaries or arrays, and then complaining that you have problems with persitency... But that's part of my point: I need to go out of my way to circumvent Python, and I need to be really careful, because using dicts and lists might still work. Nothing is enforced, and where it breaks is hard to predict. No, it's dead easy to predict, as soon as you understand that you should not modify non-persistent aware attributes, and expect that to work optimally. You may be right that doing that should raise an error, but I also don't exactly see how to make that happen. See the example for some major implicitness. What is implicit with it? I explained this above. Transaction handling in Zope (someone else pointed that out in this thread), Zope looking at the code to determine that self.a has changed (which isn't really documented anywhere obvious). I'm sorry, I still don't understand what implicitness you are talking about. It is obvious to me that you have misunderstood something. I don't know what yet, though. I think we might be misunderstanding each other because we both place different value on implicitness and explicit design of data inside code. I am mostly talking about what is, pragmatically, good programming and a supportive environment. No, I think the misunderstanding is that you are overcomplicating something that is really quite simple. But I'm not sure. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
On 12/17/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: It's not about the threads or processes being tied up and waiting, it's about the transaction breaking: because the internal call (the one from the second server back to the first) changes the object on the first server, and thus when the first server checks wether the object has changed after the transaction should close (during the last return), it finds that indeed it has, and before it could write to it, so it raises a conflict error invariably. This is still not a problem that has anything to do with Zope or persistenace, but it is quite simply just a conflict error. It will happen anywhere you do things like this. It will also only happen when you, in the processor of modifying an object, calls a method on another server, which, before that call finishes, makes a call back and modifies the *same* object. I can't currently dream up any scenario where this happens, but I'm happy that Zope raises an error when it does. A system that does NOT raise an error in this situation is broken. If you get problems like this, you need to make changes to the software so that this does not happen. The right way to do that depends on your application. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. But in ZOPE, it is definitely the case that data and program are coupled in an implicit way that makes these cases much harder to debug and avoid, because if the two methods in my example operate on different sets of data, which they probably would, or if one of them did a read before calling the external method and then afterwards a write (on an SQL database maybe), nothing would happen if I used explicit data storage! In Zope, it's just the whole object that's tainted. That's my whole point. I think it is a very significant point nonetheless, because this is just an extreme case of what happens when you couple data and programs, and persistent classes are just that: application data inside program code. This had never occured to me as a _real_ problem before. Of course, if you just create self contained code on a small scale that doesn't talk to other programs on the web, you'll likely not run into any great problems because of this. Ole 2005/12/17, Lennart Regebro [EMAIL PROTECTED]: On 12/17/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: It's not about the threads or processes being tied up and waiting, it's about the transaction breaking: because the internal call (the one from the second server back to the first) changes the object on the first server, and thus when the first server checks wether the object has changed after the transaction should close (during the last return), it finds that indeed it has, and before it could write to it, so it raises a conflict error invariably. This is still not a problem that has anything to do with Zope or persistenace, but it is quite simply just a conflict error. It will happen anywhere you do things like this. It will also only happen when you, in the processor of modifying an object, calls a method on another server, which, before that call finishes, makes a call back and modifies the *same* object. I can't currently dream up any scenario where this happens, but I'm happy that Zope raises an error when it does. A system that does NOT raise an error in this situation is broken. If you get problems like this, you need to make changes to the software so that this does not happen. The right way to do that depends on your application. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
On 12/17/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. You *will* run into these problems in exactly the same cases in any other tool. But in ZOPE, it is definitely the case that data and program are coupled in an implicit way that makes these cases much harder to debug and avoid, because if the two methods in my example operate on different sets of data, which they probably would, or if one of them did a read before calling the external method and then afterwards a write (on an SQL database maybe), nothing would happen if I used explicit data storage! It will not happen i Zope in this case either. In Zope, it's just the whole object that's tainted. You just said different sets of data. That reasonably must mean different objects, unless you envision having huge objects with only marginally connected sets of data all stored as attributes. And then you have other problems. :-) That's my whole point. I think it is a very significant point nonetheless, because this is just an extreme case of what happens when you couple data and programs, and persistent classes are just that: application data inside program code. Eh, no they aren't. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. You *will* run into these problems in exactly the same cases in any other tool. I'm sorry, but that's just wrong, and I have given examples of such situations. To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the internal call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break. But in ZOPE, it is definitely the case that data and program are coupled in an implicit way that makes these cases much harder to debug and avoid, because if the two methods in my example operate on different sets of data, which they probably would, or if one of them did a read before calling the external method and then afterwards a write (on an SQL database maybe), nothing would happen if I used explicit data storage! It will not happen i Zope in this case either. No. It does happen, and it _did_ happen in my original problem. Zope doesn't even care if anything actually changes, it just considers an object changed that set self._p_changed=1. In Zope, it's just the whole object that's tainted. You just said different sets of data. That reasonably must mean different objects, unless you envision having huge objects with only marginally connected sets of data all stored as attributes. And then you have other problems. :-) No. An object usually binds together different sets of data (as in the above example - it has several fields, and that is true for almost any given object). What you are saying is don't program the ZOPE way, and it would eventually lead to the conclusion that your product classes should not have any persistent data, which is the conclusion I have come to. That's my whole point. I think it is a very significant point nonetheless, because this is just an extreme case of what happens when you couple data and programs, and persistent classes are just that: application data inside program code. Eh, no they aren't. Please don't just claim stuff. ZOPE has persistent classes. Persistent classes store data that in ordinary programs would not survive as long. In ordinary systems, you would have to find a way to store the data and retrieve it, thus having a model that isn't implicit and entangled with your code. Of course, you could do the same in ZOPE, but I addressed that above, and you could mess up ordinary code, but I think we're agreed that it is not something you should aim for (and I think I have made it clear why I do not believe it's as easy to mess up generic Python code in this way). Ole ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
Am Samstag, den 17.12.2005, 14:56 +0100 schrieb Jan-Ole Esleben: That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. You *will* run into these problems in exactly the same cases in any other tool. I'm sorry, but that's just wrong, and I have given examples of such situations. To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the internal call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break. This is wrong IMHO. dict and list are just columns of the same tuple if you speak RDBMS. And there are very few (if any?) databases which do locks only per column. In RDBMS you distribute in different tables to avoid such (e.g. normalize) in ZODB, you just make subclasses of Persistent for your subobjects. (Attributes) HTH Tino ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. You *will* run into these problems in exactly the same cases in any other tool. I'm sorry, but that's just wrong, and I have given examples of such situations. To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the internal call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break. This is wrong IMHO. dict and list are just columns of the same tuple if you speak RDBMS. And there are very few (if any?) databases which do locks only per column. In RDBMS you distribute in different tables to avoid such (e.g. normalize) in ZODB, you just make subclasses of Persistent for your subobjects. (Attributes) I'm not saying that it absolutely can't be done, it is just very difficult to do it right from the start because most of time you are not even aware you are fiddling with the data model. This of course isn't - I stressed that before - a problem for small scale development. But if you do more complex stuff with ZOPE it's a bit like going back to C - most of the mistakes in your design don't come back to you until much later. However, unlike C, ZOPE is very high level, and the problems stem from too much implicitness and too little separation of said data and code. When you design your model, you know that you are designing data and you handle those data as data explicitly. When you add a field to your class, you have to explicitly check yourself. And of course you could now answer You should be doing that anyway and that nothing hppens in perfectly designed code etc., but that's just not how it really works. You should be able to design stuff incrementally with a little experimentation along the way without constantly impending danger of it all crashing down on you. That's how Python works, and RoR etc. In ZOPE, we're back to the temptation to just stuff a bunch of data into my object. And it's not even obvious that this is a problem, because everything is so tightly interdependent. It's exactly what Python usually avoids (explicit is better than implicit). Ole ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
Am Samstag, den 17.12.2005, 16:43 +0100 schrieb Jan-Ole Esleben: That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. You *will* run into these problems in exactly the same cases in any other tool. I'm sorry, but that's just wrong, and I have given examples of such situations. To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the internal call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break. This is wrong IMHO. dict and list are just columns of the same tuple if you speak RDBMS. And there are very few (if any?) databases which do locks only per column. In RDBMS you distribute in different tables to avoid such (e.g. normalize) in ZODB, you just make subclasses of Persistent for your subobjects. (Attributes) I'm not saying that it absolutely can't be done, it is just very difficult to do it right from the start because most of time you are not even aware you are fiddling with the data model. This of course isn't - I stressed that before - a problem for small scale development. But if you do more complex stuff with ZOPE it's a bit like going back to C - most of the mistakes in your design don't come back to you until much later. However, unlike C, ZOPE is very high level, and the problems stem from too much implicitness and too little separation of said data and code. When you design your model, you know that you are designing data and you handle those data as data explicitly. When you add a field to your class, you have to explicitly check yourself. And of course you could now answer You should be doing that anyway and that nothing hppens in perfectly designed code etc., but that's just not how it really works. You should be able to design stuff incrementally with a little experimentation along the way without constantly impending danger of it all crashing down on you. That's how Python works, and RoR etc. In ZOPE, we're back to the temptation to just stuff a bunch of data into my object. And it's not even obvious that this is a problem, because everything is so tightly interdependent. It's exactly what Python usually avoids (explicit is better than implicit). Partly agreed. But then in zope, simple things are simple and complex things complex. Dont say interlocking transactions are simple anywhere else :-) If you want to do advanced stuff you need to get deeper into the concepts. Unfortunately there is no easy way. But now we figured it out, didnt we? :-) Thats why these lists exist - nobody can probably get things right from the start. Happy coding Tino ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
On 12/17/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. This statement is incorrect. So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the internal call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break. If you use all non-persistent aware dicts and lists, yes. Using only dicts and lists is suboptimal in most OO-languages, and that goes for Python too. It is correct that with Zope, the negative impact of that type of pre'ogramming-style is increased. This is however simply solved by not doing that type of suboptimization, and instead using persistant-aware objects. No. It does happen, and it _did_ happen in my original problem. Zope doesn't even care if anything actually changes, it just considers an object changed that set self._p_changed=1. No, se above. You just said different sets of data. That reasonably must mean different objects, unless you envision having huge objects with only marginally connected sets of data all stored as attributes. And then you have other problems. :-) No. An object usually binds together different sets of data (as in the above example - it has several fields, and that is true for almost any given object). Yes. You said different sets of data. That reasonably means different objects. If it doesn't, yuo need to take a long hard look at your object hierarchy. What you are saying is don't program the ZOPE way, No, in fact, I'm saying program the Zope way, and I am beginning to suspect that the problem is that you are not. and it would eventually lead to the conclusion that your product classes should not have any persistent data, which is the conclusion I have come to. Then you didn't read what I said. That's my whole point. I think it is a very significant point nonetheless, because this is just an extreme case of what happens when you couple data and programs, and persistent classes are just that: application data inside program code. Eh, no they aren't. Please don't just claim stuff. You do. ZOPE has persistent classes. Persistent classes store data that in ordinary programs would not survive as long. Not unless you tell them to. If you don't want the data stored, then do not set it as an attribute on a persistent object. In ordinary systems, you would have to find a way to store the data and retrieve it, thus having a model that isn't implicit and entangled with your code. What is implicit with it? Of course, you could do the same in ZOPE, but I addressed that above, and you could mess up ordinary code, but I think we're agreed that it is not something you should aim for (and I think I have made it clear why I do not believe it's as easy to mess up generic Python code in this way) Well, sorry, it's not clear for me. To summarize: If you have two different processes changing the same set of data, you will get confllict errors. You claim that you will not, but this is false. You claim that you can't control what a persistent class store. That is false. You claim that the whole Zope product gets locked, that is false. It's on a object-basis, not a product basis. You claim that things get locked that shouldn't be locked in Zope. It is not clear to me why you say that. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
On 12/17/05, Jan-Ole Esleben [EMAIL PROTECTED] wrote: You should be able to design stuff incrementally with a little experimentation along the way without constantly impending danger of it all crashing down on you. I don't undertand why you say that this isn't possible in Zope. That's how Python works, and RoR etc. In ZOPE, we're back to the temptation to just stuff a bunch of data into my object. And it's not even obvious that this is a problem, because everything is so tightly interdependent. It's exactly what Python usually avoids (explicit is better than implicit). I agree that there is too much implicitness in Zope 2. I don't agree that persistance is a part of that. It isn't implicit at all. Maybe it's not easy to understand, but it isn't particularily implicit, and neither is it uncontrollable, as you seem to say. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope Persistence (was: XML-RPC within ZOPE)
Thanks; this is a problem we are well aware of. Our solution is to increase the amount of workers, obviously. However, I'm increasingly getting a feeling that for a rather big range of unlikely situations that are nonetheless to be expected, Zope doesn't work _at all_. In a WebServices setting, situations like the one I described, with one server calling back to another server within a call from that latter server and both not knowing anything about the implementation of the other, it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them for persistent objects that actually _do_ change state (unlike mine). The solution is to not have state in your objects, and thus lose instantly most of what Zope is. However, as I see it, the problem is that what Zope actually _is_ (i.e. mostly the ZODB) is an unhealthy way of coupling data and implementation (which is _exactly_ why my implementation didn't work immediately). This of course comes from its origins in TTW development where there wouldn't actually have been many user made products. Please tell me if I'm wrong with my assumption above, and why. I'm not trying to be inflamatory, this just has me really worried. Ole 2005/12/15, Dieter Maurer [EMAIL PROTECTED]: Jan-Ole Esleben wrote at 2005-12-11 19:10 +0100: Is it at all impossible to use XML-RPC _within_ a ZOPE architecture? In principle yes. Be careful however: it is easy to introduce deadlocks! When during request processing you call back into the same Zope via XML-RPC, the calling out request will not complete until the XML-RPC returns a result (this always holds for calls, XML-RPC or other, to an external or internal server). Zope has a limited amount of workers (the default is 4) able to execute requests. If there are no free workers, requests have to wait for one. Now imagine that as many requests arrive as there are workers and each of them wants to perform an XML-RPC against the same Zope. Then you have a deadlock: none of the XML-RPC requests will finish, because there are no free workers. An no worker will ever become free again, because each of them waits for its XML-RPC to finish. Therefore, you should directly call internal resources (rather than use XML-RPC). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
I don't understand the problem. How is using XML-RPC incompatible with persistence? What are you trying to exclude? - C On Dec 16, 2005, at 6:40 AM, Jan-Ole Esleben wrote: Thanks; this is a problem we are well aware of. Our solution is to increase the amount of workers, obviously. However, I'm increasingly getting a feeling that for a rather big range of unlikely situations that are nonetheless to be expected, Zope doesn't work _at all_. In a WebServices setting, situations like the one I described, with one server calling back to another server within a call from that latter server and both not knowing anything about the implementation of the other, it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them for persistent objects that actually _do_ change state (unlike mine). The solution is to not have state in your objects, and thus lose instantly most of what Zope is. However, as I see it, the problem is that what Zope actually _is_ (i.e. mostly the ZODB) is an unhealthy way of coupling data and implementation (which is _exactly_ why my implementation didn't work immediately). This of course comes from its origins in TTW development where there wouldn't actually have been many user made products. Please tell me if I'm wrong with my assumption above, and why. I'm not trying to be inflamatory, this just has me really worried. Ole 2005/12/15, Dieter Maurer [EMAIL PROTECTED]: Jan-Ole Esleben wrote at 2005-12-11 19:10 +0100: Is it at all impossible to use XML-RPC _within_ a ZOPE architecture? In principle yes. Be careful however: it is easy to introduce deadlocks! When during request processing you call back into the same Zope via XML-RPC, the calling out request will not complete until the XML-RPC returns a result (this always holds for calls, XML-RPC or other, to an external or internal server). Zope has a limited amount of workers (the default is 4) able to execute requests. If there are no free workers, requests have to wait for one. Now imagine that as many requests arrive as there are workers and each of them wants to perform an XML-RPC against the same Zope. Then you have a deadlock: none of the XML-RPC requests will finish, because there are no free workers. An no worker will ever become free again, because each of them waits for its XML-RPC to finish. Therefore, you should directly call internal resources (rather than use XML-RPC). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
I don't understand the problem. How is using XML-RPC incompatible with persistence? What are you trying to exclude? I'm sorry, but I don't understand _that_ question. What am I trying to _exclude_? XML-RPC and (the concept of) persistence aren't incompatible. XML-RPC (on a ZOPE server) and ZOPE persistence are, to the extent described by others in this thread, not by me. I just described a couple more cases (and specifically one case) where it is bad that things are this way (and cannot be solved by better design). My point is: this doesn't happen _within_ single programs, but everywhere there is even the slightest bit of communication there is a chance of it happening. Ole - C On Dec 16, 2005, at 6:40 AM, Jan-Ole Esleben wrote: Thanks; this is a problem we are well aware of. Our solution is to increase the amount of workers, obviously. However, I'm increasingly getting a feeling that for a rather big range of unlikely situations that are nonetheless to be expected, Zope doesn't work _at all_. In a WebServices setting, situations like the one I described, with one server calling back to another server within a call from that latter server and both not knowing anything about the implementation of the other, it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them for persistent objects that actually _do_ change state (unlike mine). The solution is to not have state in your objects, and thus lose instantly most of what Zope is. However, as I see it, the problem is that what Zope actually _is_ (i.e. mostly the ZODB) is an unhealthy way of coupling data and implementation (which is _exactly_ why my implementation didn't work immediately). This of course comes from its origins in TTW development where there wouldn't actually have been many user made products. Please tell me if I'm wrong with my assumption above, and why. I'm not trying to be inflamatory, this just has me really worried. Ole 2005/12/15, Dieter Maurer [EMAIL PROTECTED]: Jan-Ole Esleben wrote at 2005-12-11 19:10 +0100: Is it at all impossible to use XML-RPC _within_ a ZOPE architecture? In principle yes. Be careful however: it is easy to introduce deadlocks! When during request processing you call back into the same Zope via XML-RPC, the calling out request will not complete until the XML-RPC returns a result (this always holds for calls, XML-RPC or other, to an external or internal server). Zope has a limited amount of workers (the default is 4) able to execute requests. If there are no free workers, requests have to wait for one. Now imagine that as many requests arrive as there are workers and each of them wants to perform an XML-RPC against the same Zope. Then you have a deadlock: none of the XML-RPC requests will finish, because there are no free workers. An no worker will ever become free again, because each of them waits for its XML-RPC to finish. Therefore, you should directly call internal resources (rather than use XML-RPC). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
On Dec 16, 2005, at 8:02 PM, Jan-Ole Esleben wrote: I don't understand the problem. How is using XML-RPC incompatible with persistence? What are you trying to exclude? I'm sorry, but I don't understand _that_ question. What am I trying to _exclude_? You said: it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them XML-RPC and (the concept of) persistence aren't incompatible. XML-RPC (on a ZOPE server) and ZOPE persistence are, to the extent described by others in this thread, not by me. I just described a couple more cases (and specifically one case) where it is bad that things are this way (and cannot be solved by better design). AFAICT, people have told you to not use XML-RPC here and when you said it was not possible to avoid the use of XML-RPC, they provided suggestion about how to accomplish what you wanted anyway. So I'm not sure what the exact problem is. My point is: this doesn't happen _within_ single programs, but everywhere there is even the slightest bit of communication there is a chance of it happening. The chance of what happening, sorry? I'm still trying to understand the problem. The only problem that was noted so far was a deadlock potential by Dieter which presumed you were doing XML-RPC requests to the same system which accepts them. This is an unusual thing to do; it wouldn't happen under normal circumstances. - C ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
I don't understand the problem. How is using XML-RPC incompatible with persistence? What are you trying to exclude? I'm sorry, but I don't understand _that_ question. What am I trying to _exclude_? You said: it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them That's probably an idiomatic error on my part, sorry. I meant avoid (in German, it's the same word). AFAICT, people have told you to not use XML-RPC here and when you said it was not possible to avoid the use of XML-RPC, they provided suggestion about how to accomplish what you wanted anyway. So I'm not sure what the exact problem is. The problem is a different one now, and I was referring to the _reasons_ people had for telling me not to use XML-RPC. Part of the problem I have now is that no application on the web is isolated from others, and that ZOPE specifically _comes_ with XML-RPC capabilities on the server's part. My point is: this doesn't happen _within_ single programs, but everywhere there is even the slightest bit of communication there is a chance of it happening. The chance of what happening, sorry? I'm still trying to understand the problem. The only problem that was noted so far was a deadlock potential by Dieter which presumed you were doing XML-RPC requests to the same system which accepts them. This is an unusual thing to do; it wouldn't happen under normal circumstances. The problem setup is this; I explained it above, but it this has become a long thread: I write a ZOPE product. I want to make use of other software on the internet and the services that software provides. So I use the methods exposed by that software via SOAP, XML-RPC, whatever. One of those methods actually calls my product back, maybe because the developer has learned that my product itself exposes some (or all) of its functionality via XML-RPC. If this is all inside one call, which I can't avoid explicitly as the developer of just my product, I have a broken transaction on my hands that isn't easy to fix (and maybe impossible). This holds true for the whole product, even if the method called by the second server changed some completely unrelated data. I hope that clears it up a little. Ole ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
The problem setup is this; I explained it above, but it this has become a long thread: I write a ZOPE product. I want to make use of other software on the internet and the services that software provides. So I use the methods exposed by that software via SOAP, XML-RPC, whatever. One of those methods actually calls my product back, maybe because the developer has learned that my product itself exposes some (or all) of its functionality via XML-RPC. If this is all inside one call, which I can't avoid explicitly as the developer of just my product, I have a broken transaction on my hands that isn't easy to fix (and maybe impossible). This holds true for the whole product, even if the method called by the second server changed some completely unrelated data. I see what you're saying, but how is this specific to Zope? If you write a Perl program and expose it via mod_perl on Apache, and the program calls out to a service that calls back in to the mod_perl program (no matter how broken of a pattern this was), wouldn't the Apache process that was waiting on data be tied up in the same way? - C ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
It's not about the threads or processes being tied up and waiting, it's about the transaction breaking: because the internal call (the one from the second server back to the first) changes the object on the first server, and thus when the first server checks wether the object has changed after the transaction should close (during the last return), it finds that indeed it has, and before it could write to it, so it raises a conflict error invariably. Ole 2005/12/17, Chris McDonough [EMAIL PROTECTED]: The problem setup is this; I explained it above, but it this has become a long thread: I write a ZOPE product. I want to make use of other software on the internet and the services that software provides. So I use the methods exposed by that software via SOAP, XML-RPC, whatever. One of those methods actually calls my product back, maybe because the developer has learned that my product itself exposes some (or all) of its functionality via XML-RPC. If this is all inside one call, which I can't avoid explicitly as the developer of just my product, I have a broken transaction on my hands that isn't easy to fix (and maybe impossible). This holds true for the whole product, even if the method called by the second server changed some completely unrelated data. I see what you're saying, but how is this specific to Zope? If you write a Perl program and expose it via mod_perl on Apache, and the program calls out to a service that calls back in to the mod_perl program (no matter how broken of a pattern this was), wouldn't the Apache process that was waiting on data be tied up in the same way? - C ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
