[ZODB-Dev] zope cache
Hi We write a product to with python. After changing the data. When quarrying the object we get inconsistent result. Sometimes we get the previous values. Thanks dganit ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Tue, Dec 1, 2009 at 8:44 PM, Martin Aspeli optilude+li...@gmail.com wrote: Jim Fulton wrote: On Fri, Nov 20, 2009 at 1:52 PM, Jim Fulton j...@zope.com wrote: On Fri, Nov 20, 2009 at 1:36 PM, Tres Seaver tsea...@palladion.com wrote: ... More importantly, Chris' change touches non-trivial code that isn't exercised by your test. It *looks* OK (aside from the minor bug). We are doing users a serious disservice giving them such an important tool with minimal tests and no automated tests. I'm going to back out these changes. If someone really cares about repozo in the slightest, they'll at least convert the existing manual test into an automated test. I'm pretty sure that this is a straightforward project. With an automated version of the manual test, I'd be comfortable reapplying Chris' change. Is anyone willing to convert the manual test to an automated one? Can you clarify this? To me, it looks like: - there was a small/trivial bug - there were no tests for the existing code - Chris fixed it, and didn't add any tests The fix was (apropriately) not trivial. - Noise ensued - Tres wrote a trivial test for the trivial fix And the test was inadequate. It tested that the symptom addressed by the fix went away. It didn't test that the fix didn't break backups and restores. - You now found some different problem in the same code, and want to back out Chris' change because he didn't go and add a bunch of tests for the rest of the code, which he didn't change. Read my message more carefiully. I'm backing out the change until there are tests for the code that he did change. If I have this right, I am astonished. I'm going to give you a chance to tell me I have it all wrong before I pass judgement, though. You have it wrong. Read my message carefully. This whole discussion is rather dissappointing. Something you really want to get right is a backup tool, yet while people argue that there should be one, no one seems to be very concerned about whether it works correctly. No one seems to be willing to step up and take responsibility. I guess everyone assumes I will. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Tue, Dec 1, 2009 at 6:24 PM, Chris Withers ch...@simplistix.co.uk wrote: Jim Fulton wrote: Well, not really fair enough. I just looked at the change that Chris made. It has a bug that would be caught by your test if anyone tried to port repozo to Python 3. Which alternate reality are you talking about here? ;-) For some reason, when checking the Python version, you only check the second digit. So, with Python 3.0-3.4, you'll try to import md5. This is a minor bug that is caught by Tres' test. It is not the reason I'm pulling the changes. ... We are doing users a serious disservice giving them such an important tool with minimal tests and no automated tests. Yes, because effectively deleting work that people have contributed is a *really* great way of encouraging people to contribute... I don't want people to contribute crap. You've made some reasonable contributions recently so I know you are capable of producing non-crap. In any case, nothing is deleted in subversion. If someone creates even minimal tests that exercise the code you changed, then I'll be happy to reapply the change. At some point, if no one takes responsibility for repozo, I'm inclined to remove it from the distribution, as there is an implied promise of support, which seems to be lacking. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jim Fulton wrote: snip No one seems to be willing to step up and take responsibility. I guess everyone assumes I will. Jim I'll take the responsibility of transforming the manual test to an automated... -- Godefroid Chapelle (aka __gotcha) http://bubblenet.be ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zope cache
Hi Dganit, We write a product to with python. After changing the data. When quarrying the object we get inconsistent result. Sometimes we get the previous values. Can you please add some details about the problem and a question in the message if you have one. If you are looking for help I think some more details are needed. The best option would be a Python test case code which we can run ourselves and repeat the problem. Cheers, -Mikko ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jim Fulton wrote: No one seems to be willing to step up and take responsibility. I guess everyone assumes I will. I know a thing or two about trying to entice people to help maintain open source software. It's certainly not easy. But from a casual observer's perspective, this whole exchange has been pretty tragic, and certainly doesn't help encourage people to contribute. As the maintainer of the ZODB, your voice carries a lot of weight. You are also making it very clear that you need to be the single point of co-ordination for ZODB contributions (probably wise), and that you don't want contributions directly into trunk. That cements this position even more. I don't think the lack of contributions is directly your fault, but I do think that your communication style can sometimes put people off. I guess the best word I can think of to describe it is terse. I'm sure it's misunderstood, in that demeanour and tone is hard to convey in text. But nonetheless, reading this list over the last few days makes me pretty disheartened. There are clearly issues with the development process. There were clearly issues with that particular commit. But there are few things as damaging to contributions as backing out changes. Sometimes it is necessary. But in this case, it is my experience that it's important to salvage the relationship with the commiter. Using words like crap to describe their code is a pretty safe way to ensure they won't contribute again, even if you allude to the fact that they can also produce non-crap. Martin ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Wed, Dec 2, 2009 at 7:12 AM, Godefroid Chapelle got...@bubblenet.be wrote: Jim Fulton wrote: snip No one seems to be willing to step up and take responsibility. I guess everyone assumes I will. Jim I'll take the responsibility of transforming the manual test to an automated... Thanks. Appreciated. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Wed, Dec 2, 2009 at 7:19 AM, Martin Aspeli optilude+li...@gmail.com wrote: Jim Fulton wrote: In any case, nothing is deleted in subversion. If someone creates even minimal tests that exercise the code you changed, then I'll be happy to reapply the change. At some point, if no one takes responsibility for repozo, I'm inclined to remove it from the distribution, as there is an implied promise of support, which seems to be lacking. I think all hope of anyone taking the ZODB seriously outside those already wedded to it would be lost if there wasn't some kind of supported backup solution. Have I ever said a supported backup solution wasn't important? BTW, offline back ups if file-storage even with Blobs is straightforward without repozo. ... The correct course of action here, in my opinonion, is not to make threats, but to try to encourage people to step up. I find that a positive message works much better for encouragement than a negative one. I haven't made any threats. It is dishonest to include a backup tool in ZODB that isn't supported. Users would have expectations that aren't met. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jim Fulton wrote: BTW, offline back ups if file-storage even with Blobs is straightforward without repozo. I couldn't parse this, could you restate particularly wrt to what ou mean by offline? If there's a way other than repozo, then it would be great to know what it was... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Wed, Dec 2, 2009 at 8:01 AM, Chris Withers ch...@simplistix.co.uk wrote: Jim Fulton wrote: BTW, offline back ups if file-storage even with Blobs is straightforward without repozo. I couldn't parse this, could you restate particularly wrt to what ou mean by offline? If there's a way other than repozo, then it would be great to know what it was... If you take the database down, you can use normal backup tools to back up the database file and blobs. Repozo gives you, for FileStorage: - Online backups -- you can back up a database while it is running - Incremental backups Of course, it isn't being actively supported. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jeff Shell wrote: I'm not sure whose expectations you're satisfying as the items in ZODB.scripts are a wild and inconsistent mess. It's been trial and error just to find ones that seem to work, outside of repozo. Oh the irony of the one script that Jim is looking to remove, because someone dared to do some maintenance on it ;-) Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Godefroid Chapelle wrote: Jim Fulton wrote: snip No one seems to be willing to step up and take responsibility. I guess everyone assumes I will. Jim I'll take the responsibility of transforming the manual test to an automated... Done in svn.zope.org/repos/main/ZODB/branches/test_repozo I added code changes from revisions 105888, 105915 and 105919. Tests pass for me on MacOSX with 2.4, 2.5 and 2.6 -- Godefroid Chapelle (aka __gotcha) http://bubblenet.be ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Godefroid Chapelle wrote: Godefroid Chapelle wrote: Jim Fulton wrote: snip No one seems to be willing to step up and take responsibility. I guess everyone assumes I will. Jim I'll take the responsibility of transforming the manual test to an automated... Done in svn.zope.org/repos/main/ZODB/branches/test_repozo I added code changes from revisions 105888, 105915 and 105919. Tests pass for me on MacOSX with 2.4, 2.5 and 2.6 Thanks for throwing yourself on that grenade, Godefroid. I have checked in changes to your branch which: - - Use a tempdir for backup / restore. - - Clean out module-scope imports. - - Avoid shellling out to run repozo, but rather use its main(), passing argv. - - Suppress noisy output, unless NOISY_REPOZO_TEST_OUTPUT is in the environment. Remaining cleanup would be to remove the Monte Carlo use of random to generate permutations: the test takes a *long* time to run, because it tries to run 100 times and use random.random to get permutations of the three choices run: these should be six separate tests, likely. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksWuYsACgkQ+gerLs4ltQ7LYwCg0AK9vHca71u7q0maPM86hY2l yzkAoIzgiGiEZLhcuG7vvTnHtTeuDbK2 =QKtA -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Godefroid Chapelle wrote: Godefroid Chapelle wrote: Jim Fulton wrote: snip No one seems to be willing to step up and take responsibility. I guess everyone assumes I will. Jim I'll take the responsibility of transforming the manual test to an automated... Done in svn.zope.org/repos/main/ZODB/branches/test_repozo I added code changes from revisions 105888, 105915 and 105919. Tests pass for me on MacOSX with 2.4, 2.5 and 2.6 Thanks for throwing yourself on that grenade, Godefroid. I have checked in changes to your branch which: - - Use a tempdir for backup / restore. Sure. - - Clean out module-scope imports. Can you tell me why this is better ? - - Avoid shellling out to run repozo, but rather use its main(), passing argv. I thought of this but decided to not go that way to test that repozo can actually run on a living database. Even if the code that was there still needs to be modified to achieve that goal. - - Suppress noisy output, unless NOISY_REPOZO_TEST_OUTPUT is in the environment. Cool ! Remaining cleanup would be to remove the Monte Carlo use of random to generate permutations: the test takes a *long* time to run, because it tries to run 100 times and use random.random to get permutations of the three choices run: these should be six separate tests, likely. I had thought of this as well. This will be my next task. Tres. - -- -- Godefroid Chapelle (aka __gotcha) http://bubblenet.be ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Wed, Dec 2, 2009 at 6:30 AM, Jim Fulton j...@zope.com wrote: On Tue, Dec 1, 2009 at 6:24 PM, Chris Withers ch...@simplistix.co.uk wrote: Jim Fulton wrote: ... I don't want people to contribute crap. You've made some reasonable contributions recently so I know you are capable of producing non-crap. I should not have used the word crap. I apologize. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Wed, Dec 2, 2009 at 12:42 PM, Godefroid Chapelle got...@bubblenet.be wrote: Godefroid Chapelle wrote: Jim Fulton wrote: snip No one seems to be willing to step up and take responsibility. I guess everyone assumes I will. Jim I'll take the responsibility of transforming the manual test to an automated... Done in svn.zope.org/repos/main/ZODB/branches/test_repozo Much thanks! And thanks to Tres for the updates. I added code changes from revisions 105888, 105915 and 105919. Thanks. Given the new test, I don't think the test added in 105915 and 105919 provides any benefit. I'll plan to merge this tomorrow. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Wed, Dec 2, 2009 at 10:09 AM, Jeff Shell j...@bottlerocket.net wrote: On Dec 2, 2009, at 5:39 AM, Jim Fulton wrote: On Wed, Dec 2, 2009 at 7:19 AM, Martin Aspeli optilude+li...@gmail.com wrote: Jim Fulton wrote: ... BTW, offline back ups if file-storage even with Blobs is straightforward without repozo. No, it's not. What do we use to check for integrity of the other backups? That is separate from backup and restore. FWIW, I'm pretty happy with the verification script, multi-zodb-check-refs, in the zc.zodbdgc package. It checks whether all objects reachable from database roots are loadable. It supports multi-database configurations and blobs. Of course, most people wouldn't be aware of this. It also has an option to provide a reverse reference database, which is useful for tracking down where problem objects are used. fsrefs? fstest? fsoids? Many of the ZODB Scripts are barely useful and certainly not 'straightforward'. I couldn't agree more. ... And I think it's funny that you give so much concern to Blobs when I have abandoned them in sheer frustration. And I recognize your concerns about angering / frustrating / losing users because a tool does not seem to work, because that's how I feel about Blobs. I'm not sure what problems you've had. Blobs should have been clearly labeled experimental until recently. I wasn't comfortable using them in production until 3.9 and I put a lot of effort into making them production quality in 3.9. FWIW, we are using them extensively now. I know some problems people have had were with their integration into applications, especially wrt copy and paste. There's only so much that ZODB itself can do about that. ... I don't think that any of the ZODB scripts are in good shape. There's nothing I can confidently give to a system administrator. There's an old Wiki page of documentation for a handful of the tools, and that's about it. At least 'repozo' has good and useful information to spit out when you use the '-h' / '—help' switch. I agree with the sentiment. There's a lot in ZODB that doesn't have the quality I'd like. I'm making progress. For example, I've improved zeopack this year. When I did, I also added tests. I'm not sure whose expectations you're satisfying as the items in ZODB.scripts are a wild and inconsistent mess. It's been trial and error just to find ones that seem to work, outside of repozo. I didn't add those scripts. I've been hesitant to remove scripts because I don't know who might be using them, or who might find them useful. In general, I'm working on bits of ZODB at a time, trying to clean up the bits I touch without necessarily demolishing everything I don't like. I do think some scripts are more important than others. If an analysis script is wrong, it's not a big deal. If backup (or packing) is wrong, that's a serious problem. I think the biggest issue that you're touching on is the lack of up to date and maintained documentation. This is the biggest problem for the ZODB community, IMO. It isn't the biggest ZODB issue for ZC, which is why I haven't gotten to it *yet*. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Tres Seaver wrote: - - Avoid shellling out to run repozo, but rather use its main(), passing argv. I thought of this but decided to not go that way to test that repozo can actually run on a living database. Even if the code that was there still needs to be modified to achieve that goal. Computing the path to repozo to run as a script was a bit wonky: I just changed the 'main()' function to take optional arguments, which also gave me more flexibility in how thigs run (e.g., turning off noisy output). I suspect I do not express myself correctly. I try to say that repozo should be tested against a database being mutated (the current test only exercise backups when the db is closed). As far as I can understand, THE reason for repozo is to backup without stopping the ZODB. -- Godefroid Chapelle (aka __gotcha) http://bubblenet.be ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Wed, Dec 2, 2009 at 5:34 PM, Godefroid Chapelle got...@bubblenet.be wrote: Tres Seaver wrote: - - Avoid shellling out to run repozo, but rather use its main(), passing argv. I thought of this but decided to not go that way to test that repozo can actually run on a living database. Even if the code that was there still needs to be modified to achieve that goal. Computing the path to repozo to run as a script was a bit wonky: I just changed the 'main()' function to take optional arguments, which also gave me more flexibility in how thigs run (e.g., turning off noisy output). I suspect I do not express myself correctly. I try to say that repozo should be tested against a database being mutated (the current test only exercise backups when the db is closed). That would definitely be much better, but not needed to test Chris' change I think. In any case, you don't need to shell out to test repozo against a database that is being modified. You could use threads, or you could write the database records yourself to, for example, cause a repozo backup while there are partial transaction records. I would prefer not to shell out. As far as I can understand, THE reason for repozo is to backup without stopping the ZODB. One of 2 reasons. The other is incremental backups. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
