Re: [ZODB-Dev] grab zodb dict keys by search term
ZODB has nothing built-in. You build it on top of the ZODB. I think what you are looking for is something like Hypatia - https://github.com/Pylons/hypatia On Tue, Dec 10, 2013 at 4:46 PM, Tamer Higazi tamerito...@arcor.de wrote: Hi people! I am working a lot with IOB and OOB Trees as well with PersistentDict and PersistentList to store my Data in ZODB. I want to ask if there is a way to grab data from a dataset by search time. I know only the SQL way: select lower(COL1) from table1 where COL1 LIKE 'aa%' so usually I want to get the entryies only that starts with 'aa' do I have to iterate the entire dict with an interator or are there builtin functions that could accomplish those tasks for me ?! I would kindly thank you Tamer ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Skype/Twitter:: runyaga Office:: 713.942.2377 ext 111 http://ploud.com/ Plone site in less than 10 seconds ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] make ZODB as small and compact as expected
Pretty sure this could be merged to zopefoundation project if anyone asked Carlos. On Tue, Jul 23, 2013 at 7:47 AM, Adam GROSZER agroszer...@gmail.com wrote: On 07/22/2013 01:27 PM, Jim Fulton wrote: - The biggest thing ZODB needs right now is documentation. Unfortunately, this isn't easy. There is zodb.org, but much better documentation is needed. There is https://github.com/cguardia/**ZODB-Documentationhttps://github.com/cguardia/ZODB-Documentation but seems like it got stalled 5 months ago. -- Best regards, Adam GROSZER -- Quote of the day: Each time you are honest and conduct yourself with honesty, a success force will drive you toward greater success. Each time you lie, even with a little white lie, there are strong forces pushing you toward failure. (Joseph Sugarman) __**_ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/**listinfo/zodb-devhttps://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Skype/Twitter:: runyaga Office:: 713.942.2377 ext 111 http://ploud.com/ Plone site in less than 10 seconds ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Broken objects problem.
Thanks for your response , i check the imports and seems to be ok but still not working. print sys.path from inside flask app and from the console script you run which *does* work. then compare both of these sys.path listings. -- Alan Runyan Skype/Twitter:: runyaga Office:: 713.942.2377 ext 111 http://ploud.com/ Plone site in less than 10 seconds ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeo server hot failover
So an error is raised and logged, but with the cache being cleared so that on the second try it reconnects? My rational for this change is that If your doing a hot failover that means that a) something bad has happened to main server and the recovery of those transactions probably won't happen any way or b) it happened during a maintenance window/the failover is happening for convenience and any difference between the two servers are probably minor, such as session data. With b. the server admin would be in the position to restore the main server anyways. Sounds reasonable. Does the drop rather than verify feature get verified when connecting to a new replica? Should it? -- Alan Runyan Skype/Twitter:: runyaga Office:: 713.942.2377 ext 111 http://ploud.com/ Plone site in less than 10 seconds ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] A little help in configuring a better solution for ZEO deployment
I use 4 processes with 7 threads each for serving data Any reason for 7 threads? Try 1 thread? Is it a classical web application? Or you creating threads in your application? WSGIDaemonProcess doba.ua user=www group=www processes=4 threads=7 maximum-requests=200 I have little space on /tmp partition, so the cache of ZEO is only 40 MB You can change the tmp location in Python by setting TMPDIR environment variable for your process. I would advise against persistent disk cache and increase the ZODB cache-size. If you are using latest ZODB you can use the cache-size-bytes which mostly works. You can set it so zodb cache for each process will not exceed, say, 512MB. I can't find any ther explanation, that ZEO does not handle the load well enough. If your ZEO server process is not bottlenecking - it is your application. Does someone see any way to ease the load? dont know your application but it is doubtful that ZODB is your bottleneck. -- Alan Runyan Skype/Twitter:: runyaga Office:: 713.942.2377 ext 111 http://ploud.com/ Plone site in less than 10 seconds ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage and MSQL
Windows 2008 server standard. Python 2.6/2.7 I'm not sure. I think living without clustering and memcached for the moment is fine. I would agree. And what about blob storage? Are blobs inside the DB in little chunks enough, do you want them on the filesystem via shared network drive or new efficient filestream support in SQL server 2008+ I don't know enough to answer this but for our purposes support SQL Server in a basic capacity so Plone runs ok is enough to start with. This really depends on usage patterns and capacity. I would recommend keeping the BLOBs on the filesystem. Our need is to 'state' support for SQL server to our client. They have SQL server licenses and they'd prefer to use them. The job is for 3 plone sites, public, extranet and intranet but with no estimate yet of the data size. Availability concerns are an issue. In the case where the work goes ahead and if performance is an issue then we;d be in a better position to either implement greater support for sqlserver or recommend they switch to a more mature relstorage option. Hope that helps put things in context. If you can depend on mxODBC then I do not believe this is a huge problem. I would be skeptical of pyodbc stability/performance - its worked ok for us but we dont have anything 24x7 running with it. As someone said earlier it is not difficult just very time consuming to test. I would be more than happy to help test. Some more thoughts: - If you run mxODBC you will have much less adoption/testing by the community. due to license and pain to install mxODBC. - pyodbc i would skeptical of and test. you would get the most usage using this driver. - pywin32 is another candidate. it should work just fine but you may need to watch for scaling issues (you may have to add some smarts the mssql storage) I would love to see this support added. -- Alan Runyan Skype/Twitter:: runyaga Office:: 713.942.2377 ext 111 http://ploud.com/ Plone site in less than 10 seconds ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to update an object in a multithreading application?
I am updating an item in a separated thread but when it finishes the new value is not updated in the same object in the main thread. You need to call sync() on the connection object to see changes from another thread. http://zodb.readthedocs.org/en/latest/api.html#connections In the example below root['counter'] starts in 0, is incremented in the new thread but it remains in 0 in the main thread. if your increment a counter its best the counter be of type, BTrees.Length cheers, alan ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB on ironPython?!
On Thu, Oct 20, 2011 at 12:39 PM, Tamer Higazi th9...@googlemail.com wrote: Hi! I am interisted getting ZODB to run on ironpython. Is that at the moment possible?! As I red, all python modules could be loaded on .NET without any problems as long they wouldn't need any compiled modules. Is that at the moment possible running ZODB on .NET ironpython?! Not until a pure Python implementation of persistent and BTree's exist. There has been substantial amount of work done but it more is required. runyaga ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Advice on whether to run relstorage database and Zope on different servers
We've been successfully running Plone 3.3 + Zope 2.10 + Relstorage + MySql for a couple of years now. We're running low on memory on our existing server (which has numerous other services on it) and will soon need to deploy an additional Plone 4.1 + Zope 2.13 + Relstorage + Mysql environment. We've been using virtual environments since September. For the purposes of this discussion please assume we don't have the option of increasing the memory on our existing server and so need to add a new one.Some questions I'd really appreciate your help with please: * I'm familiar with enterprise environments where you would have an app server and a database server but are there any advantages to putting Zope and MySQL on different servers? * I'd expect a performance hit if we run Zope + MySQL on separate servers but is this hit manageable? * Can we put anything in place to mitigate this? * Would you simply recommend having the existing mysql + zope on one server and the new mysql + zope on the other? Anthony this isnt a ZODB question; its a plone related application question. Ask plone-users. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 32-bit vs 64-bit - RelStorage on MySQL
Right, but, even if the MySQL server is 64-bit, if the clients are 32-bit, they won't have this problem, will they? if your running a 64bit python - you will see substantial memory growth. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage support to Microsoft SQLServer
I have read that there is a problem to implement MS-SQL adapter for Relstorage because the “Two phase commit” feature is not exposed by MS-SQL server . unsure about that. probably depends on the client access library. Is there solution to overcome this problem, Without introducing too many layers? Can we use PyMSSQL and ADODB Python extension to implement the relstorage Adapter for MS-SQL. i recently had a discussion with some guys about this. i am unsure what their analysis was. but my opinion: - adodbapi is not good. - pymssql i've not used - pyodbc we used but it doesnt support storedprocs. works ok. - mxodbc we use and highly recommend. yes mxodbc costs money but you have support. i spoke with shane about this in the past about which library would he probably use if he were to support mssqlserver and his unresearched/not definitive answer was mxodbc. mainly because its supported and has been in production usage for almost a decade. Has any one tried this before? no. Are there any other major concerns you would have for developing MS-SQL adapter component? shane would be best to answer this. Please point us in right direction on the subject. many of our customers use mssqlserver; we would be very interested in the outcome. so please keep the zodb mailing list up-to-date with your progress. my suggestion: use mxodbc. involve shane so he can fold it into the main relstorage line in the future. maybe even budget some money to pay him so he has interest in the outcome of the mssql/odbc adapter for relstorage. cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB Book Update
Thanks to Carlos and Jens we have the ZODB book being updated daily on zodb.org, http://www.zodb.org/zodbbook/ -- also have disqus integrated into sphinx html output. People can read the first three chapters and provide feedback at the bottom of each page. We are looking for feedback. If anyone is interested in contributing; maybe starting a recipe-style section for the book. We can get you account and you can start contributing on github (svn is supported). cheers alan runyan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Book Update
Thanks to Carlos and Jens we have the ZODB book being updated daily on zodb.org, http://www.zodb.org/zodbbook/ -- also have disqus integrated into sphinx html output. People can read the first three chapters and provide feedback at the bottom of each page. Posting comments is a pain. I have to type in my contact info each time. really? could be a bug in disqus? are you using disqus, fb, twitter, openid? which oauth provider are you using? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Can RelStorage be used as a ZEO Storage back-end?
I'm starting to dig into the code, but perhaps this can be answered quickly: can RelStorage be used in place of FileStorage for ZEO storage? My understanding is Yes. Try it. IIRC you can even use a ZEO storage in place of a FileStorage. You may want to specify what version of python, zodb you are using. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB replication and high-availability across data centers
we are currently investigating options for a large-scale solution where the machines (ZEO clients and ZEO server) should be separated across two data centers. Incoming traffic should be balanced between both data centers. Automatic failover and online-replication are desirable - high-performance is a must. The well-known options are: ZRS (ZEO) Relstorage on top of some RDBMS-related clustering/replication mechanism NEO DRDB ZEORaid We have not explicitly set something like this up; although our hosting partner has SAN replication between data centers. This is another solution; while low level - may be valid to explore. They use it with Oracle and MSSQLServer to do cross datacenter replication. cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
I went for the following in the end: ... cache-size 200Mb client app wait on /zeoclient mount-point / cache-size 5 cache-size-bytes 500Mb Any obvious issues with that? No. each zeoclient needs a name in your conf. required so the .zec files dont collide. this is what is happening to you. look at component.xml again for details on assigning zeo client a name or identifier. that will give the filename a suffix like something-$identifier.zec please open issue at launchpad; if this fails zope should NOT start. alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
I'm wondering if some of my issues might be alleviated by persistent client caches. anything is possible. Now, I have the impression that these were unreliable and buggy and had been so for some time. they appear to be more stable now. we use them on several customer's in very specific cases. mainly with infrequently modified databases. Is this still the case? If not, where are the docs on how to configure them nowadays? http://svn.zope.org/ZODB/tags/3.9.7/src/ZODB/component.xml?rev=117035view=auto key name=client description Enables persistent cache files. The string passed here is used to construct the cache filenames. If it is not specified, the client creates a temporary cache that will only be used by the current object. /description /key -alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Windows/ZODB Tests Passing
For anyone who cares. ZODB trunk builds win32 and amd64 have been passing tests for almost a week without failures. Thanks Jim for getting ZODB tests working reliably on Windows! cheers alan p.s. let us know if you need py2.7 builders and what to test. you can find all the builders at hudson.enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB Book Project
http://zodbdocs.blogspot.com/ - ZODB Book Project Hello ZODB-dev, I wanted to provide a status update for the upcoming ZODB book. We have achieved our initial financial goal to get an advance to Carlos de la Guardia. He will begin planning and writing. We need more money. Please take this opportunity to goto the blog, review the book outline, provide feedback and click on the paypal link and add to the pot. I will be contacting companies who are absent from the donor roster. Please contribute. Both individuals and companies can contribute by becoming authors on the blog and writing articles. I've gotten a fairly luke-warm reception from the community on writing articles. If you have any experience - let me know. Email me an article, I will put it up and credit you. This project will succeed only by pulling the resources together from our communities. http://zodbdocs.blogspot.com/ - ZODB Book Project cheers, alan runyan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Mixing meta data into names is a bad idea. You don't want to change the name of something just because some of its meta data changes. Your right. It's just that sometimes people may have a package name that is not agreeable to the broader community. The namespace could be, zodb. No it can't. ZODB isn't a namespace package and can't be one without breaking backward compatibility. Should it be a namespace package over the long term? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB Docs Blog Open for Contributions
http://zodbdocs.blogspot.com/ Unfortunately we could not get a 501c3 organization to accept donations for the project in a timely manner. Enfold Systems has wired up is paypal account and will take donations on behalf of the ZODB book. Once we get the 2k in donations; we will transfer the money to Carlos and the writing will begin. Carlos is expected to have a revised draft of material to be covered by Monday evening. If you are interested in contributing some writing / materials around ZODB. Please let me know. I can give anyone contributor access to the ZODB Docs blog so that interested parties can post recipes, howtos and thoughts around the ZODB. Carlos is open to have any submissions be integrated into the upcoming ZODB Docs book! We are open for business. Please contribute. cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Book
How to sign up as a volunteer? I have you signed up to volunteer- thanks! alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Book
Greate idea! Just, what happens if the goal isn't met? Plus, I'd really like to see docs on DirectoryStorage. If I can not get enough contributions for an initial advance for Carlos to begin work -- the project will be aborted. As we start getting contributions there will be more activity on the outline / material expected to be covered. alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB Book
Hello ZODB Community, I wanted to announce a community-led initiative to create a ZODB book. The goal is to solicit contributions from the community and when we reach a threshold, Carlos de la Guardia will begin work on writing the book. I have started a blog, http://zodbdocs.blogspot.com/ The blog contains the contribution levels and the proposed outline. Any feedback is appreciated. If you are interested in contributing financially please subscribe to the email list on the blog. I am working on getting a paypal account. I will keep the blog up to date and will ping the mailing list when we start accepting contributions. cheers alan runyan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
I suppose you could look at zeoraid. Hadn't thought of that, will look at it, thanks! Anyone using ZEORaid in production? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes)
On Tue, May 18, 2010 at 1:14 PM, Ryan Noon rmn...@gmail.com wrote:
Hi All,
I converted my code to use LOBTrees holding LLTreeSets and it sticks to the
memory bounds and performs admirably throughout the whole process.
Unfortunately opening the database afterwards seems to be really really
slow. Here's what I'm doing:
from ZODB.FileStorage import FileStorage
from ZODB.DB import DB
storage = FileStorage('attempt3_wordid_to_docset',pack_keep_old=False)
I think the file in question is about 7 GB in size. It's using 100 percent
of a core and I've never seen it get past the FileStorage object creation.
Is there something I'm doing wrong when I initially fill this storage that
makes it so hard to index, or is there something wrong with the way I'm
creating the new FileStorage?
Is there a 'index' file that is being created? It would be in the
same directory as the database file.
How are you closing the application?
If you see the index file changing when you start up; it is probably
rebuilding the index.
-alan
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Simple RelStorage Question
Does RelStorage handle concurrent writes correctly? This is probably so obvious as to ask if the sky is blue, but I couldn't tell from the documentation if I could setup multiple Zope (Plone in my case) instances all using RelStorage to talk to the same database without incident. relstorage handles concurrent writes. It is still possible to get ConflictError's. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes)
So, the best place to call those would be during my commit break (whenever I decide to take it? [which would be less often if I could be sure of no crashing]). Are there any other problems with the way I was using ZODB in my code? I really like it, but I recognize that it's a lot more complicated than my old system. Correct. Pick appropriate place where you are finished with a batch of objects and possibly have them no longer referenced. Then you can call cacheMinimize. You can call cacheGC to reduce memory usage. Your code looks straight forward. I do not see it doing anything strange. It's more complicated than old system? Ugh. That sucks. The ZODB should be LESS complicated than sqlite+custom rolled caching. If not than you may be doing something wrong or ZODB is not living up to its promise *wink* cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Author ZoDB Books - Packt Publishing.
I think a ZODB book would be a wonderful edition to Packt's NoSQL library. I really believe the ZODB book should *not* talk about any web framework integration. There is plenty of information to convey about ZODB without talking about integrating it into external projects. It would be great if the book, at some later date, would have opportunity to be released under CC license. The ZODB project needs external facing documentation. I just got back from a conference where I met one of the MongoDB guys. Obviously mongo is a different animal. The features I saw that were interesting: - Indexing/Querying support. - Ability to auto - (in/de)crement counters on records. - Language independent bindings for MongoDB If a ZODB book is written I would highly suggest detailing a indexing/querying strategy Even if it is use zope.catalog/zope.index with some examples. fwiw, alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] cache-size-bytes problem
Hello, So far I know Hanno fixed that. Any chance to get it released as 3.9.5? Absolutely, it will be released in 3.9.5. This will likely happen in 2 or 3 weeks. Good to hear. There are a few issues that seem to have patches, will those make it? - https://bugs.launchpad.net/zodb/+bug/544305 I think we can close this: - https://bugs.launchpad.net/zodb/+bug/114814 There are other low hanging fixes. Should someone create a patch? e.g. https://bugs.launchpad.net/zodb/+bug/239086 alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Splitting a large ZODB file storage - any light-weight approach?
we have some huge ZODB file storage (60-80 GB each) and we want to split the storage into a number of smaller storage (easier to handle for backup and packing). The storages themselves only contain a few (Zope 2) folders - let's say folders A, B, C - within the root and we want to create seperate storages for A, B, C. Is there some easier approach than using the Zope 2 CopyManager API for moving data around. The ZODB storage iterator API comes to my mind... is this a suitable approach or is there some other light-weight approach (compared to using CopyManager API)? ExportImport module? http://svn.zope.org/ZODB/trunk/src/ZODB/ExportImport.py At one point I suggested a feature for Zope 2 which would be Export as Mounted Storage. Which could generate a database with your exported content in it. So you could simply wire up the new storage in .conf file, delete the original and remount it with ZMI. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Data.fs size grows non-stop
I'd just like to add that there's some changes that can be related to this: - we had some classes inheriting from Persistent that now inherit from something else as well (but no extra arguments are being added, AFAIK); - we added some zope.interface definitions to some Persistent classes; maybe this causes some kind of behavior that we were not aware of? I doubt it. Thousnads of people are doing this and do not report the same behavior. What is more likely is that a programmer is changing a persistent object very often. One of the downsides of the ZODB is that it is so transparent it is possible to unwillingly make database changes. A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool. Often the READ pool comes from some replica and WRITE is to the master. I'm unsure this pattern would work for ZODB. I know Malthe was thinking about this but unsure if he had anything concrete. cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [OT] NoSQL
I am very intrigued now to setup an EC2 cluster and install a z3c.sharding based solution demonstrating 100M users with some data. Mmmh... That is the great thing about EC2. You can do massive experiments on the cheap. Actually one of our interns is doing some work on ZODB. He is doing mostly narrow calculations on efficiency of ZODB. i.e. how structure of ZODB Filestorage could be changed to better use disk cache, etc. Possibly interesting to the community at large. cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] instrumenting a ZEO server?
On Wed, Aug 26, 2009 at 8:08 AM, Chris Withersch...@simplistix.co.uk wrote: Hi All, I'm wondering if there's any way to get a ZEO server to log about how it's performing? (eg: how many connections, how many clients are waiting, where they're waiting, etc) I'm trying to debug a problem where app servers are being dropped out of a load balancer pool for being unresponsive, and I think they're waiting on the ZEO server so I'm trying to get some stats on the ZEO server side as to what's going on... As far as the clients go, DeadlockDebugger is providing some insights: File /opt/Zope-2.9/lib/python/ZEO/ClientStorage.py, line 749, in loadEx self._lock.acquire() # for atomic processing of invalidations I assume this is the client waiting for the storage server's lock? Is this the ZEO server's single threadedness or is this a lock between threads on the app server? File /opt/Zope-2.9/lib/python/ZEO/ClientStorage.py, line 769, in loadEx data, tid, ver = self._server.loadEx(oid, version) File /opt/Zope-2.9/lib/python/ZEO/ServerStub.py, line 192, in loadEx return self.rpc.call(loadEx, oid, version) File /opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py, line 531, in call r_flags, r_args = self.wait(msgid) File /opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py, line 638, in wait asyncore.poll(delay, self._singleton) File /usr/local/lib/python2.4/asyncore.py, line 122, in poll r, w, e = select.select(r, w, e, timeout) What does this mean? I'm guessing it's the actual transfer of data from the storage server to the app server, right? What version of ZODB are you using? If you have zeo on in debugging what you can see is if there are outstanding clients waiting to write while a transaction is already writing. It says something like # clients blocked while transaction in progress in zeo server log files. How many connections: iirc zrpc is connection oriented so you should have # of application clients connected to ZEO server. Or it may be # of clients * # of ZODB threads. How many clients are waiting: You should be able to see this in zeo log files Where they're waiting: deadlockdebugger or products.signalstack will show what the clients are doing. may want to strace the client and server to see if there is long pauses of inactivity. maybe your running up against a real ZODB bug that was fixed in later releases? look at ZODB 3.8 branch NEWS.txt good luck alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
On Thu, Jun 25, 2009 at 7:19 AM, Izak Burgeri...@upfrontsystems.co.za wrote: Benji York wrote: Have you looked at the contents of the files? Once you have an idea of what is being written you might be able to determine what is doing the writing. The big 6.3GB one contains pickles. Lots and lots of pickles. I suspect we're simply dealing with a process that creates a LOT of savepoints before finally committing. The developer of the application is out at the moment, when he gets back from his lunch-break/touch-rugby game I shall ask him about this. We have been seeing something similar on Windows. Where there are lots of zeo-cache files + .tbufs lying around. This does not happen all the time. I was presuming it had something to do with signal handling/windows/tempfile. We have not found a way to reproduce the problem but have seen it at several customer environments. -alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Transaction blocked waiting for storage
On Mon, Jun 22, 2009 at 10:06 AM, Andreas Jungli...@zopyx.com wrote: On 22.06.09 16:56, Pedro Ferreira wrote: Hello all, We occasionally get some messages like this in the logs: 2009-06-22T14:12:42 (30651/137.138.128.213:35787) Transaction blocked waiting for storage. Clients waiting: 1. But today we eventually got to: 2009-06-22T16:00:11 (30651/137.138.4.153:53416) Transaction blocked waiting for storage. Clients waiting: 204. Which lasted some minutes. During that time, our system seemed to hang, even if it was possible to connect to the db through the command line. It eventually went back to normal by itself. Any clue on what might have caused this? The reason is likely a long running transaction. Since the storage server is single-threaded, a long running transaction will block pending requests. Question: Will the ZEO server respond to LOAD requests from ZEO clients while WRITEs are pending? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Documentation
Would it be useful to have a document that starts detailing sections that are missing in ZODB/ZEO programming guide? i.e. - Blobs are not mentioned. - ZEO/Persistent disk cache not mentioned - xdb reference safetybelt - Documentation of various backend storages - Relstorage, Demostorage, Filestorage - MVCC is not mentioned - Persistence cache not described - savepoints vs. nested transaction - Diagnostic recipes (how to diagnosis your application) - Known Limitations How do you guys suggest we work on documentation? Would it be useful to come up with an outline of the target documentation? Then people could start filling in some details. Maybe Jim could then review instead of produce the documentation? cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Tue, May 26, 2009 at 1:58 PM, Shane Hathaway sh...@hathawaymix.org wrote: Jim Fulton wrote: FileStorage indexes can't be saved after they reach a certain size, where size roughly based on the number of objects. I need to find a way to fix this. It might be interesting to use SQLite for FileStorage indexes. With SQLite, we wouldn't have to store the whole index as a unit. Or possibly BDB? The indices appear to be fairly straight forward to store. I know people have had nightmares with BDB. We use it in our Enfold Proxy cache and it works fine. We have not had corruption or other issues that tend to come along with BDB usage. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
In ZODB 3.9, you can pack without doing garbage collection. This should at least buy you time, if not solve your problem (depending on how much garbage your application creates. What about zc.FileStorage? Does this support packing w/o GC? And does that work on versions of ZODB 3.9? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] analyze.py
Attached is a minimally changed analyze.py file, it adds a min, max size of class types. Maybe someone can help me. We have a customer whose running out of memory. We suspect it is content related, i.e. they uploaded a really big file and in some way the entire file is a single PData or some attribute. Now the object gets loaded into memory (or many of these objects) get loaded into memory and the python process runs out of memory. So one of the quesitons: - What is approx. the largest instance in the ZODB? So I added min/max to the script. It may be useful to others (it is attached). What are some debugging tricks/reports you guys use to inspect the database? i.e. it would be really nice to be able to query the state of the ZODB, i.e. show all records 1MB. IIRC this should be doable in relstorage. But it would also be nice to have some set of tools for Filestorage as well. alan runyan analyze.py Description: Binary data ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Non-zeo version length but ZMI says There are no non-empty versions.
reported, https://bugs.launchpad.net/zodb/+bug/361184 maybe chris can take a stab at it *wink* On Tue, Apr 14, 2009 at 8:03 AM, Jim Fulton j...@zope.com wrote: On Apr 13, 2009, at 9:34 PM, Chris Withers wrote: Hi All, I get the following error when trying to open a filestorage .fs file with ZODB3-3.9.0a11: File /ZODB3-3.9.0a11-py2.5-linux-i686.egg/ZODB/config.py, line 154, in open **options) File /ZODB3-3.9.0a11-py2.5-linux-i686.egg/ZODB/FileStorage/ FileStorage.py, line 185, in __init__ read_only=read_only, File /ZODB3-3.9.0a11-py2.5-linux-i686.egg/ZODB/FileStorage/ FileStorage.py, line 1560, in read_index h = fmt._read_data_header(pos) File /ZODB3-3.9.0a11-py2.5-linux-i686.egg/ZODB/FileStorage/format.py, line 147, in _read_data_header h = DataHeaderFromString(s) File /ZODB3-3.9.0a11-py2.5-linux-i686.egg/ZODB/FileStorage/format.py, line 231, in DataHeaderFromString return DataHeader(*struct.unpack(DATA_HDR, s)) File /ZODB3-3.9.0a11-py2.5-linux-i686.egg/ZODB/FileStorage/format.py, line 241, in __init__ Non-zero version length. Versions aren't supported.) ValueError: Non-zero version length. Versions aren't supported. However, if I open the same file in Zope 2.9.2 or Zope 2.11.0, the ZMI says theere are no non-empty versions. How do I get rid of this phantom version? That's a great question. :) We probably need to write some sort of utility to remove version records. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - zodb-...@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] http://svn.zope.org/ZODB/trunk/src/ZEO/zrpc/smac.py
has an unused import of sha which is triggered loads of dep warnings ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9
The tests were from ZODB trunk + Python 2.6 64bit on Windows. We could setup buildbot's for ZODB on Windows. One problem is.. it takes ages for it the tests to run; so it would need to be on its own VM. I think we could run the tests nightly.. where would you want the output? cheers alan p.s. no guarantees need to figure out how we could schedule them to run at night as to not interfere with our other builds On Tue, Apr 7, 2009 at 5:17 PM, Jim Fulton j...@zope.com wrote: On Apr 7, 2009, at 6:05 PM, Alan Runyan wrote: I was just reviewing what is pending for ZODB 3.9. https://bugs.launchpad.net/zodb/3.9/+bugs Looks like all of the bugs listed for 3.9 are fixed or fixes are known and pending. Is a ZODB 3.9 beta around the corner? According to changes.txt the first alpha of 3.9 was cut almost 1/2 year ago ZODB + Python 2.6 on Windows is appearently an open issue. Andreas Where are the buildbot's/test runners for ZODB? We just ran quick test and got the attached output. What platform and Python version was this for? I presume ZC has buildbot's for ZODB already running? No. Do you? I typically run tests on Linux and Windows before committing, usually for Python 2.4 and 2.5. Lately, also on Python 2.6 on Linux. I would love someone to run some buildbots. I don't want to set them up myself. :) Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - zodb-...@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB 3.9
Hi. I was just reviewing what is pending for ZODB 3.9. https://bugs.launchpad.net/zodb/3.9/+bugs Looks like all of the bugs listed for 3.9 are fixed or fixes are known and pending. Is a ZODB 3.9 beta around the corner? According to changes.txt the first alpha of 3.9 was cut almost 1/2 year ago cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
It seems like such an easy goal: autoincremental integers for a container. Is this such a problem because of the ZODB architecture? or lack there of? There are two database primitives that everyone appears to want: - autoincrementing integers for containers (tables) - indexes (not in application) The new generation of databases (couchdb, tokyo cabinet, hypertable, etc) autoincrementing is usually absent. Though these data containers usually have index support. cheers alan On Thu, Apr 2, 2009 at 7:48 AM, Stephan Richter srich...@cosmos.phy.tufts.edu wrote: On Thursday 02 April 2009, Adam GROSZER wrote: Yeah, uuid was my guess. Christian was not necessarily thinking uuid. I think the common algorithm is: x = random.randint(10**9) while str(x) in container: x += 1 Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. Zope Stephan Richter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - zodb-...@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] upload a file of 6 MB with the function manage_upload
This is not a ZODB question. Ask on Zope User's mailing list. 2009/4/1 Sandra elsand...@yahoo.fr: Hi all, I want to upload a file of 6 MB with the function manage_upload def manage_upload(self,file='',REQUEST=None): Replaces the current contents of the File or Image object with file. The file or images contents are replaced with the contents of 'file'. if self.wl_isLocked(): raise ResourceLockedError, File is locked via WebDAV data, size = self._read_data(file) content_type=self._get_content_type(file, data, self.__name__, 'application/octet-stream') self.update_data(data, content_type, size) if REQUEST: message= Saved changes. return self.manage_main(self,REQUEST,manage_tabs_message=message) in python/OFS/image.py. But my Programm run without end. I'm making some mistake ? Anybody knows any way to solve it ? N.B: i am using Relstorage + Oracle 10i. Thanks in advance ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - zodb-...@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with _p_mtime
We run consistency checks on the ZODB - the same as we do on SVN -- nightly. http://svn.zope.org/ZODB/trunk/src/ZODB/scripts - fstest.py - checkbtrees.py - fsrefs.py We have never had corruption in years. But I just wanted to ask what people are doing to find any problems with consistency; so the problem can be solved ASAP. Miles, have you run any of these scripts across your production ZODB? cheers alan On Thu, Dec 4, 2008 at 12:45 PM, Dieter Maurer [EMAIL PROTECTED] wrote: Miles wrote at 2008-12-4 13:39 +: I've moved a FileStorage from one (old) machine to another (new) machine, but when I mount it on the new machine I get a lot of time errrors: Traceback (innermost last): Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module App.special_dtml, line 176, in _exec Module DocumentTemplate.DT_Let, line 76, in render Module DocumentTemplate.DT_In, line 703, in renderwob Module DocumentTemplate.DT_With, line 76, in render Module DocumentTemplate.DT_Var, line 214, in render Module App.PersistentExtra, line 43, in bobobase_modification_time Module DateTime.DateTime, line 509, in __init__ Module DateTime.DateTime, line 760, in _parse_args Module DateTime.DateTime, line 437, in safelocaltime TimeError: The time 98040302366.810165 is beyond the range of this Python implementation. I expect that your storage is damanged and contains some garbage at a place where a serial should be. This could probably be fixed -- but other parts of your storage might be damanged, too. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] __setstate__ semantics
I would not imagine this functionality is not in scope of the ZODB. We were thinking about piggy backing on top of the invalidation messages sent from ZEO server at one point.. cant remember why we gave up on it. For various reasons: - The object may not be loaded on the other zeo client; therefore why should it be notified? - If you have a lot of changes it could generates oodles of messages; significantly degrading ZODB performance. The best thing is to look into a messaging library. AMQP is pretty rockin. I spoke w/ Stefan Richter and they are using Amazon's messaging platform. I believe what you are looking for is some sort of application level messaging across ZEO clients. Kapil has started an implementation at repoze.queue (I would prefer the name repoze.messaging) which may not yet have an implementation other than the interfaces and some tests. But this is a known problem (which is very hard to get right). cheers alan On Mon, Oct 20, 2008 at 2:36 PM, Roché Compaan [EMAIL PROTECTED] wrote: If an object is modified on one ZEO client, will this always lead to a __setstate__ call on this object on another ZEO client? If not, is there any event or method that I can hook into to determine if an object was modifed by another ZEO client? -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB Blob Question
Hi guys. At the plone sprint we started reviewing some code that uses Blob. The code that we reviewed is working in production although I wanted to be clear about the expected usage of Blob. ZODB/tests/blob_consume.txt says that you *must* close a file before calling consumeFile(). It seems the current Plone Blob implementation *does not close the file* before calling consumeFile. And all appears to be working. Why must the file be closed before calling consumeFile? And would you go as far as saying that if you *do not close* the file before calling consumeFile - all bets are off. You are not using Blob as intended? cheers, alan runyan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zope memory usage
There was a recent modification to limit the ZODB cache to a set size. i.e. Limit the size of memory usage to 128MB. The original feature was implemented here: http://svn.zope.org/ZODB/branches/dm-memory_size_limited-cache/ You can get the feature+3.8 branch of the ZODB from: http://svn.zope.org/ZODB/branches/zcZODB-3.8/ The changes are also on trunk (will be ZODB 3.9). The goal of the modification is to try to put upper bounds on the size of the database cache. Before this size limited cache the cache would grow and if you had very large objects in the zodb cache -- your memory usage would be surpringsly high. Please use this feature in your testing at upfront. And let Roche know this feature has landed (if we was not monitoring svn checkins). The feature needs to be test driven. Give it a go and report your experience. cheers alan On Wed, Sep 17, 2008 at 5:10 AM, Izak Burger [EMAIL PROTECTED] wrote: Hi all, I'm sure this question has been asked before, but it drives me nuts so I figured I'll ask again. This is a problem that has been bugging me for ages. Why does zope memory use never decrease? Okay, I've seen it decrease maybe by a couple megabyte, but never by much. It seems the general way to run zope is to put in some kind of monitoring, and restart it when memory goes out of bounds. In general it always uses more and more RAM until the host starts paging to disk. This sort of baby-sitting just seems wrong to me. It doesn't seem to make any difference if you set the cache-size to a smaller number of objects or use a different number of threads. Over time things always go from good to bad and then on to worse. I have only two theories: a memory leak, or an issue with garbage collection (python side). It is possible that this is not a bug, my reasoning goes like this: The OS exposes a virtual memory picture to the application, ie, the application does not know how much of the RAM is real and how much is paged out. All it does is call malloc every now and then. Combined with a kernel that allows overcommit (which is in general a good thing), I see this going only one way. I'm hoping someone on this list has some insights into the issue. regards, Izak ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] wrap-up notes of sprint possible?
hey guys.. i know everyone is busy but it would be really swell if someone could write up a bullet list of things discussed/done at sprint. Things I'm curious about: - relstorage patch - zeoraid patch - transaction module (move to threading.local) https://bugs.launchpad.net/zodb/+bug/239086 - better import branch http://svn.zope.org/ZODB/branches/ctheune-betterimport/ cheers -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] what could cause this kind of cache corruption?
Chris, Are you seeing this error? https://bugs.launchpad.net/zodb/+bug/114814 If so maybe Jim's latest changes fixed this in ZODB3.8.1b65 -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Runaway cache size
I believe it is common to have a theshold (say every 1) objects to do a savepoint and then you can run cacheMinimize after the savepoint. surprised the zc.catalog does not provide this sort of knob. maybe it was a problem in the original ZCatalog implementation? cheers alan On Wed, Jul 30, 2008 at 1:21 PM, [EMAIL PROTECTED] wrote: I have a database consisting mainly of an IOBTree of ~700,000 items (each persistent mappings), and an zc.catalog indexing them by one of the mapping's fields (a date). I am opening the FileStorage in read_only mode. For each day in the index, I get the day's mappings and read the contents of another field. As I go through each day, the memory usage explodes (over 32 GB). Is there a way to configure the cache to automatically keep itself under the value of the cache size parameter? To avoid this problem in most cases, I wrap the IOBTree in another object which does nothing more than call db.cacheMinimize after every 1 items are iterated over. But for random access, that's not an option. From reading the archives, it sounds like cache cleaning does not happen while running a transaction. Is that my problem? I'm in read-only mode, so I can't perform a transaction anyway, but could the ZODB think I want to? Or is my problem something else? Maybe if someone can point me to a description of how the caching works. -- Anthony Foglia Princeton Consultants (609) 987-8787 x233 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB + ZEO performances problem
- sometimes (but most often under quite heavy load) the Zope clients completely hang and stop responding, without any access or error log, and without any disk or CPU activity. I suppose it's a kind of deadlock, how can I detect the source of the problem ? this isnt a zodb specific question. you will probably get more help moving this to the zope mailing list. check out DeadlockDebugger: http://highspeedrails.com/Support/Howto/deadlockDebuggerHowto cheers -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Relstorage
This appears to be a CMFEditions problem not a RelStorage problem. The ways to escalate a bug: Write a test case that demonstrates the bug. Do it yourself. a. Ensure tests fail reliably b. Email the author with failing tests cases and software versions to allow him to reproduce c. Hope that arthur cares enough to run your tests on the software packages and fixes it. I have a feeling its CMFEditions because he is trying to do fancy stuff with the transaction note(s) in the ZODB. At a gut level: I feel 100x more comfortable with Shane's code (ZODB/RelStoage) than CMFEditions. Which is notoriously insane. alan On Wed, Apr 23, 2008 at 4:10 AM, Ben Mason [EMAIL PROTECTED] wrote: We're currently benchmarking RelStorage to see if it's a viable alternative to ZEO. We've come across the following problem: When trying to revert to an old version of a page, we get a unicode error: Traceback (innermost last): Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module Products.CMFCore.FSPythonScript, line 140, in __call__ Module Shared.DC.Scripts.Bindings, line 313, in __call__ Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec Module Products.CMFCore.FSPythonScript, line 196, in _exec Module None, line 22, in revertversion - FSPythonScript at /plone/revertversion used for /plone/front-page - Line 22 Module Products.CMFEditions.CopyModifyMergeRepositoryTool, line 292, in save Module Products.CMFEditions.CopyModifyMergeRepositoryTool, line 454, in _recursiveSave Module Products.CMFEditions.ArchivistTool, line 288, in save Module Products.CMFEditions.ZVCStorageTool, line 232, in save Module Products.CMFEditions.ZVCStorageTool, line 391, in _applyOrCheckin Module Products.CMFEditions.ZVCStorageTool, line 505, in _encodeMetadata UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128) All that has been done here is... Click Edit on the front page of a clean Plone site, click Save. Go to the History tab and click on 'Revert to this revision' This doesn't happen if we're using the traditional Data.fs way of storing data. Anyone know what is going on? Thanks Ben ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO Client deadlocking in asyncore.poll - how to I debug
check out zeo server log files. a known problem is people using iptables or some sort of filtering between ZEO clients and ZEO server. this config took several hours off my life ;-( On Mon, Apr 7, 2008 at 9:16 AM, Anton Stonor [EMAIL PROTECTED] wrote: We have a setup with a ZEO server and 4 ZEO clients. During the last weeks we have seen almost daily deadlocks in some of the ZEO clients. I've tried to wait for up to 30 minutes before restarting a client. I could need an advice on how to debug this. With DeadlockDebugger I see the same pattern each time: One thread is hanging: File /usr/local/www/zope-2.9.6/lib/python/ZODB/Connection.py, line 732, in setstate self._setstate(obj) File /usr/local/www/zope-2.9.6/lib/python/ZODB/Connection.py, line 768, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File /usr/local/www/zope-2.9.6/lib/python/ZEO/ClientStorage.py, line 746, in load return self.loadEx(oid, version)[:2] File /usr/local/www/zope-2.9.6/lib/python/ZEO/ClientStorage.py, line 769, in loadEx data, tid, ver = self._server.loadEx(oid, version) File /usr/local/www/zope-2.9.6/lib/python/ZEO/ServerStub.py, line 192, in loadEx return self.rpc.call(loadEx, oid, version) File /usr/local/www/zope-2.9.6/lib/python/ZEO/zrpc/connection.py, line 531, in call r_flags, r_args = self.wait(msgid) File /usr/local/www/zope-2.9.6/lib/python/ZEO/zrpc/connection.py, line 638, in wait asyncore.poll(delay, self._singleton) File /usr/local/lib/python2.4/asyncore.py, line 122, in poll r, w, e = select.select(r, w, e, timeout) The other threads of the ZEO client are waiting for the hanging thread to release the storage lock so that they can acquire it: File /usr/local/www/zope-2.9.6/lib/python/ZEO/ClientStorage.py, line 760, in loadEx self._load_lock.acquire() When I connect to the ZEO server monitor I can see an increasing number of reads (probably from the other ZEO Clients). I've set transaction-timeout 15. How to I dig further to resolve this? zeo.conf partly below: -- zeo address 8200 read-only false invalidation-queue-size 100 # pid-filename $INSTANCE/var/ZEO.pid monitor-address 8201 transaction-timeout 15 /zeo filestorage 1 path $INSTANCE/var/Data.fs /filestorage %import tempstorage temporarystorage temp name temporary storage for sessioning /temporarystorage -- Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Hooking persistent.Persistent.__setstate__ was Re: [ZODB-Dev] Analyzing a ZODB.
Here is something more ZODB and less Zope related (kinda). I was talking with Benji a few weeks ago about a problem that should be easy to debug but was not. Here is the scenerio: - Customer has software on a remote machine. They are seeing unnecessary transaction commits. Just like the guy 'Analyzing a ZODB'. - Customer is completely incapable of doing anything other than putting a script on the filesystem. - Benji and I thought about it and he proposed 'the simplest thing that c/should work, monkey patch persistent.__setstate__' so that I could see what objects were being mutated. - Unfortunately ZODB 3.x does not have a Python fallback of persistent.Persistent -- its in C. The customer did not have a C compiler on their box. IIRC how I solved it was increase ZEO event log to see the oid's. Then I walked him through loading the oid up to see what object was being mutated. It was more painful than it should have been. Question: Is it possible for ZODB 3.9 to have a pure python implementation of persistent.Persistent? Maybe this would be a good ZODB GSOC project? Notes: - ZODB 4.x did have a pure python impl of persistent.Persistent but that project did not make it into production. - Seemingly Jim F. said there was a 'best practice' in writing C extensions in Python, first write reference impl in Python and to write tests around the impl. Then re-implement in C. ZODB 3.x, never had this 'best practice' put into play. - increasing zeo server log level and watching oid's being changed is sort-of the equivalent of turning on RDBMS logging to see SQL stmt's being executed. Unfortunately I believe without having a hook in persistent.Persistent we can never really get that level of granularity (i.e. __getattribute__ is only accessible in client) with only ZEO server logs. - We could monkey patch the subclasses that use persistent. But in the world of non-Zope2 applications usually there are very thin classes that may not have any common base class other than persistent.Persistent. This is a transparent plea for a new feature. But I believe it would significantly help people writing ZODB applications. Maybe people should always have a base class that you override those methods which delegate to persistent.Persistent. i.e. class MyMixin(persistent.Persistent) and mixin MyMixin instead of mixin persistent.Persistent directly? Then you can instrument the MyMixin with the logging? Another GSOC idea: Best Practices of ZODB Programming. cheers -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Hooking persistent.Persistent.__setstate__ was Re: [ZODB-Dev] Analyzing a ZODB.
Here is something more ZODB and less Zope related (kinda). I was talking with Benji a few weeks ago about a problem that should be easy to debug but was not. Here is the scenerio: - Customer has software on a remote machine. They are seeing unnecessary transaction commits. Just like the guy 'Analyzing a ZODB'. - Customer is completely incapable of doing anything other than putting a script on the filesystem. - Benji and I thought about it and he proposed 'the simplest thing that c/should work, monkey patch persistent.__setstate__' so that I could see what objects were being mutated. - Unfortunately ZODB 3.x does not have a Python fallback of persistent.Persistent -- its in C. The customer did not have a C compiler on their box. IIRC how I solved it was increase ZEO event log to see the oid's. Then I walked him through loading the oid up to see what object was being mutated. It was more painful than it should have been. Question: Is it possible for ZODB 3.9 to have a pure python implementation of persistent.Persistent? Maybe this would be a good ZODB GSOC project? Notes: - ZODB 4.x did have a pure python impl of persistent.Persistent but that project did not make it into production. - Seemingly Jim F. said there was a 'best practice' in writing C extensions in Python, first write reference impl in Python and to write tests around the impl. Then re-implement in C. ZODB 3.x, never had this 'best practice' put into play. - increasing zeo server log level and watching oid's being changed is sort-of the equivalent of turning on RDBMS logging to see SQL stmt's being executed. Unfortunately I believe without having a hook in persistent.Persistent we can never really get that level of granularity (i.e. __getattribute__ is only accessible in client) with only ZEO server logs. - We could monkey patch the subclasses that use persistent. But in the world of non-Zope2 applications usually there are very thin classes that may not have any common base class other than persistent.Persistent. This is a transparent plea for a new feature. But I believe it would significantly help people writing ZODB applications. Maybe people should always have a base class that you override those methods which delegate to persistent.Persistent. i.e. class MyMixin(persistent.Persistent) and mixin MyMixin instead of mixin persistent.Persistent directly? Then you can instrument the MyMixin with the logging? Another GSOC idea: Best Practices of ZODB Programming. cheers -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zeo Server as a single point of failure
in that vein, DirectoryStorage-behind-ZEO has worked perfectly for years... but this new stuff is very interesting! wishing we could get DirectoryStorage into svn.zope.org ~alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO+MultipleClients+ConflictErrors
Could you provide a bit more guidance? Maybe an example. Most people come at ZODB with previous experience in RDBMS. How do they map SQL INSERT/UPDATE activities to ZODB data structures? In a way that does not create hotspot. A few years ago people would say pick a BTree implementation that maps to your data structures and use that. But that is naive and pushes the hotspot into the BTree where concurrent updates are happening. cheers alan On Tue, Feb 26, 2008 at 12:42 PM, Benji York [EMAIL PROTECTED] wrote: Kenneth Miller wrote: Is there a standard bit of documentation as to how to handle a ZEO server with multiple clients writing to the same data structure at the same time? Not documentation, but there is a standard bit of snark that might be helpful: don't do that. ;) In other words, the best way to deal with a hot spot is to eliminate it. That may or may not be possible, but it's a good thing to try. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO+MultipleClients+ConflictErrors
Kenneth, I believe you will want to look into 'conflict resolution'. Possible source of inspiration: http://www.zope.org/Documentation/Articles/ZODB2 -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Jim, What would you consider a large active database? curiously, alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Andreas, Could you try to mount the catalog separately? My understanding is the catalog is what makes storages a misery. CMF/portal_catalog mounted as Filestorage and CMF could be mounted as PGStorage. I presume you would see much more reasonable performance? cheers -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Likely but I don#t know how to setup my instance in order to make the copied instance making use of mounted catalog. What about creating a Plone site with a FileStorage. Then mount a subfolder into PGStorage say /Plone/some_folder then you could see create af older /Plone/new_folder and dump buncha content and paste it into /Plone/foo_folder and paste it into /Plone/some_folder and see the difference? If there is a benefit..what would be arguments for running a mixed setup? My understanding is that each object in a catalog (Plone has 3 cataogs) has numerous other objects associated with it. Something like: each record in a catalog may have an object for each index/metadata attribute you are capturing. and possibly a few others. Each catalog entries contain ts of object per object being indexed.That is my understanding. -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack requirements?
You are running zeopack on the production machine with many clients? Are you running the zeoserver on the same machine, with many clients? It seems it is the machine that ZEO server is on that is throwing the MemoryError. I do not believe zeopack will bring the entire database into memory at once. It may/ probably brings the index into memory. Suggestion: run zeopack and zeoserver on a separate machine or allocate more memory to your zeoserver process. Zope NOTE: Running zeo clients with zserver-threads 2 is common and rarely does this impact performance. you will consume much less memory per zeo client than the 'out of the box' zope.conf which specifies zserver-threads 4 (I am -1 on this default) cheers alan On Dec 17, 2007 6:36 AM, Chris Bainbridge [EMAIL PROTECTED] wrote: I have a ~9GB live ZEO database. Last night I tried to pack it, but got this error: 2007-12-17T01:27:08 (4179) Error raised in delayed method Traceback (most recent call last): File /home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZEO/StorageServer.py, line 1016, in run result = self._method(*self._args) File /home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZEO/StorageServer.py, line 351, in _pack_impl self.storage.pack(time, referencesf) File /home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/FileStorage/FileStorage.py, line 1352, in pack opos = p.pack() File /home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py, line 482, in pack self.gc.findReachable() File /home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py, line 228, in findReachable self.findReachableAtPacktime([z64]) File /home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py, line 315, in findReachableAtPacktime self.reachable[oid] = pos File /home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/fsIndex.py, line 84, in __setitem__ tree[key[6:]] = value MemoryError 2007-12-17T01:27:16 (7412) opening storage '1' using FileStorage 2007-12-17T01:34:25 datastore.fs truncated, possibly due to damaged records at 8580513351 2007-12-17T01:34:25 Writing truncated data from datastore.fs to datastore.fs.tr0 Are there any limits on the pack operation? I assume that it doesn't have to hold the entire DB in memory at once? Would bringing the database offline and then packing it work (ie. we run out of memory because we have many active clients), or is there some fundamental constraint in the underlying code? Thanks for your help, Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack requirements?
Makes sense. Is zc.FileStorage an alternative packing strategy? Does it use less memory? http://svn.zope.org/zc.FileStorage/dev/src/zc/FileStorage/ cheers alan On Dec 17, 2007 8:29 AM, Jim Fulton [EMAIL PROTECTED] wrote: On Dec 17, 2007, at 9:14 AM, Alan Runyan wrote: ... not believe zeopack will bring the entire database into memory at once. It may/ probably brings the index into memory. Suggestion: run zeopack and zeoserver on a separate machine or allocate more memory to your zeoserver process. zeopack doesn't consume any reasources to speak of. It just tells the *server* to pack. The resources are consumed by the server. Jim -- Jim Fulton Zope Corporation -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Updating ZEO cache file magic number to get rid of version info
+1 On Nov 9, 2007 9:46 AM, Jim Fulton [EMAIL PROTECTED] wrote: I'm studying the ZEO cache-file implementation. I'd like to remove version support from the trunk. Would anyone object if I changed the magic number on the trunk (3.9) to reflect that cache data records no- longer contain version information? This would mean that persistent cache files created with 3.8 and earlier wouldn't work with 3.9. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Recovering from BTree corruption
And, as you said in another node, the BTree folder actually loves in
the resources database.
Correct the BTree is in /plone/resources/files to be exact.
Cross database references are inherently weak. A reference from a
foreign database doesn't prevent an object from being treated as
garbage. So, if the only reference to an object is from a foreign
database, then the object is considered garbage. It doesn't sound
like this is what's affecting you. The cross-database reference is
to the BTree. It sounds like the internal references are within
database.
Well. Someone could have 'copy/pasted' a file from the content database
into the resources/files database. That could have been one issue.
- checkbtrees.py
- fstest.py
There's an fsrefs script that checks internal references I believe.
fsrefs.py shows loads of problems in both the data.fs and the resources.fs.
probably 200 entries per database. i.e.
oid 0xD87110L BTrees._OOBTree.OOBucket
last updated: 2007-09-04 14:43:37.687332, tid=0x37020D3A0CC9DCCL
refers to invalid objects:
oid ('\x00\x00\x00\x00\x00\xb0+f', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xb0N\xbc', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xb0N\xbd', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xd7\xb1\xa0', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xc5\xe8:', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xc3\xc6l', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xc3\xc6m', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xcahC', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xaf\x07\xc1', None) missing: 'unknown'
My questions are:
- I imagine if there are 'invalid' references this is considered corruption
or inconsistency?
- How do I tell if something is a reference to another database?
- Having these invalid references, is this common to ZODB applications?
Possibly, there's a backup that has data records for the missing OIDs.
Going to ask hosting company to pull up backups for the past few weeks.
But how i'm going to find this other than seeing if the folder allows me
to iterate over the items is not throwing POSKeyError. Does that sound
like a decent litmus test?
--
Alan Runyan
Enfold Systems, Inc.
http://www.enfoldsystems.com/
phone: +1.713.942.2377x111
fax: +1.832.201.8856
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Recovering from BTree corruption
Hi guys. It seems that one of our customers has a corrupted BTree. I would love for someone to provide some insight on how we can recover the data. we have two databases: 1 for resources and 1 for 'content'. resources contain lots of very big files. The system is configured to have a mount point at /plone/resources is a subclass of BTreeFolder, using internal data struct of OOBTree. anytime I iterate over the keys I get POSKeyError. anytime I iterate over the values the same. if I run BTree.check() on the data structure's tree attribute (the OOBTree itself) I get a POSKeyError. Running the utils.checkbtrees doesnt say this btree has a problem. While debugging this I had a conversation with sidnei about mounted databases. He recalled that if your using a mounted database you should not pack. If for some reason your mounted database had a cross reference to another database and somehow you had a dangling reference to the other database it would cause POSKeyError. Is there any other ways of testing consistency of FileStorage other than: - checkbtrees.py - fstest.py And any ideas how I can salvage the data? This BTree, of course, had the most valuable data. cheers -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Memory Errors reading large ZODB
snip... line 96, in __setstate__ Persistent.__setstate__(self, state) MemoryError means you have run out of memory. check your jail or how much memory you have allocated. you can call cache minimize after a threshold.. maybe every 100 iterations. there are finer grain mechanisms to do this but i cant remember them off the top of my head. http://svn.zope.org/ZODB/trunk/src/ZODB/interfaces.py cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] exporting zope data as sql
This is not the appropriate forum; please use zope-users mailing list. On 5/17/07, Freezin Monkie [EMAIL PROTECTED] wrote: Hi all, I just joined the mailing list. I'm trying to export data stored in zope as sql. Is this possible and what's the best way to do this. Also is there a way to restrict which data is exported, i'm interested in exporting the data store through a particular zope product only. _ The next generation of Hotmail is here! http://www.newhotmail.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] zeo bug
Just got this on a website, Zope 2.9.6. Looks like /tmp is 90% full, not sure that would cause the problem? Not using persistent disk cache: zodb_db main mount-point / # ZODB cache, in number of objects cache-size 2 zeoclient server $HOST:$PORT storage 1 name $MAIN_NAME var $INSTANCE/var # ZEO client cache, in bytes cache-size 1024MB # Uncomment to have a persistent disk cache #client zeo1 /zeoclient /zodb_db zodb_db catalog mount-point /audioholics/portal_catalog container-class Products.CMFPlone.CatalogTool.CatalogTool cache-size 5 zeoclient cache-size 1024MB server $HOST:$PORT storage 2 name $CATALOG_NAME var $INSTANCE/var /zeoclient /zodb_db KeyError Sorry, a site error occurred. Traceback (innermost last): * Module ZPublisher.Publish, line 196, in publish_module_standard * Module Products.PlacelessTranslationService.PatchStringIO, line 34, in new_publish * Module ZPublisher.Publish, line 146, in publish * Module Zope2.App.startup, line 222, in zpublisher_exception_hook * Module ZPublisher.Publish, line 106, in publish * Module ZPublisher.BaseRequest, line 366, in traverse * Module ZODB.Connection, line 732, in setstate * Module ZODB.Connection, line 786, in _setstate * Module ZODB.serialize, line 604, in setGhostState * Module ZODB.serialize, line 597, in getState * Module ZODB.serialize, line 471, in _persistent_load * Module ZODB.serialize, line 537, in load_oid * Module ZODB.Connection, line 201, in get * Module ZEO.ClientStorage, line 746, in load * Module ZEO.ClientStorage, line 774, in loadEx * Module ZEO.cache, line 293, in store * Module ZEO.cache, line 980, in add * Module ZEO.cache, line 915, in _makeroom KeyError: 131867244 (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: already have current data for oid) -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] another reason to stop supporting versions
+1 for removing versions. they have been considered bad practice for over 3 years. I believe 100% its ok to remove them. alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Cleaning up storage interfaces
I just checked out DirectoryStorage and grepped for getSerial - no occurances. +1 for cleaning up the storage interfaces.. was looking at it over xmas holidays to see if I could implement a sqlite version that was undoless. I didnt get far ;( alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RESOLUTION: Re: [ZODB-Dev] more lockup information / zope2.9.6+zodb 3.6.2
Thanks to Jim, Theune, Dieter and all others who weighed in on this thread. The problem: ZEO Clients would lock up randomly requiring restart. The hints: Lots of 'Connection timed out' and 'No route to host' in ZEO Server log files. The solution: The machine the ZEO server was running iptables. Even though it was allowing the zeo server port 8100; it was filtering on all interfaces. I simply changed rules so port 8100 would not filter the internal network interface - at all. ZEO listens specifically on internal ip/port 8100. System has been stable for several days; without zeo server logs containing any connection errors. I had never had this problem before because.. well.. we use firewalls on our customers (external interfaces and our internal interfaces never have filtering) and this particular sysadmin was running iptables on all public/private interfaces; locking the machine down as much as possible. Thanks again guys! -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] more lockup information / zope2.9.6+zodb 3.6.2
Storage: 1 Server started: Wed Apr 11 10:56:50 2007 Clients: 10 Clients verifying: 0 Active transactions: -1 Huh? You're owing the system a transaction. However, by looking at the code briefly, this might happen if tpc_abort() and _abort() kind of overlap. And you did have two aborts at that point in time. Sounds like a bug/race that needs to be looked into in ZEO. Ah. The different clients might be because you have two storages and your ZEO clients are configured in a way not to connect to the exactly same storages? Or they are but they weren't able to. (See hardware/network problems.) They are both defined in zope.conf. All 12 clients were restarted last night: Just now I'm seeing: Storage: 1 Server started: Wed Apr 11 10:56:50 2007 Clients: 12 Clients verifying: 0 Active transactions: -1 Commits: 92 Aborts: 2 Loads: 498120 Stores: 2279 Conflicts: 0 Conflicts resolved: 20 Storage: 2 Server started: Wed Apr 11 10:56:50 2007 Clients: 11 Clients verifying: 0 Active transactions: 0 Commits: 51 Aborts: 0 Loads: 225080 Stores: 6408 Conflicts: 0 Conflicts resolved: 167 Something that came to my mind that might block the ZEO server for a long time are hard disk failures. Check your dmesg log. However, the network errors you see in various places really need to be tracked down. nothing in dmesg. I find the 'No route to host' disturbing although these have not happened over the past 24 hours. This has: 2007-04-12T00:17:45 ERROR ZEO.zrpc.Connection(S) (172.16.235.120:54881) Error caught in asyncore Traceback (most recent call last): File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 69, in read obj.handle_read_event() File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 391, in handle_read_event self.handle_read() File /usr/local/zope/lib/python/ZEO/zrpc/smac.py, line 147, in handle_read d = self.recv(8192) File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 343, in recv data = self.socket.recv(buffer_size) error: (110, 'Connection timed out') -- which is frustrating. as i understand zeoserver is taking too long to communicate to zeoclient and zeoclient times out. shouldnt it retry / reconnect? alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] more lockup information / zope2.9.6+zodb 3.6.2
ETHERNET CARD: 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11) - I have turned down transaction timeout to 15 seconds. - I currently have 11 ZEO Clients up; but showing below (seems strange) sometimes the clients are 6 or 7.. - Have not confirmed ZEO MONITOR OUTPUT: Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. ZEO monitor server version 3.6.2 Wed Apr 11 12:30:31 2007 Storage: 1 Server started: Wed Apr 11 10:56:50 2007 Clients: 10 Clients verifying: 0 Active transactions: -1 Commits: 24 Aborts: 2 Loads: 95941 Stores: 523 Conflicts: 0 Conflicts resolved: 20 Storage: 2 Server started: Wed Apr 11 10:56:50 2007 Clients: 8 Clients verifying: 0 Active transactions: 0 Commits: 9 Aborts: 0 Loads: 40836 Stores: 714 Conflicts: 0 Conflicts resolved: 167 -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] more lockup information / zope2.9.6+zodb 3.6.2
How does data get into the ZEO storage then? We have 10 ZEO clients that are for public consumption READ ONLY. We have a separate ZEO client that is writing that is on a separate box. The website got slashdoted this morning and we had 4 zeo clients go out. Basically waiting for the zeo server for many minutes (basically hung and reporting 500s back to browsers). 4/10/07 - delta ah2 lock up - http://paste.plone.org/13919 4/10/07 - epsilon ah2 lock up - http://paste.plone.org/13920 4/10/07 - delta ah1 lock up - http://paste.plone.org/13936 4/10/07 - epsilon ah3 lock up - http://paste.plone.org/13935 Just FYI: Varnish didnt go over 3% CPU during the traffic surge; over 200 req/second. The ZEO Server was consistently at 2% CPU. Lots of traffic was getting through the cache and back to pound which was then load balancing to the ZEO clients. For Jim: We did not adjust the transaction timeout. Would that have helped in the case of READ's? The customer was posting content throughout the slashdot. The problem was that when the clients would update they would end up through 500s I am seeing something *very* strange in zeo.log: 2007-04-10T12:20:53 ERROR ZEO.zrpc.Connection(S) (172.16.235.120:49351) Erro r caught in asyncore Traceback (most recent call last): File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 69, in read obj.handle_read_event() File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 391, in han dle_read_event self.handle_read() File /usr/local/zope/lib/python/ZEO/zrpc/smac.py, line 147, in handle_re ad d = self.recv(8192) File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 343, in rec v data = self.socket.recv(buffer_size) error: (113, 'No route to host') but several more of these: 2007-04-10T13:55:36 ERROR ZEO.zrpc.Connection(S) (172.16.235.119:44322) Erro r caught in asyncore Traceback (most recent call last): File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 69, in read obj.handle_read_event() File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 391, in han dle_read_event self.handle_read() File /usr/local/zope/lib/python/ZEO/zrpc/smac.py, line 147, in handle_re ad d = self.recv(8192) File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 343, in rec v data = self.socket.recv(buffer_size) error: (110, 'Connection timed out') -- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] more lockup information / zope2.9.6+zodb 3.6.2
I'ev not had anything but bad experiences with pound myself, lvs seems a much more preferable alternative... We have not had such negative experiences with pound. data = self.socket.recv(buffer_size) error: (113, 'No route to host') That *is* very odd, anything other than pound being used for load balancing or traffic shaping? This has to be a major problem maker in the system. Pound is simply round robin connections to pool of Zope. File /usr/local/python-2.4.4/lib/python2.4/asyncore.py, line 343, in rec v data = self.socket.recv(buffer_size) error: (110, 'Connection timed out') Storage server too busy :-( the storage server can be too busy when its only reading? cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] more lockup information / zope2.9.6+zodb 3.6.2
Hi guys. Running Zope 2.9.6 with ZODB 3.6.2 on Python 2.4 Having lots of lockups. Have approximately 12 zeo clients on 2 machines connecting to a single zeo server. All on local network. Disks and network is monitored by competent hosting company. All looks healthy except zeo communication. Linux 2.6.9-42.0.8.ELsmp some feedback from manage_debug_threads: http://paste.plone.org/13821 http://paste.plone.org/13822 http://paste.plone.org/13823 typical client zeo configuration: /var/zope/ah0/zope.conf (included zope.conf) %define HTTP_SERVER 7580 %define ICP_SERVER 7680 %define INSTANCE /var/zope/ah0 %define MAIN_NAME epsilon0 %define CATALOG_NAME epsilonA %include /var/zope/etc/zope.conf /var/zope/etc/zope.conf (main zope.conf) zodb_db main mount-point / # ZODB cache, in number of objects cache-size 2 zeoclient server gamma-gw.audioholics.com:8100 storage 1 name $MAIN_NAME var $INSTANCE/var # ZEO client cache, in bytes cache-size 1024MB # Uncomment to have a persistent disk cache #client zeo1 /zeoclient /zodb_db zodb_db catalog mount-point /audioholics/portal_catalog container-class Products.CMFPlone.CatalogTool.CatalogTool cache-size 5 zeoclient cache-size 1024MB server gamma-gw.audioholics.com:8100 storage 2 name $CATALOG_NAME var $INSTANCE/var /zeoclient /zodb_db any ideas? alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] zeo cache hang?
Hi guys. Got this on, http://paste.plone.org/13772 The entire zope comes to a halt. any ideas? cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] POSKeyError in zodb-3.6.0
Hi Chris,I am no ZODB expert. But I see a few thins that are wrong with the code. Others should be able to comment at length: - You cant just catch ConflictError and pass - I think you can catch a ReadConflictError and *retry* that is ok. - But a ConflictError needs to be *retried* manually in your client code.If you catch a ConflictError you need to abort the transaction.You should be explicit about *beginning* transactions after ending previous transaction. If your using threads. I believe you need to be a bit more careful. Does this codecorrupt ZODB if your *not* using threads?Instead of using threads try: - Write the code to be used in single process - Start 3 processes; see if that fails - Remove your try/except around ConflictError. ConflictErrors need to be handled. NOTE: In Zope the framework handled ConflictError's automatically but your doing everything *outside* of Zope. Which means your code needs to do the retries.make sense?-- Alan RunyanEnfold Systems, Inc.http://www.enfoldsystems.com/ phone: +1.713.942.2377x111fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
