Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Hi, Shane Hathaway wrote: Stefan H. Holek wrote: We have put up two boxes in a MySQL master-master replication setup [1]. As long as we only write to one of the masters all is fine. Writing to both masters (and expecting MySQL to sort it out) is giving us trouble. We see things like: Last_Errno: 1062 Last_Error: Error 'Duplicate entry '41860' for key 1' on query. Default database: 'prod_zodb'. Query: 'INSERT INTO new_oid VALUES ()' Perhaps we can solve this by making each server allocate a different set of OIDs. For example, one server might allocate odd-numbered OIDs while the other allocates even-numbered OIDs. Even better would be some kind of standard MySQL solution, but I haven't found any yet. the standard approach for master-master replicated data is to use an UUID-Datatype (see http://en.wikipedia.org/wiki/UUID ) Last_Errno: 1050 Last_Error: Error 'Table 'temp_store' already exists' on query. Default database: 'prod_zodb'. Query: 'CREATE TEMPORARY TABLE temp_store ( zoidBIGINT NOT NULL PRIMARY KEY, prev_tidBIGINT NOT NULL, md5 CHAR(32), state LONGBLOB ) ENGINE MyISAM' Uh-oh. The commit lock should have prevented this, so this error suggests that each master has a completely independent set of locks! surprise ;) That is a serious issue that will lead to database corruption. The commit and pack locks need to be cluster-wide. Does MySQL have a way to do that? Well, synchronous replication is a hard beast, master-master synchronous even harder (and you have to ask what problem you really want to solve with it, since the trade-offs are massive) Tino smime.p7s Description: S/MIME Cryptographic Signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Stefan H. Holek wrote: I am playing with database replication (MySQL) and have found the following caveats: - don't mix engine types (MyISAM, InnoDB) - don't use temporary tables Now, RelStorage does both and so I was wondering how to proceed with replication. Would it be prudent to assume that all I ever need to Maybe use postgres and slony instead? The mysql shortcomings on replication and integrity should be well known. It should also be possible to replicate on ZODB layer instead. At least I read something in the ML beside ZRE of course :-) Cheers Tino smime.p7s Description: S/MIME Cryptographic Signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] adding mount points
Christian Theune wrote: Hi, Am Dienstag, den 18.12.2007, 01:42 -0800 schrieb Tarek Ziadé: Hi, I am working on a project that needs to do mass hosting of zope sites. I would like to have several site per zope instance, and to make a zodb mount point for each new site created to ease their maintenance. Is there a way to dynamically load new mount points on a ZODB to avoid relaunching the zope instance everytime a website is added ? (then reflecting the changes in zope.conf) My guess is to override getConfiguration() in ZODBMountpoint.MountedObject but maybe i miss something ? On ZODB-level this is not an issue with the multi-database API. I'm not sure about the specifics of the ZODBMountpoint product though. Just restarting a twin instance usually causes the mountpoint to appear so with a little wiring this should be doable within one instance as well :-) Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: AW: [ZODB-Dev] diploma thesis: ZODB Indexing
Sebastian Wehrmann schrieb: Am 4. September 2007, 16:17:27 Uhr schrieb Jim Fulton: I would very much like to see an open indexing+querying framework for Python objects. I'm thinking of something *like* an SQL engine that allowed one to plug in relation and index implementations and that took queries in some form, optimized them and executing them using the given index and relations. We plan to realize three things important to us: - We don't want indexing on application-level (e.g. application-specific) - We want ad-hoc queries - We don't want to rely on transforming the data into a relational model very well! Greets Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: AW: [ZODB-Dev] diploma thesis: ZODB Indexing
Christian Theune schrieb: Am Mittwoch, den 05.09.2007, 09:24 -0400 schrieb Jim Fulton: I'd like to see a generic framework for defining collections and indexes in Python and querying them efficiently. No ZODB expertise should be needed, I have the feeling you already pondered this a bit and have some more specific ideas ... :) I also have the feeling that our goal for ad-hoc querying would be incompatible with your envisioned framework for defining collections and indexes. My impression is many people have thought about this problem. Now someone stood up and starts actual working, this is good :-) I think ad-hoc queries are not per se incompatible, they would just act like sequential scan in relational databases - therefore work but not too efficient. Maybe the api can generate a warning if desired so the application developer can add indexing. Consequently thinking about the whole scope, sometime at the end we will even need further abstraction regarding authorization and access to objects and attributes. This would dramatically change the way zope works with ZODB but open a lot more uses of ZODB independently of the Zope world. Just my 1e-21 cents :-) Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Implementing Storage Decorators
Dieter Maurer schrieb: Jim Fulton wrote at 2007-5-4 14:40 -0400: On May 4, 2007, at 2:33 PM, Dieter Maurer wrote: Jim Fulton wrote at 2007-5-2 11:52 -0400: ... I think I still rather like explicit, but I'm on the fence about which approach is best. What do other people think? From your description, I would use a subclassing (and forget about proxy and copying). That would be a nightmare, on multiple levels: - All of the separate implementations would become tightly coupled, which is what happens with inheritance. - Either someone would have to create classes for the various permutations of features, or consumers would have to mix and match multiple classes to get what they want and sort out the variate internal implementation incompatibilities. Your decorators would become mixin classes and the final classes would list the features they like -- simpler than ZCML binding together... Of course, some features may not play well with one another. But, that will make problems also with proxies or copying... Even this could be sorted out with special cleanup or secure mixins which can do dirty tricks like accessing self.__class__.__bases__ to fixup ;) (Much like combining the copy with the subclass approach) I'm for mixin with strong guidelines on implementation (e.g. how to call private attributes to not pollute the namespace) Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] History-less FileStorage?
Stefan H. Holek schrieb: Do we have a history-less (i.e. no-grow) FileStorage? How do you think it should work? I mean, technically in theory? How would you implement MVCC? Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Before-abort hook?
Stefan H. Holek schrieb: Hi all, The ExtFile product I maintain keeps its payload on the filesystem. All operations are performed using temporary files which are then saved or deleted on transaction commit or abort respectively. Do you remember, when you use the tempfile module, you get a filehandle with a nameless file which automatically vanishes if you close the handle. Maybe this solves the problem for you? Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] how to let java applet using mysql database work in zope/plone?
LonLon schrieb: Thanks for your suggestion jens, but my java applet is working outside zope, it has no problem with mySQL database. Only when I integrate my applet into zope, it can not take mySQL data anymore. That's why I request suggestions here. I am still hoping somebody knows how to fix this problem. :-( Nothing to do with ZODB (since you are using mySQL?) and even not a Zope problem. Applets in a browser can only connect back to the same host. If you start your applet locally, it is allowed to connect to any host it wants. Read about it in your java documentation. Solutions: put the database and your web site on the same host (or use firewall/proxy tricks to let it appear like this) - or better access your database thru zope - via XMLRPC and friend - and this way get on topic :-) --Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB with a Relational Database
José Carlos Senciales schrieb: thanks How can i know if i have my Python configured with large file support ? My version is: Zope Version (Zope 2.8.4-final, python 2.3.5, win32) Python Version 2.3.5 (#62, Feb 8 2005, 16:23:02) [MSC v.1200 32 bit (Intel)] System Platform win32 double-klick on the python.exe or start it in a console window and try: f=open(somefile.txt,w) f.tell() 0L if its 0L, you probably have large file support. If its just 0, you probably dont. Not sure if on windows it also depends on the filesystem you use. Try with NTFS if in doubt. Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB with a Relational Database
José Carlos Senciales schrieb: thanks my phyton console return 0L, so i have have large file support. So if you still experience problems with files 2GB, you might move to another filesystem (if moving to another OS is not an option ;) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Query Regrading ZODB FileStorage(.fs file)
tav schrieb: Why is it significantly easier to protect the key[s] used for the encryption than the storage itself? one could always passphrase-protect the key, i.e. use symmetric encryption. admittedly, this could potentially be brute-forced, but ... should be good enough for most purposes? And how does your Application (Zope) access the storage? Exactly. It needs the key - if it has the key - the attacker can just read the data thru the application. In the end this does not buy you anything but overhead. If you want to encrypt, just use a crypted filesystem as DM already suggested. Best performance, best transparency and well tested. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Fixing Broken Objects?
Am Donnerstag, den 13.10.2005, 18:09 -0400 schrieb Jim Fulton: Tino Wildenhain wrote: Am Donnerstag, den 13.10.2005, 08:03 +0200 schrieb Andreas Jung: --On 13. Oktober 2005 01:45:30 -0400 Chris Spencer [EMAIL PROTECTED] wrote: I noticed that if a ZEOServer client doesn't have the definition for an object's class, it loads it as a broken object. Is there any way to fix the broken object by obtaining the class source from the server? Your sources on the client side must be complete. Otherwise the object is broken. The server has nothing to do with problem. Fix your client. While it would certainly be nice if the ZEO server could deliver codeobjects too. I don't think it would be useful to deliver ordinary Python modules via the DB. Why not? This way one could keep all code changes in a central repository, w/o the need of either manually syncing or some special (flaky) network filesystem. It would be useful to support persistent modules, which would have somewhat different semantics than standard Python modules. There is limited experimental support for this in the Zope 3 repository. Another usecase and valid too - this way one could do an entire timetravel with code, data and objects in the ZODB. SVN made right, so to say :) In Theory and with much much bad hackery (overwriting __import__ and friends) it could be possible. Jim? For some definition of it, sure. Btw, (hijacking the thread because a bit related) how can I serialize the state of a generator object? (e.g. for caching it between requests) Pickle does not work for it (whyever) Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Proposal] Size controlled ZODB cache
Am Donnerstag, den 02.06.2005, 19:28 +0200 schrieb Dieter Maurer: Currently, the ZODB cache can only be controlled via the maximal number of objects. This makes configuration complex as the actual limiting factor is the amount of available RAM and it is very difficult to estimate the size of the objects in the cache. I therefore propose the implementation of cache replacement policies based on the estimated size of its objects. I propose to use the pickle size as the size estimate. The connection could store the pickle size in the object as _p_size (and may call a hook function _p_estimateSize, if it is defined -- but I do not think, we need this). I am aware that the actual size of an object may significantly differ from its pickle size, but usually, they will at least be in the same order. As additional limiting parameters, I propose MAX_OBJECT_SIZE and MAX_TOTAL_SIZE. Objects with size = MAX_OBJECT_SIZE are invalidated at the next possible time (at a transaction boundary) before other potential invalidations are considered. The purpose of the limit it to prevent a single (or few) large objects to flush large amounts of small objects. Such large objects are managed in a special (doubly linked) list in order to quickly locate them. After large objects are flushed, the replacement policy works as it does now. However, beside the number of objects, their total estimated size is accumulated. As soon as either the MAX_OBJECT_NUMBER or MAX_TOTAL_SIZE is reached, the remaining objects are invalidated (as far as possible). Would be nice to optionally have the posibility to define finer grained cache policy (e.g. reserve 80% of cache mem to objects 5k and 20% for all the others - or something the like - much like altq does for ip-traffic shaping). E.g. some clean hooks to cache store management and cache policy management. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
