Re: [ZODB-Dev] Ackward PersistentList read Performance
On 13 August 2013 15:40, Joerg Baach li...@baach.de wrote: So, can it really be that using a PersistentList is 300 times slower? Am I doing something completely wrong, or am I missing something? In your test setup, you commit the transaction and close the connection. This means that when you iterate through the edges, not only do you load buckets, but for each item in the bucket, you load an additional persistent object, namely the `PersistentList`. This doesn't happen when you persist a Python list. In that case, the list is simply persisted right into the bucket data. This makes the buckets take up more space, but you need less reads from disk which makes the whole thing go a lot faster. Note that once you've loaded all the `PersistentList` objects, iterating again should go much faster because it's now all in memory. \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Fwd: Zope/Plone/Zeo Concurrency Issue
de succès et de bonheur –– Malthe Borch mbo...@gmail.com ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Broken objects problem.
Hey Matias, Make sure that all of the processes have a Python interpreter with access to the relevant modules. You'll get broken objects on the ZEO side if it does not have the same software as your clients. \malthe On 8 August 2012 01:46, Matias Pansa matiashpa...@gmail.com wrote: Hello everyone , i'm Matias from Argentina. I'm developing a web app using flask - zodb (zeo) and gunicorn as server , i'm facing the following problem : from a server that i wrote i'm recieving data from internet of some gps and the same server store the data via zeo client , here everything is ok , i can check the data with vista.py script tha simply connects via zeo client and get the results from the database , the data class is Posicion and represents a gps position in time , all the objects are stored correctly.i check almos When i try to do the same thing from inside tha flask app running gunicorn all i get in the resultset of data from zeo are broken objects with no class so i can't access any attributes or methods of the class Posicion . My question is , could this problem be related to zeo or zodb ? , or i keep looking to gunicorn/flask problem ? , i check almost everything that's why i'm asking here. thanks ,Matias. ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Is any function called when an object is loaded from the database?
On 19 June 2012 19:54, Claudiu Saftoiu csaft...@gmail.com wrote: That is, a function called whenever the object is loaded, that does all the necessary backwards-compatibility work right there. It separates the backwards-compat code cleanly, and also only updates the objects as-needed... though still a minor performance hit as it does the check each time the object is loaded. Is there a way to do that last option? What's the best practice for this sort of thing, in general? You'll need to override ``__new__``. That's your hook. It's called when the database instantiates the object. Note that this is always true for Python. The ``__new__`` method is always called before an object is instantiated. \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Is any function called when an object is loaded from the database?
On 19 June 2012 23:38, Claudiu Saftoiu csaft...@gmail.com wrote: Is there any hook to call *after* the instance attributes get set/loaded from the database? True. I guess you'll have to go with Marius' suggestion – ``__setstate__``. \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Solved!!! Re: unable to import ZODB: class ConflictError, AttributeError
On 14 June 2012 17:52, Martijn Pieters m...@zopatista.com wrote: * Store as a tree of persistent objects if parts need changing over time. Future commits would be a lot smaller that way. I'd parse out the original large dictionary in a separate process, perhaps chunk the commits (so build up the structure over several commits). If you use a BTree, then you can commit savepoints as you go (after a suitable number of entries): transaction.savepoint() Please report back your findings. \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Zope-DB] How to extract users info from ZODB?
On 25 January 2012 11:14, smita kamdar smita.kamda...@gmail.com wrote: You mean to say, i click 'acl_users' then click export? That might work you as well – but if you've got something like a pluggable authentication storage, then there's any number of plugins inside your acl_users folder. You might want to export a particular plugin's data instead of trying to export the whole thing. That said, it might be better for you to start up the instance's debug console and work with your application root object: $ bin/instance debug This gets you into Python; try playing around with the app object: app ... \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Question about zeo client cache
On 10 December 2011 23:41, Maurits van Rees m.van.r...@zestsoftware.nl wrote: Disk size is cheap. I suggest increasing this to a few hundred MB or maybe even a few GB if you have lots of data. Otherwise you will get a lot of traffic between your zeoclient and server. If you have very low network latency (essentially if your ZEO is on the LAN), there's little point in having a client cache. It's faster to pull the data from the ZEO server which either: 1) has the data you need in its own cache, or 2) can load it from disk in which case the network overhead is negligible. The ZEO client cache is really only suitable for systems where your ZEO server is on the internet. \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZEO] remote connection problem with ZEO
On 21 October 2011 04:15, 刘一新 liuy...@gmail.com wrote: (1) Why can't I connect zeo on other nodes by just using the hostname:port when zeo is started with -a localhost:1234, and It binds only the loopback device (127.0.0.1). Try setting it up with 0.0.0.0 – this will bind the server to all network interfaces. (2) Why can't I connect ZEO on the local node (here the console node) with localhost:1234 when I start zeo with the hostname of the local node, i.e. console:1234? Same situation – only now you're not binding it to the loopback device :-) \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using demostorage + blobs with ZEO
On 17 October 2011 14:15, Zvezdan Petkovic zvez...@zope.com wrote: It is supported. Great! 1. Wrap zeoclient into before storage. 2. Add filestorage for changes. 3. Wrap all that into demostorage ... but I simply can't see how I can *add* filestorage for changes. Here's my hierarchy: demostorage before zeoclient ... /zeoclent /before /demostorage In which of these section can I add a storage for changes? As far as I can tell from the documentation, demostorage only takes a read-only base storage and keeps its own changes in memory. \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using demostorage + blobs with ZEO
On 17 October 2011 14:15, Zvezdan Petkovic zvez...@zope.com wrote: 2. Add filestorage for changes. Just saw: http://pypi.python.org/pypi/zc.demostorage2/0.1.3. That was the missing piece of information. \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Using demostorage + blobs with ZEO
Hello, According to source code comments, a demo storage will support blobs if the underlying storage does. But I find that it's impossible to wrap a zeoclient in a demo storage and have it working. Am I doing something wrong – or, if it's not supported, would it be difficult to do so and is anyone interested in helping out on this? Thanks, \malthe ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to check for setting the same values on persistent objects?
On 4 May 2011 11:53, Hanno Schlichting ha...@hannosch.eu wrote: I tried to analyze the overhead of changing content in Plone a bit. It turns out we write back a lot of persistent objects to the database, even tough the actual values of these objects haven't changed. How about identifying the places in the application where this happens the most and fix those (it sounds straight-forward). It reminds me that when I wrote Dobbin, I added a requirement that you check out an object that you want to modify: checkout(obj) The object identity didn't change, but it became thread-safe in terms of changing it. The ZODB persistency model on the other hand makes all objects ready for changing at any time; maybe the checkout pattern would have made for better application code (because writes are then explicit). \malthe ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Sharing (persisted) strings between threads
Currently, when a thread loads a non-ghost into its object cache, its
straight from being unpickled. That means that if two threads load the
exact same object, any (immutable) string contained in the object
state will be allocated for in duplicate (or in general, on the count
of the active threads).
If instead, all unpickled strings were made canonical via a weak
dictionary, there would be only one copy in memory, no matter the
thread count, e.g.:
string = weak_string_map.setdefault(string, string)
If the returned string was a different (canonical) copy, the duplicate
would immediately be ready for garbage collection.
This is a real win in memory savings. Using Plone, I experimented with
the approach by using the Python pickle implementation and interning
all byte strings (using ``intern``) directly in the unpickle routine
to the same effect:
def load_binstring(self):
len = mloads('i' + self.read(4))
string = self.read(len)
interned = intern(string)# (sic)
self.append(interned)
With 20 active threads, each having rendered the Plone 4 front page,
this approach reduced the memory usage with 70 MB. Note that unicode
strings aren't internable (but the alternative technique of using a
weak mapping should work fine).
In a long-running operation, dirty objects should be invalidated after
the transaction, to prevent future data redundancy.
For an implementation one needs to have a hook to use a special
reconstructor function for strings. Currently there is a technical
impediment in that BTrees and Persistent objects have their own
internal way to save strings. In my experiments, the ``persistent_id``
function was not called for string objects (which is a different
behavior than the regular cPickle.Pickler.dump has).
\malthe
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Sharing (persisted) strings between threads
On 8 December 2010 13:28, Jim Fulton j...@zope.com wrote: With 20 active threads, each having rendered the Plone 4 front page, this approach reduced the memory usage with 70 MB. Out of a total of what? In my case out of 430 MB non-shared for the process. Note that if a process is CPU bound (as most dynamic Python apps should be), then there is little or no benefit in having multiple threads, due to the (damn) GIL. The case I'm thinking of is when one thread is being used in a write transaction, while another is doing a read. If the database is bigger than the allowed memory usage, then I guess threads can also ensure that requests for in-memory objects can be served while some threads are blocked due to swapping and/or reading pickles from disk. Except that you can't create wekrefs to strings or unicode. I see. Maybe another scheme could be devised. Also, while interning is fine for an experiment, it's wasteful for strings that are rarely needed. How so? As far as I can see, interning is still subjected to reference counting. The only real difference is that a hash table is maintained (fairly minimal memory use + probable computation of string hash). Sharing immutable data between threads is very appealing intellectually. I've certainly thoughtr about it a lot. In practice, I doubt the benefit will be worth the extra overhead (let alond the effort :). I think if the case can be made for threading, then it's worth pursuing. Alternatively, applications might put all non-trivial strings into blobs, but I don't know if there's a non-trivial overhead with that approach. \malthe ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Sharing (persisted) strings between threads
On 8 December 2010 15:29, Jim Fulton j...@zope.com wrote: The hash table retains a reference to the strings in it. The references aren't weak afaik. The reference is removed by the custom string dealloc function when the ref count falls to 0. Interned strings do not linger on. What are you thinking of as applications with non-trivial strings? In content management apps for instance, you have many non-trivial strings with content, be it shorter descriptions or text bodies. That's a substantial amount of text if allocated in duplicate or more. Actually, I think *text* is the only kind of data that ought to take up significant memory. Binary data ought to be blobbed out of the database and the rest is just site structure and object overhead. The only one I can think of is template source. That might be better served by either storing the source compressed or even storing it in a separate object that doesn't need to be in memory except when editing or compiling. It might make sense to keep a global cache for ZODB-persisted templates. But I'm not sure if there are so many that it's a burden directly. This is an app-specific matter though. I'll try to get some raw numbers out of currently running apps to make the conversation more substantial. \malthe ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
