[ZODB-Dev] database conflict fail

2012-03-21 Thread Claudiu Saftoiu
Hello ZODB List, (This is also a stackoverflow question - you might prefer the formatting there: http://stackoverflow.com/questions/9810116/zodb-database-conflict-fail ) I have a server, and a client. A client sends a request. The request has a certain key associated with it, e.g. `a-1`, `a-2`,

Re: [ZODB-Dev] database conflict fail

2012-03-22 Thread Claudiu Saftoiu
On Thu, Mar 22, 2012 at 2:45 PM, Vincent Pelletier plr.vinc...@gmail.comwrote: Le jeudi 22 mars 2012 18:23:47, Claudiu Saftoiu a écrit : Ahh, now that looks promising. Are there any particularly good places to get documentation on that sort of thing? All I see when I google are mailing

Re: [ZODB-Dev] database conflict fail

2012-03-22 Thread Claudiu Saftoiu
Stop! Before you go down that road, application-level conflict resolution if an extremely advanced feature. It's on the same level as meta classes and custom import hooks in a Python. It's a last resort (or maybe the one after that). The first thing you need to ask yourself is why your

Re: [ZODB-Dev] database conflict fail

2012-03-22 Thread Claudiu Saftoiu
On Thu, Mar 22, 2012 at 5:04 PM, Vincent Pelletier plr.vinc...@gmail.comwrote: Le jeudi 22 mars 2012 21:13:34, Claudiu Saftoiu a écrit : In [14]: root._p_jar[0x139c35] Actually, you want to write: root._p_jar['\x00\x00\x00\x00\x00\x13\x9c\x35'] ie, OIDs are 8-byte binary strings. Ah

[ZODB-Dev] How does automatic retrying work?

2012-03-29 Thread Claudiu Saftoiu
Hello all, I have an HTTP request which has roughly two parts to it. One part requires a lot of processing with a small chance of a DatabaseConflict, while the other part requires little processing with a high chance of a DatabaseConflict. I'd like to do something like this: def

Re: [ZODB-Dev] ZODB-Dev Digest, Vol 108, Issue 17

2012-04-05 Thread Claudiu Saftoiu
Hello retry shall be made on a transaction basis. I would say, that while for the regular_view retry may be handled by a middleware or such (the publisher in Zope2 ?), you may want to manage it in my_view around your transaction. in my_view you do transaction commit, so you won't undo the

[ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

2012-04-19 Thread Claudiu Saftoiu
Hello all, I recently made a lot of changes to my ZODB app, and now I'm experiencing some puzzling behavior. Normally I have ~15 clients making requests to the server every few seconds. Some of these requests commit new data to the database, while others just process existing data. I also have a

Re: [ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

2012-04-19 Thread Claudiu Saftoiu
On Thu, Apr 19, 2012 at 11:33 AM, Martijn Pieters m...@zopatista.com wrote: On Thu, Apr 19, 2012 at 17:20, Claudiu Saftoiu csaft...@gmail.com wrote: My question is: what could possibly be causing the server to 'lock up', even on a simple view like 'is_alive', without using any memory

Re: [ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

2012-04-19 Thread Claudiu Saftoiu
I have no idea; all you told is that you use the ZODB, not what server framework you use to register your views. Is this Grok, Bluebream, Repoze.BFG, Zope 2 or something else? Ah yes, sorry about that. I'm using Repoze.BFG . Does that help any? I don't think that is_alive would be the cause

Re: [ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

2012-04-19 Thread Claudiu Saftoiu
I don't know, anything could lock up your site if something is waiting for a lock, for example. Use http://pypi.python.org/pypi/z3c.deadlockdebugger to figure out what the threads are doing at this time. Preferably, trigger the dump_threads() method of that module on SIGUSR1, like the Zope

Re: [ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

2012-04-20 Thread Claudiu Saftoiu
I don't think that is_alive would be the cause of this, it looks like a simple enough view. But if your server uses a thread pool, and all the other threads are now occupied by something that got locked up, then it could be that the server is not answering your is_alive request at all

Re: [ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

2012-04-20 Thread Claudiu Saftoiu
Ah, so it seems that, as I leave the server running longer longer, more more threads are taken up with a `.recv()` call. I think one of my clients opens requests and does not read them/close them. Eventually all the threads are blocking in that fashion. I will fix my clients. But, is

[ZODB-Dev] how to get root obj from a new transaction?

2012-05-02 Thread Claudiu Saftoiu
Hey all, I'm using a thread to do some server-side work. The thread will be asked by different requests to do the same thing at the same time, so I want the thread to do all the work once and return the data to the requests. The problem is that the requests each have their own transaction and the

Re: [ZODB-Dev] ZODB-Dev Digest, Vol 110, Issue 1

2012-05-03 Thread Claudiu Saftoiu
However, the thread has to start a new transaction each time it processes something - which I know how to do: while True: #wait until asked to do something import transaction transaction.begin() However, the thread needs access to the root

[ZODB-Dev] all webserver threads blocking on db.open()

2012-05-07 Thread Claudiu Saftoiu
Hello all, I'm using Repoze.BFG, with paster to launch the webserver. This is a similar issue to the one I emailed about before titled server stops handling requests - nowhere near 100% CPU or Memory used The situation is the same. I used z3c.deadlockdebugger , and what I notice is that, when

Re: [ZODB-Dev] all webserver threads blocking on db.open()

2012-05-07 Thread Claudiu Saftoiu
On Mon, May 7, 2012 at 12:13 PM, Hanno Schlichting ha...@hannosch.euwrote: I think you might get better help on one of the Pyramid support channels. Your problems all seem to be related to configuring a web server in production mode, rather than database issues. Thanks, I'll check out the

[ZODB-Dev] Is any function called when an object is loaded from the database?

2012-06-19 Thread Claudiu Saftoiu
Hello all, It is often the case that I have a Persistent object that evolves over time. E.g., it might start off as this: class Foo(Persistent): def __init__(self, a): self.a = a def calc_it(self, b): return expensive_function(a, b) I'll then have a

[ZODB-Dev] How to cause commit with non-PersistentDict?

2012-06-19 Thread Claudiu Saftoiu
Hello all, Say I have a: class Foo(Persistent): def __init__(self, bar): self.my_dict = PersistentDict({'keyis': bar}) def update_it(self, bar): self.my_dict['keyis'] = bar If I want to use a `dict`, instead (it seems it might be faster for my larger

Re: [ZODB-Dev] Is any function called when an object is loaded from the database?

2012-06-19 Thread Claudiu Saftoiu
On Tue, Jun 19, 2012 at 2:29 PM, Malthe Borch mbo...@gmail.com wrote: On 19 June 2012 19:54, Claudiu Saftoiu csaft...@gmail.com wrote: That is, a function called whenever the object is loaded, that does all the necessary backwards-compatibility work right there. It separates the backwards

Re: [ZODB-Dev] Is any function called when an object is loaded from the database?

2012-06-19 Thread Claudiu Saftoiu
You'll need to override ``__new__``. That's your hook. It's called when the database instantiates the object. Note that this is always true for Python. The ``__new__`` method is always called before an object is instantiated. Actually, this doesn't seem to be what I want. ``__new__`` is

[ZODB-Dev] repoze.catalog.query very slow

2013-01-03 Thread Claudiu Saftoiu
Hello all, Am I doing something wrong with my queries, or is repoze.catalog.query very slow? I have a `Catalog` with ~320,000 objects and 17 `CatalogFieldIndex`es. All the objects are indexed and up to date. This is the query I ran (field names renamed): And(InRange('float_field', 0.01,

[ZODB-Dev] zeopack error in zrpc.connection

2013-01-07 Thread Claudiu Saftoiu
I noticed my DB had swelled to 132 gigabytes (as of 3 days ago; it's 160 gigabytes today) and it seems to be because zeopack has started failing: tsa@sp2772c:~/db$ /home/tsa/env/bin/zeopack -u /home/tsa/db/zeo.sock Traceback (most recent call last): File

Re: [ZODB-Dev] zeopack error in zrpc.connection

2013-01-07 Thread Claudiu Saftoiu
How do I go about fixing this? Let me know if I can provide any other information that would be helpful. I took the advice in this thread: https://mail.zope.org/pipermail/zodb-dev/2012-January/014526.html The exception that comes up, from the zeo server log, is: 2013-01-07T13:01:49 ERROR

Re: [ZODB-Dev] zeopack error in zrpc.connection

2013-01-07 Thread Claudiu Saftoiu
I'm afraid this doesn't seem to help me figure out what's wrong... I suspect your database is corrupted. You'd probably want to look at the record in question to be sure. Sure, I've re-run the pack and will dump the pickled object to a file to inspect it - is that what you meant? (How

[ZODB-Dev] runzeo perma-memory leak

2013-01-09 Thread Claudiu Saftoiu
Hello all, It seems that something weird is happening with runzeo my server's memory. I have a Data.fs that is 5.5gb, and 7.6gb of RAM on my server. I do 'runzeo -C zeo.conf' where the 'zeo.conf' is: %define INSTANCE . zeo address $INSTANCE/zeo.sock read-only false

Re: [ZODB-Dev] runzeo perma-memory leak

2013-01-09 Thread Claudiu Saftoiu
Oh never mind, http://www.linuxatemyram.com/ . =) ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] documentation on the various caching options

2013-01-09 Thread Claudiu Saftoiu
I'm going a bit nuts trying to figure out what all the different caches mean. This doesn't seem to be well documented, but I'll keep googling after sending out this email. Here's the various configs I've got. I have a zeo.conf: %define INSTANCE . zeo address $INSTANCE/zeo_indexdb.sock

Re: [ZODB-Dev] ZODB-Dev Digest, Vol 118, Issue 7

2013-01-10 Thread Claudiu Saftoiu
Hey Vincent, Thanks for the detailed reply! That really clears things up. - Are there any other caching options? The option to make pickle cache persistent comes to mind. There may be others, I don't recall right now. This seems like a good idea - I would like for the server to maintain its

[ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-15 Thread Claudiu Saftoiu
Hello all, I'm looking to speed up my server and it seems memcached would be a good way to do it - at least for the `Catalog` (I've already put the catalog in a separate zodb with a separate zeoserver with persistent client caching enabled and it still doesn't run as nice as I like...) I've

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-15 Thread Claudiu Saftoiu
On Tue, Jan 15, 2013 at 2:07 PM, Leonardo Santagada santag...@gmail.comwrote: On Tue, Jan 15, 2013 at 3:10 PM, Jim Fulton j...@zope.com wrote: On Tue, Jan 15, 2013 at 12:00 PM, Claudiu Saftoiu csaft...@gmail.com wrote: Hello all, I'm looking to speed up my server and it seems

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-15 Thread Claudiu Saftoiu
, then there is a separate blob cache. On Tue, Jan 15, 2013 at 2:15 PM, Claudiu Saftoiu csaft...@gmail.com wrote: You can't cause a specific object (or collection of objects) to stay ion the cache, but if you're working set is small enough to fit in the memory or client cache, you can get the same effect

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-17 Thread Claudiu Saftoiu
Okay, that makes sense. Would that be a server-side cache, or a client-side cache? There are no server-side caches (other than the OS disk cache). Ok, that's what I gathered before, was just checking. I believe I've already succeeded in getting a client-side persistent disk-based

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-17 Thread Claudiu Saftoiu
Thanks, Jeff! I'll check out that cacheDetail stuff and see if I simply have too few objects being stored (though I thought 5 million would be enough...) Further, after having preloaded the indices once, shouldn't it preload quite rapidly upon further server restarts, if it's all in the cache

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-18 Thread Claudiu Saftoiu
If you want to load the btree item into cache, you need to do item._p_activate() That's not going to work, since `item` is a tuple. I don't want to load the item itself into the cache, I just want the btree to be in the cache. Er, to be clearer: my goal is for the preload

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-18 Thread Claudiu Saftoiu
Er, to be clearer: my goal is for the preload to load everything into the cache that the query mechanism might use. It seems the bucket approach only takes ~10 seconds on the 350k-sized index trees vs. ~60-90 seconds. This seems to indicate that less things end up being pre-loaded...

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-18 Thread Claudiu Saftoiu
That is after having run the 'preloading'. It seems that when the query takes forever, the non-ghost-size is slowly increasing (~100 objects/second) while the 'size' stays the same. Once the query is done after having taken a few minutes, each subsequent run is instant and the ngsize doesn't

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-18 Thread Claudiu Saftoiu
Any suggestions? There must be a way to effectively use indexing with zodb and what I'm doing isn't working. Have you confirmed that the ZEO client cache file is being used? Configure logging to display the ZEO messages to make sure. The client cache is transient by default, so you will

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-18 Thread Claudiu Saftoiu
I wonder if disk latency is the problem?. As a test you could put the index.fs file into a tmpfs and see if that improves things, or cat index.fs /dev/null to try and force it into the fs cache. Hmm, it would seem not... the cat happens instantly: (env)tsa@sp2772c:~/sports$ time cat

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-21 Thread Claudiu Saftoiu
attribute in a certain range, I don't have a good way to do it based on how they are stored, now. I could try making my own indexing scheme but I figured ZCatalog would be well-suited for this... On Thu, Jan 17, 2013 at 12:31 PM, Claudiu Saftoiu csaft...@gmail.com wrote: ... One potential thing

[ZODB-Dev] A certain code path seems to be blocking everything

2013-02-14 Thread Claudiu Saftoiu
I've got a weird bug with my server and I'm wondering if anyone could provide some insight into it. The general idea of the server is that I have 8 or so clients constantly pinging it for information. Each ping usually only takes up to 2 seconds to process. Essentially, everything runs fine,

Re: [ZODB-Dev] A certain code path seems to be blocking everything

2013-02-15 Thread Claudiu Saftoiu
going to change that part of the code to be more efficient. - Claudiu On Thu, Feb 14, 2013 at 4:24 PM, Claudiu Saftoiu csaft...@gmail.com wrote: I've got a weird bug with my server and I'm wondering if anyone could provide some insight into it. The general idea of the server is that I have

Re: [ZODB-Dev] Cache warm up time

2013-03-07 Thread Claudiu Saftoiu
I was having this same issue. Persistent caching helped a little bit but not too much. I didn't end up implementing this but ultimately the best thing to do seemed to be to have a different server with a different zodb that only handles indexing. That way it will never restart and lose its cache.

Re: [ZODB-Dev] Cache warm up time

2013-03-08 Thread Claudiu Saftoiu
I'd be curious to know what your results are, whichever path you decide to take! Might help inform me as to what might help on my server... One thing I haven't yet understood is - how come the ZEO server itself doesn't have a cache? It seems that would be a logical place to put one as the ZEO

Re: [ZODB-Dev] Cache warm up time

2013-03-08 Thread Claudiu Saftoiu
of the data that ZEO does between reading the bytes and sending them across the network). Cheers, Leo On Fri, Mar 8, 2013 at 1:16 PM, Claudiu Saftoiu csaft...@gmail.com wrote: I'd be curious to know what your results are, whichever path you decide to take! Might help inform me as to what might

Re: [ZODB-Dev] Cache warm up time

2013-03-08 Thread Claudiu Saftoiu
On Fri, Mar 8, 2013 at 12:31 PM, Leonardo Santagada santag...@gmail.comwrote: On Fri, Mar 8, 2013 at 2:17 PM, Claudiu Saftoiu csaft...@gmail.comwrote: Once I know the difference I'll probably be able to answer this myself, but I wonder why the ZEO server doesn't do the sort of caching

Re: [ZODB-Dev] polite advice request

2013-08-18 Thread Claudiu Saftoiu
I wonder, if you have a problem which an SQL database would be so good for that youre mimicking an SQL database with zodb, why not just use an SQL database? It doesn't sound like you'll gain much from being able to persist objects which is one of the main reasons to use an object database...