[Gfeller, Martin] > ... > You are correct that we're accessing one connection with multiple > threads.
That would appear to explain it then. > The reason is that we have many very long-running calculations which > also use a large portion of objects in our DBs. The app is most > efficient when we use almost all memory space as ZODB cache - i.e., ~1.5 > GB. If we multiply that per connection, we'd either go into massive > trashing or out of address space. > > So I will try (i) to pinpoint the crash more closely (by using a debug > build) and (ii) try to lock around our accesses (which is difficult, as > loading a ghostified object is an access as well). > > In the longer run, I assume you're suggesting we should look for another > way to access ZODB, or for another DBMS altogether (not to our liking)? I don't know enough about your app to suggest something concrete here. All I can tell you, based on what I know, is a good guess about why your current approach segfaults. Suggesting something concrete would require knowing a _great_ deal more. For example, I don't know why you're using multiple threads to begin with (e.g., if the app is CPU-bound, throwing multiple threads at it is likely counterproductive). Don't know how much data you have (e.g., perhaps it would all fit in RAM, and you could get away with loading it all immediately after opening the connection). Don't know the data access patterns; e.g., perhaps a small amount of data is mutable, but a lot more doesn't change, in which case perhaps you could load the unchanging data from one database and the mutable data from another. I don't know what kinds of computations you're doing. For example, many large-scale numerical computations in pure Fortran are deliberately designed to be "cache friendly"; e.g., applying divide-and-conquer blocking strategies to large matrix multiplications, to minimize mean working-set size over a sequence of computational stages. There's often far more code arranging to "live with" limited fast cache than there is to do actual number-crunching (matrix multiplication can be written in 4 lines of code; a _useful_ large-scale matrix multiplication routine can easily consume 4 pages of code). Don't even know if you _are_ doing significant number-crunching; if you are, perhaps recoding some of it in C would be a major win. And so on. If your app requires truly random access to almost all of an enormous collection of data, then I have no suggestion that's likely to help. I'm not sure I've ever seen such an app. But if there are exploitable properties in your app's algorithm and/or in its data, I don't know enough about your app to guess what kind they may be. It's possible (even likely) that there isn't a quick refactoring that would help enough. _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
