Thanks for the quick response to this as usual. See me responses below.
On Fri, Mar 4, 2011 at 7:32 AM, Michael Bayer <[email protected]>wrote:
>
> On Mar 4, 2011, at 2:58 AM, Lenza McElrath wrote:
>
> Hello! I'm iterating over a session to look at all the objects:
>
> for obj in session:
> do_something_cool(obj)
>
> Yesterday this caused what looks like a deadlock in SQLAlchemy code. Here
> is the stack I grabbed using gdb:
>
> 1. /python2.5/sqlalchemy/orm/session.py:1353 (Session.__iter__)
> 2. /python2.5/sqlalchemy/orm/identity.py:184 (WeakInstanceDict.values)
> 3. /python2.5/sqlalchemy/orm/identity.py:188 (
> WeakInstanceDict.itervalues) ** self._remove_mutex.acquire()
> 4. /python2.5/sqlalchemy/orm/state.py:501 (
> MutableAttrInstanceState.__resurrect)
> 5. /python2.5/sqlalchemy/orm/attributes.py:925 (Events.run)
> 6. /python2.5/sqlalchemy/orm/mapper.py:2424 (_event_on_resurrect)
> 7. /python2.5/sqlalchemy/util.py:953 (OrderedSet.__iter__)
> 8. /python2.5/sqlalchemy/orm/state.py:477 (
> MutableAttrInstanceState._cleanup)
> 9. /python2.5/sqlalchemy/orm/identity.py:139A (WeakInstanceDict.remove)**
> self._remove_mutex.acquire
> ()
>
> I'm running SQLAlchemy 0.6.5. Is this a known issue or am I doing
> something wrong?
>
>
> I've never seen that before. A reentrant mutex would fix this but I
> really hate to use those as they have a big performance hit and thats a very
> critical section. Its true this is also related to "mutable" attributes
> which is something I'd eventually like to remove entirely - they are pending
> deprecation in 0.7. Is this issue consistently reproducible and can you
> send me a test case ? This would be very high priority.
>
> Not sure if this helps but one of the triggers there is you have an object
> that has "mutable" attributes on it, which is dirty, and has been garbage
> collected. The "state" hangs around in the Session and when accessed
> resurrects itself. At that point, it seems like some other object that was
> also garbage collected starts doing the same thing before the process for
> object #1 can complete, its not clear why that would happen here, thats the
> point at which I'd want to pdb around to see what that's about.
>
This issue was in-consistently reproducible, but it is in a production
system where a workaround has been applied, so now it is not-so-reproducible
(hopefully).
Every object has mutable attributes on it, so it is not surprising that
there would be a dirty one in the session. Here is the do_something_cool
function in case it provides some insight:
def do_something_cool(obj, do_copy=False):
for obj in session:
if do_copy:
ret_obj = copy.deepcopy(obj)
else:
ret_obj = obj
ret_obj._has_been_flagged = True
return ret_obj
I believe do_copy should be False in this situation, but given the weirdness
I would exclude the possibility that it is True.
>
>
> On a somewhat related note, am experiencing the issue where lots of objects
> in a session significantly reduces performance. It appears I am
> experiencing the penalty for using MutableTypes on objects described here:
> http://readthedocs.org/docs/sqlalchemy/en/latest/core/types.html#base-type-api.
> The doc states:
>
> In order to detect changes, the ORM must create a copy of the value when it
> is first accessed, so that changes to the current value can be compared
> against the “clean” database-loaded value. Additionally, when the ORM checks
> to see if any data requires flushing, it must scan through all instances in
> the session which are known to have “mutable” attributes and compare the
> current value of each one to its “clean” value.
>
> It doesn't seem like I should have to pay this penalty because with my type
> I actually know when any updates occur. The type just supports
> models.mutable_value.update(data). Is there a way to notify the
> model/session that the value has been updated when someone calls update, so
> the full scan of all instances is not needed? Do I need to rewrite the
> update function to translate models.mutable_value.update(data) to actually
> generate models.mutable_value = new_mutable_value or is there a better way
> to do this?
>
>
> very easy. Upgrade to 0.7. All has been resolved there, to support
> in-place mutation detection you use the techniques described at
> http://www.sqlalchemy.org/docs/07/orm/extensions/mutable.html .
>
> Of course you'd be beta testing something I'm not sure anyone is using yet.
> Hopefully no deadlocks though ! :) (but again, I'd love to fix that
> deadlock, I'll be looking at that today and a test case would be v. helpful)
>
So there is no way to accomplish this in 0.6? I was looking at doing it the
way I describe above, but it is not trivial to figure out which
model/session a value is attached to. And I guess it is theoretically
possible that a value could be connected to two models/sessions. Definitely
scared of moving to an untested code... but looks like there are lots of
improvements in 0.7 that might make it worth it...
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
