Re: [sqlalchemy] Trouble with ordering_list extension

[Mike Bayer]

On Wed, Jun 28, 2017 at 12:02 AM, Andy Green 
wrote:

> Now, some investigating has revealed the following information:
> - commenting out the `__reduce__` function on OrderedList seems to make
> everything work correctly (including the unit tests on orderinglist.py).
> This function is definitely being called, though my code is not pickling
> the object (perhaps SQLAlchemy is calling this?)
>

you actually don't know the function is being called if removing it changes
behavior elsewhere, because lots of things in Python depend on
non-deterministic things like dictionary ordering, which is essentially
random, but can be seeded differently when other subtle things change, like
whether or not .pyc files are already present.   You should set
PYTHONHASHSEED=random in your environment, and to prove __reduce__ and the
internal _reconstitute aren't being called you can put an exception throw
or print statement in them.  SQLAlchemy does not call __reduce__ or do any
pickling, unless you run the one test in test_orderinglist that itself
tests pickling.


> - in the last frame of the above traceback, self._data() is a weakref, and
> the fact that it is returning None suggests the object (which I believe
> would be the instance of OrderingList in this case) has been deleted. I can
> only explain this if somehow the relationship list is being replaced,
> though I can find no indication this is true in my code.
>

unfortunately neither can I if you can't provide an MCVE (what you refer to
as an MWE :) ).


> - Simply replacing the collection_class with `list` makes the code work as
> expected
> - The unit tests for orderinglist (py.test test/ext/test_orderinglist.py)
> all pass.
>
> I'm out of time to debug this, and orderinglist doesn't provide quite what
> I need anyway (which is really more an order preserving dict such as
> OrderedDict), so I'm afraid I can't offer a lot more here. It seems to me
> that `ordering_list` is not actually working at all currently, but that the
> unit tests don't actually save and restore an object containing this
> relationship to a database, and hence it isn't being picked up. I suspect
> that `__reduce__` is being called, and it is inadvertently replacing the
> OrderingList created when the relationship is created, and hence the
> original object is being garbage collected and therefore the weakref
> mentioned above is void, causing the exception. Quick fix would be to
> simply not have a `__reduce__` function on OrderingList, but I'm afraid I
> don't understand enough about pickling to know if this would cause problems.
>

try the PYTHONHASHSEED=random variable, which is almost always how to smoke
out these kinds of bugs.


>
> Cheers,
> Andy
>
>
>
>
>
>
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to sqlalchemy@googlegroups.com.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] Trouble with ordering_list extension

[Andy Green]

Hi All,

I'm having some trouble using an OrderingList as a relationship 
"collection_class". I suspect there is a bug in the implementation of the 
`__reduce__` function on the OrderingList class. 

I'm using SQLAlchemy version 1.1.11.

Posting a MWE or my actual code is going to take a fair bit of time, so 
instead let me explain my situation and what testing I have done.

I'm using a relationship between two classes of objects in the usual way, 
but with an ordered_list as the collection class:

class TraitMappingBase(bases.PersistenceBase, bases.SQLAlchemyBase):

__tablename__ = "trait_mappings"  # Note this table is shared with 
TraitMapping and SubTraitMapping classes.

_db_type = sa.Column('type', sa.String(50))
__mapper_args__ = {'polymorphic_on': "_db_type"}

_database_id = sa.Column(sa.Integer, sa.Sequence('trait_mapping_seq'), 
primary_key=True)
_db_trait_class = sa.Column(sa.String)
_parent_id = sa.Column(sa.Integer, 
sa.ForeignKey('trait_mappings._database_id'))

_db_trait_property_mappings = relationship(
'TraitPropertyMapping',
order_by='TraitPropertyMapping.ordering',
collection_class=ordering_list('ordering'))  # type: 
List[TraitPropertyMapping]


class TraitPropertyMapping(bases.SQLAlchemyBase, bases.PersistenceBase):

# Database fields and setup (SQLAlchemy)
__tablename__ = "trait_property_mappings"
database_id = sa.Column(sa.Integer, sa.Sequence('trait_mapping_seq'), 
primary_key=True)
name = sa.Column(sa.String)
id = sa.Column(sa.String)
ordering = sa.Column(sa.Integer)  # Column for ordering index.
_trait_mappings = relationship('TraitMappingBase', 
back_populates='_db_trait_property_mappings')
_trait_mapping_id = sa.Column(sa.Integer, 
sa.ForeignKey('trait_mappings._database_id'))


Alchemy reports an exception when it tries to add an object of the 
TraitMappingBase class to the session:

Traceback (most recent call last):

  File 
"/Users/agreen/Documents/ASVO/code/git-repository/python_packages/fidia_tests/test_archive.py",
 
line 111, in example_archive

return ExampleArchive(basepath=test_data_dir)

  File 
"/Users/agreen/Documents/ASVO/code/git-repository/python_packages/fidia/archive/archive.py",
 
line 287, in __new__

fidia.mappingdb_session.add(archive)

  File 
"/Users/agreen/python/virtualenvs/asvo-server/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
 
line 1711, in add

self._save_or_update_state(state)

  File 
"/Users/agreen/python/virtualenvs/asvo-server/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
 
line 1729, in _save_or_update_state

halt_on=self._contains_state):

  File 
"/Users/agreen/python/virtualenvs/asvo-server/lib/python3.4/site-packages/sqlalchemy/orm/mapper.py",
 
line 2709, in cascade_iterator

visited_states, halt_on))

  File 
"/Users/agreen/python/virtualenvs/asvo-server/lib/python3.4/site-packages/sqlalchemy/orm/relationships.py",
 
line 1553, in cascade_iterator

get_all_pending(state, dict_)

  File 
"/Users/agreen/python/virtualenvs/asvo-server/lib/python3.4/site-packages/sqlalchemy/orm/attributes.py",
 
line 929, in get_all_pending

for c in current]

  File 
"/Users/agreen/python/virtualenvs/asvo-server/lib/python3.4/site-packages/sqlalchemy/orm/collections.py",
 
line 654, in __iter__

return iter(self._data()._sa_iterator())

AttributeError: 'NoneType' object has no attribute '_sa_iterator'

Now, some investigating has revealed the following information:
- commenting out the `__reduce__` function on OrderedList seems to make 
everything work correctly (including the unit tests on orderinglist.py). 
This function is definitely being called, though my code is not pickling 
the object (perhaps SQLAlchemy is calling this?)
- in the last frame of the above traceback, self._data() is a weakref, and 
the fact that it is returning None suggests the object (which I believe 
would be the instance of OrderingList in this case) has been deleted. I can 
only explain this if somehow the relationship list is being replaced, 
though I can find no indication this is true in my code.
- Simply replacing the collection_class with `list` makes the code work as 
expected
- The unit tests for orderinglist (py.test test/ext/test_orderinglist.py) 
all pass.

I'm out of time to debug this, and orderinglist doesn't provide quite what 
I need anyway (which is really more an order preserving dict such as 
OrderedDict), so I'm afraid I can't offer a lot more here. It seems to me 
that `ordering_list` is not actually working at all currently, but that the 
unit tests don't actually save and restore an object containing this 
relationship to a database, and hence it isn't being picked up. I suspect 
that `__reduce__` is being called, and it is inadvertently replacing the 
OrderingList created when the relationship is created, and hence the 
original object is being garbage collected and therefore the weakref 
mentioned above is void,