Hi Mike,
Thanks for your quick response yet again! Here’s the stack trace.
tests/integration/test_database.py:14: in test_database_is_up_to_date
create_test_db(session)
__init__.py:111: in create_test_db
pd_utils.do_import(dtype='locations', ifile=yaml_file)
../utils/provider_data/__init__.py:54: in do_import
inserted, updated = getattr(self, 'import_%s' % item)(ifile)
../utils/provider_data/__init__.py:22: in import_locations
return import_locations(self.session, ifile)
../utils/provider_data/locations.py:190: in import_locations
Location).filter_by(
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py:1260:
in query
return self._query_cls(entities, self, **kwargs)
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py:110: in
__init__
self._set_entities(entities)
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py:120: in
_set_entities
self._set_entity_selectables(self._entities)
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py:150: in
_set_entity_selectables
ent.setup_entity(*d[entity])
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py:3250: in
setup_entity
self._with_polymorphic = ext_info.with_polymorphic_mappers
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py:747:
in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py:1893:
in _with_polymorphic_mappers
configure_mappers()
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py:2756:
in configure_mappers
mapper._post_configure_properties()
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py:1710:
in _post_configure_properties
prop.init()
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/interfaces.py:183:
in init
self.do_init()
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:1613:
in do_init
self._setup_join_conditions()
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:1688:
in _setup_join_conditions
can_be_synced_fn=self._columns_are_mapped
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:1956:
in __init__
self._determine_joins()
../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:2060:
in _determine_joins
"specify a 'primaryjoin' expression." % self.prop)
E NoForeignKeysError: Could not determine join condition between parent/child
tables on relationship Resource.tags - there are no foreign keys linking these
tables. Ensure that referencing columns are associated with a ForeignKey or
ForeignKeyConstraint, or specify a 'primaryjoin' expression.
> The sys.modules activity is not really the primary cause, it's that alembic
> makes use of a module object in a temporary way.
Absolutely agree. What I did to diagnose this was to replace secondary=lambda:
tag_to_resource with secondary=non_lambda_tag_to_resource, using this:
def non_lambda_tag_to_resource():
import sys
sys.stderr.write('*** tag_to_resource=%s\n' % tag_to_resource)
#
sys.stderr.write('*** name=%s\n' % __name__)
#
return tag_to_resource
What I found is that in the bad case, both tag_to_resource and __name__ were
None.
Thanks,
Will
> On 21 Feb 2016, at 19:12, Mike Bayer <[email protected]> wrote:
>
>
>
> Hi there -
>
> Can you post a stack trace, and also is your test suite making use of
> clear_mappers() ?
>
> The sys.modules activity is not really the primary cause, it's that alembic
> makes use of a module object in a temporary way.
>
> On Feb 21, 2016, at 1:48 PM, Will Angenent <[email protected]
> <mailto:[email protected]>> wrote:
>
>> Hi,
>>
>> We had this interesting issue recently, and I've been trying to figure out
>> if we deserve this, if this is simply unavoidable, or whether it can be
>> considered a bug. We're using python 2.7.6, sqlalchemy 1.0.12 and alembic
>> 0.8.4.
>>
>> Summary:
>>
>> This statement in alembic.util.pyfiles.load_python_file():
>> del sys.modules[module_id]
>> randomly causes the reference count of the module object to become zero;
>> triggering cleanup of the object. This effectively causes all variables in
>> the migration file to become None, leading to an sqlalchemy mapper problem
>> initializing a mapper configuration for a many-to-many relationship in a
>> model defined in the migration file.
>>
>> Are we being stupid to be using the ORM in alembic migrations? If not, is it
>> worth for me to spend more time on this? Is there any way to get this to
>> behave non-randomly? More details are below.
>>
>> Thanks,
>> Will
>>
>> Long version...
>>
>> What happened is that someone in my team added an alembic migration. He used
>> the sqlalchemy ORM and used a declarative_base with a couple of model files
>> to get the job done. The migration was fine and everyone was happy. Then,
>> about a week later, I added an import statement in a totally unrelated area
>> of code, and suddenly running alembic upgrade starting failing with a ORM
>> mapper error. I didn't spend much time on it, but refactored a couple of
>> things and the problem vanished.
>>
>> Then a couple of days later, our tests started failing with the same error.
>> We had a closer look and found the failure to be random. The inclusion of
>> the import statment seemed to trigger the random behavior. It wasn't just
>> the import statement though, other changes, such as removing a property in
>> an ORM class could make the problem appear or go away. What we were doing in
>> this particualr failure mode, is running py.test which would, in order:
>>
>> - import this random 3rd party module
>> - use the alembic API to upgrade to ensure a postgres database is up to date
>> - later on, in an unrelated test, do a query, triggering the initialization
>> of the mappings and crashing
>>
>> At first, I thought it might be a problem with sqlalchemy. Spurred on by
>> this comment in mapper.py:
>>
>> # initialize properties on all mappers
>> # note that _mapper_registry is unordered, which
>> # may randomly conceal/reveal issues related to
>> # the order of mapper compilation
>>
>> I added a couple of sorted() statements throughout the code, but it made no
>> difference. Finally, I found that the problem was a lambda function in a
>> relationship with a secondary. Something like e.g.
>>
>> tag_to_resource = Table(
>> 'tag_to_resource', Base.metadata,
>> Column('tag_id', ForeignKey('tags.id', ondelete='CASCADE'),
>> primary_key=True, index=True),
>> Column('resource_id', ForeignKey('resources.id', ondelete='CASCADE'),
>> primary_key=True, index=True)
>> )
>>
>> class Resource(Base):
>> __tablename__ = 'resources'
>> id = Column(UUIDType(binary=True), primary_key=True, default=uuid.uuid4)
>>
>> tags = relationship("Tag", secondary=lambda: tag_to_resource,
>> backref='resources')
>>
>> The lambda function called in _process_dependent_arguments() was returning
>> None instead of tag_to_resource. Resulting in a:
>>
>> sqlalchemy.exc.NoForeignKeysError: Could not determine join condition
>> between parent/child tables on relationship Resource.tags - there are no
>> foreign keys linking these tables. Ensure that referencing columns are
>> associated with a ForeignKey or ForeignKeyConstraint, or specify a
>> 'primaryjoin' expression.
>>
>> Looking deeper I found that __name__ was also None. This kind of thing
>> happens when sys.modules is messed with. I looked at the alembic code and
>> found this in load_python_file():
>>
>> del sys.modules[module_id]
>>
>> If I remove that statement, the problem goes away.
>>
>> Could it be that the reference count of the module object is becoming zero
>> randomly, causing python to delete the data, as explained in this post?
>> http://stackoverflow.com/questions/5365562/why-is-the-value-of-name-changing-after-assignment-to-sys-modules-name
>>
>> <http://stackoverflow.com/questions/5365562/why-is-the-value-of-name-changing-after-assignment-to-sys-modules-name>
>>
>> I've narrowed the problem down to a python test script, but it still imports
>> a load of other stuff. I can trigger the good + bad case by just removing an
>> import statement. I've been trying to get this down to a simple script in an
>> attempt to prove what's going on, but the problem tends to come and go while
>> I'm deleting code; making it difficult to narrow down. For example, I was
>> convinced one day that the problem vanished by upgrading to sql alchemy
>> 1.0.12, but the very next day the same code started failing again!
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected]
>> <mailto:[email protected]>.
>> To post to this group, send email to [email protected]
>> <mailto:[email protected]>.
>> Visit this group at https://groups.google.com/group/sqlalchemy
>> <https://groups.google.com/group/sqlalchemy>.
>> For more options, visit https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To post to this group, send email to [email protected]
> <mailto:[email protected]>.
> Visit this group at https://groups.google.com/group/sqlalchemy
> <https://groups.google.com/group/sqlalchemy>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
