Hi Cristian,

Unfortunately, I *have* seen this error before (though never on my own
hardware) and we thought we'd fixed it a little while back.  It's a weird
race condition in kombu[1] mapper initialization under sqlalchemy.  We use
this library for message passing between processes.  Usually kombu is bound
to a message queue like RabbitMQ, but it can also use a traditional
database if one isn't available.

I'm going to spend some more time on this today and upgrade our kombu
dependency to the latest version and can get back to you with hopefully
better news, but if you need to fix this *right now*, one sure way to do it
is to set up one of the other transports[2] and specify the
'amqp_internal_connection' entry in your galaxy.ini.  We've had success
with rabbitmq, but mongo, redis, or really any of the other non-sqlalchemy
transports should work if you already have infrastructure for it.

One of the primary reasons we chose kombu for handling Galaxy's messaging
was so that things *would* continue to just work with existing
infrastructure without requiring a message queue, so this is definitely
something I'm going to figure out.

Thanks!

Dannon

[1] https://github.com/celery/kombu/
[2]
https://kombu.readthedocs.org/en/latest/userguide/connections.html#amqp-transports


On Thu, Apr 9, 2015 at 5:36 AM, C. Ch. <tuto...@hotmail.com> wrote:

> Hi all,
>
> We had a problem on the NAS where we couldn't write anymore. This problem
> was solved and I decided that I would profit to upgrade galaxy.
>
> Now we have a problem where jobs do not get launched and the handler log
> shows a lot of error messages. The first error message is copied below.
> Afterwards it alternates between
>
>  galaxy.jobs.handler ERROR 2015-04-07 15:58:24,959 Exception in
> monitor_step
>  galaxy.jobs.runners ERROR 2015-04-07 15:58:23,542 Unhandled exception
> checking active jobs
>
> with, mainly, the same error stack (second error message shown below).
>
> Any idea what is going wrong?
>
> My first impression is that it is trying to recuperate some job that
> galaxy believes is still running but the machine has been stopped and
> rebooted in the mean time.
>
> Any help is welcome!
>
> Cristian
>
>
> galaxy.jobs.runners ERROR 2015-04-07 15:44:04,809 (27764) Unhandled
> exception calling queue_job
> Traceback (most recent call last):
>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
> line 97, in run_next
>     method(arg)
>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line
> 115, in queue_job
>     if not self.prepare_job( job_wrapper, include_metadata=True ):
>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
> line 140, in prepare_job
>     job_id = job_wrapper.get_id_tag()
>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 794,
> in get_id_tag
>     return self.get_job().get_id_tag()
>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 790,
> in get_job
>     return self.sa_session.query( model.Job ).get( self.job_id )
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py",
> line 840, in get
>     return loading.load_on_ident(self, key)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/loading.py",
> line 231, in load_on_ident
>     return q.one()
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py",
> line 2395, in one
>     ret = list(self)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py",
> line 2434, in __iter__
>     context = self._compile_context()
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py",
> line 2810, in _compile_context
>     entity.setup_context(self, context)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py",
> line 3194, in setup_context
>     column_collection=context.primary_columns
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/interfaces.py",
> line 466, in setup
>     strat.setup_query(context, entity, path, loader, adapter, **kwargs)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py",
> line 1111, in setup_query
>     column_collection, parentmapper, chained_from_outerjoin
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py",
> line 1234, in _generate_row_adapter
>     use_mapper_path=True)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/util.py",
> line 364, in __init__
>     else mapper.with_polymorphic_mappers,
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py",
> line 725, in __get__
>     obj.__dict__[self.__name__] = result = self.fget(obj)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py",
> line 1877, in _with_polymorphic_mappers
>     configure_mappers()
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py",
> line 2589, in configure_mappers
>     mapper._post_configure_properties()
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py",
> line 1694, in _post_configure_properties
>     prop.init()
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/interfaces.py",
> line 144, in init
>     self.do_init()
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/relationships.py",
> line 1549, in do_init
>     self._process_dependent_arguments()
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/relationships.py",
> line 1605, in _process_dependent_arguments
>     self.target = self.mapper.mapped_table
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py",
> line 725, in __get__
>     obj.__dict__[self.__name__] = result = self.fget(obj)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/relationships.py",
> line 1522, in mapper
>     argument = self.argument()
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/ext/declarative/clsregistry.py",
> line 283, in __call__
>     (self.prop.parent, self.arg, n.args[0], self.cls)
> InvalidRequestError: When initializing mapper Mapper|Queue|kombu_queue,
> expression 'Message' failed to locate a name ("name 'Message' is not
> defined"). If this is a class name, consider adding this relationship() to
> the <class 'kombu.transport.sqlalchemy.Queue'> class after both dependent
> classes have been defined.
>
> Repeated error messages:
> galaxy.jobs.runners ERROR 2015-04-07 15:58:23,542 Unhandled exception
> checking active jobs
> Traceback (most recent call last):
>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
> line 501, in monitor
>     self.check_watched_items()
>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line
> 238, in check_watched_items
>     galaxy_id_tag = ajs.job_wrapper.get_id_tag()
>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 794,
> in get_id_tag
>     return self.get_job().get_id_tag()
>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 790,
> in get_job
>     return self.sa_session.query( model.Job ).get( self.job_id )
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py",
> line 150
> , in do
>     return getattr(self.registry(), name)(*args, **kwargs)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py",
> line 116
> 5, in query
>     return self._query_cls(entities, self, **kwargs)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py",
> line 108,
> in __init__
>     self._set_entities(entities)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py",
> line 118,
> in _set_entities
>     self._set_entity_selectables(self._entities)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py",
> line 151,
> in _set_entity_selectables
>     ent.setup_entity(*d[entity])
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py",
> line 3036,
>  in setup_entity
>     self._with_polymorphic = ext_info.with_polymorphic_mappers
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py",
> lin
> e 725, in __get__
>     obj.__dict__[self.__name__] = result = self.fget(obj)
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py",
> line 1877
> , in _with_polymorphic_mappers
>     configure_mappers()
>   File
> "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py",
> line 2586
> , in configure_mappers
>     raise e
> InvalidRequestError: One or more mappers failed to initialize - can't
> proceed with initialization of other mappers.  Or
> iginal exception was: When initializing mapper Mapper|Queue|kombu_queue,
> expression 'Message' failed to locate a name (
> "name 'Message' is not defined"). If this is a class name, consider adding
> this relationship() to the <class 'kombu.tra
> nsport.sqlalchemy.Queue'> class after both dependent classes have been
> defined.
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to