Dear John, thank you very much your help and the warnings!
I'm testing it with a second database and so far it seems to work :-). I was also suspecting the encoding to be the issue, that's why I did it through pg_dump and specified the encoding of the new server for the dump. The error stayed the same and so far I have just seen it in the context of toolshed operations. I don't know what else I could do to ensure proper encoding. Best regards, Sarah ----- Original Message ----- From: "John Chilton" <jmchil...@gmail.com> To: "Sarah Diehl" <di...@ie-freiburg.mpg.de> Cc: "galaxy-dev@lists.bx.psu.edu List" <galaxy-dev@lists.bx.psu.edu> Sent: Thursday, October 16, 2014 3:17:39 PM Subject: Re: [galaxy-dev] Help with Galaxy server migration Hmm.... hopefully someone more knowledgeable about the tool shed than me responds also but I had a couple quick thoughts. The first is a warning - workflows may break. At very least workflows that depend on the previous instance having gone through tool migrations instead of tool install. My understanding is tool migrations cause links to the older versions of the tools to be created so workflows continue to operate - just installing the same version of the tool again doesn't set this up. Also - in theory if you resinstall the same versions of tools tool re-runs and stuff should still work - but I am not confident and I have not tested it myself - so if that functionality is important to you, you test it out before throwing out the old configuration. Next - I am a little bit concerned about this error. It would seem to point to some sort of encoding problem with the way the database was transferred - different table encodings or something? I don't really know anything about encoding and postgres though. That said I am not surprised there are problems with the tool shed. There are all sorts of implicit dependencies between what is on disk and what is in the database. If you really do want to start fresh I would clear out all of the tool shed tables before continuing. These tables including - tool_shed_repository,repository_repository_dependency_association, repository_dependency, tool_dependency, tool_version, tool_version_association, migrate_tools. Rather than truncating the existing tables - you could also just have Galaxy target a completely separate database for tool shed install stuff by creating a new postgres database, setting install_database_connection in your galaxy ini file (either universe_wsgi.ini or galaxy.ini depending on what version of Bjoern's docker image you are targetting) - or in the newest version of that Dockerfile you can just make sure your docker run includes a -e "GALAXY_CONFIG_INSTALL_DATABASE_CONNECTION=<postgresconnectionstring>". You will also want to make sure that database gets populated - I think either of the following would work ./create_db.sh install ./migrate_db.sh install upgrade Sorry I have not definitive answers - but hopefully this is helpful. Good luck with the migration. You should let the list know how the Docker container thing goes for a production server. -John On Thu, Oct 16, 2014 at 8:36 AM, Sarah Diehl <di...@ie-freiburg.mpg.de> wrote: > Dear all, > > I'm trying to migrate our local Galaxy server from a standalone server > running on CentOS 5 to a setup that runs Galaxy in a Docker container > (bgruening/galaxy-stable). Due to Docker and the setup of the container, > almost all data, tools, configs etc. have to move to a different place. Since > our old server is pretty cluttered up anyway, I would like to move just user > data (logins, histories, datasets ...), but none of the tool installations, > tool data or configs. > > My plan was to start with a fresh and clean new Galaxy installation, transfer > the datasets and the database and then reinstall all tools through the > toolshed (of course there are many more small odds and ends that need fixing). > > The problem is that after transfering the database, I always get an error > when going to "Manage installed tool shed repositories" (see end of the > mail). I tried just overwriting the pg_data directory (PostgreSQL versions > are identical) and also exporting to sql (pg_dump) and importing in the new > DB. Both times I made sure to run manage_db.sh upgrade afterwards. The error > stays the same. > > I can't make much sense of the error message, my only guess is that it looks > for something that's not there. However, I really don't want to transfer the > tool directories and tool configurations, because that will get messy and > difficult, since many paths need to be adjusted. There's a high chance for > error and this way I also carry over the clutter. > > So is there a way to remove all tool installation information from the > database, before I transfer it? Can this even be disentangled from the > histories and datasets? Any help or other ideas on how to do this migration > are highly appreciated! > > Thanks, > Sarah > > > > 10.1.5.190 - - [16/Oct/2014:09:54:35 +0000] "GET > /admin_toolshed/browse_repositories HTTP/1.1" 500 - > "http://deep2:8080/admin/index"; "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:32.0) Gecko/20100101 Firefox/32.0" > Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode > byte 0xe2 in position 393: ordinal not in range(128) > URL: http://deep2:8080/admin_toolshed/browse_repositories > File 'lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ > app_iter = self.application(environ, sr_checker) > File '/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line > 84 in __call__ > return self.application(environ, start_response) > File '/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', > line 633 in __call__ > return self.application(environ, start_response) > File 'lib/galaxy/web/framework/base.py', line 132 in __call__ > return self.handle_request( environ, start_response ) > File 'lib/galaxy/web/framework/base.py', line 190 in handle_request > body = method( trans, **kwargs ) > File 'lib/galaxy/web/framework/decorators.py', line 87 in decorator > return func( self, trans, *args, **kwargs ) > File 'lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 167 in > browse_repositories > return self.installed_repository_grid( trans, **kwd ) > File 'lib/galaxy/web/framework/helpers/grids.py', line 306 in __call__ > kwargs=kwargs ) > File 'lib/galaxy/web/framework/webapp.py', line 771 in fill_template > return self.fill_template_mako( filename, **kwargs ) > File 'lib/galaxy/web/framework/webapp.py', line 786 in fill_template_mako > return template.render( **data ) > File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line 296 > in render > return runtime._render(self, self.callable_, args, data) > File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 660 in > _render > **_kwargs_for_callable(callable_, data)) > File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 692 in > _render_context > _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) > File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 718 in > _exec_template > callable_(context, *args, **kwargs) > File '/galaxy-central/database/compiled_templates/base.mako.py', line 66 in > render_body > __M_writer(unicode(next.body())) > File '/galaxy-central/database/compiled_templates/grid_base.mako.py', line 91 > in render_body > __M_writer(unicode(self.load())) > File '/galaxy-central/database/compiled_templates/grid_base.mako.py', line > 120 in render_load > __M_writer(unicode( h.dumps( self.get_grid_config( embedded=embedded, > insert=insert ) ) )) > File '/galaxy-central/database/compiled_templates/grid_base.mako.py', line > 303 in render_get_grid_config > value = column.get_value( trans, grid, item ) > File 'lib/tool_shed/galaxy_install/grids/admin_toolshed_grids.py', line 106 > in get_value > return suc.get_tool_shed_repository_status_label( trans.app, > tool_shed_repository ) > File 'lib/tool_shed/util/shed_util_common.py', line 829 in > get_tool_shed_repository_status_label > elif tool_shed_repository.tool_dependencies_being_installed: > File 'lib/galaxy/model/tool_shed_install/__init__.py', line 408 in > tool_dependencies_being_installed > for tool_dependency in self.tool_dependencies: > File > '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py', > line 168 in __get__ > File > '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py', > line 453 in get > File > '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py', > line 508 in _load_for_state > File > '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py', > line 574 in _emit_lazyload > File > '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', > line 2115 in all > File > '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', > line 2341 in instances > File > '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py', > line 3204 in fetchall > File > '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py', > line 3171 in _fetchall_impl > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 393: > ordinal not in range(128) > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
