Re: [galaxy-dev] Database migration error
Some additional info; Actually the errors already started at migrate version 93->94. I just restored the version to prior upgrade to check any other errors. There were some create errors. I am not sure if these come from the \n\t statements in the sysntax or whether that is just an error log output thing only. Bottom line is that some tables are not created. Some seem to already exist due to the partial pass of the upgrade the previous time. See log below; Would it be possible to just create a fresh latest version database with no data and then transfer the data of the backup I have into that? Alex = sudo sh manage_db.sh -c universe_wsgi.ini upgrade 93 -> 94... Migration script to create "handler" column in job table. (OperationalError) (1060, "Duplicate column name 'handler'") u'\nALTER TABLE job ADD handler VARCHAR(255)' () done 94 -> 95... Migration script to create table for tracking history_dataset_association subsets. (OperationalError) (1005, "Can't create table 'galaxy_dist.history_dataset_association_subset' (errno: 150)") u'\nCREATE TABLE history_dataset_association_subset (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\thistory_dataset_association_id INTEGER, \n\thistory_dataset_association_subset_id INTEGER, \n\tlocation VARCHAR(255), \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(history_dataset_association_id) REFERENCES history_dataset_association (id), \n\t FOREIGN KEY(history_dataset_association_subset_id) REFERENCES history_dataset_association (id)\n)\n\n' () (ProgrammingError) (1146, "Table 'galaxy_dist.history_dataset_association_subset' doesn't exist") 'CREATE INDEX ix_hda_id ON history_dataset_association_subset (history_dataset_association_id)' () done 95 -> 96... Migration script to add column to openid table for provider. Remove any OpenID entries with nonunique GenomeSpace Identifier Adding provider column to galaxy_user_openid table failed: (OperationalError) (1060, "Duplicate column name 'provider'") u'\nALTER TABLE galaxy_user_openid ADD provider VARCHAR(255)' () done 96 -> 97... Migration script to add the ctx_rev column to the tool_shed_repository table. Adding ctx_rev column to the tool_shed_repository table failed: (OperationalError) (1060, "Duplicate column name 'ctx_rev'") u'\nALTER TABLE tool_shed_repository ADD ctx_rev VARCHAR(10)' () done 97 -> 98... Migration script to create the genome_index_tool_data table. 0098_genome_index_tool_data_table DEBUG 2012-10-12 11:27:46,128 Creating genome_index_tool_data table failed: (OperationalError) (1005, "Can't create table 'galaxy_dist.genome_index_tool_data' (errno: 150)") u'\nCREATE TABLE genome_index_tool_data (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tjob_id INTEGER, \n\tdataset_id INTEGER, \n\tdeferred_job_id INTEGER, \n\ttransfer_job_id INTEGER, \n\tfasta_path VARCHAR(255), \n\tcreated_time DATETIME, \n\tmodified_time DATETIME, \n\tindexer VARCHAR(64), \n\tuser_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(dataset_id) REFERENCES dataset (id), \n\t FOREIGN KEY(transfer_job_id) REFERENCES transfer_job (id), \n\t FOREIGN KEY(user_id) REFERENCES galaxy_user (id), \n\t FOREIGN KEY(deferred_job_id) REFERENCES deferred_job (id), \n\t FOREIGN KEY(job_id) REFERENCES job (id)\n)\n\n' () 0098_genome_index_tool_data_table DEBUG 2012-10-12 11:27:46,128 Creating genome_index_tool_data table failed: (OperationalError) (1005, "Can't create table 'galaxy_dist.genome_index_tool_data' (errno: 150)") u'\nCREATE TABLE genome_index_tool_data (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tjob_id INTEGER, \n\tdataset_id INTEGER, \n\tdeferred_job_id INTEGER, \n\ttransfer_job_id INTEGER, \n\tfasta_path VARCHAR(255), \n\tcreated_time DATETIME, \n\tmodified_time DATETIME, \n\tindexer VARCHAR(64), \n\tuser_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(dataset_id) REFERENCES dataset (id), \n\t FOREIGN KEY(transfer_job_id) REFERENCES transfer_job (id), \n\t FOREIGN KEY(user_id) REFERENCES galaxy_user (id), \n\t FOREIGN KEY(deferred_job_id) REFERENCES deferred_job (id), \n\t FOREIGN KEY(job_id) REFERENCES job (id)\n)\n\n' () done 98 -> 99... Migration script to add the tool_dependency table. 0099_add_tool_dependency_table DEBUG 2012-10-12 11:27:46,268 Creating tool_dependency table failed: (OperationalError) (1005, "Can't create table 'galaxy_dist.tool_dependency' (errno: 150)") u'\nCREATE TABLE tool_dependency (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tcreate_time DATETIME, \n\tupdate_time DATETIME, \n\ttool_shed_repository_id INTEGER NOT NULL, \n\tinstalled_changeset_revision VARCHAR(255), \n\tname VARCHAR(255), \n\tversion VARCHAR(40), \n\ttype VARCHAR(40), \n\tuninstalled BOOL, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_shed_repository_id) REFERENCES tool_shed_repository (id)\n)\n\n' () 0099_add_tool_dependency_table DEBUG 2012-10-12 11:27:46,268 Creating tool_dependency table failed: (OperationalError) (1005, "Can't create table 'galax
[galaxy-dev] Database migration error
After a fresh galaxy-dist pull today (no update for various reasons) the instance was running ok on the default sqlight database. When I migrated back the setting to use the MySQL database it complained during startup about having found an old version of the database and I need to upgrade. Database upgrade started and the first few migrations were successful, but now it hangs every time at 99->100. Any clue what might be wrong? Here is the lengthy trace log. thanks Alex 99 -> 100... Migration script to alter the type of the tool_dependency.version column from TrimmedString(40) to Text. Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in main( repository=repo, url=db_url ) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0100_alter_tool_dependency_table_version_column.py", line 30, in upgrade ToolDependency_table = Table( "tool_dependency", metadata, autoload=True ) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/schema.py", line 108, in __call__ return type.__call__(self, name, metadata, *args, **kwargs) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/schema.py", line 236, in __init__ _bind_or_error(metadata).reflecttable(self, include_columns=include_columns) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 1265, in reflecttable self.dialect.reflecttable(conn, table, include_columns) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/databases/mysql.py", line 1664, in reflecttable sql = self._show_create_table(connection, table, charset) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/databases/mysql.py", line 1835, in _show_create_table raise exc.NoSuchTableError(full_name) sqlalchemy.exc.NoSuchTableError: tool_dependency Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook from apport.fileutils import likely_packaged, get_recent_crashes ImportError: No module named apport.fileutils Original exception was: Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in main( repository=repo, url=db_url ) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/opt/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0100_alter_tool_dependency_table_version_column.py", line 30, in upgrade ToolDependency_table = Table( "tool_dependency", metadata, autoload=True ) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/schema.py", line 108, in __call__ return type.__call__(self, name, metadata, *args, **kwargs) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/schema.py", line 236, in __init__ _bind_or_error(metadata).reflecttable(self, include_columns=include_columns) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 1265, in reflecttable self.dialect.reflecttable(conn, table, include_columns) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/databases/mysql.py", line 1664, in reflecttable sql = self._show_create_table(connection, table, charset) File "/opt/galaxy_dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/databases/mysql.py", line 1835
Re: [galaxy-dev] database migration error 79->80 (NameError: name 'BigInteger' is not defined)
Edward Kirton wrote:
> hi, we are getting the following error migrating from 79 to 80.
> curiously, my own galaxy didn't have this problem, but another
> developer here was getting this error. i moved him from sqlite to
> postgres but that didn't help.
> thanks for any assistance,
> ed
Hi Ed,
Is this with the current galaxy-dist version?
--nate
>
> Migrating 79 -> 80...
> galaxy.model.migrate.check INFO 2011-08-31 13:15:31,446
> Traceback (most recent call last):
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/web/buildapp.py",
> line 82, in app_factory
> app = UniverseApplication( global_conf = global_conf, **kwargs )
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/app.py",
> line 32, in __init__
> create_or_verify_database( db_url, kwargs.get( 'global_conf', {}
> ).get( '__file__', None ), self.config.database_engine_options )
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/model/migrate/check.py",
> line 67, in create_or_verify_database
> migrate_to_current_version( engine, db_schema )
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/model/migrate/check.py",
> line 125, in migrate_to_current_version
> schema.runchange( ver, change, changeset.step )
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py",
> line 184, in runchange
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
> line 100, in run
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
> line 112, in _func
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
> line 108, in module
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
> line 65, in verify_module
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/util/importpath.py",
> line 12, in import_path
> File "lib/galaxy/model/migrate/versions/0080_quota_tables.py", line
> 29, in
> Column( "bytes", BigInteger ),
> NameError: name 'BigInteger' is not defined
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
> http://lists.bx.psu.edu/
>
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] database migration error 79->80 (NameError: name 'BigInteger' is not defined)
please disregard my previous message, it was the developer's error; he
accidentally had deleted or reverted something in the libs folder.
On Wed, Aug 31, 2011 at 3:36 PM, Edward Kirton wrote:
> hi, we are getting the following error migrating from 79 to 80.
> curiously, my own galaxy didn't have this problem, but another
> developer here was getting this error. i moved him from sqlite to
> postgres but that didn't help.
> thanks for any assistance,
> ed
>
> Migrating 79 -> 80...
> galaxy.model.migrate.check INFO 2011-08-31 13:15:31,446
> Traceback (most recent call last):
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/web/buildapp.py",
> line 82, in app_factory
> app = UniverseApplication( global_conf = global_conf, **kwargs )
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/app.py",
> line 32, in __init__
> create_or_verify_database( db_url, kwargs.get( 'global_conf', {}
> ).get( '__file__', None ), self.config.database_engine_options )
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/model/migrate/check.py",
> line 67, in create_or_verify_database
> migrate_to_current_version( engine, db_schema )
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/model/migrate/check.py",
> line 125, in migrate_to_current_version
> schema.runchange( ver, change, changeset.step )
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py",
> line 184, in runchange
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
> line 100, in run
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
> line 112, in _func
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
> line 108, in module
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
> line 65, in verify_module
> File
> "/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/util/importpath.py",
> line 12, in import_path
> File "lib/galaxy/model/migrate/versions/0080_quota_tables.py", line
> 29, in
> Column( "bytes", BigInteger ),
> NameError: name 'BigInteger' is not defined
>
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] database migration error 79->80 (NameError: name 'BigInteger' is not defined)
hi, we are getting the following error migrating from 79 to 80.
curiously, my own galaxy didn't have this problem, but another
developer here was getting this error. i moved him from sqlite to
postgres but that didn't help.
thanks for any assistance,
ed
Migrating 79 -> 80...
galaxy.model.migrate.check INFO 2011-08-31 13:15:31,446
Traceback (most recent call last):
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/web/buildapp.py",
line 82, in app_factory
app = UniverseApplication( global_conf = global_conf, **kwargs )
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/app.py",
line 32, in __init__
create_or_verify_database( db_url, kwargs.get( 'global_conf', {}
).get( '__file__', None ), self.config.database_engine_options )
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/model/migrate/check.py",
line 67, in create_or_verify_database
migrate_to_current_version( engine, db_schema )
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-jeff/lib/galaxy/model/migrate/check.py",
line 125, in migrate_to_current_version
schema.runchange( ver, change, changeset.step )
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py",
line 184, in runchange
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
line 100, in run
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
line 112, in _func
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
line 108, in module
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py",
line 65, in verify_module
File
"/house/groupdirs/genetic_analysis/jfroula/Projects/Galaxy/galaxy-depot/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/util/importpath.py",
line 12, in import_path
File "lib/galaxy/model/migrate/versions/0080_quota_tables.py", line
29, in
Column( "bytes", BigInteger ),
NameError: name 'BigInteger' is not defined
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] Database migration
Hi all, I have problems with updating my local instance of Galaxy. Specifically I can not migrate the database schema from 80 to 81: -- $$ sh manage_db.sh upgrade 80 -> 81... Migration script to add a 'tool_version' column to the hda/ldda tables. Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in main( repository=repo, url=db_url ) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/home/zhujack/bin/galaxy/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0081_add_tool_version_to_hda_ldda.py", line 17, in upgrade metadata.reflect() File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 1733, in reflect Table(name, self, **reflect_opts) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 108, in __call__ return type.__call__(self, name, metadata, *args, **kwargs) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 236, in __init__ _bind_or_error(metadata).reflecttable(self, include_columns=include_columns) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1265, in reflecttable self.dialect.reflecttable(conn, table, include_columns) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/databases/mysql.py", line 1673, in reflecttable only=include_columns) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/databases/mysql.py", line 2139, in reflect self._add_column(table, line, charset, only) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/databases/mysql.py", line 2227, in _add_column type_instance = col_type(*type_args, **type_kw) File "/home/zhujack/bin/galaxy/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/databases/mysql.py", line 569, in __init__ super(MSBigInteger, self).__init__(display_width, **kw) TypeError: super(type, obj): obj must be an instance or subtype of type -- Your help will be greatly appreciated. Jack ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] database migration issue 64 ->65 - postgresql
Sorry, for some reason, I attached the compiled version of the script to my
initial message - here's the code...
"""
Migration script to add 'name' attribute to the JSON dict which describes
a form definition field and the form values in the database. In the 'form_values'
table, the 'content' column is now a JSON dict instead of a list.
"""
from sqlalchemy import *
from sqlalchemy.orm import *
from migrate import *
from migrate.changeset import *
from sqlalchemy.exc import *
from galaxy.util.json import from_json_string, to_json_string
from galaxy.model.custom_types import _sniffnfix_pg9_hex
import datetime
now = datetime.datetime.utcnow
import sys, logging
log = logging.getLogger( __name__ )
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler( sys.stdout )
format = "%(name)s %(levelname)s %(asctime)s %(message)s"
formatter = logging.Formatter( format )
handler.setFormatter( formatter )
log.addHandler( handler )
metadata = MetaData( migrate_engine )
db_session = scoped_session( sessionmaker( bind=migrate_engine, autoflush=False, autocommit=True ) )
def upgrade():
print __doc__
metadata.reflect()
try:
FormDefinition_table = Table( "form_definition", metadata, autoload=True )
except Exception, e:
log.debug( "Loading 'form_definition' table failed: %s" % str( e ) )
try:
FormValues_table = Table( "form_values", metadata, autoload=True )
except Exception, e:
log.debug( "Loading 'form_values' table failed: %s" % str( e ) )
def get_value(lst, index):
try:
return str(lst[index]).replace("'", "''")
except IndexError,e:
return ''
# Go through the entire table and add a 'name' attribute for each field
# in the list of fields for each form definition
cmd = "SELECT f.id, f.fields FROM form_definition AS f"
result = db_session.execute( cmd )
for row in result:
form_definition_id = row[0]
fields = str( row[1] )
if not fields.strip():
continue
fields_list = from_json_string( _sniffnfix_pg9_hex( fields ) )
if len( fields_list ):
for index, field in enumerate( fields_list ):
field[ 'name' ] = 'field_%i' % index
field[ 'helptext' ] = field[ 'helptext' ].replace("'", "''").replace('"', "")
field[ 'label' ] = field[ 'label' ].replace("'", "''")
fields_json = to_json_string( fields_list )
if migrate_engine.name == 'mysql':
cmd = "UPDATE form_definition AS f SET f.fields='%s' WHERE f.id=%i" %( fields_json, form_definition_id )
else:
cmd = "UPDATE form_definition SET fields='%s' WHERE id=%i" %( fields_json, form_definition_id )
db_session.execute( cmd )
# replace the values list in the content field of the form_values table with a name:value dict
cmd = "SELECT form_values.id, form_values.content, form_definition.fields" \
" FROM form_values, form_definition" \
" WHERE form_values.form_definition_id=form_definition.id" \
" ORDER BY form_values.id ASC"
result = db_session.execute( cmd )
for row in result:
form_values_id = int( row[0] )
if not str( row[1] ).strip():
continue
row1 = str(row[1]).replace('\n', '').replace('\r', '')
values_list = from_json_string( str( row1 ).strip() )
if not str( row[2] ).strip():
continue
fields_list = from_json_string( str( row[2] ).strip() )
if fields_list and type(values_list) == type(list()):
values_dict = {}
for field_index, field in enumerate( fields_list ):
field_name = field[ 'name' ]
values_dict[ field_name ] = get_value(values_list, field_index )
cmd = "UPDATE form_values SET content='%s' WHERE id=%i" %( to_json_string( values_dict ), form_values_id )
db_session.execute( cmd )
def downgrade():
metadata.reflect()
try:
FormDefinition_table = Table( "form_definition", metadata, autoload=True )
except Exception, e:
log.debug( "Loading 'form_definition' table failed: %s" % str( e ) )
try:
FormValues_table = Table( "form_values", metadata, autoload=True )
except Exception, e:
log.debug( "Loading 'form_values' table failed: %s" % str( e ) )
# remove the name attribute in the content column JSON dict in the form_values table
# and restore it to a list of values
cmd = "SELECT form_values.id, form_values.content, form_definition.fields" \
" FROM form_values, form_definition" \
" WHERE form_values.form_definition_id=form_definition.id" \
" ORDER BY form_values.id ASC"
result = db_session.execute( cmd )
for row in result:
form_values_id = int( row[0] )
if not str( row[1] ).strip():
continue
values_dict = from_json_string( str
Re: [galaxy-dev] database migration issue 64 ->65 - postgresql
Hell all,
Michael discovered a bug in the
lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py
migration script. To get the fix, you can pull from our development repo at
https://bitbucket.org/galaxy/galaxy-central/wiki/Home. As another option, I've
attached a corrected script to this message. Ypou'll need to place it in
~/lib/galaxy/model/migrate/versions, replacing the script with the bug in it.
Let us know if you have any problems with this.
Thanks!
Greg Von Kuster
0065_add_name_to_form_fields_and_values.pyc
Description: Binary data
On Jun 28, 2011, at 6:58 AM, michael burrell (TSL) wrote:
> Good Afternoon,
>
> Firstly am a massive fan of galaxy, really like the things that it can do,
> but I am a bit stuck updating our instance here and its starting to hold us
> back.
>
> I was hoping someone could offer me some assistance with a database migration
> (I am pulling my hair out). I am using postgresql and galaxy is updated to …
>
> galaxy@jic55119:~/software/galaxy-ceneral$ hg tip
> changeset: 5751:e132a1398caa
> tag: tip
> user:jeremy goecks
> date:Mon Jun 27 17:25:32 2011 -0400
> summary: Additional security for loading visualizations.
>
> And I receive the following error which is blocking me,
>
> galaxy@jic55119:~/software/galaxy-ceneral$ sh manage_db.sh upgrade
> 64 -> 65...
>
> Migration script to add 'name' attribute to the JSON dict which describes
> a form definition field and the form values in the database. In the
> 'form_values'
> table, the 'content' column is now a JSON dict instead of a list.
>
> Traceback (most recent call last):
> File "./scripts/manage_db.py", line 63, in
>main( repository=repo, url=db_url )
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/shell.py",
> line 150, in main
>ret = command_func(**kwargs)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/api.py",
> line 221, in upgrade
>return _migrate(url, repository, version, upgrade=True, err=err, **opts)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/api.py",
> line 349, in _migrate
>schema.runchange(ver, change, changeset.step)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/schema.py",
> line 184, in runchange
>change.run(self.engine, step)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/script/py.py",
> line 101, in run
>func()
> File
> "lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py",
> line 57, in upgrade
>db_session.execute( cmd )
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/scoping.py",
> line 127, in do
>return getattr(self.registry(), name)(*args, **kwargs)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/session.py",
> line 755, in execute
>clause, params or {})
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 824, in execute
>return Connection.executors[c](self, object, multiparams, params)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 874, in _execute_clauseelement
>return self.__execute_context(context)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 896, in __execute_context
>self._cursor_execute(context.cursor, context.statement,
> context.parameters[0], context=context)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 950, in _cursor_execute
>self._handle_dbapi_exception(e, statement, parameters, cursor, context)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 931, in _handle_dbapi_exception
>raise exc.DBAPIError.instance(statement, parameters, e,
> connection_invalidated=is_disconnect)
> sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "f" of relation
> "form_definition" does not exist
> LINE 1: UPDATE form_definition AS f SET f.fields='[{"layout": "", "n...
>^
> 'UPDATE form_definition AS f SET f.fields=\'[{"layout": "", "name":
> "field_0", "default": "yer nom", "required": "required", "label": "Sample
> Name", "visible": true, "helptext": "??", "type": "TextField"}, {"layout":
> "", "name": "field_1", "default": "", "required": "required", "label":
> "run_this?", "visible": true, "helptext": "??", "type": "CheckboxField"}]\'
> WHERE f.id=2' {}
>
> Thank you for all you
Re: [galaxy-dev] database migration issue 64 ->65 - postgresql
Hi Michael,
I'm not exactly sure why this migration script is not working for you( and
unfortunately, er maybe fortunately, I didn't write it ), so we may be forced
to try a few things. How are you bumping into this error? Are you trying to
upgrade a clean Galaxy instance (where you have not yet seen this error), or
are you trying multiple times to upgrade the same instance, and you're
continually seeing this issue?
Can you paste the result of the following sql query against your db?
select * from migrate_version;
I'm not sure the following will help, but look at line # 56 in
~/lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py
- it should look like this:
cmd = "UPDATE form_definition AS f SET f.fields='%s' WHERE f.id=%i" %(
fields_json, form_definition_id )
Change the line to look like this, and try the migration.
cmd = "UPDATE form_definition AS f SET f.fields='%s' WHERE id=%i" %(
fields_json, form_definition_id )
Greg Von Kuster
On Jun 28, 2011, at 6:58 AM, michael burrell (TSL) wrote:
> Good Afternoon,
>
> Firstly am a massive fan of galaxy, really like the things that it can do,
> but I am a bit stuck updating our instance here and its starting to hold us
> back.
>
> I was hoping someone could offer me some assistance with a database migration
> (I am pulling my hair out). I am using postgresql and galaxy is updated to …
>
> galaxy@jic55119:~/software/galaxy-ceneral$ hg tip
> changeset: 5751:e132a1398caa
> tag: tip
> user:jeremy goecks
> date:Mon Jun 27 17:25:32 2011 -0400
> summary: Additional security for loading visualizations.
>
> And I receive the following error which is blocking me,
>
> galaxy@jic55119:~/software/galaxy-ceneral$ sh manage_db.sh upgrade
> 64 -> 65...
>
> Migration script to add 'name' attribute to the JSON dict which describes
> a form definition field and the form values in the database. In the
> 'form_values'
> table, the 'content' column is now a JSON dict instead of a list.
>
> Traceback (most recent call last):
> File "./scripts/manage_db.py", line 63, in
>main( repository=repo, url=db_url )
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/shell.py",
> line 150, in main
>ret = command_func(**kwargs)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/api.py",
> line 221, in upgrade
>return _migrate(url, repository, version, upgrade=True, err=err, **opts)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/api.py",
> line 349, in _migrate
>schema.runchange(ver, change, changeset.step)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/schema.py",
> line 184, in runchange
>change.run(self.engine, step)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/script/py.py",
> line 101, in run
>func()
> File
> "lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py",
> line 57, in upgrade
>db_session.execute( cmd )
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/scoping.py",
> line 127, in do
>return getattr(self.registry(), name)(*args, **kwargs)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/session.py",
> line 755, in execute
>clause, params or {})
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 824, in execute
>return Connection.executors[c](self, object, multiparams, params)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 874, in _execute_clauseelement
>return self.__execute_context(context)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 896, in __execute_context
>self._cursor_execute(context.cursor, context.statement,
> context.parameters[0], context=context)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 950, in _cursor_execute
>self._handle_dbapi_exception(e, statement, parameters, cursor, context)
> File
> "/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
> line 931, in _handle_dbapi_exception
>raise exc.DBAPIError.instance(statement, parameters, e,
> connection_invalidated=is_disconnect)
> sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "f" of relation
> "form_definition" does not exist
> LINE 1: UPDATE form_definition AS f SET f.fields='[{"layout": "", "n...
>^
> 'UPDATE form_d
[galaxy-dev] database migration issue 64 ->65 - postgresql
Good Afternoon,
Firstly am a massive fan of galaxy, really like the things that it can do, but
I am a bit stuck updating our instance here and its starting to hold us back.
I was hoping someone could offer me some assistance with a database migration
(I am pulling my hair out). I am using postgresql and galaxy is updated to …
galaxy@jic55119:~/software/galaxy-ceneral$ hg tip
changeset: 5751:e132a1398caa
tag: tip
user:jeremy goecks
date:Mon Jun 27 17:25:32 2011 -0400
summary: Additional security for loading visualizations.
And I receive the following error which is blocking me,
galaxy@jic55119:~/software/galaxy-ceneral$ sh manage_db.sh upgrade
64 -> 65...
Migration script to add 'name' attribute to the JSON dict which describes
a form definition field and the form values in the database. In the
'form_values'
table, the 'content' column is now a JSON dict instead of a list.
Traceback (most recent call last):
File "./scripts/manage_db.py", line 63, in
main( repository=repo, url=db_url )
File
"/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/shell.py",
line 150, in main
ret = command_func(**kwargs)
File
"/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/api.py",
line 221, in upgrade
return _migrate(url, repository, version, upgrade=True, err=err, **opts)
File
"/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/api.py",
line 349, in _migrate
schema.runchange(ver, change, changeset.step)
File
"/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/schema.py",
line 184, in runchange
change.run(self.engine, step)
File
"/home/galaxy/software/galaxy-ceneral/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/script/py.py",
line 101, in run
func()
File
"lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py",
line 57, in upgrade
db_session.execute( cmd )
File
"/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/scoping.py",
line 127, in do
return getattr(self.registry(), name)(*args, **kwargs)
File
"/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/orm/session.py",
line 755, in execute
clause, params or {})
File
"/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
line 824, in execute
return Connection.executors[c](self, object, multiparams, params)
File
"/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
line 874, in _execute_clauseelement
return self.__execute_context(context)
File
"/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
line 896, in __execute_context
self._cursor_execute(context.cursor, context.statement,
context.parameters[0], context=context)
File
"/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
line 950, in _cursor_execute
self._handle_dbapi_exception(e, statement, parameters, cursor, context)
File
"/home/galaxy/software/galaxy-ceneral/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.5.egg/sqlalchemy/engine/base.py",
line 931, in _handle_dbapi_exception
raise exc.DBAPIError.instance(statement, parameters, e,
connection_invalidated=is_disconnect)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "f" of relation
"form_definition" does not exist
LINE 1: UPDATE form_definition AS f SET f.fields='[{"layout": "", "n...
^
'UPDATE form_definition AS f SET f.fields=\'[{"layout": "", "name": "field_0",
"default": "yer nom", "required": "required", "label": "Sample Name",
"visible": true, "helptext": "??", "type": "TextField"}, {"layout": "", "name":
"field_1", "default": "", "required": "required", "label": "run_this?",
"visible": true, "helptext": "??", "type": "CheckboxField"}]\' WHERE f.id=2' {}
Thank you for all your assistance.
Michael Burrell.
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
