[galaxy-dev] error reading tool from path (epigraph.xml, epigraph_test.xml, access_libraries.xml)
Dear all, I installed a local galaxy using the latest codebase yesterday. During the startup, the loading of 3 tool wrappers are failed: epigraph.xml, epigraph_test.xml, access_libraries.xml. Similar errors were reported in http://www.mail-archive.com/galaxy-dev@lists.bx.psu.edu/msg01839.html Have other people experienced the same problem? Any suggested solutions? Thanks, Leon paster.log below === ... galaxy.tools DEBUG 2011-07-22 16:02:47,084 Loaded tool: hbvar 1.0.0 galaxy.tools DEBUG 2011-07-22 16:02:47,088 Loaded tool: fix_errors 1.0.0 galaxy.tools DEBUG 2011-07-22 16:02:47,089 Loading section: Send Data galaxy.tools ERROR 2011-07-22 16:02:47,089 error reading tool from path: data_destination/epigraph.xml Traceback (most recent call last): File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 77, in load_tool tool = self.load_tool( os.path.join( self.tool_root_dir, path ) ) File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 169, in load_tool return ToolClass( config_file, root, self.app ) File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 340, in __init__ self.parse( root ) File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 385, in parse self.interpreter = command.get(interpreter, None) AttributeError: 'NoneType' object has no attribute 'get' galaxy.tools ERROR 2011-07-22 16:02:47,090 error reading tool from path: data_destination/epigraph_test.xml Traceback (most recent call last): File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 77, in load_tool tool = self.load_tool( os.path.join( self.tool_root_dir, path ) ) File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 169, in load_tool return ToolClass( config_file, root, self.app ) File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 340, in __init__ self.parse( root ) File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 385, in parse self.interpreter = command.get(interpreter, None) AttributeError: 'NoneType' object has no attribute 'get' ... galaxy.tools DEBUG 2011-07-22 16:02:50,556 Loading section: SNP/WGA: Data; Filters galaxy.tools DEBUG 2011-07-22 16:02:50,587 Loaded tool: upload1 1.1.3 galaxy.tools ERROR 2011-07-22 16:02:50,587 error reading tool from path: data_source/access_libraries.xml Traceback (most recent call last): File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 77, in load_tool tool = self.load_tool( os.path.join( self.tool_root_dir, path ) ) File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 169, in load_tool return ToolClass( config_file, root, self.app ) File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 340, in __init__ self.parse( root ) File /opt/galaxy/prog/galaxy-2011-7-21/lib/galaxy/tools/__init__.py, line 385, in parse self.interpreter = command.get(interpreter, None) AttributeError: 'NoneType' object has no attribute 'get' ... -- Hailiang (Leon) Mei Netherlands Bioinformatics Center (http://www.nbic.nl/) Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] DB schema upgrade problem (79 - 85)
Hi Nate, Here's the output of the migration: == galaxy@galaxy2:~/prog/galaxy-2011-12-5$ sh manage_db.sh upgrade 79 - 80... Migration script to create tables for disk quotas. done 80 - 81... Migration script to add a 'tool_version' column to the hda/ldda tables. done 81 - 82... Migration script to add the tool_shed_repository table. done 82 - 83... Migration script to add 'prepare_input_files_cmd' column to the task table and to rename a column. done 83 - 84... Migration script to add 'ldda_id' column to the implicitly_converted_dataset_association table. done 84 - 85... Migration script to add 'info' column to the task table. done == Thanks, Leon On Mon, Dec 5, 2011 at 9:11 PM, Nate Coraor n...@bx.psu.edu wrote: On Dec 5, 2011, at 3:02 PM, Leon Mei wrote: Dear all, Today I upgraded our NBIC server from an older release (fetched on July 5th, 2011) to the latest version in galaxy-dist. After execute hg update and sh manage_db upgrade and merge some local configurations, I successfully restarted the server. However once I access the server from a web browser, Galaxy crashed. Log file complains that some tables don't exist: == ProgrammingError: (ProgrammingError) (1146, Table 'galaxy.user_quota_association' doesn't exist) u'SELECT user_quota_association.id AS user_quota_association_id, user_quota_association.user_id AS user_quota_association_user_id, user_quota_association.quota_id AS user_quota_association_quota_id, user_quota_association.create_time AS user_quota_association_create_time, user_quota_association.update_time AS user_quota_association_update_time \nFROM user_quota_association \nWHERE %s = user_quota_association.user_id' [1L] == A check in mysql confirms that only the following 3 tables were added during the migration from schema 79 to 85: * default_quota_association * quota * tool_shed_repository The following 9 tables are missing: cloud_*, user_quota_association, group_quota_association Hi Leon, Were there any errors output when you ran mange_db.sh upgrade? --nate Has anyone experienced a similar problem? Any hint on how to solve this is highly appreciate! Thanks, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - https://wiki.nbic.nl/index.php/Next_Generation_Sequencing Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - https://wiki.nbic.nl/index.php/Next_Generation_Sequencing Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] error on viewing workflow details via API
Hi,
I received an error when I was trying to view the details of existing
workflows through API.
I can see 4 workflows owned by myself successfully. But viewing the details
of any workflow got me a 500 error.
./display.py my_key http://galaxy.nbic.nl/galaxy/api/workflows
Collection Members
--
#1: /galaxy/api/workflows/e037fdb493429c2a
name: test
id: e037fdb493429c2a
#2: /galaxy/api/workflows/13120e62d0fbb985
name: imported: Partial Workflow for Student Project (1)
id: 13120e62d0fbb985
#3: /galaxy/api/workflows/c9468fdb6dc5c5f1
name: GAPSS SNP calling
id: c9468fdb6dc5c5f1
#4: /galaxy/api/workflows/3f5830403180d620
name: GAPPS CAGE
id: 3f5830403180d620
4 element(s) in collection
./display.py my_key
http://galaxy.nbic.nl/galaxy/api/workflows/e037fdb493429c2a
HTTP Error 500: Internal Server Error
500 Internal Server Error
The server has either erred or is incapable of performing
the requested operation.
Here is the error message in paster.log:
/*
galaxy.web.framework ERROR 2012-01-06 17:46:53,349 Uncaught exception in
exposed API method:
Traceback (most recent call last):
File
/opt/galaxy/prog/galaxy-2011-12-5/lib/galaxy/web/framework/__init__.py,
line 147, in decorator
return simplejson.dumps( func( self, trans, *args, **kwargs ) )
File /opt/galaxy/prog/galaxy-2011-12-5/lib/galaxy/web/api/workflows.py,
line 68, in show
inputs[step.id] = {'label':step.tool_inputs['name'], 'value':}
TypeError: 'NoneType' object is unsubscriptable
my_IP - - [06/Jan/2012:17:46:53 +0200] GET
/galaxy/api/workflows/e037fdb493429c2a?key=my_key HTTP/1.1 500 - -
Python-urllib/2.6
*/
Any suggestion on how to solve this problem?
Thanks,
Leon
--
Hailiang (Leon) Mei
Netherlands Bioinformatics Center
BioAssist NGS Taskforce
- https://wiki.nbic.nl/index.php/Next_Generation_Sequencing
Skype: leon_meiMobile: +31 6 41709231
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] error on viewing workflow details via API
Hi Dannon,
Thanks for the hint. I got it working by removing input dataset step and
adding it back in the workflow canvas. The previous workflow in question
was purely extracted from history.
So now I can check the details via API:
./display.py my_key
http://galaxy.nbic.nl/galaxy/api/workflows/e037fdb493429c2a
Member Information
--
url: /galaxy/api/workflows/e037fdb493429c2a
inputs: {'8648': {'value': '', 'label': 'Input Dataset'}}
id: e037fdb493429c2a
name: test
I still don't understand why it is not working with an extracted workflow.
But I can live with this work around.
Cheers,
Leon
On Mon, Jan 9, 2012 at 3:18 PM, Dannon Baker dannonba...@me.com wrote:
Leon,
I'm not seeing this behavior in any of my instances. I assume they work
correctly through the standard galaxy interface, but is there anything
special about the workflows in question? Is there perhaps a tool with no
inputs in use?
Thanks!
-Dannon
On Jan 6, 2012, at 11:59 AM, Leon Mei wrote:
Hi,
I received an error when I was trying to view the details of existing
workflows through API.
I can see 4 workflows owned by myself successfully. But viewing the
details of any workflow got me a 500 error.
./display.py my_key http://galaxy.nbic.nl/galaxy/api/workflows
Collection Members
--
#1: /galaxy/api/workflows/e037fdb493429c2a
name: test
id: e037fdb493429c2a
#2: /galaxy/api/workflows/13120e62d0fbb985
name: imported: Partial Workflow for Student Project (1)
id: 13120e62d0fbb985
#3: /galaxy/api/workflows/c9468fdb6dc5c5f1
name: GAPSS SNP calling
id: c9468fdb6dc5c5f1
#4: /galaxy/api/workflows/3f5830403180d620
name: GAPPS CAGE
id: 3f5830403180d620
4 element(s) in collection
./display.py my_key
http://galaxy.nbic.nl/galaxy/api/workflows/e037fdb493429c2a
HTTP Error 500: Internal Server Error
500 Internal Server Error
The server has either erred or is incapable of performing
the requested operation.
Here is the error message in paster.log:
/*
galaxy.web.framework ERROR 2012-01-06 17:46:53,349 Uncaught exception
in exposed API method:
Traceback (most recent call last):
File
/opt/galaxy/prog/galaxy-2011-12-5/lib/galaxy/web/framework/__init__.py,
line 147, in decorator
return simplejson.dumps( func( self, trans, *args, **kwargs ) )
File
/opt/galaxy/prog/galaxy-2011-12-5/lib/galaxy/web/api/workflows.py, line
68, in show
inputs[step.id] = {'label':step.tool_inputs['name'], 'value':}
TypeError: 'NoneType' object is unsubscriptable
my_IP - - [06/Jan/2012:17:46:53 +0200] GET
/galaxy/api/workflows/e037fdb493429c2a?key=my_key HTTP/1.1 500 - -
Python-urllib/2.6
*/
Any suggestion on how to solve this problem?
Thanks,
Leon
--
Hailiang (Leon) Mei
Netherlands Bioinformatics Center
BioAssist NGS Taskforce
- https://wiki.nbic.nl/index.php/Next_Generation_Sequencing
Skype: leon_meiMobile: +31 6 41709231
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
--
Hailiang (Leon) Mei
Netherlands Bioinformatics Center
BioAssist NGS Taskforce
- https://wiki.nbic.nl/index.php/Next_Generation_Sequencing
Skype: leon_meiMobile: +31 6 41709231
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] how to set FTP quota?
Dear Galaxy team, We just installed ProFTPd (the same as Galaxy Main) on our Galaxy server. Now we would like to implement a FTP quota. I am wondering how Galaxy main handles this. Could you give us some hints? And does the disk quota counting script include the FTP directory? Thanks very much, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - https://wiki.nbic.nl/index.php/Next_Generation_Sequencing Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] delete data library via API
Hi guys, Is that possible to delete datasets in a shared data library via API? Of course when the modify permission is granted. We see delete.py in the API script folder but can't figure out how to get it work. Thanks! Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nlhttps://wiki.nbic.nl/index.php/Next_Generation_Sequencing Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] delete data library via API
Dear all, Haven't heard anything back yet. So give it another try. Anyone knows if deleting dataset is supported by API? Or it will be a to-be-added feature? Thanks for any hint! Cheers, Leon On Thu, Apr 12, 2012 at 10:32 PM, Leon Mei hailiang@nbic.nl wrote: Hi guys, Is that possible to delete datasets in a shared data library via API? Of course when the modify permission is granted. We see delete.py in the API script folder but can't figure out how to get it work. Thanks! Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_mei Mobile: +31 6 41709231 -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_mei Mobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Migrate Galaxy database from MySQL to Postgres
Dear list members, Did anyone migrate the Galaxy database from MySQL to Postgres lately? Any verified script available? Simple solution by adding a postgres flag in mysqldump (http://dev.list.galaxyproject.org/MySQL-2-Postgres-td4139570.html) didn't work for me. It throws lots of syntax errors on int(), double quote, etc. My mysql version: 14.14, distribution 5.5.25a My postgres version: 9.1 I am now on a quest following the suggested list at: http://wiki.postgresql.org/wiki/Converting_from_other_Databases_to_PostgreSQL#Scripts.2C_programs I appreciate if someone can share his or her experience on this. Thanks a lot, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Migrate Galaxy database from MySQL to Postgres
Dear all, After playing with several different tools, I got the conversion working by using this python program: http://pypi.python.org/pypi/py-mysql2pgsql Just follow the program manual. Both the installation and the execution are quite straight forward. Hope this may help someone in future. Cheers, Leon On Tue, Sep 4, 2012 at 10:54 PM, Leon Mei hailiang@nbic.nl wrote: Dear list members, Did anyone migrate the Galaxy database from MySQL to Postgres lately? Any verified script available? Simple solution by adding a postgres flag in mysqldump (http://dev.list.galaxyproject.org/MySQL-2-Postgres-td4139570.html) didn't work for me. It throws lots of syntax errors on int(), double quote, etc. My mysql version: 14.14, distribution 5.5.25a My postgres version: 9.1 I am now on a quest following the suggested list at: http://wiki.postgresql.org/wiki/Converting_from_other_Databases_to_PostgreSQL#Scripts.2C_programs I appreciate if someone can share his or her experience on this. Thanks a lot, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_meiMobile: +31 6 41709231 -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] missing new job lock function
Dear list, I recently updated our local Galaxy to galaxy-dist (version 9320:47ddf167c9f1, May 1st 2013). I noticed that the lock preventing new jobs dispatching in Manage jobs of the admin panel is missing. Is this a new feature? Or am I having problem with my configuration? If this is indeed a Galaxy update, is there any other way to lock the job queue in Galaxy? Thanks, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nlhttps://wiki.nbic.nl/index.php/Next_Generation_Sequencing Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Startup error, Galaxy DB got corrupted?
:17.608725 | 2013-06-12 06:55:19.396557 | error | f | f | t| | | || | (1 row) galaxy=# UPDATE dataset SET update_time='2013-06-17 12:42:38.145908', state='error' WHERE dataset.id = 68792; ERROR: could not read block 326 in file base/257536/259291: read only 0 of 8192 bytes galaxy=# UPDATE dataset SET state='error' WHERE dataset.id = 68792; UPDATE 1 == When I tried to restore the database from the dumped copy I made yesterday, I got the following error: == ERROR: insert or update on table galaxy_session_to_history violates foreign key constraint galaxy_session_to_history_ session_id_fkey DETAIL: Key (session_id)=(956338) is not present in table galaxy_session. ERROR: insert or update on table job violates foreign key constraint job_session_id_fkey DETAIL: Key (session_id)=(852060) is not present in table galaxy_session. ERROR: insert or update on table workflow_step_connection violates foreign key constraint workflow_step_connection_input_step_id_fkey DETAIL: Key (input_step_id)=(18422) is not present in table workflow_step. ERROR: insert or update on table workflow_step_connection violates foreign key constraint workflow_step_connection_output_step_id_fkey DETAIL: Key (output_step_id)=(18422) is not present in table workflow_step. == It seems our postgres DB got corrupted? Any suggestion on a fix? Thanks a lot! Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nlhttps://wiki.nbic.nl/index.php/Next_Generation_Sequencing Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Startup error, Galaxy DB got corrupted?
Dear Dannon, Thanks for your suggestion! We did give it a try with the backup copy of 10 days back and that worked! We keep the postgres log and the failed postgres backup (using pg_dump with both custom and plain text format) for now and will look into what exactly happened during those 10 days. After this update, it seems we also have solved the problem of missing user email addresses in the add quota page. Need a bit more test further to see it all goes well. Cheers, Leon On Tue, Jun 18, 2013 at 3:24 AM, Dannon Baker dannon.ba...@gmail.comwrote: The error ERROR: could not read block 326 in file base/257536/259291: read only 0 of 8192 bytes line makes me wonder if your database may have been corrupt prior to the upgrade/dump. Some brief googling suggests that this could be hardware failure induced among other things. Can you check the postgres logs for any likely explanations? How exactly did you perform your database backup and attempted recreation? -Dannon On Mon, Jun 17, 2013 at 5:12 PM, Leon Mei hailiang@nbic.nl wrote: Dear list, Yesterday, I noticed that the user quota setting on our Galaxy had a problem - the list of user email address is missing from the manage quota page. So I thought to give a Galaxy codebase update a try after backing up the Galaxy dir and the database. Running hg head gives me: == postgres@ubuntu:/mnt/galaxyTools/galaxy-2013-05-13$ hg head changeset: 10003:b4a373d86c51 tag: tip parent: 10001:471484ff8be6 user:greg date:Wed Jun 12 11:48:09 2013 -0400 summary: Add targets to Repository Actions menu items. == I updated the DB schema to support the code update. But the restart of Galaxy failed with an error in paster.log: == Traceback (most recent call last): File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/webapps/galaxy/buildapp.py, line 35, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/app.py, line 164, in __init__ self.job_manager = manager.JobManager( self ) File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/manager.py, line 36, in __init__ self.job_handler.start() File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/handler.py, line 34, in start self.job_queue.start() File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/handler.py, line 77, in start self.__check_jobs_at_startup() File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/handler.py, line 125, in __check_jobs_at_startup self.dispatcher.recover( job, job_wrapper ) File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/handler.py, line 620, in recover self.job_runners[runner_name].recover( job, job_wrapper ) File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/runners/local.py, line 128, in recover job_wrapper.change_state( model.Job.states.ERROR, info = This job was killed when Galaxy was restarted. Please retry the job. ) File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/jobs/__init__.py, line 824, in change_state dataset.state = state File /mnt/galaxyTools/galaxy-2013-05-13/lib/galaxy/model/__init__.py, line 1163, in set_dataset_state object_session( self ).flush() #flush here, because hda.flush() won't flush the Dataset object File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py, line 1718, in flush self._flush(objects) File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py, line 1789, in _flush flush_context.execute() File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py, line 331, in execute rec.execute(self) File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py, line 475, in execute uow File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py, line 59, in save_obj mapper, table, update) File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py, line 485, in _emit_update_statements execute(statement, params) File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py, line 1449, in execute params) File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py, line 1584, in _execute_clauseelement compiled_sql, distilled_params File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py, line 1698, in _execute_context context) File /mnt/galaxyTools/galaxy-2013-05-13/eggs/SQLAlchemy-0.7.9-py2.7-linux
[galaxy-dev] support pbkdf2 in proftpd 1.3.5rc3
Dear galaxy developers, We have tried today to upgrade our proftpd configuration to make uploading for our galaxy users possible again, both for users with old as well as new style hashed passwords. We upgraded proftpd on the server to 1.3.5rc3 and have the following SQL part in our configuration file based on the post of http://dev.list.galaxyproject.org/ProFTPD-integration-with-Galaxy-td4660295.html SQLEngine on SQLLogFile /var/log/proftpd-sql.log SQLBackend postgres SQLConnectInfo galaxy@localhost:5840 galaxyftp [ourpassword] SQLAuthTypesSHA1 SHA256 PBKDF2 SQLPasswordPBKDF2SHA256 1000 24 SQLPasswordUserSaltsql:/GetUserSalt SQLAuthenticate users SQLDefaultUID 108 SQLDefaultGID 116 SQLDefaultHomedir /opt/cloudman/pkg/proftpd/var SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 32) ELSE password END) AS password2,'108','116','/mnt/galaxyData/tmp/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U' SQLNamedQuery GetUserSalt SELECT (CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U' We have executed the LookupGalaxyUser and GetUserSalt commands manually, and the results look good. Now, old users can login via ftp, but for a new user, the authentication still fails: 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: cmd_check 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'sha1' 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: 'sha1' SQLAuthType handler reports failure 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'pbkdf2' 2013-07-26 13:15:06,993 mod_sql/4.3[31761]: 'pbkdf2' SQLAuthType handler reports failure What are we missing? Thanks! Rob and Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nlhttps://wiki.nbic.nl/index.php/Next_Generation_Sequencing Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] support pbkdf2 in proftpd 1.3.5rc3
Hi Nate, Thanks for the suggestion! Unfortunately, it still failed :( I got the following error message in proftp log: 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: enteringpostgres cmd_escapestring 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: enteringpostgres cmd_open 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: connection 'default' count is now 2 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: exiting postgres cmd_open 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: enteringpostgres cmd_close 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: connection 'default' count is now 1 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: exiting postgres cmd_close 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: exiting postgres cmd_escapestring 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: cache hit for user ' hailiang.m...@nbic.nl' 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: cmd_check 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: checking password using SQLAuthType 'sha1' 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: 'sha1' SQLAuthType handler reports failure 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: checking password using SQLAuthType 'sha256' 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: 'sha256' SQLAuthType handler reports failure 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: checking password using SQLAuthType 'pbkdf2' 2013-08-09 08:32:41,841 mod_sql/4.3[32384]: 'pbkdf2' SQLAuthType handler reports failure 2013-08-09 08:32:41,841 mod_sql/4.3[32384]: cmd_check 2013-08-09 08:32:41,841 mod_sql/4.3[32384]: cmd_auth The old user account generated before our code update still works. I wonder how it is configured at the Galaxy main server? Thanks, Leon On Thu, Aug 8, 2013 at 8:45 PM, Nate Coraor n...@bx.psu.edu wrote: On Jul 26, 2013, at 3:51 PM, Leon Mei wrote: Dear galaxy developers, We have tried today to upgrade our proftpd configuration to make uploading for our galaxy users possible again, both for users with old as well as new style hashed passwords. We upgraded proftpd on the server to 1.3.5rc3 and have the following SQL part in our configuration file based on the post of http://dev.list.galaxyproject.org/ProFTPD-integration-with-Galaxy-td4660295.html SQLEngine on SQLLogFile /var/log/proftpd-sql.log SQLBackend postgres SQLConnectInfo galaxy@localhost:5840 galaxyftp [ourpassword] SQLAuthTypesSHA1 SHA256 PBKDF2 SQLPasswordPBKDF2 SHA256 1000 24 SQLPasswordUserSalt sql:/GetUserSalt SQLAuthenticate users SQLDefaultUID 108 SQLDefaultGID 116 SQLDefaultHomedir /opt/cloudman/pkg/proftpd/var SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 32) ELSE password END) AS password2,'108','116','/mnt/galaxyData/tmp/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U' SQLNamedQuery GetUserSalt SELECT (CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U' We have executed the LookupGalaxyUser and GetUserSalt commands manually, and the results look good. Now, old users can login via ftp, but for a new user, the authentication still fails: 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: cmd_check 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'sha1' 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: 'sha1' SQLAuthType handler reports failure 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'pbkdf2' 2013-07-26 13:15:06,993 mod_sql/4.3[31761]: 'pbkdf2' SQLAuthType handler reports failure What are we missing? Thanks! Rob and Leon Hallo Leon and Rob, Thanks for working on this, when I'd looked a couple months ago I could not find an entirely-ProFTPD way to do this. I think it may have actually come about because I asked about it on their IRC channel. ;) This may work if you change SQLPasswordPBKDF2: SQLPasswordPBKDF2 SHA256 1 24 It'd be great if ProFTPD also supported pulling those values dynamically from the database, but Galaxy's PBKDF2 code currently has them hardcoded, so they will be static anyway. --nate -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http
Re: [galaxy-dev] support pbkdf2 in proftpd 1.3.5rc3
Hi Nate/James, Thanks! Where should I look into to disable PBKDF2 in the Galaxy code? Cheers, Leon On Fri, Aug 9, 2013 at 4:28 PM, James Taylor ja...@jamestaylor.org wrote: For the moment, the best solution is to disable PBKDF2 in Galaxy if you are also using FTP. That is how Galaxy main is currently configured. -- James Taylor, Assistant Professor, Biology/CS, Emory University On Fri, Aug 9, 2013 at 10:01 AM, Nate Coraor n...@bx.psu.edu wrote: On Aug 9, 2013, at 2:38 AM, Leon Mei wrote: Hi Nate, Thanks for the suggestion! Unfortunately, it still failed :( I got the following error message in proftp log: 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: enteringpostgres cmd_escapestring 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: enteringpostgres cmd_open 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: connection 'default' count is now 2 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: exiting postgres cmd_open 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: enteringpostgres cmd_close 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: connection 'default' count is now 1 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: exiting postgres cmd_close 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: exiting postgres cmd_escapestring 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: cache hit for user ' hailiang.m...@nbic.nl' 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: cmd_check 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: checking password using SQLAuthType 'sha1' 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: 'sha1' SQLAuthType handler reports failure 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: checking password using SQLAuthType 'sha256' 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: 'sha256' SQLAuthType handler reports failure 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: checking password using SQLAuthType 'pbkdf2' 2013-08-09 08:32:41,841 mod_sql/4.3[32384]: 'pbkdf2' SQLAuthType handler reports failure 2013-08-09 08:32:41,841 mod_sql/4.3[32384]: cmd_check 2013-08-09 08:32:41,841 mod_sql/4.3[32384]: cmd_auth The old user account generated before our code update still works. I wonder how it is configured at the Galaxy main server? Thanks, Leon It isn't in use on the Main server, but now that I'm aware that ProFTPD has PBKDF2 support, I will put this on my to-do list for next week to test. --nate On Thu, Aug 8, 2013 at 8:45 PM, Nate Coraor n...@bx.psu.edu wrote: On Jul 26, 2013, at 3:51 PM, Leon Mei wrote: Dear galaxy developers, We have tried today to upgrade our proftpd configuration to make uploading for our galaxy users possible again, both for users with old as well as new style hashed passwords. We upgraded proftpd on the server to 1.3.5rc3 and have the following SQL part in our configuration file based on the post of http://dev.list.galaxyproject.org/ProFTPD-integration-with-Galaxy-td4660295.html SQLEngine on SQLLogFile /var/log/proftpd-sql.log SQLBackend postgres SQLConnectInfo galaxy@localhost:5840 galaxyftp [ourpassword] SQLAuthTypesSHA1 SHA256 PBKDF2 SQLPasswordPBKDF2 SHA256 1000 24 SQLPasswordUserSalt sql:/GetUserSalt SQLAuthenticate users SQLDefaultUID 108 SQLDefaultGID 116 SQLDefaultHomedir /opt/cloudman/pkg/proftpd/var SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 32) ELSE password END) AS password2,'108','116','/mnt/galaxyData/tmp/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U' SQLNamedQuery GetUserSalt SELECT (CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U' We have executed the LookupGalaxyUser and GetUserSalt commands manually, and the results look good. Now, old users can login via ftp, but for a new user, the authentication still fails: 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: cmd_check 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'sha1' 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: 'sha1' SQLAuthType handler reports failure 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'pbkdf2' 2013-07-26 13:15:06,993 mod_sql/4.3[31761]: 'pbkdf2' SQLAuthType handler reports failure What are we missing? Thanks! Rob and Leon Hallo Leon and Rob, Thanks for working on this, when I'd looked a couple months ago I could not find an entirely-ProFTPD way to do this. I think it may have actually come about because I asked about it on their IRC channel. ;) This may work if you change SQLPasswordPBKDF2: SQLPasswordPBKDF2 SHA256 1 24 It'd be great if ProFTPD also
Re: [galaxy-dev] Galaxy Release Cycle Length
Hi Dave, I am one of the requesters on a less frequent release. Certainly this does not need to replace the current release scheme in the development branch: Galaxy system users who prefer keeping their code up to date can still go there. It is just nice to have a separate branch with a less frequent stable release cycle with the support of fixing major bugs as Hans-Rudolf pointed out. This fixing support is only needed on the latest stable release. Regarding the frequency of such a release, I would say it is a bit dependent on the speed of introducing new major features in Galaxy. I wouldn't mind to have a 4 or 6 month release cycle in the stable branch if they synchronize better with the major milestones in Galaxy. Thanks, Leon Message: 3 Date: Wed, 21 Aug 2013 08:57:22 +0200 From: Hans-Rudolf Hotz h...@fmi.ch To: Dave Clements cleme...@galaxyproject.org Cc: Galaxy Dev List galaxy-...@bx.psu.edu Subject: Re: [galaxy-dev] Galaxy Release Cycle Length Message-ID: 521464d2.5010...@fmi.ch Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi Dave two months is a good time between releases. Much more important than the release cycle length is fixing identified bugs on the release branch as well. Regards, Hans-Rudolf On 08/20/2013 08:36 PM, Dave Clements wrote: Hello all, At one of the GCC2013 Birds of a Feather sessions http://wiki.galaxyproject.org/Events/GCC2013/BoF/PublicGalaxyServers the group was very clear that they would like to see less frequent releases of Galaxy. We're currently aiming to do a release every 2 months and have been pretty successful at making that target. In the past, we have tried doing releases more often and less often. Is there a sweet spot for the time between releases? Please reply to the group. We are interested in a discussion. Thanks, Dave C -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nlhttps://wiki.nbic.nl/index.php/Next_Generation_Sequencing Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Problem with installing MAF cached dataset
Dear colleagues, I am having some trouble installing MAF datasets at our server. When I run Extract MAF blocks given a set of genomic intervals, I got this error message: Fatal Error: The MAF source specified (10_WAY_MULTIZ_hg19) appears to be invalid. Below is what I did by following the wiki page: http://wiki.galaxyproject.org/Admin/Datatypes/Add%20MAFs 1, download all mafs for hg19 from http://hgdownload.cse.ucsc.edu/goldenPath/hg19/multiz46way/maf/ 2, index them using maf_build_index.py --species=hg19,panTro2, gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2 $file. Our user requested primates only. 3, Here is my maf_index.loc file: == 10-way multiZ (hg19)10_WAY_MULTIZ_hg19 hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2 hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2 /mnt/galaxyIndices/alignments/maf/chr1.maf,/mnt/galaxyIndices/alignments/m af/chr10.maf,/mnt/galaxyIndices/alignments/maf/chr11.maf,/mnt/galaxyIndices/alignments/maf/chr11_gl000202_random.maf,/mnt/galaxyIndices/ali gnments/maf/chr12.maf,/mnt/galaxyIndices/alignments/maf/chr13.maf,/mnt/galaxyIndices/alignments/maf/chr14.maf,/mnt/galaxyIndices/alignments /maf/chr15.maf,/mnt/galaxyIndices/alignments/maf/chr16.maf,/mnt/galaxyIndices/alignments/maf/chr17.maf,/mnt/galaxyIndices/alignments/maf/ch r17_ctg5_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000203_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000204_random.maf,/ mnt/galaxyIndices/alignments/maf/chr17_gl000205_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000206_random.maf,/mnt/galaxyIndices/a lignments/maf/chr18.maf,/mnt/galaxyIndices/alignments/maf/chr18_gl000207_random.maf,/mnt/galaxyIndices/alignments/maf/chr19.maf,/mnt/galaxy Indices/alignments/maf/chr19_gl000208_random.maf,/mnt/galaxyIndices/alignments/maf/chr19_gl000209_random.maf,/mnt/galaxyIndices/alignments/ maf/chr1_gl000191_random.maf,/mnt/galaxyIndices/alignments/maf/chr1_gl000192_random.maf,/mnt/galaxyIndices/alignments/maf/chr2.maf,/mnt/gal axyIndices/alignments/maf/chr20.maf,/mnt/galaxyIndices/alignments/maf/chr21.maf,/mnt/galaxyIndices/alignments/maf/chr21_gl000210_random.maf ,/mnt/galaxyIndices/alignments/maf/chr22.maf,/mnt/galaxyIndices/alignments/maf/chr3.maf,/mnt/galaxyIndices/alignments/maf/chr4.maf,/mnt/gal axyIndices/alignments/maf/chr4_ctg9_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr4_gl000193_random.maf,/mnt/galaxyIndices/alignments/maf/c hr4_gl000194_random.maf,/mnt/galaxyIndices/alignments/maf/chr5.maf,/mnt/galaxyIndices/alignments/maf/chr6.maf,/mnt/galaxyIndices/alignments /maf/chr6_apd_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr6_cox_hap2.maf,/mnt/galaxyIndices/alignments/maf/chr6_dbb_hap3.maf,/mnt/galaxyI ndices/alignments/maf/chr6_mann_hap4.maf,/mnt/galaxyIndices/alignments/maf/chr6_mcf_hap5.maf,/mnt/galaxyIndices/alignments/maf/chr6_qbl_hap 6.maf,/mnt/galaxyIndices/alignments/maf/chr6_ssto_hap7.maf,/mnt/galaxyIndices/alignments/maf/chr7.maf,/mnt/galaxyIndices/alignments/maf/chr 7_gl000195_random.maf,/mnt/galaxyIndices/alignments/maf/chr8.maf,/mnt/galaxyIndices/alignments/maf/chr8_gl000196_random.maf,/mnt/galaxyIndi ces/alignments/maf/chr8_gl000197_random.maf == Any hint on which step is wrong or missing is highly appreciate! Thanks, Leon -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Problem with installing MAF cached dataset
-rw-rw-r-- 1 postgres postgres 926K Sep 9 23:18 chr6_dbb_hap3.maf.index -rw-rw-r-- 1 postgres postgres 16G Oct 31 2009 chr6.maf -rw-rw-r-- 1 postgres postgres 249M Sep 9 23:17 chr6.maf.index -rw-rw-r-- 1 postgres postgres 78M Oct 31 2009 chr6_mann_hap4.maf -rw-rw-r-- 1 postgres postgres 848K Sep 9 23:18 chr6_mann_hap4.maf.index -rw-rw-r-- 1 postgres postgres 80M Oct 31 2009 chr6_mcf_hap5.maf -rw-rw-r-- 1 postgres postgres 854K Sep 9 23:18 chr6_mcf_hap5.maf.index -rw-rw-r-- 1 postgres postgres 87M Oct 31 2009 chr6_qbl_hap6.maf -rw-rw-r-- 1 postgres postgres 949K Sep 9 23:18 chr6_qbl_hap6.maf.index -rw-rw-r-- 1 postgres postgres 80M Oct 31 2009 chr6_ssto_hap7.maf -rw-rw-r-- 1 postgres postgres 871K Sep 9 23:19 chr6_ssto_hap7.maf.index -rw-rw-r-- 1 postgres postgres 2.4M Oct 31 2009 chr7_gl000195_random.maf -rw-rw-r-- 1 postgres postgres 56K Sep 9 23:44 chr7_gl000195_random.maf.index -rw-rw-r-- 1 postgres postgres 14G Oct 31 2009 chr7.maf -rw-rw-r-- 1 postgres postgres 214M Sep 9 23:44 chr7.maf.index -rw-rw-r-- 1 postgres postgres 551K Oct 31 2009 chr8_gl000196_random.maf -rw-rw-r-- 1 postgres postgres 50K Sep 10 00:08 chr8_gl000196_random.maf.index -rw-rw-r-- 1 postgres postgres 1.4M Oct 31 2009 chr8_gl000197_random.maf -rw-rw-r-- 1 postgres postgres 56K Sep 10 00:08 chr8_gl000197_random.maf.index -rw-rw-r-- 1 postgres postgres 13G Oct 31 2009 chr8.maf -rw-rw-r-- 1 postgres postgres 201M Sep 10 00:08 chr8.maf.index == And thanks for the suggestion of adding an extra dir level, I will do that. At the moment, I only need primates, but I couldn't find a way to get subset of mafs (http://hgdownload.cse.ucsc.edu/goldenPath/hg19/), so I downloaded the 46 way files and index primates first. If you knows a trick to get a selected subset, I would love to hear it. Thanks! Leon On Wed, Sep 11, 2013 at 6:38 PM, Daniel Blankenberg d...@bx.psu.edu wrote: Hi Leon, What is the output of ls -lah /mnt/galaxyIndices/alignments/maf/? Also, you may what to use additional sub-directory structure for your file locations in case you want to add more alignments later, e.g., /mnt/galaxyIndices/hg19/maf/10_way_multiz/ or similiar Also, note that although you are only indexing (for extraction) for the specified listed species, the alignments will contain all of the species present in the original file (you list 46way multiZ as the source). Thanks for using Galaxy, Dan On Sep 10, 2013, at 6:44 PM, Leon Mei wrote: Dear colleagues, I am having some trouble installing MAF datasets at our server. When I run Extract MAF blocks given a set of genomic intervals, I got this error message: Fatal Error: The MAF source specified (10_WAY_MULTIZ_hg19) appears to be invalid. Below is what I did by following the wiki page: http://wiki.galaxyproject.org/Admin/Datatypes/Add%20MAFs 1, download all mafs for hg19 from http://hgdownload.cse.ucsc.edu/goldenPath/hg19/multiz46way/maf/ 2, index them using maf_build_index.py --species=hg19,panTro2, gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2 $file. Our user requested primates only. 3, Here is my maf_index.loc file: == 10-way multiZ (hg19)10_WAY_MULTIZ_hg19 hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2 hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2 /mnt/galaxyIndices/alignments/maf/chr1.maf,/mnt/galaxyIndices/alignments/m af/chr10.maf,/mnt/galaxyIndices/alignments/maf/chr11.maf,/mnt/galaxyIndices/alignments/maf/chr11_gl000202_random.maf,/mnt/galaxyIndices/ali gnments/maf/chr12.maf,/mnt/galaxyIndices/alignments/maf/chr13.maf,/mnt/galaxyIndices/alignments/maf/chr14.maf,/mnt/galaxyIndices/alignments /maf/chr15.maf,/mnt/galaxyIndices/alignments/maf/chr16.maf,/mnt/galaxyIndices/alignments/maf/chr17.maf,/mnt/galaxyIndices/alignments/maf/ch r17_ctg5_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000203_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000204_random.maf,/ mnt/galaxyIndices/alignments/maf/chr17_gl000205_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000206_random.maf,/mnt/galaxyIndices/a lignments/maf/chr18.maf,/mnt/galaxyIndices/alignments/maf/chr18_gl000207_random.maf,/mnt/galaxyIndices/alignments/maf/chr19.maf,/mnt/galaxy Indices/alignments/maf/chr19_gl000208_random.maf,/mnt/galaxyIndices/alignments/maf/chr19_gl000209_random.maf,/mnt/galaxyIndices/alignments/ maf/chr1_gl000191_random.maf,/mnt/galaxyIndices/alignments/maf/chr1_gl000192_random.maf,/mnt/galaxyIndices/alignments/maf/chr2.maf,/mnt/gal axyIndices/alignments/maf/chr20.maf,/mnt/galaxyIndices/alignments/maf/chr21.maf,/mnt/galaxyIndices/alignments/maf/chr21_gl000210_random.maf ,/mnt/galaxyIndices/alignments/maf/chr22.maf,/mnt/galaxyIndices/alignments/maf/chr3.maf,/mnt/galaxyIndices/alignments/maf/chr4.maf,/mnt/gal axyIndices/alignments/maf/chr4_ctg9_hap1.maf,/mnt/galaxyIndices/alignments/maf
Re: [galaxy-dev] Problem with installing MAF cached dataset
Hi Dan, Thanks for the solution! It did help me to identify a missing index file. I think the indexing run of that particular config maf file failed for some reason. Since I don't expect much interesting thing on that, I just remove that for now. == hmei@ubuntu:/mnt/galaxyTools/galaxy-central/scripts/tools/maf$ python check_loc_file.py ../../../tool-data/maf_index.loc /usr/local/lib/python2.7/dist-packages/distribute-0.6.49-py2.7.egg/pkg_resources.py:1025: UserWarning: /home/hmei/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable). warnings.warn(msg, UserWarning) Line 6 is invalid: [Errno 2] No such file or directory: '/mnt/galaxyIndices/alignments/maf/chr11_gl000202_random.maf.index' hmei@ubuntu:/mnt/galaxyTools/galaxy-central/scripts/tools/maf$ == So now I have my MAF functions working. Thanks again! Leon On Wed, Sep 11, 2013 at 9:18 PM, Daniel Blankenberg d...@bx.psu.edu wrote: Hi Leon, Can you see if there is any info reported reported by running the MAF consistency checking script at scripts/tools/maf/check_loc_file.py path/to/maf_index.loc? You'll need to make sure that required dependencies (e.g. galaxy_root/lib and galaxy_root/eggs) are included in your PYTHONPATH. As far as cutting down a MAF to only contain the set of desired species and depending upon your needs, you can use the tools included with Galaxy, or alternatively the maf_thread_for_species.py script in bx-python. Thanks for using Galaxy, Dan -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
