Re: [galaxy-dev] Cluster setup - shared temporary directory
On Tue, Jul 26, 2011 at 8:16 PM, Luobin Yang yangl...@isu.edu wrote: I benchmarked MrBayes 3.1.2 program on my cluster for two cases: 1. use local /tmp for temporary files 2. use the network shared /home/galaxy/galaxy-dist/database/tmp MrBayes is about 10 times slower for case 2 than for case 1. What I did was to set the network shared folder as the default but in the MrBayes wrapper, I changed the environment variable TEMP to be a local folder. Luobin Does that mean Galaxy will configure the TEMP environment variable for tools to point at the universe_wsgi.ini new_file_path setting? In your case, /home/galaxy/galaxy-dist/database/tmp This is the kind of thing I think should be documented somewhere for tool authors. Thanks Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Cluster setup - shared temporary directory
On Wed, Jul 27, 2011 at 9:06 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Jul 26, 2011 at 8:16 PM, Luobin Yang yangl...@isu.edu wrote: I benchmarked MrBayes 3.1.2 program on my cluster for two cases: 1. use local /tmp for temporary files 2. use the network shared /home/galaxy/galaxy-dist/database/tmp MrBayes is about 10 times slower for case 2 than for case 1. What I did was to set the network shared folder as the default but in the MrBayes wrapper, I changed the environment variable TEMP to be a local folder. Luobin Does that mean Galaxy will configure the TEMP environment variable for tools to point at the universe_wsgi.ini new_file_path setting? In your case, /home/galaxy/galaxy-dist/database/tmp This is the kind of thing I think should be documented somewhere for tool authors ... ... and mentioned in the universe_wsgi.ini text for the new_file_path setting. I've just seen Shantanu Pavgi's thread which also covers this issue, and the related TMP and TMPDIR environment variables set up or overridden by SGE. I think we agree that Galaxy needs some documentation and guidance in this area for tool authors. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Invalid job runner: drmaa
Hi Ka Ming, Thanks for your answer, but I already checked this for my galaxy user (and for root) on both the galaxy frontend node and on each node all over the cluster. It's the same path as yours for me. If you have any other idea It would be great ! :) Regards, Remy 2011/7/26 Ka Ming Nip km...@bcgsc.ca Hi, You may want to check whether your env variables are exported properly. For example, $ env DRMAA_LIBRARY_PATH=/opt/sge/lib/lx24-amd64/libdrmaa.so.1.0 Hope that helps! Ka Ming From: galaxy-dev-boun...@lists.bx.psu.edu [ galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of remy d1 [ remy...@gmail.com] Sent: July 26, 2011 7:44 AM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Invalid job runner: drmaa Hi, I try to configure Galaxy with SGE. It works one time yesterday but now I have the following message all the time : galaxy.jobshttp://galaxy.jobs DEBUG 2011-07-26 16:50:54,002 dispatching job 118 to drmaa runner galaxy.jobshttp://galaxy.jobs ERROR 2011-07-26 16:50:54,002 put(): (118) Invalid job runner: drmaa [ An error occurred running this job: Unable to run job due to a misconfiguration of the Galaxy job running system. Please contact a site administrator. ] On the web page. Here is my universe_wsgi.ini file : [server:main] port = 8000 host = X.X.X.X use_threadpool = True [filter:proxy-prefix] prefix = /galaxy [app:main] paste.app_factory = galaxy.web.buildapp:app_factory filter-with = proxy-prefix database_connection = mysql://galaxy:xx@localhost :3306/galaxy?unix_socket=/var/run/mysqld/mysqld.sock file_path = database/files new_file_path = database/tmp job_working_directory = database/users_space/user_galaxy cluster_files_directory = database/users_space/user_galaxy use_nglims = False nglims_config_file = tool-data/nglims.yaml static_enabled = True static_cache_time = 360 static_dir = %(here)s/static/ static_images_dir = %(here)s/static/images static_favicon_dir = %(here)s/static/favicon.ico static_scripts_dir = %(here)s/static/scripts/ static_style_dir = %(here)s/static/june_2007_style/blue log_events = True log_actions = True debug = True use_interactive = True library_import_dir = /home/galaxy/galaxy-dist/tmpdir user_library_import_dir = True allow_library_path_paste = True admin_users = me@... enable_pages = True enable_api = True set_metadata_externally = True retry_metadata_internally = True outputs_to_working_directory = True start_job_runners = drmaa default_cluster_job_runner = drmaa:/// [galaxy:tool_runners] biomart = local:/// encode_db1 = local:/// hbvar = local:/// microbial_import1 = local:/// ucsc_table_direct1 = local:/// ucsc_table_direct_archaea1 = local:/// ucsc_table_direct_test1 = local:/// upload1 = local:/// [galaxy_amqp] I checked many things, but it did not change anything... Any help would gracefull. Regards. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] database problem
Hi ALL, I want to delete some data from history_dataset_association. Will it effect any other tables.and I want to delete records for last 3 days by sqlite query manually.How may tables will be effected by this and provide me tables name .How can i delete the records. Regards shashi shekhar ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] automatic installation of galaxy tools...
Hi all, I am working on a way to add/install tools to galaxy in a more automated way. I need that for the galaxy-taverna interoperability work we are doing (https://trac.nbic.nl/elabfactory/wiki/eGalaxy). The generator produces a galaxy tool (.zip or an .xml + .rb) that I want to install as easily as possible. Since, this is sth that other people might be interested I was thinking of generalising it. The idea is to create a script (background process) where we specify an association between a galaxy tool directory and a galaxy section. Every time something is copied in that directory it will be picked up by the script, which will check if it is a valid tool, and add it to tool_conf.xml in the specified section (+optionally restart the server). I am assuming that people would have already worked on/thought of something similar for their own needs. So, my questions are: a) anyone knows if sth similar already exists and how reusable it is, and b) does it sound like something useful (apparently, to people that manage tools frequently!) Cheers, Kostas ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] automatic installation of galaxy tools...
Hi Kostas, I recently talked with Bodo Vossen from MPI who has created this functionality and made available: https://bitbucket.org/Bodolski/galaxy-central-adding-tools He added this functionality to the Admin page of Galaxy which allows to add tools without restarting the server. Unfortunately I have not yet tested it myself, but it seemed to work 'out-of-the-box'. Cheers, Jelle On Wed, Jul 27, 2011 at 2:41 PM, Kostas Karasavvas kostas.karasav...@nbic.nl wrote: Hi all, I am working on a way to add/install tools to galaxy in a more automated way. I need that for the galaxy-taverna interoperability work we are doing (https://trac.nbic.nl/elabfactory/wiki/eGalaxy). The generator produces a galaxy tool (.zip or an .xml + .rb) that I want to install as easily as possible. Since, this is sth that other people might be interested I was thinking of generalising it. The idea is to create a script (background process) where we specify an association between a galaxy tool directory and a galaxy section. Every time something is copied in that directory it will be picked up by the script, which will check if it is a valid tool, and add it to tool_conf.xml in the specified section (+optionally restart the server). I am assuming that people would have already worked on/thought of something similar for their own needs. So, my questions are: a) anyone knows if sth similar already exists and how reusable it is, and b) does it sound like something useful (apparently, to people that manage tools frequently!) Cheers, Kostas ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] job runner error
Hi all, I have added separate job runner in my galaxy instance. So far it was running successfully from many months. Now its giving error as below and i am unable to trace it. Please suggest. Traceback (most recent call last): File /home/gwadmin/galaxy-central/lib/galaxy/jobs/__init__.py, line 202, in __monitor_step self.dispatcher.put( JobWrapper( job, self ) ) File /home/gwadmin/galaxy-central/lib/galaxy/jobs/__init__.py, line 847, in put self.job_runners[runner_name].put( job_wrapper ) File /home/gwadmin/galaxy-central/lib/galaxy/jobs/runners/gw.py, line 370, in put job_wrapper.change_state( model.Job.states.QUEUED ) File /home/gwadmin/galaxy-central/lib/galaxy/jobs/__init__.py, line 428, in change_state self.sa_session.flush() File /home/gwadmin/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/orm/scoping.py, line 127, in do return getattr(self.registry(), name)(*args, **kwargs) File /home/gwadmin/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/orm/session.py, line 1356, in flush self._flush(objects) File /home/gwadmin/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/orm/session.py, line 1438, in _flush transaction.commit() File /home/gwadmin/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/orm/session.py, line 382, in commit t[1].commit() File /home/gwadmin/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py, line 1036, in commit self._do_commit() File /home/gwadmin/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py, line 1060, in _do_commit self.connection._commit_impl() File /home/gwadmin/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py, line 750, in _commit_impl self._handle_dbapi_exception(e, None, None, None, None) File /home/gwadmin/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py, line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) OperationalError: (OperationalError) database is locked None None Thanks Shashi ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Upload of images - jpg/png - binary file contains inappropriate content
Helo Jelle, I've taken a look at your patches and the code looks good. However, I'm wondering why you find it necessary to upload image files to Galaxy. Do tools exist that take image files as input? If the Galaxy community finds this feature as a good addition to Galaxy, we can probably incorporate it into the distribution. However, we'll need an additional feature before we can do that. We'll need to implement a type of config setting that enables / disables the uploading of image files via the Galaxy api. This can either be an enhancement to the existing display_in_upload config setting, or a separate allow_api_upload config setting (probably the latter for clarity). I haven't looked too far into what it will take to do this, but if your reasons for needing this feature are beneficial to the community, I can spend some time on it - or if you want to take a pass at it, feel free! Thanks very much for your contributions! Greg Von Kuster On Jul 25, 2011, at 11:23 AM, Jelle Scholtalbers wrote: Hi Greg, this was indeed causing the problem. I added some code which now allows to sniff image files. It is practically untested (it does work for me on bmp, jpg, png, tiff) - didn't try with PIL although the code is there. Attached are diffs against changeset 058a5d7a4f84 (bit outdated - can provide newer if desired..). Sniffed formats: http://infohost.nmt.edu/tcc/help/pubs/pil/formats.html or http://docs.python.org/library/imghdr.html depending if PIL is available or not. Cheers, Jelle On Fri, Jul 22, 2011 at 6:46 PM, Greg Von Kuster g...@bx.psu.edu wrote: Based on the error message you gave, I assume the following code in ~/lib/galaxy/datatypes/sniff.py is presenting the problem. if check_binary( filename ): if ext not in unsniffable_binary_formats and not datatypes_registry.get_datatype_by_extension( ext ).sniff( filename ): raise InappropriateDatasetContentError, 'The binary uploaded file contains inappropriate content.' Have you tried adding your 'jpg' extension to the following in ~/lib/galaxy/datatypes/binary.py? # Currently these supported binary data types must be manually set on upload unsniffable_binary_formats = [ 'ab1', 'scf' ] On Jul 22, 2011, at 11:52 AM, Jelle Scholtalbers wrote: Hi Greg, I used that link when trying to create the support. The jpg datatype was already present in the datatypes_conf.xml ( datatype extension=jpg type=galaxy.datatypes.images:Image mimetype=image/jpeg/ ) , so I first made it available at upload - display_in_upload=True. As far as I understood, if galaxy doesn't have to guess what the format is, this would be sufficient? From step2: Galaxy tools are configured to automatically set the data type of an output dataset. However, in some scenarios, Galaxy will attempt to determine the data type of a file using a sniffer However manually setting the format on upload still gave the mentioned error. Therefore I followed the rest of the guide (adding a sniffer) but still seem to fail here.. Cheers, Jelle On Jul 22, 2011 3:26 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Jelle, There are a few things you need to do to add support for a new data type. The steps are described here: http://wiki.g2.bx.psu.edu/Admin/Datatypes/Adding%20Datatypes Greg Von Kuster On Jul 22, 2011, at 8:49 AM, Jelle Scholtalbers wrote: Hi all, I would like to be able to upload images to my Galaxy instance - in particular jpg/png to data libraries. I can't however find out how to overcome the problem of inappropriate content in binary file. How to go about? When I create a sniffer class in the galaxy.datatypes.images.py that always returns true (for testing purposes) I still don't get the jpg uploaded without an error (sniffer is incl. in the datatypes_conf.xml ). class Jpg( Image ): file_ext = jpg def sniff(self, filename): Determine if the file is in jpg format. return True It seems I'm missing something obvious and could use some help. Cheers, Jelle ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu Greg Von Kuster Galaxy Development Team g...@bx.psu.edu datatypes_conf.xml.diffimages.py.diffupload.py.diff Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] job runner error
On Wed, Jul 27, 2011 at 2:14 PM, shashi shekhar meshash...@gmail.com wrote: Hi all, I have added separate job runner in my galaxy instance. So far it was running successfully from many months. Now its giving error as below and i am unable to trace it. Please suggest. Traceback (most recent call last): File /home/gwadmin/galaxy-central/lib/galaxy/jobs/__init__.py, line 202, in __monitor_step self.dispatcher.put( JobWrapper( job, self ) ) ... File /home/gwadmin/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py, line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) OperationalError: (OperationalError) database is locked None None Thanks Shashi You're not using SQLite as the database are you? I've seen that kind of error message from locking problems (e.g. when the SQLite DB was on a CIFS mapped drive), and it wouldn't surprise me that with a separate thread for the job runner you get contention over the SQLite DB. If you are using PostgreSQL (or MySQL) as recommended for a production Galaxy server, then I'm not sure what could be wrong. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Upload of images - jpg/png - binary file contains inappropriate content
On Wed, Jul 27, 2011 at 2:28 PM, Greg Von Kuster g...@bx.psu.edu wrote: Helo Jelle, I've taken a look at your patches and the code looks good. However, I'm wondering why you find it necessary to upload image files to Galaxy. Do tools exist that take image files as input? If the Galaxy community finds this feature as a good addition to Galaxy, we can probably incorporate it into the distribution. I think it is a nice feature - and can come up with some use cases too. How about future tools for image analysis? e.g. microscope photos of cultures to automatically do cell/organism counting, or plant leaves for pathogenicity assays. However, we'll need an additional feature before we can do that. We'll need to implement a type of config setting that enables / disables the uploading of image files via the Galaxy api. ... Why not just allow it? A general configuration allowing/blocking any file type from being uploaded makes more sense to me than just wanting to block images. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] job runner error
On Wed, Jul 27, 2011 at 2:14 PM, shashi shekhar meshash...@gmail.com Hi all, I have added separate job runner in my galaxy instance. So far it was running successfully from many months. Now its giving error as below and i am unable to trace it. Please suggest. Traceback (most recent call last): ... OperationalError: (OperationalError) database is locked None None Thanks Shashi On Wed, Jul 27, 2011 at 7:00 PM, Peter Cock p.j.a.c...@googlemail.com You're not using SQLite as the database are you? I've seen that kind of error message from locking problems (e.g. when the SQLite DB was on a CIFS mapped drive), and it wouldn't surprise me that with a separate thread for the job runner you get contention over the SQLite DB. If you are using PostgreSQL (or MySQL) as recommended for a production Galaxy server, then I'm not sure what could be wrong. On Wed, Jul 27, 2011 at 2:42 PM, shashi shekhar meshash...@gmail.com wrote: I am using sqlite database . I think that to avoid this database problem you will have to move to PostgreSQL (which is what the Galaxy team recommend), or possibly MySQL (not sure what the downsides are). This might be hard if you already have a number of users and datafiles. I've CC'd the list again so hopefully Nate or one of the other Galaxy Devs can confirm this. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Upload of images - jpg/png - binary file contains inappropriate content
Hi Peter, On Jul 27, 2011, at 9:34 AM, Peter Cock wrote: On Wed, Jul 27, 2011 at 2:28 PM, Greg Von Kuster g...@bx.psu.edu wrote: Helo Jelle, I've taken a look at your patches and the code looks good. However, I'm wondering why you find it necessary to upload image files to Galaxy. Do tools exist that take image files as input? If the Galaxy community finds this feature as a good addition to Galaxy, we can probably incorporate it into the distribution. I think it is a nice feature - and can come up with some use cases too. How about future tools for image analysis? e.g. microscope photos of cultures to automatically do cell/organism counting, or plant leaves for pathogenicity assays. Ok, I'm convinced. However, we'll need an additional feature before we can do that. We'll need to implement a type of config setting that enables / disables the uploading of image files via the Galaxy api. ... Why not just allow it? A general configuration allowing/blocking any file type from being uploaded makes more sense to me than just wanting to block images. The enhancement would be to add a config setting to allow / disallow uploading any data type ( not just images ) via the api. The problem with just allowing uploading of images is that we'll end up hosting a bunch of porn on our main public Galaxy instance. This has happened to us in the past... Peter Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?
Hello all, I'm working on a local instance of Galaxy at George Mason University. We'd been looking into integrating Qiime and I've found the toolkit very helpful, thanks! I did have one question, though: would you mind uploading the tool_conf file or chunk of text with the qiime functions in it? This would be helpful to myself, as well as others interested in the toolkit! Thank you, Amanda Zuzolo Bioengineering Major, George Mason University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Upload of images - jpg/png - binary file contains inappropriate content
Hi, On Jul 27, 2011, at 3:34 PM, Peter Cock wrote: On Wed, Jul 27, 2011 at 2:28 PM, Greg Von Kuster g...@bx.psu.edu wrote: Helo Jelle, I've taken a look at your patches and the code looks good. However, I'm wondering why you find it necessary to upload image files to Galaxy. Do tools exist that take image files as input? If the Galaxy community finds this feature as a good addition to Galaxy, we can probably incorporate it into the distribution. I think it is a nice feature - and can come up with some use cases too. +1 How about future tools for image analysis? e.g. microscope photos of cultures to automatically do cell/organism counting, or plant leaves for pathogenicity assays. However, we'll need an additional feature before we can do that. We'll need to implement a type of config setting that enables / disables the uploading of image files via the Galaxy api. ... Why not just allow it? Indeed! A general configuration allowing/blocking any file type from being uploaded makes more sense to me than just wanting to block images. Why block file types? If you really want to then how about automagically *allowing* all file types specified in datatypes_conf.xml? Cheers, Pi Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ - mobile: +31 6 143 66 783 e-mail: pieter.neeri...@gmail.com skype: pieter.online - ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Upload of images - jpg/png - binary file contains inappropriate content
Hi Greg, I do see the concern of uploading data through the api but I don't think filtering on file types is the right solution. It seems to be an issue with allowing everyone to register. Wouldn't it be better to at least validate e-mail addresses of new users on the main instance? This would probably resolve most issues related to unwanted content on the server? Furthermore I agree with Peter, that if you want to block certain file types on your own instance, they should be removed from the datatypes_config.xml and thereby disabling them. Btw I needed the images for the sample tracking part - e.g. uploading gel fotos. Cheers, Jelle On Wed, Jul 27, 2011 at 3:55 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, On Jul 27, 2011, at 9:34 AM, Peter Cock wrote: On Wed, Jul 27, 2011 at 2:28 PM, Greg Von Kuster g...@bx.psu.edu wrote: Helo Jelle, I've taken a look at your patches and the code looks good. However, I'm wondering why you find it necessary to upload image files to Galaxy. Do tools exist that take image files as input? If the Galaxy community finds this feature as a good addition to Galaxy, we can probably incorporate it into the distribution. I think it is a nice feature - and can come up with some use cases too. How about future tools for image analysis? e.g. microscope photos of cultures to automatically do cell/organism counting, or plant leaves for pathogenicity assays. Ok, I'm convinced. However, we'll need an additional feature before we can do that. We'll need to implement a type of config setting that enables / disables the uploading of image files via the Galaxy api. ... Why not just allow it? A general configuration allowing/blocking any file type from being uploaded makes more sense to me than just wanting to block images. The enhancement would be to add a config setting to allow / disallow uploading any data type ( not just images ) via the api. The problem with just allowing uploading of images is that we'll end up hosting a bunch of porn on our main public Galaxy instance. This has happened to us in the past... Peter Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Upload of images - jpg/png - binary file contains inappropriate content
On Wed, Jul 27, 2011 at 3:26 PM, Jelle Scholtalbers j.scholtalb...@gmail.com wrote: Hi Greg, I do see the concern of uploading data through the api but I don't think filtering on file types is the right solution. It seems to be an issue with allowing everyone to register. Wouldn't it be better to at least validate e-mail addresses of new users on the main instance? This would probably resolve most issues related to unwanted content on the server? That might be another good idea. Furthermore I agree with Peter, that if you want to block certain file types on your own instance, they should be removed from the datatypes_config.xml and thereby disabling them. But we don't want to disable the formats - many tools may produce JPEG etc as output. Greg just wants to be able to block uploads by file type. Btw I needed the images for the sample tracking part - e.g. uploading gel fotos. Another good use case - do you analyse them in Galaxy too? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] set_metadata_externally =True
Well I got all the rw permissions changed correctly except for the metadata files written to the temp directory (new_file_path). I still can't seem to track down where those are written. Any help would be greatly appreciated. Best, Ilya From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, July 25, 2011 4:29 PM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] set_matadata_externally =True I am still working on the patch to submit jobs as the actual user under SGE . I have it working for the default case but it fails for the case of set_metadata_externally = TRUE. Can someone point me to where the metadata files are created so that I can change the permissions for those files? Also, I have to addusers to the galaxy group in order for this to work because the galaxy_*.sh scripts are created as executable only by members of the galaxy group. I want to change this so that all users can execute these scripts. Any help on where in the code these scripts are created? Upon completion I will create a galaxy central fork and upload my changes to it. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Upload of images - jpg/png - binary file contains inappropriate content
No, data does not require a login to view if the dataset permissions are set permissively. This allows users to share direct links to datasets (which is meant to be a good feature ;). On Jul 27, 2011, at 11:45 AM, Paul Gordon wrote: On the other hand, isn't user data supposed to require a login to view? i.e., was the inappropriate content being served or just stored on the public server? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tool XML parameter options from locally stored files
Hi.
I'm still searching for a solution to this problem.
Currently we have to implement a tool to read the GTF file selected
via a from_data_table parameter and output a list of feature types.
Secondly we have a second tool that reads this file using a
from_dataset parameter so the user can select a specific datatype.
Not really ideal, I would prefer to do this in one tool if possible.
If I could reference the file chosen in the first select then this
should work.
For instance in the output data tags I can put label=${gtf.value}
and it returns the name of the selected file from the datatable.
If I do the same in inputs section: options from_file=${gtf.value}
galaxy tries to find the file [galaxy_dir]/tool-data/${gtf.value}.
So the solution would ideally look like this:
param name=gtf type=select label=GTF File --gtf
options from_data_table=pycrac_gtf/ !--link to .loc file with
gtf file per species --
/param
param name=feature type=select label=Select feature
options from_file=${gtf.value}
column name=name index=2/
column name=value index=2/
filter type=unique_value name=unique column=1/
/options
/param
Hope that makes sense.
Thanks
Shaun
Quoting SHAUN WEBB swe...@staffmail.ed.ac.uk on Fri, 15 Jul 2011
10:17:44 +0100:
Hi Greg,
thanks for the suggestion. This line doesn't seem to work:
options from_file=some_value
Galaxy looks for a file at galaxy_dir/tool-data/some_value rather
than the value given in the .loc file
I would also like to be able to use the galaxy data tables rather
then referring directly to a .loc file.
Any ideas?
Shaun
Quoting Greg Von Kuster g...@bx.psu.edu on Wed, 13 Jul 2011 10:28:30 -0400:
Hi Shaun,
This should be possible. It's been quite a long time since I
worked on tools, so maybe others can provide better assistance if I
am incorrect here. You'll need to have a conditional something
like the following ( not tested whatsoever ):
conditional name=selected_file_select_list
param name=selected_gtf_file type=select label=Gtf
file by species
options from_file=some_file
column name=name index=2/
column name=some_value index=1/
/options
/param
when value=some_value
param name=values_from_selected_gtf_file
type=select label=Values from selected gtf file /
options from_file=some_value
column name=name index=2/
column name=some_value index=1/
/options
/when
/conditional
On Jul 13, 2011, at 9:45 AM, SHAUN WEBB wrote:
Hi Greg, I'm not sure if this answers my question. Using
options_from_file is fine if I want to refer to the same file each
time.
In this case I want the first select to choose which gtf file to
look at (by species) and the second select to pull options (gtf
annotation types) from this file.
So I would somehow have to reference the file using the first
parameter rather than giving a hard-coded path.
Is that possible?
Shaun
Quoting Greg Von Kuster g...@bx.psu.edu on Wed, 13 Jul 2011
09:23:19 -0400:
Hello Shaun,
Use the 'from_file' option. The referenced file should be in the
~/tool-data directory in the Galaxy root. There are several
example tools in the distribution that use this option. Here is
the example code from the annotation profiler tool:
param name=table_names type=drill_down display=checkbox
hierarchy=recurse multiple=true label=Choose Tables to Use
help=Selecting no tables will result in using all tables.
from_file=annotation_profiler_options.xml/
and here is an example from the microbial import tool:
param name=kingdom type=select label=Select the Desired
Kingdom
options from_file=microbial_data.loc startswith=ORG
column name=name index=3/
column name=value index=3/
filter type=unique_value name=unique column=3/
/options
/param
On Jul 13, 2011, at 6:17 AM, SHAUN WEBB wrote:
Hi,
I am trying to write a tool xml file and want to do the following:
I have a number of GTF files stored locally and referenced in a
data table. I give the user a drop down list of those available
using the following code:
param name=gtf type=select label=GTF File
options from_data_table=gtf_files/
/param
I then want them to choose an annotation type based on those
present in the selected gtf file above. It seems simple enough
to do this if the gtf file is in your history already:
param name=feature type=select multiple=true
label=Extract features
options from_dataset=gtf
column name=name index=2/
column name=value index=2/
filter type=unique_value name=unique column=2/
/options
/param
But is there a way to query locally stored files via
datatables/.loc files to pull out options when the file is
dynamically chosen
[galaxy-dev] problems with two coupled select menus using dynamic_options and refresh_on_change
Hi,
I am trying to create a form where there are two select menus, both using
dynamic_options, where one is the parent menu that is chosen first and based
on the selection(s) there is a refresh and the second menu gets populated
based on the first menu's choice(s).
param name=names type=select multiple=true optional=true
dynamic_options=dyn_options('names') refresh_on_change=true
label=Filter By Names/
param name=values type=select multiple=true optional=true
dynamic_options=dyn_options('values', $names) label=Filter by Values /
There seems to be a problem when the form loads for the first time as $names
is not selected yet. I get a syntax error like Invalid string, when I
remove $names as a parameters from my second dynamic_options function call
everything is fine. Does someone know how to achieve the functionality I'm
aiming for in Galaxy without having to use the deprecated pages tags? Is
there a way in Cheetah to set a default value for $names and if so how would
I do this in the tool xml?
regards,
Leandro
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] manage library permission interface glitch
I should preface this by saying we are running a galaxy-dist that is a couple months old now (we are planning on updating soon), but here is an annoying issue we are seeing managing data library permissions: The Roles not associated lists are filtered so it shows only roles associated with the access library permission -- this is good, except this is filtered for the access library permission too, meaning once you add roles to the access library permission and save you can't add any other roles to this permission since the Roles not associated for access library now only includes roles associated with access library. To add any new roles we need to first remove every role from this permission, making it public, which then causes all roles to get listed in roles not associated, and then we can associate all of the roles we want. The correct behavior would be to filter every roles not associated list for every permission except for access library -- this should always list all roles not associated. I will open a new issue for this on Bitbucket unless a developer can confirm that it has been fixed since my last galaxy-dist update a couple months ago. -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] DRMAA options for SGE
Answering myself here... I don't see the error anymore after adding -w n: [galaxy:tool_runners] ... mytoolname = drmaa://-w n -l mem_free=1G -l mem_token=1G -l h_vmem=1G/ ... Is this what other Galaxy admins do when using drmaa native options for their SGE? Ka Ming From: galaxy-dev-boun...@lists.bx.psu.edu [galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Ka Ming Nip [km...@bcgsc.ca] Sent: July 26, 2011 10:39 AM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] DRMAA options for SGE Hi, I am trying to configure the proper memory resource requests for my Galaxy tool. This is what I have under the tool_runners section of universe_wsgi.ini [galaxy:tool_runners] ... mytoolname = drmaa://-l mem_free=1G -l mem_token=1G -l h_vmem=1G/ ... When I execute my tool on Galaxy, I get the error below in the shell that I ran sh run.sh: galaxy.jobs.runners.drmaa ERROR 2011-07-26 09:54:01,930 Uncaught exception queueing job Traceback (most recent call last): File /home/kmnip/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py, line 112, in run_next self.queue_job( obj ) File /home/kmnip/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py, line 177, in queue_job job_id = self.ds.runJob(jt) File /home/kmnip/galaxy/galaxy-central/eggs/drmaa-0.4b3-py2.4.egg/drmaa/__init__.py, line 331, in runJob File /home/kmnip/galaxy/galaxy-central/eggs/drmaa-0.4b3-py2.4.egg/drmaa/helpers.py, line 213, in c File /home/kmnip/galaxy/galaxy-central/eggs/drmaa-0.4b3-py2.4.egg/drmaa/errors.py, line 90, in error_check DeniedByDrmException: code 17: error: no suitable queues All the flags I used work with qsub commands on the SGE cluster I use. The tool runs when I comment out the line in universe_wsgi.ini. Thanks, Ka Ming ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] manage library permission interface glitch
Glen, the behavior you see is undoubtedly the same in the development repo, so feel free to open a ticket if you want. In any case, this is now on my list, although several tasks down in it. I should be able to look into it some time within the next 2 weeks though. Thanks for pointing it out! Greg On Jul 27, 2011, at 2:08 PM, Glen Beane wrote: I should preface this by saying we are running a galaxy-dist that is a couple months old now (we are planning on updating soon), but here is an annoying issue we are seeing managing data library permissions: The Roles not associated lists are filtered so it shows only roles associated with the access library permission -- this is good, except this is filtered for the access library permission too, meaning once you add roles to the access library permission and save you can't add any other roles to this permission since the Roles not associated for access library now only includes roles associated with access library. To add any new roles we need to first remove every role from this permission, making it public, which then causes all roles to get listed in roles not associated, and then we can associate all of the roles we want. The correct behavior would be to filter every roles not associated list for every permission except for access library -- this should always list all roles not associated. I will open a new issue for this on Bitbucket unless a developer can confirm that it has been fixed since my last galaxy-dist update a couple months ago. -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Upload of images - jpg/png - binary file contains inappropriate content
Jelle, I've added support for uploading image datatypes based on your patches in change set 5833:e7214c69ed7d, but with some changes. There is a new image_util.py file in ~/lib/galaxy/datatypes/util which allows for detecting image data types without having to create a new Image() class. I've also aded a couple of fixes to the code. There is currently no support in the Galaxy api for uploading files outside of a data library, so I did not have to add the new config setting discussed in the previous thread. We'll look into that further when the api is enhanced to enable uploads. Thanks very much for your contribution! Greg Von Kuster On Jul 25, 2011, at 11:23 AM, Jelle Scholtalbers wrote: Hi Greg, this was indeed causing the problem. I added some code which now allows to sniff image files. It is practically untested (it does work for me on bmp, jpg, png, tiff) - didn't try with PIL although the code is there. Attached are diffs against changeset 058a5d7a4f84 (bit outdated - can provide newer if desired..). Sniffed formats: http://infohost.nmt.edu/tcc/help/pubs/pil/formats.html or http://docs.python.org/library/imghdr.html depending if PIL is available or not. Cheers, Jelle On Fri, Jul 22, 2011 at 6:46 PM, Greg Von Kuster g...@bx.psu.edu wrote: Based on the error message you gave, I assume the following code in ~/lib/galaxy/datatypes/sniff.py is presenting the problem. if check_binary( filename ): if ext not in unsniffable_binary_formats and not datatypes_registry.get_datatype_by_extension( ext ).sniff( filename ): raise InappropriateDatasetContentError, 'The binary uploaded file contains inappropriate content.' Have you tried adding your 'jpg' extension to the following in ~/lib/galaxy/datatypes/binary.py? # Currently these supported binary data types must be manually set on upload unsniffable_binary_formats = [ 'ab1', 'scf' ] On Jul 22, 2011, at 11:52 AM, Jelle Scholtalbers wrote: Hi Greg, I used that link when trying to create the support. The jpg datatype was already present in the datatypes_conf.xml ( datatype extension=jpg type=galaxy.datatypes.images:Image mimetype=image/jpeg/ ) , so I first made it available at upload - display_in_upload=True. As far as I understood, if galaxy doesn't have to guess what the format is, this would be sufficient? From step2: Galaxy tools are configured to automatically set the data type of an output dataset. However, in some scenarios, Galaxy will attempt to determine the data type of a file using a sniffer However manually setting the format on upload still gave the mentioned error. Therefore I followed the rest of the guide (adding a sniffer) but still seem to fail here.. Cheers, Jelle On Jul 22, 2011 3:26 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Jelle, There are a few things you need to do to add support for a new data type. The steps are described here: http://wiki.g2.bx.psu.edu/Admin/Datatypes/Adding%20Datatypes Greg Von Kuster On Jul 22, 2011, at 8:49 AM, Jelle Scholtalbers wrote: Hi all, I would like to be able to upload images to my Galaxy instance - in particular jpg/png to data libraries. I can't however find out how to overcome the problem of inappropriate content in binary file. How to go about? When I create a sniffer class in the galaxy.datatypes.images.py that always returns true (for testing purposes) I still don't get the jpg uploaded without an error (sniffer is incl. in the datatypes_conf.xml ). class Jpg( Image ): file_ext = jpg def sniff(self, filename): Determine if the file is in jpg format. return True It seems I'm missing something obvious and could use some help. Cheers, Jelle ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu Greg Von Kuster Galaxy Development Team g...@bx.psu.edu datatypes_conf.xml.diffimages.py.diffupload.py.diff Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Regarding submitting jobs as actual user
I completed the changes to galaxy to submit DRMAA jobs (SGE) as the actual galaxy user. You can see the code at the fork below. Below is a brief description of how to turn it which is also described in the universe_wsgi.ini.sample. https://icho...@bitbucket.org/ichorny/galaxy-central # Uncomment drmaa_external_runjob_script and drmaa_external_killjob_script pameters and have them point to the # absolute path for scripts/drmaa_external_runner.py and scripts/drmaa_external_killer.py. # The scripts directory is located in the top level galaxy directory. The parameters when # uncommented allow for submission to the drmaa queue with the user name of the user submitting # the job and not the galaxy user. In order for this to work the actual user must log into galaxy # and the galaxy authentication must be consistent with the authentication on the server in which the # drmaa queue is running (i.e. the username must have an account on the server and be allowed to # submit jobs to the queue). The galaxy user must also be given sudo permission to execute # scripts/drmaa_external_runner.py and scripts/drmaa_external_killer.py in /etc/sudoers # Example: # galaxy ALL = (root) NOPASSWD: SETENV: /opt/galaxy/scripts/drmaa_external_runner.py # galaxy ALL = (root) NOPASSWD: SETENV: /opt/galaxy/scripts/drmaa_external_killer.py # Also the # Defaultsrequiretty # in /etc/sudoers must be commented out #drmaa_external_runjob_script = None #drmaa_external_killjob_script = None Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?
here's what i have; i started categorizing the tools under labels but hadn't finished. i'm also working on getting these tools working in galaxy (albeit slowly as we won't need them until we get a miseq machine in oct), but maybe we can be in communication with each other and jjohnson. section name=QIIME (testing; under development) id=qiime_tools label text=UNDER DEVELOPMENT id=QIIME_TESTING/ label text=Preprocessing/QC id=qiime_preprocessing/ tool file=qiime/check_id_map.xml / tool file=qiime/split_libraries.xml / tool file=qiime/denoise.xml / tool file=qiime/sort_denoiser_output.xml / tool file=qiime/trim_sff_primers.xml / tool file=qiime/fix_arb_fasta.xml / tool file=qiime/quality_scores_plot.xml / tool file=qiime/truncate_fasta_qual_files.xml / tool file=qiime/process_sff.xml / tool file=qiime/identify_chimeric_seqs.xml / tool file=qiime/parallel_identify_chimeric_seqs.xml / tool file=qiime/split_libraries_illumina.xml / tool file=qiime/make_library_id_lists.xml / tool file=qiime/make_fastq.xml / tool file=qiime/merge_denoiser_output.xml / tool file=qiime/adjust_seq_orientation.xml / tool file=qiime/extract_seqs_by_sample_id.xml / label text=Pick OTUs id=qiime_otus/ tool file=qiime/parallel_pick_otus_blast.xml / tool file=qiime/pick_otus_through_otu_table.xml / tool file=qiime/parallel_pick_otus_uclust_ref.xml / tool file=qiime/pick_rep_set.xml / tool file=qiime/parallel_assign_taxonomy_rdp.xml / tool file=qiime/parallel_assign_taxonomy_blast.xml / tool file=qiime/make_phylogeny.xml / tool file=qiime/make_otu_table.xml / tool file=qiime/pick_otus.xml / tool file=qiime/filter_alignment.xml / label text=Align OTUs id=qiime_align/ tool file=qiime/parallel_align_seqs_pynast.xml / tool file=qiime/blast_wrapper.xml / tool file=qiime/parallel_blast.xml / tool file=qiime/exclude_seqs_by_blast.xml / tool file=qiime/align_seqs.xml / label text=OTU Analysis id=qiime_otu_analysis/ tool file=qiime/assign_taxonomy.xml / tool file=qiime/per_library_stats.xml / tool file=qiime/make_otu_heatmap_html.xml / tool file=qiime/make_otu_network.xml / tool file=qiime/summarize_taxa.xml / tool file=qiime/filter_otu_table.xml / label text=Diversity Analysis id=qiime_diversity/ tool file=qiime/multiple_rarefactions.xml / tool file=qiime/multiple_rarefactions_even_depth.xml / tool file=qiime/parallel_multiple_rarefactions.xml / tool file=qiime/alpha_diversity.xml / tool file=qiime/alpha_rarefaction.xml / tool file=qiime/parallel_alpha_diversity.xml / tool file=qiime/collate_alpha.xml / tool file=qiime/make_rarefaction_plots.xml / tool file=qiime/single_rarefaction.xml / tool file=qiime/beta_diversity.xml / tool file=qiime/beta_diversity_through_3d_plots.xml / tool file=qiime/compare_3d_plots.xml / tool file=qiime/beta_significance.xml / tool file=qiime/jackknifed_beta_diversity.xml / tool file=qiime/parallel_beta_diversity.xml / tool file=qiime/make_prefs_file.xml / tool file=qiime/principal_coordinates.xml / tool file=qiime/make_2d_plots.xml / tool file=qiime/make_3d_plots.xml / tool file=qiime/make_distance_histograms.xml / tool file=qiime/upgma_cluster.xml / tool file=qiime/tree_compare.xml / tool file=qiime/consensus_tree.xml / label text=More id=qiime_more/ tool file=qiime/poller_example.xml / tool file=qiime/filter_fasta.xml / tool file=qiime/add_taxa.xml / tool file=qiime/make_qiime_rst_file.xml / tool file=qiime/otu_category_significance.xml / tool file=qiime/poller.xml / tool file=qiime/supervised_learning.xml / tool file=qiime/merge_mapping_files.xml / tool file=qiime/dissimilarity_mtx_stats.xml / tool file=qiime/summarize_otu_by_cat.xml / tool file=qiime/transform_coordinate_matrices.xml / tool file=qiime/merge_otu_maps.xml / tool file=qiime/start_parallel_jobs.xml / tool file=qiime/plot_rank_abundance_graph.xml / tool file=qiime/sort_otu_table.xml / tool file=qiime/make_bootstrapped_tree.xml / tool file=qiime/trflp_file_to_otu_table.xml / tool file=qiime/pool_by_metadata.xml / tool file=qiime/submit_to_mgrast.xml / tool file=qiime/pick_reference_otus_through_otu_table.xml / tool file=qiime/merge_otu_tables.xml / tool file=qiime/filter_by_metadata.xml / tool file=qiime/plot_taxa_summary.xml / tool file=qiime/make_per_library_sff.xml / tool file=qiime/convert_unifrac_sample_mapping_to_otu_table.xml / tool file=qiime/categorized_dist_scatterplot.xml / tool file=qiime/shared_phylotypes.xml / tool file=qiime/print_qiime_config.xml / tool file=qiime/convert_otu_table_to_unifrac_sample_mapping.xml /
