Re: [galaxy-dev] PATH env variable for SGE jobs
Or you could create a directory in tool-deps called bowtie under bowtie create a default directory and in default create an env.sh file which set yours path correctly. -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Nate Coraor Sent: Friday, December 02, 2011 7:42 AM To: Langhorst, Brad Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] PATH env variable for SGE jobs On Dec 1, 2011, at 8:55 PM, Langhorst, Brad wrote: I have the software for our SGE cluster (eg. Bowtie) in /mnt/ngswork/galaxy/sw/bin And I've set the PATH environment in the galaxy user's .bashrc export PATH=/mnt/ngswork/galaxy/sw/bin:$PATH When I echo $PATH via qsub as galaxy I do see my path in STDIN.o on the job runner node /mnt/ngswork/galaxy/sw/bin:/tmp/41.1.default.q:/usr/local/bin:/bin:/us r/bin But I always see /bin/sh: bowtie: not found As an error message. How can I get the path set correctly? Hi Brad, I'd suggest putting it in ~/.sge_request rather than ~/.bashrc. This will ensure that running jobs differently (qsub vs. the DRMAA API) won't affect setting $PATH or other environment variables. See past discussions on the list for more on .sge_request: http://gmod.827538.n3.nabble.com/template/NamlServlet.jtp?macro=search_pagenode=815885query=sge_request --nate Brad -- Brad Langhorst New England Biolabs langho...@neb.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Delete toolshed repository
I want to delete a test repository I created on http://toolshed.g2.bx.psu.edu/repository?webapp=community. Is that possible? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Grooming Error when linking bam files instead of copying into galaxy
The uploaded files need grooming, so change your Copy data into Galaxy? selection to be Copy files into Galaxy instead of Link to files without copying into Galaxy so grooming can be performed. Thoughts? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Actual User Code
I think the problem has to do with self.app.config.outputs_to_working_directory which dos not jive well with linking files from the admin menu instead of uploading them. It still performs the move (i.e) in lib/galaxy/jobs/__init__.py and moves a link to the actual files and writes over it. Any thoughts on how to fix this. Looks like more of a core bug. if self.app.config.outputs_to_working_directory: for dataset_path in self.get_output_fnames(): try: shutil.move( dataset_path.false_path, dataset_path.real_path ) log.debug( finish(): Moved %s to %s % ( dataset_path.false_path, dataset_path.real_path ) ) From: Oren Livne [mailto:li...@uchicago.edu] Sent: Friday, November 11, 2011 1:33 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Actual User Code Check the permissions of the test.bam file. Maybe the galaxy user can't write into a temp directory when it runs the external_chown_script. On 11/11/2011 3:30 PM, Chorny, Ilya wrote: When I try to link file paths into a shared data library from the admin menu it fails. I get the following in the log file: galaxy.jobs DEBUG 2011-11-11 13:24:38,280 (2645) Changing ownership of working directory with: /usr/bin/sudo -E /home/galaxy/production/Sept06/galaxy-central/sc ripts/external_chown_script.py ./database/job_working_directory/2645 galaxy 504 galaxy.jobs ERROR 2011-11-11 13:24:38,771 fail(): Missing output file in working directory: [Errno 13] Permission denied: '/bioinfoSD/ichorny/test.bam' galaxy.datatypes.metadata DEBUG 2011-11-11 13:24:38,919 Cleaning up external metadata files STDOUT has the following : Job 2645's output dataset(s) could not be read Also my file get erased: /bioinfoSD/ichorny/test.bam Any thoughts? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Looks like actual user breaks splitting
I am not sure what the copy is for. I did not add the move/copy I just changed
shutil.mv to os.system(mv) because it has less baggage. In fact if I comment
it out the upload still works.
Ilya
-Original Message-
From: Nate Coraor [mailto:n...@bx.psu.edu]
Sent: Monday, November 07, 2011 7:57 AM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
On Nov 4, 2011, at 8:03 PM, Chorny, Ilya wrote:
Looks like the mv is something I added and forgot about. I changed it to
shutil.copyfile and it works fine. This also has something to do with the
isilon. I pushed it to the actual user repository so you can check it out.
Okay, I'll have a look. What's the copy for? We try to avoid moving bits as
much as possible during upload since it can slow down the entire upload tool
run pretty significantly if there are multiple copies taking place.
--nate
Have a good weekend.
Thanks,
Ilya
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 4:50 PM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
I am getting a strange error when I am uploading a file
mv: inter-device move failed:
`/illumina/scratch/Swim/galaxy/tmpk2sbZl' to
`/home/galaxy/ichorny/galaxy-central/database/tmp/upload_file_data_isA
nwO'; unable to remove target: Permission denied
I figured out that the mv is coming from
lib/galaxy/datatypes/sniff.py
if in_place:
fnamemode = stat.S_IMODE( os.stat(fname).st_mode )
os.system('mv %s %s' %(temp_name,fname))
# Return number of lines in file.
return ( i + 1, None )
else:
return ( i + 1, temp_name )
Any thoughts on what might be going on? If I comment out the move the upload
works just fine.
Thanks,
Ilya
-Original Message-
From: Chorny, Ilya
Sent: Friday, November 04, 2011 11:11 AM
To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu); Duddy, John
Cc: galaxy-dev@lists.bx.psu.edu
Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
Well I think I figured out why we get OSError: [Errno 39] Directory not
empty: './database/job_working_directory/451'. I changed it to an rm -rf and
got the following error:
`./database/job_working_directory/455/.nfs000161e986fc0068' ':
Device or resource busy
It might have something to do with our mounts being Isilon. Any thought on
how to deal with this?
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 10:50 AM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
I came up with a fix for User killed running job, but error encountered
removing from DRM queue: 'DRMAAJobRunner' object has no attribute
'job_user_uid' and pushed it.
-Original Message-
From: Chorny, Ilya
Sent: Friday, November 04, 2011 10:24 AM
To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
I am also getting the following error in the log file which makes no sense
because the directory is empty. I will try to track it down. Thoughts?
Ilya
Traceback (most recent call last):
File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/__init__.py, line
732, in cleanup
shutil.rmtree( self.working_directory ) File
/usr/lib64/python2.6/shutil.py, line 225, in rmtree
onerror(os.rmdir, path, sys.exc_info()) File
/usr/lib64/python2.6/shutil.py, line 223, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty:
'./database/job_working_directory/451'
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 9:07 AM
To: Duddy, John; Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
Nate,
I get the following error when I try to kill a job.
User killed running job, but error encountered removing from DRM queue:
'DRMAAJobRunner' object has no attribute 'job_user_uid'.
Ilya
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Duddy, John
Sent: Thursday, November 03, 2011 3:55 PM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
I just came to that conclusion myself. If you need me to do anything, let me
know - but it sounds like you have a handle on it.
Muchas
Re: [galaxy-dev] Error when clicking on published workflows
I pull the latest version of galaxy central and I still get this error when clicking on published workflows. Any thoughts? Error - type 'exceptions.TypeError': isinstance() arg 2 must be a class, type, or tuple of classes and types and below. Best, Ilya From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, November 04, 2011 2:45 PM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Error when clicking on published workflows URL: http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test Module paste.exceptions.errormiddleware:143 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test app_iter = self.application(environ, start_response) Module paste.debug.prints:98 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test environ, self.app) Module paste.wsgilib:539 in intercept_output http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return self.application(environ, start_response) Module galaxy.web.framework.middleware.remoteuser:111 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return self.app( environ, start_response ) Module paste.httpexceptions:632 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test body = method( trans, **kwargs ) Module galaxy.web.controllers.workflow:221 in display_by_username_and_slug http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test user_item_rating = user_item_rating, ave_item_rating=ave_item_rating, num_ratings=num_ratings ) Module galaxy.web.framework:718 in fill_template_mako http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return template.render( **data ) Module mako.template:296 in render http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return runtime._render(self, self.callable_, args, data) Module mako.runtime:660 in _render http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test **_kwargs_for_callable(callable_, data)) Module mako.runtime:692 in _render_context http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) Module mako.runtime:718 in _exec_template http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test callable_(context, *args, **kwargs) Module _base_panels_mako:100 in render_body http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(self.center_panel())) Module _display_base_mako:219 in render_center_panel http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(self.render_content())) Module _display_base_mako:179 in render_render_content http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(self.render_item( item, item_data ))) Module workflow_display_mako:136 in render_render_item http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(do_inputs( tool.inputs, step.state.inputs, , step ))) Module workflow_display_mako:91 in do_inputs http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return render_do_inputs(context,inputs,values,prefix,step,other_values) Module workflow_display_mako:216 in render_do_inputs http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(do_inputs( input.inputs, repeat_values[ i ], prefix + input.name + _ + str(index) + |, step, other_values ))) Module workflow_display_mako:181 in do_inputs http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return render_do_inputs(context,inputs,values,prefix,step,other_values) Module workflow_display_mako:246 in render_do_inputs http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(row_for_param( input, values[ input.name ], other_values, prefix, step ))) Module workflow_display_mako:179 in row_for_param http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return render_row_for_param(context,param,value,other_values,prefix,step) Module workflow_display_mako:300 in render_row_for_param http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test if isinstance( param, DataToolParameter ): TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client
[galaxy-dev] Workflow output selection bug
I think there is another bug in the output selection of the workflows. If I select only certain number of outputs (by selecting the orange star) to be show in history following a workflow and then edit the workflow to show more outputs, the number of outputs in the history does not change and the history only shows the number of originally selected outputs. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error when clicking on published workflows
URL: http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test Module paste.exceptions.errormiddleware:143 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test app_iter = self.application(environ, start_response) Module paste.debug.prints:98 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test environ, self.app) Module paste.wsgilib:539 in intercept_output http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return self.application(environ, start_response) Module galaxy.web.framework.middleware.remoteuser:111 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return self.app( environ, start_response ) Module paste.httpexceptions:632 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test body = method( trans, **kwargs ) Module galaxy.web.controllers.workflow:221 in display_by_username_and_slug http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test user_item_rating = user_item_rating, ave_item_rating=ave_item_rating, num_ratings=num_ratings ) Module galaxy.web.framework:718 in fill_template_mako http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return template.render( **data ) Module mako.template:296 in render http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return runtime._render(self, self.callable_, args, data) Module mako.runtime:660 in _render http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test **_kwargs_for_callable(callable_, data)) Module mako.runtime:692 in _render_context http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) Module mako.runtime:718 in _exec_template http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test callable_(context, *args, **kwargs) Module _base_panels_mako:100 in render_body http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(self.center_panel())) Module _display_base_mako:219 in render_center_panel http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(self.render_content())) Module _display_base_mako:179 in render_render_content http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(self.render_item( item, item_data ))) Module workflow_display_mako:136 in render_render_item http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(do_inputs( tool.inputs, step.state.inputs, , step ))) Module workflow_display_mako:91 in do_inputs http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return render_do_inputs(context,inputs,values,prefix,step,other_values) Module workflow_display_mako:216 in render_do_inputs http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(do_inputs( input.inputs, repeat_values[ i ], prefix + input.name + _ + str(index) + |, step, other_values ))) Module workflow_display_mako:181 in do_inputs http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return render_do_inputs(context,inputs,values,prefix,step,other_values) Module workflow_display_mako:246 in render_do_inputs http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test __M_writer(unicode(row_for_param( input, values[ input.name ], other_values, prefix, step ))) Module workflow_display_mako:179 in row_for_param http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test return render_row_for_param(context,param,value,other_values,prefix,step) Module workflow_display_mako:300 in render_row_for_param http://ussd-dev-swim02.illumina.com/galaxy2/u/ichorny/w/test if isinstance( param, DataToolParameter ): TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Looks like actual user breaks splitting
Looks like the mv is something I added and forgot about. I changed it to
shutil.copyfile and it works fine. This also has something to do with the
isilon. I pushed it to the actual user repository so you can check it out.
Have a good weekend.
Thanks,
Ilya
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 4:50 PM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
I am getting a strange error when I am uploading a file
mv: inter-device move failed: `/illumina/scratch/Swim/galaxy/tmpk2sbZl' to
`/home/galaxy/ichorny/galaxy-central/database/tmp/upload_file_data_isAnwO';
unable to remove target: Permission denied
I figured out that the mv is coming from
lib/galaxy/datatypes/sniff.py
if in_place:
fnamemode = stat.S_IMODE( os.stat(fname).st_mode )
os.system('mv %s %s' %(temp_name,fname))
# Return number of lines in file.
return ( i + 1, None )
else:
return ( i + 1, temp_name )
Any thoughts on what might be going on? If I comment out the move the upload
works just fine.
Thanks,
Ilya
-Original Message-
From: Chorny, Ilya
Sent: Friday, November 04, 2011 11:11 AM
To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu); Duddy, John
Cc: galaxy-dev@lists.bx.psu.edu
Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
Well I think I figured out why we get OSError: [Errno 39] Directory not empty:
'./database/job_working_directory/451'. I changed it to an rm -rf and got the
following error:
`./database/job_working_directory/455/.nfs000161e986fc0068' ': Device
or resource busy
It might have something to do with our mounts being Isilon. Any thought on how
to deal with this?
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 10:50 AM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
I came up with a fix for User killed running job, but error encountered
removing from DRM queue: 'DRMAAJobRunner' object has no attribute
'job_user_uid' and pushed it.
-Original Message-
From: Chorny, Ilya
Sent: Friday, November 04, 2011 10:24 AM
To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: RE: [galaxy-dev] Looks like actual user breaks splitting
I am also getting the following error in the log file which makes no sense
because the directory is empty. I will try to track it down. Thoughts?
Ilya
Traceback (most recent call last):
File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/__init__.py, line
732, in cleanup
shutil.rmtree( self.working_directory )
File /usr/lib64/python2.6/shutil.py, line 225, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File /usr/lib64/python2.6/shutil.py, line 223, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: './database/job_working_directory/451'
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, November 04, 2011 9:07 AM
To: Duddy, John; Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
Nate,
I get the following error when I try to kill a job.
User killed running job, but error encountered removing from DRM queue:
'DRMAAJobRunner' object has no attribute 'job_user_uid'.
Ilya
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Duddy, John
Sent: Thursday, November 03, 2011 3:55 PM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
I just came to that conclusion myself. If you need me to do anything, let me
know - but it sounds like you have a handle on it.
Muchas gracias.
John Duddy
Sr. Staff Software Engineer
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Tel: 858-736-3584
E-mail: jdu...@illumina.com
-Original Message-
From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu]
Sent: Thursday, November 03, 2011 3:53 PM
To: Duddy, John
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Looks like actual user breaks splitting
Nate Coraor (n...@bx.psu.edu) wrote:
Duddy, John wrote:
I'm not following you - it's been 6 months since I wrote that code
;-}
I know the feeling!
IT looks to me like a DatasetPath() object is always placed in that array,
and with one exception near then, it looks like the change I made generates
those objects the same way.
It's creating a dict in self.output_dataset_paths, and that dict looks
like
[galaxy-dev] Looks like actual user breaks splitting
Hey John, Any thoughts? Ilya Traceback (most recent call last): File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/runners/tasks.py, line 73, in run_job tasks = splitter.do_split(job_wrapper) File /home/galaxy/ichorny/galaxy-central/lib/galaxy/jobs/splitters/multi.py, line 73, in do_split input_type.split(input_datasets, get_new_working_directory_name, parallel_settings) File /home/galaxy/ichorny/galaxy-central/lib/galaxy/datatypes/data.py, line 473, in split raise Exception(Text file splitting does not support multiple files) Exception: Text file splitting does not support multiple files Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Clusters, Runners, and user credentials
BTW, I am not sure if PBS works with drmaa. If not then the code will need to be ported to work with pbs. Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, October 31, 2011 3:27 PM To: Lloyd Brown; Galaxy Dev List Subject: Re: [galaxy-dev] Clusters, Runners, and user credentials Lyod, See Nate's email below Title: Actual user code. We have been working on implementing this feature in galaxy. The code is still in development but feel free to test it out and let us know how it works for you. Best, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Lloyd Brown Sent: Monday, October 31, 2011 2:35 PM To: Galaxy Dev List Subject: [galaxy-dev] Clusters, Runners, and user credentials I'm a systems administrator for an HPC cluster, and have been asked by a faculty member here to try to get galaxy to work on our cluster. Unfortunately, there are one or two outstanding questions that I can't seem to find the answer to, and I'm hoping someone here can help me out. In particular, is galaxy, and the PBS runner specifically, capable of submitting jobs under specific user names? Essentially, if I set up galaxy to push jobs to our cluster, will they all show up under one user credential (eg. the galaxy user), or can we set it up so that the user logged into galaxy, is used to submit the job? This one is kindof a show-stopper, since our internal policies require that all jobs have a specific user credential, with one person per username. Thanks, Lloyd -- Lloyd Brown Systems Administrator Fulton Supercomputing Lab Brigham Young University http://marylou.byu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Clusters, Runners, and user credentials
I modified drmaa.py to pass the galaxy users path variable to the actual user. As long as the galaxy user's environment is correct then the actual user's environment should be correct. -Original Message- From: Glen Beane [mailto:glen.be...@jax.org] Sent: Monday, October 31, 2011 4:20 PM To: Chorny, Ilya Cc: Lloyd Brown; Galaxy Dev List Subject: Re: [galaxy-dev] Clusters, Runners, and user credentials Many of us are using the PBS job runner (for TORQUE) and would definitely be interested in a port. How do you deal with making sure the user's environment is configured properly? We use a python virtualenv and load specific module files with tested tool versions in our galaxy users startup scripts on our cluster. Sent from my iPhone On Oct 31, 2011, at 6:29 PM, Chorny, Ilya icho...@illumina.com wrote: BTW, I am not sure if PBS works with drmaa. If not then the code will need to be ported to work with pbs. Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, October 31, 2011 3:27 PM To: Lloyd Brown; Galaxy Dev List Subject: Re: [galaxy-dev] Clusters, Runners, and user credentials Lyod, See Nate's email below Title: Actual user code. We have been working on implementing this feature in galaxy. The code is still in development but feel free to test it out and let us know how it works for you. Best, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Lloyd Brown Sent: Monday, October 31, 2011 2:35 PM To: Galaxy Dev List Subject: [galaxy-dev] Clusters, Runners, and user credentials I'm a systems administrator for an HPC cluster, and have been asked by a faculty member here to try to get galaxy to work on our cluster. Unfortunately, there are one or two outstanding questions that I can't seem to find the answer to, and I'm hoping someone here can help me out. In particular, is galaxy, and the PBS runner specifically, capable of submitting jobs under specific user names? Essentially, if I set up galaxy to push jobs to our cluster, will they all show up under one user credential (eg. the galaxy user), or can we set it up so that the user logged into galaxy, is used to submit the job? This one is kindof a show-stopper, since our internal policies require that all jobs have a specific user credential, with one person per username. Thanks, Lloyd -- Lloyd Brown Systems Administrator Fulton Supercomputing Lab Brigham Young University http://marylou.byu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Re Acutal user code
It seems to be working but when I run BWA or Bowtie it finishes green but the files are empty. Doesn't give me much information. I also tried fastqc and it ran just fine. -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Monday, October 31, 2011 1:21 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Sweet!! Can't wait to see it. Have a look: https://bitbucket.org/natefoo/galaxy-central-ichorny/ Caveats: * job_working_directory has to be in a shared filesystem * outputs_to_working_directory must be True * set_metadata_externally must be True One thing that would be good from a security standpoint would be if the external chown script took a Galaxy Job ID for its argument instead of a path, and then looked up the working directory path from the ID. That way the script couldn't be used maliciously to chown any file. --nate Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Thursday, October 27, 2011 1:24 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours. Chorny, Ilya wrote: Hi Nate, Any updates? Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know. Ilya -Original Message- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client
Re: [galaxy-dev] Re Acutal user code
Hi Nate, Any updates? Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know. Ilya -Original Message- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Modifying the GATK wrappers to pick genome infomration from bam/sam dbkey attribute
Hi Dan,
I want to modify the wrappers to get genome information from the bam/sam
attribute. I did this for the tophat/cufflinks wrappers. I add another option
to select genome called attribute and then I pass $bam.metadata.dkey in the
command line (see example below). Is this something you might be interested in?
Can you suggest an alternative way to do this? My only concern is my inability
to validate that the $input.metadata.dbkey exists.
Thanks,
Ilya
See example below.
#if $reference_annotation.use_ref == Use reference annotation:
#if
$reference_annotation.annotationSource.reference_annotation_file == indexed:
-G ${ filter( lambda x: str( x[0] ) == str(
$reference_annotation.annotationSource.indices ), $__app__.tool_data_tables[
'gtf_index' ].get_fields() )[0][-1] }
#else:
#if
$reference_annotation.annotationSource.reference_annotation_file == attribute:
-G ${ filter( lambda x: str( x[0] ) == str(
$input.metadata.dbkey ), $__app__.tool_data_tables[ 'gtf_index' ].get_fields()
)[0][-1] }
#else:
-G ${reference_annotation.annotationSource.ownFile}
#end if
#end if
#end if
conditional name=annotationSource
param name=reference_annotation_file type=select
label=Please select a reference Aonnotation
option value=indexedUse a built-in index/option
option value=historyUse one from the
history/option
option value=attributeUse input bam
metadata.dbkey attribute/option
/param
when value=indexed
param name=indices type=select label=Select
genome for gtf annotation
options from_data_table=gtf_index
filter type=sort_by column=3 /
validator type=no_options message=No indexes
are available for the selected input dataset /
/options
/param
/when
when value=history
param name=ownFile type=data format=gff3, gtf
label=Select a reference annotation file /
/when
/conditional
Ilya Chorny Ph.D.
Bioinformatics Scientist I
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Work: 858.202.4582
Email: icho...@illumina.commailto:icho...@illumina.com
Website: www.illumina.comhttp://www.illumina.com
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Modifying the GATK wrappers to pick genome infomration from bam/sam dbkey attribute
What I am trying to do is, instead of having to select the genome from a drop
down that points to the loc file, to take the metadata.dbkey attribute of an
input file and use it to select the genome. I want to avoid having a user of my
workflow select a genome n times in each step of a workflow. Attached is a
sample cufflinks_wrapper.xml and gtf_indices.loc file.
With regards to GATK, I want to edit the tools that are in the broad best
practices workflow.
http://www.broadinstitute.org/gsa/wiki/index.php/Best_Practice_Variant_Detection_with_the_GATK_v3
Thanks,
Ilya
From: Daniel Blankenberg [mailto:d...@bx.psu.edu]
Sent: Thursday, October 27, 2011 11:51 AM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: Modifying the GATK wrappers to pick genome infomration from
bam/sam dbkey attribute
Hi Ilya,
You can validate dbkeys using validators including metadata,
dataset_metadata_in_file/data_table, unspecified_build and others.
Could you clarify what GATK tools you are trying to modify and exactly what you
are trying accomplish? A complete example xml file, and any additional files
(e.g. *.loc) would need to be provided to give you direction.
Thanks for using Galaxy,
Dan
On Oct 27, 2011, at 1:03 PM, Chorny, Ilya wrote:
Hi Dan,
I want to modify the wrappers to get genome information from the bam/sam
attribute. I did this for the tophat/cufflinks wrappers. I add another option
to select genome called attribute and then I pass $bam.metadata.dkey in the
command line (see example below). Is this something you might be interested in?
Can you suggest an alternative way to do this? My only concern is my inability
to validate that the $input.metadata.dbkey exists.
Thanks,
Ilya
See example below.
#if $reference_annotation.use_ref == Use reference annotation:
#if
$reference_annotation.annotationSource.reference_annotation_file == indexed:
-G ${ filter( lambda x: str( x[0] ) == str(
$reference_annotation.annotationSource.indices ), $__app__.tool_data_tables[
'gtf_index' ].get_fields() )[0][-1] }
#else:
#if
$reference_annotation.annotationSource.reference_annotation_file == attribute:
-G ${ filter( lambda x: str( x[0] ) == str(
$input.metadata.dbkey ), $__app__.tool_data_tables[ 'gtf_index' ].get_fields()
)[0][-1] }
#else:
-G ${reference_annotation.annotationSource.ownFile}
#end if
#end if
#end if
conditional name=annotationSource
param name=reference_annotation_file type=select
label=Please select a reference Aonnotation
option value=indexedUse a built-in index/option
option value=historyUse one from the
history/option
option value=attributeUse input bam
metadata.dbkey attribute/option
/param
when value=indexed
param name=indices type=select label=Select
genome for gtf annotation
options from_data_table=gtf_index
filter type=sort_by column=3 /
validator type=no_options message=No indexes
are available for the selected input dataset /
/options
/param
/when
when value=history
param name=ownFile type=data format=gff3, gtf
label=Select a reference annotation file /
/when
/conditional
Ilya Chorny Ph.D.
Bioinformatics Scientist I
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Work: 858.202.4582
Email: icho...@illumina.commailto:icho...@illumina.com
Website: www.illumina.comhttp://www.illumina.com
cufflinks_wrapper.xml.diff
Description: cufflinks_wrapper.xml.diff
gtf_indices.loc
Description: gtf_indices.loc
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?
Any more thought on how to accomplish this without my current hack? Ideally it would be great if this feature could become part of galaxy-central. This is a feature that we would really like to implement so that our reports can link to the output files. Thanks, Ilya -Original Message- From: James Taylor [mailto:ja...@jamestaylor.org] Sent: Friday, October 21, 2011 9:47 AM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper? Excerpts from Chorny, Ilya's message of 2011-10-21 01:51:03 +: I get the following error from routes AttributeError: 'thread._local' object has no attribute 'mapper' Okay, what I suspected. This is a product of the fact that job running is decoupled from the web frontend. The job runner doesn't necessarily live in the same process or even on the same machine as the web interface. You could have multiple web interfaces serving different base urls feeding one job runner. I don't have any simple answer to this. You could try to initialize route's mapper in the job runner, but I think this would be a big mistake. Instead you probably want to capture the URL somehow when the job is created. I need to think about it some more. -- James Taylor, Assistant Professor, Biology / Computer Science, Emory University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Broad IGV
I saw some posts and a commit which allows one to use broad IGV within galaxy but I do not see a tool, a link, or an option. Can someone explain how to set that up? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?
I need the base_url and the proxy. That is why I am using cookie_path. Any
thoughts on how to get both.
Thanks,
Ilya
-Original Message-
From: Greg Von Kuster [mailto:g...@bx.psu.edu]
Sent: Friday, October 21, 2011 6:11 AM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] FW: how do I pass the URL of the input and output
files to my tool wrapper?
Ilya, you can use trans.request.host to get the base url.
On Oct 20, 2011, at 11:52 AM, Chorny, Ilya wrote:
I thought I would add this to the list in case anyone in the galaxy :) knew
the answer. I get a mapping error every time I invoke url_for. What variable
stores the base url for the galaxy instance?
Thanks,
Ilya
-Original Message-
From: Chorny, Ilya
Sent: Wednesday, October 19, 2011 4:21 PM
To: 'James Taylor'
Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files
to my tool wrapper?
Hi James,
I got the code working but I can't figure out how to set the URL. I have a
hack right now (i.e. url = 'http://' + socket.getfqdn() +
config_info.cookie_path + '/datasets/' + sec.encode_id(dataset.id) +
'/display/?preview=True' ). I tried using url_for() but was unable to get
that to work. Attached are the diff's for the files I made changes to.
If you have any suggestions on how to properly set the url please let me know.
Thanks,
Ilya
-Original Message-
From: James Taylor [mailto:ja...@jamestaylor.org]
Sent: Tuesday, October 18, 2011 11:38 AM
To: Chorny, Ilya
Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files
to my tool wrapper?
Hi Ilya, I think any case where $__app__ is used should raise red flags, it
is definitely breaking abstraction. In this case, you're passing a very
specific implementation detail of the web layer all the way down to your tool.
I think the right way to do what you want is to add a 'web_display_url'
attribute to the class DatasetFilenameWrapper in tools/__init__.py.
Every dataset gets wrapped by this when put into the namespace for building
command lines. If you added an attribute it would them be available as
${input.web_display_url} in the command line template.
Historically tool wrappers were designed so that they have no knowledge of
the UI, allowing non-web UIs to be added on top. At this point it seems
unlikely that this will ever happen, but it is still a good idea to avoid
this kind of coupling. Otherwise we can't change how the web interface does
security without changing individual tools.
Excerpts from Chorny, Ilya's message of 2011-10-17 22:56:40 +:
So we came up with a work around and I wouldn't mind getting your opinion on
it. Specifically with respect to passing cookie_path. I could not figure out
how to get the proxy prefix from the universe_wsgi.ini file via the $__app__
call. Also you mention in the wiki the the $__app__ may get deprecated.
Thanks,
Ilya
We copied the encryption code (i.e)
def encode_id(dataset_id,id_secret):
id_cipher = Blowfish.new(id_secret)
s = dataset_id
s = ( ! * ( 8 - len(s) % 8 ) ) + s
encrypted_id = id_cipher.encrypt( s ).encode( 'hex' )
return encrypted_id
and we passed to the too wrapperl:
-i input -i $input.dataset.id
as well as
--id_secret=$__app__.config.id_secret
--cookie_path=$__app__.config.cookie_path
-Original Message-
From: Chorny, Ilya
Sent: Monday, October 17, 2011 11:10 AM
To: Chorny, Ilya; James Taylor
Cc: galaxy-dev
Subject: RE: [galaxy-dev] how do I pass the URL of the input and output
files to my tool wrapper?
Would it be difficult to add the url as an attribute of the input and output
parameters ($param.url)? If you point me to where these values are set, I
can try to make the changes to the code.
Thanks,
Ilya
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny,
Ilya
Sent: Sunday, October 16, 2011 7:36 AM
To: James Taylor
Cc: galaxy-dev
Subject: Re: [galaxy-dev] how do I pass the URL of the input and output
files to my tool wrapper?
PDF report.
Sent from my iPhone
On Oct 15, 2011, at 7:36 PM, James Taylor ja...@jamestaylor.org wrote:
Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +:
Anyone know if it possible to pass the url (shown in the eye ball) to a
tool wrapper?
Ilya, this isn't currently supported because it breaks the
abstraction between the web interface and tools (tools shouldn't
know they are being invoked by a web application). What are you trying to
achieve with this?
Is this for a tool generating an HTML report?
--
James Taylor, Assistant Professor, Biology / Computer Science, Emory
University
___
Please keep all replies on the list by using reply all
in your mail client. To manage your
Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?
Also, I do not have access to trans in the Tools class in
lib/galaxy/tools/__init__.py.
Thanks,
Ilya
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, October 21, 2011 8:40 AM
To: Greg Von Kuster
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] FW: how do I pass the URL of the input and output
files to my tool wrapper?
I need the base_url and the proxy. That is why I am using cookie_path. Any
thoughts on how to get both.
Thanks,
Ilya
-Original Message-
From: Greg Von Kuster [mailto:g...@bx.psu.edu]
Sent: Friday, October 21, 2011 6:11 AM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] FW: how do I pass the URL of the input and output
files to my tool wrapper?
Ilya, you can use trans.request.host to get the base url.
On Oct 20, 2011, at 11:52 AM, Chorny, Ilya wrote:
I thought I would add this to the list in case anyone in the galaxy :) knew
the answer. I get a mapping error every time I invoke url_for. What variable
stores the base url for the galaxy instance?
Thanks,
Ilya
-Original Message-
From: Chorny, Ilya
Sent: Wednesday, October 19, 2011 4:21 PM
To: 'James Taylor'
Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files
to my tool wrapper?
Hi James,
I got the code working but I can't figure out how to set the URL. I have a
hack right now (i.e. url = 'http://' + socket.getfqdn() +
config_info.cookie_path + '/datasets/' + sec.encode_id(dataset.id) +
'/display/?preview=True' ). I tried using url_for() but was unable to get
that to work. Attached are the diff's for the files I made changes to.
If you have any suggestions on how to properly set the url please let me know.
Thanks,
Ilya
-Original Message-
From: James Taylor [mailto:ja...@jamestaylor.org]
Sent: Tuesday, October 18, 2011 11:38 AM
To: Chorny, Ilya
Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files
to my tool wrapper?
Hi Ilya, I think any case where $__app__ is used should raise red flags, it
is definitely breaking abstraction. In this case, you're passing a very
specific implementation detail of the web layer all the way down to your tool.
I think the right way to do what you want is to add a 'web_display_url'
attribute to the class DatasetFilenameWrapper in tools/__init__.py.
Every dataset gets wrapped by this when put into the namespace for building
command lines. If you added an attribute it would them be available as
${input.web_display_url} in the command line template.
Historically tool wrappers were designed so that they have no knowledge of
the UI, allowing non-web UIs to be added on top. At this point it seems
unlikely that this will ever happen, but it is still a good idea to avoid
this kind of coupling. Otherwise we can't change how the web interface does
security without changing individual tools.
Excerpts from Chorny, Ilya's message of 2011-10-17 22:56:40 +:
So we came up with a work around and I wouldn't mind getting your opinion on
it. Specifically with respect to passing cookie_path. I could not figure out
how to get the proxy prefix from the universe_wsgi.ini file via the $__app__
call. Also you mention in the wiki the the $__app__ may get deprecated.
Thanks,
Ilya
We copied the encryption code (i.e)
def encode_id(dataset_id,id_secret):
id_cipher = Blowfish.new(id_secret)
s = dataset_id
s = ( ! * ( 8 - len(s) % 8 ) ) + s
encrypted_id = id_cipher.encrypt( s ).encode( 'hex' )
return encrypted_id
and we passed to the too wrapperl:
-i input -i $input.dataset.id
as well as
--id_secret=$__app__.config.id_secret
--cookie_path=$__app__.config.cookie_path
-Original Message-
From: Chorny, Ilya
Sent: Monday, October 17, 2011 11:10 AM
To: Chorny, Ilya; James Taylor
Cc: galaxy-dev
Subject: RE: [galaxy-dev] how do I pass the URL of the input and output
files to my tool wrapper?
Would it be difficult to add the url as an attribute of the input and output
parameters ($param.url)? If you point me to where these values are set, I
can try to make the changes to the code.
Thanks,
Ilya
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny,
Ilya
Sent: Sunday, October 16, 2011 7:36 AM
To: James Taylor
Cc: galaxy-dev
Subject: Re: [galaxy-dev] how do I pass the URL of the input and output
files to my tool wrapper?
PDF report.
Sent from my iPhone
On Oct 15, 2011, at 7:36 PM, James Taylor ja...@jamestaylor.org wrote:
Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +:
Anyone know if it possible to pass the url (shown in the eye ball) to a
tool wrapper?
Ilya, this isn't currently supported because it breaks the
abstraction between
Re: [galaxy-dev] update of cufflinks and tophat to use dbkey attribute of input files
Can you guys show me an example of how this is used in the xml file? Thanks, Ilya From: Jeremy Goecks [mailto:jeremy.goe...@emory.edu] Sent: Thursday, October 20, 2011 7:03 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: update of cufflinks and tophat to use dbkey attribute of input files I made some more modifications to the wrappers to allow for the tools to pull the genome information from the input files. The only drawback is that there is no way to validate whether the input has the dbkey attribute as a parameter (can you suggest a way to validate?). Attached are the diff's to the current version in galaxy-central. See Dan's recent commit for validation via data tables: https://bitbucket.org/galaxy/galaxy-central/changeset/150069a23aad Thanks, J. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Re Acutal user code
Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Re Acutal user code
One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know. Ilya -Original Message- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (n...@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote: Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] update of cufflinks and tophat to use dbkey attribute of input files
Hi Jeremy, I made some more modifications to the wrappers to allow for the tools to pull the genome information from the input files. The only drawback is that there is no way to validate whether the input has the dbkey attribute as a parameter (can you suggest a way to validate?). Attached are the diff's to the current version in galaxy-central. Best, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com tophat_wrapper.xml.diff Description: tophat_wrapper.xml.diff cufflinks_wrapper.xml.diff Description: cufflinks_wrapper.xml.diff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?
I thought I would add this to the list in case anyone in the galaxy :) knew the
answer. I get a mapping error every time I invoke url_for. What variable
stores the base url for the galaxy instance?
Thanks,
Ilya
-Original Message-
From: Chorny, Ilya
Sent: Wednesday, October 19, 2011 4:21 PM
To: 'James Taylor'
Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files
to my tool wrapper?
Hi James,
I got the code working but I can't figure out how to set the URL. I have a hack
right now (i.e. url = 'http://' + socket.getfqdn() + config_info.cookie_path +
'/datasets/' + sec.encode_id(dataset.id) + '/display/?preview=True' ). I tried
using url_for() but was unable to get that to work. Attached are the diff's for
the files I made changes to.
If you have any suggestions on how to properly set the url please let me know.
Thanks,
Ilya
-Original Message-
From: James Taylor [mailto:ja...@jamestaylor.org]
Sent: Tuesday, October 18, 2011 11:38 AM
To: Chorny, Ilya
Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files
to my tool wrapper?
Hi Ilya, I think any case where $__app__ is used should raise red flags, it is
definitely breaking abstraction. In this case, you're passing a very specific
implementation detail of the web layer all the way down to your tool.
I think the right way to do what you want is to add a 'web_display_url'
attribute to the class DatasetFilenameWrapper in tools/__init__.py.
Every dataset gets wrapped by this when put into the namespace for building
command lines. If you added an attribute it would them be available as
${input.web_display_url} in the command line template.
Historically tool wrappers were designed so that they have no knowledge of the
UI, allowing non-web UIs to be added on top. At this point it seems unlikely
that this will ever happen, but it is still a good idea to avoid this kind of
coupling. Otherwise we can't change how the web interface does security without
changing individual tools.
Excerpts from Chorny, Ilya's message of 2011-10-17 22:56:40 +:
So we came up with a work around and I wouldn't mind getting your opinion on
it. Specifically with respect to passing cookie_path. I could not figure out
how to get the proxy prefix from the universe_wsgi.ini file via the $__app__
call. Also you mention in the wiki the the $__app__ may get deprecated.
Thanks,
Ilya
We copied the encryption code (i.e)
def encode_id(dataset_id,id_secret):
id_cipher = Blowfish.new(id_secret)
s = dataset_id
s = ( ! * ( 8 - len(s) % 8 ) ) + s
encrypted_id = id_cipher.encrypt( s ).encode( 'hex' )
return encrypted_id
and we passed to the too wrapperl:
-i input -i $input.dataset.id
as well as
--id_secret=$__app__.config.id_secret
--cookie_path=$__app__.config.cookie_path
-Original Message-
From: Chorny, Ilya
Sent: Monday, October 17, 2011 11:10 AM
To: Chorny, Ilya; James Taylor
Cc: galaxy-dev
Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files
to my tool wrapper?
Would it be difficult to add the url as an attribute of the input and output
parameters ($param.url)? If you point me to where these values are set, I can
try to make the changes to the code.
Thanks,
Ilya
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Sunday, October 16, 2011 7:36 AM
To: James Taylor
Cc: galaxy-dev
Subject: Re: [galaxy-dev] how do I pass the URL of the input and output files
to my tool wrapper?
PDF report.
Sent from my iPhone
On Oct 15, 2011, at 7:36 PM, James Taylor ja...@jamestaylor.org wrote:
Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +:
Anyone know if it possible to pass the url (shown in the eye ball) to a
tool wrapper?
Ilya, this isn't currently supported because it breaks the
abstraction between the web interface and tools (tools shouldn't
know they are being invoked by a web application). What are you trying to
achieve with this?
Is this for a tool generating an HTML report?
--
James Taylor, Assistant Professor, Biology / Computer Science, Emory
University
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this and other
Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this and other Galaxy
lists, please use the interface at:
http://lists.bx.psu.edu/
--
James Taylor, Assistant Professor, Biology / Computer Science, Emory University
lib.galaxy.tools.__init__.py.diff
Description: lib.galaxy.tools.__init__.py.diff
Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?
I get the following error from routes AttributeError: 'thread._local' object has no attribute 'mapper' -Original Message- From: James Taylor [mailto:ja...@jamestaylor.org] Sent: Thursday, October 20, 2011 5:46 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper? Excerpts from Chorny, Ilya's message of 2011-10-20 15:52:03 +: I thought I would add this to the list in case anyone in the galaxy :) knew the answer. I get a mapping error every time I invoke url_for. What variable stores the base url for the galaxy instance? Can you provide the specific error you get from url_for? There is no single variable that stores the app url, it is constructed from the routing system which url_for is connected to. If you look in form_builder.py for example you can see the use of url_for from non template code. -- James Taylor, Assistant Professor, Biology / Computer Science, Emory University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Re Acutal user code
Nate, I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Upload File Button
I create an upload file button using upload.xml in one of my xml scripts. param name=file_data type=file size=30 label=File ajax-upload=true help=TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator). /param But when I add it to a work flow and try to have it set the parameters at runtime, the set at runtime options fails. Any ideas why? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Upload File Button
I want to upload an experiment specific file from the workflow and bypass loading it into the history. -Original Message- From: James Taylor [mailto:ja...@jamestaylor.org] Sent: Wednesday, October 19, 2011 4:50 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Upload File Button Excerpts from Chorny, Ilya's message of 2011-10-19 23:41:28 +: I create an upload file button using upload.xml in one of my xml scripts. ... But when I add it to a work flow and try to have it set the parameters at runtime, the set at runtime options fails. Any ideas why? There is currently no support for upload tool parameters in workflows. What behavior are you looking for in this case? The upload tool is very special, and the parameters are quite interdependent. Are you trying to create a new dataset from the uploaded file or something else? -- James Taylor, Assistant Professor, Biology / Computer Science, Emory University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Trouble getting setting dbkey of output data
Anybody, anybody?
Thanks,
Ilya
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Monday, October 17, 2011 9:31 AM
To: galaxy-dev@lists.bx.psu.edu
Subject: [galaxy-dev] Trouble getting setting dbkey of output data
I am having trouble setting the dbkey of an output data set. The input and
output, respectively, is shown below. I copied this from the bowtie_wrapper.xml
but the difference is that I do not have a condition on selecting the
reference genome. My tool requires a .loc file. The tools completes correctly
but the dbkey is not set. Any thoughts on what I may be doing wrong?
Thanks,
Ilya
param name=input2 type=select label=Select a reference genome
options from_data_table=RNASEQ_contams
filter type=sort_by column=2 /
validator type=no_options message=No indexes are available for
the selected input dataset /
/options
/param
data format=sam name=output2 label=${tool.name} on ${on_string}:Output
Sam File /
actions
action type=metadata name=dbkey
option type=from_data_table name=RNASEQ_contams column=1
offset=0
filter type=param_value column=0 value=#
compare=startswith keep=False/
filter type=param_value ref=input2 column=0/
/option
/action
/actions
Ilya Chorny Ph.D.
Bioinformatics Scientist I
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Work: 858.202.4582
Email: icho...@illumina.commailto:icho...@illumina.com
Website: www.illumina.comhttp://www.illumina.com
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Trouble getting setting dbkey of output data
Great, thanks!!!
From: Daniel Blankenberg [mailto:d...@bx.psu.edu]
Sent: Tuesday, October 18, 2011 10:24 AM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Trouble getting setting dbkey of output data
Hi Ilya,
The 'actions' tagset must be within the ''data' tagset, you have it outside.
Try something like:
data format=sam name=output2 label=${tool.name} on ${on_string}:Output
Sam File
actions
action type=metadata name=dbkey
option type=from_data_table name=RNASEQ_contams column=1
offset=0
filter type=param_value column=0 value=#
compare=startswith keep=False/
filter type=param_value ref=input2 column=0/
/option
/action
/actions
/data
Please let us know if we can provide additional assistance.
Thanks for using Galaxy,
Dan
On Oct 18, 2011, at 12:29 PM, Chorny, Ilya wrote:
Anybody, anybody?
Thanks,
Ilya
From:
galaxy-dev-boun...@lists.bx.psu.edumailto:galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu]mailto:[mailto:galaxy-dev-boun...@lists.bx.psu.edu]
On Behalf Of Chorny, Ilya
Sent: Monday, October 17, 2011 9:31 AM
To: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: [galaxy-dev] Trouble getting setting dbkey of output data
I am having trouble setting the dbkey of an output data set. The input and
output, respectively, is shown below. I copied this from the bowtie_wrapper.xml
but the difference is that I do not have a condition on selecting the
reference genome. My tool requires a .loc file. The tools completes correctly
but the dbkey is not set. Any thoughts on what I may be doing wrong?
Thanks,
Ilya
param name=input2 type=select label=Select a reference genome
options from_data_table=RNASEQ_contams
filter type=sort_by column=2 /
validator type=no_options message=No indexes are available for
the selected input dataset /
/options
/param
data format=sam name=output2 label=${tool.name} on ${on_string}:Output
Sam File /
actions
action type=metadata name=dbkey
option type=from_data_table name=RNASEQ_contams column=1
offset=0
filter type=param_value column=0 value=#
compare=startswith keep=False/
filter type=param_value ref=input2 column=0/
/option
/action
/actions
Ilya Chorny Ph.D.
Bioinformatics Scientist I
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Work: 858.202.4582
Email: icho...@illumina.commailto:icho...@illumina.com
Website: www.illumina.comhttp://www.illumina.com
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] Trouble getting setting dbkey of output data
I am having trouble setting the dbkey of an output data set. The input and
output, respectively, is shown below. I copied this from the bowtie_wrapper.xml
but the difference is that I do not have a condition on selecting the
reference genome. My tool requires a .loc file. The tools completes correctly
but the dbkey is not set. Any thoughts on what I may be doing wrong?
Thanks,
Ilya
param name=input2 type=select label=Select a reference genome
options from_data_table=RNASEQ_contams
filter type=sort_by column=2 /
validator type=no_options message=No indexes are available for
the selected input dataset /
/options
/param
data format=sam name=output2 label=${tool.name} on ${on_string}:Output
Sam File /
actions
action type=metadata name=dbkey
option type=from_data_table name=RNASEQ_contams column=1
offset=0
filter type=param_value column=0 value=#
compare=startswith keep=False/
filter type=param_value ref=input2 column=0/
/option
/action
/actions
Ilya Chorny Ph.D.
Bioinformatics Scientist I
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Work: 858.202.4582
Email: icho...@illumina.commailto:icho...@illumina.com
Website: www.illumina.comhttp://www.illumina.com
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] attributes associted with xml inputs and outputs
Is there a list of associated attributes such as $input.ext, $input.dbkey and $input.id? Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper?
Would it be difficult to add the url as an attribute of the input and output parameters ($param.url)? If you point me to where these values are set, I can try to make the changes to the code. Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Sunday, October 16, 2011 7:36 AM To: James Taylor Cc: galaxy-dev Subject: Re: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper? PDF report. Sent from my iPhone On Oct 15, 2011, at 7:36 PM, James Taylor ja...@jamestaylor.org wrote: Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +: Anyone know if it possible to pass the url (shown in the eye ball) to a tool wrapper? Ilya, this isn't currently supported because it breaks the abstraction between the web interface and tools (tools shouldn't know they are being invoked by a web application). What are you trying to achieve with this? Is this for a tool generating an HTML report? -- James Taylor, Assistant Professor, Biology / Computer Science, Emory University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper?
So we came up with a work around and I wouldn't mind getting your opinion on it. Specifically with respect to passing cookie_path. I could not figure out how to get the proxy prefix from the universe_wsgi.ini file via the $__app__ call. Also you mention in the wiki the the $__app__ may get deprecated. Thanks, Ilya We copied the encryption code (i.e) def encode_id(dataset_id,id_secret): id_cipher = Blowfish.new(id_secret) s = dataset_id s = ( ! * ( 8 - len(s) % 8 ) ) + s encrypted_id = id_cipher.encrypt( s ).encode( 'hex' ) return encrypted_id and we passed to the too wrapperl: -i input -i $input.dataset.id as well as --id_secret=$__app__.config.id_secret --cookie_path=$__app__.config.cookie_path -Original Message- From: Chorny, Ilya Sent: Monday, October 17, 2011 11:10 AM To: Chorny, Ilya; James Taylor Cc: galaxy-dev Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper? Would it be difficult to add the url as an attribute of the input and output parameters ($param.url)? If you point me to where these values are set, I can try to make the changes to the code. Thanks, Ilya -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Sunday, October 16, 2011 7:36 AM To: James Taylor Cc: galaxy-dev Subject: Re: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper? PDF report. Sent from my iPhone On Oct 15, 2011, at 7:36 PM, James Taylor ja...@jamestaylor.org wrote: Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +: Anyone know if it possible to pass the url (shown in the eye ball) to a tool wrapper? Ilya, this isn't currently supported because it breaks the abstraction between the web interface and tools (tools shouldn't know they are being invoked by a web application). What are you trying to achieve with this? Is this for a tool generating an HTML report? -- James Taylor, Assistant Professor, Biology / Computer Science, Emory University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] how do I pass the URL of the input and output files to my tool wrapper?
PDF report. Sent from my iPhone On Oct 15, 2011, at 7:36 PM, James Taylor ja...@jamestaylor.org wrote: Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +: Anyone know if it possible to pass the url (shown in the eye ball) to a tool wrapper? Ilya, this isn't currently supported because it breaks the abstraction between the web interface and tools (tools shouldn't know they are being invoked by a web application). What are you trying to achieve with this? Is this for a tool generating an HTML report? -- James Taylor, Assistant Professor, Biology / Computer Science, Emory University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] strange history error
Thanks, we actually had the wrong cookie_path set and that seemed to be causing the problem. Ilya -Original Message- From: Hans-Rudolf Hotz [mailto:h...@fmi.ch] Sent: Wednesday, October 12, 2011 1:09 AM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] strange history error Hi just asking the obvious questions to double check: - are you sure, this second instance is not accessing the same PostgreSQL/MySQL database as the first one? - are you sure, the user you are logged-in does exist in the PostgreSQL/MySQL database? I kind of remember getting a similar error after switching to external authentication. The difference was due to capitalization in parts of the e-mail address. Regards, Hans On 10/12/2011 01:39 AM, Chorny, Ilya wrote: I am getting a strange error that I cannot figure out how to debug. It's from a second instance we have running on the same server. When I upload data it does not show me my history. I get the following messages from the log file. Thoughts? galaxy.web.framework DEBUG 2011-10-11 15:47:50,259 Error: this request returned None from get_history(): http://ussd-dev-swim02.illumina.com/galaxy2/history/list galaxy.web.framework DEBUG 2011-10-11 15:47:50,261 Error: this request returned None from get_history(): http://ussd-dev-swim02.illumina.com/galaxy2/history/list galaxy.web.framework DEBUG 2011-10-11 16:34:25,717 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,717 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,718 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,718 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,718 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,718 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,719 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,719 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Trasfering Galaxy workflows to other instances
Is there a way of uploading workflows by saving the workflow to a file and copying the workflow files to a directory or editing a database file to point to them. When external authentication is invoked linking to workflows does not seem to work and copy and pasting from a file is very tedious. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Some changes to tophat and cufflinks wrappers for indexing loc files
1.)Please see attached. 2.)So I did this for tophat and cufflinks. This would be useful for cuffcompare. Cuffdiff uses the gtf files output from cufflinks and not reference gtf files. Following a conversation we had with James, I plan on editing the xml scripts to pull the genome attributes from a previous step as opposed to having to set them. From: Jeremy Goecks [mailto:jeremy.goe...@emory.edu] Sent: Wednesday, October 12, 2011 1:22 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Some changes to tophat and cufflinks wrappers for indexing loc files I made some changes to the cufflinks and tophat wrapper to pull gtf files from an indexed loc file as opposed to from history. The diff's are attached. I made it an optional feature. Please let me know if this makes its way into galaxy-central. Ilya, This looks like a good start. A couple questions: (1) Do you have a sample GTF indices file similar to e.g. bowtie_indices.loc.sample ? (2) Would it be useful to provide GTF index files in Cuffcompare, Cuffdiff, and Tophat as well? Thanks, J. gtf_indices.loc Description: gtf_indices.loc ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] strange history error
I am getting a strange error that I cannot figure out how to debug. It's from a second instance we have running on the same server. When I upload data it does not show me my history. I get the following messages from the log file. Thoughts? galaxy.web.framework DEBUG 2011-10-11 15:47:50,259 Error: this request returned None from get_history(): http://ussd-dev-swim02.illumina.com/galaxy2/history/list galaxy.web.framework DEBUG 2011-10-11 15:47:50,261 Error: this request returned None from get_history(): http://ussd-dev-swim02.illumina.com/galaxy2/history/list galaxy.web.framework DEBUG 2011-10-11 16:34:25,717 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,717 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,718 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,718 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,718 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,718 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,719 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create galaxy.web.framework DEBUG 2011-10-11 16:34:25,719 Error: this request returned None from get_his tory(): http://ussd-dev-swim02.illumina.com/galaxy2/tool_runner/upload_async_create Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable
The egg is there, but the error still shows up. -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Nate Coraor Sent: Friday, September 30, 2011 6:40 AM To: Haden, Kirt Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Haden, Kirt wrote: Hi, When I run Galaxy under Python 2.7, I am unable to run SAM to BAM tools without getting the following error (below). The same configuration with Python 2.6 does not have an issue. The error seems to indicate bx_python eggs are not fetchable. What am I doing wrong? Hey Kirt, Are you running it on a machine with no Internet access? If not, can you run: python ./scripts/fetch_eggs.py With python set to your 2.7 version? Thanks, --nate Error report below: GALAXY TOOL ERROR REPORT This error report was sent from the Galaxy instance hosted on the server ussd-dev-lnx01.illumina.com -- --- This is in reference to dataset id 301 from history id 68 -- --- You should be able to view the history containing the related history item 7: SAM-to-BAM on data 6: converted BAM by logging in as a Galaxy admin user to the Galaxy instance referenced above and pointing your browser to the following link. ussd-dev-lnx01.illumina.com/history/view?id=42e99d98e0ee2844 -- --- The user 'kha...@illumina.com' provided the following information: -- --- job id: 200 tool id: sam_to_bam -- --- job command line: python /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py --input1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_300.dat --dbkey=PhiX --output1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_301.dat --index_dir=/home/galaxy/eval/sept09/galaxy-central/tool-data -- --- job stderr: Traceback (most recent call last): File /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py, line 14, in ? import pkg_resources; pkg_resources.require( bx-python ) File /home/galaxy/eval/sept09/galaxy-central/lib/galaxy/eggs/__init__.py, line 415, in require raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) ) galaxy.eggs.EggNotFetchable: ['bx_python'] -- --- job stdout: -- --- job info: None -- --- job traceback: None -- --- (This is an automated message). Thanks, Kirt Haden Sr. Mgr. Engineering Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858.202.4655 Fax: 858.202.4545 Email: kha...@illumina.commailto:kha...@illumina.com Website: www.illumina.comhttp://www.illumina.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable
Same permissions as the 2.6 egg. -Original Message- From: Daniel Blankenberg [mailto:d...@bx.psu.edu] Sent: Friday, September 30, 2011 8:59 AM To: Chorny, Ilya Cc: Nate Coraor; Haden, Kirt; 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Is this instance using the run job as user patch, and if so, does the user running the job have permission to access the egg? Thanks for using Galaxy, Dan On Sep 30, 2011, at 11:36 AM, Chorny, Ilya wrote: The egg is there, but the error still shows up. -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Nate Coraor Sent: Friday, September 30, 2011 6:40 AM To: Haden, Kirt Cc: 'galaxy-dev@lists.bx.psu.edu' Subject: Re: [galaxy-dev] Python2.7 SAM to BAM Egg Not Fetchable Haden, Kirt wrote: Hi, When I run Galaxy under Python 2.7, I am unable to run SAM to BAM tools without getting the following error (below). The same configuration with Python 2.6 does not have an issue. The error seems to indicate bx_python eggs are not fetchable. What am I doing wrong? Hey Kirt, Are you running it on a machine with no Internet access? If not, can you run: python ./scripts/fetch_eggs.py With python set to your 2.7 version? Thanks, --nate Error report below: GALAXY TOOL ERROR REPORT This error report was sent from the Galaxy instance hosted on the server ussd-dev-lnx01.illumina.com - - --- This is in reference to dataset id 301 from history id 68 - - --- You should be able to view the history containing the related history item 7: SAM-to-BAM on data 6: converted BAM by logging in as a Galaxy admin user to the Galaxy instance referenced above and pointing your browser to the following link. ussd-dev-lnx01.illumina.com/history/view?id=42e99d98e0ee2844 - - --- The user 'kha...@illumina.com' provided the following information: - - --- job id: 200 tool id: sam_to_bam - - --- job command line: python /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py --input1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_300.dat --dbkey=PhiX --output1=/home/galaxy/production/Sept06/galaxy-central/database/files/000/dataset_301.dat --index_dir=/home/galaxy/eval/sept09/galaxy-central/tool-data - - --- job stderr: Traceback (most recent call last): File /home/galaxy/eval/sept09/galaxy-central/tools/samtools/sam_to_bam.py, line 14, in ? import pkg_resources; pkg_resources.require( bx-python ) File /home/galaxy/eval/sept09/galaxy-central/lib/galaxy/eggs/__init__.py , line 415, in require raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) ) galaxy.eggs.EggNotFetchable: ['bx_python'] - - --- job stdout: - - --- job info: None - - --- job traceback: None - - --- (This is an automated message). Thanks, Kirt Haden Sr. Mgr. Engineering Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858.202.4655 Fax: 858.202.4545 Email: kha...@illumina.commailto:kha...@illumina.com Website: www.illumina.comhttp://www.illumina.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using
Re: [galaxy-dev] Some changes to tophat and cufflinks wrappers for indexing loc files
I made some changes to the cufflinks and tophat wrapper to pull gtf files from an indexed loc file as opposed to from history. The diff's are attached. I made it an optional feature. Please let me know if this makes its way into galaxy-central. Best, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com cufflinks_wrapper.xml.diff Description: cufflinks_wrapper.xml.diff tophat_wrapper.xml.diff Description: tophat_wrapper.xml.diff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Additional eggs
Could you add PyX and Matplotlib(http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.0.1/) to the eggs? They would be very useful for creating reports. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] use_tasked_jobs
I found it as well. You just copied the user logic from JobWrapper to TaskWrapper? Sent from my iPhone On Sep 22, 2011, at 8:39 AM, Dannon Baker dannonba...@me.com wrote: The problem was caused by an unimplemented method in the TaskWrapper. I've fixed it in changeset 6026:3f926d934d98. -Dannon On Sep 20, 2011, at 6:58 PM, Chorny, Ilya wrote: Hi Nate, We are having an issue that when use_tasked_jobs = True, the job_wrapper.user = None in drama.py. Do you know why that might be. It crashes my code. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.com Website: www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] use_tasked_jobs
We also had to make some changes to drama.py so the use_tasks_jobs behaves
nicely with drama.
# define job attributes
-ofile = %s/%s.o % (self.app.config.cluster_files_directory,
job_wrapper.job_id)
-efile = %s/%s.e % (self.app.config.cluster_files_directory,
job_wrapper.job_id)
+ofile = %s/%s.o % (self.app.config.cluster_files_directory,
job_wrapper.get_id_tag())
+efile = %s/%s.e % (self.app.config.cluster_files_directory,
job_wrapper.get_id_tag())
jt = self.ds.createJobTemplate()
jt.remoteCommand = %s/database/pbs/galaxy_%s.sh % (os.getcwd(),
job_wrapper.get_id_tag())
jt.outputPath = :%s % ofile
@@ -424,7 +425,7 @@
Stores the content of a DRMAA JobTemplate object in a file as a
JSON string.
Path is hard-coded, but it's no worse than other path in this module.
Uses Galaxy's JobID, so file is expected to be unique.
-filename = %s/database/pbs/%s.jt_json % (os.getcwd(),
job_wrapper.job_id)
+filename = %s/database/pbs/%s.jt_json % (os.getcwd(),
job_wrapper.get_id_tag())
data = {}
for attr in DRMAA_jobTemplate_attributes:
try:
-Original Message-
From: Dannon Baker [mailto:dannonba...@me.com]
Sent: Friday, September 23, 2011 4:18 AM
To: Chorny, Ilya
Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] use_tasked_jobs
TaskWrapper should inherit it from JobWrapper, so I just removed the
(unimplemented, other than the pass) overriding TaskWrapper.user property that
was preventing that.
On Sep 23, 2011, at 7:11 AM, Chorny, Ilya wrote:
I found it as well. You just copied the user logic from JobWrapper to
TaskWrapper?
Sent from my iPhone
On Sep 22, 2011, at 8:39 AM, Dannon Baker dannonba...@me.com wrote:
The problem was caused by an unimplemented method in the TaskWrapper. I've
fixed it in changeset 6026:3f926d934d98.
-Dannon
On Sep 20, 2011, at 6:58 PM, Chorny, Ilya wrote:
Hi Nate,
We are having an issue that when use_tasked_jobs = True, the
job_wrapper.user = None in drama.py. Do you know why that might be. It
crashes my code.
Thanks,
Ilya
Ilya Chorny Ph.D.
Bioinformatics Scientist I
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Work: 858.202.4582
Email: icho...@illumina.com
Website: www.illumina.com
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this and other
Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] use_tasked_jobs and drmaa
So we have some new issues with use_tasked_jobs and drama and wanted to share with the community. 1.)The task get split on the head node which will bog down the head node if multiple user submit a job which requires splitting. It would be great if we could submit that to drmaa. 2.)Can't kill the job while it's being split. 3.)Can't kill the job when it finally gets submitted to the drama. (Looks like there needs to be some logic to assign multiple drama job id's with the galalxy job ID). Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Passing the number of mpi processors to SGE from xml file
It's easy to hard code in the ini but I want to be able to set the number of processors as a parameter. From: Glen Beane [mailto:glen.be...@jax.org] Sent: Wednesday, September 14, 2011 10:34 AM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Passing the number of mpi processors to SGE from xml file On Sep 14, 2011, at 11:30 AM, Chorny, Ilya wrote: Nate, Any thoughts on how to do this? Can I add an xml tag that gets parsed and set's -PE mpi $NSLOTS in the job template. Can you point me to where the xml get's parsed? Thanks, Ilya I'm not that familiar with SGE, but this would be pretty trivial with TORQUE assuming a fixed number of MPI processes for the tool. No modifications to the job template would be necessary: tool specific job runner specifies number of nodes/ppn or procs; mpiexec (built with TM support) launches appropriate number of processes on all nodes assigned to job by TORQUE. From: Glen Beane [mailto:glen.be...@jax.org]mailto:[mailto:glen.be...@jax.org] Sent: Tuesday, September 13, 2011 5:00 PM To: Chorny, Ilya; galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: RE: [galaxy-dev] Passing the number of mpi processors to SGE from xml file I think with the current state of Galaxy this would need to be fixed. You would need to hard code the number of processors in the xml wrapper and in the corresponding job runner for that tool From: galaxy-dev-boun...@lists.bx.psu.edumailto:galaxy-dev-boun...@lists.bx.psu.edu [galaxy-dev-boun...@lists.bx.psu.edu] on behalf of Chorny, Ilya [icho...@illumina.com] Sent: Tuesday, September 13, 2011 7:58 PM To: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Passing the number of mpi processors to SGE from xml file Any thoughts on how to pass the number of processor I want to use from my xml wrapper to the job template. I am trying to wrap pBWA and use mpi. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Passing the number of mpi processors to SGE from xml file
Any thoughts on how to pass the number of processor I want to use from my xml wrapper to the job template. I am trying to wrap pBWA and use mpi. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] cuffcompare wrapper
It's not an issue in the tmp dir but in the job_working_directory. I run most of those other tools with no problems. I don't think we should make it a requirement across the board and I think we can come up with alternative ways to clean up the job_working_directory. I am hoping that you could add the symlink to the cuffcompare wrapper as it is the only one where the symlink causes me a problem as far as I have tested. We don't want to have our code base differ to much from galaxy-central. Thanks, Ilya From: Jeremy Goecks [mailto:jeremy.goe...@emory.edu] Sent: Wednesday, September 07, 2011 6:26 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] cuffcompare wrapper Ilya, A search of the Galaxy codebase indicates that thirteen tools use symlinks (e.g. GATK, Sicer, Picard, Cuff*, Bowtie), so the changes required to support this new code are significant. (Changes would also likely be needed for tools in the tool shed.) Also, asking tool wrappers to delete symlinks would be an idiosyncratic requirement as tools assume they have a temporary working directory at their disposal. For these reasons, it seems best to have the tool framework clean up symlinks as necessary to support the new code. Best, J. On Sep 7, 2011, at 2:28 PM, Chorny, Ilya wrote: Ok, I figured out why you need the symlink. Can you add an unlink after the process completes? i.e for i, arg in enumerate( args ): input_file_name = ./input%i % ( i+1 ) os.unlink(input_file_name) From: galaxy-dev-boun...@lists.bx.psu.edumailto:galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu]mailto:[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Wednesday, September 07, 2011 9:18 AM To: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] cuffcompare wrapper Hi Jeremy, The symlink in the cuffcompare wrapper was causing galaxy to crash because I run as the actual user and have to chmod the job_working directory at the end so Galaxy can clean it up. Turns out is seems like the symlink is not needed. Am I missing something. See below. Your code: for i, arg in enumerate( args ): input_file_name = ./input%i % ( i+1 ) os.symlink( arg, input_file_name ) cmd += %s % input_file_name My code: for i, arg in enumerate( args ): cmd += arg Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] macs in galaxy
Is there a python script associated with the macs.xml file? From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Kanwei Li Sent: Wednesday, August 24, 2011 5:41 PM To: KOH Jia Yu Jayce Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] macs in galaxy You can use the MACS wrappers here (for 1.4): https://bitbucket.org/cistrome/cistrome-harvard/src/779d208c2cbd/tools/peakcalling/ Until we officially add it to our distribution. Thanks, K On Wed, Aug 24, 2011 at 8:32 PM, KOH Jia Yu Jayce ko...@gis.a-star.edu.sgmailto:ko...@gis.a-star.edu.sg wrote: Yes. Thank you for your help ☺ -Original Message- From: Kanwei Li [mailto:kan...@gmail.commailto:kan...@gmail.com] Sent: Wednesday, August 24, 2011 11:28 PM To: KOH Jia Yu Jayce Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] macs in galaxy Hi Jayce, Are you running this on your local instance? It seems you are running MACS 1.4, which our wrapper does not support yet, but we are planning to add a wrapper for 1.4 soon. Thanks, K On Wed, Aug 24, 2011 at 4:20 AM, KOH Jia Yu Jayce ko...@gis.a-star.edu.sgmailto:ko...@gis.a-star.edu.sg wrote: In running macs in galaxy, the following error was found ERROR:root:mfold format error! Your input is '32'. It should be like '10,30' A format for mfold like 10,30 is expected… but the default value configured in xml remains as 32. will there be an updated version of this xml in future? Also after altering the default display mfold value to 10,30, type integer in the param tag for mfold become erroneous. May I ask what is the correct type for input format 10,30? Thanks alot ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] cuffcompare wrapper
Hi Jeremy, The symlink in the cuffcompare wrapper was causing galaxy to crash because I run as the actual user and have to chmod the job_working directory at the end so Galaxy can clean it up. Turns out is seems like the symlink is not needed. Am I missing something. See below. Your code: for i, arg in enumerate( args ): input_file_name = ./input%i % ( i+1 ) os.symlink( arg, input_file_name ) cmd += %s % input_file_name My code: for i, arg in enumerate( args ): cmd += arg Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] GATK Unified Genotyper
Hi Daniel,
I have been implementing the GATK unified genotyper and I was having some
issues. Occasionally I would get an error that the Java virtual machine would
not start. I got around that by adding the Xmx3g parameter to the command line.
I also added the -nt 3 because otherwise it's really slow on human data.
--stdout ${output_log}
#for $i, $input_bam in enumerate( $reference_source.input_bams ):
-d -I ${input_bam.input_bam} ${input_bam.input_bam.ext}
gatk_input_${i}
-d ${input_bam.input_bam.metadata.bam_index} bam_index
gatk_input_${i} ##hardcode galaxy ext type as bam_index
#end for
-p 'java -Xmx3g
-jar ${GALAXY_DATA_INDEX_DIR}/shared/jars/gatk/GenomeAnalysisTK.jar
-T UnifiedGenotyper
-o ${output_vcf}
##-o out_vcf.txt
-et NO_ET ##ET no phone home
##-log ${output_log} ##don't use this to log to file, instead directly
capture stdout
#if $reference_source.reference_source_selector != history:
-R ${reference_source.ref_file.fields.path}
#end if
-nt 3
--standard_min_confidence_threshold_for_calling
${standard_min_confidence_threshold_for_calling}
--standard_min_confidence_threshold_for_emitting
${standard_min_confidence_threshold_for_emitting}
Ilya Chorny Ph.D.
Bioinformatics Scientist I
Illumina, Inc.
9885 Towne Centre Drive
San Diego, CA 92121
Work: 858.202.4582
Email: icho...@illumina.commailto:icho...@illumina.com
Website: www.illumina.comhttp://www.illumina.com
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] cuffcompare wrapper
Ok, I figured out why you need the symlink. Can you add an unlink after the process completes? i.e for i, arg in enumerate( args ): input_file_name = ./input%i % ( i+1 ) os.unlink(input_file_name) From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Wednesday, September 07, 2011 9:18 AM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] cuffcompare wrapper Hi Jeremy, The symlink in the cuffcompare wrapper was causing galaxy to crash because I run as the actual user and have to chmod the job_working directory at the end so Galaxy can clean it up. Turns out is seems like the symlink is not needed. Am I missing something. See below. Your code: for i, arg in enumerate( args ): input_file_name = ./input%i % ( i+1 ) os.symlink( arg, input_file_name ) cmd += %s % input_file_name My code: for i, arg in enumerate( args ): cmd += arg Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] TEMP variable in cluster install
Hi Nate, I wrote the code such that if a machine had a large local scratch space then you could comment it out and it would go to the default setting of writing to /tmp or whatever is defined by SGE. You are correct that the environmental variables defined in the Galaxy startup script do not get passed when running as an actual user. Further I separated TMPDIR from new_files_path because new_files_path is used by Galaxy for other reasons such as uploading files and metadata. Best, Ilya -Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Tuesday, September 06, 2011 7:21 AM To: Chorny, Ilya Cc: Assaf Gordon; Shantanu Pavgi; galaxydev psu Subject: Re: [galaxy-dev] TEMP variable in cluster install Chorny, Ilya wrote: Nate, We ran into this issue with /tmp not having enough space. We came up with a generic solution and were wondering if you might want to add it to galaxy-central. We did not want to modify SGE as it could affect other jobs that use the cluster. We created a TMPDIR option in universe_wsgi.ini and then modified drama.py to include an export TMPDIR=TMPDIR to the drm_tmplate if TMPDIR is defined in universe_wsgi.ini. Do you want to pull this code into galaxy central. It's like 5 lines of code. This is also something you can put in the Galaxy user's ~/.sge_request, although with your actual user code this would be more difficult. There's actually already a temp directory option, new_file_path, this would just need to be added to the template. But I don't know that it's a good idea, since each node may have a large local scratch space and the path to that space may differ (this is the case in our environment, where we actually run jobs on multiple clusters). Thanks and have a good weekend, Ilya BTW, any luck reviewing the run as actual user code? I should be doing it today. --nate -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Assaf Gordon Sent: Thursday, August 04, 2011 2:31 PM To: Shantanu Pavgi; galaxydev psu Subject: Re: [galaxy-dev] TEMP variable in cluster install As a follow-up for SGE + TMP variables, I've also encountered problems with settings TEMP,TMP and TMPDIR variables. Tried setting ~/.sge_request and changing the Galaxy environment variables before starting python - nothing worked - TMPDIR and TMP were always set to /tmp/all.q.X by SGE . What finally worked is changing the SGE configuration queue, and simply setting the tmpdir variable to my desired temp directory. Do that by running: $ sudo qconf -mq all.q and changing the line: tmpdir/tmp to tmpdir/my/temporary/directory/path Problem solved :) No more messing around with TMP,TMPDIR variables in any Galaxy related source files. Hope this helps, -gordon Shantanu Pavgi wrote, On 07/26/2011 05:40 PM: On Jul 21, 2011, at 6:24 PM, Shantanu Pavgi wrote: We have configured galaxy to work with our SGE cluster using drmaa job runner interface. We are using 'unified method' for this install and both TEMP environment variable and new_file_path in universe_wsgi.ini file have been configured correctly. However, we are seeing some errors where local /tmp space on compute nodes is being referenced by the galaxy tools. Specifically we saw it mentioned in error messages from following tools: * bwa_wrapper and upload tools: 'No space left on device: /tmp...' * sam_to_bam tool: 'Error sorting alignments from /tmp/filename..' Shouldn't it be referencing TEMP environment variable or new_file_path configuration value? Is it getting overridden by TMP or TMPDIR variables in python wrapper code? Has anyone else experienced similar issue? Further debugging showed that there are two more temporary directory related environment variables - TMP and TMPDIR which were pointing local /tmp location on compute nodes. We tried to set these variables in our shell environment (drmaa URL uses -V to export current shell env) however SGE overwrote TMP and TMPDIR before actual job execution. The TEMP variable remained unmodified by SGE scheduler. The galaxy tools seemed to be using temporary directory space pointed by TMP and TMPDIR and hence we saw local /tmp related errors mentioned in my earlier post. We have temporarily fixed this problem by hard coding TEMP, TMP and TMPDIR values in job template files - 1. lib/galaxy/jobs/runners/drmaa.py 2. lib/galaxy/jobs/runners/sge.py This will affect all jobs submitted through the galaxy. It will be helpful to know how it can be handled in the tool wrapper scripts. It seems like it can be handled in the tool wrapper scripts using directory prefix while creating the tempfile ( http://docs.python.org/library/tempfile.html ). Any thoughts/comments? Are there any other SGE
[galaxy-dev] Tools requiring java can't find java
This is probably a question for our sys admin but when I run tools such as GATK I get an error java command not found. I can execute java from the command line and it works. Further I can grab the drmaa submission script (ie galaxy_job_id.sh) and qsub it from the command line and it finds JAVA. Any thoughts? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tools requiring java can't find java
How do I set my path in a non login shell? From: George Marselis [mailto:george.marse...@kaust.edu.sa] Sent: Tuesday, August 30, 2011 9:40 AM To: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Tools requiring java can't find java Yes, your java path is not set, because you are using an non-login shell. Indeed, you should be talking to your sysadmins. George Marselis, systems administrator Building #2, Level 4, room 4327 Computational Bioscience Research Center, KAUST Land: +966-2-808-2944, Mobile: +966-56-321-7713, Skype: project2501a From: Chorny, Ilya icho...@illumina.commailto:icho...@illumina.com Date: Tue, 30 Aug 2011 19:23:42 +0300 To: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Tools requiring java can't find java This is probably a question for our sys admin but when I run tools such as GATK I get an error java command not found. I can execute java from the command line and it works. Further I can grab the drmaa submission script (ie galaxy_job_id.sh) and qsub it from the command line and it finds JAVA. Any thoughts? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Updated Cufflinks Wrapper
I just remembered what I did for the tophat wrapper. All I did was change the default behavior of the use coverage search option. I believe no is the default behavior and you have it as yes. From: Jeremy Goecks [mailto:jeremy.goe...@emory.edu] Sent: Thursday, August 11, 2011 7:22 AM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Updated Cufflinks Wrapper Hi Ilya, Thanks for the contribution; most of it has been committed in changeset 0ef81af1de09 A couple notes: *I couldn't transplant your changes because your commit did not include recent changes to the Cufflinks wrapper. It appears you didn't merge correctly with the main repository at some point; please try to merge correctly as it speeds up the inclusion process significantly on our end. *I didn't include your changes to the Tophat wrapper because merging was incorrect and I couldn't figure out what you were trying to do. Let us know if you have any questions. Best, J. On Jul 27, 2011, at 7:26 PM, Chorny, Ilya wrote: I made some changes to the cufflinks wrapper. I added the -g/--GTF-guide use reference transcript annotation to guide assembly as an option. The version number needs to still be updated in my commit. https://bitbucket.org/ichorny/galaxy-central/changeset/fc5e24d453e5 Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] upload multiple files
I noticed that when uploading multiple files only one job gets submitted to the queue. Is there a way for galaxy to split that up on multiple nodes? Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] database/files/_metadata_files
Great, thanks!! From: Daniel Blankenberg [mailto:d...@bx.psu.edu] Sent: Tuesday, August 23, 2011 1:26 PM To: Chorny, Ilya Subject: Re: [galaxy-dev] database/files/_metadata_files Run a tool (or auto-detect metadata) that creates any BAM or MAF output file. The file will originally be created in a temporary location and then moved into the final location in Galaxy's job finish methods. On Aug 23, 2011, at 4:20 PM, Chorny, Ilya wrote: I am setting metadata externally so how do I get a file to be generated in that directory? From: Daniel Blankenberg [mailto:d...@bx.psu.edu]mailto:[mailto:d...@bx.psu.edu] Sent: Tuesday, August 23, 2011 1:18 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] database/files/_metadata_files Hi Ilya, Tools should not ordinarily write to this directory, but will often read from it. These files are generated by Galaxy's Metadata functionality. If you are setting metadata externally (settings in universe_wsgi.ini) and using a cluster, then this will occur as part of the tool/job script, as a separate entry (calling set_metadata.sh) in the e.g. .sh file submitted to pbs. Certain filetypes, e.g. BAM and MAF, use these for storing index data. Thanks for using Galaxy, Dan On Aug 23, 2011, at 4:04 PM, Chorny, Ilya wrote: Can someone point me to a tool which writes to this directory? I made some changes to the code to run as actual user but cannot find a tools that writes to this directory. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] What script updates the umask of database/files/* and deletes database/job_working_directory
When running as the actual user in drama, I made it so the .dat files are not initially created so that when a tool moves data from the a temp directory to the out .dat file it will not fail. (i.e. unix does not allow moving a file owned by one user to a file owned by another user because mv carries the attributes of the original file). Now I want to change the ownership of the files through a sudo call at the end. Which script is run at the end to clean things up? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] What script updates the umask of database/files/* and deletes database/job_working_directory
So I figured out that it is job/runners/__init__.py that does the cleanup. But now I can't figure out how to pass the galaxy username to my script to change the permission back. I thought os.getlogin() would give me the right username but it returns my user name and not the galaxy user. Thoughts? Thanks, Ilya From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Tuesday, August 23, 2011 1:46 PM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] What script updates the umask of database/files/* and deletes database/job_working_directory When running as the actual user in drama, I made it so the .dat files are not initially created so that when a tool moves data from the a temp directory to the out .dat file it will not fail. (i.e. unix does not allow moving a file owned by one user to a file owned by another user because mv carries the attributes of the original file). Now I want to change the ownership of the files through a sudo call at the end. Which script is run at the end to clean things up? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Metadata error
I get these two error in the log file. The job finishes correctly and a metadata file is created but I still get the error. Can anyone provide me with more detail about what this error means? The second error is incorrect because the files were deleted. setting metadata externally failed for HistoryDatasetAssociation 344: [Errno 1] Operation not permitted: '/mnt/data/galaxy/database/files/000' and Failed to cleanup MetadataTempFile temp files from ../../mnt/data/galaxy/database/tmp/metadata_out_HistoryDatasetAssociation_344_NUDu3A: No JSON object could be decoded: line 1 column 0 (char 0) Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.commailto:icho...@illumina.com Website: www.illumina.comhttp://www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] When writing wrapper do not use shutil.copy()
If possible when writing wrappers do not use shutil.copy() when copying a file to a dataset_(job_id).dat file. shutil.copy() tries to change the permission of the copied to file. Better to use shutil.copyfile(). Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] upload.py fails with files having commas in file name
The ftp upload module fails when a file has a comma in the file name for example test.bam works but when copied to test,test.bam that file fails. Cheers, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] collect_associated_files and umask
When I run galaxy as the actual user using the code I committed to my fork, I run into a problem with dataset_*.dat files that have associated data wherein the associated data files are copied from the job_working_directory into the files directory. That directory is owned by the actual user and not by the galaxy user. I don't have a problem with the ownership but the permissions of the directory and associated files get changed to 777. Any thoughts on why the permissions get changed. The permissions are 755 and 644 for dirs and files respectively when in the working directory so why do they change when the directory is moved? Any help would be greatly appreciated. Best, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] dataset_*_file directory being compied to file_path when running fastqc
I am testing the latest galaxy central distribution by running FASTQC. It creates a directory call dataset_*_file in the job_working_directory and then copies that directory to file_path when the job is complete. Is that on purpose or is that a bug? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] dataset_*_file directory being compied to file_path when running fastqc
Thanks, that makes sense. Ilya -Original Message- From: Ross [mailto:ross.laza...@gmail.com] Sent: Thursday, July 28, 2011 10:14 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] dataset_*_file directory being compied to file_path when running fastqc It's a feature although arguably necessitated by what some might consider a design bug in FastQC FastQC insists on writing the html report with links to the generated images which it writes to a subdirectory. That won't work for Galaxy's html datatype, so the wrapper script unpacks that structure into a Galaxy html datatype and adjusts the links in the html so they work correctly when viewed in a history. On Fri, Jul 29, 2011 at 2:47 PM, Chorny, Ilya icho...@illumina.com wrote: I am testing the latest galaxy central distribution by running FASTQC. It creates a directory call dataset_*_file in the job_working_directory and then copies that directory to file_path when the job is complete. Is that on purpose or is that a bug? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Director of Bioinformatics, Channing Lab; Tel: +1 617 505 4850; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444; ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] set_metadata_externally =True
Well I got all the rw permissions changed correctly except for the metadata files written to the temp directory (new_file_path). I still can't seem to track down where those are written. Any help would be greatly appreciated. Best, Ilya From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, July 25, 2011 4:29 PM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] set_matadata_externally =True I am still working on the patch to submit jobs as the actual user under SGE . I have it working for the default case but it fails for the case of set_metadata_externally = TRUE. Can someone point me to where the metadata files are created so that I can change the permissions for those files? Also, I have to addusers to the galaxy group in order for this to work because the galaxy_*.sh scripts are created as executable only by members of the galaxy group. I want to change this so that all users can execute these scripts. Any help on where in the code these scripts are created? Upon completion I will create a galaxy central fork and upload my changes to it. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Regarding submitting jobs as actual user
I completed the changes to galaxy to submit DRMAA jobs (SGE) as the actual galaxy user. You can see the code at the fork below. Below is a brief description of how to turn it which is also described in the universe_wsgi.ini.sample. https://icho...@bitbucket.org/ichorny/galaxy-central # Uncomment drmaa_external_runjob_script and drmaa_external_killjob_script pameters and have them point to the # absolute path for scripts/drmaa_external_runner.py and scripts/drmaa_external_killer.py. # The scripts directory is located in the top level galaxy directory. The parameters when # uncommented allow for submission to the drmaa queue with the user name of the user submitting # the job and not the galaxy user. In order for this to work the actual user must log into galaxy # and the galaxy authentication must be consistent with the authentication on the server in which the # drmaa queue is running (i.e. the username must have an account on the server and be allowed to # submit jobs to the queue). The galaxy user must also be given sudo permission to execute # scripts/drmaa_external_runner.py and scripts/drmaa_external_killer.py in /etc/sudoers # Example: # galaxy ALL = (root) NOPASSWD: SETENV: /opt/galaxy/scripts/drmaa_external_runner.py # galaxy ALL = (root) NOPASSWD: SETENV: /opt/galaxy/scripts/drmaa_external_killer.py # Also the # Defaultsrequiretty # in /etc/sudoers must be commented out #drmaa_external_runjob_script = None #drmaa_external_killjob_script = None Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] set_matadata_externally =True
I am still working on the patch to submit jobs as the actual user under SGE . I have it working for the default case but it fails for the case of set_metadata_externally = TRUE. Can someone point me to where the metadata files are created so that I can change the permissions for those files? Also, I have to addusers to the galaxy group in order for this to work because the galaxy_*.sh scripts are created as executable only by members of the galaxy group. I want to change this so that all users can execute these scripts. Any help on where in the code these scripts are created? Upon completion I will create a galaxy central fork and upload my changes to it. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Scroll bars not present using i.e.
http://test.g2.bx.psu.edu/ Cheers, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Newer Version of Cufflinks Wrapper
I just hit reply. Would be easier if when I hit reply it automatically sent the email back to the list. That's how other mailing lists that.I'm on work. Thanks for the hg info. Thanks, Ilya Sent from my iPhone On Jul 22, 2011, at 6:32 AM, Jeremy Goecks jeremy.goe...@emory.edumailto:jeremy.goe...@emory.edu wrote: Ilya, A patch is fine for a one-time fix; this is easy with Mercurial: % hg diff my_patch.patch However, if you plan to contribute to Galaxy on a regular basis, we encourage you to fork Galaxy and maintain your own Bitbucket repository. Finally, please send all emails to our mailing lists rather than individual Galaxy developers or other list members. Sending email to the mailing lists ensures that everyone can see and contribute to questions and issues; in addition, emails to mailing lists are tracked and archived so that they are available in the future. Thanks, J. On Jul 21, 2011, at 5:16 PM, Chorny, Ilya wrote: I would be happy to share any changes that I make. By patch do you mean a diff/patch share patch or just check in the code? From: Jeremy Goecks [mailto:jeremy.goe...@emory.edu] Sent: Wednesday, July 20, 2011 4:55 PM To: Chorny, Ilya Cc: mailto:galaxy-user-boun...@lists.bx.psu.edu galaxy-user-boun...@lists.bx.psu.edumailto:galaxy-user-boun...@lists.bx.psu.edu; mailto:galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Newer Version of Cufflinks Wrapper Ilya, Despite the wrapper version numbers (which I'll update shortly to avoid further confusion), the current Cufflinks wrapper in both galaxy-central and galaxy-dist supports Cufflinks v1.0.3 However, not all new options for v1.0.3 are implemented the current wrappers; should you decide to implement additional options in the Cufflinks/compare/diff wrappers, please consider submitting a patch that we could incorporate into Galaxy. Best, J. On Jul 21, 2011, at 1:30 AM, Chorny, Ilya wrote: Has anyone produced a newer version of the cufflinks wrapper? The current wrapper is for version 0.9.1. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics – Intern mailto:icho...@illumina.comicho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Newer Version of Cufflinks Wrapper
Has anyone produced a newer version of the cufflinks wrapper? The current wrapper is for version 0.9.1. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] TopHat V1.1 with GTF files
I figured it out. Please disregard my last email. Thanks, Ilya From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, July 11, 2011 10:00 AM To: galaxy-dev@lists.bx.psu.edu; galaxy-user-boun...@lists.bx.psu.edu Subject: [galaxy-dev] TopHat V1.1 with GTF files I was wondering how to pass a GTF file to the current Tophat implementation in Galaxy. It says that -G/--GTF is an implemented parameter but that option does not show up. BTW, what version of tophat is supplied with the most recent stable release. Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Setting up Galaxy with SGE
Ravi, Do your galaxy jobs submit to the cluster as the galaxy user or as the user? Thanks, Ilya Sent from my iPhone On Jun 15, 2011, at 2:43 PM, Ravi Madduri madd...@mcs.anl.govmailto:madd...@mcs.anl.gov wrote: Hi We have been able to get the following setup working pretty reliably: * Automated deployment of Galaxy clusters on EC2. These clusters have the following setup: * A NFS server providing global directories for home directories, software, and scratch space. We are also experimenting with glusterfs * A NIS server providing an authentication domain within the cluster. The list of users must be specified when the cluster is created * A GridFTP server, which is automatically registered as a Globus Online (http://www.globusonline.orgwww.globusonline.orghttp://www.globusonline.org) endpoint for high speed and reliable data transfer. * Ability to script transfer tasks as part of the workflow using Globus Online Galaxy tools (To get large quantities of data in and out of galaxy EC2 cluster reliably) * A Condor pool with jobs running using user's credentials (using users accounts that are provisioned in the EC2 cluster) * A Galaxy server with the modifications outlined above, which we plan to contribute back to the galaxy community soon I gave a talk at the recent Galaxy users conference about this setup. You can find slides here: PowerPointhttp://wiki.g2.bx.psu.edu/GCC2011?action=AttachFiledo=gettarget=GalaxyGridFTPCondorAndGlobusOnline.pptx Please let me know if you are interested and would like more information. Regards On Jun 15, 2011, at 5:28 AM, Marina Gourtovaia wrote: There are two slightly different problems here. One with the files ystem and the second is whether the machine Galaxy is running on can submit jobs to the cluster (ssh is mentioned in the original e-mail, suggesting that it's impossible to communicate with the cluster from the machine the job is running on). The Galaxy instance is constantly communicating with the cluster job scheduling system (SGE or other) in order to get updates on the status of the jobs. It should be possible to do it over ssh, but, in my experience, this slows down the code. We run the Galaxy server on one of the cluster nodes. I imagine that other people using Galaxy with the cluster do the same. Are they? Marina On 15/06/2011 08:51, Peter Cock wrote: On Wed, Jun 15, 2011 at 2:30 AM, Ka Ming Nipmailto:km...@bcgsc.cakm...@bcgsc.camailto:km...@bcgsc.ca wrote: Hello, I have been trying to set up Galaxy to run its jobs on my SGE cluster using the Unified Method, based on the steps described in the wiki: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/Clusterhttps://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster My local instance of Galaxy is installed under my home directory, but my cluster is on a different file system. I need to ssh to the head node before I can submit jobs. Is there any way to set up Galaxy with a SGE cluster on a different file system? I haven't looked into it in great detail, but we currently have a similar network setup, and would also like to link Galaxy to SGE. The main problem is the lack of a shared file system - in order to run a job on the cluster we (Galaxy) has to get the data onto the cluster, run the job, and get the results back. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ http://lists.bx.psu.edu/ -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/http://lists.bx.psu.edu/ -- Ravi K Madduri The Globus Alliance | Argonne National Laboratory | University of Chicago http://www.mcs.anl.gov/~maddurihttp://www.mcs.anl.gov/~madduri ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Passing username to drmaa.py
Ok, so I figured out how to get the username in drama.py and pass it along to SGE(in our case) using the Gordon patch. Now I am getting an error that Job output not returned from cluster which comes from the finish job sub routine in drama.py . I am assuming this is because the .dat file are owned by the galaxy user and do not have write permission by other users. Where are these files created? I am hoping to add a system call to chmod to see if this fixes the problem. Thanks, Ilya -Original Message- From: Nate Coraor [mailto:n...@bx.psu.edu] Sent: Thursday, June 09, 2011 9:47 AM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Passing username to drmaa.py Chorny, Ilya wrote: I am trying to figure out how to pass the current user name to drama.py. I have had little success so far. None of the environmental variables contain any current user information. Any thoughts? Thanks, Ilya Hi Ilya, You can use $__user_email__ in the command template in the tool's configuration, if you need it there. In the drmaa.py code itself, you can use the '.user' property on the job wrapper to get the email address. --nate Ilya Chorny Ph.D. Bioinformatics - Intern icho...@illumina.commailto:icho...@illumina.com 858-202-4582 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
