Re: [galaxy-dev] Looking for recommendations: How to run galaxy workflows in batch
Hi Dev-Team, are you planning or maybe working on an update which enables the possiblilty to run workflows in batch mode with paired end data? Cheers, Sascha ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Fwd: Galaxy crach while visualization
-- Forwarded message -- From: Chebbi Mohamed Amine chebbimam...@gmail.com Date: 2012/7/18 Subject: Re: [galaxy-dev] Galaxy crach while visualization To: Jeremy Goecks jeremy.goe...@emory.edu Hello Jeremy! Thanks for your response. Yes the paster.log is not complaining. Excuse me where I can found the javascript log. In /var/log I haven't any file related to javascript.log. Thanks Amine 2012/7/17 Jeremy Goecks jeremy.goe...@emory.edu Hello Amine, The paster log looks normal, and Postgre log is standard given the crash. Was there anything out of the ordinary that you can spot in the paster log and/or Javascript log in your Web browser? Thanks, J. On Jul 17, 2012, at 6:53 AM, Chebbi Mohamed Amine wrote: Hello Galaxy team ! I have e strange problem whith my Galaxy trackster visualization . While viewing bookmarked regions in my visualisation the whole Galaxy process crashed and needed to be restarted, mainly when I use Chrome browser. Above the Paster.log : xxx.xxx.xx.xxx, xxx.xx.xx.x - - [12/Jul/2012:12:56:36 +0200] GET /galaxy/tracks/data?chrom=chr19low=1183972high=1185154mode=Autoresolution=2.954022988505747dataset_id=f4b17be90d9b3c9dhda_ldda=hdafilter_cols=%5B%22Score%22%5D HTTP/1.1 200 - http://xxx.xxx.com/galaxy/u/amine/v/rna-seq-analysis-tophats-output; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11 xxx.xxx.xx.xxx, xxx.xx.xx.x - - [12/Jul/2012:12:56:36 +0200] GET /galaxy/tracks/data?chrom=chr19low=1182790high=1183972mode=Autoresolution=2.954022988505747dataset_id=ec90add676cb0ccehda_ldda=hdafilter_cols=%5B%22Mapping+Quality%22%5D HTTP/1.1 200 - http://xxx.xxx.com/galaxy/u/amine/v/rna-seq-analysis-tophats-output; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11 xxx.xxx.xx.xxx, xxx.xx.xx.x - - [12/Jul/2012:12:56:36 +0200] GET /galaxy/tracks/data?chrom=chr19low=1181609high=1182791mode=Autoresolution=2.954022988505747dataset_id=ec90add676cb0ccehda_ldda=hdafilter_cols=%5B%22Mapping+Quality%22%5D HTTP/1.1 200 - http://xxx.xxx.com/galaxy/u/amine/v/rna-seq-analysis-tophats-output; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11 xxx.xxx.xx.xxx, xxx.xx.xx.x - - [12/Jul/2012:12:56:36 +0200] GET /galaxy/tracks/data?chrom=chr19low=1183972high=1185154mode=Autoresolution=2.954022988505747dataset_id=ec90add676cb0ccehda_ldda=hdafilter_cols=%5B%22Mapping+Quality%22%5D HTTP/1.1 200 - http://xxx.xxx.com/galaxy/u/amine/v/rna-seq-analysis-tophats-output; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11 xxx.xxx.xx.xxx, xxx.xx.xx.x - - [12/Jul/2012:12:56:36 +0200] GET /galaxy/tracks/data?chrom=chr19low=1180427high=1181609mode=Autoresolution=2.954022988505747dataset_id=cc26e7b927e75a3dhda_ldda=hdafilter_cols=%5B%22Score%22%5D HTTP/1.1 200 - http://xxx.xxx.com/galaxy/u/amine/v/rna-seq-analysis-tophats-output; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11 xxx.xxx.xx.xxx, xxx.xx.xx.x - - [12/Jul/2012:12:56:36 +0200] GET /galaxy/tracks/data?chrom=chr19low=1181609high=1182791mode=Autoresolution=2.954022988505747dataset_id=cc26e7b927e75a3dhda_ldda=hdafilter_cols=%5B%22Score%22%5D HTTP/1.1 200 - http://xxx.xxx.com/galaxy/u/amine/v/rna-seq-analysis-tophats-output; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11 xxx.xxx.xx.xxx, xxx.xx.xx.x - - [12/Jul/2012:12:56:36 +0200] GET /galaxy/tracks/data?chrom=chr19low=1182790high=1183972mode=Autoresolution=2.954022988505747dataset_id=cc26e7b927e75a3dhda_ldda=hdafilter_cols=%5B%22Score%22%5D HTTP/1.1 200 - http://xxx.xxx.com/galaxy/u/amine/v/rna-seq-analysis-tophats-output; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11 xxx.xxx.xx.xxx, xxx.xx.xx.x - - [12/Jul/2012:12:56:37 +0200] GET /galaxy/tracks/data?chrom=chr19low=814128high=815310mode=Autoresolution=2.954022988505747dataset_id=7710e7c5028b7c3bhda_ldda=hdafilter_cols=%5B%22Score%22%5D HTTP/1.1 200 - http://xxx.xxx.com/galaxy/u/amine/v/rna-seq-analysis-tophats-output; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11 PostgreSQL saw the crach : 2012-07-12 12:56:40 CEST LOG: could not receive data from client: Connection reset by peer 2012-07-12 12:56:40 CEST LOG: unexpected EOF on client connection 2012-07-12 12:56:40 CEST LOG: unexpected EOF on client connection 2012-07-12 12:56:40 CEST LOG: disconnection: session time: 0:00:46.907 user=user database=database host=server port=33106 2012-07-12 12:56:40 CEST LOG: unexpected EOF on client connection 2012-07-12 12:56:40
Re: [galaxy-dev] Looking for recommendations: How to run galaxy workflows in batch
Hi, In case you'd be interested: we use a script that creates interlaced data of paired end data. We run this outside galaxy on groups of samples ordered in directories at once. We then import the interlaced data into galaxy, enabling batch workflow. The first step of the workflow is a deinterlacing of the datafiles. The script is available here: http://geertvandeweyer.zymichost.com/index.php?page=readid=27 Best Regards, Geert On 07/18/2012 02:38 PM, Sascha Kastens wrote: Hi Dev-Team, are you planning or maybe working on an update which enables the possiblilty to run workflows in batch mode with paired end data? Cheers, Sascha ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] re-writing username for login or stripping domain from remote_user?
On Jul 16, 2012, at 6:42 PM, Smithies, Russell wrote: I have a situation I’m sure others have faced but I can’t see how to solve it without hacking the src and I’d rather not do that just yet as it complicates upgrades. We’re using Apache with NTLM and “require valid user” so it’s a corporate domain and only authorized users are allowed access. If I set “use_remote_user = True” on universe_wsgi.ini then users are denied as Apache is passing the domain and username e.g. REMOTE_USER = DOMAIN\\username I can’t use a rewrite rule to fix it from Apache because then it’s an invalid username and the user can’t log into the web, and if it’s passing DOMAIN\\username to Galaxy it doesn’t match up with the Galaxy username so I get a 403 error. Is there a hidden option to strip the domain from the login or am I going to have to start hacking? Hi Russell, In the Apache configuration, you should be able to modify the regex here: RewriteCond %{LA-U:REMOTE_USER} (.+) To strip your domain, e.g.: RewriteCond %{LA-U:REMOTE_USER} DOMAIN\\(.+) --nate Thanx, Russell Smithies Infrastructure Technician Invermay Agricultural Centre Puddle Alley, Private Bag 50034, Mosgiel 9053, New Zealand T +64 3 489 3809 F +64 3 489 3739 www.agresearch.co.nz Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] re-writing username for login or stripping domain from remote_user?
IANAAG, but the following should be secure and effective as well (at least it works for me), setting REMOTE_USER with an Apache authentication module in conjunction with ProxyPass RequestHeader set REMOTE_USER %{REMOTE_USER}s RequestHeader edit REMOTE_USER ^.*(.+?) $1 Cheers, Paul On 12-07-18 9:08 AM, Nate Coraor n...@bx.psu.edu wrote: On Jul 16, 2012, at 6:42 PM, Smithies, Russell wrote: I have a situation I¹m sure others have faced but I can¹t see how to solve it without hacking the src and I¹d rather not do that just yet as it complicates upgrades. We¹re using Apache with NTLM and ³require valid user² so it¹s a corporate domain and only authorized users are allowed access. If I set ³use_remote_user = True² on universe_wsgi.ini then users are denied as Apache is passing the domain and username e.g. REMOTE_USER = DOMAIN\\username I can¹t use a rewrite rule to fix it from Apache because then it¹s an invalid username and the user can¹t log into the web, and if it¹s passing DOMAIN\\username to Galaxy it doesn¹t match up with the Galaxy username so I get a 403 error. Is there a hidden option to strip the domain from the login or am I going to have to start hacking? Hi Russell, In the Apache configuration, you should be able to modify the regex here: RewriteCond %{LA-U:REMOTE_USER} (.+) To strip your domain, e.g.: RewriteCond %{LA-U:REMOTE_USER} DOMAIN\\(.+) --nate Thanx, Russell Smithies Infrastructure Technician Invermay Agricultural Centre Puddle Alley, Private Bag 50034, Mosgiel 9053, New Zealand T +64 3 489 3809 F +64 3 489 3739 www.agresearch.co.nz Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tool test question
On Jul 18, 2012, at 2:55 PM, christopher fletez-brant wrote: Hi, just wondering if anyone has any insight into this issue, as it is driving me crazy, especially given that the correct list control element is selected (hg19). From the output you sent, it doesn't look like your tool is actually running, just the upload. What does your test XML look like? --nate On Sat, Jul 14, 2012 at 8:54 PM, christopher fletez-brant cafletezbr...@gmail.com wrote: Hi Nate, Thanks for the reply. Error messages are output at the bottom of the message. Setup is that the tool has a number of inputs, one of which is actually a directory containing reference files. We pass the directory by allowing users to select from a menu the genome whose information is held in that directory; this is done by means of a *.loc file, whose value in the tool config is 'indices_path' below. Run from the Galaxy server, this tool works perfectly. However, I think that perhaps it cannot find the directory for some reason? I just note that diff reports the values listed as missing, but no '+' values indicating what is in their place. Any advice would be appreciated. Thank you, Kipper alaxy.tools.actions.upload_common INFO 2012-07-14 20:39:41,302 tool upload1 created job id 1 galaxy.jobs.manager DEBUG 2012-07-14 20:39:47,663 (1) Job assigned to handler 'main' galaxy.jobs DEBUG 2012-07-14 20:39:52,828 (1) Working directory for job is: /home/fake/galaxy-dist/database/job_working_directory/000/1 galaxy.jobs.handler DEBUG 2012-07-14 20:39:52,829 dispatching job 1 to local runner galaxy.jobs.handler INFO 2012-07-14 20:39:53,008 (1) Job dispatched galaxy.jobs.runners.local DEBUG 2012-07-14 20:39:53,255 executing: python /home/fake/galaxy-dist/tools/data_source/upload.py /home/fake/galaxy-dist /tmp/tmpeEp5KR/database/tmp/tmpfYngg0 /tmp/tmpeEp5KR/database/tmp/tmpMKsPIL 1:/home/fake/galaxy-dist/database/job_working_directory/000/1/dataset_1_files:/tmp/tmpeEp5KR/database/files/000/dataset_1.dat galaxy.jobs.runners.local DEBUG 2012-07-14 20:39:54,480 execution finished: python /home/fake/galaxy-dist/tools/data_source/upload.py /home/fake/galaxy-dist /tmp/tmpeEp5KR/database/tmp/tmpfYngg0 /tmp/tmpeEp5KR/database/tmp/tmpMKsPIL 1:/home/fake/galaxy-dist/database/job_working_directory/000/1/dataset_1_files:/tmp/tmpeEp5KR/database/files/000/dataset_1.dat galaxy.jobs DEBUG 2012-07-14 20:39:54,759 job 1 ended base.twilltestcase INFO 2012-07-14 20:39:55,782 ## files diff on /home/fake/galaxy-dist/test-data/nullseq_output.bed and /tmp/tmpeEp5KR/database/tmp/tmpuT588_nullseq_output.bed lines_diff=0, found diff = 200 -- begin tool stdout --- --- end tool stdout -- begin tool stderr --- --- end tool stderr FAIL == FAIL: test_tool_00 (functional.test_toolbox.TestForTool_kmersvm_nullseq) Generate Null Sequence ( kmersvm_nullseq ) Test-1 -- Traceback (most recent call last): File /home/fake/galaxy-dist/test/functional/test_toolbox.py, line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File /home/fake/galaxy-dist/test/functional/test_toolbox.py, line 102, in do_it self.verify_dataset_correctness( outfile, hid=elem_hid, maxseconds=testdef.maxseconds, attributes=attributes, shed_tool_id=shed_tool_id ) File /home/fake/galaxy-dist/test/base/twilltestcase.py, line 733, in verify_dataset_correctness raise AssertionError( errmsg ) AssertionError: History item 1 different than expected, difference (using diff): --- local_file +++ history_data @@ -1,100 +1,100 @@ -chr104522044 4522438 -chr105650896 5651310 -chr105772258 5772663 -chr109135867 9136467 -chr109946900 9947273 -chr101079368910794313 -chr101579002315790478 -chr101604843516049028 -chr101762015517620367 -chr102171840421719020 -chr102296716922967533 -chr102299938322999802 -chr102451184524512748 -chr102477363624774006 -chr102597594325976389 -chr102662003226620396 -chr102690799226908408 -chr102830983328310034 -chr103090840730909220 -chr103118477631185151 -chr103119171731192170 -chr103165109831651551 -chr103242250232422880 -chr103331645333316817 -chr103435370034354168 -chr103457061234571026 -chr103485317834853542 -chr10
[galaxy-dev] Where does Galaxy get its TMP directory from?
Hi,I am running into some out-of-disk space errors with some of my jobs and this was because the /tmp space is running out of space.I want to use a different temporary directory but I can't get GALAXY to pick up my settings...I use this in the .ini file# Temporary files are stored in this directory.new_file_path = /mnt/ngs/analysis/tdeboer/galaxy-data/database2/tmpI set my TMPDIR environment variable to something similar, but when I check what my python script is using as its temp dir I get this:print "Temp dir: %s" % tempfile.gettempdir()Temp dir: /tmp/3642962.1.all.qIt seems that the SGE engine overrides the settings in GALAXY, no?So where do I set the tmp directory to be something different then /tmp?ThanksThon___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Where does Galaxy get its TMP directory from?
On Jul 18, 2012, at 4:17 PM, Anthonius deBoer wrote: Hi, I am running into some out-of-disk space errors with some of my jobs and this was because the /tmp space is running out of space. I want to use a different temporary directory but I can't get GALAXY to pick up my settings... I use this in the .ini file # Temporary files are stored in this directory. new_file_path = /mnt/ngs/analysis/tdeboer/galaxy-data/database2/tmp I set my TMPDIR environment variable to something similar, but when I check what my python script is using as its temp dir I get this: print Temp dir: %s % tempfile.gettempdir() Temp dir: /tmp/3642962.1.all.q It seems that the SGE engine overrides the settings in GALAXY, no? So where do I set the tmp directory to be something different then /tmp? Hi Thon, Galaxy can't override SGE's setting of TMPDIR, it would have to happen somewhere in the execution of the job on the node. You may be able to do this in the Galaxy user's .bash_profile, by modifying ~/.sge_request, or by setting up enviornment_setup_file in universe_wsgi.ini. --nate Thanks Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tool test question
Hi Nate, The test XML is: tests test param name=input value=nullseq_test.bed ftype=bed / param name=fold value=1 / param name=gc_err value=0.02 / param name=rpt_err value=0'02 / param name=rseed value=1 / param name=indices_path value=hg19 / output name=output file=nullseq_output.bed / /test /tests And the XML for the problematic parameter, indices_path, is: param name=indices_path type=select label=Available Datasets options from_file=nullseq_indices.loc column name=dbkey index=0/ column name=value index=0/ column name=name index=1/ column name=path index=2/ !--filter type=data_meta ref=input key=dbkey column=0 /-- /options /param Thanks for the reply! On Wed, Jul 18, 2012 at 3:55 PM, Nate Coraor n...@bx.psu.edu wrote: On Jul 18, 2012, at 2:55 PM, christopher fletez-brant wrote: Hi, just wondering if anyone has any insight into this issue, as it is driving me crazy, especially given that the correct list control element is selected (hg19). From the output you sent, it doesn't look like your tool is actually running, just the upload. What does your test XML look like? --nate On Sat, Jul 14, 2012 at 8:54 PM, christopher fletez-brant cafletezbr...@gmail.com wrote: Hi Nate, Thanks for the reply. Error messages are output at the bottom of the message. Setup is that the tool has a number of inputs, one of which is actually a directory containing reference files. We pass the directory by allowing users to select from a menu the genome whose information is held in that directory; this is done by means of a *.loc file, whose value in the tool config is 'indices_path' below. Run from the Galaxy server, this tool works perfectly. However, I think that perhaps it cannot find the directory for some reason? I just note that diff reports the values listed as missing, but no '+' values indicating what is in their place. Any advice would be appreciated. Thank you, Kipper alaxy.tools.actions.upload_common INFO 2012-07-14 20:39:41,302 tool upload1 created job id 1 galaxy.jobs.manager DEBUG 2012-07-14 20:39:47,663 (1) Job assigned to handler 'main' galaxy.jobs DEBUG 2012-07-14 20:39:52,828 (1) Working directory for job is: /home/fake/galaxy-dist/database/job_working_directory/000/1 galaxy.jobs.handler DEBUG 2012-07-14 20:39:52,829 dispatching job 1 to local runner galaxy.jobs.handler INFO 2012-07-14 20:39:53,008 (1) Job dispatched galaxy.jobs.runners.local DEBUG 2012-07-14 20:39:53,255 executing: python /home/fake/galaxy-dist/tools/data_source/upload.py /home/fake/galaxy-dist /tmp/tmpeEp5KR/database/tmp/tmpfYngg0 /tmp/tmpeEp5KR/database/tmp/tmpMKsPIL 1:/home/fake/galaxy-dist/database/job_working_directory/000/1/dataset_1_files:/tmp/tmpeEp5KR/database/files/000/dataset_1.dat galaxy.jobs.runners.local DEBUG 2012-07-14 20:39:54,480 execution finished: python /home/fake/galaxy-dist/tools/data_source/upload.py /home/fake/galaxy-dist /tmp/tmpeEp5KR/database/tmp/tmpfYngg0 /tmp/tmpeEp5KR/database/tmp/tmpMKsPIL 1:/home/fake/galaxy-dist/database/job_working_directory/000/1/dataset_1_files:/tmp/tmpeEp5KR/database/files/000/dataset_1.dat galaxy.jobs DEBUG 2012-07-14 20:39:54,759 job 1 ended base.twilltestcase INFO 2012-07-14 20:39:55,782 ## files diff on /home/fake/galaxy-dist/test-data/nullseq_output.bed and /tmp/tmpeEp5KR/database/tmp/tmpuT588_nullseq_output.bed lines_diff=0, found diff = 200 -- begin tool stdout --- --- end tool stdout -- begin tool stderr --- --- end tool stderr FAIL == FAIL: test_tool_00 (functional.test_toolbox.TestForTool_kmersvm_nullseq) Generate Null Sequence ( kmersvm_nullseq ) Test-1 -- Traceback (most recent call last): File /home/fake/galaxy-dist/test/functional/test_toolbox.py, line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File /home/fake/galaxy-dist/test/functional/test_toolbox.py, line 102, in do_it self.verify_dataset_correctness( outfile, hid=elem_hid, maxseconds=testdef.maxseconds, attributes=attributes, shed_tool_id=shed_tool_id ) File /home/fake/galaxy-dist/test/base/twilltestcase.py, line 733, in verify_dataset_correctness raise AssertionError( errmsg ) AssertionError: History item 1 different than expected, difference (using diff): --- local_file +++ history_data @@ -1,100 +1,100 @@ -chr104522044 4522438 -chr105650896 5651310 -chr105772258 5772663 -chr109135867 9136467 -chr109946900 9947273 -chr101079368910794313 -chr10