date:20130506

 On May 2, 2013, at 12:54 PM, Peter Cock wrote:

 Another even simpler change (which would also make a big
 difference) is to actually have the name of the current
 repository on the Upload a single file or a tarball page.

 Thanks,

On Thu, May 2, 2013 at 9:32 PM, Greg Von Kuster g...@bx.psu.edu wrote:
 Hi Peter,

 This fix is now running on the test tool shed.  Thanks for pointing out this 
 flaw!

 Greg Von Kuster

Great - that was an easy fix but should cut down on the number
of accidental commits due to human error. Let's see how long it
is before someone else messes up (or I do it again - grin) and
if something more advanced is needed like my first suggestions.

Thanks,

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Missing test results on (Test) Tool Shed

 On 2013-05-01 08:52, Greg Von Kuster wrote:

 Hi Peter,

 I created the following Trello card for this and things will be fixed as a
 priority.  Thanks for letting us know!


 https://trello.com/card/toolshed-issues-with-functional-test-results/506338ce32ae458f6d15e4b3/823

On Wed, May 1, 2013 at 11:50 PM, Dave Bouvier d...@bx.psu.edu wrote:
 Peter,

 Thanks for the data, I am in the process of tracking down the source of the
 issue, and I hope to have a fix ready shortly.

  --Dave B.

Hi Dave,

I see from the Trello card you've made some progress here - but I am
still seeing odd results on the Test Tool Shed (including not seeing
any test results), which has also been reflected in the RSS feed.
e.g. Conflicting RSS entries:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired/fe469503441b
Functional test results for changeset revision fae4084a0bc0 of
fastq_paired_unpaired
0 tests passed, 0 tests failed, 5 tests missing test components.

vs

http://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired/fe469503441b
Functional test results for changeset revision fe469503441b of
fastq_paired_unpaired
0 tests passed, 0 tests failed, 0 tests missing test components.

Currently the Test Tool Shed shows me no test information at all
for this example (and sadly many others).

This seems to be part of a larger issue, Latest revision: failing tool tests
currently only lists mira_assembler - yet viewing this shows no failure
(or any other results):
http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler/da604cc07658

Last week I had numerous failing tests (mostly due to missing tool
dependencies, some due to issues with the text framework). Many
of the (presumably still problematic) repositories are now listed under
Latest revision: all tool tests pass yet on viewing them there are no
test results shown at all, e.g.
http://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go/54c3d14e5621
http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp/af3174637834

Something seems to have gone wrong on Friday or over the weekend,

Regards,

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] user creation using API

2013-05-06 Thread Rémy Dernat

Hi,

This is an old topic now, but here is a little update (fork from Git / John
Chilton) of this code:

https://gist.github.com/remyd1/5524162

Regards
Remy



2013/1/8 Hagai Cohen haga...@gmail.com

 This is really helpful,
 Works great.
 Thanks,
 Hagai



 On Mon, Jan 7, 2013 at 5:13 PM, John Chilton chil...@msi.umn.edu wrote:

 I am pretty confident there is no way to do this via the API
 currently, though I imagine it wouldn't be terribly complicated to add
 and the community would appreciate the contribution. If API-based
 access isn't a strong requirement and you have direct access to the
 Galaxy server here is a script-template I have used to automatically
 create users:

 https://gist.github.com/4475646

 The above gist is probably enough to get going, but here is the full
 context of how I create such a file if that is of interest:

 https://github.com/jmchilton/cloudbiolinux/blob/deploy/cloudbio/deploy/galaxy.py

 -John

 On Mon, Jan 7, 2013 at 3:37 AM, Hagai Cohen haga...@gmail.com wrote:
  Hi,
  I would like to auto create users on my local Galaxy instance.
 
  I saw that in the create user on the API, I should enable
 use_remote_user
  (which as I understand means that I have to use Apache or nginx).
  Is there another way besides using Apache to create users automatically?
 
  Thanks,
  Hagai
 
 
  ___
  Please keep all replies on the list by using reply all
  in your mail client.  To manage your subscriptions to this
  and other Galaxy lists, please use the interface at:
 
http://lists.bx.psu.edu/



 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:

   http://lists.bx.psu.edu/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Missing test results on (Test) Tool Shed

2013-05-06 Thread Dave Bouvier

Peter,

Yes, there are a few issues that are causing some repositories not to be
tested correctly. I've discovered the problem, but the solution will
require a bit of work. My goal is to have this resolved as soon as possible.

--Dave B.

On 5/6/13 05:10:57.000, Peter Cock wrote:

On 2013-05-01 08:52, Greg Von Kuster wrote:

Hi Peter,

I created the following Trello card for this and things will be fixed as a
priority. Thanks for letting us know!

https://trello.com/card/toolshed-issues-with-functional-test-results/506338ce32ae458f6d15e4b3/823

On Wed, May 1, 2013 at 11:50 PM, Dave Bouvier d...@bx.psu.edu wrote:

Peter,

Thanks for the data, I am in the process of tracking down the source of the
issue, and I hope to have a fix ready shortly.

--Dave B.

Hi Dave,

I see from the Trello card you've made some progress here - but I am
still seeing odd results on the Test Tool Shed (including not seeing
any test results), which has also been reflected in the RSS feed.
e.g. Conflicting RSS entries:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired/fe469503441b
Functional test results for changeset revision fae4084a0bc0 of
fastq_paired_unpaired
0 tests passed, 0 tests failed, 5 tests missing test components.

http://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired/fe469503441b
Functional test results for changeset revision fe469503441b of
fastq_paired_unpaired
0 tests passed, 0 tests failed, 0 tests missing test components.

Currently the Test Tool Shed shows me no test information at all
for this example (and sadly many others).

This seems to be part of a larger issue, Latest revision: failing tool tests
currently only lists mira_assembler - yet viewing this shows no failure
(or any other results):
http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler/da604cc07658

Last week I had numerous failing tests (mostly due to missing tool
dependencies, some due to issues with the text framework). Many
of the (presumably still problematic) repositories are now listed under
Latest revision: all tool tests pass yet on viewing them there are no
test results shown at all, e.g.
http://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go/54c3d14e5621
http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp/af3174637834

Something seems to have gone wrong on Friday or over the weekend,

Regards,

Peter

___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/

[galaxy-dev] May 2013 Galaxy Update

2013-05-06 Thread Dave Clements

Hello all,

The May 2013 Galaxy Update is now
availablehttp://wiki.galaxyproject.org/GalaxyUpdates/2013_05
.

*Highlights:*

   -

   the GCC2013 http://wiki.galaxyproject.org/GalaxyUpdates/2013_05#GCC2013
*early registration deadline is 24
Mayhttp://wiki.galaxyproject.org/Events/GCC2013/Register
   *.
   -

   39 new papershttp://wiki.galaxyproject.org/GalaxyUpdates/2013_05#New_Papers
   -

   Open 
Positionshttp://wiki.galaxyproject.org/GalaxyUpdates/2013_05#Who.27s_Hiring
at
   five different institutions
   -

   Other Upcoming
Eventshttp://wiki.galaxyproject.org/GalaxyUpdates/2013_05#Other_Upcoming_Events
   -

   Galaxy 
Distributionshttp://wiki.galaxyproject.org/GalaxyUpdates/2013_05#Galaxy_Distributions
   -

   Tool Shed 
Contributionshttp://wiki.galaxyproject.org/GalaxyUpdates/2013_05#Tool_Shed_Contributions
   -

   Other Newshttp://wiki.galaxyproject.org/GalaxyUpdates/2013_05#Other_News

If you have anything you would like to see in the June *Galaxy
Updatehttp://wiki.galaxyproject.org/GalaxyUpdates
*, please let us know.

Dave Clements and the Galaxy Team http://wiki.galaxyproject.org/GalaxyTeam


-- 
http://galaxyproject.org/GCC2013
http://galaxyproject.org/
http://getgalaxy.org/
http://usegalaxy.org/
http://wiki.galaxyproject.org/
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Taxonomy fetching not working

2013-05-06 Thread Deepthi Thomas Kannanayakal

Hi Galaxy team,

I am having problems with taxonomy information fetching.(Metagenomic
Analysses-Fetch taxonomic representation).

According to the following link

http://dev.list.galaxyproject.org/quot-Fetch-taxonomic-representation-quot-DOESN-T-WORK-td4565489.html

I have installed libraries from https://bitbucket.org/natefoo/taxonomy
I have installed them in my local directory(not in /usr/local).  But I am
still having problem

/bin/sh: taxBuilder: command not found

What am I missing here?

Thanks,

Deeps

*Deepthi Theresa Thomas Kannanayakal

Bioinformatics Database Specialist
Sun Visual Genomic Centre,
Suite 1150,  Health Science Centre
3330 Hospital Drive NW
Department of Biochemistry and Molecular Biology
 Faculty of Medicine,  University of Calgary
T2N 4N1
Calgary,  Alberta,  Canada**
*
*Ph: (403) - 210 6661   Fax: (403) - 210 9538*
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] LWR runner configuration for shared folder in cluster

2013-05-06 Thread Krampis, Konstantinos

Hi all,

  I am trying to set up a Galaxy cluster using the LWR runner. The nodes have
a shared filesystem and in universe.wsgi this parameter is set :

job_working_directory = /mnt/shared
...
clustalw = lwr://http://192.168.33.12:8913


this folder has been chown-ed to the galaxy user, and also is a+w,
while it has been verified that can been read / written by ssh-ing to
each node of the cluster. The sticky bit is set.

When I try to run jobs (I used clustalw as example) there seems to be 
confusion between where Galaxy puts files and where LWR tries to read
them from. Here are two setups that error out:




1). When in server.ini for LWR the following is set as:
staging_directory = /mnt/shared/000

galaxy error:

galaxy.jobs DEBUG 2013-05-06 10:21:22,320 (128) Working directory for job is: 
/mnt/shared/000/128
galaxy.jobs.handler DEBUG 2013-05-06 10:21:22,320 dispatching job 128 to lwr 
runner
galaxy.jobs.handler INFO 2013-05-06 10:21:22,427 (128) Job dispatched
galaxy.datatypes.metadata DEBUG 2013-05-06 10:21:22,875 Cleaning up external 
metadata files
galaxy.jobs.runners.lwr ERROR 2013-05-06 10:21:22,902 failure running job 128


lwr error (on the cluster node):

  File /home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/app.py, line 81, in setup
manager.setup_job_directory(job_id)
  File /home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/manager.py, line 101, in 
setup_job_directory
os.mkdir(job_directory)
OSError: [Errno 17] File exists: '/mnt/shared/000/128'




2). When in server.ini for LWR the following is set as:
staging_directory = /mnt/shared

galaxy error:

galaxy.jobs DEBUG 2013-05-06 10:28:46,872 (129) Working directory for job is: 
/mnt/shared/000/129
galaxy.jobs.handler DEBUG 2013-05-06 10:28:46,872 dispatching job 129 to lwr 
runner
galaxy.jobs.handler INFO 2013-05-06 10:28:46,967 (129) Job dispatched
192.168.33.1 - - [06/May/2013:10:28:48 -0200] GET 
/api/histories/2a56795cad3c7db3 HTTP/1.1 200 - 
http://192.168.33.11:8080/history; Mozilla/5.0 (Macintosh; Intel Mac OS X 
10_7_5) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.65 
Safari/537.31
galaxy.jobs.runners.lwr DEBUG 2013-05-06 10:28:50,653 run_results {'status': 
'status', 'returncode': 0, 'complete': 'true', 'stderr': '', 'stdout': ''}
galaxy.datatypes.metadata DEBUG 2013-05-06 10:28:50,970 Cleaning up external 
metadata files
galaxy.jobs.runners.lwr ERROR 2013-05-06 10:28:51,050 failure running job 129


lwr error (on the cluster node):

resp.app_iter = FileIterator(result)
  File /home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/framework.py, line 111, 
in __init__
self.input = open(path, 'rb')
IOError: [Errno 2] No such file or directory: 
u'/mnt/shared/129/outputs/dataset_170.dat'




The full error stacks are at the end of this email. It might be something very 
simple that I am missing,
but any feedback would be greatly appreciated. Thanks !

Ntino



-- 
Konstantinos (Ntino) Krampis, Ph.D.
Asst. Professor, Informatics
J.Craig Venter Institute

kkram...@jcvi.org
agbio...@gmail.com
+1-540-200-8277

Web:
http://bit.ly/cloud-research
http://cloudbiolinux.org/
http://twitter.com/agbiotec



 GALAXY ERROR

galaxy.jobs DEBUG 2013-05-06 10:21:22,320 (128) Working directory for job is: 
/mnt/shared/000/128
galaxy.jobs.handler DEBUG 2013-05-06 10:21:22,320 dispatching job 128 to lwr 
runner
galaxy.jobs.handler INFO 2013-05-06 10:21:22,427 (128) Job dispatched
galaxy.datatypes.metadata DEBUG 2013-05-06 10:21:22,875 Cleaning up external 
metadata files
galaxy.jobs.runners.lwr ERROR 2013-05-06 10:21:22,902 failure running job 128
Traceback (most recent call last):
  File /home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py, line 286, in 
run_job
file_stager = FileStager(client, command_line, job_wrapper.extra_filenames, 
input_files, output_files, job_wrapper.tool.tool_dir)
  File /home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py, line 40, in 
__init__
job_config = client.setup()
  File /home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py, line 212, in 
setup
return self.__raw_execute_and_parse(setup, { job_id : self.job_id })
  File /home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py, line 150, in 
__raw_execute_and_parse
response = self.__raw_execute(command, args, data)
  File /home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py, line 146, in 
__raw_execute
response = self.url_open(request, data)
  File /home/vagrant/galaxy-dist/lib/galaxy/jobs/runners/lwr.py, line 134, in 
url_open
return urllib2.urlopen(request, data)
  File /usr/lib/python2.7/urllib2.py, line 126, in urlopen
return _opener.open(url, data, timeout)
  File /usr/lib/python2.7/urllib2.py, line 406, in open
response = meth(req, response)
  File /usr/lib/python2.7/urllib2.py, line 519, in http_response
'http', request, response, code, msg, hdrs)
  File /usr/lib/python2.7/urllib2.py, line 444, in error
return self._call_chain(*args)
  File /usr/lib/python2.7/urllib2.py, line 378, in

Re: [galaxy-dev] LWR runner configuration for shared folder in cluster

2013-05-06 Thread John Chilton

Hello Dr. Krampis,

  At the present, the LWR is most valuable when there is not a shared
file system between the server executing the jobs and the server
hosting Galaxy. In this case you seem to have a shared filesystem so I
would think setting up something like sun grid engine to manage the
jobs and using the DRMAA job runner would be the best route forward.

  The LWR and the corresponding galaxy job runner will coordinate to
stage jobs, but the upshot is that the LWR should be the only thing
writing to its staging directory. In this case you have configured the
LWR and Galaxy to both use the same directory. You should change this
configuration immediately, I am worried the LWR is going to delete or
overwrite files maintained by Galaxy. I am sorry for the confusion, I
will update the documentation to explicitly warn against this.

  If you still feel there is a compelling reason to use the LWR in
this situation, you will just want to change the staging_directory in
the LWR configuration to something else. It has long been on my TODO
list to allow one to disable (or selectively disable by path
regex/globs) file staging with the LWR, it seems like that would what
would also help in your situation. Let me know if that is of interest
to you.

-John


On Mon, May 6, 2013 at 8:32 AM, Krampis, Konstantinos kkram...@jcvi.org wrote:
 Hi all,

   I am trying to set up a Galaxy cluster using the LWR runner. The nodes have
 a shared filesystem and in universe.wsgi this parameter is set :

 job_working_directory = /mnt/shared
 ...
 clustalw = lwr://http://192.168.33.12:8913
 

 this folder has been chown-ed to the galaxy user, and also is a+w,
 while it has been verified that can been read / written by ssh-ing to
 each node of the cluster. The sticky bit is set.

 When I try to run jobs (I used clustalw as example) there seems to be
 confusion between where Galaxy puts files and where LWR tries to read
 them from. Here are two setups that error out:




 1). When in server.ini for LWR the following is set as:
 staging_directory = /mnt/shared/000

 galaxy error:

 galaxy.jobs DEBUG 2013-05-06 10:21:22,320 (128) Working directory for job is: 
 /mnt/shared/000/128
 galaxy.jobs.handler DEBUG 2013-05-06 10:21:22,320 dispatching job 128 to lwr 
 runner
 galaxy.jobs.handler INFO 2013-05-06 10:21:22,427 (128) Job dispatched
 galaxy.datatypes.metadata DEBUG 2013-05-06 10:21:22,875 Cleaning up external 
 metadata files
 galaxy.jobs.runners.lwr ERROR 2013-05-06 10:21:22,902 failure running job 128


 lwr error (on the cluster node):

   File /home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/app.py, line 81, in 
 setup
 manager.setup_job_directory(job_id)
   File /home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/manager.py, line 101, 
 in setup_job_directory
 os.mkdir(job_directory)
 OSError: [Errno 17] File exists: '/mnt/shared/000/128'




 2). When in server.ini for LWR the following is set as:
 staging_directory = /mnt/shared

 galaxy error:

 galaxy.jobs DEBUG 2013-05-06 10:28:46,872 (129) Working directory for job is: 
 /mnt/shared/000/129
 galaxy.jobs.handler DEBUG 2013-05-06 10:28:46,872 dispatching job 129 to lwr 
 runner
 galaxy.jobs.handler INFO 2013-05-06 10:28:46,967 (129) Job dispatched
 192.168.33.1 - - [06/May/2013:10:28:48 -0200] GET 
 /api/histories/2a56795cad3c7db3 HTTP/1.1 200 - 
 http://192.168.33.11:8080/history; Mozilla/5.0 (Macintosh; Intel Mac OS X 
 10_7_5) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.65 
 Safari/537.31
 galaxy.jobs.runners.lwr DEBUG 2013-05-06 10:28:50,653 run_results {'status': 
 'status', 'returncode': 0, 'complete': 'true', 'stderr': '', 'stdout': ''}
 galaxy.datatypes.metadata DEBUG 2013-05-06 10:28:50,970 Cleaning up external 
 metadata files
 galaxy.jobs.runners.lwr ERROR 2013-05-06 10:28:51,050 failure running job 129


 lwr error (on the cluster node):

 resp.app_iter = FileIterator(result)
   File /home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/framework.py, line 111, 
 in __init__
 self.input = open(path, 'rb')
 IOError: [Errno 2] No such file or directory: 
 u'/mnt/shared/129/outputs/dataset_170.dat'




 The full error stacks are at the end of this email. It might be something 
 very simple that I am missing,
 but any feedback would be greatly appreciated. Thanks !

 Ntino



 --
 Konstantinos (Ntino) Krampis, Ph.D.
 Asst. Professor, Informatics
 J.Craig Venter Institute

 kkram...@jcvi.org
 agbio...@gmail.com
 +1-540-200-8277

 Web:
 http://bit.ly/cloud-research
 http://cloudbiolinux.org/
 http://twitter.com/agbiotec



  GALAXY ERROR

 galaxy.jobs DEBUG 2013-05-06 10:21:22,320 (128) Working directory for job is: 
 /mnt/shared/000/128
 galaxy.jobs.handler DEBUG 2013-05-06 10:21:22,320 dispatching job 128 to lwr 
 runner
 galaxy.jobs.handler INFO 2013-05-06 10:21:22,427 (128) Job dispatched
 galaxy.datatypes.metadata DEBUG 2013-05-06 10:21:22,875 Cleaning up external 
 metadata files
 galaxy.jobs.runners.lwr ERROR 2013-05-06 10:21:22,902 failure

Re: [galaxy-dev] Any hint on how to use Galaxy within virtualenv?

2013-05-06 Thread Jean-Francois Payotte

Hi Dannon,

We did update our galaxy development instance to the latest distribution 
(April 1st).
Unfortunately, because of some LDAP issue (we are testing to implement 
LDAP authentication at the same time) I didn't had time yet, to test if 
this would solve the markupsafe issue.

As soon as I can find a couple minutes to test it, I will let you know the 
results.
Thank you for your follow-up. :)

Jean-Francois




From:   Dannon Baker dannon.ba...@gmail.com
To: Jean-Francois Payotte jean-francois.payo...@dnalandmarks.ca
Cc: Galaxy Dev galaxy-dev@lists.bx.psu.edu
Date:   29/04/2013 09:24 AM
Subject:Re: [galaxy-dev] Any hint on how to use Galaxy within 
virtualenv?



Hi Jean-Francois,

Were you able to update to the next galaxy-dist and test?  This issue 
should be resolved in 8771:ae446180fd11, which was added just after the 
distribution you were using previously.

-Dannon
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] LWR runner configuration for shared folder in cluster

2013-05-06 Thread Krampis, Konstantinos

Hi John,

  thanks for the quick reply! It is actually a virtual cluster using virtual 
machines
with VirtualBox, which are brought up via Vagrant (www.vagrantup.com) - that
also sets up the shared filesystem. I chose LWR for the reason that it not a 
real
cluster, and needed something lightweight instead of having to stand up SGE etc.

  I am very interested in the 3rd option you present in your response below. I 
am 
still confused though that if Galaxy and LWR doe not use shared filesystem, how 
does 
Galaxy know when the job is finished, or where to find the output files ? 

Or does the output gets piped back to Galaxy ? That would work (all virtual 
machines 
are on the same box, so no latency), as long as the output gets deleted by LWR 
after 
being send to Galaxy (to save space in the worker virtual machines).

  If you could provide some pointers to documentation / code, or explain how
could be set up it would be of great help !

 many thanks,

Ntino

  


From: jmchil...@gmail.com [jmchil...@gmail.com] On Behalf Of John Chilton 
[chil...@msi.umn.edu]
Sent: Monday, May 06, 2013 9:50 AM
To: Krampis, Konstantinos
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] LWR runner configuration for shared folder in cluster

Hello Dr. Krampis,

  At the present, the LWR is most valuable when there is not a shared
file system between the server executing the jobs and the server
hosting Galaxy. In this case you seem to have a shared filesystem so I
would think setting up something like sun grid engine to manage the
jobs and using the DRMAA job runner would be the best route forward.

  The LWR and the corresponding galaxy job runner will coordinate to
stage jobs, but the upshot is that the LWR should be the only thing
writing to its staging directory. In this case you have configured the
LWR and Galaxy to both use the same directory. You should change this
configuration immediately, I am worried the LWR is going to delete or
overwrite files maintained by Galaxy. I am sorry for the confusion, I
will update the documentation to explicitly warn against this.

  If you still feel there is a compelling reason to use the LWR in
this situation, you will just want to change the staging_directory in
the LWR configuration to something else. It has long been on my TODO
list to allow one to disable (or selectively disable by path
regex/globs) file staging with the LWR, it seems like that would what
would also help in your situation. Let me know if that is of interest
to you.

-John


On Mon, May 6, 2013 at 8:32 AM, Krampis, Konstantinos kkram...@jcvi.org wrote:
 Hi all,

   I am trying to set up a Galaxy cluster using the LWR runner. The nodes have
 a shared filesystem and in universe.wsgi this parameter is set :

 job_working_directory = /mnt/shared
 ...
 clustalw = lwr://http://192.168.33.12:8913
 

 this folder has been chown-ed to the galaxy user, and also is a+w,
 while it has been verified that can been read / written by ssh-ing to
 each node of the cluster. The sticky bit is set.

 When I try to run jobs (I used clustalw as example) there seems to be
 confusion between where Galaxy puts files and where LWR tries to read
 them from. Here are two setups that error out:




 1). When in server.ini for LWR the following is set as:
 staging_directory = /mnt/shared/000

 galaxy error:

 galaxy.jobs DEBUG 2013-05-06 10:21:22,320 (128) Working directory for job is: 
 /mnt/shared/000/128
 galaxy.jobs.handler DEBUG 2013-05-06 10:21:22,320 dispatching job 128 to lwr 
 runner
 galaxy.jobs.handler INFO 2013-05-06 10:21:22,427 (128) Job dispatched
 galaxy.datatypes.metadata DEBUG 2013-05-06 10:21:22,875 Cleaning up external 
 metadata files
 galaxy.jobs.runners.lwr ERROR 2013-05-06 10:21:22,902 failure running job 128


 lwr error (on the cluster node):

   File /home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/app.py, line 81, in 
 setup
 manager.setup_job_directory(job_id)
   File /home/vagrant/jmchilton-lwr-5213f6dce32d/lwr/manager.py, line 101, 
 in setup_job_directory
 os.mkdir(job_directory)
 OSError: [Errno 17] File exists: '/mnt/shared/000/128'




 2). When in server.ini for LWR the following is set as:
 staging_directory = /mnt/shared

 galaxy error:

 galaxy.jobs DEBUG 2013-05-06 10:28:46,872 (129) Working directory for job is: 
 /mnt/shared/000/129
 galaxy.jobs.handler DEBUG 2013-05-06 10:28:46,872 dispatching job 129 to lwr 
 runner
 galaxy.jobs.handler INFO 2013-05-06 10:28:46,967 (129) Job dispatched
 192.168.33.1 - - [06/May/2013:10:28:48 -0200] GET 
 /api/histories/2a56795cad3c7db3 HTTP/1.1 200 - 
 http://192.168.33.11:8080/history; Mozilla/5.0 (Macintosh; Intel Mac OS X 
 10_7_5) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.65 
 Safari/537.31
 galaxy.jobs.runners.lwr DEBUG 2013-05-06 10:28:50,653 run_results {'status': 
 'status', 'returncode': 0, 'complete': 'true', 'stderr': '', 'stdout': ''}
 galaxy.datatypes.metadata DEBUG 2013-05-06

Re: [galaxy-dev] galaxysession cookie secure flag

2013-05-06 Thread Vipin TS

Hello,

I figured out the place where the cookie gets set in Galaxy and then add
secure to that. Apache configuration didn't work well.

added the following code into  function set_cookie in:
lib/galaxy/web/framework/__init__.py

*def set_cookie( self, value, name='galaxysession', path='/', age=90,
version='1' ):
*
* try: *
* self.response.cookies[name]['secure'] = True *
* except CookieError, e: *
* log.warning( Error setting secure attribute in cookie '%s': %s
% ( name, e ) )*

I tested by running the following, now I can see the flag secure in the
set-cookie

curl -k -D - https://gx.cbio.mskcc.org/ -o /dev/null
  % Total% Received % Xferd  Average Speed   TimeTime Time
 Current
 Dload  Upload   Total   SpentLeft
 Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:--
  0HTTP/1.1 200 OK
Date: Mon, 06 May 2013 14:50:16 GMT
Server: PasteWSGIServer/0.5 Python/2.6.6
content-type: text/html; charset=UTF-8
Set-Cookie:
galaxysession=7cf35ade3e68eef6c0bd6866318609b987df86a0d50ecc280f02efaa5966a9aa59ce7177812bed97;
expires=Sun, 04-Aug-2013 10:50:16 GMT; httponly; Max-Age=7776000; Path=/;
secure; Version=1
Connection: close
Transfer-Encoding: chunked

100 253950 253950 0  35881  0 --:--:-- --:--:-- --:--:--
69575

--Vipin

 Hi dev-team,

 We have placed our galaxy instance ssl and I need to make sure that the
 secure flag is set
 on the cookie (commonly represented by the word “secure” under the
 Security column) but
 I am not able to do the same. something like below:

 [image: Inline image 2]

 when I checked on my instance I saw as below:
 [image: Inline image 3]
 I have made necessary changes to my ssl.conf to put the flag as secure,
 but it seems not appearing here.

 Header edit Set-Cookie ^(.*)$ $1;Secure;HttpOnly

 does anybody have an experience in setting up the same. thanks in advance,
 --/Vipin


image.pngimage.png___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] [galaxy-iuc] Script to help maintain toolshed repos across toolsheds

On Thu, May 2, 2013 at 10:24 AM, Peter Cock p.j.a.c...@googlemail.com wrote:
 On Thu, May 2, 2013 at 7:16 AM, Ira Cooke iraco...@gmail.com wrote:
 Hi all,

 I've written a script to help deal with the problem of maintaining toolshed 
 tools
 across multiple toolsheds (eg test and release)

 The problem I encountered was that switching between test and production
 versions of a suite of tools can be quite painful because every repository
 definition like this

 repository toolshed=http://toolshed.g2.bx.psu.edu; 
 name=proteomics_datatypes owner=iracooke 
 changeset_revision=463328a6967f/

 needs to be updated to a different toolshed url and (by extension) a 
 different
 changeset revision.

 The idea with this script is that you should be able to point it at a 
 directory
 containing a toolshed repository and it will create a copy of that repository
 in which the toolshed urls (and changeset revisions) have been updated
 to correct values for a different toolshed.

 I'm not sure how others are dealing with this issue (perhaps there is another
 easier way) .. but I've found this helped me alot so I thought I'd share

 https://bitbucket.org/iracooke/galaxy_repo_bundler/

 Cheers
 Ira

 Thanks Ira,

 I've not made as heavy use of inter-repository dependencies as you, but
 thus far I have ignored the problem (only a couple of my repositories are
 affected), in the hope this limitation will be fixed sooner rather than later.

 Peter

Thinking out loud, another way to solve this would be to allow multiple
equivalent repository entries as a group where any one would be OK.

e.g. For v0.0.5 of my seq_filter_by_id tool,

repository toolshed=http://toolshed.g2.bx.psu.edu; owner=peterjc
name=seq_filter_by_id changeset_revision=abdd608c869b/
for http://toolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id/abdd608c869b

or:

repository toolshed=http://testtoolshed.g2.bx.psu.edu;
owner=peterjc name=seq_filter_by_id
changeset_revision=66d1ca92fb38/
for http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id/66d1ca92fb38

Something like this maybe?:

any_one_of
  repository toolshed=http://toolshed.g2.bx.psu.edu; owner=peterjc
name=seq_filter_by_id changeset_revision=abdd608c869b/
  repository toolshed=http://testtoolshed.g2.bx.psu.edu;
owner=peterjc name=seq_filter_by_id
changeset_revision=66d1ca92fb38/
/any_one_of

Of course more generally we might also post this on a local public Tool
Shed as well. The point is then the same repository_dependencies.xml
could be used on either tool shed without modification.

Fixing declaring a dependency on an external Tool Shed would be better -
but perhaps we'll need both or some sort of mirroring federated system
in the long term?

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] user password different type encoding

2013-05-06 Thread Vipin TS

Thanks James!

I have pulled the recent changes to my repository and this is working fine.
I have tested with creating a new user
and I tried to login with the recently created user and this works fine at
my end. The entry looks like as follows:

galaxy=# select username,email,password from galaxy_user where email = '
c...@gmail.com';
 username | email  |   password

--++---
 cbio | c...@gmail.com |
PBKDF2$sha256$1$7CWpfEOjZ3xmRH6b$ufFHb8Ax9GkVBGmnR5fq159NA3C4F6o0

This works fine.

With this new implementation, the new registration password hashes are
generated in PBKDF2. May be one can force
the existing users to reset the password, so that all users password will
be updated in the table. As you mentioned in the
previous mail, because of random salt generation in the password hashing
step, I don't think I can take all of the existing
user’s SHA-1 hashes, run them through the algorithm and replace them with
the updated hashes. I thought this would
be transparent to the users instead of forcing to reset the password.

any thoughts on this?

Thank you for timely help,
--Vipin


Rather than committing this directly I created the following pull request:


 https://bitbucket.org/galaxy/galaxy-central/pull-request/165/password-security-use-pbkdf2-scheme-with

 It would be great if a couple of people could sign-off on it before
 merging. I don't think I'm doing anything stupid, but a sanity check
 is appreciated.

 --
 James Taylor, Assistant Professor, Biology/CS, Emory University


 On Sun, May 5, 2013 at 12:12 PM, James Taylor ja...@jamestaylor.org
 wrote:
  Vipin, I think the main problem here is that you cannot treat PBKDF2
  as a hash in this way. Every time you hash the same password you get a
  different result because you are generating a new random salt.
  Instead, you need to decode the in database representation to extract
  the salt and then do a comparison on the hashed part.
 
  I have this working in a backward compatible way, and I think it is a
  good idea so I will be committing it to central shortly.
 
  --
  James Taylor, Assistant Professor, Biology/CS, Emory University
 
 
  On Thu, May 2, 2013 at 2:34 PM, Vipin TS vipin...@gmail.com wrote:
 
  Thanks James, I have updated the password of one user in galaxy_user
 table
  with the new algorithm,
  I also adjusted the function new_secure_hash in
  /lib/galaxy/util/hash_util.py in such a way that it returns
  the new hash instead of sha1. Now I tried to login, it fails to get the
  account, I think there is something going
  wrong in the password hash comparison. Can you please assit here.
 
  +++ b/lib/galaxy/util/hash_util.py  Thu May 02 14:33:07 2013 -0400
  @@ -25,13 +25,60 @@
   Returns either a sha1 hash object (if called with no arguments),
 or a
   hexdigest of the sha1 hash of the argument `text_type`.
   
  +import hashlib
  +from os import urandom
  +from base64 import b64encode, b64decode
  +from itertools import izip
  +from pbkdf2 import pbkdf2_bin
  +
  +SALT_LENGTH = 12
  +KEY_LENGTH = 24
  +HASH_FUNCTION = 'sha256'
  +COST_FACTOR = 1
  +
   if text_type:
  +#return sha1( text_type ).hexdigest()
  +
  +sec_hash_1 = sha1( text_type ).hexdigest()
  +
  +if isinstance(sec_hash_1, unicode):
  +sec_hash_1 = sec_hash_1.encode('utf-8')
  +salt = b64encode(urandom(SALT_LENGTH))
  +
  +return 'PBKDF2${0}${1}${2}${3}'.format(
  +HASH_FUNCTION,
  +COST_FACTOR,
  +salt,
  +b64encode(pbkdf2_bin(sec_hash_1, salt, COST_FACTOR,
 KEY_LENGTH,
  getattr(hashlib, HASH_FUNCTION
 
 
  thanks, Vipin
 
 
  That should be the only place, it is called from the some methods of
  the User model object. So you could modify it to always hash new
  passwords in a different way, but check old passwords with sha1 first,
  then something else.
 
  Although it might be nice to move the functionality into
  security.validate_user_input since it is really specific to user
  passwords, especially with those changes.
 
  I'd be happy to see this go into main with sha256 or something
  similar. Also, we could consider adding a random per-user salt field
  if you are really concerned about this.
 
  --
  James Taylor, Assistant Professor, Biology/CS, Emory University
 
 
  On Thu, May 2, 2013 at 10:21 AM, Vipin TS vipin...@gmail.com wrote:
   Hello dev-team,
   I would like to add the different type of password encryption to the
   users
   in my galaxy instance. I started working with the current password
   encoding
   script:
   /home/apps/galaxy-dist/lib/galaxy/util/hash_util.py
  
   I will keep the current sha1 and add another layer of encryption to
 the
   sha1
   hash, otherwise I need to force all my users to change the password
 and
   follow the new hashing

[galaxy-dev] Published Workflows missing from All Workflows

Hello all,

This is to report a usability bug with workflows - specifically the curiously
separate handling of Published Workflows in the user interface.

1. Log into your Galaxy account,
2. On the top ribbon, click on Shared Data, Published Workflows,
and make a note of the entries.
3. Return to the main screen via the top ribbon entry Analyse Data
4. On the bottom of the left hand tool column, under Workflows click
on All workflows. Observe results.
5. On the top ribbon, click on Workflows. Observe results.

Actual result (in 4 and 5):
Two sections, Your workflows and Workflows shared with you by others

Expected result (in 4 and 5):
Three sections, Your workflows, Workflows shared with you by
others, Published Workflows.

i.e. Published Workflows are hidden or separated away from the others.
This appears to be a design choice (like shared datasets they must be
imported before use), but if so this could be clarified as follows:

Suggested result (in 4 and 5):
Two sections, Your workflows and Workflows shared with you by others,
plus a footnote (with link) saying something about additional workflows are
available to import from Published Workflows.

Regards,

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] List of genomes

2013-05-06 Thread Lee Katz

Hi, I am wondering what I can do to add more genomes to this list
(hopefully this image will make it through the mailing list).  I am
importing fastq files from a MiSeq run.  We are using a modified version of
Galaxy called Tron.

Everything I see on the wiki is about adding custom reference genome
assemblies, but I don't understand what to do about adding a species
without the reference assembly. Without the correct species, I am forced to
use the option unspecified (?)

Thanks for your help.

[image: Inline image 3]


-- 
Lee Katz, Ph.D.
image.png___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] LWR runner configuration for shared folder in cluster

2013-05-06 Thread John Chilton

On Mon, May 6, 2013 at 9:35 AM, Krampis, Konstantinos kkram...@jcvi.org wrote:
 Hi John,

   thanks for the quick reply! It is actually a virtual cluster using virtual 
 machines
 with VirtualBox, which are brought up via Vagrant (www.vagrantup.com) - that
 also sets up the shared filesystem. I chose LWR for the reason that it not a 
 real
 cluster, and needed something lightweight instead of having to stand up SGE 
 etc.

Fascinating. Unless you don't have root on the physical server, I
would still really recommend SGE. I cannot see any reason it shouldn't
work on virtual servers, it  works with CloudMan and there the master
and workers are both virtualized.


   I am very interested in the 3rd option you present in your response below. 
 I am
 still confused though that if Galaxy and LWR doe not use shared filesystem, 
 how does
 Galaxy know when the job is finished, or where to find the output files ?

 Or does the output gets piped back to Galaxy ? That would work (all virtual 
 machines
 are on the same box, so no latency), as long as the output gets deleted by 
 LWR after
 being send to Galaxy (to save space in the worker virtual machines).

The LWR job runner communicates with the LWR client over HTTP to work
out these details. The job description is rewritten with new file
paths (input, output, tool files, config files), staged and submitted
to the remote LWR server which executes it and tracks its progress.
Meanwhile, the LWR job runner polls the LWR server waiting for
completion of the job. Upon completion, it downloads the outputs and
places them in the paths Galaxy is expecting.

This is quite different than the other job runners, so there are all
sorts of assumptions tools may make that could cause this to fail - so
if you are going to run more than a handful of tools this way I would
really recommend the SGE or otherwise you are going to spend time
tracking down little annoying problems I suspect.

As for the option of not staging the files, I am going to make the
rookie mistake of suggesting that this is straight forward to
implement in your case - as long as all of the files are available on
the remote system. I have created an issue for this, I don't know when
I will get to this but you can subscribe to the issue to stay in the
loop:

https://bitbucket.org/jmchilton/lwr/issue/8/allow-alternate-staging-rules

To outline how to just disable staging, I think most of the changes
will need to be made to






   If you could provide some pointers to documentation / code, or explain how
 could be set up it would be of great help !

  many thanks,

 Ntino



 
 From: jmchil...@gmail.com [jmchil...@gmail.com] On Behalf Of John Chilton 
 [chil...@msi.umn.edu]
 Sent: Monday, May 06, 2013 9:50 AM
 To: Krampis, Konstantinos
 Cc: galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] LWR runner configuration for shared folder in 
 cluster

 Hello Dr. Krampis,

   At the present, the LWR is most valuable when there is not a shared
 file system between the server executing the jobs and the server
 hosting Galaxy. In this case you seem to have a shared filesystem so I
 would think setting up something like sun grid engine to manage the
 jobs and using the DRMAA job runner would be the best route forward.

   The LWR and the corresponding galaxy job runner will coordinate to
 stage jobs, but the upshot is that the LWR should be the only thing
 writing to its staging directory. In this case you have configured the
 LWR and Galaxy to both use the same directory. You should change this
 configuration immediately, I am worried the LWR is going to delete or
 overwrite files maintained by Galaxy. I am sorry for the confusion, I
 will update the documentation to explicitly warn against this.

   If you still feel there is a compelling reason to use the LWR in
 this situation, you will just want to change the staging_directory in
 the LWR configuration to something else. It has long been on my TODO
 list to allow one to disable (or selectively disable by path
 regex/globs) file staging with the LWR, it seems like that would what
 would also help in your situation. Let me know if that is of interest
 to you.

 -John


 On Mon, May 6, 2013 at 8:32 AM, Krampis, Konstantinos kkram...@jcvi.org 
 wrote:
 Hi all,

   I am trying to set up a Galaxy cluster using the LWR runner. The nodes have
 a shared filesystem and in universe.wsgi this parameter is set :

 job_working_directory = /mnt/shared
 ...
 clustalw = lwr://http://192.168.33.12:8913
 

 this folder has been chown-ed to the galaxy user, and also is a+w,
 while it has been verified that can been read / written by ssh-ing to
 each node of the cluster. The sticky bit is set.

 When I try to run jobs (I used clustalw as example) there seems to be
 confusion between where Galaxy puts files and where LWR tries to read
 them from. Here are two setups that error out:




 1). When in server.ini for LWR the following is set as:

Re: [galaxy-dev] LWR runner configuration for shared folder in cluster

2013-05-06 Thread John Chilton

Opps that last message got sent before I meant to.

I meant to finish saying that this file would need to be changed:

https://bitbucket.org/galaxy/galaxy-central/src/default/lib/galaxy/jobs/runners/lwr.py#cl-80

queue_job() will likely need to skip everything to do with the
FileStager class and just call client.setup() directly. You will want
to submit the original command-line, not a rewritten one. Also in
finish job, I think you will just want to skip the for loops with the
downloads.

Thanks,
-John

On Mon, May 6, 2013 at 11:55 AM, John Chilton chil...@msi.umn.edu wrote:
On Mon, May 6, 2013 at 9:35 AM, Krampis, Konstantinos kkram...@jcvi.org
wrote:
Hi John,

thanks for the quick reply! It is actually a virtual cluster using virtual
machines
with VirtualBox, which are brought up via Vagrant (www.vagrantup.com) - that
also sets up the shared filesystem. I chose LWR for the reason that it not a
real
cluster, and needed something lightweight instead of having to stand up SGE
etc.

Fascinating. Unless you don't have root on the physical server, I
would still really recommend SGE. I cannot see any reason it shouldn't
work on virtual servers, it works with CloudMan and there the master
and workers are both virtualized.

I am very interested in the 3rd option you present in your response below.
I am
still confused though that if Galaxy and LWR doe not use shared filesystem,
how does
Galaxy know when the job is finished, or where to find the output files ?

Or does the output gets piped back to Galaxy ? That would work (all
virtual machines
are on the same box, so no latency), as long as the output gets deleted by
LWR after
being send to Galaxy (to save space in the worker virtual machines).

The LWR job runner communicates with the LWR client over HTTP to work
out these details. The job description is rewritten with new file
paths (input, output, tool files, config files), staged and submitted
to the remote LWR server which executes it and tracks its progress.
Meanwhile, the LWR job runner polls the LWR server waiting for
completion of the job. Upon completion, it downloads the outputs and
places them in the paths Galaxy is expecting.

This is quite different than the other job runners, so there are all
sorts of assumptions tools may make that could cause this to fail - so
if you are going to run more than a handful of tools this way I would
really recommend the SGE or otherwise you are going to spend time
tracking down little annoying problems I suspect.

As for the option of not staging the files, I am going to make the
rookie mistake of suggesting that this is straight forward to
implement in your case - as long as all of the files are available on
the remote system. I have created an issue for this, I don't know when
I will get to this but you can subscribe to the issue to stay in the
loop:

https://bitbucket.org/jmchilton/lwr/issue/8/allow-alternate-staging-rules

To outline how to just disable staging, I think most of the changes
will need to be made to

If you could provide some pointers to documentation / code, or explain how
could be set up it would be of great help !

many thanks,

Ntino

From: jmchil...@gmail.com [jmchil...@gmail.com] On Behalf Of John Chilton
[chil...@msi.umn.edu]
Sent: Monday, May 06, 2013 9:50 AM
To: Krampis, Konstantinos
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] LWR runner configuration for shared folder in
cluster

Hello Dr. Krampis,

At the present, the LWR is most valuable when there is not a shared
file system between the server executing the jobs and the server
hosting Galaxy. In this case you seem to have a shared filesystem so I
would think setting up something like sun grid engine to manage the
jobs and using the DRMAA job runner would be the best route forward.

The LWR and the corresponding galaxy job runner will coordinate to
stage jobs, but the upshot is that the LWR should be the only thing
writing to its staging directory. In this case you have configured the
LWR and Galaxy to both use the same directory. You should change this
configuration immediately, I am worried the LWR is going to delete or
overwrite files maintained by Galaxy. I am sorry for the confusion, I
will update the documentation to explicitly warn against this.

If you still feel there is a compelling reason to use the LWR in
this situation, you will just want to change the staging_directory in
the LWR configuration to something else. It has long been on my TODO
list to allow one to disable (or selectively disable by path
regex/globs) file staging with the LWR, it seems like that would what
would also help in your situation. Let me know if that is of interest
to you.

-John

On Mon, May 6, 2013 at 8:32 AM, Krampis, Konstantinos kkram...@jcvi.org
wrote:
Hi all,

I am trying to set up a Galaxy cluster using the LWR runner.

Re: [galaxy-dev] Conditional tool options based on input file format?

On Fri, Mar 15, 2013 at 2:50 PM, Björn Grüning
bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Hi Peter,

  I've looked over the wiki and as far as I can see, 'when' tags used inside
  'conditional' tags only work on another input variable's values.
 
  I would like to be able to do is a conditional based on the file format
  of an input file. For instance, a tool might take XML or tabular files,
  and need to show additional parameters for one input type but not the
  other.
 
  Does this enhancement idea seem useful?
 
  Peter

 Hello again,

 I'd still like to be able to show different tool parameters for different
 input file formats - is anyone else interested in using or working on
 this?

 Yes! Im using that for a few of my tools, but currently the user need to
 choose the file format, that is suboptimal :(

Thank Bjoern  John for your comments back in March.
I've just filed a Trello card for this:
https://trello.com/card/-/506338ce32ae458f6d15e4b3/837

This was prompted by revisiting my Venn Diagram tool, where it
would be nice to offer a column number parameter when using
a tabular input file (but this is not appropriate for FASTA files):
http://toolshed.g2.bx.psu.edu/view/peterjc/venn_list

Peter

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Tophat problem

2013-05-06 Thread Zain A Alvi

Dear Sir or Madam,

I hope this reaches you well. Lately, I have been trying to use tophat and then 
use bowtie on Galaxy project to create an aligned BAM file. The original data 
came from a SRA file that I have acquired from the Japanese DNA Databank. This 
SRA was then converted to FASTQ using the tools available on Galaxy project. 
Now when I go under Tophat on Galaxy Project, I am unable to select the 
converted RNA-Seq FASTQ file. I was wondering, is there a specific format for 
the file to be in. Currently it is just a *.fastq file.  I am confused as to 
why I am not being able to select the FASTQ file.

Also if there is a guide on how to use Galaxy Project to create an aligned BAM 
file and then check for expression through Cufflinks package. I would really 
appreciate it.

Sincerely,

Zain
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Taxonomy fetching not working

2013-05-06 Thread Adam Brenner

Deeps,

It does not appear that you have installed taxBuilder on your system.
What commands did you use to compile it on your system?

If you type 'which taxBuilder' it should return no results (assuming
installed incorrectly). You will need to 'make clean' then 'make all'
then manually move the binary files it produces into some
environmental $PATH.


Hope that helps,

-Adam

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebre...@uci.edu


On Fri, May 3, 2013 at 3:58 PM, Deepthi Thomas Kannanayakal 
dttho...@ucalgary.ca wrote:

 Hi Galaxy team,

 I am having problems with taxonomy information fetching.(Metagenomic
 Analysses-Fetch taxonomic representation).

 According to the following link


 http://dev.list.galaxyproject.org/quot-Fetch-taxonomic-representation-quot-DOESN-T-WORK-td4565489.html

 I have installed libraries from https://bitbucket.org/natefoo/taxonomy
 I have installed them in my local directory(not in /usr/local).  But I am
 still having problem

 /bin/sh: taxBuilder: command not found

 What am I missing here?

 Thanks,

 Deeps

 *Deepthi Theresa Thomas Kannanayakal

 Bioinformatics Database Specialist
 Sun Visual Genomic Centre,
 Suite 1150,  Health Science Centre
 3330 Hospital Drive NW
 Department of Biochemistry and Molecular Biology
  Faculty of Medicine,  University of Calgary
 T2N 4N1
 Calgary,  Alberta,  Canada**
 *
 *Ph: (403) - 210 6661   Fax: (403) - 210 9538*

 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Hi Team,

I urgently need to resize the data partition on my galaxy cloudman instance 
without loosing anything.  I notice that there a gui on the cloudman admin page 
that will allow me to increase it to 1000GB.  If I use that will it restart all 
of the processes?  I need to maintain this as a functional instance since it 
needs to be available for use tomorrow at 8:00 AM and I do not have the time to 
rebuild it.  The gui says it will create a snapshot.  The following is what it 
says:

Expand Disk Space
Through this form you may increase the disk space available to Galaxy. All of 
the cluster services (but not the cluster) WILL BE STOPPED until the new disk 
is ready, at which point they will all be restarted. This may result in Galaxy 
jobs that are currently running to fail. Note that the new disk size must be 
larger than the current disk size.

During this process, a snapshot of your data volume will be created, which can 
optionally be left in your account. If you decide to leave the snapshot for 
reference, you may also provide a brief note that will later be visible in the 
snapshot's description. New Disk Size (minimum 100GB, maximum 1000GB):

OK
Note (optional):

 or delete the created snapshot after filesystem resizing? If checked, the 
created snapshot will not be kept

What is the likelihood that this will be successful?  Has anyone utilized it?  
Is there another way to resize the /mnt/galaxyData/ and /mnt/galaxyIndices 
partitions that will allow me to do what I need to?

Thanks,

Iry Witham


The information in this email, including attachments, may be confidential and 
is intended solely for the addressee(s). If you believe you received this email 
by mistake, please notify the sender by return email as soon as possible.
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Hi Iry,

This should work, I use it regularly.  Just make sure you're not ssh'ed in
and 'in' /mnt/galaxyData or anything like that.  There's not really another
way to do this that I'd recommend trying if you need the instance to
continue working for something tomorrow.

Let me know if you have any issues and I can help.

-Dannon


On Mon, May 6, 2013 at 1:54 PM, Iry Witham iry.wit...@jax.org wrote:

  Hi Team,

  I urgently need to resize the data partition on my galaxy cloudman
 instance without loosing anything.  I notice that there a gui on the
 cloudman admin page that will allow me to increase it to 1000GB.  If I use
 that will it restart all of the processes?  I need to maintain this as a
 functional instance since it needs to be available for use tomorrow at 8:00
 AM and I do not have the time to rebuild it.  The gui says it will create a
 snapshot.  The following is what it says:

  Expand Disk Space
 Through this form you may increase the disk space available to Galaxy. All
 of the cluster services (but not the cluster) WILL BE STOPPED until the new
 disk is ready, at which point they will all be restarted. This may result
 in Galaxy jobs that are currently running to fail. Note that the new disk
 size must be larger than the current disk size.

  During this process, a snapshot of your data volume will be created,
 which can optionally be left in your account. If you decide to leave the
 snapshot for reference, you may also provide a brief note that will later
 be visible in the snapshot's description. New Disk Size (minimum 100GB,
 maximum 1000GB):

  OK
 Note (optional):

   or delete the created snapshot after filesystem resizing? If checked,
 the created snapshot will not be kept

  What is the likelihood that this will be successful?  Has anyone
 utilized it?  Is there another way to resize the /mnt/galaxyData/
 and /mnt/galaxyIndices partitions that will allow me to do what I need to?

  Thanks,

  Iry Witham

  The information in this email, including attachments, may be
 confidential and is intended solely for the addressee(s). If you believe
 you received this email by mistake, please notify the sender by return
 email as soon as possible.

 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Published Workflows missing from All Workflows

2013-05-06 Thread Jeremy Goecks

 i.e. Published Workflows are hidden or separated away from the others.
 This appears to be a design choice (like shared datasets they must be
 imported before use), but if so this could be clarified as follows:

This was a very old design choice that will be revisited in the (hopefully 
near) future. I see no reason that published workflows need to be imported 
before use.

J.
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool Shed Request - sanity check uploads?

On Mon, May 6, 2013 at 9:55 AM, Peter Cock p.j.a.c...@googlemail.com wrote:
 On May 2, 2013, at 12:54 PM, Peter Cock wrote:

 Another even simpler change (which would also make a big
 difference) is to actually have the name of the current
 repository on the Upload a single file or a tarball page.

 Thanks,

 On Thu, May 2, 2013 at 9:32 PM, Greg Von Kuster g...@bx.psu.edu wrote:
 Hi Peter,

 This fix is now running on the test tool shed.  Thanks for pointing out this 
 flaw!

 Greg Von Kuster

 Great - that was an easy fix but should cut down on the number
 of accidental commits due to human error. Let's see how long it
 is before someone else messes up (or I do it again - grin) and
 if something more advanced is needed like my first suggestions.

Well that didn't take long - I just accidentally stuffed up an upload
to http://toolshed.g2.bx.psu.edu/view/peterjc/venn_list - the sooner
the quick fix to the upload form makes it to the live Tool Shed the
better.

I'd also urge the related issue of visually distinguishing the main
and test Tool Sheds to be prioritised:
https://trello.com/card/-/506338ce32ae458f6d15e4b3/817

Regards,

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Improving the change revision view on the Tool Shed

Hello all,

I've noticed a couple of weaknesses in the current revision view
for the Tool Shed.

First, the commit comment is not shown - this would make sense
next to the revision number and above the diffs.

Second, there is no easy way to jump to the next/prev revision,
you must first go back to the View change log page. Perhaps
simple prev/next links would work well, or as used elsewhere
in the Tool Shed a revision drop down control [*]?

Regards,

Peter

[*] Use with caution, I've found this focus issue quite infuriating:
https://trello.com/card/-/506338ce32ae458f6d15e4b3/821
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Hi Dannon,

I will give it a shot.  Are you familiar with the load balancer and how to make 
it work?  Or the auto scaling feature?

Thanks,

Iry

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 2:06 PM
To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Hi Iry,

This should work, I use it regularly.  Just make sure you're not ssh'ed in and 
'in' /mnt/galaxyData or anything like that.  There's not really another way to 
do this that I'd recommend trying if you need the instance to continue working 
for something tomorrow.

Let me know if you have any issues and I can help.

-Dannon


On Mon, May 6, 2013 at 1:54 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
Hi Team,

I urgently need to resize the data partition on my galaxy cloudman instance 
without loosing anything.  I notice that there a gui on the cloudman admin page 
that will allow me to increase it to 1000GB.  If I use that will it restart all 
of the processes?  I need to maintain this as a functional instance since it 
needs to be available for use tomorrow at 8:00 AM and I do not have the time to 
rebuild it.  The gui says it will create a snapshot.  The following is what it 
says:

Expand Disk Space
Through this form you may increase the disk space available to Galaxy. All of 
the cluster services (but not the cluster) WILL BE STOPPED until the new disk 
is ready, at which point they will all be restarted. This may result in Galaxy 
jobs that are currently running to fail. Note that the new disk size must be 
larger than the current disk size.

During this process, a snapshot of your data volume will be created, which can 
optionally be left in your account. If you decide to leave the snapshot for 
reference, you may also provide a brief note that will later be visible in the 
snapshot's description. New Disk Size (minimum 100GB, maximum 1000GB):

OK
Note (optional):

 or delete the created snapshot after filesystem resizing? If checked, the 
created snapshot will not be kept

What is the likelihood that this will be successful?  Has anyone utilized it?  
Is there another way to resize the /mnt/galaxyData/ and /mnt/galaxyIndices 
partitions that will allow me to do what I need to?

Thanks,

Iry Witham


The information in this email, including attachments, may be confidential and 
is intended solely for the addressee(s). If you believe you received this email 
by mistake, please notify the sender by return email as soon as possible.

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


The information in this email, including attachments, may be confidential and 
is intended solely for the addressee(s). If you believe you received this email 
by mistake, please notify the sender by return email as soon as possible.
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Usability bug/inconsistency with workflows versus tools

I seem to be in feedback mode: another little rough edge in Galaxy:

When using a tool in Galaxy, at the end of the parameters there is a nice
eye catching blue button labelled execute.

When running a workflow in Galaxy, the equivalent button is plain grey
and reads Run workflow. I think it would be clearer if this was the same
blue style, and perhaps also labelled execute.

Regards,

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Embedding a help text in a Workflow?

Hello all,

I've recently started sharing some sample workflows on the Galaxy Tool Shed,
http://toolshed.g2.bx.psu.edu/view/peterjc/rxlr_venn_workflow
http://toolshed.g2.bx.psu.edu/view/peterjc/secreted_protein_workflow

When writing a Galaxy tool we have a lot of flexibility within the help
tag to describe the tool, its output, and relevant references using some
nice markup language for things like bold fonts and tables.

I would very much like something similar for embedding README or
help text into a workflow - which again should be shown to the user,
both on the 'View workflow action' but also under the Run workflow /
Execute button when actually running a workflow.

Am I overlooking something or is the only workflow level description
the plain text 'Annotation / Notes' field?

Thanks,

Peter

P.S. I'm not talking about the repository level README text which I
can use for a workflow repository - that is only really seen by the
Galaxy administrator (not by the users within the Galaxy web-interface)
and is not specific to each workflow.
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Hi Dannon,

I have attempted the resizing.  I clicked on the disk and set it for 1000GB.  
The instance restarted, but the volume remains at 100GB.  Is this due to the 
lack of disk space?  I don't see a snapshot when I look at the AWS console 
after the restart.  Any ideas?

Thanks,

IRy

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 2:06 PM
To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Hi Iry,

This should work, I use it regularly.  Just make sure you're not ssh'ed in and 
'in' /mnt/galaxyData or anything like that.  There's not really another way to 
do this that I'd recommend trying if you need the instance to continue working 
for something tomorrow.

Let me know if you have any issues and I can help.

-Dannon


On Mon, May 6, 2013 at 1:54 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
Hi Team,

I urgently need to resize the data partition on my galaxy cloudman instance 
without loosing anything.  I notice that there a gui on the cloudman admin page 
that will allow me to increase it to 1000GB.  If I use that will it restart all 
of the processes?  I need to maintain this as a functional instance since it 
needs to be available for use tomorrow at 8:00 AM and I do not have the time to 
rebuild it.  The gui says it will create a snapshot.  The following is what it 
says:

Expand Disk Space
Through this form you may increase the disk space available to Galaxy. All of 
the cluster services (but not the cluster) WILL BE STOPPED until the new disk 
is ready, at which point they will all be restarted. This may result in Galaxy 
jobs that are currently running to fail. Note that the new disk size must be 
larger than the current disk size.

During this process, a snapshot of your data volume will be created, which can 
optionally be left in your account. If you decide to leave the snapshot for 
reference, you may also provide a brief note that will later be visible in the 
snapshot's description. New Disk Size (minimum 100GB, maximum 1000GB):

OK
Note (optional):

 or delete the created snapshot after filesystem resizing? If checked, the 
created snapshot will not be kept

What is the likelihood that this will be successful?  Has anyone utilized it?  
Is there another way to resize the /mnt/galaxyData/ and /mnt/galaxyIndices 
partitions that will allow me to do what I need to?

Thanks,

Iry Witham


The information in this email, including attachments, may be confidential and 
is intended solely for the addressee(s). If you believe you received this email 
by mistake, please notify the sender by return email as soon as possible.

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


The information in this email, including attachments, may be confidential and 
is intended solely for the addressee(s). If you believe you received this email 
by mistake, please notify the sender by return email as soon as possible.
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Usability bug/inconsistency with workflows versus tools

Thanks for the suggestion -- this style should definitely be applied in
both places and I've done so in 9659:bcb6e71eda61.

-Dannon


On Mon, May 6, 2013 at 2:30 PM, Peter Cock p.j.a.c...@googlemail.comwrote:

 I seem to be in feedback mode: another little rough edge in Galaxy:

 When using a tool in Galaxy, at the end of the parameters there is a nice
 eye catching blue button labelled execute.

 When running a workflow in Galaxy, the equivalent button is plain grey
 and reads Run workflow. I think it would be clearer if this was the same
 blue style, and perhaps also labelled execute.

 Regards,

 Peter
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

So you have a cluster with 9 instances and you're using ec2 load balancer
with all of them to divert traffic?


On Mon, May 6, 2013 at 3:56 PM, Iry Witham iry.wit...@jax.org wrote:

  I have setup a load balancer, but it shows that only 1 of 9 instances
 are 'In Service.  I have set the health check to:

  Ping: http:80/
 Timeout: 40 seconds
 Interval: 60 seconds
 Unhealthy threshold: 2
 Healty threshold: 10

  As far as autoscaling, I am not really interested since I see no real
 need at this point.

  Thanks,

  Iry

   From: Dannon Baker dannon.ba...@gmail.com
 Date: Monday, May 6, 2013 3:41 PM

 To: Iry Witham iry.wit...@jax.org
 Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData
 partition

   Great, I'm glad it's working, and I'm sorry you ran into this bug.

  As far as autoscaling, etc., what are you interested in?  I should be
 able to help out with just about anything related to Cloudman.


 On Mon, May 6, 2013 at 3:03 PM, Iry Witham iry.wit...@jax.org wrote:

  That seems to be working.

   From: Dannon Baker dannon.ba...@gmail.com
 Date: Monday, May 6, 2013 3:01 PM

 To: Iry Witham iry.wit...@jax.org
 Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData
 partition

   The first thing that comes to mind is a bug that existed in a
 particular version of cloudman regarding the max size allowable -- can you
 try 999GB?


 On Mon, May 6, 2013 at 2:57 PM, Iry Witham iry.wit...@jax.org wrote:

  Hi Dannon,

  I have attempted the resizing.  I clicked on the disk and set it for
 1000GB.  The instance restarted, but the volume remains at 100GB.  Is
 this due to the lack of disk space?  I don't see a snapshot when I look at
 the AWS console after the restart.  Any ideas?

  Thanks,

  IRy

   From: Dannon Baker dannon.ba...@gmail.com
 Date: Monday, May 6, 2013 2:06 PM
 To: Iry Witham iry.wit...@jax.org
 Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData
 partition

Hi Iry,

  This should work, I use it regularly.  Just make sure you're not
 ssh'ed in and 'in' /mnt/galaxyData or anything like that.  There's not
 really another way to do this that I'd recommend trying if you need the
 instance to continue working for something tomorrow.

  Let me know if you have any issues and I can help.

  -Dannon


 On Mon, May 6, 2013 at 1:54 PM, Iry Witham iry.wit...@jax.org wrote:

  Hi Team,

  I urgently need to resize the data partition on my galaxy cloudman
 instance without loosing anything.  I notice that there a gui on the
 cloudman admin page that will allow me to increase it to 1000GB.  If I use
 that will it restart all of the processes?  I need to maintain this as a
 functional instance since it needs to be available for use tomorrow at 8:00
 AM and I do not have the time to rebuild it.  The gui says it will create a
 snapshot.  The following is what it says:

  Expand Disk Space
 Through this form you may increase the disk space available to Galaxy.
 All of the cluster services (but not the cluster) WILL BE STOPPED until the
 new disk is ready, at which point they will all be restarted. This may
 result in Galaxy jobs that are currently running to fail. Note that the new
 disk size must be larger than the current disk size.

  During this process, a snapshot of your data volume will be created,
 which can optionally be left in your account. If you decide to leave the
 snapshot for reference, you may also provide a brief note that will later
 be visible in the snapshot's description. New Disk Size (minimum 100GB,
 maximum 1000GB):

  OK
 Note (optional):

   or delete the created snapshot after filesystem resizing? If
 checked, the created snapshot will not be kept

  What is the likelihood that this will be successful?  Has anyone
 utilized it?  Is there another way to resize the /mnt/galaxyData/
 and /mnt/galaxyIndices partitions that will allow me to do what I need to?

  Thanks,

  Iry Witham

  The information in this email, including attachments, may be
 confidential and is intended solely for the addressee(s). If you believe
 you received this email by mistake, please notify the sender by return
 email as soon as possible.

 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


 The information in this email, including attachments, may be
 confidential and is intended solely for the addressee(s). If you believe
 you received this email by mistake, please notify the sender by return
 email as soon as possible.


 The information in this email, including

Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

I was not clear how the load balancer worked in this instance, but my goal was 
to improve performance.  I am not sure this was the correct use, but it seemed 
to work.

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 4:12 PM
To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

So you have a cluster with 9 instances and you're using ec2 load balancer with 
all of them to divert traffic?

On Mon, May 6, 2013 at 3:56 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
I have setup a load balancer, but it shows that only 1 of 9 instances are 'In 
Service.  I have set the health check to:

Ping: http:80/
Timeout: 40 seconds
Interval: 60 seconds
Unhealthy threshold: 2
Healty threshold: 10

As far as autoscaling, I am not really interested since I see no real need at 
this point.

Thanks,

Iry

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 3:41 PM

To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Great, I'm glad it's working, and I'm sorry you ran into this bug.

As far as autoscaling, etc., what are you interested in?  I should be able to 
help out with just about anything related to Cloudman.

On Mon, May 6, 2013 at 3:03 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
That seems to be working.

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 3:01 PM

To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

The first thing that comes to mind is a bug that existed in a particular 
version of cloudman regarding the max size allowable -- can you try 999GB?

On Mon, May 6, 2013 at 2:57 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
Hi Dannon,

I have attempted the resizing.  I clicked on the disk and set it for 1000GB.  
The instance restarted, but the volume remains at 100GB.  Is this due to the 
lack of disk space?  I don't see a snapshot when I look at the AWS console 
after the restart.  Any ideas?

Thanks,

IRy

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 2:06 PM
To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Hi Iry,

This should work, I use it regularly.  Just make sure you're not ssh'ed in and 
'in' /mnt/galaxyData or anything like that.  There's not really another way to 
do this that I'd recommend trying if you need the instance to continue working 
for something tomorrow.

Let me know if you have any issues and I can help.

-Dannon

On Mon, May 6, 2013 at 1:54 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
Hi Team,

I urgently need to resize the data partition on my galaxy cloudman instance 
without loosing anything.  I notice that there a gui on the cloudman admin page 
that will allow me to increase it to 1000GB.  If I use that will it restart all 
of the processes?  I need to maintain this as a functional instance since it 
needs to be available for use tomorrow at 8:00 AM and I do not have the time to 
rebuild it.  The gui says it will create a snapshot.  The following is what it 
says:

Expand Disk Space
Through this form you may increase the disk space available to Galaxy. All of 
the cluster services (but not the cluster) WILL BE STOPPED until the new disk 
is ready, at which point they will all be restarted. This may result in Galaxy 
jobs that are currently running to fail. Note that the new disk size must be 
larger than the current disk size.

During this process, a snapshot of your data volume will be created, which can 
optionally be left in your account. If you decide to leave the snapshot for 
reference, you may also provide a brief note that will later be visible in the 
snapshot's description. New Disk Size (minimum 100GB, maximum 1000GB):

OK
Note (optional):

 or delete the created snapshot after filesystem resizing? If checked, the 
created snapshot will not be kept

What is the likelihood that this will be successful?  Has anyone utilized it?  
Is there another way to resize the /mnt/galaxyData/ and /mnt/galaxyIndices 
partitions that will allow me to do what I need

Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Ahh, ok.  So, unfortunately Cloudman isn't designed to be used with a load
balancer (at least at this point).  In any given cluster you have exactly
one master node and all of the rest are workers (which do not run web
processes, hence the 1/9 health).

What was slow that you were trying to improve the performance of?  One
thing I'd definitely recommend for head node performance, if you do have
extra worker nodes, would be to go to the admin panel
(your_ec2_instance/cloud/admin) and disable job running on the head node
using the Set master to not run jobs option.



On Mon, May 6, 2013 at 4:15 PM, Iry Witham iry.wit...@jax.org wrote:

  I was not clear how the load balancer worked in this instance, but my
 goal was to improve performance.  I am not sure this was the correct use,
 but it seemed to work.

   From: Dannon Baker dannon.ba...@gmail.com
 Date: Monday, May 6, 2013 4:12 PM

 To: Iry Witham iry.wit...@jax.org
 Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData
 partition

   So you have a cluster with 9 instances and you're using ec2 load
 balancer with all of them to divert traffic?


 On Mon, May 6, 2013 at 3:56 PM, Iry Witham iry.wit...@jax.org wrote:

  I have setup a load balancer, but it shows that only 1 of 9 instances
 are 'In Service.  I have set the health check to:

  Ping: http:80/
 Timeout: 40 seconds
 Interval: 60 seconds
 Unhealthy threshold: 2
 Healty threshold: 10

  As far as autoscaling, I am not really interested since I see no real
 need at this point.

  Thanks,

  Iry

   From: Dannon Baker dannon.ba...@gmail.com
 Date: Monday, May 6, 2013 3:41 PM

 To: Iry Witham iry.wit...@jax.org
 Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData
 partition

   Great, I'm glad it's working, and I'm sorry you ran into this bug.

  As far as autoscaling, etc., what are you interested in?  I should be
 able to help out with just about anything related to Cloudman.


 On Mon, May 6, 2013 at 3:03 PM, Iry Witham iry.wit...@jax.org wrote:

  That seems to be working.

   From: Dannon Baker dannon.ba...@gmail.com
 Date: Monday, May 6, 2013 3:01 PM

 To: Iry Witham iry.wit...@jax.org
 Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData
 partition

   The first thing that comes to mind is a bug that existed in a
 particular version of cloudman regarding the max size allowable -- can you
 try 999GB?


 On Mon, May 6, 2013 at 2:57 PM, Iry Witham iry.wit...@jax.org wrote:

  Hi Dannon,

  I have attempted the resizing.  I clicked on the disk and set it for
 1000GB.  The instance restarted, but the volume remains at 100GB.  Is
 this due to the lack of disk space?  I don't see a snapshot when I look at
 the AWS console after the restart.  Any ideas?

  Thanks,

  IRy

   From: Dannon Baker dannon.ba...@gmail.com
 Date: Monday, May 6, 2013 2:06 PM
 To: Iry Witham iry.wit...@jax.org
 Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu
 Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData
 partition

Hi Iry,

  This should work, I use it regularly.  Just make sure you're not
 ssh'ed in and 'in' /mnt/galaxyData or anything like that.  There's not
 really another way to do this that I'd recommend trying if you need the
 instance to continue working for something tomorrow.

  Let me know if you have any issues and I can help.

  -Dannon


 On Mon, May 6, 2013 at 1:54 PM, Iry Witham iry.wit...@jax.org wrote:

  Hi Team,

  I urgently need to resize the data partition on my galaxy cloudman
 instance without loosing anything.  I notice that there a gui on the
 cloudman admin page that will allow me to increase it to 1000GB.  If I use
 that will it restart all of the processes?  I need to maintain this as a
 functional instance since it needs to be available for use tomorrow at 
 8:00
 AM and I do not have the time to rebuild it.  The gui says it will create 
 a
 snapshot.  The following is what it says:

  Expand Disk Space
 Through this form you may increase the disk space available to Galaxy.
 All of the cluster services (but not the cluster) WILL BE STOPPED until 
 the
 new disk is ready, at which point they will all be restarted. This may
 result in Galaxy jobs that are currently running to fail. Note that the 
 new
 disk size must be larger than the current disk size.

  During this process, a snapshot of your data volume will be created,
 which can optionally be left in your account. If you decide to leave the
 snapshot for reference, you may also provide a brief note that will later
 be visible in the snapshot's description. New Disk Size (minimum 100GB,
 maximum 1000GB):

  OK
 Note (optional):

   or delete the created snapshot after filesystem resizing? If
 checked, the created snapshot will not be kept

  What is the likelihood that this will be

Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

I did add nodes about the same time I setup load balancing so that was likely 
the fix.  I will definitely set the head node not to run jobs.

Thanks,
Iry

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 4:19 PM
To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Ahh, ok.  So, unfortunately Cloudman isn't designed to be used with a load 
balancer (at least at this point).  In any given cluster you have exactly one 
master node and all of the rest are workers (which do not run web processes, 
hence the 1/9 health).

What was slow that you were trying to improve the performance of?  One thing 
I'd definitely recommend for head node performance, if you do have extra worker 
nodes, would be to go to the admin panel (your_ec2_instance/cloud/admin) and 
disable job running on the head node using the Set master to not run jobs 
option.

On Mon, May 6, 2013 at 4:15 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
I was not clear how the load balancer worked in this instance, but my goal was 
to improve performance.  I am not sure this was the correct use, but it seemed 
to work.

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 4:12 PM

To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

So you have a cluster with 9 instances and you're using ec2 load balancer with 
all of them to divert traffic?

On Mon, May 6, 2013 at 3:56 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
I have setup a load balancer, but it shows that only 1 of 9 instances are 'In 
Service.  I have set the health check to:

Ping: http:80/
Timeout: 40 seconds
Interval: 60 seconds
Unhealthy threshold: 2
Healty threshold: 10

As far as autoscaling, I am not really interested since I see no real need at 
this point.

Thanks,

Iry

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 3:41 PM

To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Great, I'm glad it's working, and I'm sorry you ran into this bug.

As far as autoscaling, etc., what are you interested in?  I should be able to 
help out with just about anything related to Cloudman.

On Mon, May 6, 2013 at 3:03 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
That seems to be working.

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 3:01 PM

To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

The first thing that comes to mind is a bug that existed in a particular 
version of cloudman regarding the max size allowable -- can you try 999GB?

On Mon, May 6, 2013 at 2:57 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
Hi Dannon,

I have attempted the resizing.  I clicked on the disk and set it for 1000GB.  
The instance restarted, but the volume remains at 100GB.  Is this due to the 
lack of disk space?  I don't see a snapshot when I look at the AWS console 
after the restart.  Any ideas?

Thanks,

IRy

From: Dannon Baker dannon.ba...@gmail.commailto:dannon.ba...@gmail.com
Date: Monday, May 6, 2013 2:06 PM
To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu 
galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition

Hi Iry,

This should work, I use it regularly.  Just make sure you're not ssh'ed in and 
'in' /mnt/galaxyData or anything like that.  There's not really another way to 
do this that I'd recommend trying if you need the instance to continue working 
for something tomorrow.

Let me know if you have any issues and I can help.

-Dannon

On Mon, May 6, 2013 at 1:54 PM, Iry Witham 
iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote:
Hi Team,

I urgently need to resize the data partition on my galaxy cloudman instance 
without loosing anything.  I notice that there a gui on the cloudman admin page 
that will allow me to increase it to 1000GB.  If I use that will it restart all 
of the processes?  I need to maintain this as a functional

Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData partition