date:20110926

Re: [galaxy-dev] picard error.

2011-09-26 Thread Ross

A fix for these warnings is in revision 6048 of galaxy-central -
again, thanks for pointing them out Michael.

For the record, the warnings were from the HSmetrics and GCbias tools
and arose because there is a user configurable maxheap parameter that
is not available by default - it is commented out in the tool form -
but it had not been commented out from the test - since it is not
enabled by default that generated a warning. The test parameter is now
also commented out by default and must be uncommented for sites
running into ram limits with very large datasets.

On Tue, Sep 27, 2011 at 8:04 AM, Ross  wrote:
> Michael,
>
> Thanks for pointing these warning errors out. I think they're harmless
> and can be ignored but I'll definitely take a look later today and
> check in a fix.
> As I recall, we had some discussion some months ago about whether to
> allow the user to set the size of the java heap and decided that it
> was not necessary - these messages suggest the parameter is still
> somewhere in the tool tests - oddly enough I can't see it but I'm
> seeing the same messages!
> I'll let you know what I find once I've had a chance to take a look.
>
>
> On Mon, Sep 26, 2011 at 11:06 PM, michael burrell (TOC)
>  wrote:
>> Good afternoon,
>>
>>
>>
>> When starting galaxy I find the following errors logged in the paster.log
>>
>>
>>
>> galaxy.tools DEBUG 2011-09-26 13:58:21,577 Loaded tool: PicardASMetrics 0.03
>>
>> galaxy.tools.test DEBUG 2011-09-26 13:58:21,625 Error in add_param for
>> maxheap: Unable to determine parameter type of test input 'maxheap'. Ensure
>> that the parameter exists and that any container groups are defined first.
>>
>>
>>
>> galaxy.tools DEBUG 2011-09-26 13:58:21,703 Loaded tool: PicardInsertSize
>> 0.3.0
>>
>> galaxy.tools.test DEBUG 2011-09-26 13:58:21,737 Error in add_param for
>> maxheap: Unable to determine parameter type of test input 'maxheap'. Ensure
>> that the parameter exists and that any container groups are defined first.
>>
>>
>>
>> Our galaxy version is.
>>
>>
>>
>> galaxy@jic55666:~/software/galaxy-central$ hg tip
>>
>> changeset:   6042:2469c53051ea
>>
>> tag: tip
>>
>> user:    Daniel Blankenberg 
>>
>> date:    Sun Sep 25 13:27:14 2011 -0400
>>
>> summary: Some random lines tool tweaks.
>>
>>
>>
>> Thank you for your assistance.
>>
>>
>>
>> Michael.
>>
>>
>>
>> 
>> Please note, that my email address is no longer
>> givenname.surn...@bbsrc.ac.uk. Please update your records to reflect my new
>> email address of givenname.surn...@nbi.ac.uk as shown on this email.
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>
>>  http://lists.bx.psu.edu/
>>
>
>
>
> --
> Ross Lazarus MBBS MPH;
> Associate Professor, Harvard Medical School;
> Director of Bioinformatics, Channing Lab; Tel: +1 617 505 4850;
> Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
>



-- 
Ross Lazarus MBBS MPH;
Associate Professor, Harvard Medical School;
Director of Bioinformatics, Channing Lab; Tel: +1 617 505 4850;
Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Galaxy fronting multiple clusters

2011-09-26 Thread Luobin Yang

I have never setup a Galaxy instance fronting multiple clusters, but it's
something I would like to explore. I have a dedicated cluster to run Galaxy
jobs and I've got another a shared cluster to which I hope Galaxy can assign
jobs when the dedicated cluster is too busy.

>From my understanding of Galaxy, the tool runner for each tool is hardcoded
in the universe.ini file and if you do not configure a tool runner for a
tool, Galaxy will use the default tool runner, which is determined by the
default_cluster_job_runner parameter. I believe you can configure multiple
job runners for a specific tool under the [galaxy:tool_runners] in
universe.ini, for instance, for each cluster, you have a different tool
runner for a specific tool,  however, Galaxy probably will just use one of
them, most likely the last one. So cluster selection for a tool is
determined by the job runner, which is hard coded in the universe.ini file.
As a result, the running of a workflow is determined by the tools in the
workflow. If each tool in the workflow is configured to use the same
cluster, then the workflow is run on the same cluster, otherwise, it will
span multiple clusters.

I think if you can configure the machine that runs Galaxy instance to be the
submit host of multiple clusters, then it's possible to have Galaxy front
multiple clusters. For me, the biggest hurdle is how to let two clusters
having a shared storage space and configure a machine in one cluster to be
the submit host of another cluster.

Thanks,
Luoin

On Mon, Sep 19, 2011 at 9:44 AM, Ann Black  wrote:

> Hello -
>
> I am working on standing up our own galaxy installation.  We would like to
> have galaxy front multiple clusters, and I have some questions I was hoping
> someone could help with.
>
> 1)  From reading other forum posts on this subject, it seems I need to
> minimally do the following ... is this correct?:
>   A) have galaxy server w/ sge register as a job submitting host to the
> head node of each cluster
>   B) Configure multiple tool runners for each tool per remote cluster?
>
> 2) When galaxy would submit a job, how would a backend remote cluster be
> selected?  When running workflows, would the same cluster be used to run the
> entire workflow - or could the workflow then span remote clusters?
>
> 3) I am trying to understand some of the source code, where is the logic
> that would then dispatch the job and select a job runner to use?
>
> 4) Other advice or steps needed in order to get galaxy to front multiple
> remote clusters?
>
>
> Thanks so much,
>
> Ann
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>  http://lists.bx.psu.edu/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] cloud instance missing /opt/sge/default/common directory

2011-09-26 Thread Joseph Hargitai

It is there:

data_filesystems:
  galaxyData:
  - size: 200
vol_id: vol-5b059a30
services:
- service: SGE
- service: Postgres
- service: Galaxy
static_filesystems:
- filesystem: galaxyIndices
  size: 700
  snap_id: !!python/unicode 'snap-5b030634'
- filesystem: galaxyTools
  size: 2
  snap_id: !!python/unicode 'snap-7dad9712'



From: Enis Afgan [eaf...@emory.edu]
Sent: Monday, September 26, 2011 8:52 AM
To: Joseph Hargitai
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] cloud instance missing /opt/sge/default/common 
directory

Are you continuously running this instance and just restarting it or do you 
start a clean instance each time?
I'd suggest ensuring the persistent_data.yaml in your cluster's bucket is 
correct (see below), terminating this instance, and starting a new one with the 
same user data as used for this cluster. That way you should get a clean (i.e., 
standardized) start and that is something that should follow the process 
correctly.

persistent_data.yaml contains the data required to reestablish an existing 
cluster and must follow the following format but contain proper values 
(especially if you customized your cluster via custom snapshots). This file is 
in your cluster's bucket on S3 so you can download it, check it, and upload if 
anything was changed or got out of sync.

data_filesystems:
  galaxyData:
  - size: 50
vol_id: vol-909342f8
services:
- service: SGE
- service: Postgres
- service: Galaxy
static_filesystems:
- filesystem: galaxyIndices
  size: 700
  snap_id: !!python/unicode 'snap-5b030634'
- filesystem: galaxyTools
  size: 2
  snap_id: !!python/unicode 'snap-1688b978'



On Fri, Sep 23, 2011 at 1:09 PM, Joseph Hargitai 
mailto:joseph.hargi...@einstein.yu.edu>> wrote:
Hi,

the mount is no longer there:

ubuntu@ip-10-68-42-15:~$ more /etc/hosts
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
ubuntu@ip-10-68-42-15:~$ df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/sda1  15G  9.5G  5.6G  63% /
devtmpfs  7.3G  128K  7.3G   1% /dev
none  7.6G 0  7.6G   0% /dev/shm
none  7.6G   96K  7.6G   1% /var/run
none  7.6G 0  7.6G   0% /var/lock
none  7.6G 0  7.6G   0% /lib/init/rw
/dev/sdb  414G  201M  393G   1% /mnt
ubuntu@ip-10-68-42-15:~$ cd /mnt
ubuntu@ip-10-68-42-15:/mnt$ ls
cm  lost+found



ubuntu@ip-10-68-42-15:/mnt$ more /etc/fstab
# /etc/fstab: static file system information.
#
proc/proc   procnodev,no
exec,nosuid 0   0
/dev/sda1   /   xfs defaults
0   0
/dev/sdb/mntautodefaults,nobootwait,comment=cloudconfig0
0



From: Enis Afgan [eaf...@emory.edu]
Sent: Thursday, September 22, 2011 8:48 AM

To: Joseph Hargitai
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] cloud instance missing /opt/sge/default/common 
directory

Hi Joe,
And this is happening on a freshly booted instance (from a previously existing 
cluster) using the same AMI? The order of execution seems a bit odd, seeing 
Galaxy being removed before SGE is setup; SGE should be the first thing that 
gets setup so I'm wondering...

If you log into the instance, what is in /etc/hosts? Does it match the instance 
DNS?

And if you try executing that same command (cd /opt/sge; ./inst_sge -m -x -auto 
/opt/sge/galaxyEC2.conf) by hand (as root), is any more info produced? Also, 
qmaster log should be available under /opt/sge/ge6 (or something like this) 
/default/spool/qmaster/ so please take a look there as well and see if more 
info is available.



On Thu, Sep 22, 2011 at 12:46 AM, Joseph Hargitai 
mailto:joseph.hargi...@einstein.yu.edu>> wrote:
the error is

'
[DEBUG] galaxy:139 2011-09-22 00:03:21,055: Galaxy UI does not seem to be 
accessible.
[DEBUG] master:1491 2011-09-22 00:03:21,055: S&S: SGE..Shut down; 
FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK; 
Galaxy..Starting;
[DEBUG] root:354 2011-09-22 00:03:24,724: Managing services: []
[INFO] galaxy:30 2011-09-22 00:03:24,724: Removing 'Galaxy' service
[INFO] galaxy:122 2011-09-22 00:03:24,724: Shutting down Galaxy...
[DEBUG] misc:511 2011-09-22 00:03:26,067: Successfully stopped Galaxy.
[DEBUG] root:354 2011-09-22 00:03:33,936: Managing services: []
[DEBUG] sge:61 2011-09-22 00:03:33,937: Unpacking SGE from 
'/opt/galaxy/pkg/ge6.2u5'
[DEBUG] sge:76 2011-09-22 00:03:33,937: Cleaning '/opt/sge' directory.
[DEBUG] sge:82 2011-09-22 00:03:34,117: Unpacking SGE to '/opt/sge'.
[INFO] sge:96 2011-09-22 00:03:35,5

[galaxy-dev] prevent downloaded files with .txt added

2011-09-26 Thread David Hoover

Does anyone know how to prevent the .txt suffix from being added to files 
downloaded from Galaxy?  For example, a fasta file is downloaded as 
xxx.fasta.txt, instead of xxx.fasta.

Wondering,
David Hoover
Helix Systems Staff
http://helix.nih.gov
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Setting http_proxy for Python

2011-09-26 Thread Matloob Khushi

Hi Guys

I am trying to install Galaxy on Red Hat Enterprise Linux Server release
5.7 (Tikanga). I have installed Python 2.6.  Our server is behind a
proxy that needs to be authenticated. I have set up the proxy for
Mercurial as described in
http://www.selenic.com/mercurial/hgrc.5#http-proxy

 

Now Mercurial is working and I have cloned the repository, however, when
I issue sh run.sh command I get the following error. Though I have set
the environment variable http_proxy by issuing export command on command
line: 

 

export HTTP_PROXY=UserName:myunip...@servername.edu.au:8080

I have also tried various export syntax such as:

export http_proxy=http://UserName:MyUniPass@ servername.edu.au:8080

 

I suspect Python is not reading the proxy setting to access internet.
Help please.

 

Some eggs are out of date, attempting to fetch...

Traceback (most recent call last):

  File "./scripts/fetch_eggs.py", line 30, in ?

c.resolve() # Only fetch eggs required by the config

  File "/opt/galaxy-dist/lib/galaxy/eggs/__init__.py", line 345, in
resolve

egg.resolve()

  File "/opt/galaxy-dist/lib/galaxy/eggs/__init__.py", line 195, in
resolve

return self.version_conflict( e.args[0], e.args[1] )

  File "/opt/galaxy-dist/lib/galaxy/eggs/__init__.py", line 226, in
version_conflict

r = pkg_resources.working_set.resolve( ( dist.as_requirement(), ),
env, egg.fetch )

  File "/opt/galaxy-dist/lib/pkg_resources.py", line 565, in resolve

raise DistributionNotFound(req)  # XXX put more info here

pkg_resources.DistributionNotFound: simplejson==2.1.1

Fetch failed.

 

Regards

Matloob

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] picard error.

2011-09-26 Thread michael burrell (TOC)

Good afternoon,

When starting galaxy I find the following errors logged in the paster.log

galaxy.tools DEBUG 2011-09-26 13:58:21,577 Loaded tool: PicardASMetrics 0.03
galaxy.tools.test DEBUG 2011-09-26 13:58:21,625 Error in add_param for maxheap: 
Unable to determine parameter type of test input 'maxheap'. Ensure that the 
parameter exists and that any container groups are defined first.

galaxy.tools DEBUG 2011-09-26 13:58:21,703 Loaded tool: PicardInsertSize 0.3.0
galaxy.tools.test DEBUG 2011-09-26 13:58:21,737 Error in add_param for maxheap: 
Unable to determine parameter type of test input 'maxheap'. Ensure that the 
parameter exists and that any container groups are defined first.

Our galaxy version is.

galaxy@jic55666:~/software/galaxy-central$ hg tip
changeset:   6042:2469c53051ea
tag: tip
user:Daniel Blankenberg 
date:Sun Sep 25 13:27:14 2011 -0400
summary: Some random lines tool tweaks.

Thank you for your assistance.

Michael.

-
Please note, that my email address is no longer givenname.surn...@bbsrc.ac.uk.  
Please update your records to reflect my new email address of 
givenname.surn...@nbi.ac.uk as shown on this email.___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] cloud instance missing /opt/sge/default/common directory

2011-09-26 Thread Enis Afgan

Are you continuously running this instance and just restarting it or do you
start a clean instance each time?
I'd suggest ensuring the persistent_data.yaml in your cluster's bucket is
correct (see below), terminating this instance, and starting a new one with
the same user data as used for this cluster. That way you should get a clean
(i.e., standardized) start and that is something that should follow the
process correctly.

persistent_data.yaml contains the data required to reestablish an existing
cluster and must follow the following format but contain proper values
(especially if you customized your cluster via custom snapshots). This file
is in your cluster's bucket on S3 so you can download it, check it, and
upload if anything was changed or got out of sync.

data_filesystems:
  galaxyData:
  - size: 50
vol_id: vol-909342f8
services:
- service: SGE
- service: Postgres
- service: Galaxy
static_filesystems:
- filesystem: galaxyIndices
  size: 700
  snap_id: !!python/unicode 'snap-5b030634'
- filesystem: galaxyTools
  size: 2
  snap_id: !!python/unicode 'snap-1688b978'



On Fri, Sep 23, 2011 at 1:09 PM, Joseph Hargitai <
joseph.hargi...@einstein.yu.edu> wrote:

>  Hi,
>
> the mount is no longer there:
>
> ubuntu@ip-10-68-42-15:~$ more /etc/hosts
> 127.0.0.1 localhost
>
> # The following lines are desirable for IPv6 capable hosts
> ::1 ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> ff02::3 ip6-allhosts
> ubuntu@ip-10-68-42-15:~$ df -h
> FilesystemSize  Used Avail Use% Mounted on
> /dev/sda1  15G  9.5G  5.6G  63% /
> devtmpfs  7.3G  128K  7.3G   1% /dev
> none  7.6G 0  7.6G   0% /dev/shm
> none  7.6G   96K  7.6G   1% /var/run
> none  7.6G 0  7.6G   0% /var/lock
> none  7.6G 0  7.6G   0% /lib/init/rw
> /dev/sdb  414G  201M  393G   1% /mnt
> ubuntu@ip-10-68-42-15:~$ cd /mnt
> ubuntu@ip-10-68-42-15:/mnt$ ls
> cm  lost+found
>
>
>
> ubuntu@ip-10-68-42-15:/mnt$ more /etc/fstab
> # /etc/fstab: static file system information.
> # 
>  > 
> proc/proc   proc
> nodev,no
> exec,nosuid 0   0
> /dev/sda1   /   xfs
> defaults
> 0   0
> /dev/sdb/mntautodefaults,nobootwait,comment=cloudconfig0
>
> 0
>
>
>  --
> *From:* Enis Afgan [eaf...@emory.edu]
> *Sent:* Thursday, September 22, 2011 8:48 AM
>
> *To:* Joseph Hargitai
> *Cc:* galaxy-dev@lists.bx.psu.edu
> *Subject:* Re: [galaxy-dev] cloud instance missing /opt/sge/default/common
> directory
>
>   Hi Joe,
> And this is happening on a freshly booted instance (from
> a previously existing cluster) using the same AMI? The order of execution
> seems a bit odd, seeing Galaxy being removed before SGE is setup; SGE should
> be the first thing that gets setup so I'm wondering...
>
>  If you log into the instance, what is in /etc/hosts? Does it match the
> instance DNS?
>
>  And if you try executing that same command (cd /opt/sge; ./inst_sge -m -x
> -auto /opt/sge/galaxyEC2.conf) by hand (as root), is any more info
> produced? Also, qmaster log should be available under /opt/sge/ge6 (or
> something like this) /default/spool/qmaster/ so please take a look there as
> well and see if more info is available.
>
>
>
> On Thu, Sep 22, 2011 at 12:46 AM, Joseph Hargitai <
> joseph.hargi...@einstein.yu.edu> wrote:
>
>>  the error is
>>
>> '
>> [DEBUG] galaxy:139 2011-09-22 00:03:21,055: Galaxy UI does not seem to
>> be accessible.
>> [DEBUG] master:1491 2011-09-22 00:03:21,055: S&S: SGE..Shut down;
>> FS-galaxyIndices..OK; FS-galaxyTools..OK; FS-galaxyData..OK; Postgres..OK;
>> Galaxy..Starting;
>> [DEBUG] root:354 2011-09-22 00:03:24,724: Managing services: []
>> [INFO] galaxy:30 2011-09-22 00:03:24,724: Removing 'Galaxy' service
>> [INFO] galaxy:122 2011-09-22 00:03:24,724: Shutting down Galaxy...
>> [DEBUG] misc:511 2011-09-22 00:03:26,067: Successfully stopped Galaxy.
>> [DEBUG] root:354 2011-09-22 00:03:33,936: Managing services: []
>> [DEBUG] sge:61 2011-09-22 00:03:33,937: Unpacking SGE from
>> '/opt/galaxy/pkg/ge6.2u5'
>> [DEBUG] sge:76 2011-09-22 00:03:33,937: Cleaning '/opt/sge' directory.
>> [DEBUG] sge:82 2011-09-22 00:03:34,117: Unpacking SGE to '/opt/sge'.
>> [INFO] sge:96 2011-09-22 00:03:35,557: Configuring SGE...
>> [DEBUG] sge:104 2011-09-22 00:03:35,558: Created SGE install template as
>> file '/opt/sge/galaxyEC2.conf'
>> [DEBUG] sge:112 2011-09-22 00:03:35,558: Setting up SGE.
>> [ERROR] misc:514 2011-09-22 00:03:35,651: Setting up SGE did not go
>> smoothly, running command 'cd /opt/sge; ./inst_sge -m -x -auto
>> /opt/sge/galaxyEC2.conf' returned code '2' and following stderr: '[: 359:
>> 11: unexpected operator
>> [: 359: 11: unexpected operator
>> [: 359: 11:

Re: [galaxy-dev] picard error.

Re: [galaxy-dev] Galaxy fronting multiple clusters

Re: [galaxy-dev] cloud instance missing /opt/sge/default/common directory

[galaxy-dev] prevent downloaded files with .txt added

[galaxy-dev] Setting http_proxy for Python

[galaxy-dev] picard error.

Re: [galaxy-dev] cloud instance missing /opt/sge/default/common directory

7 matches

Site Navigation

Mail list logo

Footer information