[galaxy-dev] error on fastq file while grooming or fastqc report or fastqc converter
Dear I am receiving the same error over and over again WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'coverage', using datatype from /mnt/galaxyData/tmp/tmpuTitVp. Whether I run - Fastqc: Fastqc QC http://ec2-107-21-142-14.compute-1.amazonaws.com/tool_runner?tool_id=fa stqc using FastQC from Babraham - FASTQ Groomer http://ec2-107-21-142-14.compute-1.amazonaws.com/tool_runner?tool_id=fa stq_groomer convert between various FASTQ quality formats Anybode any idea ? Kind Regards Yves ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
You don't need a full PBS server on that machine, just the client libraries and configuration. i.e. if you can run 'qstat' from machine A and have it return the queue on machine B, that should be all you need. We asked the IT guys they did just that :) now qsub works fine from machine A! But I still have a problem: They have no libdrmaa.so anywhere! What should I do? If I understood correctly it's part of the Fedstage DRMAA service provider, will it be enough installing it on our Galaxy server? Torque's syntax allows you to specify the server name right in the runner URL, but pbs_python links against libtorque, which is part of the torque client, which must be installed somewhere on the local system. Ok, thanks a lot! Best, L-A Best, L-A Le 30/01/2012 18:20, Nate Coraor a écrit : On Jan 30, 2012, at 12:07 PM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the leads! But setting DRMAA_LIBRARY_PATH means I'm in trouble since the libraries are on machine B which is maintained by our IT dept. I cannot access them from machine A. Is it a desperate situation? Will it work if I have a copy of those libs somewhere? :/ Hi L-A, The Galaxy server will need to be a submission host, so I believe it will have to have PBS Pro installed. If it has this, then the FedStage DRMAA library should be installable on the same host. It may be possible copy the libraries, although I don't know whether you'd be able to configure the server address without access to the directories in which the library will look for its configuration. --nate Best, L-A Le 30/01/2012 17:18, Nate Coraor a écrit : On Jan 30, 2012, at 7:37 AM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Hi L-A, Hrm, I've only used it with SGE, which uses an environment variable to define the cell location, and LSF, which I don't remember, but I assume it used the default. I think if you configure the PBS Pro client on the submission host and point DRMAA_LIBRARY_PATH at the correct libdrmaa, it will use your client configuration. There are other PBS Pro users on the list who can hopefully chime in with more details. --nate Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] LDAP authentification
Thanks nate,
I just would like to share others informations :
* I have added a VirtualHost, proxy and a DocumentRoot
* Consequently, RewriteRule have been modified :
Remplace
RewriteRule ^ galaxy/ (. *) http://ip:port $ 1 [P]
by
RewriteRule ^ / (. *) http://ip:port/ $ 1 [P]
It should be better for security.
Faithfully,
Sarah Maman
Nate Coraor a écrit :
On Feb 13, 2012, at 7:38 AM, Sarah Maman wrote:
Hello,
I managed to connect to Galaxy to LDAP ;-)
Three points were blocking for me:
* Being root of my virtual machine can carry out tests
* I confused login / password of two LDAP, so I thought that my authentication
method was not good while I was using the wrong password ...
* It is better not to go through a proxy
Hi Sarah,
Thanks very much for reporting back with your findings. This should be very
helpful for people who stumble on to similar problems in the future.
1 - Set configuration file of Galaxy: universe_wsgi.ini to delegate user
authentication to an upstream proxy Apache:
Users and Security
use_remote_user = True
remote_user_maildomain = toulouse.inra.fr
2 - Create a file type htaccess file named galaxy.conf (in / etc / httpd /
conf.d /):
For reasons of performance and safety, it is advisable not to use a. htaccess
but a galaxy.conf file in the main server configuration (Apache), because the
latter will be charged a once when the server starts. With an .htaccess file,
this file will be charged at each access.
RewriteEngine on
Location /galaxy
# Define the authentication method
AuthType Basic
AuthName Galaxy
AuthBasicProvider ldap
AuthLDAPURL ldap :/ / server URL: 389/...
AuthzLDAPAuthoritative off
Require valid-user
RequestHeader set REMOTE_USER %{AUTHENTICATE_uid}e
/ Location
RewriteRule ^ / $ galaxy / galaxy / [R]
RewriteRule ^ / galaxy / static / style / (. *) /
var/www/html/galaxy/static/june_2007_style/blue / $ 1 [L]
RewriteRule ^ / galaxy / static / scripts / (. *) /vVar / www / html / galaxy /
static / scripts / packed / $ 1 [L]
RewriteRule ^ / galaxy / static / (. *) / var / www / html / galaxy / static /
$ 1 [L]
RewriteRule ^ / galaxy / favicon.ico / var / www / html / galaxy / static /
favicon.ico [L]
RewriteRule ^ / galaxy / robots.txt / var / www / html / galaxy / static /
robots.txt [L]
RewriteRule ^ / galaxy (. *) http://ip:port $ 1 [P]
As Galaxy is not installed in root directory but in a galaxy directory (var /
www / html / galaxy /), so following changes are needed:
This is probably not a good idea. From the documentation:
Please note that Galaxy should never be located on disk inside Apache's
DocumentRoot. By default, this would expose all of Galaxy (including datasets)
to anyone on the web.
Galaxy is a proxied application and as such, only the static content like
javascript and images are served directly by Apache (and this is set up with
the RewriteRules), everything else is passed through to the Galaxy application
via a proxied http connection. Right now I could presumably use the URL
http://server/galaxy/galaxy-dist/database/files/000/dataset_1.dat to view a
dataset directly.
1 - Add a RewriteRule
2 - Do not go through a proxy
Can you clarify this? I'm a bit confused, since if you are connecting to
Apache to access Galaxy, you are going through a proxy.
3 - REMOTE_USER variable is AUTHENTICATE_uid ( AUTHENTICATE_ sAMAccountName for
Windows AD)
I've added this to the wiki page, thanks!
--nate
4 - To generate dynamic URLs, it is necessary to configure prefix in
universe_wsgi.ini :
[Filter: proxy-prefix]
use = egg: # prefix PasteDeploy
prefix = / galaxy
[App: main]
filter-with = proxy-prefix
cookie_path = / galaxy
If you are not root on the virtual machine, create a symlink from / etc / httpd
/ conf.d / to galaxy.conf
3 - Some useful checks
Verify Apache version and Apache modules because each directive must have an
associated module:
Directive → Related module (which mod_ldap)
AuthType → mod_auth_basic.so
AuthBasicProvider → mod_authnz_ldap and mod_authz_ldap
Rewrite (for proxy) → mod_rewrite.so
RequestHeader→ mod_headers
Check that the galaxy is installed on ldap using this command: ldapsearch-x-h LDAP URL :
port-b dc
When you make a modification in galaxy.conf, restart Apache (or graful).
In httpd.conf, so that access management is authorized by the file. #
# AccessFileName: The name of the file to look for in EACH directory
# For additional configuration directives. See also the AllowOverride
# Directive.
#
AccessFileName. Htaccess
Check: Chmod 777 galaxy.conf
4 - Finally, restart run.sh (sh run.sh )
Thanks A LOT for your help,
Sarah
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] error on fastq file while grooming or fastqc report or fastqc converter
Hello Yves, You have one or more entries in your datatypes_conf.xml file for a datatype named coverage These should be eliminated from your datatypes.conf.xml file because they are not valid datatypes (unless you have added proprietary datatypes to your Galaxy instance with this extension). They were originally in the datatypes.conf.xml.sample file for datatype indexers\, but datatype indexers have been eliminated from the Galaxy framework because datatype converters do the same thing. Greg Von Kuster On Feb 17, 2012, at 3:25 AM, Wetzels, Yves [JRDBE Extern] wrote: Dear I am receiving the same error over and over again “WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'coverage', using datatype from /mnt/galaxyData/tmp/tmpuTitVp.” Whether I run - Fastqc: Fastqc QC using FastQC from Babraham - FASTQ Groomer convert between various FASTQ quality formats Anybode any idea ? Kind Regards Yves ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?
On Thu, Feb 16, 2012 at 9:02 PM, Peter wrote: On Thu, Feb 16, 2012 at 6:42 PM, Chris wrote: Cool! Seems like a perfectly fine start. I guess you could grab the # of sequences from the dataset somehow (I'm guessing that is set somehow upon import into Galaxy). Yes, I should be able to get that from Galaxy's metadata if known - much like how the FASTQ splitter works. It only needs to be an estimate anyway - which is what I think Galaxy does for large files - if we get it wrong then rather than using n sub-jobs as suggested, we might use n+1 or n-1. Done, and it seems to be working nicely now. If we don't know the sequence count, I divide the file based on the total size in bytes - which avoids any extra IO. https://bitbucket.org/peterjc/galaxy-central/changeset/26a0c0aa776d Taking advantage of this I have switched the BLAST tools from saying split the query into batches of 500 sequences (which worked fine but only gave benefits if doing genome scale queries) to just split the query into four parts (which will be done based on the sequence count if known, or the file size if not). This way any multi-query BLAST will get divided and run in parallel, not just the larger jobs. This gives a nice improvement (over yesterday's progress) with small tasks like 10 query sequences against a big database like NR or NT. https://bitbucket.org/peterjc/galaxy-central/changeset/1fb89ae798be Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy and Cluster options
Hi Tanguy, You can set that at the very bottom of your universe_wsgi.ini file. I did it myself with Torque to set a different behavior for a couple of tools, it works fine. The related Wiki page is here: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster :) Best, L-A Le 17/02/2012 09:43, Tanguy LE CARROUR a écrit : Hi! Is there a way to specify parameters for the job submission on the cluster from the tool configuration file? Let's say I want tool A to be submitted with -pe smp 8 but tool B submitted with -l mem_free=500. Is it already supported? I could not find it in the wiki. ... or do I have to start digging in the code and try to implement it?! ^_^' Best regards, Tanguy ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy and Cluster options
Hi Tanguy, L-A is correct in that you can set using the tool id different cluster options for different tools. For example I submit different tools to different queues: default_cluster_job_runner = drmaa://-q all.q -V/ upload1 = drmaa://-q short.q -V/ tophat = drmaa://-q long.q -V/ cufflinks = drmaa://-q long.q -V/ But I don't think this is what you want, as this is an one time setting that you can't change at runtime. I think there was some talk about dynamic DRMAA options base on the input size and tool, but can't tell what is the status on that feature. Hope it helps, Carlos 2012/2/17 Louise-Amélie Schmitt louise-amelie.schm...@embl.de: Hi Tanguy, You can set that at the very bottom of your universe_wsgi.ini file. I did it myself with Torque to set a different behavior for a couple of tools, it works fine. The related Wiki page is here: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster :) Best, L-A Le 17/02/2012 09:43, Tanguy LE CARROUR a écrit : Hi! Is there a way to specify parameters for the job submission on the cluster from the tool configuration file? Let's say I want tool A to be submitted with -pe smp 8 but tool B submitted with -l mem_free=500. Is it already supported? I could not find it in the wiki. ... or do I have to start digging in the code and try to implement it?! ^_^' Best regards, Tanguy ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
If they used a package install, I think the DRMAA libs for torque are normally separate. For instance: http://pkgs.org/download/torque-drmaa If going from source, you can enable DRMAA compilation with --enable-drmaa; I don't recall if that is on by default (I don't think it is). chris On Feb 17, 2012, at 3:49 AM, Louise-Amélie Schmitt wrote: You don't need a full PBS server on that machine, just the client libraries and configuration. i.e. if you can run 'qstat' from machine A and have it return the queue on machine B, that should be all you need. We asked the IT guys they did just that :) now qsub works fine from machine A! But I still have a problem: They have no libdrmaa.so anywhere! What should I do? If I understood correctly it's part of the Fedstage DRMAA service provider, will it be enough installing it on our Galaxy server? Torque's syntax allows you to specify the server name right in the runner URL, but pbs_python links against libtorque, which is part of the torque client, which must be installed somewhere on the local system. Ok, thanks a lot! Best, L-A Best, L-A Le 30/01/2012 18:20, Nate Coraor a écrit : On Jan 30, 2012, at 12:07 PM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the leads! But setting DRMAA_LIBRARY_PATH means I'm in trouble since the libraries are on machine B which is maintained by our IT dept. I cannot access them from machine A. Is it a desperate situation? Will it work if I have a copy of those libs somewhere? :/ Hi L-A, The Galaxy server will need to be a submission host, so I believe it will have to have PBS Pro installed. If it has this, then the FedStage DRMAA library should be installable on the same host. It may be possible copy the libraries, although I don't know whether you'd be able to configure the server address without access to the directories in which the library will look for its configuration. --nate Best, L-A Le 30/01/2012 17:18, Nate Coraor a écrit : On Jan 30, 2012, at 7:37 AM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Hi L-A, Hrm, I've only used it with SGE, which uses an environment variable to define the cell location, and LSF, which I don't remember, but I assume it used the default. I think if you configure the PBS Pro client on the submission host and point DRMAA_LIBRARY_PATH at the correct libdrmaa, it will use your client configuration. There are other PBS Pro users on the list who can hopefully chime in with more details. --nate Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail
[galaxy-dev] Dynamic DRMAA options, was Re: Galaxy and Cluster options
This was recently brought up in a related thread by Nate, the code is in a pull request on bitbucket. Changing the subject to see if that rings any additional bells… Here's the original post by John Chilton and link to the fork: http://galaxy-development-list-archive.2308389.n4.nabble.com/Defining-Job-Runners-Dynamically-td4141945.html https://bitbucket.org/galaxy/galaxy-central/pull-request/12/dynamic-job-runners chris On Feb 17, 2012, at 9:28 AM, Carlos Borroto wrote: Hi Tanguy, L-A is correct in that you can set using the tool id different cluster options for different tools. For example I submit different tools to different queues: default_cluster_job_runner = drmaa://-q all.q -V/ upload1 = drmaa://-q short.q -V/ tophat = drmaa://-q long.q -V/ cufflinks = drmaa://-q long.q -V/ But I don't think this is what you want, as this is an one time setting that you can't change at runtime. I think there was some talk about dynamic DRMAA options base on the input size and tool, but can't tell what is the status on that feature. Hope it helps, Carlos 2012/2/17 Louise-Amélie Schmitt louise-amelie.schm...@embl.de: Hi Tanguy, You can set that at the very bottom of your universe_wsgi.ini file. I did it myself with Torque to set a different behavior for a couple of tools, it works fine. The related Wiki page is here: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster :) Best, L-A Le 17/02/2012 09:43, Tanguy LE CARROUR a écrit : Hi! Is there a way to specify parameters for the job submission on the cluster from the tool configuration file? Let's say I want tool A to be submitted with -pe smp 8 but tool B submitted with -l mem_free=500. Is it already supported? I could not find it in the wiki. ... or do I have to start digging in the code and try to implement it?! ^_^' Best regards, Tanguy ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
Thanks a lot Chris, actually we found another place where we found rpms for PBS Pro: http://apps.man.poznan.pl/trac/pbs-drmaa And it works now! :D Have a great weekend, L-A Le 17/02/2012 17:07, Fields, Christopher J a écrit : If they used a package install, I think the DRMAA libs for torque are normally separate. For instance: http://pkgs.org/download/torque-drmaa If going from source, you can enable DRMAA compilation with --enable-drmaa; I don't recall if that is on by default (I don't think it is). chris On Feb 17, 2012, at 3:49 AM, Louise-Amélie Schmitt wrote: You don't need a full PBS server on that machine, just the client libraries and configuration. i.e. if you can run 'qstat' from machine A and have it return the queue on machine B, that should be all you need. We asked the IT guys they did just that :) now qsub works fine from machine A! But I still have a problem: They have no libdrmaa.so anywhere! What should I do? If I understood correctly it's part of the Fedstage DRMAA service provider, will it be enough installing it on our Galaxy server? Torque's syntax allows you to specify the server name right in the runner URL, but pbs_python links against libtorque, which is part of the torque client, which must be installed somewhere on the local system. Ok, thanks a lot! Best, L-A Best, L-A Le 30/01/2012 18:20, Nate Coraor a écrit : On Jan 30, 2012, at 12:07 PM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the leads! But setting DRMAA_LIBRARY_PATH means I'm in trouble since the libraries are on machine B which is maintained by our IT dept. I cannot access them from machine A. Is it a desperate situation? Will it work if I have a copy of those libs somewhere? :/ Hi L-A, The Galaxy server will need to be a submission host, so I believe it will have to have PBS Pro installed. If it has this, then the FedStage DRMAA library should be installable on the same host. It may be possible copy the libraries, although I don't know whether you'd be able to configure the server address without access to the directories in which the library will look for its configuration. --nate Best, L-A Le 30/01/2012 17:18, Nate Coraor a écrit : On Jan 30, 2012, at 7:37 AM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Hi L-A, Hrm, I've only used it with SGE, which uses an environment variable to define the cell location, and LSF, which I don't remember, but I assume it used the default. I think if you configure the PBS Pro client on the submission host and point DRMAA_LIBRARY_PATH at the correct libdrmaa, it will use your client configuration. There are other PBS Pro users on the list who can hopefully chime in with more details. --nate Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] transferring sample datasets stuck in 'In queue' status
So I stopped using Apache as a proxy web server and use Galaxy as the web server directly. This time the status of the sample data transfer changed to Complete. So I believe Apache needs configuration modifications for this case, maybe Apache needs mod_dav module and other related modules? However, when I go to the data library where the datasets are supposed to go, there is no datasets in it, even though the sample sequence files have been transferred from the sequencer to the folder specified by library_import_dir in universe_wsgi.ini. They are just not imported to the data libraries automatically. When I looked at the data_transfer.log, I found the following error message: 2012-02-17 09:28:37,927 - datatx_7318 - 400 Bad Request The server could not comply with the request since it is either malformed or otherwise incorrect. Malformed LibraryFolder id ( 994debf9f6ab02b9912262b0bd04c784 ) specified, unable to decode 2012-02-17 09:28:37,928 - datatx_7318 - Setting status Complete for dataset All of sample 17 Luobin On Thu, Feb 16, 2012 at 7:58 PM, Luobin Yang yangl...@isu.edu wrote: Hi all, I configured Sample Tracking System in Galaxy to transfer datasets from a sequencer to data libraries, however, after I selected the datasets to be transferred on the sequencer and clicked Transfer button, the transfer status has been in queue forever. I didn't find any error message in galaxy_listener.log, however, I found the following error message in data_transfer.log: 2012-02-16 19:39:01,221 - datatx_3623 - Error. !DOCTYPE HTML PUBLIC -//IETF//DTD HTML 2.0//EN htmlhead title405 Method Not Allowed/title /headbody h1Method Not Allowed/h1 pThe requested method PUT is not allowed for the URL /api/samples/1e8ab44153008be8./p hr addressApache/2.2.14 (Ubuntu) Server at xxx.xxx.xxx.xxx Port 80/address /body/html Any idea what went wrong? Thanks, Luobin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
Re: PBSPro RPMs, that's actually good to know, we will likely need use of that here as well. chris On Feb 17, 2012, at 10:30 AM, Louise-Amélie Schmitt wrote: Thanks a lot Chris, actually we found another place where we found rpms for PBS Pro: http://apps.man.poznan.pl/trac/pbs-drmaa And it works now! :D Have a great weekend, L-A Le 17/02/2012 17:07, Fields, Christopher J a écrit : If they used a package install, I think the DRMAA libs for torque are normally separate. For instance: http://pkgs.org/download/torque-drmaa If going from source, you can enable DRMAA compilation with --enable-drmaa; I don't recall if that is on by default (I don't think it is). chris On Feb 17, 2012, at 3:49 AM, Louise-Amélie Schmitt wrote: You don't need a full PBS server on that machine, just the client libraries and configuration. i.e. if you can run 'qstat' from machine A and have it return the queue on machine B, that should be all you need. We asked the IT guys they did just that :) now qsub works fine from machine A! But I still have a problem: They have no libdrmaa.so anywhere! What should I do? If I understood correctly it's part of the Fedstage DRMAA service provider, will it be enough installing it on our Galaxy server? Torque's syntax allows you to specify the server name right in the runner URL, but pbs_python links against libtorque, which is part of the torque client, which must be installed somewhere on the local system. Ok, thanks a lot! Best, L-A Best, L-A Le 30/01/2012 18:20, Nate Coraor a écrit : On Jan 30, 2012, at 12:07 PM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the leads! But setting DRMAA_LIBRARY_PATH means I'm in trouble since the libraries are on machine B which is maintained by our IT dept. I cannot access them from machine A. Is it a desperate situation? Will it work if I have a copy of those libs somewhere? :/ Hi L-A, The Galaxy server will need to be a submission host, so I believe it will have to have PBS Pro installed. If it has this, then the FedStage DRMAA library should be installable on the same host. It may be possible copy the libraries, although I don't know whether you'd be able to configure the server address without access to the directories in which the library will look for its configuration. --nate Best, L-A Le 30/01/2012 17:18, Nate Coraor a écrit : On Jan 30, 2012, at 7:37 AM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Hi L-A, Hrm, I've only used it with SGE, which uses an environment variable to define the cell location, and LSF, which I don't remember, but I assume it used the default. I think if you configure the PBS Pro client on the submission host and point DRMAA_LIBRARY_PATH at the correct libdrmaa, it will use your client configuration. There are other PBS Pro users on the list who can hopefully chime in with more details. --nate Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Re: [galaxy-dev] No Display at UCSC Main link.
Hello Luobin, Thanks for sending the additional information. It sounds as if you are using the default version of: tool-data/shared/ucsc/ucsc_build_sites.txt Adding hg19 to main in this file will add the link to the datasets. Hopefully this helps, Best, Jen Galaxy team On 2/17/12 1:58 PM, Luobin Yang wrote: Hi, Jen, I noticed that when I use hg18 or other database builds, the UCSC display link shows up, but when I use hg19 database builds, there is no UCSC display link. Thanks, Luobin On Wed, Feb 15, 2012 at 11:30 AM, Luobin Yang yangl...@isu.edu mailto:yangl...@isu.edu wrote: Hi, Jen, Thanks for your replay. Yes, it's an instance that I installed here at ISU, not the main instance at PSU. The dataset is BED and the database is hg19. In my universe_wsgi.ini file, I have the following lines: # UCSC browsers: tool-data/shared/ucsc/ucsc_build_sites.txt ucsc_display_sites = main,test,archaea,ucla In my apache configuration file, I added the following: http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy#CA-64199e734386081ab1c9ec50d26246b3120bb238_1Location /root/display_as http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy#CA-64199e734386081ab1c9ec50d26246b3120bb238_2 Satisfy Any Order deny,allow Deny fromall Allow fromhgw1.cse.ucsc.edu http://hgw1.cse.ucsc.edu Allow fromhgw2.cse.ucsc.edu http://hgw2.cse.ucsc.edu Allow fromhgw3.cse.ucsc.edu http://hgw3.cse.ucsc.edu Allow fromhgw4.cse.ucsc.edu http://hgw4.cse.ucsc.edu Allow fromhgw5.cse.ucsc.edu http://hgw5.cse.ucsc.edu Allow fromhgw6.cse.ucsc.edu http://hgw6.cse.ucsc.edu Allow fromhgw7.cse.ucsc.edu http://hgw7.cse.ucsc.edu Allow fromhgw8.cse.ucsc.edu http://hgw8.cse.ucsc.edu /Location One thing I would like to ask you about is for above Apache configuration, because my Galaxy is served as /galaxy subdirectory instead of the root directory, do I need to make changed to above location? for instance, instead of /root/display_as, use /galaxy/root/display_as ? Thanks so much! Luobin On Tue, Feb 14, 2012 at 10:51 PM, Jennifer Jackson j...@bx.psu.edu mailto:j...@bx.psu.edu wrote: Hello again, The link is not a Galaxy Main public instance share link, which would start with: http://main.g2.bx.psu.edu Having the datatype as BED and the database as hg19 would (for all of the cases I can think of) bring up the View at UCSC link, if the instance is set up to have display at UCSC configured. So, the root cause of the problem is probably with how this other instance is set up. If this is not your instance, you will want to contact the folks running it to see if they have plans to configure UCSC display. If this is yours, follow the instructions on this wiki to get set up: http://wiki.g2.bx.psu.edu/__Admin/Config/Apache%20Proxy http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy See the section: Display at UCSC Hopefully this helps, Best, Jen Galaxy team On 2/14/12 9:26 PM, Luobin Yang wrote: Hi, Jen, The dataset is BED datatype and the database is hg19. Actually this history contains the same steps as those in Galaxy101 tutorial. I created the link for the history and here it is: So basically I do not see an option called Display at UCSC Main, but it has two other options, View in GeneTrack and Display at Ensemble Current Thanks, Luobin On Tue, Feb 14, 2012 at 9:35 PM, Jennifer Jackson j...@bx.psu.edu mailto:j...@bx.psu.edu mailto:j...@bx.psu.edu mailto:j...@bx.psu.edu wrote: Hi Luobin, To display at UCSC, the datatype has to be a UCSC display type and the database has to be a UCSC genome browser. Often, small changes can be made to tune a dataset to help it fit into these requirements. Other times, a dataset will need to be transformed into another type or it can't be viewed at all because the genome is not available at UCSC. You should try to modify these attributes yourself first. Common UCSC datatypes from Galaxy are: BED, bigBED, BAM. For the list of UCSC known datatypes, see: http://genome.ucsc.edu/FAQ/FAQformat.html http://genome.ucsc.edu/FAQ/__FAQformat.html http://genome.ucsc.edu/FAQ/__FAQformat.html http://genome.ucsc.edu/FAQ/FAQformat.html.
