[galaxy-dev] Single job latency
This Galaxy speed and production question reminded me that I have been meaning to raise the following question on the list: For a single job (workflow item), It seems like there is a 20 second lag before the job is launched. It gives the impression that Galaxy is slow and unresponsive (in the sports car sense). I imagine that it is a combination of python's poor thread architecture and web polling latency. Has anyone looked into the latency of running a process? Ted ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] 回复： Speed up the galaxy
Hi I presume the best way to optimise your current problem is to evaluate whether you really need to groom your data!? If its old data presumably yes, but if it is recent data in Illumina 1.8+ encoding(http://en.wikipedia.org/wiki/FASTQ_format) it is not necessary speedup 100% :-) Groom would take long at our servers as well but due to the new Illumina format we didn’t bother to optimise it further by parallelisation for instance... Alex Van: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] Namens ? ? Verzonden: dinsdag 4 december 2012 16:08 Aan: Nate Coraor CC: galaxy-dev@lists.bx.psu.edu Onderwerp: [galaxy-dev] 回复： Speed up the galaxy Hi， I read the page of the document. But I don't think there is anything I need. Now, I installed a local instance of galaxy, and I just run it on a PC. I need to deal with large data, if I just use the default configure of galaxy, every task would take long time. For example I ran the FASTQ Groomer with a large file, the precedure is so slow, and galaxy would not use the potential of my machine. So I want to know how can I get galaxy ran faster. I already can upload files quickly and I just need to know how to ran tools quickly. 发件人： Nate Coraor n...@bx.psu.edumailto:n...@bx.psu.edu 收件人： 泽 蔡 caizexi...@yahoo.com.cnmailto:caizexi...@yahoo.com.cn 抄送： galaxy-...@bx.psu.edumailto:galaxy-...@bx.psu.edu galaxy-...@bx.psu.edumailto:galaxy-...@bx.psu.edu 发送日期： 2012年12月4日, 星期二, 下午 9:38 主题: Re: [galaxy-dev] Speed up the galaxy On Dec 4, 2012, at 4:36 AM, 泽 蔡 wrote: Hi all, How can I speed up the galaxy? Like how to use more cores and memeries. Hi, This is a pretty broad question. However, I would recommend that you start at: http://usegalaxy.org/production --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] from shed to shed
Hi, how do I get tools from one local toolshed (or the main galaxy toolshed) into another tool-shed? Is it possible to clone the content of a complete tool-shed - all repositories at once into another tool-shed? Can I then import all the tools at once into a local galaxy instance? We have a group with their own tool-shed. Unfortunately, I accessed it directly without a proxy on its port which leads to a problem described in issue 825 in galaxy-central. I want to a) clone the tool-shed as it is into my own tool-shed (rsync?) b) selectively or bulk install these tools into my galaxy server. thank you very much, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] workflow input param issue
This should be resolved in changeset 1ac27213bafb in galaxy-central. Thanks for pointing this out! -Dannon On Dec 3, 2012, at 11:37 AM, Marc Logghe marc.log...@ablynx.com wrote: Hi, The conf of the parameters in question looks like this: param name=project1 type=select label=Project options from_data_table=my_projects filter type=unique_value name=unique column=0/ /options /param param name=target1 type=select label=Target multiple=true options from_data_table=my_projects column name=value index=2/ column name=name index=2/ filter type=param_value ref=project1 name=target column=0/ /options help Select 1 or more targets if you would like to restrict your data set /help /param The idea is that if a project is chosen from the project1 dropdown list, the target1 parameter is updated. This seems to work fine as a standalone tool. As soon as the tool is wrapped into a workflow, this gets broken: if project1 is chosen from the list, a server call is made but the response seems to reset the form fields. Both project1 and target1. Thanks, Marc From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Marc Logghe Sent: Monday, December 03, 2012 11:46 AM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] workflow input param issue Hi, I have a workflow that basically needs a select parameter as input. 2 steps in the workflow actually need the very same input. I don’t think there is a (easy) way to let the user only input the parameter once and that it is passed to both steps. Anyhow, currently - as a workaround and not very user friendly - the user needs to input the very same parameter twice, for each step where that parameter is required. The first issue however, is that as soon as the first parameter is set, the second is set as well apparently (they have the same name, that could explain) which is fine, but not to the chosen one, eg. both are kind of reset to default. No errors or something, simply reset, which makes it impossible to enter the parameter. THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at abl...@ablynx.com. Thank you for your co-operation. NANOBODY and NANOCLONE are registered trademarks of Ablynx N.V. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Permission denied error for velveth
On Dec 4, 2012, at 9:30 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: I see that this issue came up before by Oleksandr Moskalenko in October but was unresolved then. Is there a workaround now? regards, Andreas The velvet wrapper is not going to be usable for real user jobs until someone rewrites it to work without direct manipulation of files in the database/ tree. No work has been done on this nor any is planned as far as I know. Regards, Alex ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Main toolshed broken??
Hi Franco, The server error is thrown when your file named tool_data_table_conf.xml.sample is parsed. The content of this file is the following. !-- Locations of indexes in the bisulphite conversion -- table name=bismark_indexes comment_char=# columnsvalue, dbkey, name, path/columns file path=tool-data/bismark_indices.loc / /table !-- Locations of bed files for bisulphite analysis -- table name=bismark_bed_files comment_char=# columnsvalue, name, path/columns file path=tool-data/bismark_bed_files.loc / /table This is an invalid xml file because it requires a containing tables tag set since multiple table tag sets are defined. I've improved the current server error behavior by displaying an appropriate error message for this problem, but the fix will not make it out to the main Galaxy tool shed until the next Galaxy release, currently scheduled for about December 20. In the meantime, I believe you should see better behavior if you upload a new version of this file to your repository that looks something like this: tables !-- Locations of indexes in the bisulphite conversion -- table name=bismark_indexes comment_char=# columnsvalue, dbkey, name, path/columns file path=tool-data/bismark_indices.loc / /table !-- Locations of bed files for bisulphite analysis -- table name=bismark_bed_files comment_char=# columnsvalue, name, path/columns file path=tool-data/bismark_bed_files.loc / /table /tables Thanks, Greg Von Kuster On Dec 3, 2012, at 6:40 PM, Franco Caramia wrote: Hi list, Trying to upload, delete, update files from a repository I own in the toolshed I keep getting Server Error.. This is the repo: http://fcara...@toolshed.g2.bx.psu.edu/repos/fcaramia/methylation_analysis_bismark Is anyone having the same issues?? I could just delete the repository but no option for that is given.. Thanks, Franco This email (including any attachments or links) may contain confidential and/or legally privileged information and is intended only to be read or used by the addressee. If you are not the intended addressee, any use, distribution, disclosure or copying of this email is strictly prohibited. Confidentiality and legal privilege attached to this email (including any attachments) are not waived or lost by reason of its mistaken delivery to you. If you have received this email in error, please delete it and notify us immediately by telephone or email. Peter MacCallum Cancer Centre provides no guarantee that this transmission is free of virus or that it has not been intercepted or altered and will not be liable for any delay in its receipt. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] 回复： 回复： Speed up the galaxy
Hi Alex I look the page of wikipedia, but I have a little confused. We sequenced with solexa. I paste a read of my data, can you tell me whether I need run Groom? The read is like this:@HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 AGAAGTACATCGCGATGCCGTTNCCNNCGAAGGCGATAGNNNACAAGNCCAAATGNTTCTNCATCNNNCNCGAGNNGNCGAGGNCGCCGTGCGACCCTGC +HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 Ya^a`edddeddc\c`a`dc]\Ba^BBZ]ZZ`ZZZ]a]]BBB^[`\UB_V[V\`ZBSZX^ 发件人： Bossers, Alex alex.boss...@wur.nl 收件人： 泽 蔡 caizexi...@yahoo.com.cn 抄送： galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu; Nate Coraor n...@bx.psu.edu 发送日期： 2012年12月4日, 星期二, 下午 11:21 主题: RE: [galaxy-dev] 回复： Speed up the galaxy Hi I presume the best way to optimise your current problem is to evaluate whether you really need to groom your data!? If its old data presumably yes, but if it is recent data in Illumina 1.8+ encoding(http://en.wikipedia.org/wiki/FASTQ_format) it is not necessary speedup 100% :-) Groom would take long at our servers as well but due to the new Illumina format we didn’t bother to optimise it further by parallelisation for instance... Alex Van:galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] Namens ? ? Verzonden: dinsdag 4 december 2012 16:08 Aan: Nate Coraor CC: galaxy-dev@lists.bx.psu.edu Onderwerp: [galaxy-dev] 回复：Speed up the galaxy Hi， I read the page of the document. But I don't think there is anything I need. Now, I installed a local instance of galaxy, and I just run it on a PC. I need to deal with large data, if I just use the default configure of galaxy, every task would take long time. For example I ran the FASTQ Groomer with a large file, the precedure is so slow, and galaxy would not use the potential of my machine. So I want to know how can I get galaxy ran faster. I already can upload files quickly and I just need to know how to ran tools quickly. 发件人：Nate Coraor n...@bx.psu.edu 收件人：泽蔡caizexi...@yahoo.com.cn 抄送：galaxy-...@bx.psu.edu galaxy-...@bx.psu.edu 发送日期：2012年12月4日, 星期二, 下午9:38 主题:Re: [galaxy-dev] Speed up the galaxy On Dec 4, 2012, at 4:36 AM, 泽蔡wrote: Hi all, How can I speed up the galaxy? Like how to use more cores and memeries. Hi, This is a pretty broad question. However, I would recommend that you start at: http://usegalaxy.org/production --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Required Galaxy umask settings for HTML downloads?
Hello all, I am having issues downloading HTML files from Galaxy the same as is described in this email chain: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-August/010965.html I am getting the error (13)Permission denied: xsendfile: cannot open file: /basedir/galaxy_data/database/tmp/tmp8iEccn/library_download.zip which is indeed a basic filesystem permissions issue. The problem is that the permissions created for that directory and every directory created in tmp/ look like this: drwx--+ 2 galaxy galaxy 3 Dec 4 09:23 tmp8iEccn And I have placed the Apache user in the galaxy group, but as you can see no group permissions ever get set by Galaxy on the directories that it creates (it is getting a 700 permissions setting). As Nate Coraor suggested in the message linked to above, I have tried altering the default umask but I ran into issues with getting non-existant results. I use sudo service galaxy start as the galaxy user each time to start the server and a ps -ef | grep galaxy confirms that Galaxy is running as the galaxy user. Since I use sudo though I changed the sudoers file to include: rootALL=(ALL) ALL galaxy ALL=(ALL) ALL Defaults umask_override Defaults umask = 0002 This changed absolutely nothing. Then I started looking deeper into the PAM configuration and added a umask directive to /etc/pam.d/sudo (and also tried it in password-auth-ac and system-auth-ac) like this: session optional pam_umask.so umask=0002. Still nothing changed in the permissions in tmp/ when I tried to download an HTML file: no group permissions were set. Then I dug deeper still and saw that sometimes if setting the mask in /etc/pam.d/ config files is not enough that you can try to set a system-wide mask in /etc/login.defs (following the suggestion here: http://stackoverflow.com/questions/10220531/how-to-set-system-wide-umask). Still no dice. I've pretty much exhausted my know-how in this department. Any other suggestions of how to fix this or where the correct place to set the umask is? Thanks, Josh Nielsen ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Required Galaxy umask settings for HTML downloads?
On Dec 4, 2012, at 10:52 AM, Josh Nielsen wrote: Hello all, I am having issues downloading HTML files from Galaxy the same as is described in this email chain: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-August/010965.html I am getting the error (13)Permission denied: xsendfile: cannot open file: /basedir/galaxy_data/database/tmp/tmp8iEccn/library_download.zip which is indeed a basic filesystem permissions issue. The problem is that the permissions created for that directory and every directory created in tmp/ look like this: drwx--+ 2 galaxy galaxy 3 Dec 4 09:23 tmp8iEccn And I have placed the Apache user in the galaxy group, but as you can see no group permissions ever get set by Galaxy on the directories that it creates (it is getting a 700 permissions setting). As Nate Coraor suggested in the message linked to above, I have tried altering the default umask but I ran into issues with getting non-existant results. I use sudo service galaxy start as the galaxy user each time to start the server and a ps -ef | grep galaxy confirms that Galaxy is running as the galaxy user. Since I use sudo though I changed the sudoers file to include: rootALL=(ALL) ALL galaxy ALL=(ALL) ALL Defaults umask_override Defaults umask = 0002 This changed absolutely nothing. Then I started looking deeper into the PAM configuration and added a umask directive to /etc/pam.d/sudo (and also tried it in password-auth-ac and system-auth-ac) like this: session optional pam_umask.so umask=0002. Still nothing changed in the permissions in tmp/ when I tried to download an HTML file: no group permissions were set. Then I dug deeper still and saw that sometimes if setting the mask in /etc/pam.d/ config files is not enough that you can try to set a system-wide mask in /etc/login.defs (following the suggestion here: http://stackoverflow.com/questions/10220531/how-to-set-system-wide-umask). Still no dice. I've pretty much exhausted my know-how in this department. Any other suggestions of how to fix this or where the correct place to set the umask is? Hi Josh, Thanks for doing such extensive tests. Have you tried setting the umask in the init script itself? --nate Thanks, Josh Nielsen ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Required Galaxy umask settings for HTML downloads?
On 04/12/12 16:52, Josh Nielsen wrote: I am getting the error (13)Permission denied: xsendfile: cannot open file: /basedir/galaxy_data/database/tmp/tmp8iEccn/library_download.zip which is indeed a basic filesystem permissions issue. The problem is that the permissions created for that directory and every directory created in tmp/ look like this: drwx--+ 2 galaxy galaxy 3 Dec 4 09:23 tmp8iEccn And I have placed the Apache user in the galaxy group, but as you can see no group permissions ever get set by Galaxy on the directories that it creates (it is getting a 700 permissions setting). Isn't the trailing + character an indication of ACLs being set on the directory? What do the following say...? getfacl /tmp/tmp8iEccn getfacl /tmp If you do have ACLs involved, it may be the case that various masks are being enforced via that mechanism. Paul P.S. I'm not sure that making the galaxy user a sudoer would have any effect unless the user was attempting to gain privileges, which would be a pretty scary way of running Galaxy, I would have thought. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Required Galaxy umask settings for HTML downloads?
Hi Nate,
Thanks for the reply. No I hadn't thought to add anything to
/etc/init.d/galaxy itself. It is a short enough script that I can paste it
below. What would I need to do to edit it with umask settings?
Also I should note, changing the umask in the PAM files actually did change
the default permissions for the galaxy user when I did an su - galaxy in
a bash shell and then created or 'touch'-ed any files (which you could
logically expect). But for some reason it didn't seem to make a difference
with the directories created in that tmp/ directory even though the galaxy
user was given ownership. That made me wonder if something was going on
internal to Galaxy, or something else, that was overwriting/ignoring the
system umask settings (which actually work fine in a shell environment as
the user itself). Maybe I'll look into that ACL stuff Paul mentioned.
Here is my /etc/init.d/galaxy script:
. /etc/rc.d/init.d/functions
GALAXY_USER=galaxy
GALAXY_DIST_HOME=/home/galaxy/galaxy-dist
GALAXY_RUN=${GALAXY_DIST_HOME}/run.sh
GALAXY_PID=${GALAXY_DIST_HOME}/paster.pid
case $1 in
start)
echo -n Starting galaxy services:
daemon --user $GALAXY_USER ${GALAXY_RUN}
--daemon --pid-file=${GALAXY_PID}
touch /var/lock/subsys/galaxy
;;
stop)
echo -n Shutting down galaxy services:
daemon --user $GALAXY_USER ${GALAXY_RUN}
--stop-daemon
rm -f /var/lock/subsys/galaxy
;;
status)
daemon --user galaxy ${GALAXY_RUN} --status
;;
restart)
$0 stop; $0 start
;;
reload)
$0 stop; $0 start
;;
*)
echo Usage: galaxy
{start|stop|status|reload|restart}
;;
esac
--
Thanks!
Josh
On Tue, Dec 4, 2012 at 9:56 AM, Nate Coraor n...@bx.psu.edu wrote:
On Dec 4, 2012, at 10:52 AM, Josh Nielsen wrote:
Hello all,
I am having issues downloading HTML files from Galaxy the same as is
described in this email chain:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-August/010965.html
I am getting the error (13)Permission denied: xsendfile: cannot open
file: /basedir/galaxy_data/database/tmp/tmp8iEccn/library_download.zip
which is indeed a basic filesystem permissions issue. The problem is that
the permissions created for that directory and every directory created in
tmp/ look like this:
drwx--+ 2 galaxy galaxy 3 Dec 4 09:23 tmp8iEccn
And I have placed the Apache user in the galaxy group, but as you can
see no group permissions ever get set by Galaxy on the directories that it
creates (it is getting a 700 permissions setting).
As Nate Coraor suggested in the message linked to above, I have tried
altering the default umask but I ran into issues with getting non-existant
results. I use sudo service galaxy start as the galaxy user each time to
start the server and a ps -ef | grep galaxy confirms that Galaxy is
running as the galaxy user. Since I use sudo though I changed the sudoers
file to include:
rootALL=(ALL) ALL
galaxy ALL=(ALL) ALL
Defaults umask_override
Defaults umask = 0002
This changed absolutely nothing. Then I started looking deeper into the
PAM configuration and added a umask directive to /etc/pam.d/sudo (and also
tried it in password-auth-ac and system-auth-ac) like this: session
optional pam_umask.so umask=0002. Still nothing changed in the
permissions in tmp/ when I tried to download an HTML file: no group
permissions were set. Then I dug deeper still and saw that sometimes if
setting the mask in /etc/pam.d/ config files is not enough that you can try
to set a system-wide mask in /etc/login.defs (following the suggestion
here:
http://stackoverflow.com/questions/10220531/how-to-set-system-wide-umask).
Still no dice. I've pretty much exhausted my know-how in this department.
Any other suggestions of how to fix this or where the correct place to set
the umask is?
Hi Josh,
Thanks for doing such extensive tests. Have you tried setting the umask
in the init script itself?
--nate
Thanks,
Josh Nielsen
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] from shed to shed
Hi Ido, On Dec 4, 2012, at 10:26 AM, Ido Tamir wrote: Hi, how do I get tools from one local toolshed (or the main galaxy toolshed) into another tool-shed? Is it possible to clone the content of a complete tool-shed - all repositories at once into another tool-shed? Not at the current time, but I've attached a script that I use in my development environment that sort of does what you want. When you run it for the first time, make sure it installs to a tool shed you don't care about destroying just so you can see how it works. It uses the configuration settings in commnity_wsgi.ini, so make sure you set things accordingly. here's what you do: 1. Place the 3 attached files in ~/scripts/tool_shed 2. Define the tool shed you want to copy repositories form in copy_repository.xml 3. Define any number of valid repository tags in copy_repository.xml 4. run the copy_repositoyr.sh shell script from the ~/scripts/tool_shed directory 5. When all repositories have been copied into you local tool shed, start it up and make sure to reset all metadata on all repositories as the script does not do so. You can dfo this from the Tool Shed's admin interface. I do not have time to support it, so you're on your own if you want to use it as is. If you enhance it to the point that it would be useful to the community, please contribute it back to me and I'll make sure it is included in the Galaxy code base. #!/usr/bin/env python ''' This script requires the Galaxy instance to use Postgres for database storage. To run this script, use sh copy_repository.sh from this directory ''' import sys, os, ConfigParser assert sys.version_info[:2] = ( 2, 4 ) new_path = [ os.path.join( os.getcwd(), lib ) ] new_path.extend( sys.path[1:] ) # remove scripts/ from the path sys.path = new_path from galaxy import eggs import pkg_resources pkg_resources.require( psycopg2 ) import psycopg2 pkg_resources.require( 'elementtree' ) from elementtree import ElementTree, ElementInclude import galaxy.webapps.community.app from galaxy import util from mercurial import hg, ui, httprepo, commands def directory_hash_id( id ): s = str( id ) l = len( s ) # Shortcut -- ids 0-999 go under ../000/ if l 4: return [ 000 ] # Pad with zeros until a multiple of three padded = ( ( ( 3 - len( s ) ) % 3 ) * 0 ) + s # Drop the last three digits -- 1000 files per directory padded = padded[:-3] # Break into chunks of three return [ padded[i*3:(i+1)*3] for i in range( len( padded ) // 3 ) ] def add_hgweb_config_entry( repository, repository_path ): # Add an entry in the hgweb.config file for a new repository. This enables calls to repository.repo_path. # An entry looks something like: repos/test/mira_assembler = database/community_files/000/repo_123 hgweb_config = %s/hgweb.config % os.getcwd() entry = repos/%s/%s = %s % ( repository.user.username, repository.name, repository_path.lstrip( './' ) ) if os.path.exists( hgweb_config ): output = open( hgweb_config, 'a' ) else: output = open( hgweb_config, 'w' ) output.write( '[paths]\n' ) output.write( %s\n % entry ) output.close() def create_hgrc_file( repository ): # At this point, an entry for the repository is required to be in the hgweb.config # file so we can call repository.repo_path. # Create a .hg/hgrc file that looks something like this: # [web] # allow_push = test # name = convert_characters1 # push_ssl = False # Upon repository creation, only the owner can push to it ( allow_push setting ), # and since we support both http and https, we set push_ssl to False to override # the default (which is True) in the mercurial api. hgrc_file = os.path.abspath( os.path.join( repository.repo_path, .hg, hgrc ) ) output = open( hgrc_file, 'w' ) output.write( '[web]\n' ) output.write( 'allow_push = %s\n' % repository.user.username ) output.write( 'name = %s\n' % repository.name ) output.write( 'push_ssl = false\n' ) output.flush() output.close() def clone_repository( repository_clone_url, repository_path ): Clone the repository up to the specified changeset_revision. No subsequent revisions will be present in the cloned repository. commands.clone( get_configured_ui(), str( repository_clone_url ), dest=str( repository_path ), pull=True, noupdate=False, rev=[] ) def contains( containing_str, contained_str ): return containing_str.lower().find( contained_str.lower() ) = 0 def copy_repository( app, repository_clone_url, name, description, user_id ): sa_session = app.model.context # Add the repository record to the db print Adding new database record for repository: , name repository = app.model.Repository( name=name, description=description,
Re: [galaxy-dev] 回复： 回复： Speed up the galaxy
Hi, You can use fastQC to find out what is the quality encoding of your sequences http://www.bioinformatics.babraham.ac.uk/projects/fastqc/ Alternatively, you can also use this script: http://www.uppmax.uu.se/userscript/check-fastq-quality-score-format David Date: Tue, 4 Dec 2012 23:41:27 +0800 From: caizexi...@yahoo.com.cn To: alex.boss...@wur.nl CC: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] 回复： 回复： Speed up the galaxy Hi Alex I look the page of wikipedia, but I have a little confused. We sequenced with solexa. I paste a read of my data, can you tell me whether I need run Groom?The read is like this:@HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 AGAAGTACATCGCGATGCCGTTNCCNNCGAAGGCGATAGNNNACAAGNCCAAATGNTTCTNCATCNNNCNCGAGNNGNCGAGGNCGCCGTGCGACCCTGC +HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 Ya^a`edddeddc\c`a`dc]\Ba^BBZ]ZZ`ZZZ]a]]BBB^[`\UB_V[V\`ZBSZX^ 发件人： Bossers, Alex alex.boss...@wur.nl 收件人： 泽 蔡 caizexi...@yahoo.com.cn 抄送： galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu; Nate Coraor n...@bx.psu.edu 发送日期： 2012年12月4日, 星期二, 下午 11:21 主题: RE: [galaxy-dev] 回复： Speed up the galaxy Hi I presume the best way to optimise your current problem is to evaluate whether you really need to groom your data!? If its old data presumably yes, but if it is recent data in Illumina 1.8+ encoding(http://en.wikipedia.org/wiki/FASTQ_format) it is not necessary speedup 100% :-) Groom would take long at our servers as well but due to the new Illumina format we didn’t bother to optimise it further by parallelisation for instance... Alex Van: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] Namens ? ? Verzonden: dinsdag 4 december 2012 16:08 Aan: Nate Coraor CC: galaxy-dev@lists.bx.psu.edu Onderwerp: [galaxy-dev] 回复： Speed up the galaxy Hi， I read the page of the document. But I don't think there is anything I need. Now, I installed a local instance of galaxy, and I just run it on a PC. I need to deal with large data, if I just use the default configure of galaxy, every task would take long time. For example I ran the FASTQ Groomer with a large file, the precedure is so slow, and galaxy would not use the potential of my machine. So I want to know how can I get galaxy ran faster. I already can upload files quickly and I just need to know how to ran tools quickly. 发件人： Nate Coraor n...@bx.psu.edu 收件人： 泽 蔡 caizexi...@yahoo.com.cn 抄送： galaxy-...@bx.psu.edu galaxy-...@bx.psu.edu 发送日期： 2012年12月4日, 星期二, 下午 9:38 主题: Re: [galaxy-dev] Speed up the galaxy On Dec 4, 2012, at 4:36 AM, 泽 蔡 wrote: Hi all, How can I speed up the galaxy? Like how to use more cores and memeries. Hi, This is a pretty broad question. However, I would recommend that you start at: http://usegalaxy.org/production --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Same error, better formatting.
Hello Franco, The tool dependency installation fails due to BEDTools.v2.17.0.tar.gz extracting into a directory named bedtools-2.17.0, rather than BEDTools.v2.17.0. Also, you following tag is not optimal since it assumes a specific os platform. action type=shell_commandcp bin/* $INSTALL_DIR/bin/action You'll need to change your tool_dependency.xml definition to use the target_file option, and correct the above tag with something like the following: ?xml version=1.0? tool_dependency package name=bedtools version=2.17.0 install version=1.0 actions action type=download_by_url target_filename=bedtools-2.17.0.tar.gzhttp://bedtools.googlecode.com/files/BEDTools.v2.17.0.tar.gz/action action type=shell_commandmake clean/action action type=shell_commandmake all/action action type=move_directory_files source_directorybin/source_directory destination_directory$INSTALL_DIR/bin/destination_directory /action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install readme GCC version 4.1 or greater is recommended. 3.x versions will typically not compile BEDTools /readme /package /tool_dependency Greg Von Kuster On Dec 3, 2012, at 9:21 PM, Franco Caramia wrote: Hi Greg, Our Galaxy version is galaxy-dist 7712:e6444e7a1685 . The tool is “contra” from this repo: http://fcara...@toolshed.g2.bx.psu.edu/repos/fcaramia/contra I own the repo and uploaded the tool_dependencies file Then the following steps: Installing the tool from toolshed: Success Installing tool dependencies: Failed Deleting tool: Success Re-installing tool: Failed Thanks, Franco On 4/12/12 12:40 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Franco, It's impossible to help you with this issue and the issues you sent in your previous messages unless you provide more information. What version of Galaxy are you running? From what tool shed are you installing repositories. What repository are you installing? What are the precise steps you took to produce the behavior? Keep in mind that the Galaxy test tool shed tracks the galaxy-central repository on bitbucket while the Galaxy main tool shed tracks the galaxy-dist repository on bitbucket. Greg Von Kuster On Dec 2, 2012, at 11:36 PM, Franco Caramia wrote: URL: http://bioinf-galaxy/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a Module paste.exceptions.errormiddleware:143 in __call__ http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# app_iter = self.application(environ, start_response) Module paste.debug.prints:98 in __call__ http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# environ, self.app) Module paste.wsgilib:539 in intercept_output http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# body = method( trans, **kwargs ) Module galaxy.web.framework:184 in decorator http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# return func( self, trans, *args, **kwargs ) Module galaxy.web.controllers.admin_toolshed:350 in browse_repositories http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# return self.reselect_tool_panel_section( trans, **kwd ) Module galaxy.web.framework:184 in decorator http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# return func( self, trans, *args, **kwargs ) Module galaxy.web.controllers.admin_toolshed:1352 in reselect_tool_panel_section http://bioinf-galaxy:83/admin_toolshed/browse_repositories?operation=activate+or+reinstallid=88ee2d7021e4dc1a# status=status ) Module galaxy.web.framework:746 in fill_template
Re: [galaxy-dev] Single job latency
Hi Ted You can cut down the lag by 5 secs if you delete the time.sleep( 5 ) in '~/lib/galaxy/jobs/manager.py' ( line 155) (see also my e-mail from Nov 7: in this email conversation: http://dev.list.galaxyproject.org/Jobs-are-slow-to-start-on-my-galaxy-instance-Used-to-be-much-faster-tt4656337.html There has been a follow up discussion on the IRC Channel: as it turns out, this time delay is necessary for certain users, depending on their set-up - Hence be careful, when deleting it. Regards, Hans-Rudolf On 12/04/2012 04:19 PM, Ted Goldstein (PhD candidate) wrote: This Galaxy speed and production question reminded me that I have been meaning to raise the following question on the list: For a single job (workflow item), It seems like there is a 20 second lag before the job is launched. It gives the impression that Galaxy is slow and unresponsive (in the sports car sense). I imagine that it is a combination of python's poor thread architecture and web polling latency. Has anyone looked into the latency of running a process? Ted ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Required Galaxy umask settings for HTML downloads?
Hi Paul, Thanks for replying. Interestingly I've never dealt with filesystem ACLs before and I didn't even know that ext3/4 systems had that feature. Here is my output from those commands: bash getfacl tmp8iEccn # file: tmp8iEccn # owner: galaxy # group: galaxy user::rwx group::--- mask::rwx other::--- bash getfacl tmp # file: tmp # owner: root # group: galaxy user::rwx group::rwx other::rwx What enforces these ACLs/where can they be tweaked? P.S. The reason I run Galaxy with sudo is because if I try to do so as just the Galaxy user it cannot create the process lock files: touch: cannot touch `/var/lock/subsys/galaxy`. I suppose I could put the lock files somewhere else or manually give galaxy group permission to /var/lock/subsys (not so sure that's a good idea though), but sudo seemed to solve the problem. You can see my init script in my reply to Nate. Thanks, Josh On Tue, Dec 4, 2012 at 10:06 AM, Paul Boddie paul.bod...@biotek.uio.nowrote: On 04/12/12 16:52, Josh Nielsen wrote: I am getting the error (13)Permission denied: xsendfile: cannot open file: /basedir/galaxy_data/database/**tmp/tmp8iEccn/library_**download.zip which is indeed a basic filesystem permissions issue. The problem is that the permissions created for that directory and every directory created in tmp/ look like this: drwx--+ 2 galaxy galaxy 3 Dec 4 09:23 tmp8iEccn And I have placed the Apache user in the galaxy group, but as you can see no group permissions ever get set by Galaxy on the directories that it creates (it is getting a 700 permissions setting). Isn't the trailing + character an indication of ACLs being set on the directory? What do the following say...? getfacl /tmp/tmp8iEccn getfacl /tmp If you do have ACLs involved, it may be the case that various masks are being enforced via that mechanism. Paul P.S. I'm not sure that making the galaxy user a sudoer would have any effect unless the user was attempting to gain privileges, which would be a pretty scary way of running Galaxy, I would have thought. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Main toolshed broken??
Hi Adam, I believe this is related to the invalid xml definition in your repository's tool_data_table_conf.xml.sample file which I discussed in a previous reply. Greg Von Kuster On Dec 4, 2012, at 8:08 AM, Adam Carr (NBI) wrote: Hi, I hadn't caught this as I posted my query to the list, I'm having trouble uploading to a brand new repository. Could my issue described here: http://dev.list.galaxyproject.org/Tool-Shed-Upload-Files-To-Repository-Crash-td4657324.html, be related? Many Thanks, Adam. --- Adam Carr Linux Support Developmnent E adam.c...@nbi.ac.uk T +44 1603 450161 W jic.ac.uk ifr.ac.uk tgac.ac.uk tsl.ac.uk NBI Partnership Ltd Norwich Research Park Norwich NR4 7UH The NBI Partnership Ltd provides non-scientific services to the Institute of Food Research, the John Innes Centre, The Genome Analysis Centre and The Sainsbury Laboratory -Original Message- Hi list, Trying to upload, delete, update files from a repository I own in the toolshed I keep getting Server Error.. This is the repo: http://fcaramia@.../repos/fcaramia/methylation_analysis_bismark Is anyone having the same issues?? I could just delete the repository but no option for that is given.. Thanks, Franco ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Required Galaxy umask settings for HTML downloads?
On Dec 4, 2012, at 11:23 AM, Josh Nielsen wrote:
Hi Nate,
Thanks for the reply. No I hadn't thought to add anything to
/etc/init.d/galaxy itself. It is a short enough script that I can paste it
below. What would I need to do to edit it with umask settings?
Also I should note, changing the umask in the PAM files actually did change
the default permissions for the galaxy user when I did an su - galaxy in a
bash shell and then created or 'touch'-ed any files (which you could
logically expect). But for some reason it didn't seem to make a difference
with the directories created in that tmp/ directory even though the galaxy
user was given ownership. That made me wonder if something was going on
internal to Galaxy, or something else, that was overwriting/ignoring the
system umask settings (which actually work fine in a shell environment as the
user itself). Maybe I'll look into that ACL stuff Paul mentioned.
Paul's suggestions are worth checking in to. I'd be interested in knowing what
the POSIX permissions are on /tmp itself, and what the ACLs are, if any.
Those temporary files are created by creating a temporary directory using
Python's tempfile.mkdtemp(), which creates them with a mode of 700, which is
then masked by the current umask. The change I added in the email you
referenced in your original post changes the directory to mode 0777 masked by
the umask (after it's created).
Depending on how the pieces used in RHEL's startup() shell function handle the
environment, you may be able to set it on the line above `daemon ...` inside
the 'start)' branch of the case statement. If that doesn't work, you may need
to get more creative and do something like:
daemon --user $GALAXY_USER umask 027; ${GALAXY_RUN} --daemon
--pid-file=${GALAXY_PID}
Alternatively, you can set it inside /home/galaxy/galaxy-dist/run.sh, since
this startup script uses run.sh.
--nate
Here is my /etc/init.d/galaxy script:
. /etc/rc.d/init.d/functions
GALAXY_USER=galaxy
GALAXY_DIST_HOME=/home/galaxy/galaxy-dist
GALAXY_RUN=${GALAXY_DIST_HOME}/run.sh
GALAXY_PID=${GALAXY_DIST_HOME}/paster.pid
case $1 in
start)
echo -n Starting galaxy services:
daemon --user $GALAXY_USER ${GALAXY_RUN} --daemon
--pid-file=${GALAXY_PID}
touch /var/lock/subsys/galaxy
;;
stop)
echo -n Shutting down galaxy services:
daemon --user $GALAXY_USER ${GALAXY_RUN}
--stop-daemon
rm -f /var/lock/subsys/galaxy
;;
status)
daemon --user galaxy ${GALAXY_RUN} --status
;;
restart)
$0 stop; $0 start
;;
reload)
$0 stop; $0 start
;;
*)
echo Usage: galaxy
{start|stop|status|reload|restart}
;;
esac
--
Thanks!
Josh
On Tue, Dec 4, 2012 at 9:56 AM, Nate Coraor n...@bx.psu.edu wrote:
On Dec 4, 2012, at 10:52 AM, Josh Nielsen wrote:
Hello all,
I am having issues downloading HTML files from Galaxy the same as is
described in this email chain:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-August/010965.html
I am getting the error (13)Permission denied: xsendfile: cannot open file:
/basedir/galaxy_data/database/tmp/tmp8iEccn/library_download.zip which is
indeed a basic filesystem permissions issue. The problem is that the
permissions created for that directory and every directory created in tmp/
look like this:
drwx--+ 2 galaxy galaxy 3 Dec 4 09:23 tmp8iEccn
And I have placed the Apache user in the galaxy group, but as you can see
no group permissions ever get set by Galaxy on the directories that it
creates (it is getting a 700 permissions setting).
As Nate Coraor suggested in the message linked to above, I have tried
altering the default umask but I ran into issues with getting non-existant
results. I use sudo service galaxy start as the galaxy user each time to
start the server and a ps -ef | grep galaxy confirms that Galaxy is
running as the galaxy user. Since I use sudo though I changed the sudoers
file to include:
rootALL=(ALL) ALL
galaxy ALL=(ALL) ALL
Defaults umask_override
Defaults umask = 0002
This changed absolutely nothing. Then I started looking deeper into the PAM
configuration and added a umask directive to /etc/pam.d/sudo (and also
tried it in password-auth-ac and system-auth-ac) like this: session
optional pam_umask.so umask=0002. Still nothing changed in the
permissions in tmp/ when I tried to download an HTML file: no group
permissions were set. Then I dug deeper still and saw that sometimes if
setting the mask in /etc/pam.d/ config files is not
Re: [galaxy-dev] Required Galaxy umask settings for HTML downloads?
On 04/12/12 17:35, Josh Nielsen wrote: Hi Paul, Thanks for replying. Interestingly I've never dealt with filesystem ACLs before and I didn't even know that ext3/4 systems had that feature. Here is my output from those commands: bash getfacl tmp8iEccn # file: tmp8iEccn # owner: galaxy # group: galaxy user::rwx group::--- mask::rwx other::--- So, the galaxy group really doesn't have access. If I interpret this correctly, and although the mask is set to not filter anything - it's expressed like the symbolic form of the umask shell built-in (umask -S) and describes permissions, not restrictions - it isn't going to change the capabilities of the galaxy group. Moreover, the mask isn't a default mask which I think is the feature that most directly corresponds to umask. See the following for the complicated details: http://users.suse.com/~agruen/acl/linux-acls/online/ bash getfacl tmp # file: tmp # owner: root # group: galaxy user::rwx group::rwx other::rwx What enforces these ACLs/where can they be tweaked? You can use the setfacl command to change them. However, I don't see anything here that is causing the problem. P.S. The reason I run Galaxy with sudo is because if I try to do so as just the Galaxy user it cannot create the process lock files: touch: cannot touch `/var/lock/subsys/galaxy`. I suppose I could put the lock files somewhere else or manually give galaxy group permission to /var/lock/subsys (not so sure that's a good idea though), but sudo seemed to solve the problem. You can see my init script in my reply to Nate. I think there might be a misunderstanding here. You will need to use sudo to start the service because the lock file will need writing, as you point out, but then the service runs as the galaxy user (as seen in the init script). However, in the previous message, it looked like you had added the galaxy user to the sudoers file, which would only be necessary if that user itself needed to elevate its privileges, which I don't think would be necessary under any circumstances (but then I'm a Galaxy beginner, so I can't really say that for sure). I actually think Nate's response is promising here: you need to override the conservative permissions on temporary files after their creation. Alternatively, it might be possible to grant the Apache user access to files in the temporary directory hierarchy, but that might need some experimentation, and I'm not even sure that this wouldn't involve some modifications to the code making the temporary files in the first place. Paul ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Required Galaxy umask settings for HTML downloads?
Great. I'll give those ideas a shot to see if it gets me anywhere.
P.S. You referenced in the email that I linked to a fix in the next release
of Galaxy. Is that out yet or still in development?
-Josh
On Tue, Dec 4, 2012 at 10:44 AM, Nate Coraor n...@bx.psu.edu wrote:
On Dec 4, 2012, at 11:23 AM, Josh Nielsen wrote:
Hi Nate,
Thanks for the reply. No I hadn't thought to add anything to
/etc/init.d/galaxy itself. It is a short enough script that I can paste it
below. What would I need to do to edit it with umask settings?
Also I should note, changing the umask in the PAM files actually did
change the default permissions for the galaxy user when I did an su -
galaxy in a bash shell and then created or 'touch'-ed any files (which you
could logically expect). But for some reason it didn't seem to make a
difference with the directories created in that tmp/ directory even though
the galaxy user was given ownership. That made me wonder if something was
going on internal to Galaxy, or something else, that was
overwriting/ignoring the system umask settings (which actually work fine in
a shell environment as the user itself). Maybe I'll look into that ACL
stuff Paul mentioned.
Paul's suggestions are worth checking in to. I'd be interested in knowing
what the POSIX permissions are on /tmp itself, and what the ACLs are, if
any.
Those temporary files are created by creating a temporary directory using
Python's tempfile.mkdtemp(), which creates them with a mode of 700, which
is then masked by the current umask. The change I added in the email you
referenced in your original post changes the directory to mode 0777 masked
by the umask (after it's created).
Depending on how the pieces used in RHEL's startup() shell function handle
the environment, you may be able to set it on the line above `daemon ...`
inside the 'start)' branch of the case statement. If that doesn't work,
you may need to get more creative and do something like:
daemon --user $GALAXY_USER umask 027; ${GALAXY_RUN} --daemon
--pid-file=${GALAXY_PID}
Alternatively, you can set it inside /home/galaxy/galaxy-dist/run.sh,
since this startup script uses run.sh.
--nate
Here is my /etc/init.d/galaxy script:
. /etc/rc.d/init.d/functions
GALAXY_USER=galaxy
GALAXY_DIST_HOME=/home/galaxy/galaxy-dist
GALAXY_RUN=${GALAXY_DIST_HOME}/run.sh
GALAXY_PID=${GALAXY_DIST_HOME}/paster.pid
case $1 in
start)
echo -n Starting galaxy services:
daemon --user $GALAXY_USER ${GALAXY_RUN}
--daemon --pid-file=${GALAXY_PID}
touch /var/lock/subsys/galaxy
;;
stop)
echo -n Shutting down galaxy services:
daemon --user $GALAXY_USER ${GALAXY_RUN}
--stop-daemon
rm -f /var/lock/subsys/galaxy
;;
status)
daemon --user galaxy ${GALAXY_RUN} --status
;;
restart)
$0 stop; $0 start
;;
reload)
$0 stop; $0 start
;;
*)
echo Usage: galaxy
{start|stop|status|reload|restart}
;;
esac
--
Thanks!
Josh
On Tue, Dec 4, 2012 at 9:56 AM, Nate Coraor n...@bx.psu.edu wrote:
On Dec 4, 2012, at 10:52 AM, Josh Nielsen wrote:
Hello all,
I am having issues downloading HTML files from Galaxy the same as is
described in this email chain:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-August/010965.html
I am getting the error (13)Permission denied: xsendfile: cannot open
file: /basedir/galaxy_data/database/tmp/tmp8iEccn/library_download.zip
which is indeed a basic filesystem permissions issue. The problem is that
the permissions created for that directory and every directory created in
tmp/ look like this:
drwx--+ 2 galaxy galaxy 3 Dec 4 09:23 tmp8iEccn
And I have placed the Apache user in the galaxy group, but as you can
see no group permissions ever get set by Galaxy on the directories that it
creates (it is getting a 700 permissions setting).
As Nate Coraor suggested in the message linked to above, I have tried
altering the default umask but I ran into issues with getting non-existant
results. I use sudo service galaxy start as the galaxy user each time to
start the server and a ps -ef | grep galaxy confirms that Galaxy is
running as the galaxy user. Since I use sudo though I changed the sudoers
file to include:
rootALL=(ALL) ALL
galaxy ALL=(ALL) ALL
Defaults umask_override
Defaults umask = 0002
This changed absolutely nothing. Then I started looking deeper into
the PAM configuration and added a umask directive to /etc/pam.d/sudo (and
also tried it in password-auth-ac and system-auth-ac)
Re: [galaxy-dev] Required Galaxy umask settings for HTML downloads?
On Dec 4, 2012, at 11:57 AM, Josh Nielsen wrote:
Great. I'll give those ideas a shot to see if it gets me anywhere.
P.S. You referenced in the email that I linked to a fix in the next release
of Galaxy. Is that out yet or still in development?
The fix was in the September 7, 2012 distribution release, so it'd be a good
idea to make sure your release is at least that new.
--nate
-Josh
On Tue, Dec 4, 2012 at 10:44 AM, Nate Coraor n...@bx.psu.edu wrote:
On Dec 4, 2012, at 11:23 AM, Josh Nielsen wrote:
Hi Nate,
Thanks for the reply. No I hadn't thought to add anything to
/etc/init.d/galaxy itself. It is a short enough script that I can paste it
below. What would I need to do to edit it with umask settings?
Also I should note, changing the umask in the PAM files actually did change
the default permissions for the galaxy user when I did an su - galaxy in
a bash shell and then created or 'touch'-ed any files (which you could
logically expect). But for some reason it didn't seem to make a difference
with the directories created in that tmp/ directory even though the galaxy
user was given ownership. That made me wonder if something was going on
internal to Galaxy, or something else, that was overwriting/ignoring the
system umask settings (which actually work fine in a shell environment as
the user itself). Maybe I'll look into that ACL stuff Paul mentioned.
Paul's suggestions are worth checking in to. I'd be interested in knowing
what the POSIX permissions are on /tmp itself, and what the ACLs are, if any.
Those temporary files are created by creating a temporary directory using
Python's tempfile.mkdtemp(), which creates them with a mode of 700, which is
then masked by the current umask. The change I added in the email you
referenced in your original post changes the directory to mode 0777 masked by
the umask (after it's created).
Depending on how the pieces used in RHEL's startup() shell function handle
the environment, you may be able to set it on the line above `daemon ...`
inside the 'start)' branch of the case statement. If that doesn't work, you
may need to get more creative and do something like:
daemon --user $GALAXY_USER umask 027; ${GALAXY_RUN} --daemon
--pid-file=${GALAXY_PID}
Alternatively, you can set it inside /home/galaxy/galaxy-dist/run.sh, since
this startup script uses run.sh.
--nate
Here is my /etc/init.d/galaxy script:
. /etc/rc.d/init.d/functions
GALAXY_USER=galaxy
GALAXY_DIST_HOME=/home/galaxy/galaxy-dist
GALAXY_RUN=${GALAXY_DIST_HOME}/run.sh
GALAXY_PID=${GALAXY_DIST_HOME}/paster.pid
case $1 in
start)
echo -n Starting galaxy services:
daemon --user $GALAXY_USER ${GALAXY_RUN}
--daemon --pid-file=${GALAXY_PID}
touch /var/lock/subsys/galaxy
;;
stop)
echo -n Shutting down galaxy services:
daemon --user $GALAXY_USER ${GALAXY_RUN}
--stop-daemon
rm -f /var/lock/subsys/galaxy
;;
status)
daemon --user galaxy ${GALAXY_RUN} --status
;;
restart)
$0 stop; $0 start
;;
reload)
$0 stop; $0 start
;;
*)
echo Usage: galaxy
{start|stop|status|reload|restart}
;;
esac
--
Thanks!
Josh
On Tue, Dec 4, 2012 at 9:56 AM, Nate Coraor n...@bx.psu.edu wrote:
On Dec 4, 2012, at 10:52 AM, Josh Nielsen wrote:
Hello all,
I am having issues downloading HTML files from Galaxy the same as is
described in this email chain:
http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-August/010965.html
I am getting the error (13)Permission denied: xsendfile: cannot open
file: /basedir/galaxy_data/database/tmp/tmp8iEccn/library_download.zip
which is indeed a basic filesystem permissions issue. The problem is that
the permissions created for that directory and every directory created in
tmp/ look like this:
drwx--+ 2 galaxy galaxy 3 Dec 4 09:23 tmp8iEccn
And I have placed the Apache user in the galaxy group, but as you can see
no group permissions ever get set by Galaxy on the directories that it
creates (it is getting a 700 permissions setting).
As Nate Coraor suggested in the message linked to above, I have tried
altering the default umask but I ran into issues with getting
non-existant results. I use sudo service galaxy start as the galaxy
user each time to start the server and a ps -ef | grep galaxy confirms
that Galaxy is running as the galaxy user. Since I use sudo though I
changed the sudoers file to include:
rootALL=(ALL) ALL
galaxy
Re: [galaxy-dev] Can't find new pause / resume functionality in Dec 2012 Galaxy
This isn't (at least at first) Pause/Resume as you might be expecting - where you could manually pause a currently running job and continue it later. What we're doing at least in the first pass is using 'Paused' as an internal state that jobs only go into in two scenarios: 1) User quota is full - all jobs waiting to run will be paused instead of waiting in the queue forever. After clearing their quota, paused jobs can be resumed. 2) A parent job (say, in a workflow) fails, then dependent jobs will be paused instead of having the cascaded error. You'll be able to rerun the failed parent and (if it succeeds) have paused jobs resume, instead of having to re-queue everything all over again. This isn't quite fully implemented yet, but will be finished in the near future. Other than being able to effectively resume workflows and potentially do other things with this framework, the changes should also reduce the load on Galaxy's job running system. -Dannon On Dec 3, 2012, at 7:31 PM, Greg Edwards gedwar...@gmail.com wrote: Hi all, I was keen to try the new Pause / Resume functionality mentuioned in http://wiki.galaxyproject.org/News/2012_12_03_DistributionNewsBrief but can't see it anywhere, in Galaxy or searches. It's not on the Test site at https://test.g2.bx.psu.edu/ or production at https://main.g2.bx.psu.edu/ Can you point me to where I can try it ? Thanks, -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] 回复： 回复： Speed up the galaxy
It seems old Illumina encoding 1.5 So yes some tools requiring the fastqsanger would need grooming. If you are up to programming you can seriously speed this up by using a precalculated transfer or hash table. That way you do not have to do any calculation but just translate each quality line using generic regexp/grep/sed like tools or $seq =~ t/STARTSCORES/SANGERSCORES/g in perl. The wiki table could be an alternative option. You still have to set the upload type to fastsanger likely since it will probably sniff the header and see its fastq but not 1.8+ encoding. Good luck! Alex Van: 泽 蔡 [caizexi...@yahoo.com.cn] Verzonden: dinsdag 4 december 2012 16:41 To: Bossers, Alex Cc: galaxy-dev@lists.bx.psu.edu Onderwerp: 回复： [galaxy-dev] 回复： Speed up the galaxy Hi Alex I look the page of wikipedia, but I have a little confused. We sequenced with solexa. I paste a read of my data, can you tell me whether I need run Groom? The read is like this:@HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 AGAAGTACATCGCGATGCCGTTNCCNNCGAAGGCGATAGNNNACAAGNCCAAATGNTTCTNCATCNNNCNCGAGNNGNCGAGGNCGCCGTGCGACCCTGC +HWUSI-EAS1734_0003_FC620JEAAXX:8:1:1174:9013#0/1 Ya^a`edddeddc\c`a`dc]\Ba^BBZ]ZZ`ZZZ]a]]BBB^[`\UB_V[V\`ZBSZX^ 发件人： Bossers, Alex alex.boss...@wur.nl 收件人： 泽 蔡 caizexi...@yahoo.com.cn 抄送： galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu; Nate Coraor n...@bx.psu.edu 发送日期： 2012年12月4日, 星期二, 下午 11:21 主题: RE: [galaxy-dev] 回复： Speed up the galaxy Hi I presume the best way to optimise your current problem is to evaluate whether you really need to groom your data!? If its old data presumably yes, but if it is recent data in Illumina 1.8+ encoding(http://en.wikipedia.org/wiki/FASTQ_format) it is not necessary speedup 100% :-) Groom would take long at our servers as well but due to the new Illumina format we didn’t bother to optimise it further by parallelisation for instance... Alex Van: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] Namens ? ? Verzonden: dinsdag 4 december 2012 16:08 Aan: Nate Coraor CC: galaxy-dev@lists.bx.psu.edu Onderwerp: [galaxy-dev] 回复： Speed up the galaxy Hi， I read the page of the document. But I don't think there is anything I need. Now, I installed a local instance of galaxy, and I just run it on a PC. I need to deal with large data, if I just use the default configure of galaxy, every task would take long time. For example I ran the FASTQ Groomer with a large file, the precedure is so slow, and galaxy would not use the potential of my machine. So I want to know how can I get galaxy ran faster. I already can upload files quickly and I just need to know how to ran tools quickly. 发件人： Nate Coraor n...@bx.psu.edumailto:n...@bx.psu.edu 收件人： 泽 蔡 caizexi...@yahoo.com.cnmailto:caizexi...@yahoo.com.cn 抄送： galaxy-...@bx.psu.edumailto:galaxy-...@bx.psu.edu galaxy-...@bx.psu.edumailto:galaxy-...@bx.psu.edu 发送日期： 2012年12月4日, 星期二, 下午 9:38 主题: Re: [galaxy-dev] Speed up the galaxy On Dec 4, 2012, at 4:36 AM, 泽 蔡 wrote: Hi all, How can I speed up the galaxy? Like how to use more cores and memeries. Hi, This is a pretty broad question. However, I would recommend that you start at: http://usegalaxy.org/production --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Can't find new pause / resume functionality in Dec 2012 Galaxy
Thanks Dannon. No worries. Yes my interest was different. On a private local Galaxy, or a private Cloud one, sometimes a job seems to be going awry, taking forever etc. I was interested in the ability to suspend it, check what's wrong, maybe fix it, maybe resume or cancel the job. Rgds, Greg E. On 5 December 2012 06:57, Dannon Baker dannonba...@me.com wrote: This isn't (at least at first) Pause/Resume as you might be expecting - where you could manually pause a currently running job and continue it later. What we're doing at least in the first pass is using 'Paused' as an internal state that jobs only go into in two scenarios: 1) User quota is full - all jobs waiting to run will be paused instead of waiting in the queue forever. After clearing their quota, paused jobs can be resumed. 2) A parent job (say, in a workflow) fails, then dependent jobs will be paused instead of having the cascaded error. You'll be able to rerun the failed parent and (if it succeeds) have paused jobs resume, instead of having to re-queue everything all over again. This isn't quite fully implemented yet, but will be finished in the near future. Other than being able to effectively resume workflows and potentially do other things with this framework, the changes should also reduce the load on Galaxy's job running system. -Dannon On Dec 3, 2012, at 7:31 PM, Greg Edwards gedwar...@gmail.com wrote: Hi all, I was keen to try the new Pause / Resume functionality mentuioned in http://wiki.galaxyproject.org/News/2012_12_03_DistributionNewsBrief but can't see it anywhere, in Galaxy or searches. It's not on the Test site at https://test.g2.bx.psu.edu/ or production at https://main.g2.bx.psu.edu/ Can you point me to where I can try it ? Thanks, -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] RPy testsuite some tests passing - enough for Galaxy?
Dear Galaxy devs,
We are in the process of deploying Galaxy internally here at AgResearch, and I
am currently installing required dependencies. My question relates to RPy. It
seems that RPy2 is not yet supported by Galaxy, so we're stuck with RPy 1.0.3a,
and this seems to have rusted quite badly. (I note however that the remarks
previously made on the Galaxy lists about changes required to RPy appear to
have been rolled into 1.0.3a.)
After installing RPy 1.0.3a, I ran the bundled test suite. Many tests succeed,
but there are several errors, in the following categories:
1. Errors about numpy, e.g. object of type numpy.int64 has no len() (Maybe
numpy has changed since RPy 1.0.3a was written. I am using numpy 1.4.1 on
CentOS 6.)
2. Errors due to missing files, e.g. logit.r and table.txt test data files do
not appear to ship with RPy-1.0.3a (!)
3. An error in testLapackLoad, which may be because of (2), I'm not sure yet.
4. An R object index out of range error in testGetItemSlice, which looks quite
serious.
So here's my question. Just how much of RPy has to work, for its use in Galaxy
to be OK?
I attach my RPy test suite output in case anyone is (a) interested, (b) has any
useful comments.
cheers,
Simon
Simon Guest
Senior UNIX Technical Consultant
AgResearch, NZ
===
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
===
impulsive$ python testall.py --random
Modules to be tested: ['test_io.py', 'test_boolean.py', 'test_util.py',
'test_lapack_load.py', 'test_array.py', 'test_cleanup.py', 'test_topy.py',
'test_tor.py', 'test_vars.py', 'test_modes.py', 'test_numeric.py',
'test_init.py', 'test_noninteractive.py', 'test_robj.py', 'test_sigint.py']
Testing: test_topy
..Warning message:
In file(file, rt) :
cannot open file 'table.txt': No such file or directory
E..
==
ERROR: testDataFrameToList (test_topy.TypeConversionToPyTestCase)
--
Traceback (most recent call last):
File /home/guestsi/rpmbuild/BUILD/rpy-1.0.3/tests/test_topy.py, line 65, in
testDataFrameToList
assert(r.read_table('table.txt', header=1) ==
RPy_RException: Error in file(file, rt) : cannot open the connection
--
Ran 21 tests in 0.027s
FAILED (errors=1)
Testing: test_boolean
..
--
Ran 2 tests in 0.000s
OK
Testing: test_numeric
.E.E.E...E.E.E..
==
ERROR: test16BitIntScalar (test_numeric.ArrayTestCase)
--
Traceback (most recent call last):
File /home/guestsi/rpmbuild/BUILD/rpy-1.0.3/tests/test_numeric.py, line
229, in test16BitIntScalar
b = r.c(a[0])
TypeError: object of type 'numpy.int16' has no len()
==
ERROR: test32BitFloatScalar (test_numeric.ArrayTestCase)
--
Traceback (most recent call last):
File /home/guestsi/rpmbuild/BUILD/rpy-1.0.3/tests/test_numeric.py, line
268, in test32BitFloatScalar
b = r.c(a[0])
TypeError: object of type 'numpy.float32' has no len()
==
ERROR: test32BitIntScalar (test_numeric.ArrayTestCase)
--
Traceback (most recent call last):
File /home/guestsi/rpmbuild/BUILD/rpy-1.0.3/tests/test_numeric.py, line
219, in test32BitIntScalar
b = r.c(a[0])
TypeError: object of type 'numpy.int32' has no len()
==
ERROR: test64BitIntScalar (test_numeric.ArrayTestCase)
--
Traceback (most recent call last):
File /home/guestsi/rpmbuild/BUILD/rpy-1.0.3/tests/test_numeric.py, line
209, in test64BitIntScalar
b = r.c(a[0])
TypeError: object of type 'numpy.int64' has no len()
==
ERROR: test8BitIntScalar (test_numeric.ArrayTestCase)
Re: [galaxy-dev] Permission denied error for velveth
I don't have a velvet or a submit-jobs-as-user environment setup so I cannot test this, but it seems like this should have a an easy fix. Can someone who is experiencing the problem try the attached patch or has something like this been tried and there is some other problem? -John (09:43:01 AM) jmchilton: Is the velvet problem related to something specific about how velvet runs or do no tools that depend on extra_files_path work with as-user jobs? (10:10:31 AM) natefoo: they have to either write to files_path or extra_files_path, let me find the email... (10:12:24 AM) natefoo: http://dev.list.galaxyproject.org/Folder-permissions-after-cluster-run-td4657156.html#a4657219 (10:12:26 AM) mrscribe: Title: Galaxy Development List Archive - Folder permissions after cluster run (at dev.list.galaxyproject.org) (11:07:31 AM) jmchilton: Is fixing the velvet problem as simple as changing extra_files_path to files_path in the wrappers then? On Tue, Dec 4, 2012 at 9:38 AM, Oleksandr Moskalenko o...@hpc.ufl.edu wrote: On Dec 4, 2012, at 9:30 AM, Andreas Kuntzagk andreas.kuntz...@mdc-berlin.de wrote: I see that this issue came up before by Oleksandr Moskalenko in October but was unresolved then. Is there a workaround now? regards, Andreas The velvet wrapper is not going to be usable for real user jobs until someone rewrites it to work without direct manipulation of files in the database/ tree. No work has been done on this nor any is planned as far as I know. Regards, Alex ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ velvet.patch Description: Binary data ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Running galaxy on multiple, physical servers
Hi,I am trying to figure out how to run galaxy distributed over several physical servers, with different IP addresses...The wiki on scaling and load balancing only speaks of balancing on the same physical server, but does not address how to put a handful of managers on a different physical server...Any ideas how to do this?I guess I can just copy the .ini file, define managers in that and try to start it on that server, but I am hesitant until I know this can work at all...Thankshttp://wiki.galaxyproject.org/Admin/Config/Performance/Web%20Application%20ScalingThon! ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] RFE: Adding tags to AWS resources created by Cloudman
Hi Shea, This is a reasonable feature so we'll add it on the todo list. To make the search for all untagged resources easier, did you have a list of the ones that are currently not getting tagged? Thanks, Enis On Tue, Dec 4, 2012 at 7:34 AM, Shea Lovan shea.lo...@lscg.ucsb.edu wrote: We are managing Galaxy/Cloudman clusters on AWS for a few research groups on campus. As is so often the case, we need to separate usage for billing purposes. This this is fairly simple to do with Amazon's CSV billing report with the clusterName tag included. However, it doesn't appear that I can configure Cloudman to set that tag on every resource it creates. This is forcing us to keep the scaling of the clusters under our control so that we can set the tags manually. What I was hoping, is that this feature could be added. It seems like a fairly simple patch; once the ID (instance, volume, or snapshot) is available, it's one more API call to create/set the clusterName tag. Any chance of getting this included? Thanks! -Shea -- Shea A. Lovan Manager, Server Operations and Architecture UCSB Life Sciences Computing Group Santa Barbara, CA 93106 805.893.2405 __**_ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Key error in latest build of galaxy-central
Got this when trying to install GMAP from the tool shed...Any ideas?Error Traceback:View as: Interactive|Text|XML(full)⇝KeyError: 'includes_repository_dependencies'URL:http://srv151/admin_toolshed/prepare_for_install?tool_shed_url=http://toolshed.g2.bx.psu.edu/repository_ids=4131098bea459833changeset_revisions=561503a442f0Moduleweberror.evalexception.middleware:364inrespond viewapp_iter=self.application(environ,detect_start_response)Modulepaste.debug.prints:98in__call__ viewenviron,self.app)Modulepaste.wsgilib:539inintercept_output viewapp_iter=application(environ,replacement_start_response)Modulepaste.recursive:80in__call__ viewreturnself.application(environ,start_response)Modulegalaxy.web.framework.middleware.remoteuser:91in__call__ viewreturnself.app(environ,start_response)Modulepaste.httpexceptions:632in__call__ viewreturnself.application(environ,start_response)Modulegalaxy.web.framework.base:160in__call__ viewbody=method(trans,**kwargs)Modulegalaxy.web.framework:206indecorator viewreturnfunc(self,trans,*args,**kwargs)Modulegalaxy.webapps.galaxy.controllers.admin_toolshed:1070inprepare_for_install viewincludes_repository_dependencies=util.string_as_bool(repo_information_dict['includes_repository_dependencies'])KeyError: 'includes_repository_dependencies'___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
