Re: [galaxy-dev] Irregular failures on cluster (pbs_python?)

2013-05-21 Thread Fields, Christopher J
On May 21, 2013, at 2:13 PM, Christopher Fields cjfie...@illinois.edu wrote: On May 20, 2013, at 3:08 PM, Nate Coraor n...@bx.psu.edu wrote: ... Hi Chris, We're disconnecting under all normal conditions and most error conditions - it looks like only a few conditions would not properly

Re: [galaxy-dev] Irregular failures on cluster (pbs_python?)

2013-05-21 Thread Fields, Christopher J
On May 20, 2013, at 3:08 PM, Nate Coraor n...@bx.psu.edu wrote: On May 20, 2013, at 3:31 PM, Fields, Christopher J wrote: On May 20, 2013, at 9:45 AM, Nate Coraor n...@bx.psu.edu wrote: On May 19, 2013, at 11:41 PM, Fields, Christopher J wrote: I've been seeing this error popping up

Re: [galaxy-dev] Irregular failures on cluster (pbs_python?)

2013-05-20 Thread Fields, Christopher J
On May 20, 2013, at 9:45 AM, Nate Coraor n...@bx.psu.edu wrote: On May 19, 2013, at 11:41 PM, Fields, Christopher J wrote: I've been seeing this error popping up quite a bit recently (we're using Torque 3.0.5), which is giving a general 'cluster failure' Galaxy error on jobs

[galaxy-dev] Irregular failures on cluster (pbs_python?)

2013-05-19 Thread Fields, Christopher J
I've been seeing this error popping up quite a bit recently (we're using Torque 3.0.5), which is giving a general 'cluster failure' Galaxy error on jobs: galaxy.jobs.runners.pbs WARNING 2013-05-19 18:56:44,073 (10588) pbs_submit failed (try 1/5), PBS error 15033: No free connections This

Re: [galaxy-dev] upload files by cluster with pbs-python returns empty files

2013-01-17 Thread Fields, Christopher J
I saw this same problem with our upload1 set to run on the cluster (it had worked properly a few months prior). I found out that the cluster sysadmins had set the nodes for local access only (no web access), direct file uploads and FTP to local disk worked but URL-based uploads did not. The

Re: [galaxy-dev] Questions about Install GATK on local instance

2012-12-13 Thread Fields, Christopher J
Older versions of GATK can be grabbed from Github: https://github.com/broadgsa/gatk/tags Not sure if they have an official 'older version' download site. chris On Dec 13, 2012, at 9:51 AM, yg...@cpdr.org wrote: Hi, Could anyone give me some suggestions about that? Thanks very much.

Re: [galaxy-dev] Impersonate a User

2012-12-13 Thread Fields, Christopher J
The user impersonation feature doesn't work with external authentication. There is a bug report on this: https://trello.com/c/zwqNwflD chris On Dec 13, 2012, at 10:01 AM, greg margeem...@gmail.com wrote: The problem is I disabled the logout button in universe.wsgi (I think) (I believe I

Re: [galaxy-dev] How to get a shedtool to run locally

2012-10-24 Thread Fields, Christopher J
On Oct 19, 2012, at 9:35 AM, Carlos Borroto carlos.borr...@gmail.com wrote: On Fri, Oct 19, 2012 at 12:00 AM, Fields, Christopher J cjfie...@illinois.edu wrote: My local one is set up like this and it works (for augustus): toolshed.g2.bx.psu.edu/repos/bjoern-gruening/augustus/augustus

Re: [galaxy-dev] The blastdb_p.loc sample is wrong.

2012-10-20 Thread Fields, Christopher J
Yes, have to agree there, an error would be more informative. chris On Oct 20, 2012, at 2:38 PM, Shane Sturrock sh...@biomatters.com wrote: That would certainly have saved me a fair bit of time since the only indication I got was an empty set of quotes in the error log. I always feel like

Re: [galaxy-dev] The blastdb_p.loc sample is wrong.

2012-10-19 Thread Fields, Christopher J
On Oct 4, 2012, at 3:11 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Oct 4, 2012 at 8:56 PM, Shane Sturrock sh...@biomatters.com wrote: I've just been setting up our internal galaxy server and following the instructions to get the NCBI BLAST+ tool working but I kept getting stuck

Re: [galaxy-dev] How to get a shedtool to run locally

2012-10-18 Thread Fields, Christopher J
On Oct 17, 2012, at 3:53 PM, Carlos Borroto carlos.borr...@gmail.com wrote: On Mon, Oct 15, 2012 at 11:49 AM, Liisa Koski liisa.ko...@basf.com wrote: Hello, I've installed a tool to my local Galaxy installation via my local ToolShed. I would like to run this tool locally, and not have it

[galaxy-dev] Odd issue with updateucsc.dh

2012-10-17 Thread Fields, Christopher J
I had an odd error with wigToBigWig that I tracked back to the *.len files generated by the updateucsc.sh script. With my last UCSC update, a majority of the assemblies have this as the single line: META http-equiv=Content-Script-Type content=text/javascript but others, like

Re: [galaxy-dev] Redirect log files?

2012-09-18 Thread Fields, Christopher J
`round). chris On Sep 16, 2012, at 1:34 PM, James Taylor ja...@jamestaylor.org wrote: On Sunday, September 16, 2012, Fields, Christopher J wrote: This seems to correspond with what I have tried; I attempter setting log_destination in the config file (as James suggested) and the server

Re: [galaxy-dev] Trackster caching

2012-09-18 Thread Fields, Christopher J
On Sep 15, 2012, at 11:05 PM, Jeremy Goecks jeremy.goe...@emory.edu wrote: is there a way to determine whether Trackster is the cause? The only place where Trackster caches data is in the SummaryTreeDataProvider. In galaxy-central, the relevant line is 709 in

Re: [galaxy-dev] Redirect log files?

2012-09-16 Thread Fields, Christopher J
updates. chris On Sep 12, 2012, at 2:32 PM, Nate Coraor n...@bx.psu.edu wrote: On Sep 12, 2012, at 1:09 PM, Fields, Christopher J wrote: Simple question: is there a way within universe_wsgi.ini to point to a specific location where Galaxy log (and possibly PID) files can be dropped? I've

[galaxy-dev] Trackster caching

2012-09-14 Thread Fields, Christopher J
We're seeing an odd issue that seems to be related to Trackster's caching of track information, mainly that memory in the web server processes increases. It appears that tracks are being cached in memory per web process. If I create a new visualization (and monitor our web processed via top)

Re: [galaxy-dev] Odd (possible latency/race?) problems post-upgrade

2012-08-02 Thread Fields, Christopher J
30, 2012 at 7:03 PM, Fields, Christopher J cjfie...@illinois.edu wrote: I'm seeing a possible latency issue or race condition when starting Galaxy after the latest hg upgrade (July 20) from galaxy-dist; the prior upgrade doesn't have this problem. We have small setup with one job manager

Re: [galaxy-dev] Shell script to start Galaxy in multi-server environment

2012-08-02 Thread Fields, Christopher J
The docs about that are here: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling I am working on combining our separate ini files myself, but ran into some odd problems with cluster job submission that weren't present with the original split files (e.g. the

[galaxy-dev] Odd (possible latency/race?) problems post-upgrade

2012-07-30 Thread Fields, Christopher J
I'm seeing a possible latency issue or race condition when starting Galaxy after the latest hg upgrade (July 20) from galaxy-dist; the prior upgrade doesn't have this problem. We have small setup with one job manager/runner and two web front-ends for testing load balancing: …from

Re: [galaxy-dev] Configuring Galaxy for FTP upload

2012-06-11 Thread Fields, Christopher J
On Jun 11, 2012, at 9:10 AM, Nate Coraor wrote: On Jun 10, 2012, at 8:56 PM, Ciara Ledero wrote: Hi all, So I was not able to configure Galaxy immediately after this post of mine due to the tasks that I first needed to do. Looking back, I forgot to ask this question: Do I need to

Re: [galaxy-dev] Running Galaxy on PBS/Torque cluster

2012-05-10 Thread Fields, Christopher J
On May 10, 2012, at 11:02 AM, Nate Coraor wrote: On May 3, 2012, at 3:42 PM, Dorset, Daniel C wrote: I have had Galaxy running on a local BAS, and after success with that we’re ready to create an instance on our local cluster computing service. The local cluster uses PBS/Torque. I’m

Re: [galaxy-dev] galaxy cluster integration

2012-05-07 Thread Fields, Christopher J
We are using PBS locally and had our cluster admin set this up, but I think the same principle applies across most job runners/mgrs. As Peter mentioned, there should be a way for cluster admin to add your server as an allowed submitting host, the PBS variant is here:

Re: [galaxy-dev] Interested in speaking with other institutions deploying Galaxy locally?

2012-04-29 Thread Fields, Christopher J
We've also started a local deployment for University of Illinois Urbana-Champaign. So I would be interested in participating. Chris On Apr 27, 2012, at 2:32 PM, David Hoover hoove...@helix.nih.gov wrote: Not a bad idea. Most of the conversations on this list are very limited and detailed

Re: [galaxy-dev] Error while uploading

2012-03-12 Thread Fields, Christopher J
You can change the default location of tmp files by modifying the environment; we ran into this with samtools directly dumping temp sort files to a local node, causing the job to shut down. Setting TMPDIR worked for me. chris On Mar 11, 2012, at 5:10 PM, Dorset, Daniel C wrote: Hi Luciano,

Re: [galaxy-dev] Variable number of inputs in Workflow (or optional inputs)

2012-03-06 Thread Fields, Christopher J
This seems to fall within the recent discussion of map/reduce operators for workflows, e.g. breaking large jobs up for embarrassingly parallel tasks, then merging them back at some later point. Dannon mentioned some basic functionality does exist within Galaxy to do this, but it's at an early

Re: [galaxy-dev] anyone using SFTP?

2012-02-20 Thread Fields, Christopher J
Yes, we are using this locally (sftp and scp). Basically follow the same rules as for FTP on the wiki, but leave out the ProFTP setup; this was per Nate's instructions, whom I believe is to change the FTP-specific documentation at some point. Basically, any mechanism that transfers data to

Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)

2012-02-17 Thread Fields, Christopher J
If they used a package install, I think the DRMAA libs for torque are normally separate. For instance: http://pkgs.org/download/torque-drmaa If going from source, you can enable DRMAA compilation with --enable-drmaa; I don't recall if that is on by default (I don't think it is). chris On

[galaxy-dev] Dynamic DRMAA options, was Re: Galaxy and Cluster options

2012-02-17 Thread Fields, Christopher J
This was recently brought up in a related thread by Nate, the code is in a pull request on bitbucket. Changing the subject to see if that rings any additional bells… Here's the original post by John Chilton and link to the fork:

Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)

2012-02-17 Thread Fields, Christopher J
And it works now! :D Have a great weekend, L-A Le 17/02/2012 17:07, Fields, Christopher J a écrit : If they used a package install, I think the DRMAA libs for torque are normally separate. For instance: http://pkgs.org/download/torque-drmaa If going from source, you can enable DRMAA

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Fields, Christopher J
On Feb 16, 2012, at 4:47 AM, Peter Cock wrote: On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker dannonba...@me.com wrote: Good luck, let me know how it goes, and again - contributions are certainly welcome :) I think I found the first bug, method split in lib/galaxy/datatypes/sequence.py

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-16 Thread Fields, Christopher J
On Feb 16, 2012, at 12:24 PM, Peter Cock wrote: On Thu, Feb 16, 2012 at 4:28 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Dan, I think I need a little more advice - what is the role of the script scripts/extract_dataset_part.py and the JSON files created when splitting FASTQ files in

Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?

2012-02-15 Thread Fields, Christopher J
Ah, was just about to ask about this as well, nice to know something is already in place (as experimental as it might be). Thanks Dannon! chris On Feb 15, 2012, at 11:08 AM, Dannon Baker wrote: It's definitely an experimental feature at this point, and there's no wiki, but basic support for

Re: [galaxy-dev] Status on importing BAM file into Library does not update

2012-02-13 Thread Fields, Christopher J
On Feb 13, 2012, at 9:45 AM, Nate Coraor wrote: On Feb 8, 2012, at 9:32 PM, Fields, Christopher J wrote: 'samtools sort' seems to be running on our server end as well (not on the cluster). I may look into it a bit more myself. Snapshot of top off our server (you can see our local runner

Re: [galaxy-dev] Status on importing BAM file into Library does not update

2012-02-08 Thread Fields, Christopher J
'samtools sort' seems to be running on our server end as well (not on the cluster). I may look into it a bit more myself. Snapshot of top off our server (you can see our local runner as well): PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 3950 galaxy

Re: [galaxy-dev] Status on importing BAM file into Library does not update

2012-02-08 Thread Fields, Christopher J
Forgot to add, but this also seems tied to the same problem Ryan's describing. IIRC Galaxy also runs 'samtools sort' after certain jobs, correct? chris On Feb 8, 2012, at 8:32 PM, Fields, Christopher J wrote: 'samtools sort' seems to be running on our server end as well (not on the cluster

Re: [galaxy-dev] Multiple input file bug

2012-02-03 Thread Fields, Christopher J
tried a simple workflow with nothing but an input and FastQC and ran with multiple inputs successfully. Thanks! Dannon On Feb 2, 2012, at 11:15 PM, Fields, Christopher J wrote: Just filed as Issue #714 on bitbucket. I noticed that multiple input seems to be broken on workflows when

Re: [galaxy-dev] Multiple input file bug

2012-02-03 Thread Fields, Christopher J
. That said, it's definitely on the wishlist to expand multi-input capabilities and allow for things like paired dataset inputs, etc., and at that time I will revisit allowing direct dataset inputs as multiples. Thanks! -Dannon On Feb 3, 2012, at 12:17 PM, Fields, Christopher J

[galaxy-dev] Multiple input file bug

2012-02-02 Thread Fields, Christopher J
Just filed as Issue #714 on bitbucket. I noticed that multiple input seems to be broken on workflows when selecting more that one file (a single file works). This is happening on our local instance, using Jan 27 galaxy-dist release, as well as both main/test.g2.bx.psu.edu Galaxy servers

Re: [galaxy-dev] running functional tests on a galaxy server

2012-01-20 Thread Fields, Christopher J
wrote: On Jan 19, 2012, at 12:57 AM, Fields, Christopher J wrote: Just to add for the list, nate mentioned on IRC that running tests on an external server likely doesn't work at the moment but that ./scripts/functional_tests.py may have something to work with via environment variables

Re: [galaxy-dev] LDAP authentification

2012-01-19 Thread Fields, Christopher J
We also ran into this. Our local server configuration was case-sensitive and the LDAP uid key was converted to upper-case, so we used: RequestHeader set REMOTE_USER %{AUTHENTICATE_UID}e which fixed the problem. We found the solution by searching for 'AUTHENTICATE_uid', it appears to be a

Re: [galaxy-dev] LDAP authentification

2012-01-19 Thread Fields, Christopher J
David, This is the approach recommended on the wiki for performance: http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy Would be nice if your alternative approach using virtual hosts were similarly documented. chris On Jan 19, 2012, at 10:01 AM, David Hoover wrote: Why do you need to

[galaxy-dev] running functional tests on a galaxy server

2012-01-18 Thread Fields, Christopher J
I have been following the basic protocol for running remote tests on a Galaxy server (in this case, our local one) using: http://wiki.g2.bx.psu.edu/Admin/Running%20Tests From reading this, I assume the tests are to be run from the galaxy root directory, like so: GALAXY_TEST_EXTERNAL=1

Re: [galaxy-dev] running functional tests on a galaxy server

2012-01-18 Thread Fields, Christopher J
run_functional_tests.sh There are several options you can use as well. Greg On Jan 18, 2012, at 4:38 PM, Fields, Christopher J wrote: I have been following the basic protocol for running remote tests on a Galaxy server (in this case, our local one) using: http://wiki.g2.bx.psu.edu/Admin/Running%20Tests

Re: [galaxy-dev] running functional tests on a galaxy server

2012-01-18 Thread Fields, Christopher J
Just to add for the list, nate mentioned on IRC that running tests on an external server likely doesn't work at the moment but that ./scripts/functional_tests.py may have something to work with via environment variables. chris On Jan 18, 2012, at 8:03 PM, Fields, Christopher J wrote: Greg

Re: [galaxy-dev] PBS job failed: Unknown error: -10

2012-01-05 Thread Fields, Christopher J
Do you mean this (In universe_wsgi.ini)? ... # Debug enables access to various config options useful for development and # debugging: use_lint, use_profile, use_printdebug and use_interactive. It # also causes the files used by PBS/SGE (submission script, output, and error) # to remain on disk

Re: [galaxy-dev] Exiting a java application without errors

2011-12-22 Thread Fields, Christopher J
The first option (wrapper) also gives one the opportunity to actually check for errors. Chris On Dec 22, 2011, at 2:21 AM, Mikel Egaña Aranguren meg...@fi.upm.es wrote: Hi; My interpretation is that your BioJava tool is sending normal information to stderr, and Galaxy interprets it as an

Re: [galaxy-dev] running two Galaxy installations on one server - was: Re: Can not clean my galaxy datasets

2011-12-22 Thread Fields, Christopher J
We are anticipating a similar issue and decided it would be easier to just have separate VMs where possible. I know this isn't always possible, but it is a very easy way to both manage multiple galaxy instances and keep the various galaxy instances completely separate. Chris On Dec 22, 2011,

Re: [galaxy-dev] Job output not returned from cluster

2011-11-29 Thread Fields, Christopher J
On Nov 29, 2011, at 3:13 AM, Peter Cock wrote: On Monday, November 28, 2011, Joseph Hargitai joseph.hargi...@einstein.yu.edu wrote: Ed, we had the classic goof on our cluster with this. 4 nodes could not see the /home/galaxy folder due to a missing entry in /etc/fstab. When the jobs

Re: [galaxy-dev] Inquiring

2011-11-17 Thread Fields, Christopher J
Try http://test.g2.bx.psu.edu/ chris On Nov 15, 2011, at 3:13 PM, Yan Luo wrote: Hi, Dannon, Thanks for your response. I just tried test.bx.psu.edu, but it was not available. Could you please double check the address? Looking forward to hearing from you. Thanks, Yan On Tue,

Re: [galaxy-dev] Clusters, Runners, and user credentials

2011-11-03 Thread Fields, Christopher J
, Fields, Christopher J wrote: I recall at the Galaxy conf there were questions on how secure this is (having the 'galaxy' user submit jobs as someone else). This would involve switching users on the cluster or would require user login information, correct? The way we planned on working

Re: [galaxy-dev] Clusters, Runners, and user credentials

2011-11-03 Thread Fields, Christopher J
folks here re: security concerns and possible solutions for user job submissions (there wasn't much added yet beyond what you already covered, unfortunately). chris Fields, Christopher J wrote: Ilya, Nate, To add a bit of background to the below, we have several clusters on campus

Re: [galaxy-dev] Clusters, Runners, and user credentials

2011-11-01 Thread Fields, Christopher J
I recall at the Galaxy conf there were questions on how secure this is (having the 'galaxy' user submit jobs as someone else). This would involve switching users on the cluster or would require user login information, correct? The way we planned on working around this was to just specify a

Re: [galaxy-dev] Data Retention Policies

2011-10-24 Thread Fields, Christopher J
On Oct 24, 2011, at 2:57 AM, Peter Cock wrote: On Mon, Oct 24, 2011 at 4:53 AM, John Chilton chil...@msi.umn.edu wrote: Hello All, This is more of a general policy question. The head of our Research Informatics Support Systems group here at the University of Minnesota wants to know how

Re: [galaxy-dev] Blast database permissions

2011-10-19 Thread Fields, Christopher J
On Oct 19, 2011, at 4:33 AM, Peter Cock wrote: On Wed, Oct 19, 2011 at 10:24 AM, Jeroen Crappé jeroen.cra...@ugent.be wrote: Dear all, I have recently installed a local galaxy on our servers. I could succesfully add a blastable database via the blastdb.loc file. All is working, so no

Re: [galaxy-dev] Blast database permissions

2011-10-19 Thread Fields, Christopher J
On Oct 19, 2011, at 8:03 AM, Peter Cock wrote: On Wed, Oct 19, 2011 at 1:41 PM, Fields, Christopher J cjfie...@illinois.edu wrote: On Oct 19, 2011, at 4:33 AM, Peter Cock wrote: On Wed, Oct 19, 2011 at 10:24 AM, Jeroen Crappé jeroen.cra...@ugent.be wrote: Dear all, I have recently

Re: [galaxy-dev] disk space and file formats

2011-09-08 Thread Fields, Christopher J
The use of (unaligned) BAM for readgroups seems like a good idea. At the very least it prevents inconsistently hacking this information into the FASTQ descriptor (a common problem with any simple format). chris On Sep 8, 2011, at 1:35 PM, Edward Kirton wrote: copied from another thread:

Re: [galaxy-dev] Suggestion / Request for Comments on Galaxy Best Practices - Gradual migration to standard indentation

2011-09-02 Thread Fields, Christopher J
On Sep 2, 2011, at 9:57 AM, Nate Coraor wrote: ... Chatting to some of the Galaxy team at BOSC/ISMB 2011 there is some support for this internally. Again, there are automated tools to do this. would anyone be willing to add the appropriate hooks to the central repository as well? As

Re: [galaxy-dev] disk space and file formats

2011-09-02 Thread Fields, Christopher J
On Sep 2, 2011, at 3:02 PM, Edward Kirton wrote: What, like a BAM file of unaligned reads? Uses gzip compression, and tracks the pairing information explicitly :) Some tools will already take this as an input format, but not all. ah, yes, precisely. i actually think illumina's pipeline

Re: [galaxy-dev] disk space and file formats

2011-09-02 Thread Fields, Christopher J
On Sep 2, 2011, at 8:02 PM, Peter Cock wrote: On Fri, Sep 2, 2011 at 9:27 PM, Fields, Christopher J cjfie...@illinois.edu wrote: On Sep 2, 2011, at 3:02 PM, Edward Kirton wrote: What, like a BAM file of unaligned reads? Uses gzip compression, and tracks the pairing information explicitly