Re: [galaxy-dev] [galaxy-user] operating on, and getting at, large files in galaxy...

2011-02-17 Thread Edward Kirton
Hi Nick, Yes, these nextgen reads files are huge and getting bigger every quarter! But there will be storage issues nomatter whether you use Galaxy or not. In fact, i think users are more likely to cleanup files and histories in galaxy than they are to cleanup NFS folders -- out of sight, out

Re: [galaxy-dev] Tools to organize long histories

2011-02-17 Thread Edward Kirton
you can have your workflow hide intermediate files. also, renaming files helps. i usually recommend to users to split their work into several histories rather than trying to do everything in one huge history (e.g. read qc, assembly, gene annotation, read mapping, expression analysis) On Wed,

Re: [galaxy-dev] Creating new datatype with variable number of input files

2011-02-17 Thread Edward Kirton
use the repeat tag to accept a list of infiles (and perhaps labels/timepoint/etc for each). On Thu, Jan 13, 2011 at 6:22 AM, Kempenaar, M (med) m.kempen...@med.umcg.nl wrote: Hello, This is the first question I ask on this list, please let me know if I'm 'doing it wrong'. Currently I'm

[galaxy-dev] MarkupSafe egg cannot be fetched

2011-04-23 Thread Edward Kirton
hello, after pulling the latest changes from galaxy-central, i get the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched $ python scripts/fetch_eggs.py Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched

[galaxy-dev] quick question: how can i supply the user's email address to a tool?

2011-05-05 Thread Edward Kirton
is there a variable i can use in the tool config xml file? thanks! ed ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

Re: [galaxy-dev] quick question: how can i supply the user's email address to a tool?

2011-05-06 Thread Edward Kirton
Hi Nate, can I access the values in the universe_wsgi.ini file? How would I retrieve the value of smtp_server for example? (Or do i just parse the file myself?) Thanks! On Fri, May 6, 2011 at 9:06 AM, Nate Coraor n...@bx.psu.edu wrote: __tool_data_path__ = GALAXY_DATA_INDEX_DIR =

Re: [galaxy-dev] quick question: how can i supply the user's email address to a tool?

2011-05-09 Thread Edward Kirton
awesome; thanks everyone for your help. On Fri, May 6, 2011 at 9:03 PM, Nate Coraor n...@bx.psu.edu wrote: Edward Kirton wrote: Hi Nate, can I access the values in the universe_wsgi.ini file? How would I retrieve the value of smtp_server for example? (Or do i just parse the file myself

Re: [galaxy-dev] galaxy with SGE cluster

2011-06-17 Thread Edward Kirton
yes, your web server needs to be configured as an sge submit host to work seamlessly with galaxy. alternatives include submitting the jobs to the cluster outside of galaxy using another script that will either ssh or use expect. these alternatives are messy and to be avoided unless necessary.

[galaxy-dev] fyi: can't upload to toolshed

2011-07-14 Thread Edward Kirton
Server ErrorAn error occurred. See the error logs for more information. (Turn debug on to display exception reports here) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2011-07-19 Thread Edward Kirton
pushed what code I had to the toolshed repository qiime and gave you push permissions. I'm not sure when I'll have time to work on it again. Thanks, JJ On 7/15/11 3:18 PM, Edward Kirton wrote: fantastic, thanks! i'm also available to help out, so perhaps you can find me something to work

Re: [galaxy-dev] use remote data without duplication

2011-07-19 Thread Edward Kirton
besides using libraries, you can create a tool to add files from nfs path which merely symlinks the file. while libraries are nice, i haven't figured out how to automatically import libraries, so i created a tool for users to import sequence data using their run id (i.e. in our lims). On Tue,

[galaxy-dev] workflows are broken in current galaxy-central version

2011-07-21 Thread Edward Kirton
URL: https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8 Module paste.exceptions.errormiddleware:*143* in __call__ Module paste.debug.prints:*98* in __call__ Module paste.wsgilib:*539* in intercept_output Module paste.recursive:*80* in __call__ Module paste.httpexceptions:*632* in

Re: [galaxy-dev] workflows are broken in current galaxy-central version

2011-07-21 Thread Edward Kirton
yup, works. thanks for the prompt reply On Thu, Jul 21, 2011 at 10:15 AM, Kanwei Li kan...@gmail.com wrote: Fixed on trunk, thanks for reporting! K On Thu, Jul 21, 2011 at 12:41 PM, Edward Kirton eskir...@lbl.gov wrote: URL: https://galaxy.jgi-psf.org/workflow/editor?id=3918559ba2a747d8

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?

2011-07-27 Thread Edward Kirton
here's what i have; i started categorizing the tools under labels but hadn't finished. i'm also working on getting these tools working in galaxy (albeit slowly as we won't need them until we get a miseq machine in oct), but maybe we can be in communication with each other and jjohnson. section

Re: [galaxy-dev] organizing histories in galaxy

2011-07-28 Thread Edward Kirton
i usually prefix the history names with an identifier so i can search for them (e.g. AmiMT: read QC). but i agree, folders similar to the data libraries would be useful, so i created a ticket. https://bitbucket.org/galaxy/galaxy-central/issue/621/folders-to-organize-saved-histories On Fri, Jul

[galaxy-dev] Job output not returned from cluster

2011-07-28 Thread Edward Kirton
I've been getting these errors sometimes lately, particularly when the cluster is heavily loaded. The jobs have completed successfully, as I can see the output if I click the pen icon, but the job is in a failed state. Have any other sites been experiencing this problem? Or can the galaxy

Re: [galaxy-dev] Job output not returned from cluster

2011-07-29 Thread Edward Kirton
thanks for your comments, fellas. permissions would certainly cause this problem, but that's not the cause for me. most wrappers just serve to redirect stderr, so i don't think it's the wrapper script itself, but the stdout/stderr files are part of the problem. the error message is thrown in

Re: [galaxy-dev] Galaxy public instances: data and job quotas

2011-08-24 Thread Edward Kirton
love the quotas; this was sorely needed. thanks much. ed On Fri, Aug 19, 2011 at 4:44 PM, Jennifer Jackson j...@bx.psu.edu wrote: Galaxy public instances: data and job quotas User data and job quota limits are now implemented at the public Galaxy Test instance http://test.g2.bx.psu.edu:

Re: [galaxy-dev] Galaxy public instances: data and job quotas

2011-08-24 Thread Edward Kirton
i created some quotas which were assigned to groups (set as for yes, registered users) but apparently the groups didn't initially stick; i had to edit the quota after creation to reassign the group. On Fri, Aug 19, 2011 at 4:44 PM, Jennifer Jackson j...@bx.psu.edu wrote: Galaxy public

Re: [galaxy-dev] Galaxy public instances: data and job quotas

2011-08-24 Thread Edward Kirton
also, delete and purge a quota seems to have no effect. On Wed, Aug 24, 2011 at 3:58 PM, Edward Kirton eskir...@lbl.gov wrote: i created some quotas which were assigned to groups (set as for yes, registered users) but apparently the groups didn't initially stick; i had to edit the quota after

[galaxy-dev] downloading large files

2011-08-26 Thread Edward Kirton
i thought i recalled reading about downloading files from a history via ftp, but i could been mistaken -- couldn't find anything on the wiki or mailing list archives. does this feature exist? what's the best way for users to download many or large files other than via the browser?

Re: [galaxy-dev] Selective storage of galaxy files

2011-08-26 Thread Edward Kirton
An easy and immediate solution may be to: (a) create a Link data tool. The user specifies the data ID and your tool queries a db to find the location and creates a symlink to the data, which is stored on different groups'/projects' disks. While subsequent datafiles will still be stored in the

Re: [galaxy-dev] using Galaxy for map/reduce

2011-08-26 Thread Edward Kirton
Not intending to hijack the thread, but in response to John's comment -- I, too, made a general solution for embarassingly parallel problems but instead of splitting the large files on disk, I just use seek to move the file pointer so each task can grab it's part. On Tue, Aug 2, 2011 at 10:54 AM,

Re: [galaxy-dev] downloading large files

2011-08-26 Thread Edward Kirton
okay, thanks. i'll create a tool to export files as a tarball in the user's ftp folder, and couple it with a cron job to make sure the files are deleted after a week. i'll contribute it to the toolshed when done. On Fri, Aug 26, 2011 at 11:59 AM, Nate Coraor n...@bx.psu.edu wrote: Edward

Re: [galaxy-dev] database migration error 79-80 (NameError: name 'BigInteger' is not defined)

2011-08-31 Thread Edward Kirton
please disregard my previous message, it was the developer's error; he accidentally had deleted or reverted something in the libs folder. On Wed, Aug 31, 2011 at 3:36 PM, Edward Kirton eskir...@lbl.gov wrote: hi, we are getting the following error migrating from 79 to 80. curiously, my own

Re: [galaxy-dev] HOW TO RETRIEVE DATA FROM HISTORY??!!

2011-09-01 Thread Edward Kirton
why not create a simple export tool? perhaps with the option to cp or symlink. On Thu, Aug 4, 2011 at 9:57 PM, colin molter colin.mol...@gmail.com wrote: Hi all, i am still stuck with the same problem. Is there a way to directly move/copy data from your galaxy history to a given location in

Re: [galaxy-dev] disk space and file formats

2011-09-02 Thread Edward Kirton
What, like a BAM file of unaligned reads? Uses gzip compression, and tracks the pairing information explicitly :) Some tools will already take this as an input format, but not all. ah, yes, precisely. i actually think illumina's pipeline produces files in this format now. wrappers which

Re: [galaxy-dev] disk space and file formats

2011-09-02 Thread Edward Kirton
 i actually think illumina's pipeline produces files in this format (unaligned-bam) now. Oh do they? - that's interesting. Do you have a reference/link? i caught wind of this at the recent illumina user's conference but i asked someone in our sequencing team to confirm and he hadn't heard of

Re: [galaxy-dev] disk space and file formats

2011-09-02 Thread Edward Kirton
In your position I agree that is a pragmatic choice. Thanks for helping me muddle through my options. You might be able to modify the file upload code to gzip any FASTQ files... that would prevent uncompressed FASTQ getting into new histories. Right! I wonder if Galaxy would benefit from

Re: [galaxy-dev] Tophat non Sanger input

2011-09-07 Thread Edward Kirton
seems unnecessary since illumina switched over to fastqsanger now. http://www.illumina.com/truseq/quality_101/quality_scores.ilmn On Wed, Aug 31, 2011 at 12:45 AM, Stephen Taylor stephen.tay...@imm.ox.ac.uk wrote: Hi, Is there any plans to enhance the tophat wrapper to accept non Sanger

Re: [galaxy-dev] Automatically removing items from history

2011-09-07 Thread Edward Kirton
i'm resurrecting this thread to see if there's any more support for the idea of deleting intermediate files in a workflow. i think this is an important feature to have. oftentimes a workflow creates many intermediate files no one will ever look at. and leaving it up to the user to cleanup their

Re: [galaxy-dev] disk space and file formats

2011-09-08 Thread Edward Kirton
copied from another thread: On Thu, Sep 8, 2011 at 7:30 AM, Anton Nekrutenko an...@bx.psu.edu wrote: What we are thinking of lately is switching to unaligned BAM for everyting. One of the benefits here is the ability to add readgroups from day 1 simplifying multisample analyses down the road.

Re: [galaxy-dev] The need for wrappers

2011-10-13 Thread Edward Kirton
wrappers are also used when a tool produces several files in an output folder (i.e. executable takes outdir parameter, not explicitly named outfile paths) and you would like to move these to the desired paths under files/ (i.e. a composite datatype is not desired) and/or cleanup unnecessary files.

Re: [galaxy-dev] only showing first lines of large output file

2011-11-28 Thread Edward Kirton
hi matthias, i've also been getting metadata errors (below) for several weeks, but it works with the set_metadata_externally option (in the universe config file) set to False. for large files, it could take many minutes for your job server to check the metadata, and no new jobs will be run or

Re: [galaxy-dev] Job output not returned from cluster

2011-11-28 Thread Edward Kirton
hi, we've had this issue too -- in short, the cluster node(s) finish writing outfiles to disk, but the file system (inode metadata) isn't updated at the galaxy server yet when galaxy checks for the files. turning the metadata caching off (as recommended on the galaxy wiki) isn't an option for me

Re: [galaxy-dev] lims integration

2011-12-01 Thread Edward Kirton
Hi Chris, unfortunately none of us here have played around with the API yet. I would recommend inquiring on the galaxy-central's mailing list ( mailto:galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu). - workflows, histories, libraries, and datasets have IDs in the database but they may

Re: [galaxy-dev] Job output not returned from cluster

2011-12-01 Thread Edward Kirton
yes, i think galaxy already grabs these files. i seem to recall this process would get stuck if the output was too large (i was running something with a --debug/verbose option and galaxy would not finish the job even though it was off the cluster -- had to redirect to a log file). so i guess

Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException

2012-01-12 Thread Edward Kirton
sometimes the scheduler can't keep up with all the work in it's 15sec cycle, so it doesn't respond to some messages. here's a fix i've been trying that seems to work. in lib/galaxy/jobs/runners/drmaa.py: def check_watched_items( self ): Called by the monitor thread to look

[galaxy-dev] how to use projects for fair-share on compute-cluster

2012-01-12 Thread Edward Kirton
solution works well enough for me). Edward Kirton US DOE JGI ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http

Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException

2012-01-13 Thread Edward Kirton
be related to a valid job being moved into failure state ? Did you follow it down the rabbit hole by any chance to see what caused the hang in your env ? Thanks again, Ann From: Edward Kirton eskir...@lbl.gov Date: Thu, 12 Jan 2012 13:00:27 -0800 To: Ann Black annbl...@eng.uiowa.edu Cc

Re: [galaxy-dev] how to use projects for fair-share on compute-cluster

2012-01-13 Thread Edward Kirton
correction: i didn't adequately test what happens if the user_proj_map_db was not defined in the universe file; here's the changes: 157 # BEGIN ADD USER'S PROJ 158 try: 159 conn = sqlite3.connect(self.app.config.user_proj_map_db) 160 c = conn.cursor() 161

Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException

2012-01-18 Thread Edward Kirton
and related discussion on it: https://bitbucket.org/galaxy/galaxy-central/pull-request/23/catch-drmcommunication-exception -- Shantanu On Jan 13, 2012, at 2:11 PM, Edward Kirton wrote: i had seen the job process die with this error: if state != old_state: UnboundLocalError: local

Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException

2012-01-18 Thread Edward Kirton
to check job status. %( galaxy_job_id, job_id )) # give scheduler time to catch up time.sleep( 15 ) # keep old job state new_watched.append( drm_job_state ) continue }}} On Wed, Jan 18, 2012 at 1:07 PM, Edward Kirton eskir...@lbl.gov wrote: excellent, thanks for the correction

Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException

2012-01-20 Thread Edward Kirton
yes, nate but that fails the job but it is, in fact, still running and the error should be ignored except Exception, e: # so we don't kill the monitor thread log.exception((%s/%s) Unable to check job status % ( galaxy_job_id, job_id ) )

Re: [galaxy-dev] how to use projects for fair-share on compute-cluster

2012-01-20 Thread Edward Kirton
default that allowed users to select their current project/role would remove the single-project-per-user limitation. --nate On Jan 13, 2012, at 3:17 PM, Edward Kirton wrote: correction: i didn't adequately test what happens if the user_proj_map_db was not defined in the universe file; here's

Re: [galaxy-dev] Timeout and Galaxy - Cluster could not complete job

2012-02-28 Thread Edward Kirton
i believe the latest stable update of galaxy included changes to drmaa.py which allows a job to be rechecked indefinitely with regard to scheduler communication errors, so perhaps your cluster could not complete job errors are due to a filesystem race condition, whereby the cluster node completes

Re: [galaxy-dev] HMMER wrappers

2012-02-29 Thread Edward Kirton
hi, peter - i will fix the description and upload hmmsearch and infernal today. On Wed, Feb 29, 2012 at 2:05 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Tue, Feb 28, 2012 at 10:31 PM, Dave Clements cleme...@galaxyproject.org wrote: Hi Peter, I think hmmerscan has been wrapped, but

Re: [galaxy-dev] HMMER wrappers

2012-02-29 Thread Edward Kirton
great suggestion; i'll make those changes On Wed, Feb 29, 2012 at 10:05 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Wed, Feb 29, 2012 at 5:56 PM, Edward Kirton eskir...@lbl.gov wrote: hi, peter - i will fix the description and upload hmmsearch and infernal today. Great. Assuming

Re: [galaxy-dev] HMMER wrappers

2012-03-05 Thread Edward Kirton
i created a new toolshed repo, hmmer since i couldn't rename it. as suggested, it has the hmmscan/hmmsearch as one tool, plus hmmpress. will add hmmbuild, hmmalign asap; others upon request. dave, is there a way to delete an old tool? (hmmscan) On Wed, Feb 29, 2012 at 10:12 AM, Edward Kirton

Re: [galaxy-dev] HMMER wrappers

2012-03-06 Thread Edward Kirton
:47 AM, Edward Kirton wrote: i created a new toolshed repo, hmmer since i couldn't rename it. as suggested, it has the hmmscan/hmmsearch as one tool, plus hmmpress. will add hmmbuild, hmmalign asap; others upon request. dave, is there a way to delete an old tool? (hmmscan) On Wed, Feb 29

Re: [galaxy-dev] HMMER wrappers

2012-03-07 Thread Edward Kirton
good tip On Tue, Mar 6, 2012 at 3:28 PM, Peter Cock p.j.a.c...@googlemail.comwrote: marked as hidden in the XML ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other

Re: [galaxy-dev] Merging BLAST database support into Galaxy?

2012-04-18 Thread Edward Kirton
sounds great, thanks peter. i granted you access to my toolshed repo, but perhaps we want only one tool in the toolshed when all done. On Wed, Apr 18, 2012 at 3:20 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Wed, Apr 18, 2012 at 10:53 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi

Re: [galaxy-dev] Merging BLAST database support into Galaxy?

2012-04-26 Thread Edward Kirton
your suggestion for blastdbn and blastdbp sounds fine. it's okay if a few of our users need to edit the metadata of the dbs in their history. thanks for asking and doing this. On Thu, Apr 26, 2012 at 5:37 AM, Peter Cock p.j.a.c...@googlemail.comwrote: Hi Edward, I've started work on this in

Re: [galaxy-dev] Task Manager: This Galaxy instance is not the job manager.

2012-06-06 Thread Edward Kirton
yes, i've had the same error ever since the last galaxy-dist release. i previously had multiple servers and switched to the one manager, two handlers. rewrite rules didn't need to be changed. On Thu, May 24, 2012 at 8:14 AM, Sarah Diehl di...@immunbio.mpg.de wrote: ** Hi all, I have a

Re: [galaxy-dev] scripts in welcome.html?

2012-06-27 Thread Edward Kirton
yes, javascript and ssi work and i use them for other purposes (grid load meter) but i've found it convenient to have two welcome.html files -- e.g. welcome.html.dev, welcome.html.main -- a use a symlink to point to the correct one. On Wed, Jun 13, 2012 at 3:37 PM, Smithies, Russell

Re: [galaxy-dev] HMMER wrappers

2012-06-27 Thread Edward Kirton
sorry for the late reply, i've been away for almost a month. thanks for your work on this, i'll update the tools and upload to toolshed asap. On Tue, May 29, 2012 at 3:35 AM, Peter Cock p.j.a.c...@googlemail.comwrote: On Fri, May 25, 2012 at 4:54 PM, Peter Cock p.j.a.c...@googlemail.com

Re: [galaxy-dev] Port Config SMTP Server

2012-06-28 Thread Edward Kirton
in the universe_wsgi.ini file, define the server:port like thus: smtp_server = smtp.gmail.com:587 On Thu, Jun 21, 2012 at 3:23 PM, CHEBBI Mohamed Amine am...@abgenomica.comwrote: Hi Galxy-team ! I have to use the smtp-server in Galaxy. However I don't see how to set the port to 587 in the

Re: [galaxy-dev] output file

2012-07-19 Thread Edward Kirton
change your tool to use explicitly named output folder and files bash mytool.sh $input $output1.extra_files_path $output1 $output2 where $output1.extra_files_path will be a folder (e.g. working dir) $output1, $output2, etc. are files that are to be brought into your history in galaxy, to be

Re: [galaxy-dev] Modules

2012-08-29 Thread Edward Kirton
galaxy already has tool-dependencies which can be used. for example, velvet's tool XML would include: requirements requirement type=packagevelvet/requirement /requirements the above will use the default version; or to specify a specific version: requirements requirement type=package

Re: [galaxy-dev] disk space warning

2012-09-10 Thread Edward Kirton
i use a cron job to monitor my disks and email me if they approach capacity. i'm not sure of the benefit of informing your users; better for a sysop to take care of it before it becomes a problem. i also use cron jobs to purge older files from the ftp and tmp dirs. On Thu, Sep 6, 2012 at 6:48