Re: [galaxy-dev] Galaxy and ENCODE
It's an excellent idea . Does anybody have a document around the work package ? I'm sure that many people will be happy to participate. Victoria On Mon, Nov 19, 2012 at 8:03 PM, Scooter Willis hwil...@scripps.edu wrote: Thank you for the clarification. Would be an interesting project for a Galaxy hackathon to move over the ENCODE tools/workflow to run in Galaxy. On 11/19/12 11:21 AM, James Taylor ja...@jamestaylor.org wrote: Scooter, The ENCODE VM which is available on Amazon Web Services and as a downloadable VM[1] uses Galaxy's Cloudman component for managing storage and compute resources. However, the Galaxy analysis interface was not used. All of the code for analysis performed in the ENCODE integrated paper is included in the VM, however most of the components are not exposed as Galaxy tools or workflows. It would definitely be possible to do so if you are interested, however any given analysis may not have been built with reuse on different datasets in mind. Thanks! [1] http://encodeproject.org/ENCODE/integrativeAnalysis/VM/ -- James Taylor, Assistant Professor, Biology/CS, Emory University On Sun, Nov 18, 2012 at 10:29 AM, Scooter Willis hwil...@scripps.edu wrote: I found out about Galaxy based on the ENCODE project making a big deal about packaging the software they used on a CD and available for download. I misplaced the CD and following some link on an ENCODE/Nature web site found Galaxy. What I haven't found is any public workflows that would indicate this is what ENCODE was doing to call SNPs or assemble RNAseq data. It appears that number of public workflows are minimal. Maybe I am not pointing to the correct repository. Can someone provide some guidance on ENCODE use of Galaxy and where I can find ENCODE workflows? I seem to remember the ENCODE Nature publication having a CD but never looked at it. If this was the case did ENCODE package a vmware'ish version of Galaxy on the CD? Thanks Scooter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Unable to run jobs in Galaxy: runners/local.py # stdout_file.close() # python2.7/tempfile.py # self.unlink(self.name) # OSError: [Errno 2] No such file or directory
Hi all Since updating to the latest Galaxy-dist version, I'm unable to run any jobs. galaxy.jobs.handler INFO 2012-11-20 12:22:09,018 (6403) Job dispatched galaxy.objectstore CRITICAL 2012-11-20 12:22:15,362 /home/galaxy/galaxy-dist/database/job_working_directory/006/6403 delete error [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/job_working_directory/006/6403' CRITICAL12:22:15 __init__.delete:322 /home/galaxy/galaxy-dist/database/job_working_directory/006/6403 delete error [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/job_working_directory/006/6403' galaxy.jobs.runners.local ERROR 2012-11-20 12:22:15,363 failure running job 6403 Traceback (most recent call last): File /home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py, line 129, in run_job stdout_file.close() File /usr/lib/python2.7/tempfile.py, line 410, in close self.unlink(self.name) OSError: [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/job_working_directory/006/6403/tmpKuTwAJ_stdout' ERROR 12:22:15 local.run_job:134 failure running job 6403 Traceback (most recent call last): File /home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py, line 129, in run_job stdout_file.close() File /usr/lib/python2.7/tempfile.py, line 410, in close self.unlink(self.name) OSError: [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/job_working_directory/006/6403/tmpKuTwAJ_stdout' Exception OSError: (2, 'No such file or directory', '/home/galaxy/galaxy-dist/database/job_working_directory/006/6403/tmpK7i5n4_stderr') in bound method _TemporaryFileWrapper.__del__ of closed file 'fdopen', mode 'w+b' at 0x55b8f60 ignored galaxy.objectstore CRITICAL 2012-11-20 12:22:15,723 /home/galaxy/galaxy-dist/database/job_working_directory/006/6404 delete error [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/job_working_directory/006/6404' CRITICAL12:22:15 __init__.delete:322 /home/galaxy/galaxy-dist/database/job_working_directory/006/6404 delete error [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/job_working_directory/006/6404' I'm running the version from ~Nov 6th: 8153:5dcbbdfe1087 tip | Allow rerun to access hidden datasets. I already found this recent issue on the maillinglist which appeared similar, but my problems involve running jobs, not uploading datasets: http://dev.list.galaxyproject.org/Re-Galaxy-at-Duke-td4656946.html My setup involves 1 runner instance and 2 web instances, all inside the same VM, as per my configuration files here: https://trac.nbic.nl/brs2010p33/browser/trunk/galaxy-dist/ The folder /work/project is an sshfs mount to the server running the VM. sshfs#backup@192.168.122.1:/work/odose/ /work/project/ fuse defaults,idmap=user,allow_other,cache=no 0 0 I've setup the following paths to that mounted folder: # Dataset files are stored in this directory. file_path = /work/project/galaxy-dist/database/files # Temporary files are stored in this directory. new_file_path = /work/project/galaxy-dist/database/tmp The job_working_directory is a normal folder in the VM, but I still get the above error. Can anyone tell me what I should do to be able to run jobs in Galaxy again? Best, Tim ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] format sensitive tools not working properly after upgrade
I performed an upgrade over the weekend. Since then, I've observed an odd behavior - bowtie2 will not list the new fastqsanger files in my history. It will work with files that had already been groomed before the upgrade. I also see that the groomer is not recognizing fastq files. The format does seem to be correctly set for these datasets. Is there more than one place that a dataset's format is stored (and therefore get out of sync)? Could I have messed up the data types xml file? It looks ok. Seems unlikely that this would be a global problem since nobody else has reported it… but it does seem a bit subtle (only affecting newly tweaked files) Any ideas? Brad -- Brad Langhorst langho...@neb.commailto:langho...@neb.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] First questions from a new developer
Thanks James - I had figured out that I needed to restart the server after changing config files, and the debug info you sent was helpful. Thanks for taking the time to respond. Joshua On Mon, Nov 19, 2012 at 10:38 AM, James Taylor ja...@jamestaylor.orgwrote: On Fri, Nov 16, 2012 at 2:35 PM, Joshua Orvis jor...@gmail.com wrote: If I removed that line how is it still part of the call? Is Galaxy caching the xml tool files? Did you restart Galaxy? Tool configuration files are read at startup. You can also force a tool to be reloaded in the admin interface. Also, is there a way to see the precise commands being executed by Galaxy for any particular tool? These are logged at the debug level, either to the console or paster.log depending on how you start Galaxy. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] DRMAA job will now be errored - Segmentation fault
On Thu, Nov 15, 2012 at 11:21 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Nov 15, 2012 at 10:12 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Nov 15, 2012 at 10:06 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, Something has changed in the job handling, and in a bad way. On my development machine submitting jobs to the cluster didn't seem to be working anymore (never sent to SGE). I killed Galaxy and restarted: ... (segmentation fault) Looking into the problem with submitting the jobs, there seems to be a problem with task splitting somehow recursing - the same file is split four times, the filename getting longer and longer: Turning off task splitting I could run the same job OK on SGE. So, the good news is the problems seem to be specific to the task splitting code. Also I have reproduced the segmentation fault when restarting Galaxy (after stopping Galaxy with one of these broken jobs). Starting server in PID 17996. serving on http://127.0.0.1:8081 galaxy.jobs.runners.drmaa ERROR 2012-11-15 11:07:27,762 (327/None) Unable to check job status Traceback (most recent call last): File /mnt/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py, line 296, in check_watched_items state = self.ds.jobStatus( job_id ) File /mnt/galaxy/galaxy-central/eggs/drmaa-0.4b3-py2.6.egg/drmaa/__init__.py, line 522, in jobStatus _h.c(_w.drmaa_job_ps, jobName, _ct.byref(status)) File /mnt/galaxy/galaxy-central/eggs/drmaa-0.4b3-py2.6.egg/drmaa/helpers.py, line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer File /mnt/galaxy/galaxy-central/eggs/drmaa-0.4b3-py2.6.egg/drmaa/errors.py, line 90, in error_check raise _ERRORS[code-1](code %s: %s % (code, error_buffer.value)) InvalidArgumentException: code 4: Job id, None, is not a valid job id galaxy.jobs.runners.drmaa WARNING 2012-11-15 11:07:27,764 (327/None) job will now be errored ./run.sh: line 86: 17996 Segmentation fault (core dumped) python ./scripts/paster.py serve universe_wsgi.ini $@ The problem is the job_id variable is None (note this is a string, not the Python special object None) in check_watched_items(). Peter Is anyone else seeing this? I am wary of applying the update to our production Galaxy until I know how to resolve this (other than just be disabling task splitting). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Where is the eye icon for datasets in error state?
Hey devs! It seems that many things related to history datasets has changed since I have updated my local Galaxy instance and the current version. Unfortunately it seems that one cannot Display data in browser (eye icon) if the dataset is in error state. Where can I get that feature back? Some of my tools write logging information to a separate logfile which will be in error state if the tool fails but contain useful information. Thanks, Sascha ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Where is the eye icon for datasets in error state?
Let me echo Sascha. I've been thinking about this problem. I suggest that we need an Error Document dataset type that contains two kinds information: 1. which will help the end user correct the problem (E.g. Please be sure to use MAF files not BAM files etc). 2. which will help tool developers collect a bunch of log files.. Error Documents could be aggregated (especially useful in a cluster computation). Then the report a bug button would aggregate the log file document and send it to the tool developer. Before sending, the end user would be able to see the contents and be sure that they were not posting confidential information offsite. Thanks Ted On Nov 20, 2012, at 8:02 AM, MSc. Sascha Kastens wrote: Hey devs! It seems that many things related to history datasets has changed since I have updated my local Galaxy instance and the current version. Unfortunately it seems that one cannot Display data in browser (eye icon) if the dataset is in error state. Where can I get that feature back? Some of my tools write logging information to a separate logfile which will be in error state if the tool fails but contain useful information. Thanks, Sascha ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Direct access the fastq files stored in galaxy server local storage
We have a local galaxy server. I know I can get data by ftp, upload directly from browser, or by a URL. If I have some fastq files in the Galaxy server local storage, is it possible to let Galaxy use it directly? Thanks, Fenglou ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Move files between histories?
Hello Is it possible to move a file from one history to another? I have generated BAM files in separate histories that now must be merged. How can I do this without downloading and then uploading the files? Thanks, Gregory Thyssen, PhD Molecular Biologist Cotton Fiber Bioscience USDA-ARS-Southern Regional Research Center 1100 Robert E Lee Blvd New Orleans, LA 70124 gregory.thys...@ars.usda.gov 504-286-4280 This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] NGS assembly?
Hello I would like to assemble my Illimuna reads along with some Sanger reads. Is Velvet or ABySS hidden somewhere in the menu? Please advise. Thanks, Gregory Thyssen, PhD Molecular Biologist Cotton Fiber Bioscience USDA-ARS-Southern Regional Research Center 1100 Robert E Lee Blvd New Orleans, LA 70124 gregory.thys...@ars.usda.gov 504-286-4280 This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Direct access the fastq files stored in galaxy server local storage
I don't think this is possible. Galaxy is designed to import data files and track their relationships and disk usage internally. You can import lots of files at once though.. see: http://wiki.galaxyproject.org/Admin/Data%20Libraries/Uploading%20Library%20Files Brad On Nov 20, 2012, at 3:49 PM, Fenglou Mao feng...@gmail.commailto:feng...@gmail.com wrote: We have a local galaxy server. I know I can get data by ftp, upload directly from browser, or by a URL. If I have some fastq files in the Galaxy server local storage, is it possible to let Galaxy use it directly? Thanks, Fenglou ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langho...@neb.commailto:langho...@neb.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Move files between histories?
Yes. If you click on the little sprocket icon in your history panel and go to Copy Datasets, you'll be able to do this. -Dannon On Nov 20, 2012, at 2:52 PM, Thyssen, Gregory - ARS gregory.thys...@ars.usda.gov wrote: Hello Is it possible to move a file from one history to another? I have generated BAM files in separate histories that now must be merged. How can I do this without downloading and then uploading the files? Thanks, Gregory Thyssen, PhD Molecular Biologist Cotton Fiber Bioscience USDA-ARS-Southern Regional Research Center 1100 Robert E Lee Blvd New Orleans, LA 70124 gregory.thys...@ars.usda.gov 504-286-4280 This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] NGS assembly?
Hello Gregory, The Tool Shed has some assembly tools that you would be able to use with a local or cloud instance. Please see: http://toolshed.g2.bx.psu.edu/ Under Category name: Assembly Help with the tool shed and local/cloud Galaxy instances is in our wiki: http://wiki.galaxyproject.org/Tool%20Shed http://getgalaxy.org http://usegalaxy.org/cloud Hopefully this helps to get you started, this is a good mailing list to use for follow-up questions for both the tool shed and local/cloud installs. Start a new thread when you have new questions for the best visibility. Take care, Jen Galaxy team On 11/20/12 11:53 AM, Thyssen, Gregory - ARS wrote: Hello I would like to assemble my Illimuna reads along with some Sanger reads. Is Velvet or ABySS hidden somewhere in the menu? Please advise. Thanks, Gregory Thyssen, PhD Molecular Biologist Cotton Fiber Bioscience USDA-ARS-Southern Regional Research Center 1100 Robert E Lee Blvd New Orleans, LA 70124 gregory.thys...@ars.usda.gov 504-286-4280 This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Direct access the fastq files stored in galaxy server local storage
This is actually possible using data libraries. What you'd want to do is upload by filepath (described in the wiki page Brad linked, heading Upload files from filesystem paths) and check the No box under Copy data into Galaxy. -Dannon On Nov 20, 2012, at 4:13 PM, Langhorst, Brad langho...@neb.com wrote: I don't think this is possible. Galaxy is designed to import data files and track their relationships and disk usage internally. You can import lots of files at once though.. see: http://wiki.galaxyproject.org/Admin/Data%20Libraries/Uploading%20Library%20Files Brad On Nov 20, 2012, at 3:49 PM, Fenglou Mao feng...@gmail.com wrote: We have a local galaxy server. I know I can get data by ftp, upload directly from browser, or by a URL. If I have some fastq files in the Galaxy server local storage, is it possible to let Galaxy use it directly? Thanks, Fenglou ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langho...@neb.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Multiple language in one tool
Hi Philippe: You can just wrap a bash script if you want to to that… or even write a simple pipeline in the xml wrapper with interpreter='bash' or no interpreter at all. I've done that sort of thing before with good results. brad On Nov 20, 2012, at 10:03 PM, Philipe Moncuquet philippe.m...@gmail.commailto:philippe.m...@gmail.com wrote: Hi dev, I was wondering if the following is possible or not. What if I want to build a tool that that uses Perl and R at the same time ? If I had to do it on my local machine I would create a pipeline but wouldn't it be cool if a wrapper could use different languages. Philippe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langho...@neb.commailto:langho...@neb.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Multiple language in one tool
Hi Brad, Thanks a lot for that tip, will try that. Cheers, Philippe 2012/11/21 Langhorst, Brad langho...@neb.com Hi Philippe: You can just wrap a bash script if you want to to that… or even write a simple pipeline in the xml wrapper with interpreter='bash' or no interpreter at all. I've done that sort of thing before with good results. brad On Nov 20, 2012, at 10:03 PM, Philipe Moncuquet philippe.m...@gmail.com wrote: Hi dev, I was wondering if the following is possible or not. What if I want to build a tool that that uses Perl and R at the same time ? If I had to do it on my local machine I would create a pipeline but wouldn't it be cool if a wrapper could use different languages. Philippe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langho...@neb.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
