[galaxy-dev] Metadata in Galaxy Tool Shed
I recently created a new repository in the galaxy tool shed for the cutadapt tool and imported my old hg repository into it. However, it doesn't seem to be picking up the metadata correctly. Does this take time, or is there something I need to do to get the metadata generated correctly? Also, any suggestions on how to organize the files in the repository are welcome. Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University attachment: lparsons.vcf___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Dataset Cleanup Question
I am running a local instance of Galaxy and I've been trying to sort out some issue with dataset cleanup. For the most part, things are working OK running the shell scripts in the recommended order: delete_userless_histories.sh purge_histories.sh purge_libraries.sh purge_folders.sh delete_datasets.sh purge_datasets.sh I have the number of days set to 10. When I look at the reports webapp however, it reports that there are 62 datasets were deleted more than 15 days ago, but have not yet been purged, disk space: 12975717335. These have stuck around now for 45 days (and counting). I have even tried running the scripts with the -f option to force galaxy to re-evaluate the datasets to no avail. Any suggestions? Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University attachment: lparsons.vcf___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Bug importing from data library to history
I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] MergeSamFiles.jar and TMPDIR
For what it's worth, I ran into an issue with the use of /tmp as well. When merging a lot of BAM files, /tmp filled up and the merge failed. To make matters worse, since STDERR is redirected and the exit status of java is not checked, the item in the Galaxy history appeared OK. Though examination of the .log file revealed the issue, users weren't aware of the problem right away, causing further confusion. To get around this, I implemented two fixes: 1 - Wrapper for sam_merge2 very similar to hide_stderr.py used by the ncbi_blast_plus tools. I did have to hack it a bit, however, since when I output the captured stderr, galaxy would hang if it was too large. My fix was to output the first and last 7500 characters of stderr (truncating the middle). Now when the merge fails, users see a failure in Galaxy (red history item). 2 - Looking at other Picard tools, it seems that at least some of them explicitly set TMP_DIR to $__new_files_directory__, which in this case seemed appropriate, at least for my use case. So I added that simple change to the sam_merge.xml command line. I'm open to better solutions if people see issues with these, but I would suggest that Galaxy be updated to include fixes for these issues since it seems others are likely to run into these problems. If people generally agree I'd be happy to provide create tickets and/or provide changesets or whatever the Galaxy maintainers would prefer. Lance Glen Beane wrote: On Sep 27, 2011, at 1:14 PM, Nate Coraor wrote: Glen Beane wrote: We recently updated to the latest galaxy-dist, and learned that the sam_merge.xml tool now uses picard MergeSamFiles.jar to merge the files instead of the samtools merge wrapper sam_merge.py. this is a problem for us because MergeSamFiles.jar does not honor $TMPDIR when creating temporary file names (the jvm developers inexplicably hard code the value of java.io.tmpdir to /tmp in Unix/Linux rather than doing the Right Thing) . On our cluster, TMPDIR is set to something like /scratch/batch_job_id/. This location has plenty of free space, however /tmp does not and now we can't successfully merge largeish bam files. In case anyone else is bit by this, I think there are two options the Picard tools take an optional TMP_DIR= argument that lets us specify the location we want to use for a temporary directory. Initially we ended up modifying the .xml to add TMP_DIR=\$TMPDIR to the arguments to MergeSamFiles.jar. This works, but we could potentially need to do this with multiple Picard tools and not just MergeSamFiles. Now I am probably going to go with the following solution: add something like export _JAVA_OPTIONS=-Djava.io.tmpdir=$TMPDIR to the .bashrc file for my Galaxy user. This is what I've just done on our local cluster as well. I was also confounded by the lack of a proper environment variable to do this. It looks like the export _JAVA_OPTIONS=-Djava.io.tmpdir=$TMPDIR solution breaks some tools (like snpEFF, which is a 3rd party tool we use). The Jvm prints a diagnostic message to stderr that looks something like this: Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/scratch/32095.scyld.localdomain so in this case the tool fails, since it does not have a wrapper. The sam_merge.xml tool redirects stderr, so it doesn't have this problem. I am thinking about putting a wrapper script for java in my galaxy user's path that adds -Djava.io.tmpdir=$TMPDIR to the arguments. -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] MergeSamFiles.jar and TMPDIR
The patches have been put into a pull request. Lance Nate Coraor wrote: Lance Parsons wrote: For what it's worth, I ran into an issue with the use of /tmp as well. When merging a lot of BAM files, /tmp filled up and the merge failed. To make matters worse, since STDERR is redirected and the exit status of java is not checked, the item in the Galaxy history appeared OK. Though examination of the .log file revealed the issue, users weren't aware of the problem right away, causing further confusion. To get around this, I implemented two fixes: 1 - Wrapper for sam_merge2 very similar to hide_stderr.py used by the ncbi_blast_plus tools. I did have to hack it a bit, however, since when I output the captured stderr, galaxy would hang if it was too large. My fix was to output the first and last 7500 characters of stderr (truncating the middle). Now when the merge fails, users see a failure in Galaxy (red history item). 2 - Looking at other Picard tools, it seems that at least some of them explicitly set TMP_DIR to $__new_files_directory__, which in this case seemed appropriate, at least for my use case. So I added that simple change to the sam_merge.xml command line. I'm open to better solutions if people see issues with these, but I would suggest that Galaxy be updated to include fixes for these issues since it seems others are likely to run into these problems. If people generally agree I'd be happy to provide create tickets and/or provide changesets or whatever the Galaxy maintainers would prefer. Hi Lance, Could you put these patches in an issue (or create a pull request)? Thanks, --nate -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug importing from data library to history
Good catch, that does indeed seem to be the issue. And it only seems to happen with my older data libraries. I didn't see a bug report for this, but I'd be happy to file one and/or take a stab at a fix if someone can point me in the right direction in the code. Lance Nate Coraor wrote: Lance Parsons wrote: I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. Hey Lance, No real ideas, but Alex Bossers just posted up saying that old library datasets are importing into his histories as deleted. Could you check and see if this is the case in your instance as well? Thanks, --nate -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug importing from data library to history
Ah, sorry, I just saw your note to Alex regarding the pieces of code to check out. I'll start doing some digging and let you know if I find anything out. See line 1925 in lib/galaxy/web/controllers/library_common.py. Try adding a bit of debugging to find out if hda.deleted == True when the copied HistoryDatasetAssociation is created, and whether the source LibraryDatasetDatasetAssociation is itself deleted (which should be the only way the resulting hda could be deleted). Lance Lance Parsons wrote: Good catch, that does indeed seem to be the issue. And it only seems to happen with my older data libraries. I didn't see a bug report for this, but I'd be happy to file one and/or take a stab at a fix if someone can point me in the right direction in the code. Lance Nate Coraor wrote: Lance Parsons wrote: I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. Hey Lance, No real ideas, but Alex Bossers just posted up saying that old library datasets are importing into his histories as deleted. Could you check and see if this is the case in your instance as well? Thanks, --nate -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug importing from data library to history
I believe I may have found the bug... Line 258 of cleanup_datasets.py is the query for library_dataset_ids when force_retry is set. While it correctly eliminates the the app.model.LibraryDataset.table.c.purged == False from the where clause, it incorrectly eliminates the app.model.LibraryDataset.table.c.deleted == True. My suggested fix would be to correct that query by adding app.model.LibraryDataset.table.c.deleted == True to the where clause. Cleaning up users databases might be a bit trickier, but I think you could find the affected library datasets by the following query: select * from library_dataset where deleted=f and purged=t; With those, I would think setting purged to f and then setting deleted = f for the associated library_dataset_dataset_association records. Lance Nate Coraor wrote: Lance Parsons wrote: Good catch, that does indeed seem to be the issue. And it only seems to happen with my older data libraries. I didn't see a bug report for this, but I'd be happy to file one and/or take a stab at a fix if someone can point me in the right direction in the code. Any community offers to debug/fix are always happily welcomed. =) My reply to Alex: See line 1925 in lib/galaxy/web/controllers/library_common.py. Try adding a bit of debugging to find out if hda.deleted == True when the copied HistoryDatasetAssociation is created, and whether the source LibraryDatasetDatasetAssociation is itself deleted (which should be the only way the resulting hda could be deleted). I'll add that a key to figuring this out is probably is determining what's different about these older library items and the newer ones that don't exhibit the problem (if this is also true in your environment as it is in his). If no answers turn up in a few days, then we can open an issue for it in Bitbucket. --nate Lance Nate Coraor wrote: Lance Parsons wrote: I'm running a local instance of Galaxy that has been working quite well. However, I've recently run into a problem when importing datasets from a data library into a history. The Size on Disk of the history grows by the size of the imported dataset, however, the new dataset is not visible (and the number of datasets does not change in the saved histories view). Any thoughts on what might be going on here? Thanks. Hey Lance, No real ideas, but Alex Bossers just posted up saying that old library datasets are importing into his histories as deleted. Could you check and see if this is the case in your instance as well? Thanks, --nate -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Display VCF in IGV problem
I am unable to get VCF files to display in IGV in my local galaxy installation. BAM files display fine, however, whenever I click on the link to display a VCF file I get the following error message: You must import this dataset into your current history before you can view it at the desired display application. When I return to Galaxy, my history is indeed changed to a new empty history. I think this may have to do with the fact that I had to setup Apache to allow unauthenticated access to the bam/vcf files (see my notes on how I did this at https://sites.google.com/site/princetonhtseq/tutorials/visualization-with-galaxy-and-igv/igv-as-a-display-application-in-galaxy). However, I can't figure out why BAM files work and VCF files don't. Any advice or insight would be appreciated. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Tool Requirement tag and dependency management
I recently ran into a problem where the tool-dependency env.sh file was not getting sourced for a particular tool. I realized that the problem was in how the tool 'requirement' tag was specified. The requirement tag was specified as in the Tool Config Syntax documentation (http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax#A.3Crequirement.3E_tag_set): requirements requirement type=binarytaxBuilder/requirement /requirements However, it appears that the type must be set to package in order for the dependency system to source the env.sh file (http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies): requirements requirement type=packagetaxBuilder/requirement /requirements I'm working on getting this working with bedtools, and I'm wondering what the recommended way to do this would be. Since there are various binaries for this package, I'm thinking I could do something like: requirements requirement type=packagebedtools/requirement requirement type=binarygenomeCoverageBed/requirement /requirements Secondly, once this is sorted out, it would be nice to update the Tool Config Syntax documentation. Thanks in advance for any advice. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] BAM to BigWig (and tool ID clashes)
Hi Peter, Thanks for the thoughtful comments. I believe the requirement for the genome was imposed by the use of an underlying BedTools utility. I also think that in a newer version of that tool, the requirement was removed, since you correctly point out it is not really necessary. I will see if I can update the tool to remove that requirement and also see about changing the tool id. Sorry for the conflict, that was an oversight on my part, though it would be nice if the Tool Shed could check and warn when someone tries to create a new tool. I would suggest flagging the new repo as invalid until the id is updated instead of outright rejection. As for the author info, you're right, I should really add that as well. That tool was put together very quickly to meet the need of a customer and I didn't properly clean things up before I uploaded. I'll let you know once I get an update out. Of course, any patches etc. are welcome. ;-) Lance Peter Cock wrote: Hi Brad Lance, I've been using Brad's bam_to_bigwig tool in Galaxy but realized today (with a new dataset using a splice-aware mapper) that it doesn't seem to be ignoring CIGAR N operators where a read is split over an intron. Looking over Brad's Python script which calculates the coverage to write an intermediate wiggle file, this is done with the samtools via pysam. It is not obvious to me if this can be easily modified to ignore introns. Is this possible Brad? I wasn't aware of Lance's rival bam_to_bigwig tool in the ToolShed till now, and that does talk about this issue. It has a boolean option to ignore gaps when computing coverage, recommended for RNA-Seq where reads are mapped across long splice junctions. Lance, from your tool's help it sounds like it needs a genome database build filled in. I don't understand this requirement - Brad's tool works just fine for standalone BAM files (for example reads mapped to an in house assembly). Is that not supported in your tool? Galaxy team - why does the ToolShed allow duplicate repository names (here bam_to_bigwig) AND duplicate tool IDs (again, here bam_to_bigwig)? Won't this cause chaos when sharing workflows? I would suggest checking this when a tool is uploaded and rejecting repository name or tool ID clashes. Regards, Peter P.S. Brad, your tool is missing an explicitrequirements tag listing the UCSC binary wigToBigWig, and the Python library pysam. Lance, your tool doesn't seem to include any author information like your name or email address. I'm inferring it is yours from the Galaxy tool shed user id, lparsons. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] BAM to BigWig (and tool ID clashes)
I'm happy to have a merged version and put us all down as contributing authors. I won't have a chance to look at the code until next week, but I'd be happy to help out with any merging, etc. Thanks to both of you for you help an input. Lance Peter Cock wrote: On Fri, Apr 20, 2012 at 2:17 AM, Brad Chapmanchapm...@50mail.com wrote: Lance and Peter; Peter, thanks for noticing the problem and duplicate tools. Lance, I'm happy to merge these so there are not two different versions out there. I prefer your use for genomeCoverageBed over my custom hacks. That's a nice approach I totally missed. I avoid the need for the sam indexes by creating the file directly from the information in the BAM header. I don't think there is any way around creating it since it's required by the UCSC tools as well, but everything you need is in the BAM header. Indeed - I remember looking at that with you back in March 2011, including the special case of BAM files lacking an embedded SAM header (where the BAM header alone suffices). There might be a sneaky way to do this with samtools -H and awk but I'm not nearly skilled enough to pull that out. Using pysam works nicely, and therefore I stuck with Python ;) Let me know what you think. I can also update my python wrapper script to use the genomeCoverageBed approach instead if you think that's easier. I've made the update to Brad's script from the Tool Shed (attached), switching to using genomeCoverageBed and bedGraphToBigWig (based on the approach used in Lance's script), although in doing so I dropped the region support (which wasn't exposed to the Galaxy interface anyway). Since genomeCoverageBed doesn't support this directly, we could use samtools view for this I think - if you want this functionality. Sadly then I noticed that the Tool Shed version was out of date - lacking the new normalization option added here: https://github.com/chapmanb/bcbb/commits/master/nextgen/scripts/bam_to_wiggle.py This was enough for my immediate needs today, but I'd happily try and merge this into the git version and update the XML file to match and add the new split option. We could list this as three contributing authors if you both like? Peter -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] FreeBayes bug - update to a toolshed tool wrapper
I have been using the FreeBayes tool in our local Galaxy instance for some time. As FreeBayes is a rapidly evolving tool, it has been very helpful to have Daniel Blankenberg work on keeping an updated wrapper. Recently, I've run into a rather troublesome bug in FreeBayes (https://github.com/ekg/freebayes/issues/22) that is causing it to crash on nearly every real dataset. Luckily, the bug has been fixed in the latest version of FreeBayes (0.9.6_9608597d12e127c847ae03aa03440ab63992fedf). Unfortunately, the latest version has a few parameter changes that require an update to the wrapper code. I've taken the liberty to make some changes to the wrapper and they seem to work. However, now that I have the tool installed from the Tool Shed, I'm not sure of the best way to 1) incorporate this update into my local galaxy instance and 2) contribute this change back to the Tool Shed. The best solution is to get the Tool Shed updated and then pull that update into my production Galaxy instance, however, I'm wondering if there is a way I could push hot-fixes to my production instance without causing me headaches down the line when the Tool Shed is inevitably updated. As far as coordinating updates of Tool Shed tools, I suppose the process might be different for different tool authors. I was wondering, however, if anyone had thoughts on how this might work best. I've created a clone of the galaxy tool shed repository in Bitbucket (https://bitbucket.org/lance_parsons/freebayes/) and given Daniel and Greg access to it in hopes that would make things easier.Let me know if there is a better way to coordinate, etc. Thanks again. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] FreeBayes bug - update to a toolshed tool wrapper
Thanks for the quick turnaround Dan. Much appreciated. When I tried to test updating Freebayes from the toolshed, I get a message that no updates are available. Perhaps it takes some time before they are recognized? Or am I missing something? Lance Daniel Blankenberg wrote: Hi Lance, Thank you for the pull request. I think this is an excellent way to handle patches until the toolshed itself provides a better mechanism, particularly for the devteam owned repositories; other repository owners may prefer different methods of providing patches, but they can be contacted directly by using the contact form on the toolshed. Future enhancements to the toolshed will integrate better methods of providing this functionality, but that likely some ways off. Any suggestions and community contributions on this would be appreciated. FreeBayes has been updated in the Toolshed to 0.9.6_9608597d12e127c847ae03aa03440ab63992fedf, with mostly minor differences from your pull request. Please let us know if you experience any issues installing or using the updated version (0.0.3). Thanks, Dan On Aug 30, 2012, at 11:35 AM, Lance Parsons wrote: I have been using the FreeBayes tool in our local Galaxy instance for some time. As FreeBayes is a rapidly evolving tool, it has been very helpful to have Daniel Blankenberg work on keeping an updated wrapper. Recently, I've run into a rather troublesome bug in FreeBayes (https://github.com/ekg/freebayes/issues/22) that is causing it to crash on nearly every real dataset. Luckily, the bug has been fixed in the latest version of FreeBayes (0.9.6_9608597d12e127c847ae03aa03440ab63992fedf). Unfortunately, the latest version has a few parameter changes that require an update to the wrapper code. I've taken the liberty to make some changes to the wrapper and they seem to work. However, now that I have the tool installed from the Tool Shed, I'm not sure of the best way to 1) incorporate this update into my local galaxy instance and 2) contribute this change back to the Tool Shed. The best solution is to get the Tool Shed updated and then pull that update into my production Galaxy instance, however, I'm wondering if there is a way I could push hot-fixes to my production instance without causing me headaches down the line when the Tool Shed is inevitably updated. As far as coordinating updates of Tool Shed tools, I suppose the process might be different for different tool authors. I was wondering, however, if anyone had thoughts on how this might work best. I've created a clone of the galaxy tool shed repository in Bitbucket (https://bitbucket.org/lance_parsons/freebayes/) and given Daniel and Greg access to it in hopes that would make things easier.Let me know if there is a better way to coordinate, etc. Thanks again. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] FreeBayes bug - update to a toolshed tool wrapper
Thanks for the explanation. That makes sense. I went ahead and
installed the new version along with dependencies in my development
environments and all went well. Then in production, the installation
seems to have failed and I'm now stuck with freebayes in the cloning
status. I tried to deactivate and uninstall it so I could try again to
monitor what went wrong and I just get an error:
Error - type 'exceptions.AttributeError': 'NoneType' object has no
attribute 'startswith'
URL:
http://galaxy.princeton.edu/admin_toolshed/browse_repositories?operation=deactivate+or+uninstallid=ac6e1b48798577db
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py',
line 143 in __call__
app_iter = self.application(environ, start_response)
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/recursive.py',
line 80 in __call__
return self.application(environ, start_response)
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/middleware/remoteuser.py',
line 91 in __call__
return self.app( environ, start_response )
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py',
line 632 in __call__
return self.application(environ, start_response)
File '/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py',
line 160 in __call__
body = method( trans, **kwargs )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/__init__.py',
line 184 in decorator
return func( self, trans, *args, **kwargs )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/controllers/admin_toolshed.py',
line 356 in browse_repositories
return self.deactivate_or_uninstall_repository( trans, **kwd )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/__init__.py',
line 184 in decorator
return func( self, trans, *args, **kwargs )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/controllers/admin_toolshed.py',
line 408 in deactivate_or_uninstall_repository
repository_install_dir = os.path.abspath ( relative_install_dir )
File '/usr/local/python/lib/python2.6/posixpath.py', line 342 in abspath
if not isabs(path):
File '/usr/local/python/lib/python2.6/posixpath.py', line 52 in isabs
return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'
Any thoughts on how I can get rid of the partially installed freebayes
tool? Or how to troubleshoot the failing installation?
Lance
Daniel Blankenberg wrote:
Hi Lance,
This has to do with the way that the updating currently works. Since
the migrated version is 0.0.2, the automatic update will only update
that particular Tool version (newest version of 0.0.2). To get 0.0.3,
you need to install it as a new tool. This way you are able to have
both versions installed and available e.g. for having a reproducible
rerun functionality. In the latest Galaxy-central / upcoming
Galaxy-dist release, only the newest version is shown in the lefthand
tool panel.
Future enhancements will include allowing the selection of different
tool versions from within the tool interface (e.g. allowing the rerun
of an older version to use a newer version) and to show and allow
updating (installing) new tool versions from the same interface (e.g.
2 to 3).
Thanks for using Galaxy,
Dan
On Aug 31, 2012, at 11:47 AM, Lance Parsons wrote:
Thanks for the quick turnaround Dan. Much appreciated. When I tried
to test updating Freebayes from the toolshed, I get a message that no
updates are available. Perhaps it takes some time before they are
recognized? Or am I missing something?
Lance
Daniel Blankenberg wrote:
Hi Lance,
Thank you for the pull request. I think this is an excellent way to
handle patches until the toolshed itself provides a better
mechanism, particularly for the devteam owned repositories; other
repository owners may prefer different methods of providing patches,
but they can be contacted directly by using the contact form on the
toolshed. Future enhancements to the toolshed will integrate better
methods of providing this functionality, but that likely some ways
off. Any suggestions and community contributions on this would be
appreciated.
FreeBayes has been updated in the Toolshed to
0.9.6_9608597d12e127c847ae03aa03440ab63992fedf, with mostly minor
differences from your pull request. Please let us know if you
experience any issues installing or using the updated version (0.0.3).
Thanks,
Dan
On Aug 30, 2012, at 11:35 AM, Lance Parsons wrote:
I have been using the FreeBayes tool in our local Galaxy instance
for some time. As FreeBayes is a rapidly evolving tool, it has
been very helpful to have Daniel Blankenberg work on keeping an
updated wrapper.
Recently, I've run into a rather troublesome bug in FreeBayes
(https://github.com/ekg/freebayes/issues/22) that is causing it to
crash on nearly every real dataset. Luckily, the bug has been
fixed in the latest version
[galaxy-dev] Automatic installation of third party dependancies
I've put together a tool wrapper for the htseq-count script that is part of the HTSeq python package and uploaded that to the tool shed. However, I have discovered that the tool dependancies do not install properly. There are a couple of issues that I've run into. 1) The biggest issue is the inability to set both the PYTHONPATH and PATH environment variables as part of installation. If I put two separate action type=set_environment tags, then the second overwrites the first in the env.sh file. If instead, I put two environment_variable tags within the action, only one (the second) gets executed during installation. 2) Another issue is the inability to install Numpy as a dependancy to HTSeq. I can get Numpy to install, but it won't be in the PYTHONPATH when HTSeq is installed (or executed). Any thoughts on how to address these issues would be appreciated. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] FreeBayes bug - update to a toolshed tool wrapper
Thanks for information Greg. Actually, it turned out to be two things
causing my issues:
1) My copy of tool-data/sam_fa_indices.loc.sample was not owned by the
galaxy user (though it was writable by that user). This caused the
exception which caused the installation to get stuck at cloning:
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/controllers/admin_toolshed.py',
line 709 in handle_repository_contents
copy_sample_files( trans.app, sample_files,
sample_files_copied=sample_files_copied )
File '/data/local/galaxy/galaxy-prod/lib/galaxy/util/shed_util.py', line
311 in copy_sample_files
copy_sample_file( app, filename, dest_path=dest_path )
File '/data/local/galaxy/galaxy-prod/lib/galaxy/util/shed_util.py', line
299 in copy_sample_file
shutil.copy( full_source_path, full_destination_path )
File '/usr/local/python/lib/python2.6/shutil.py', line 85 in copy
copymode(src, dst)
File '/usr/local/python/lib/python2.6/shutil.py', line 59 in copymode
os.chmod(dst, mode)
OSError: [Errno 1] Operation not permitted:
'/data/local/galaxy/galaxy-prod/tool-data/sam_fa_indices.loc.sample'
2) Once that happened, the directory to clone the repository into still
existed (and wasn't empty), which caused the error below, also causing
the process to get stuck in cloning status.
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py',
line 143 in __call__
app_iter = self.application(environ, start_response)
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/recursive.py',
line 80 in __call__
return self.application(environ, start_response)
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/middleware/remoteuser.py',
line 91 in __call__
return self.app( environ, start_response )
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py',
line 632 in __call__
return self.application(environ, start_response)
File '/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py',
line 160 in __call__
body = method( trans, **kwargs )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/__init__.py',
line 184 in decorator
return func( self, trans, *args, **kwargs )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/controllers/admin_toolshed.py',
line 846 in manage_repositories
self.install_tool_shed_repositories( trans,
repositories_for_installation, reinstalling=reinstalling, **decoded_kwd )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/framework/__init__.py',
line 184 in decorator
return func( self, trans, *args, **kwargs )
File
'/data/local/galaxy/galaxy-prod/lib/galaxy/web/controllers/admin_toolshed.py',
line 618 in install_tool_shed_repositories
clone_repository( repository_clone_url, os.path.abspath(
relative_install_dir ), ctx_rev )
File '/data/local/galaxy/galaxy-prod/lib/galaxy/util/shed_util.py', line
287 in clone_repository
rev=util.listify( str( ctx_rev ) ) )
File
'/data/local/galaxy/galaxy-prod/eggs/mercurial-2.1.2-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py',
line 1157 in clone
branch=opts.get('branch'))
File
'/data/local/galaxy/galaxy-prod/eggs/mercurial-2.1.2-py2.6-linux-x86_64-ucs2.egg/mercurial/hg.py',
line 270 in clone
raise util.Abort(_(destination '%s' is not empty) % dest)
Abort: destination
'/data/local/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/freebayes/d3bf1e86b243/freebayes'
is not empty
Once I manually cleaned those issues up things progressed normally.
Hope this helps to make the process more robust.
Lance
Greg Von Kuster wrote:
Hi Lance,
You'll need to execute the following sql manually in your Galaxy database.
update tool_shed_repository set status = 'New' where status = 'Cloning';
Next, you'll have to update your production Galaxy instance code base
to at least changeset 2531e085f262 (the human-readable changeset
revision is 7503). This revision is required to install the freebayes
tool from the main Galaxy tool shed. You can also choose to wait
until the next Galaxy release to upgrade if you do not need the
freebayes tool in your local instance immediately.
Sorry for the inconvenience on this.
Greg Von Kuster
On Aug 31, 2012, at 3:59 PM, Lance Parsons wrote:
Thanks for the explanation. That makes sense. I went ahead and
installed the new version along with dependencies in my development
environments and all went well. Then in production, the installation
seems to have failed and I'm now stuck with freebayes in the
cloning status. I tried to deactivate and uninstall it so I could
try again to monitor what went wrong and I just get an error:
Error - type 'exceptions.AttributeError': 'NoneType' object has no
attribute 'startswith'
URL:
http://galaxy.princeton.edu/admin_toolshed/browse_repositories?operation=deactivate+or+uninstallid=ac6e1b48798577db
File
'/data/local/galaxy/galaxy-prod/eggs/Paste-1.6-py2.6.egg/paste/exceptions
[galaxy-dev] FreeBayes Installation Bug
Today I ran into a new issues with installing a specific version of Freebayes. It turns out to be an issue with how git deals with submodules, something I'm only just learning about myself. In order to ensure that FreeBayes is built with the correct versions of all of the various submodules for the desired revision, it seems that two commands are needed: git checkout 9608597d12e127c847ae03aa03440ab63992fedf git submodule update --recursive In the current version of tool_dependencies.xml, there is a git reset --hard 9608597d12e127c847ae03aa03440ab63992fedf which seems to work similarly to the checkout command above, however, it does not put the submodules back at the correct revisions. That is only now an issue since they were updated only recently. I've attached a patch to the tool_dependencies.xml file that should once again allow FreeBayes to be built and installed automatically. It would be great if you could update the Tool Shed repo. Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University diff --git a/tool_dependencies.xml b/tool_dependencies.xml --- a/tool_dependencies.xml +++ b/tool_dependencies.xml @@ -4,7 +4,8 @@ install version=1.0 actions action type=shell_commandgit clone --recursive git://github.com/ekg/freebayes.git/action -action type=shell_commandgit reset --hard 9608597d12e127c847ae03aa03440ab63992fedf/action +action type=shell_commandgit checkout 9608597d12e127c847ae03aa03440ab63992fedf/action +action type=shell_commandgit submodule update --recursive/action action type=shell_commandmake/action action type=move_directory_files source_directorybin/source_directory ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Automatic installation of third party dependancies
Thanks Greg. I used you're updated version and added numpy as a separate dependency. It seems to work in my development system. I haven't updated my qa or production versions yet, so I can't check there. Perhaps you could test it and let me know if there are issues, etc. Glad I can be of help. Lance Greg Von Kuster wrote: Hello Lance, See my inline comments. On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote: I've put together a tool wrapper for the htseq-count script that is part of the HTSeq python package and uploaded that to the tool shed. However, I have discovered that the tool dependancies do not install properly. There are a couple of issues that I've run into. 1) The biggest issue is the inability to set both the PYTHONPATH and PATH environment variables as part of installation. If I put two separateaction type=set_environment tags, then the second overwrites the first in the env.sh file. If instead, I put twoenvironment_variable tags within the action, only one (the second) gets executed during installation. The above issues have been corrected in changeset 7621:108cda898646, which is currently available only in the Galaxy central repository. There is a new Galaxy distribution scheduled to go out today, but unfortunately this changeset will not be included. From here on, Galaxy releases are tentatively scheduled for every 2 weeks, so if you don't want to pull from the Galaxy central repository, the fix should be available in the next release 2 weeks from now. I've slightly altered your tool_dependencies.xml file (mostly because I've implemented support for a new make_directory tag. It is attached below - you should updated your tool_dependencies.xml file in your htseq_count repository with this updated version. 2) Another issue is the inability to install Numpy as a dependancy to HTSeq. I can get Numpy to install, but it won't be in the PYTHONPATH when HTSeq is installed (or executed). Were you attempting to install numpy as another package defined in your tool-dependencies.xml file? If so, the fixes in the above changeset should allow you to do that now since you can install it before htseq and set PYTHONPATH to point to numpy as part of the installation. Let me know if you need additional help or bump into further problems. The tool dependency installation components are fairly immature, so adding enhancements for new tools like yours is very valuable. Thanks for your help on this, Greg Von Kuster Any thoughts on how to address these issues would be appreciated. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Automatic installation of third party dependancies
I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks. URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/name=htseq_countowner=lparsonschangeset_revision=f7a5b54a8d4flatest_changeset_revision=14bec14f4290latest_ctx_rev=4 File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', line 1469 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', line 1655 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _(unknown revision '%s') % changeid) RepoLookupError: unknown revision '4' Lance Parsons wrote: Thanks Greg. I used you're updated version and added numpy as a separate dependency. It seems to work in my development system. I haven't updated my qa or production versions yet, so I can't check there. Perhaps you could test it and let me know if there are issues, etc. Glad I can be of help. Lance Greg Von Kuster wrote: Hello Lance, See my inline comments. On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote: I've put together a tool wrapper for the htseq-count script that is part of the HTSeq python package and uploaded that to the tool shed. However, I have discovered that the tool dependancies do not install properly. There are a couple of issues that I've run into. 1) The biggest issue is the inability to set both the PYTHONPATH and PATH environment variables as part of installation. If I put two separateaction type=set_environment tags, then the second overwrites the first in the env.sh file. If instead, I put twoenvironment_variable tags within the action, only one (the second) gets executed during installation. The above issues have been corrected in changeset 7621:108cda898646, which is currently available only in the Galaxy central repository. There is a new Galaxy distribution scheduled to go out today, but unfortunately this changeset will not be included. From here on, Galaxy releases are tentatively scheduled for every 2 weeks, so if you don't want to pull from the Galaxy central repository, the fix should be available in the next release 2 weeks
Re: [galaxy-dev] Automatic installation of third party dependancies
Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it. I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?) 4) I attempted to get updates and ran into the issue I described. I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool. Let me know if there is anything I can do to help sort this out. Lance Greg Von Kuster wrote: Hi Lance, What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions: 1:14e18dc9ed13 http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14e18dc9ed13id=2df7e24ce6c1f224 2:f5d08224af89 http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f5d08224af89id=2df7e24ce6c1f224 4:14bec14f4290 http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14bec14f4290id=2df7e24ce6c1f224 Since you could not have installed 3:f7a5b54a8d4f http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f7a5b54a8d4fid=2df7e24ce6c1f224, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates? I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem Greg Von Kuster On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote: I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks. URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/name=htseq_countowner=lparsonschangeset_revision=f7a5b54a8d4flatest_changeset_revision=14bec14f4290latest_ctx_rev=4 http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/name=htseq_countowner=lparsonschangeset_revision=f7a5b54a8d4flatest_changeset_revision=14bec14f4290latest_ctx_rev=4 File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', line 1469 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', line 1655 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux
[galaxy-dev] Tophat2 Issues
I've run into one bug and a one other smaller issue with the Tophat2 wrapper. The bug manifests itself when running the tool with use Defaults selected results in the following error: cp: cannot stat `/data/galaxy-dev/galaxy-dev/database/job_working_directory/000/281/tophat_out/fusions.out': No such file or directory It seems that somehow, the filter on the output dataset is not working properly when the value (params['fusion_search']['do_search']) is not explicitly set in the parameters. The second issue is more minor. It seems that the Tophat2 wrapper assumes that Tophat2 will use Bowtie2 (and not Bowtie). However, the actual Tophat2 program will only use Bowtie2 if it is found in the path. Otherwise it defaults to using bowtie which results in an error (Could not find Bowtie index files) since the wrapper points the tool to the bowtie2 index instead. To make things a bit more robust, I suggest adding bowtie2 as a requirement to the tophat2 wrapper. It also might be a good idea to add versions to those requirements. I've attached a small patch to do that (though it's fairly trivial). Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University --- tophat2_wrapper.xml 2012-09-12 11:41:46.0 -0400 +++ tophat2_wrapper_withreqs.xml2012-09-14 13:23:25.0 -0400 @@ -3,7 +3,8 @@ descriptionGapped-read mapper for RNA-seq data/description version_commandtophat2 --version/version_command requirements -requirement type=packagetophat2/requirement +requirement type=package version=2.0.0-beta7bowtie2/requirement +requirement type=package version=2.0.4tophat2/requirement /requirements command interpreter=python tophat2_wrapper.py ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Automatic installation of third party dependancies
I've run into this issue again, and I'm having a hard time working around it. However, I have confirmed that at least some updates to a tool in the tool shed will invalidate previously valid revisions and thus prevent users from installing or updating the tool at all. For example, push version 0.1 of the tool and create a valid revision 1:xx. Then install the tool in galaxy. Make a small change (say to tool_dependencies.xml) and push a new revision (but keep the tool version the same), now at revision 2:xxx. The tool shed will show 2:xx as the only valid revision to install, but the galaxy system with revision 1:xx will be stuck, unable to get upgrades (Server Error described previously). I'm trying to work around this now with my htseq-count tool, but so far no luck. I've created a few spurious revisions in the attempt, and I think now I may just try bumping the version (already did to no avail, toolshed still thinks it's the same) and uploading a tar file. That seems to more reliably parse metadata. Will let you know what, if anything, works. Thanks. Lance Greg Von Kuster wrote: Hello Lance, I've just committed a fix for getting updates to installed tool shed repositories in change set 7713:23107188eab8, which is currently available only in the Galaxy central repository. However, my fix will probably not correct the issue you're describing, and I'm still not able to reproduce this behavior. See my inline comments... On Sep 13, 2012, at 4:41 PM, Lance Parsons wrote: Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it. I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?) You are correct in stating that the tool version number should not change just because you've added a tool_dependencies.xml file. This is definitely not causing the behavior you're describing. 4) I attempted to get updates and ran into the issue I described. I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool. The following section of the tool shed wiki should provide the details about why you are seeing this behavior. Keep in mind that you will only get certain updates to installed repositories from the tool shed. This behavior enables updates to installed tool versions. To get a completely new version of an installed tool (if one exists), you need to install a new (different) changeset revision from the tool shed repository. http://wiki.g2.bx.psu.edu/Tool%20Shed#Getting_updates_for_tool_shed_repositories_installed_in_a_local_Galaxy_instance Let me know if there is anything I can do to help sort this out. Lance Greg Von Kuster wrote: Hi Lance, What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions: 1:14e18dc9ed13 http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14e18dc9ed13id=2df7e24ce6c1f224 2:f5d08224af89 http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f5d08224af89id=2df7e24ce6c1f224 4:14bec14f4290 http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14bec14f4290id=2df7e24ce6c1f224 Since you could not have installed 3:f7a5b54a8d4f http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f7a5b54a8d4fid=2df7e24ce6c1f224, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates? I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem Greg Von Kuster On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote: I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks. URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/name=htseq_countowner=lparsonschangeset_revision=f7a5b54a8d4flatest_changeset_revision=14bec14f4290latest_ctx_rev=4 http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/name=htseq_countowner
Re: [galaxy-dev] Automatic installation of third party dependancies
OK, I was able to get a new version installed. It seems there are two issues: 1) New revisions with the same version ionvalidate previous revisions. This means that Galaxy servers with the old, and now invalid, revisions are not able to update the tool (nor install it again). 2) Pushes from Mercurial (even version 2.3.3) do not seem to trigger metadata refreshes in the tool shed, however, uploads of tar.gz files do. Hope this helps. Lance Lance Parsons wrote: I've run into this issue again, and I'm having a hard time working around it. However, I have confirmed that at least some updates to a tool in the tool shed will invalidate previously valid revisions and thus prevent users from installing or updating the tool at all. For example, push version 0.1 of the tool and create a valid revision 1:xx. Then install the tool in galaxy. Make a small change (say to tool_dependencies.xml) and push a new revision (but keep the tool version the same), now at revision 2:xxx. The tool shed will show 2:xx as the only valid revision to install, but the galaxy system with revision 1:xx will be stuck, unable to get upgrades (Server Error described previously). I'm trying to work around this now with my htseq-count tool, but so far no luck. I've created a few spurious revisions in the attempt, and I think now I may just try bumping the version (already did to no avail, toolshed still thinks it's the same) and uploading a tar file. That seems to more reliably parse metadata. Will let you know what, if anything, works. Thanks. Lance Greg Von Kuster wrote: Hello Lance, I've just committed a fix for getting updates to installed tool shed repositories in change set 7713:23107188eab8, which is currently available only in the Galaxy central repository. However, my fix will probably not correct the issue you're describing, and I'm still not able to reproduce this behavior. See my inline comments... On Sep 13, 2012, at 4:41 PM, Lance Parsons wrote: Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it. I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?) You are correct in stating that the tool version number should not change just because you've added a tool_dependencies.xml file. This is definitely not causing the behavior you're describing. 4) I attempted to get updates and ran into the issue I described. I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool. The following section of the tool shed wiki should provide the details about why you are seeing this behavior. Keep in mind that you will only get certain updates to installed repositories from the tool shed. This behavior enables updates to installed tool versions. To get a completely new version of an installed tool (if one exists), you need to install a new (different) changeset revision from the tool shed repository. http://wiki.g2.bx.psu.edu/Tool%20Shed#Getting_updates_for_tool_shed_repositories_installed_in_a_local_Galaxy_instance Let me know if there is anything I can do to help sort this out. Lance Greg Von Kuster wrote: Hi Lance, What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions: 1:14e18dc9ed13 http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14e18dc9ed13id=2df7e24ce6c1f224 2:f5d08224af89 http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f5d08224af89id=2df7e24ce6c1f224 4:14bec14f4290 http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14bec14f4290id=2df7e24ce6c1f224 Since you could not have installed 3:f7a5b54a8d4f http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f7a5b54a8d4fid=2df7e24ce6c1f224, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates? I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem Greg Von Kuster On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote: I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I
Re: [galaxy-dev] Automatic installation of third party dependancies
I'm sorry I wasn't more clear. I do believe that those links explain the behavior I am seeing. However, let me try to describe it a different way. It seems that there will, at most, be one installable revision of a given version of a tool. Here I use revision to denote a mercurial revision and version to denote the version described in the tool's config xml. However, while it seems this is the desired behavior, it seems to lead to an undesirable situation in some circumstances. Consider this scenario: 1) Upload version 0.1 of MYTOOL at revision 0: to the tool shed. 2) Install version 0.1 of MYTOOL at revision 0:XXX to my local Galaxy instance. 3) Modify MYTOOL, leaving the version at 0.1, but changing the revision to 1:. These updates would be for things that don't change the input or output of a tool (such as updates to documentation, the addition of tool dependencies, etc.) 4) Attempt to update MYTOOL from my local Galaxy instance. This now results in an exception: RepoLookupError: unknown revision '1' . Deactivating and/or uninstalling MYTOOL works, but any attempt to reinstall the latest version results in an error stating that an older revision of the tool was installed. I now have to way to (easily) update my local Galaxy instance to this updated revision of the tool. My only workaround to this was to increment the version of MYTOOL and push another revision. This allowed me to install the new version and deactivate the old version in my local Galaxy instance. As for the mercurial issue, I'm running OS X, using the Homebrew version of Mercurial: hg --version Mercurial Distributed SCM (version 2.3.1) (see http://mercurial.selenic.com for more information) Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Let me know if I can help track things down any further, etc. Thanks. Lance Greg Von Kuster wrote: Hi Lance, On Sep 21, 2012, at 6:04 PM, Lance Parsons wrote: OK, I was able to get a new version installed. It seems there are two issues: 1) New revisions with the same version ionvalidate previous revisions. This means that Galaxy servers with the old, and now invalid, revisions are not able to update the tool (nor install it again). I'm not quite sure what you're stating here. Do the following tool shed wiki page clarify the behavior you are seeing? http://wiki.g2.bx.psu.edu/ToolShedRepositoryFeatures#Pushing_changes_to_a_repository_using_hg_from_the_command_line http://wiki.g2.bx.psu.edu/RepositoryRevisions#Installable_repository_changeset_revisions 2) Pushes from Mercurial (even version 2.3.3) do not seem to trigger metadata refreshes in the tool shed, however, uploads of tar.gz files do. I am not able to reproduce this behavior. In my environment, metadata is always automatically generated for new changesets I push to my local tool shed (or the test tool shed) from the command line. What is the result of typing the following in the environment from which you are pushing changes to the tool shed? $hg --version You should see something like the following, showing that you are running at least hg version 2.2.3. gvk:/tmp/repos/convert_chars gvk$ hg --version Mercurial Distributed SCM (version 2.2.3) (see http://mercurial.selenic.com for more information) Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Hope this helps. Lance Lance Parsons wrote: I've run into this issue again, and I'm having a hard time working around it. However, I have confirmed that at least some updates to a tool in the tool shed will invalidate previously valid revisions and thus prevent users from installing or updating the tool at all. For example, push version 0.1 of the tool and create a valid revision 1:xx. Then install the tool in galaxy. Make a small change (say to tool_dependencies.xml) and push a new revision (but keep the tool version the same), now at revision 2:xxx. The tool shed will show 2:xx as the only valid revision to install, but the galaxy system with revision 1:xx will be stuck, unable to get upgrades (Server Error described previously). I'm trying to work around this now with my htseq-count tool, but so far no luck. I've created a few spurious revisions in the attempt, and I think now I may just try bumping the version (already did to no avail, toolshed still thinks it's the same) and uploading a tar file. That seems to more reliably parse metadata. Will let you know what, if anything, works. Thanks. Lance Greg Von Kuster wrote: Hello Lance, I've just committed a fix for getting updates to installed tool shed repositories in change set 7713:23107188eab8, which
Re: [galaxy-dev] Automatic installation of third party dependancies
I'm actually still a bit confused as to what the expected workflow is. Should I be able to clone the tool shed repo to my local development workstation (TOOL_DEV_LOCAL_DIR), commit a change, and then push that change back to the Tool Shed? I've added some comments inline to clarify. Greg Von Kuster wrote: Hi Lance, I need to figure out precisely what steps you are taking to produce this behavior, as I have not been able to do so. Please see my inline comments, and let me know more information about each step you are taking to produce this behavior. On Sep 25, 2012, at 10:24 AM, Lance Parsons wrote: I'm sorry I wasn't more clear. I do believe that those links explain the behavior I am seeing. However, let me try to describe it a different way. It seems that there will, at most, be one installable revision of a given version of a tool. Here I use revision to denote a mercurial revision and version to denote the version described in the tool's config xml. However, while it seems this is the desired behavior, it seems to lead to an undesirable situation in some circumstances. Consider this scenario: 1) Upload version 0.1 of MYTOOL at revision 0: to the tool shed. I assume you are uploading the initial tarball or separate files from a certain local directory. Let's call it TOOL_DEV_LOCAL_DIR. Not quite. I would typically create the repository using the Tool Shed web interface. Then, on my local workstation, I would create the TOOL_DEV_LOCAL_DIR by using the 'hg clone http://toolshedurl/repo' command. 2) Install version 0.1 of MYTOOL at revision 0:XXX to my local Galaxy instance. This now creates a new mercurial repository at a specified location on your Galaxy server, let's call it REPO_INSTALL_DIR. Yes, I use the Galaxy Admin interface to install the tool from a tool shed to my Galaxy instance. 3) Modify MYTOOL, leaving the version at 0.1, You are doing the above step in TOOL_DEV_LOCAL_DIR, correct? Yes, I modify the files in TOOL_DEV_LOCAL_DIR, leaving the version the same. but changing the revision to 1:. The tool shed is creating the new change set hash for you when you push the changes from TOOL_DEV_LOCAL_DIR to the repository in the tool shed, correct? I don't think so. I would typically commit the changes by issuing an 'hg commit' while in my TOOL_DEV_LOCAL_DIR and then push them to the tool shed using the 'hg push' command. These updates would be for things that don't change the input or output of a tool (such as updates to documentation, the addition of tool dependencies, etc.) OK, this make sense. Right, I think we're on the same page here, though this should probably be spelled out somewhere in a Best Practices type document for tool development (which could also help to clarify the steps to setup a sane development environment, test tools, and push them to the public tool shed. I've not even quite gotten a solid workflow for myself, so publishing one might be useful. 4) Attempt to update MYTOOL from my local Galaxy instance. You are using the Galaxy UI to Get updates to your cloned repository, which is located in REPO_INSTALL_DIR, correct? Yes, then I use the Galaxy web admin interface to Get updates. This now results in an exception: RepoLookupError: unknown revision '1' . If answers to all of my above questions are Yes, then this is where I cannot reproduce the behavior you are seeing. If you are actually adding changsets to your cloned repository located in REPO_INSTALL_DIR, then pushing them to the repository in the tool shed, then attempting to get updates from the tool shed to your cloned repository in REPO_INSTALL_DIR, then this process may be causing the exception you are seeing. You should not be adding changesets to the cloned repository, but to your local development environment in TOOL_DEV_LOCAL_DIR. I don't add changesets to the REPO_INSTALL_DIR, but instead make changes in TOOL_DEV_LOCAL_DIR, commit them using mercurial, and then push them to the tool shed using hg push. Deactivating and/or uninstalling MYTOOL works, but any attempt to reinstall the latest version results in an error stating that an older revision of the tool was installed. I now have to way to (easily) update my local Galaxy instance to this updated revision of the tool. My only workaround to this was to increment the version of MYTOOL and push another revision. This allowed me to install the new version and deactivate the old version in my local Galaxy instance. As for the mercurial issue, I'm running OS X, using the Homebrew version of Mercurial: hg --version Mercurial Distributed SCM (version 2.3.1) (see http://mercurial.selenic.com for more information) Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. I'm
Re: [galaxy-dev] Cannot install ncbi_blast_plus or other tools from Tool Shed: HTTP 500 error
I ran into this issue as well on my locally installed instance. After some poking around, it looks like it might be a cookie issue. Specifically, the toolshedgalaxyurl and galaxycommunitysession cookies don't seem to be getting set by toolshed.g2.bx.psu.edu and this is causing the server error. If the cookies are set, the URL resolves normally, but if not, the URL results in a Server Error 500. Lance Joshua Shapiro wrote: I'm having the same issue. I started a brand new cloudman instance to see if the problem was something I had done, but I got the same error. Is the tool shed working for anyone? -Josh On Mon, Oct 8, 2012 at 1:13 PM, Joel Rosenbergthisisj...@hotmail.com wrote: (Apologies if this is in duplicate, Peter Cock may have forwarded it on after I posted it to the user list) So I'm trying to install the recently migrated ncbi_blast_plus tool from the Tool Shed to my local galaxy instance running in EC2 and am getting errors at the installation step. After clicking Install to local Galaxy I immediately get: Server Error An error occurred. See the error logs for more information. (Turn debug on to display exception reports here) and an HTTP 500 error code is returned. Chrome tells me the URL it's attempting to request is: http://toolshed.g2.bx.psu.edu/repository/install_repositories_by_revision?changeset_revisions=a4b9836f8f47repository_ids=cd0d8ada19d98f27 I tried the same process for other non-blast related tools that appeared to be simple (no 3rd party binary dependencies) and got the same server error. I restarted galaxy with its log level set to DEBUG and wasn't illuminated further, but that doesn't surprise me since my browser's requesting data from the public toolshed, not my server. I just performed a mercurial update in my cluster, too. Any ideas? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Updating help text for a tool shed repository
I recently updated the htseq-count tool in the Galaxy toolshed to version 0.3. This version has some functional changes and thus the tool version was incremented. I installed this new version of the tool in my local Galaxy instance. However, I realized later that I missed updating one part of the help text. Today I fixed that text (but left the tool version the same), tarred up the file, and uploaded it to the toolshed. I then attempted to Get Updates from my local Galaxy instances for that tool got the following error: URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/name=htseq_countowner=lparsonschangeset_revision=5d969cb56112latest_changeset_revision=f320093f1e8elatest_ctx_rev=11 File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 206 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 1677 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util_common.py', line 1428 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _(unknown revision '%s') % changeid) RepoLookupError: unknown revision '11' I believe this is because the revision I have installed (10) is not longer considered a valid revision by the toolshed (I presume since it is an older revision of with the same tool version). Any help would be appreciated. Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Updating help text for a tool shed repository
Since I've had issues with workflow and the toolshed in the past, I have changed my workflow so that my only interaction with the toolshed is either via uploading a tarball via the web interface or installing a tool into my local Galaxy via the web interface (or occasionally just downloading tarballs). In this specific case, I uploaded a tar.gz file of the entire tool wrapper directory. I did NOT increment the tool version. In the past, incrementing the tool version has generally worked fine. However, when I leave the tool version the same, but change something (such as the help text for the tool), and then upload a tarball to the toolshed, I am unable to pull updates into my local galaxy instance. I get an error similar to what I have described. I hope this is enough information, but let me know if I can provide more detail, etc. I would very much like to get to the root of this issue, since it has plagued my attempts to use the toolshed smoothly when (seemingly inevitable) minor changes need to be made. In the past I have just had to wait until I update the tool version, as no changes can be made to the installed revision in my production Galaxy instance. Greg Von Kuster wrote: Hi Lance, What process did you use for updating the tool shed repository and how many separate repositories did you use? You should not be updating a local repository, pushing the changes to the tool shed repository, and then attempting to pull the changes back to the same repository. Your repository environment should be something like this: DEV REPO ( all development updates are performed here, so this is where the tool version is incremented ) | V pushed changeset | V EXISTING TOOL SHED REPO | V pulled changeset to some previously cloned repo | V PREVIOUSLY CLONED (possibly production) REPO In other words, nothing should ever get pushed and pulled from the tool shed repo into the same repo. If this is the process you used, it would undoubtedly result in the behavior you saw. If you used separate repos as I describe above, we'll need more details as to the process you used in order to determine the cause of the problem. Greg Von Kuster On Dec 10, 2012, at 1:45 PM, Lance Parsons wrote: I recently updated the htseq-count tool in the Galaxy toolshed to version 0.3. This version has some functional changes and thus the tool version was incremented. I installed this new version of the tool in my local Galaxy instance. However, I realized later that I missed updating one part of the help text. Today I fixed that text (but left the tool version the same), tarred up the file, and uploaded it to the toolshed. I then attempted to Get Updates from my local Galaxy instances for that tool got the following error: URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/name=htseq_countowner=lparsonschangeset_revision=5d969cb56112latest_changeset_revision=f320093f1e8elatest_ctx_rev=11 File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 206 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 1677 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util_common.py', line 1428 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py
Re: [galaxy-dev] Updating help text for a tool shed repository
Thanks for looking into this Greg. The environment where I'm having this problem is our testing and production systems. Both environments are CentOS release 5.8 (Final) with Mercurial version 2.4.1. Interestingly, when I tried the same thing with a development setup on my OSX box (10.6.8) with Mercurial 2.4 I did not have the same issue. One difference is that on my testing/production boxes I do have an older version of the tool installed, so perhaps that is the issue. I also tried uninstalling, but that led to the (expected) error that that a previous changeset is installed: The tool shed repository htseq_count with owner lparsons and changeset revision f320093f1e8e was previously installed using changeset revision 5d969cb56112. The repository has been uninstalled, however, so reinstall the original repository instead of installing it again. You can get the latest updates for the repository using the Get updates option from the repository's Repository Actions pop-up menu. Click here to manage the repository. I then attempted to reinstall and I got a javascript message the cloning failed, and the repo is now stuck in the cloning state. I did not see anything in the paster logs regarding this at all. Here are the steps I took on my local system: 1. Upload htseq-count v0.3 to local toolshed 2. Install on local galaxy 3. Upload htseq-count v0.3-release2 to toolshed [Errno 2] No such file or directory: '/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/database/community_files/000/repo_3/./tool_data_table_conf.xml.sample' STDOUT from paster htseq-count.xml does not exist! sam_fa_indices.loc.sample does not exist! static/images/count_modes.png does not exist! test-data/htseq-test-paired.bam does not exist! test-data/htseq-test-paired_counts.tsv does not exist! test-data/htseq-test-paired_othercounts.tsv does not exist! test-data/htseq-test.bam does not exist! test-data/htseq-test.gff does not exist! test-data/htseq-test.sam does not exist! test-data/htseq-test_counts.tsv does not exist! test-data/htseq-test_othercounts.tsv does not exist! tool_data_table_conf.xml.sample does not exist! 4. Second attempt to upload The file 'htseq-count_0.3.release2.tar.gz' has been successfully uncompressed and uploaded to the repository. 1 undesirable items (.hg .svn .git directories, .DS_Store, hgrc files, etc) were removed from the archive. 13 files were removed from the repository root. 5. Get repository updates The installed repository named 'htseq_count' has been updated to change set revision '74a6a8221337'. Greg Von Kuster wrote: Hi Lance, I tried several scenarios today in an attempt to reproduce this but I was not able to. Have you tried uninstalling the repository (check the checkbox to completely uninstall it) and then reinstall it? I'm trying to reproduce this on a mac running mercurial version 2.2.3 and Python 2.5. What's your environment? Greg Von Kuster On Dec 13, 2012, at 2:30 PM, Lance Parsons wrote: Since I've had issues with workflow and the toolshed in the past, I have changed my workflow so that my only interaction with the toolshed is either via uploading a tarball via the web interface or installing a tool into my local Galaxy via the web interface (or occasionally just downloading tarballs). In this specific case, I uploaded a tar.gz file of the entire tool wrapper directory. I did NOT increment the tool version. In the past, incrementing the tool version has generally worked fine. However, when I leave the tool version the same, but change something (such as the help text for the tool), and then upload a tarball to the toolshed, I am unable to pull updates into my local galaxy instance. I get an error similar to what I have described. I hope this is enough information, but let me know if I can provide more detail, etc. I would very much like to get to the root of this issue, since it has plagued my attempts to use the toolshed smoothly when (seemingly inevitable) minor changes need to be made. In the past I have just had to wait until I update the tool version, as no changes can be made to the installed revision in my production Galaxy instance. Greg Von Kuster wrote: Hi Lance, What process did you use for updating the tool shed repository and how many separate repositories did you use? You should not be updating a local repository, pushing the changes to the tool shed repository, and then attempting to pull the changes back to the same repository. Your repository environment should be something like this: DEV REPO ( all development updates are performed here, so this is where the tool version is incremented ) | V pushed changeset | V EXISTING TOOL SHED REPO | V pulled changeset to some previously cloned repo | V PREVIOUSLY CLONED (possibly
Re: [galaxy-dev] Bug in handling user job limits (due to dynamic job runner code?)
Just an update on this issue. Upon further investigation, it looks like the dynamic job runner code in commit 6f3b4e8 broke this. I haven't been able to parse through everything going on yet to propose a fix, but I've discovered two things: 1) Moving the self.__clear_user_job_count() call outside of the if block, does indeed fix that issue 2) Fixing that leads the __check_user_jobs method to fail since it does not check `self.track_jobs_in_database`, but instead assumes that jobs are in the database, resulting in a different exception regarding invalid columns. Lance Parsons wrote: I submitted a trello ticket [(https://trello.com/c/6vxkqdjT) regarding this, but wanted to make sure I brought it to someone's attention (it's causing me some queue issues with my instance of galaxy). When registered_user_job_limit and anonymous_user_job_limit are set in universe.wsgi jobs cannot be run, instead the following error occurs: galaxy.jobs.handler ERROR 2012-12-04 12:44:51,869 failure running job 22396 Traceback (most recent call last): File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 183, in __monitor_step job_state = self.__check_if_ready_to_run( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 253, in __check_if_ready_to_run state = self.__check_user_jobs( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 274, in __check_user_jobs if not self.user_job_count: AttributeError: 'JobHandlerQueue' object has no attribute 'user_job_count' Commenting out the lines for registered_user_job_limit and anonymous_user_job_limit in universe.wsgi allows job to be queue one again. It looks like this is due to the fact that `self.__clear_user_job_count()` on line 159 of `handler.py` is only called when jobs are tracked in the database. If jobs are not tracked in the database (as in my case), the error occurs. Perhaps the fix would be to simply move the call outside the `if` block. It appears this was broken in the 2012-11-13 revision (73e05bc). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Bug in handling user job limits (due to dynamic job runner code?)
Thanks Nate, it's much appreciated. I'm glad I was able to help out a bit, just sorry I wasn't able to provide a complete fix (though maybe that's for the best if there is a rewrite underway). Lance Nate Coraor wrote: Hi Lance, I'm rewriting much of this code, I should have all of it committed (including this fix) some time this week. Thanks for figuring out exactly what's going on here. --nate On Jan 18, 2013, at 5:28 PM, Lance Parsons wrote: Just an update on this issue. Upon further investigation, it looks like the dynamic job runner code in commit 6f3b4e8 broke this. I haven't been able to parse through everything going on yet to propose a fix, but I've discovered two things: 1) Moving the self.__clear_user_job_count() call outside of the if block, does indeed fix that issue 2) Fixing that leads the __check_user_jobs method to fail since it does not check `self.track_jobs_in_database`, but instead assumes that jobs are in the database, resulting in a different exception regarding invalid columns. Lance Parsons wrote: I submitted a trello ticket [(https://trello.com/c/6vxkqdjT) regarding this, but wanted to make sure I brought it to someone's attention (it's causing me some queue issues with my instance of galaxy). When registered_user_job_limit and anonymous_user_job_limit are set in universe.wsgi jobs cannot be run, instead the following error occurs: galaxy.jobs.handler ERROR 2012-12-04 12:44:51,869 failure running job 22396 Traceback (most recent call last): File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 183, in __monitor_step job_state = self.__check_if_ready_to_run( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 253, in __check_if_ready_to_run state = self.__check_user_jobs( job ) File /data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py, line 274, in __check_user_jobs if not self.user_job_count: AttributeError: 'JobHandlerQueue' object has no attribute 'user_job_count' Commenting out the lines for registered_user_job_limit and anonymous_user_job_limit in universe.wsgi allows job to be queue one again. It looks like this is due to the fact that `self.__clear_user_job_count()` on line 159 of `handler.py` is only called when jobs are tracked in the database. If jobs are not tracked in the database (as in my case), the error occurs. Perhaps the fix would be to simply move the call outside the `if` block. It appears this was broken in the 2012-11-13 revision (73e05bc). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy: Dependency error htseq_count
Hi Joachim, It seems that the error is because your system is trying to install the library files to lib64/python instead of just lib/python. For reasons I don't quite understand, the setup.py installation fails when that directory doesn't exist (I would think it would just create it, but that doesn't appear to be the case). As a quick fix, you might try creating the directory /apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/. Create it as the user that galaxy runs under so make sure galaxy has permissions later. Then try reinstalling the dependency. I'm not sure that will work, but that is the gist of the problem. I noticed there is an update to the htseq-count package, so I'll try to put out a new version of the wrapper with a fix for that as well as the new version. Let me know if you still have trouble or if you find some sort of workaround or fix. I'd like to make the installation more robust. Lance Joachim Jacob | VIB | wrote: Hi Lance, [apologies for directly emailing to you and not through the galaxy-dev list. I will post it on the dev list later on] I am trying to install htseq-count, but it fails during the dependency installation of htseq. The error I get is: error: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 2] No such file or directory: '/apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/test-easy-install-30482.pth' The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was: /apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/ This directory does not currently exist. Please create it and try again, or choose a different installation directory (using the -d or --install-dir option). Any idea whether it is related to my setup (and what I can do about it)? Note: only the htseq_count dependency is not being installed. Thanks for providing the htseq_count wrapper! Now getting it through the toolshed will make it again a little closer to a perfect world :-) Cheers, Joachim Joachim Jacob Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 12/17/2012 05:28 PM, Lance Parsons wrote: Thanks for looking into this Greg. The environment where I'm having this problem is our testing and production systems. Both environments are CentOS release 5.8 (Final) with Mercurial version 2.4.1. Interestingly, when I tried the same thing with a development setup on my OSX box (10.6.8) with Mercurial 2.4 I did not have the same issue. One difference is that on my testing/production boxes I do have an older version of the tool installed, so perhaps that is the issue. I also tried uninstalling, but that led to the (expected) error that that a previous changeset is installed: The tool shed repository htseq_count with owner lparsons and changeset revision f320093f1e8e was previously installed using changeset revision 5d969cb56112. The repository has been uninstalled, however, so reinstall the original repository instead of installing it again. You can get the latest updates for the repository using the Get updates option from the repository's Repository Actions pop-up menu. Click here to manage the repository. I then attempted to reinstall and I got a javascript message the cloning failed, and the repo is now stuck in the cloning state. I did not see anything in the paster logs regarding this at all. Here are the steps I took on my local system: 1. Upload htseq-count v0.3 to local toolshed 2. Install on local galaxy 3. Upload htseq-count v0.3-release2 to toolshed [Errno 2] No such file or directory: '/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/database/community_files/000/repo_3/./tool_data_table_conf.xml.sample' STDOUT from paster htseq-count.xml does not exist! sam_fa_indices.loc.sample does not exist! static/images/count_modes.png does not exist! test-data/htseq-test-paired.bam does not exist! test-data/htseq-test-paired_counts.tsv does not exist! test-data/htseq-test-paired_othercounts.tsv does not exist! test-data/htseq-test.bam does not exist! test-data/htseq-test.gff does not exist! test-data/htseq-test.sam does not exist! test-data/htseq-test_counts.tsv does not exist! test-data/htseq-test_othercounts.tsv does not exist! tool_data_table_conf.xml.sample does not exist! 4. Second attempt to upload The file 'htseq-count_0.3.release2.tar.gz' has been successfully uncompressed and uploaded to the repository. 1 undesirable items (.hg .svn .git directories, .DS_Store, hgrc files, etc) were removed from the archive. 13 files were removed from the repository root. 5. Get repository updates The installed repository named 'htseq_count' has been updated to change set revision '74a6a8221337
Re: [galaxy-dev] Galaxy: Dependency error htseq_count
I have posted an updated version of the htseq-count tool in the toolshed that uses and updated version of the HTSEQ count package (0.5.4p1) and hopefully addresses the installation issue you have experienced. Please try it out and let me know if you still have problems. Lance Joachim Jacob | VIB | wrote: Hi Lance, [apologies for directly emailing to you and not through the galaxy-dev list. I will post it on the dev list later on] I am trying to install htseq-count, but it fails during the dependency installation of htseq. The error I get is: error: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 2] No such file or directory: '/apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/test-easy-install-30482.pth' The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was: /apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/ This directory does not currently exist. Please create it and try again, or choose a different installation directory (using the -d or --install-dir option). Any idea whether it is related to my setup (and what I can do about it)? Note: only the htseq_count dependency is not being installed. Thanks for providing the htseq_count wrapper! Now getting it through the toolshed will make it again a little closer to a perfect world :-) Cheers, Joachim Joachim Jacob Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 12/17/2012 05:28 PM, Lance Parsons wrote: Thanks for looking into this Greg. The environment where I'm having this problem is our testing and production systems. Both environments are CentOS release 5.8 (Final) with Mercurial version 2.4.1. Interestingly, when I tried the same thing with a development setup on my OSX box (10.6.8) with Mercurial 2.4 I did not have the same issue. One difference is that on my testing/production boxes I do have an older version of the tool installed, so perhaps that is the issue. I also tried uninstalling, but that led to the (expected) error that that a previous changeset is installed: The tool shed repository htseq_count with owner lparsons and changeset revision f320093f1e8e was previously installed using changeset revision 5d969cb56112. The repository has been uninstalled, however, so reinstall the original repository instead of installing it again. You can get the latest updates for the repository using the Get updates option from the repository's Repository Actions pop-up menu. Click here to manage the repository. I then attempted to reinstall and I got a javascript message the cloning failed, and the repo is now stuck in the cloning state. I did not see anything in the paster logs regarding this at all. Here are the steps I took on my local system: 1. Upload htseq-count v0.3 to local toolshed 2. Install on local galaxy 3. Upload htseq-count v0.3-release2 to toolshed [Errno 2] No such file or directory: '/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/database/community_files/000/repo_3/./tool_data_table_conf.xml.sample' STDOUT from paster htseq-count.xml does not exist! sam_fa_indices.loc.sample does not exist! static/images/count_modes.png does not exist! test-data/htseq-test-paired.bam does not exist! test-data/htseq-test-paired_counts.tsv does not exist! test-data/htseq-test-paired_othercounts.tsv does not exist! test-data/htseq-test.bam does not exist! test-data/htseq-test.gff does not exist! test-data/htseq-test.sam does not exist! test-data/htseq-test_counts.tsv does not exist! test-data/htseq-test_othercounts.tsv does not exist! tool_data_table_conf.xml.sample does not exist! 4. Second attempt to upload The file 'htseq-count_0.3.release2.tar.gz' has been successfully uncompressed and uploaded to the repository. 1 undesirable items (.hg .svn .git directories, .DS_Store, hgrc files, etc) were removed from the archive. 13 files were removed from the repository root. 5. Get repository updates The installed repository named 'htseq_count' has been updated to change set revision '74a6a8221337'. Greg Von Kuster wrote: Hi Lance, I tried several scenarios today in an attempt to reproduce this but I was not able to. Have you tried uninstalling the repository (check the checkbox to completely uninstall it) and then reinstall it? I'm trying to reproduce this on a mac running mercurial version 2.2.3 and Python 2.5. What's your environment? Greg Von Kuster On Dec 13, 2012, at 2:30 PM, Lance Parsons wrote: Since I've had issues with workflow and the toolshed in the past, I have changed my workflow so that my only interaction with the toolshed is either via uploading a tarball via the web interface or installing a tool into my local Galaxy via the web
[galaxy-dev] Improving Administrative Data Clean Up (pgcleanup.py vs cleanup_datasets.py)
I have been running a Galaxy server for our sequencing researchers for a while now and it's become increasingly successful. The biggest resource challenge for us has been, and continues to be disk space. As such, I'd like to implement some additional cleanup scripts. I thought I run a few questions by this list before I got too far into things. In general, I'm wondering how to implement updates/additions to the cleanup system that will be in line with the direction that the Galaxy project is headed. The pgcleanup.py script is the newest piece of code in this area (and even adds cleanup of exported histories, which are absent from the older cleanup scripts). Also, the pgcleanup.py script uses a cleanup_event table that I don't believe is used by the older cleanup_datasets.py script. However, the new pgcleanup.py script only works for Postgres, and worse, only for version 9.1+. I run my system on RedHat (CentOS) and thus we use version 8.4 of Postgres. Are there plans to support other databases or older versions of Postgres? I'd like to implement a script to delete (set the deleted flag) for certain datasets (e.g. raw data imported from our archive, for old, inactive users, etc.). I'm wondering if it would make sense to try and extend pgcleanup.py or cleanup_datasets.py. Or perhaps it would be best to just implement a separate script, though that seems like I'd have to re-implement a lot of boilerplate code for configuration reading, connections, logging, etc. Any tips on generally acceptable (supported) procedures for marking a dataset as deleted? Of course, I'll make any of the enhancements available (and would be happy to submit pull requests if there is interest). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy: Dependency error htseq_count
Glad that everything works now for you. The Get Updates functionality won't work since this is a new version of the tool (updated version of HTSeq package). Since the updated version could generate different results with the same input, Galaxy keeps the old version around for reproducibility reasons. I don't know why you have to restart Galaxy for the tool to show up. As far as I know, that should not be necessary, and also shouldn't be dependent on anything in the tool wrapper configuration. I'm aware of the issue with the screenshot missing. I'm not sure how to get the tool wrapper to point to the static images installed from the toolshed (or if that's even possible). In the meantime, a workaround would be to copy the count_modes.png file from the repository to the [GALAXY_HOME]/static/images directory. Lance Joachim Jacob | VIB | wrote: *sorry for my late reply - we have experienced some email issues lately * Thanks, a clean install from the new versions works! Unfortunately, the 'get updates' from the admin interface of Galaxy does not find any updates with the old version. So, I searched the main toolshed, and I was able to install the new version next to the old (instead of updating). The install succeeded, but I always have to restart Galaxy to get the tool available in the toolbox. One small remark: the screenshot in the tool's interface below the parameters is not shown. Cheers Joachim Joachim Jacob Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 03/11/2013 06:34 PM, Lance Parsons wrote: I have posted an updated version of the htseq-count tool in the toolshed that uses and updated version of the HTSEQ count package (0.5.4p1) and hopefully addresses the installation issue you have experienced. Please try it out and let me know if you still have problems. Lance Joachim Jacob | VIB | wrote: Hi Lance, [apologies for directly emailing to you and not through the galaxy-dev list. I will post it on the dev list later on] I am trying to install htseq-count, but it fails during the dependency installation of htseq. The error I get is: error: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 2] No such file or directory: '/apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/test-easy-install-30482.pth' The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was: /apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/ This directory does not currently exist. Please create it and try again, or choose a different installation directory (using the -d or --install-dir option). Any idea whether it is related to my setup (and what I can do about it)? Note: only the htseq_count dependency is not being installed. Thanks for providing the htseq_count wrapper! Now getting it through the toolshed will make it again a little closer to a perfect world :-) Cheers, Joachim Joachim Jacob Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 12/17/2012 05:28 PM, Lance Parsons wrote: Thanks for looking into this Greg. The environment where I'm having this problem is our testing and production systems. Both environments are CentOS release 5.8 (Final) with Mercurial version 2.4.1. Interestingly, when I tried the same thing with a development setup on my OSX box (10.6.8) with Mercurial 2.4 I did not have the same issue. One difference is that on my testing/production boxes I do have an older version of the tool installed, so perhaps that is the issue. I also tried uninstalling, but that led to the (expected) error that that a previous changeset is installed: The tool shed repository htseq_count with owner lparsons and changeset revision f320093f1e8e was previously installed using changeset revision 5d969cb56112. The repository has been uninstalled, however, so reinstall the original repository instead of installing it again. You can get the latest updates for the repository using the Get updates option from the repository's Repository Actions pop-up menu. Click here to manage the repository. I then attempted to reinstall and I got a javascript message the cloning failed, and the repo is now stuck in the cloning state. I did not see anything in the paster logs regarding this at all. Here are the steps I took on my local system: 1. Upload htseq-count v0.3 to local toolshed 2. Install on local galaxy 3. Upload htseq-count v0.3-release2 to toolshed [Errno 2] No such file or directory: '/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/database/community_files/000/repo_3/./tool_data_table_conf.xml.sample' STDOUT from paster htseq
[galaxy-dev] Toolshed - Installation of static files
For one of my tool wrappers (htseq-count) in the toolshed, I point to an image file in the Help section. At the moment, I'm including the image in the repository as static/imagescount_modes.png and pointing to it in the wrapper.xml file with .. image:: /static/images/count_modes.png. However, the toolshed installation doesn't copy this file to the static directory of the galaxy install (which doesn't seem like a good idea anyway). Is there any way for me to include a static file and then point to that file in the help when the tool is installed automatically via the toolshed.. Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Toolshed - Installation of static files
Thanks for the quick response Greg. I tried this and uploaded an update to the testtoolshed and installed that in a test instance of Galaxy. Unfortunately, it didn't seem to resolve the issue. Galaxy still renders the html with the image source pointing to the same location (e.g. http://127.0.0.1:8080/static/images/count_modes.png). This doesn't resolve. You can try installing the htseq_count package from the testtoolshed to see the issue for yourself. Lance Greg Von Kuster wrote: Hi Lance, I answered this in your previous email - here's the answer: To make this work in the tool shed, make the static directory that contains the image file a relative path in the tool config. For example, your repository has the following structure: htseq_count http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=2df7e24ce6c1f224# htseq-count.xml http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=2df7e24ce6c1f224# sam_fa_indices.loc.sample http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=2df7e24ce6c1f224# static/ http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=2df7e24ce6c1f224# images/ http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=2df7e24ce6c1f224# count_modes.png http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=2df7e24ce6c1f224# test-data/ http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=2df7e24ce6c1f224# tool_data_table_conf.xml.sample http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=2df7e24ce6c1f224# tool_dependencies.xml http://toolshed.g2.bx.psu.edu/repository/browse_repository?id=2df7e24ce6c1f224# The help section of your tool config should simply use a relative path to the image file ( in this case, just a a . in front of the path to the image file ) .. image:: ./static/images/count_modes.png :width: 500 I believe that's all that is necessary. Greg Von Kuster On Mar 22, 2013, at 12:44 PM, Lance Parsons wrote: For one of my tool wrappers (htseq-count) in the toolshed, I point to an image file in the Help section. At the moment, I'm including the image in the repository as static/imagescount_modes.png and pointing to it in the wrapper.xml file with .. image:: /static/images/count_modes.png. However, the toolshed installation doesn't copy this file to the static directory of the galaxy install (which doesn't seem like a good idea anyway). Is there any way for me to include a static file and then point to that file in the help when the tool is installed automatically via the toolshed.. Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Improving Administrative Data Clean Up (pgcleanup.py vs cleanup_datasets.py)
Nate Coraor wrote: On Mar 22, 2013, at 11:56 AM, Lance Parsons wrote: I have been running a Galaxy server for our sequencing researchers for a while now and it's become increasingly successful. The biggest resource challenge for us has been, and continues to be disk space. As such, I'd like to implement some additional cleanup scripts. I thought I run a few questions by this list before I got too far into things. In general, I'm wondering how to implement updates/additions to the cleanup system that will be in line with the direction that the Galaxy project is headed. The pgcleanup.py script is the newest piece of code in this area (and even adds cleanup of exported histories, which are absent from the older cleanup scripts). Also, the pgcleanup.py script uses a cleanup_event table that I don't believe is used by the older cleanup_datasets.py script. However, the new pgcleanup.py script only works for Postgres, and worse, only for version 9.1+. I run my system on RedHat (CentOS) and thus we use version 8.4 of Postgres. Are there plans to support other databases or older versions of Postgres? Hi Lance, pgcleanup.py makes extensive use of Writable CTEs, so there is not really a way to port it to older versions. For 8.4 or MySQL, you can still use the older cleanup_datasets.py. After looking at it a bit more, I see what you mean. Are there plans to implement and additional cleanup scripts for non-postgres 9.1 users? Just curious so I don't reinvent the wheel, I'd be happy to help with existing efforts. I'd like to implement a script to delete (set the deleted flag) for certain datasets (e.g. raw data imported from our archive, for old, inactive users, etc.). I'm wondering if it would make sense to try and extend pgcleanup.py or cleanup_datasets.py. Or perhaps it would be best to just implement a separate script, though that seems like I'd have to re-implement a lot of boilerplate code for configuration reading, connections, logging, etc. Any tips on generally acceptable (supported) procedures for marking a dataset as deleted? You could probably reuse a lot of the code from either of the cleanup scripts for this. Right. It seems to make sense to me to focus on the cleanup_datasets.py since that will work for everyone. I would like to essentially mimic the user deleting a dataset. I'd then email them to let them know that some old data had been marked for deletion and let the rest of the scripts proceed as normal, cleaning that up if they don't undelete it. It looks like I would want to mark the HistoryDatasetAssociations as deleted? Is that correct? Would I need to do anything else to simulate the user deleting the dataset? Thanks for the help, Lance Thanks, --nate -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Bowtie2 and Tophat2 versions
I noticed that the latest versions for Bowtie2 and Tophat2 listed on the Tool Dependencies page (http://wiki.galaxyproject.org/Admin/Tools/Tool%20Dependencies) are 2.0.4 for Bowtie2 and 2.0.6 for Tophat2. Are these the latest supported versions? Does anyone know if it's advisable to go with the latest (Bowtie2 2.1.0 and Tophat 2.0.8) with Galaxy? I'm looking forward to these tools being moved to the ToolShed so updates to the underlying tools are accompanied by Galaxy tool version updates, but until then I'm just trying to make sure my users can use the latest versions that are compatible. Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Improving Administrative Data Clean Up (pgcleanup.py vs cleanup_datasets.py)
I have implemented a basic script to perform administrative dataset deletion and email notification. Right now it's limited to the history_dataset_assocation update_time and an optional tool_id string. I have pushed it to a galaxy-central fork and will issue a pull request once I've tested it a bit more. I'm open to comments or suggestions, it could certainly be extended. Hopefully people find this useful. See https://bitbucket.org/lance_parsons/galaxy-central-dataset-cleanup/commits/aa938fc707ae314ceeecd1e13631663f7596b0f4 admin_dataset_cleanup.py Documentation - Mark datasets as deleted that are older than specified cutoff and (optionaly) with a tool_id that matches the specified search string. This script is useful for administrators to cleanup after users who leave many old datasets around. It was modeled after the cleanup_datasets.py script originally distributed with Galaxy. Basic Usage: admin_cleanup_datasets.py universe_wsgi.ini -d 60 \ --template=email_template.txt Required Arguments: config_file - the Galaxy configuration file (universe_wsgi.ini) Optional Arguments: -d --days - number of days old the dataset must be (default: 60) --tool_id - string to search for in dataset tool_id --template - Mako template file to use for email notification -i --info_only - Print results, but don't email or delete anything -e --email_only - Email notifications, but don't delete anything Useful for notifying users of pending deletion --smtp - Specify smtp server If not specified, use smtp settings specified in config file --fromaddr - Specify from address If not specified, use error_email_to specified in config file Email Template Variables: cutoff - the cutoff in days email - the users email address datasets - a list of tuples containing 'dataset' and 'history' names xiandongm...@lbl.gov wrote: Hi Lance, I have same questions as yours about cleanup data in Galaxy. We maintain a local instance of the Galaxy system. I am thinking a way to delete datasets which are not accessed/updated for a certain period of time, no matter if users deleted them. In addition, sending an email to users before deleting their dataset. It looks like the current scripts for clean up only purges deleted histories/datasets. I was trying to find a way to use API functions to delete old files which are not deleted, but am not successful. I found the update_time is not access time in the dataset table. A reference file may be used frequently but the update_time is pretty old. This would be a problem if deleting file by the update_time. I think files should be deleted by the access time. Is it enough to resolve this problem by checking the update_time of the HistoryDatasetAssociation table? Thanks for your help, X. Meng Joint Genome Institute, LBNL quote author='Lance Parsons' Nate Coraor wrote: On Mar 22, 2013, at 11:56 AM, Lance Parsons wrote: I have been running a Galaxy server for our sequencing researchers for a while now and it's become increasingly successful. The biggest resource challenge for us has been, and continues to be disk space. As such, I'd like to implement some additional cleanup scripts. I thought I run a few questions by this list before I got too far into things. In general, I'm wondering how to implement updates/additions to the cleanup system that will be in line with the direction that the Galaxy project is headed. The pgcleanup.py script is the newest piece of code in this area (and even adds cleanup of exported histories, which are absent from the older cleanup scripts). Also, the pgcleanup.py script uses a cleanup_event table that I don't believe is used by the older cleanup_datasets.py script. However, the new pgcleanup.py script only works for Postgres, and worse, only for version 9.1+. I run my system on RedHat (CentOS) and thus we use version 8.4 of Postgres. Are there plans to support other databases or older versions of Postgres? Hi Lance, pgcleanup.py makes extensive use of Writable CTEs, so there is not really a way to port it to older versions. For 8.4 or MySQL, you can still use the older cleanup_datasets.py. After looking at it a bit more, I see what you mean. Are there plans to implement and additional cleanup scripts for non-postgres 9.1 users? Just curious so I don't reinvent the wheel, I'd be happy to help with existing efforts. I'd like to implement a script to delete (set the deleted flag) for certain datasets (e.g. raw data imported from our archive, for old, inactive users, etc.). I'm wondering if it would make sense to try and extend pgcleanup.py or cleanup_datasets.py. Or perhaps it would be best to just implement a separate script, though that seems like I'd have to re-implement a lot of boilerplate code for configuration reading, connections, logging, etc. Any tips on generally acceptable (supported
Re: [galaxy-dev] Tools declaring Python dependencies
I modified the PYTHONPATH in htseq_count (http://toolshed.g2.bx.psu.edu/view/lparsons/htseq_count) and cutadapt (http://toolshed.g2.bx.psu.edu/view/lparsons/cutadapt). It would be great to have some common dependencies setup in the toolshed such as numpy, etc. Lance Björn Grüning wrote: Hi Peter, On Wed, Apr 17, 2013 at 6:05 PM, James Taylorja...@jamestaylor.org wrote: No it does not have root, and it is very important that we not install anything at the system level since we need to maintain version isolation. We should always be able to inject things into the environment so that Python/Perl/R look in the right place. So is there any guidance on how that is best done? I do not now it it is best practise but have a look at package_numpy_1_7 and package_matplotlib_1_2 I will try to setup a few common python deps, so we can include them easily in tool_dependecies.xml Python has various options including virtual env, but I doubt we can make any assumptions about the host system's python - even which version exactly. Does that mean installing Python modules locally and then manipulating the PYTHONPATH is the best plan? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Improving Administrative Data Clean Up (pgcleanup.py vs cleanup_datasets.py)
Just an update on the cleanup script. I have implemented a basic script to perform administrative dataset deletion and email notification. Right now it's limited to the history_dataset_assocation update_time and an optional tool_id string. I have pushed it to a galaxy-central and issued a pull request to galaxy-central (https://bitbucket.org/galaxy/galaxy-central/pull-request/158/basic-administrative-dataset-cleanup/diff). I'm open to comments or suggestions, it could certainly be extended. Hopefully people find this useful. admin_dataset_cleanup.py Documentation - Mark datasets as deleted that are older than specified cutoff and (optionally) with a tool_id that matches the specified search string. This script is useful for administrators to cleanup after users who leave many old datasets around. It was modeled after the cleanup_datasets.py script originally distributed with Galaxy. Basic Usage: admin_cleanup_datasets.py universe_wsgi.ini -d 60 \ --template=email_template.txt Required Arguments: config_file - the Galaxy configuration file (universe_wsgi.ini) Optional Arguments: -d --days - number of days old the dataset must be (default: 60) --tool_id - string to search for in dataset tool_id --template - Mako template file to use for email notification -i --info_only - Print results, but don't email or delete anything -e --email_only - Email notifications, but don't delete anything Useful for notifying users of pending deletion --smtp - Specify smtp server If not specified, use smtp settings specified in config file --fromaddr - Specify from address If not specified, use error_email_to specified in config file Email Template Variables: cutoff - the cutoff in days email - the users email address datasets - a list of tuples containing 'dataset' and 'history' names Lance Parsons wrote: Nate Coraor wrote: On Mar 22, 2013, at 11:56 AM, Lance Parsons wrote: I have been running a Galaxy server for our sequencing researchers for a while now and it's become increasingly successful. The biggest resource challenge for us has been, and continues to be disk space. As such, I'd like to implement some additional cleanup scripts. I thought I run a few questions by this list before I got too far into things. In general, I'm wondering how to implement updates/additions to the cleanup system that will be in line with the direction that the Galaxy project is headed. The pgcleanup.py script is the newest piece of code in this area (and even adds cleanup of exported histories, which are absent from the older cleanup scripts). Also, the pgcleanup.py script uses a cleanup_event table that I don't believe is used by the older cleanup_datasets.py script. However, the new pgcleanup.py script only works for Postgres, and worse, only for version 9.1+. I run my system on RedHat (CentOS) and thus we use version 8.4 of Postgres. Are there plans to support other databases or older versions of Postgres? Hi Lance, pgcleanup.py makes extensive use of Writable CTEs, so there is not really a way to port it to older versions. For 8.4 or MySQL, you can still use the older cleanup_datasets.py. After looking at it a bit more, I see what you mean. Are there plans to implement and additional cleanup scripts for non-postgres 9.1 users? Just curious so I don't reinvent the wheel, I'd be happy to help with existing efforts. I'd like to implement a script to delete (set the deleted flag) for certain datasets (e.g. raw data imported from our archive, for old, inactive users, etc.). I'm wondering if it would make sense to try and extend pgcleanup.py or cleanup_datasets.py. Or perhaps it would be best to just implement a separate script, though that seems like I'd have to re-implement a lot of boilerplate code for configuration reading, connections, logging, etc. Any tips on generally acceptable (supported) procedures for marking a dataset as deleted? You could probably reuse a lot of the code from either of the cleanup scripts for this. Right. It seems to make sense to me to focus on the cleanup_datasets.py since that will work for everyone. I would like to essentially mimic the user deleting a dataset. I'd then email them to let them know that some old data had been marked for deletion and let the rest of the scripts proceed as normal, cleaning that up if they don't undelete it. It looks like I would want to mark the HistoryDatasetAssociations as deleted? Is that correct? Would I need to do anything else to simulate the user deleting the dataset? Thanks for the help, Lance Thanks, --nate -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative
Re: [galaxy-dev] Galaxy hg repository problem abort: path ... traverses symbolic link
I ran into the same problem as Peter, and you're post was very helpful John. Thanks for the workaround! Lance Peter Cock wrote: On Thu, Apr 11, 2013 at 5:02 PM, John Chiltonchil...@msi.umn.edu wrote: Hey Peter, Hopefully Nate will have some wisdom, all I have is a workaround (I have fought with this on multiple forks and this is the best I have come up with). Check out a clean version of the tools branch somehow (it seems like the working contents can get stuck in an invalid state that update and checkout cannot recover from). Once you are back to a good state, it works when you merge in two steps. hg checkout tools hg merge -r ee389d2 hg ci -m Partial merge. # Not sure if this step is needed. hg merge default -John I don't understand quite what it is doing, but it seems to have worked on a quick test repository (I re-cloned Galaxy and my fork on bitbucket). I've not just push any of these changes to bitbucket - they're all still local to this experiment for now. Unless I hit an issue or Nate has some alternative recommendations, I'll probably do this tomorrow (its home time here in the UK). Thanks John, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Issue with set_user_disk_usage.py and Postgres 8.x
Thanks for confirmation Björn. Pull request was submitted: https://bitbucket.org/galaxy/galaxy-central/pull-request/97/fix-for-postgres-versions-9/diff Trello card also created: https://trello.com/c/Z0EynAV2 Björn Grüning wrote: Hi Lance, I can confirm both the regression and the fix. A pull request would be great I think. I can also do it if you want. Thanks, Björn The recent updates to set_user_disk_usage.py for Postgres users have an issue with Postgres 8.x. The SQL in the pgcalc method (line 51) leads to the following error: sqlalchemy.exc.ProgrammingError: (ProgrammingError) column d.total_size must appear in the GROUP BY clause or be used in an aggregate function LINE 4: FROM ( SELECT d.total_siz... ^ The problem is that version of Postgres before 9.x were a bit more restrictive in the use of GROUP BY. This can be fixed using DISTINCT ON instead. See this StackOverflow post for more info: http://stackoverflow.com/questions/1769361/postgresql-group-by-different-from-mysql I've included a patch below. Let me know if a pull request would be preferred. --- a/scripts/set_user_disk_usage.py +++ b/scripts/set_user_disk_usage.py @@ -52,7 +52,7 @@ sql = UPDATE galaxy_user SET disk_usage = (SELECT COALESCE(SUM(total_size), 0) - FROM ( SELECT d.total_size + FROM ( SELECT DISTINCT ON (d.id) d.total_size, d.id FROM history_dataset_association hda JOIN history h ON h.id = hda.history_id JOIN dataset d ON hda.dataset_id = d.id @@ -62,7 +62,7 @@ AND d.purged = false AND d.id NOT IN (SELECT dataset_id FROM library_dataset_dataset_association) -GROUP BY d.id) sizes) +) sizes) WHERE id = :id RETURNING disk_usage; -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Issue with set_user_disk_usage.py and Postgres 8.x
No problem, glad it was helpful. Dannon Baker wrote: Thanks for the bug fix, (and for the reminder that it was still sitting in the inbox, sorry about that). I've verified the fix as well and pulled it into -central. -Dannon On Thu, Jun 6, 2013 at 10:13 AM, Lance Parsons lpars...@princeton.edu mailto:lpars...@princeton.edu wrote: Thanks for confirmation Björn. Pull request was submitted: https://bitbucket.org/galaxy/galaxy-central/pull-request/97/fix-for-postgres-versions-9/diff Trello card also created: https://trello.com/c/Z0EynAV2 Björn Grüning wrote: Hi Lance, I can confirm both the regression and the fix. A pull request would be great I think. I can also do it if you want. Thanks, Björn The recent updates to set_user_disk_usage.py for Postgres users have an issue with Postgres 8.x. The SQL in the pgcalc method (line 51) leads to the following error: sqlalchemy.exc.ProgrammingError: (ProgrammingError) column d.total_size must appear in the GROUP BY clause or be used in an aggregate function LINE 4: FROM ( SELECT d.total_siz... ^ The problem is that version of Postgres before 9.x were a bit more restrictive in the use of GROUP BY. This can be fixed using DISTINCT ON instead. See this StackOverflow post for more info: http://stackoverflow.com/questions/1769361/postgresql-group-by-different-from-mysql I've included a patch below. Let me know if a pull request would be preferred. --- a/scripts/set_user_disk_usage.py +++ b/scripts/set_user_disk_usage.py @@ -52,7 +52,7 @@ sql = UPDATE galaxy_user SET disk_usage = (SELECT COALESCE(SUM(total_size), 0) - FROM ( SELECT d.total_size + FROM ( SELECT DISTINCT ON (d.id http://d.id) d.total_size,d.id http://d.id FROM history_dataset_association hda JOIN history h ON h.id http://h.id = hda.history_id JOIN dataset d ON hda.dataset_id =d.id http://d.id @@ -62,7 +62,7 @@ AND d.purged = false ANDd.id http://d.id NOT IN (SELECT dataset_id FROM library_dataset_dataset_association) -GROUP BYd.id http://d.id) sizes) +) sizes) WHERE id = :id RETURNING disk_usage; -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Toolshed bug related to readme files
I have run into an issue when trying to reinstall a tool that was originally installed via migration scripts (toolshed version of picard). When I attempt to reinstall (after successfully uninstalling) I get a 500 error. After a bit of poking around, it seems that this URL generates the 500 error: http://toolshed.g2.bx.psu.edu/repository/get_readme_files?name=picardowner=devteamchangeset_revision=e0232cbac965 Looks like this changeset is relevant: https://bitbucket.org/galaxy/galaxy-central/commits/f72fd96d3fac5e17ec5c9aaece023b0ae74db696 -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Toolshed bug related to readme files
I'm running the latest stable. The 500 error seems to be coming from the toolshed. Do you also get a 500 error at the toolshed URL in my previous message? There is nothing other than standard info line from the request in my pasted log. I'm not able to get to the log right now, but I'll send it once I get a chance tomorrow. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University Original message From: Greg Von Kuster g...@bx.psu.edu Date: 06/11/2013 10:00 PM (GMT-05:00) To: Lance Parsons lpars...@princeton.edu Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Toolshed bug related to readme files Hi Lance, What version of Galaxy are you running? Is the 500 error coming from your Galaxy server or the tool shed server? Can you send the relevant information from your Galaxy paster log when attempting to reinstall? Thanks, Greg Von Kuster On Jun 11, 2013, at 3:52 PM, Lance Parsons lpars...@princeton.edu wrote: I have run into an issue when trying to reinstall a tool that was originally installed via migration scripts (toolshed version of picard). When I attempt to reinstall (after successfully uninstalling) I get a 500 error. After a bit of poking around, it seems that this URL generates the 500 error: http://toolshed.g2.bx.psu.edu/repository/get_readme_files?name=picardowner=devteamchangeset_revision=e0232cbac965 Looks like this changeset is relevant: https://bitbucket.org/galaxy/galaxy-central/commits/f72fd96d3fac5e17ec5c9aaece023b0ae74db696 -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Toolshed Collaboration Issues
I'm having some difficulty collaborating on a tool development in the toolshed. It seems to happen on the production toolshed as well as the testtoolshed. When I try to grant another user the ability to update a repository (or when the other person tries to add me) we get the following error: URL: http://testtoolshed.g2.bx.psu.edu/repository/manage_repository?id=f49c56ded06bdcd9 File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/debug/prints.py', line 106 in __call__ environ, self.app) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/wsgilib.py', line 543 in intercept_output app_iter = application(environ, replacement_start_response) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 98 in decorator return func( self, trans, *args, **kwargs ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/webapps/tool_shed/controllers/repository.py', line 2124 in manage_repository user = trans.sa_session.query( trans.model.User ).get( trans.security.decode_id( user_id ) ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/security/__init__.py', line 60 in decode_id return int( self.id_cipher.decrypt( obj_id.decode( 'hex' ) ).lstrip( ! ) ) File '/usr/lib/python2.7/encodings/hex_codec.py', line 42 in hex_decode output = binascii.a2b_hex(input) TypeError: Non-hexadecimal digit found Can anyone else successfully grant a user update permission to a repository? I've submitted an issue on Trello (https://trello.com/c/XYkqW3Lr). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Toolshed Collaboration Issues
Perhaps it has something to do with a specific account? The user 'nilesh' tried to add me to his rseqc repository (http://toolshed.g2.bx.psu.edu/view/nilesh/rseqc ) and got a Server Error. I then created the rseqc_dev3 repository (http://testtoolshed.g2.bx.psu.edu/view/lparsons/rseqc_dev3 )on the testtoolshed and tried to add him (nilesh) and got the error indicated below. Any help in tracking this down would be appreciated. Björn Grüning wrote: Hi Lance, I worked for me the last time. I usually create a repository under the IUC account and add me as contributor. No problem so far. Cheers, Bjoern I'm having some difficulty collaborating on a tool development in the toolshed. It seems to happen on the production toolshed as well as the testtoolshed. When I try to grant another user the ability to update a repository (or when the other person tries to add me) we get the following error: URL: http://testtoolshed.g2.bx.psu.edu/repository/manage_repository?id=f49c56ded06bdcd9 File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/debug/prints.py', line 106 in __call__ environ, self.app) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/wsgilib.py', line 543 in intercept_output app_iter = application(environ, replacement_start_response) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/framework/__init__.py', line 98 in decorator return func( self, trans, *args, **kwargs ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/webapps/tool_shed/controllers/repository.py', line 2124 in manage_repository user = trans.sa_session.query( trans.model.User ).get( trans.security.decode_id( user_id ) ) File '/var/opt/galaxy/g2cmnty/galaxy_testtoolshed/lib/galaxy/web/security/__init__.py', line 60 in decode_id return int( self.id_cipher.decrypt( obj_id.decode( 'hex' ) ).lstrip( ! ) ) File '/usr/lib/python2.7/encodings/hex_codec.py', line 42 in hex_decode output = binascii.a2b_hex(input) TypeError: Non-hexadecimal digit found Can anyone else successfully grant a user update permission to a repository? I've submitted an issue on Trello (https://trello.com/c/XYkqW3Lr). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] installing htseq-count from tool-shed
Hi Karen, Sorry your having trouble with installing the tool. It seems that the somehow, the numpy dependency didn't install properly or isn't being found during the HTSeq install. Is there any other info in the log to indicate an issue with installing Numpy? Also, what OS are you running Galaxy on? Lance On June 10, 2014 4:04:13 AM EDT, Karen Chait kch...@techunix.technion.ac.il wrote: Hi, I am trying to install the tool htseq-count from the tool-shed. I had a problem in the first installation so I uninstalled the tool and all its dependencies (numpy, samtools 0.1.19 and pysam) from the 'manage installed tool shed repositories) and installed it again but then I get installation status 'Installed, missing tool dependencies' for htseq-count. When running this tool I get the error Fatal error: The HTSeq python package is not properly installed, contact Galaxy administrators The path set in universe_wsgi.ini: tool_dependency_dir = /home/galaxy/galaxy-dist/tool_deps in the installation log htseq/0.6.1/lparsons/htseq_count/d5edaf8dc974/INSTALLATION.log I get the error: Setup script for HTSeq: Failed to import 'numpy'. Please install numpy and then try again to install HTSeq. I reinstalled numpy but that did not help. How should I continue? Thanks, Karen ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] tool_data_table_conf.xml entries for migrated tools
I've just upgraded a galaxy instance from the Nov 2013 release to the current. I had a few issues, mostly related to tool migration. 1) The entires in the tool_conf.xml file were not deleted as they should have been. This is a pain, but I can fix it (though it will take some time) 2) The repo directories in tool-data/toolshed.g2.bx.psu.edu/repos/devteam/ were never created (e.g. there was no bowtie2 directory there). I did uninstall and reinstall bowtie2 as a test, and the tool-data/toolshed.g2.bx.psu.edu/repos/devteam/bowtie2 directory was created. However, it seems that Galaxy now looks for the bowtie2 indicies in tool-data/toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/96d2e31a3938/bowtie2_indicies.loc instead of in tool-data/bowtie2_indicies.loc. How should I maintain these indicies going forward? Do I need to manually update the tool_data_table_conf.xml for each migrated tool? Thanks for any help or suggestions. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Updated Freebayes Wrapper - Bugs
Thanks for the major update of the Freebayes wrapper, excellent! I've run into two issues, however. 1) When using set allelic scope I get the following error: Fatal error: Exit code 1 () freebayes: unrecognized option `--min-repeat-length' did you mean --min-repeat-size ? 2) When using a vcf file as input I get the following error: Fatal error: Exit code 1 () open: No such file or directory [bgzf_check_bgzf] failed to open the file: input_variant_vcf.vcf.gz [tabix++] was bgzip used to compress this file? input_variant_vcf.vcf.gz I'd be happy to help resolve these. Is there a bitbucket repo somewhere to submit pull requests? -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Updated Freebayes Wrapper (0.9.14) - Bug Report
Thanks for the major update of the Freebayes wrapper, excellent! I've run into two issues, however. 1) When using set allelic scope I get the following error: Fatal error: Exit code 1 () freebayes: unrecognized option `--min-repeat-length' did you mean --min-repeat-size ? 2) When using a vcf file as input I get the following error: Fatal error: Exit code 1 () open: No such file or directory [bgzf_check_bgzf] failed to open the file: input_variant_vcf.vcf.gz [tabix++] was bgzip used to compress this file? input_variant_vcf.vcf.gz I'd be happy to help resolve these. Is there a bitbucket repo somewhere to submit pull requests? Also, I'm not sure if this is best sent to this list or put into Trello, etc. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Adding Reference Genomes, builds.txt vs manual_builds.txt
I noticed that the instructions at https://wiki.galaxyproject.org/Admin/DataIntegration direct admins to update the builds.txt file when adding a new reference genome to local Galaxy instances. Up to this point I had been actually modifying the manual_builds.txt file and then rebuilding the builds.txt file using the updateucsc.sh script. This allows me to continue to use the updateuscs.sh script without overwriting my changes and also generates a BUILD.len file in the chrom/ directory. It does require some scripting since one needs to add the chromosome lengths to manual_builds.txt, but this is relatively straight forward using an indexed fasta file (.fai) or similar. Perhaps a larger issue is that the manual_builds.txt file is currently under source control in mercurial (instead of being generated by a .sample version). So I have two questions: 1) What would be best practice for adding a new genome? Updating builds.txt or manual_builds.txt? 2) Can the manual_builds.txt file be generated from a sample and moved out of source control? (I'll add a Trello card if this seems reasonable). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Jobs stuck in new state - Data Library datasets to blame?
Recently, I updated our Galaxy instance to use two processes (one for web, the other as a job handler). This has been working well, except in a few cases. I've noticed that a number of jobs get stuck in the new status. In a number of cases, I've resolved the issue by downloading and uploading one of the input files and rerunning the job using the newly uploaded file. In at least one of these cases, the offending input file was one that was copied from a Data Library. Can anyone point me to something to look for in the database, etc. that would cause a job to think a dataset was not ready for use as a job input? I'd very much like to fix these datasets since having to re-upload data libraries would be very tedious. Thanks in advance. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Jobs stuck in new state - Data Library datasets to blame?
Thanks, that was the first thing I checked. However, restarting the handler didn't help. Downloading the offending data and re-uploading as a new data set and then rerunning using the new dataset as input did work. Also, all other jobs continued to run fine. Lance Kandalaft, Iyad wrote: I’ve had jobs get stuck in the new state when one of the handler servers crashes. If you have dedicated handlers, check to make sure they are still running. Restart the handler to see if the jobs get resumed automatically. *Iyad Kandalaft* *From:*galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] *On Behalf Of *Aaron Petkau *Sent:* Wednesday, October 01, 2014 5:32 PM *To:* Lance Parsons *Cc:* galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] Jobs stuck in new state - Data Library datasets to blame? Are you attempting to upload datasets to a Data Library, and then copy to a history and run jobs on them right away? I've run into issues before where if I attempt to run a job on a dataset in a library before it is finished being uploaded and processed, then the job gets stuck in a queued state and never executes. Aaron On Wed, Oct 1, 2014 at 2:51 PM, Lance Parsons lpars...@princeton.edu mailto:lpars...@princeton.edu wrote: Recently, I updated our Galaxy instance to use two processes (one for web, the other as a job handler). This has been working well, except in a few cases. I've noticed that a number of jobs get stuck in the new status. In a number of cases, I've resolved the issue by downloading and uploading one of the input files and rerunning the job using the newly uploaded file. In at least one of these cases, the offending input file was one that was copied from a Data Library. Can anyone point me to something to look for in the database, etc. that would cause a job to think a dataset was not ready for use as a job input? I'd very much like to fix these datasets since having to re-upload data libraries would be very tedious. Thanks in advance. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Apache proxy, SSL, and UCSC get data
I'm having trouble after setting up SSL on our Apache proxy for Galaxy. When I try to import data from UCSC, Galaxy tries to go to the http (non-ssl) url for UCSC and thus generates a security error in the browser and the request is blocked. I have added RequestHeader set X-URL-SCHEME https to the httpd.conf and that appears to be working. Is there something else I need to do? Should I patch the data_source tools to use https in the xml tool definitions? Thanks for any advice. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Jobs stuck in new state - Data Library datasets to blame?
I'v run into this same issue again (just with some other Data Library datasets). This time, there are a few users involved with quite a few stuck jobs. Does anyone have any advice on pushing these jobs through? Maybe even a pointer to the relevant code? I'm running latest_2014.08.11. Thanks in advance. Lance Lance Parsons wrote: Thanks, that was the first thing I checked. However, restarting the handler didn't help. Downloading the offending data and re-uploading as a new data set and then rerunning using the new dataset as input did work. Also, all other jobs continued to run fine. Lance Kandalaft, Iyad wrote: I’ve had jobs get stuck in the new state when one of the handler servers crashes. If you have dedicated handlers, check to make sure they are still running. Restart the handler to see if the jobs get resumed automatically. *Iyad Kandalaft* *From:*galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] *On Behalf Of *Aaron Petkau *Sent:* Wednesday, October 01, 2014 5:32 PM *To:* Lance Parsons *Cc:* galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] Jobs stuck in new state - Data Library datasets to blame? Are you attempting to upload datasets to a Data Library, and then copy to a history and run jobs on them right away? I've run into issues before where if I attempt to run a job on a dataset in a library before it is finished being uploaded and processed, then the job gets stuck in a queued state and never executes. Aaron On Wed, Oct 1, 2014 at 2:51 PM, Lance Parsons lpars...@princeton.edu mailto:lpars...@princeton.edu wrote: Recently, I updated our Galaxy instance to use two processes (one for web, the other as a job handler). This has been working well, except in a few cases. I've noticed that a number of jobs get stuck in the new status. In a number of cases, I've resolved the issue by downloading and uploading one of the input files and rerunning the job using the newly uploaded file. In at least one of these cases, the offending input file was one that was copied from a Data Library. Can anyone point me to something to look for in the database, etc. that would cause a job to think a dataset was not ready for use as a job input? I'd very much like to fix these datasets since having to re-upload data libraries would be very tedious. Thanks in advance. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Is release_2014.10.06 official?
I see that the tags release_2014.10.06 and latest_2014.10.06 were created some time ago, but there was never any announcement. Are these considered officially released? Should I upgrade my instance to this version? Most importantly, are there any release notes so I can see what, if anything, I need to do to prepare for this update? Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Jobs stuck in new state - Data Library datasets to blame?
Thanks for looking into this John. I'm pretty sure there are multiple reasons for stuck jobs. (In fact I'm running into another I'll discuss separately, likely in a Trello ticket). But it turns out in my most recent run-in with this issue, the problem turned out to be caused by datasets that were marked as deleted in the dataset table, but not marked as deleted in the history dataset association table, and thus were used as inputs to jobs. The following query fixed the stuck jobs: update dataset set deleted = 'f' and purgable = 'f' where id in (select distinct(d.id) from dataset d join history_dataset_association hda on d.id = hda.dataset_id join job_to_input_dataset jtid on hda.id = jtid.dataset_id join job j on jtid.job_id = j.id where d.deleted = 't' and hda.deleted = 'f' and j.state = 'new'); However, there seem to many instances where I find datasets marked as deleted but where the history dataset association is not marked as deleted. I'm wary of updating them all without knowing how they got set this way (or even if this is sometimes an appropriate state). Is this ever a valid state for a dataset? BTW: There is also a discussion at https://biostar.usegalaxy.org/p/9608/ about this. Lance John Chilton wrote: Hello Lance, I cannot think of a good way to rescue these jobs. If you are curious about the code where jobs are selected for execution - I would check out the job handler (lib/galaxy/jobs/handler.py) - see __monitor_step for instance. It seems like to prevent this from happening in the future - we should only allow copying datasets from libraries into histories if the the library dataset is in an 'OK' state (https://trello.com/c/0vxbP4El). -John On Thu, Nov 6, 2014 at 11:13 AM, Lance Parsonslpars...@princeton.edu wrote: I'v run into this same issue again (just with some other Data Library datasets). This time, there are a few users involved with quite a few stuck jobs. Does anyone have any advice on pushing these jobs through? Maybe even a pointer to the relevant code? I'm running latest_2014.08.11. Thanks in advance. Lance Lance Parsons wrote: Thanks, that was the first thing I checked. However, restarting the handler didn't help. Downloading the offending data and re-uploading as a new data set and then rerunning using the new dataset as input did work. Also, all other jobs continued to run fine. Lance Kandalaft, Iyad wrote: I’ve had jobs get stuck in the new state when one of the handler servers crashes. If you have dedicated handlers, check to make sure they are still running. Restart the handler to see if the jobs get resumed automatically. Iyad Kandalaft From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Aaron Petkau Sent: Wednesday, October 01, 2014 5:32 PM To: Lance Parsons Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Jobs stuck in new state - Data Library datasets to blame? Are you attempting to upload datasets to a Data Library, and then copy to a history and run jobs on them right away? I've run into issues before where if I attempt to run a job on a dataset in a library before it is finished being uploaded and processed, then the job gets stuck in a queued state and never executes. Aaron On Wed, Oct 1, 2014 at 2:51 PM, Lance Parsonslpars...@princeton.edu wrote: Recently, I updated our Galaxy instance to use two processes (one for web, the other as a job handler). This has been working well, except in a few cases. I've noticed that a number of jobs get stuck in the new status. In a number of cases, I've resolved the issue by downloading and uploading one of the input files and rerunning the job using the newly uploaded file. In at least one of these cases, the offending input file was one that was copied from a Data Library. Can anyone point me to something to look for in the database, etc. that would cause a job to think a dataset was not ready for use as a job input? I'd very much like to fix these datasets since having to re-upload data libraries would be very tedious. Thanks in advance. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please
