Re: [galaxy-dev] Configurable toolshed package installation to support tool-dependency-resolver-plugins
I think that the best approach for this is an additional ToolDependency status as John proposed a while back as this approach allows for the flexibility needed to support just about anything users may want to do. I haven't had a chance to look at this yet, and I'm currently working on a couple of high priority items. But I can probably take a looke at this probably within the next couple of weeks. If there is an example tool dependency packafge that you have that uses modules, it would really help me with this implementation. Of course, if you want to tak ethis John, it would be much appreciated as well. Greg Von Kuster On Nov 5, 2013, at 12:03 AM, John Chilton chil...@msi.umn.edu wrote: Simon, As you have probably noticed a new stable galaxy was released. It includes 95% of what we discussed including this implicit check to see if tool shed packages are enabled. Your help implementing, testing, and driving these changes was greatly appreciated! I couldn't however pull the trigger and mark packages resolved via modules as Installed - so they will still appear to be in an error state (though your check is in there and they won't attempt to be installed, they will be just marked as errors). The upshot is you can change this one line of code in your Galaxy instance to get the behavior you desire (patch attached). The reason I don't want to mark these packages as Installed is that I am worried about Galaxy deployments that maybe want to use modules are first but transition to tool shed packages down the road. I am unsure what will happen if things are marked as Installed even if no files corresponding to the installation exist. I think the state NEVER_INSTALLED may be preferable - but I need to understand more about what that means. For your own instance, if you are certainly committed to using modules and not using the tool shed - it should be easy to apply the above patch. Is this a fair compromise for the time being? Until I can resolve this last issue, the Trello card remains open, but it should now be quite trivial to modify Galaxy to get the behavior you desire and hopefully this can serve as a model for how others can hook in other dependency resolution mechanisms. I would be eager to hear how this experiment progresses and how you feel about the implementation. Thanks for your contributions, -John On Mon, Oct 14, 2013 at 8:33 PM, Guest, Simon simon.gu...@agresearch.co.nz wrote: At Mon, 14 Oct 2013 20:22:06 -0500, John Chilton wrote: Simon, Very cool! I have two concerns. Rather than adding a new configuration option I think I would prefer to just check the configured dependency resolvers and then infer from them if the tool shed will be used. The configuration option strikes me as having to configure the same thing twice, and this change would make your setup slightly easier. Do you have any objection to me reworking your patch to do this? On the other hand, perhaps it is made more clear to the deployer that they are definitely disabling tool dependency installations if they have to add the explicit option this way. Hi John, I have no problem with you reworking it in that way. There are two reasons I didn't do that myself: 1. I would have had to change the interface to the dependency resolvers somehow to support this query, and I wasn't sure that was a good thing. 2. I wanted to make it explicit that toolshed package installation was disabled in this case, as I thought that would make it more likely this change gets accepted into the mainline. Whichever way you Greg and Dave are happy with is OK by me. Actually, I like your implicit approach better, so hope that's the one that gets agreed. cheers, Simon modules_installed.patch ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Deleting history entries not working, javascript void link?
Hi all, I've just updated our main Galaxy to release_2013.11.04 but also see the following problem on galaxy-central, and on https://usegalaxy.org/ as well. (1) Go to any history with deleted files (2) Cog icon, tick Include Deleted datasets (3) Look at a deleted dataset in your history column, note the yellow Warning box with text: This dataset has been deleted. Click *here* to undelete it or *here* to immediately remove it from disk (4) Notice the first *here* link (undelete) has a sensible URL, e.g. as shown via your browser's status bar or right click to copy the URL. (5) Notice the second *here* (remove) has this as the URL: javascript:void(0) Unsurprisingly, the remove link has no effect as it is a JavaScript command to do nothing, and the user cannot therefore purge the file from their history. I've confirmed this on multiple browsers on multiple machines. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Question regarding walltime exceeded not being correctly reported via the WebUI
Hey Daniel, Thanks so much for the details problem report, it was very helpful. Reviewing the code there appears to be a bug in the PBS job runner - in some cases pbs_job_state.stop_job is never set but is attempted to be read. I don't have torque so I don't have a great test setup for this problem, any chance you can make the following changes for me and let me know if they work? Between the following two lines: log.error( '(%s/%s) PBS job failed: %s' % ( galaxy_job_id, job_id, JOB_EXIT_STATUS.get( int( status.exit_status ), 'Unknown error: %s' % status.exit_status ) ) ) self.work_queue.put( ( self.fail_job, pbs_job_state ) ) log.error( '(%s/%s) PBS job failed: %s' % ( galaxy_job_id, job_id, JOB_EXIT_STATUS.get( int( status.exit_status ), 'Unknown error: %s' % status.exit_status ) ) ) pbs_job_state.stop_job = False self.work_queue.put( ( self.fail_job, pbs_job_state ) ) And at the top of the file can you add a -11 option to the JOB_EXIT_STATUS to indicate a job timeout. I have attached a patch that would apply against the latest stable - it will probably will work against your branch as well. If you would rather not act as my QC layer, I can try to come up with a way to do some testing on my end :). Thanks again, -John On Mon, Nov 4, 2013 at 10:10 AM, Daniel Patrick Sullivan dansulli...@gmail.com wrote: Hi, Galaxy Developers, I have what I hops is somewhat of a basic question regarding Galaxy's interaction with a pbs job cluster and information reported via the webUI. Basically, in certain situations, the walltime of a specific job is exceeded. This is of course to be expected and all fine and understandeable. My problem is that the information is not being relayed back to the end user via the Galaxy web UI, which causes confusion in our Galaxy user community. Basically the Torque scheduler generates the following message when a walltime is exceeded: 11/04/2013 08:39:45;000d;PBS_Server.30621;Job;163.sctest.cri.uchicago.edu;preparing to send 'a' mail for job 163.sctest.cri.uchicago.edu to s.cri.gal...@crigalaxy-test.uchicago.edu (Job exceeded its walltime limit. Job was aborted 11/04/2013 08:39:45;0009;PBS_Server.30621;Job;163.sctest.cri.uchicago.edu;job exit status -11 handled Now, my problem is that this status -11 return code is not being correctly handled by Galaxy. What happens is that Galaxy throws an exception, specificially: 10.135.217.178 - - [04/Nov/2013:08:39:42 -0500] GET /api/histories/90240358ebde1489 HTTP/1.1 200 - https://crigalaxy-test.uchicago.edu/history; Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0 galaxy.jobs.runners.pbs DEBUG 2013-11-04 08:39:46,137 (2150/163.sctest.cri.uchicago.edu) PBS job state changed from R to C galaxy.jobs.runners.pbs ERROR 2013-11-04 08:39:46,139 (2150/163.sctest.cri.uchicago.edu) PBS job failed: Unknown error: -11 galaxy.jobs.runners ERROR 2013-11-04 08:39:46,139 (unknown) Unhandled exception calling fail_job Traceback (most recent call last): File /group/galaxy_test/galaxy-dist/lib/galaxy/jobs/runners/__init__.py, line 60, in run_next method(arg) File /group/galaxy_test/galaxy-dist/lib/galaxy/jobs/runners/pbs.py, line 561, in fail_job if pbs_job_state.stop_job: AttributeError: 'AsynchronousJobState' object has no attribute 'stop_job' After this exception occurs, the Galaxy job status via the Web UI is still reported as Job is currently running. It appears that the job will remain in this state (from the end users perspective) indefinitely. Has anybody seen this issue before? I noticed that return code -11 does not exist in /group/galaxy_test/galaxy-dist/lib/galaxy/jobs/runners/pbs.py under the JOB_EXIT_STATUS dictionary. I tried adding an entry for this, however when I do the exception changes to: galaxy.jobs.runners.pbs ERROR 2013-11-04 10:02:17,274 (2151/164.sctest.cri.uchicago.edu) PBS job failed: job walltime exceeded galaxy.jobs.runners ERROR 2013-11-04 10:02:17,275 (unknown) Unhandled exception calling fail_job Traceback (most recent call last): File /group/galaxy_test/galaxy-dist/lib/galaxy/jobs/runners/__init__.py, line 60, in run_next method(arg) File /group/galaxy_test/galaxy-dist/lib/galaxy/jobs/runners/pbs.py, line 562, in fail_job if pbs_job_state.stop_job: AttributeError: 'AsynchronousJobState' object has no attribute 'stop_job' I am wondering if this is a bug or if it is just because I am using a newer version of TORQUE (I am using TORQUE 4.2.2). In terms of Galaxy, I am using: [s.cri.galaxy@crigalaxy-test galaxy-dist]$ hg parents changeset: 10408:6822f41bc9bb branch: stable parent: 10393:d05bf67aefa6 user:Dave Bouvier d...@bx.psu.edu date:Mon Aug 19 13:06:17 2013 -0400 summary: Fix for case where running functional tests might overwrite certain files in database/files.
Re: [galaxy-dev] Security vulnerability in Galaxy filtering tools
This seems to happen often e.g. http://wiki.galaxyproject.org/DevNewsBriefs/2012_10_23#Compute_Tool_Security_Fix a) are there general guidelines in the wiki on how to avoid these problems when creating tools? b) is there a way to check automatically if all input fields are correctly escaped in a tool? A search for security in the wiki brings up: • Admin/Data Libraries/Library Security 0.0k - rev: 1 (current) last modified: 2013-01-02 23:54:33 • Admin/DataLibraries/LibrarySecurity 19.2k - rev: 4 (current) last modified: 2013-01-03 00:12:36 • HelpOnConfiguration/SecurityPolicy 1.9k - rev: 1 (current) last modified: 0 • Learn/Security Features 7.0k - rev: 3 (current) last modified: 2011-09-13 16:52:08 • News/2013_04_08_Galaxy_Security_Release 1.3k - rev: 3 (current) last modified: 2013-04-08 16:56:41 escape does not bring up anything. thank you very much, ido On Nov 5, 2013, at 12:45 AM, Nate Coraor n...@bx.psu.edu wrote: A security vulnerability was recently discovered by John Chilton with Galaxy's Filter data on any column using simple expressions and Filter on ambiguities in polymorphism datasets tools that can allow for arbitrary execution of code on the command line. The fix for these tools has been committed to the Galaxy source. The timing of this commit coincides with the next Galaxy stable release (which has also been pushed out today). To apply the fix and simultaneously update to the new Galaxy stable release, ensure you are on the stable branch and upgrade to the latest changeset: % hg branch stable % hg pull -u For Galaxy installations that administrators are not yet ready to upgrade to the latest release, there are three workarounds. First, for Galaxy installations running on a relatively new version of the stable release (e.g. release_2013.08.12), Galaxy can be updated to the specific changeset that that contains the fix. This will include all of the stable (non-feature) commits that have been accumulated since the 8/12 release plus any new features included with (and prior to) the 8/12 release, but without all of the new features included in the 11/4 release. Ensure you are on the stable branch and then upgrade to the specific changeset: % hg pull -u -r e094c73fed4d Second, the patch can be downloaded and applied manually: % wget -o security.patch https://bitbucket.org/galaxy/galaxy-central/commits/e094c73fed4dc66b589932edb83412cb8b827cd3raw/ and then: % hg patch security.patch or: % patch -p1 security.patch Third, the tools can be completely disabled by removing them from the tool configuration file (by default, tool_conf.xml) and restarting all Galaxy server processes. The relevant lines in tool_conf.xml are: tool file=stats/dna_filtering.xml / tool file=stats/filtering.xml / The full 11/4 Galaxy Distribution News Brief will be available later today and will contain details of changes since the last release. --nate Galaxy Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] latest galaxy-central version
Hello, Robert I'm having difficulty reproducing this with a fresh install of galaxy_central:default:11226:c67b9518c1e0. Is this an intermittent error or does it happen reliably each time with the steps above? Is it still the same javascript error you mentioned above? I'll investigate further. On Mon, Nov 4, 2013 at 7:29 PM, Robert Baertsch baert...@soe.ucsc.eduwrote: I updated to the stable release and reproduced the issue. Step to reproduce 1. go to admin 2. Manage data librarieshttp://su2c-dev.ucsc.edu:8383/library_admin/browse_libraries 3. add dataset 4. select Upload files from filesystem paths 5. paste full path to any bam file. 6. leave defaults: auto-detect and copy files into galaxy 7. select role to restrict access 8. click upload to start Screen shows strange formatting and Job is running for Bam file. python /data/galaxy-central/tools/data_source/upload.py /data/galaxy-central /data/galaxy-central/database/tmp/tmpJoasJl /data/galaxy-central/database/tmp/tmpzZO8t1 8877:/data/galaxy-central/database/job_working_directory/004/4548/dataset_8877_files:/data/galaxy-central/database/files/008/dataset_8877.dat If I do a firefox refresh and go back to the library, the formatting is normal. I'm assuming the fix is to just render the page before waiting for the job to complete. -Robert On Nov 4, 2013, at 12:45 PM, Martin Čech mar...@bx.psu.edu wrote: Hello, I have also seen some of these errors while developing libraries. The library code is not in central however it might be related to recent changes to the history panel. Carl Eberhard might now more, adding him to the conversation. --Marten On Mon, Nov 4, 2013 at 2:45 PM, Robert Baertsch baert...@soe.ucsc.eduwrote: It keeps doing posts, and I'm not seeing any new errors. POST http://su2c-dev.ucsc.edu:8383/library_common/library_item_updates 200 OK 121ms When I did a browser refresh, I got the following javascript error: (I am logged in) Galaxy.currUser is undefined on Line 631 in history-panel.js When I opened the data library where the bam file was copying, everything is rendered ok. It seems the browser refresh fixed things. -Robert On Nov 4, 2013, at 11:14 AM, James Taylor ja...@jamestaylor.org wrote: Robert, I'm not sure what is going on here, other than that the javascript that converts buttons into dropdown menus has not fired. Are there any javascript errors? Marten is working on rewriting libraries, and we will be eliminating the progressive loading popupmenus for something much more efficient, but this also might indicate a bug so let us know if there is anything odd in the console. -- James Taylor, Associate Professor, Biology/CS, Emory University On Mon, Nov 4, 2013 at 1:58 PM, Robert Baertsch baert...@soe.ucsc.eduwrote: HI James, I just pulled in the latest code to see how you changed from iframe to divs. Very exciting update. I tried importing a bam file into the library using the admin tool that reads the file directly from a path on the server. During the copy operation, the screen gets into a weird state. ps -ef shows these two processes running: python /data/galaxy-central/tools/data_source/upload.py /data/galaxy-central /data/galaxy-central/database/tmp/tmpywxnFE /data/galaxy-central/database/tmp/tmpXkEvSY 8876:/data/galaxy-central/database/job_working_directory/004/4547/dataset_8876_files:/data/galaxy-central/database/files/008/dataset_8876.dat samtools sort /data/galaxy-central/database/files/008/dataset_8876.dat /tmp/tmpNbcggh/sorted Do you finish rendering the screen after the copy operation is complete? -Robert Screen Shot 2013-11-04 at 10.53.02 AM.png ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Deleting history entries not working, javascript void link?
Hello, Peter. Thanks for the report. There was an error in the javascript error handling of purge_async which is now fixed in stable:11227:151b7d3b2f1b. In the instances you mention, you should now see an error message in the history 'Unable to purge this dataset': This is because of two possible reasons: - the setting 'allow_user_dataset_purge' in universe_wsgi.ini is 'False' (or commented out) - there was an error in the call to purge_async on the server The 'href=javascript:void(0);' is a common idiom for creating links/anchors where the logic is handled on the client side. It tells the browser that, although the element has all the other properties of a link, it should not load it like another page and the click event will be handled by javascript on the current page. In the default branch, the error reporting has been (hopefully) improved to display appropriate messages for failed purging as well as better reporting overall. Sorry for the inconvenience. On Tue, Nov 5, 2013 at 7:02 AM, Peter Cock p.j.a.c...@googlemail.comwrote: Hi all, I've just updated our main Galaxy to release_2013.11.04 but also see the following problem on galaxy-central, and on https://usegalaxy.org/ as well. (1) Go to any history with deleted files (2) Cog icon, tick Include Deleted datasets (3) Look at a deleted dataset in your history column, note the yellow Warning box with text: This dataset has been deleted. Click *here* to undelete it or *here* to immediately remove it from disk (4) Notice the first *here* link (undelete) has a sensible URL, e.g. as shown via your browser's status bar or right click to copy the URL. (5) Notice the second *here* (remove) has this as the URL: javascript:void(0) Unsurprisingly, the remove link has no effect as it is a JavaScript command to do nothing, and the user cannot therefore purge the file from their history. I've confirmed this on multiple browsers on multiple machines. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] set_environment_for_install problem, seeking for ideas
Hi Greg, I'm right now in implementing a setup_perl_environment and stumbled about a tricky problem (that is not only related to perl but also for ruby, python and R). The Problem: Lets assume a perl package (A) requires a xml parser written in C/C++ (Z). (Z) is a dependency that I can import but as far as I can see there is no way to call set_environment_for_install before setup_perl_environment, because setup_perl_environment defines an installation type. I would like to discuss that issue to get a few ideas. I can think about these solutions: - hackish solution: I can call install_environment.add_env_shell_file_paths( action_dict[ 'env_shell_file_paths' ] ) inside of the setup_*_environment path and remove it from action type afterwards - import all env.sh variables from every (package) definition. Regardless if set_environment_for_install is set or not. I must admit, I do not understand why set_environment_for_install is actually needed. I think we can assume that if I specify a package name=R_3_0_1 version=3.0.1 repository name=package_r_3_0_1 owner=iuc prior_installation_required=True / /package I want the ENV vars sourced. Furthermore, that can solve an other issue: Namely, the need of ENV vars from a package definition in the same file. Lets imagine package P has dependency D and you want to download compile both in one tool_dependencies.xml file. You can either do it in one package definition or you need to split them up in 2 tool_dependencies.xml files, rigth? Maybe we can just assume a strict order in a tool_dependencies.xml file, where every ENV vars are sourced for the following one? Does that make sense? Any other ideas? Thanks, Bjoern ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Security vulnerability in Galaxy filtering tools
Hi Ido, Thanks for the feedback. Replies below. On Nov 5, 2013, at 9:54 AM, Ido Tamir wrote: This seems to happen often e.g. http://wiki.galaxyproject.org/DevNewsBriefs/2012_10_23#Compute_Tool_Security_Fix I'm not sure I'd agree that it's often - we've had 4 or 5 vulnerabilities over the life of the project. 2 allowed arbitrary code execution, the others were less severe. a) are there general guidelines in the wiki on how to avoid these problems when creating tools? The guidelines for writing a Galaxy tool are no different from best practices for writing secure code. In specific for this vulnerability, execution of user input should be handled with extreme care, and this tool had some gaps in its input validation and sanitization. For what it's worth, the filter tool (on which the other vulnerable tools were based) is one of the few tools surviving from the very early days of Galaxy, and would not be implemented the same way if written today. b) is there a way to check automatically if all input fields are correctly escaped in a tool? I am not sure how Galaxy could do this. Galaxy sanitizes the command line so that input fields passed to a tool as command line arguments cannot be crafted to exploit the shell's parsing rules. What the tool itself does with its inputs are out of Galaxy's control. --nate A search for security in the wiki brings up: • Admin/Data Libraries/Library Security 0.0k - rev: 1 (current) last modified: 2013-01-02 23:54:33 • Admin/DataLibraries/LibrarySecurity 19.2k - rev: 4 (current) last modified: 2013-01-03 00:12:36 • HelpOnConfiguration/SecurityPolicy 1.9k - rev: 1 (current) last modified: 0 • Learn/Security Features 7.0k - rev: 3 (current) last modified: 2011-09-13 16:52:08 • News/2013_04_08_Galaxy_Security_Release 1.3k - rev: 3 (current) last modified: 2013-04-08 16:56:41 escape does not bring up anything. thank you very much, ido On Nov 5, 2013, at 12:45 AM, Nate Coraor n...@bx.psu.edu wrote: A security vulnerability was recently discovered by John Chilton with Galaxy's Filter data on any column using simple expressions and Filter on ambiguities in polymorphism datasets tools that can allow for arbitrary execution of code on the command line. The fix for these tools has been committed to the Galaxy source. The timing of this commit coincides with the next Galaxy stable release (which has also been pushed out today). To apply the fix and simultaneously update to the new Galaxy stable release, ensure you are on the stable branch and upgrade to the latest changeset: % hg branch stable % hg pull -u For Galaxy installations that administrators are not yet ready to upgrade to the latest release, there are three workarounds. First, for Galaxy installations running on a relatively new version of the stable release (e.g. release_2013.08.12), Galaxy can be updated to the specific changeset that that contains the fix. This will include all of the stable (non-feature) commits that have been accumulated since the 8/12 release plus any new features included with (and prior to) the 8/12 release, but without all of the new features included in the 11/4 release. Ensure you are on the stable branch and then upgrade to the specific changeset: % hg pull -u -r e094c73fed4d Second, the patch can be downloaded and applied manually: % wget -o security.patch https://bitbucket.org/galaxy/galaxy-central/commits/e094c73fed4dc66b589932edb83412cb8b827cd3raw/ and then: % hg patch security.patch or: % patch -p1 security.patch Third, the tools can be completely disabled by removing them from the tool configuration file (by default, tool_conf.xml) and restarting all Galaxy server processes. The relevant lines in tool_conf.xml are: tool file=stats/dna_filtering.xml / tool file=stats/filtering.xml / The full 11/4 Galaxy Distribution News Brief will be available later today and will contain details of changes since the last release. --nate Galaxy Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Security vulnerability in Galaxy filtering tools
On Nov 5, 2013, at 6:28 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Ido, Thanks for the feedback. Replies below. On Nov 5, 2013, at 9:54 AM, Ido Tamir wrote: This seems to happen often e.g. http://wiki.galaxyproject.org/DevNewsBriefs/2012_10_23#Compute_Tool_Security_Fix I'm not sure I'd agree that it's often - we've had 4 or 5 vulnerabilities over the life of the project. 2 allowed arbitrary code execution, the others were less severe. But these were written by experts, not by people like me, that don't know what the galaxy framework really does/does not do with the input, so I guess I make many more mistakes. a) are there general guidelines in the wiki on how to avoid these problems when creating tools? The guidelines for writing a Galaxy tool are no different from best practices for writing secure code. In specific for this vulnerability, execution of user input should be handled with extreme care, and this tool had some gaps in its input validation and sanitization. For what it's worth, the filter tool (on which the other vulnerable tools were based) is one of the few tools surviving from the very early days of Galaxy, and would not be implemented the same way if written today. I think it would be nice to have a small outline on the wiki of what galaxy does with the input data and how it could affect a tool. What sanitisation is there by default so I don't have to worry about it, but what could happen I if I don't care to check/remove sanitise ' | or ..., maybe with examples. b) is there a way to check automatically if all input fields are correctly escaped in a tool? I am not sure how Galaxy could do this. Galaxy sanitizes the command line so that input fields passed to a tool as command line arguments cannot be crafted to exploit the shell's parsing rules. Thats good best, ido What the tool itself does with its inputs are out of Galaxy's control. --nate A search for security in the wiki brings up: • Admin/Data Libraries/Library Security 0.0k - rev: 1 (current) last modified: 2013-01-02 23:54:33 • Admin/DataLibraries/LibrarySecurity 19.2k - rev: 4 (current) last modified: 2013-01-03 00:12:36 • HelpOnConfiguration/SecurityPolicy 1.9k - rev: 1 (current) last modified: 0 • Learn/Security Features 7.0k - rev: 3 (current) last modified: 2011-09-13 16:52:08 • News/2013_04_08_Galaxy_Security_Release 1.3k - rev: 3 (current) last modified: 2013-04-08 16:56:41 escape does not bring up anything. thank you very much, ido On Nov 5, 2013, at 12:45 AM, Nate Coraor n...@bx.psu.edu wrote: A security vulnerability was recently discovered by John Chilton with Galaxy's Filter data on any column using simple expressions and Filter on ambiguities in polymorphism datasets tools that can allow for arbitrary execution of code on the command line. The fix for these tools has been committed to the Galaxy source. The timing of this commit coincides with the next Galaxy stable release (which has also been pushed out today). To apply the fix and simultaneously update to the new Galaxy stable release, ensure you are on the stable branch and upgrade to the latest changeset: % hg branch stable % hg pull -u For Galaxy installations that administrators are not yet ready to upgrade to the latest release, there are three workarounds. First, for Galaxy installations running on a relatively new version of the stable release (e.g. release_2013.08.12), Galaxy can be updated to the specific changeset that that contains the fix. This will include all of the stable (non-feature) commits that have been accumulated since the 8/12 release plus any new features included with (and prior to) the 8/12 release, but without all of the new features included in the 11/4 release. Ensure you are on the stable branch and then upgrade to the specific changeset: % hg pull -u -r e094c73fed4d Second, the patch can be downloaded and applied manually: % wget -o security.patch https://bitbucket.org/galaxy/galaxy-central/commits/e094c73fed4dc66b589932edb83412cb8b827cd3raw/ and then: % hg patch security.patch or: % patch -p1 security.patch Third, the tools can be completely disabled by removing them from the tool configuration file (by default, tool_conf.xml) and restarting all Galaxy server processes. The relevant lines in tool_conf.xml are: tool file=stats/dna_filtering.xml / tool file=stats/filtering.xml / The full 11/4 Galaxy Distribution News Brief will be available later today and will contain details of changes since the last release. --nate Galaxy Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists
[galaxy-dev] Egg distribution issue
Hi Dev Team,
I have run into an egg distribution issue. I had performed an upgrade late
last month and am experiencing issues fetching eggs. I am getting the
following error message:
galaxy@jaxgalaxydev01:/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist python
./scripts/fetch_eggs.py
Fetched
http://eggs.galaxyproject.org/sqlalchemy_migrate/sqlalchemy_migrate-0.7.2-py2.6.egg
One of Galaxy's managed eggs depends on something which is missing, this is
almost certainly a bug in the egg distribution.
Dependency sqlalchemy-migrate requires setuptools
Traceback (most recent call last):
File ./scripts/fetch_eggs.py, line 37, in module
c.resolve() # Only fetch eggs required by the config
File
/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/lib/galaxy/eggs/__init__.py,
line 345, in resolve
egg.resolve()
File
/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/lib/galaxy/eggs/__init__.py,
line 168, in resolve
dists = pkg_resources.working_set.resolve( (
self.distribution.as_requirement(), ), env, self.fetch )
File
/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist-jax/galaxy-dist-jax/lib/pkg_resources.py,
line 569, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (sqlalchemy-migrate 0.7.2
(/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.6.egg),
Requirement.parse('setuptools'))
Could someone point me in the right direction?
Thanks,
Iry
__
Iry T. Witham
Scientific Applications Administrator
Scientific Computing Group
Computational Sciences Dept.
The Jackson Laboratory
600 Main Street
Bar Harbor, ME 04609
Phone: 207-288-6744
email: iry.wit...@jax.org
[cid:3302E28C-F24A-4AF1-97B8-A33A6CD618F6]
The information in this email, including attachments, may be confidential and
is intended solely for the addressee(s). If you believe you received this email
by mistake, please notify the sender by return email as soon as possible.
inline: 372D007A-1B00-4668-BA6B-F0527C1F24BE[34][3].png___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] set_environment_for_install problem, seeking for ideas
Hello Bjoern, On Nov 5, 2013, at 12:13 PM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Hi Greg, I'm right now in implementing a setup_perl_environment and stumbled about a tricky problem (that is not only related to perl but also for ruby, python and R). The Problem: Lets assume a perl package (A) requires a xml parser written in C/C++ (Z). (Z) is a dependency that I can import but as far as I can see there is no way to call set_environment_for_install before setup_perl_environment, because setup_perl_environment defines an installation type. The above is fairly difficult to understand - can you provide an actual xml recipe that provides some context? I would like to discuss that issue to get a few ideas. I can think about these solutions: - hackish solution: I can call install_environment.add_env_shell_file_paths( action_dict[ 'env_shell_file_paths' ] ) inside of the setup_*_environment path and remove it from action type afterwards Again, it's difficult to provide good feedback on the above approach without an example recipe. However, your hackish solution term probably means it is not ideal. ;) - import all env.sh variables from every (package) definition. Regardless if set_environment_for_install is set or not. I don't think the above approach would be ideal. It seems that it could fairly easily create conflicting environment variables within a single installation, so the latest value for an environment variable may not be what is expected. I must admit, I do not understand why set_environment_for_install is actually needed. I think we can assume that if I specify a package name=R_3_0_1 version=3.0.1 repository name=package_r_3_0_1 owner=iuc prior_installation_required=True / /package I want the ENV vars sourced. Hmmm…so you are saying that you want the be able to define the above package tag set inside of an actions tag set and have everything work? I think this may cause problems because I believe the set_environment_for_install tag set restricts activity to only the time when a dependent repository will be using the defined environment from the required repository in order to compile one or more of it's dependencies. Eliminating this restriction may cause problems after compilation. ALthough I cannot state this as a definite fact. Furthermore, that can solve an other issue: Namely, the need of ENV vars from a package definition in the same file. Lets imagine package P has dependency D and you want to download compile both in one tool_dependencies.xml file. You can either do it in one package definition or you need to split them up in 2 tool_dependencies.xml files, rigth? Maybe we can just assume a strict order in a tool_dependencies.xml file, where every ENV vars are sourced for the following one? Does that make sense? It may make sense, but without an example it's diffiecult to answer this for sure. Can you provide some xml recipes that use your different proposals? Any other ideas? Not yet, but possibly after your next response. ;) Thanks, Bjoern Thanks! Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] set_environment_for_install problem, seeking for ideas
Hi Greg, Hello Bjoern, On Nov 5, 2013, at 12:13 PM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Hi Greg, I'm right now in implementing a setup_perl_environment and stumbled about a tricky problem (that is not only related to perl but also for ruby, python and R). The Problem: Lets assume a perl package (A) requires a xml parser written in C/C++ (Z). (Z) is a dependency that I can import but as far as I can see there is no way to call set_environment_for_install before setup_perl_environment, because setup_perl_environment defines an installation type. The above is fairly difficult to understand - can you provide an actual xml recipe that provides some context? Attached, please see a detailed explanation at the bottom. I would like to discuss that issue to get a few ideas. I can think about these solutions: - hackish solution: I can call install_environment.add_env_shell_file_paths( action_dict[ 'env_shell_file_paths' ] ) inside of the setup_*_environment path and remove it from action type afterwards Again, it's difficult to provide good feedback on the above approach without an example recipe. However, your hackish solution term probably means it is not ideal. ;) :) - import all env.sh variables from every (package) definition. Regardless if set_environment_for_install is set or not. I don't think the above approach would be ideal. It seems that it could fairly easily create conflicting environment variables within a single installation, so the latest value for an environment variable may not be what is expected. What means conflicting ENV vars, I only can imagine multiple set_to that overwrite each other. append_to and prepend_to should be save or? I must admit, I do not understand why set_environment_for_install is actually needed. I think we can assume that if I specify a package name=R_3_0_1 version=3.0.1 repository name=package_r_3_0_1 owner=iuc prior_installation_required=True / /package I want the ENV vars sourced. Hmmm…so you are saying that you want the be able to define the above package tag set inside of an actions tag set and have everything work? Oh no, I mean just have it as package like it is but source the env.sh file for every other package set automatically. So you do not need set_environment_for_install. In the attached example: package name=expat version=2.1.0 repository changeset_revision=8fc96166cddd name=package_expat_2_1 owner=iuc prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package Is not imported with set_environment_for_install so its actually useless (now). But the env.sh needs to be sourced during the setup_perl_environment part. I think this may cause problems because I believe the set_environment_for_install tag set restricts activity to only the time when a dependent repository will be using the defined environment from the required repository in order to compile one or more of it's dependencies. Eliminating this restriction may cause problems after compilation. ALthough I cannot state this as a definite fact. Furthermore, that can solve an other issue: Namely, the need of ENV vars from a package definition in the same file. Lets imagine package P has dependency D and you want to download compile both in one tool_dependencies.xml file. You can either do it in one package definition or you need to split them up in 2 tool_dependencies.xml files, rigth? Maybe we can just assume a strict order in a tool_dependencies.xml file, where every ENV vars are sourced for the following one? Does that make sense? It may make sense, but without an example it's diffiecult to answer this for sure. Can you provide some xml recipes that use your different proposals? Sure, attached. Its quite complicated. - TPP needs libgd to compile C-Code (no problem). - TPP needs some perl libs and perl - setup_perl_environment (at runtime) - no problem until one of these perl packages (here XML-Parser) needs a C library (expat) - I don't see how to source expat during setup_perl_environment - TPP needs perl (at compile time) ... It would be more readable or logical to separate this recipe into two parts: TPP and Perl libraries Something like that: package name=trans_proteomic_pipeline_perl_libs version=4.6.3 . set PERL5LIBS /package package name=trans_proteomic_pipeline version=4.6.3 Here I need the PERL5LIBS /package - I don't see any way to get the PERL5LIBS from the perl libraries into a separate package section which tries to compile TPP. Any other ideas? Not yet, but possibly after your next response. ;) :) Here we go! Thanks Greg! Bjoern Thanks, Bjoern Thanks! Greg Von Kuster tool_dependencies.xml Description: XML document
Re: [galaxy-dev] set_environment_for_install problem, seeking for ideas
Thanks Björn, We'll have to digest this a bit and we'll get back to you. Greg Von Kuster On Nov 5, 2013, at 4:35 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Greg, Hello Bjoern, On Nov 5, 2013, at 12:13 PM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Hi Greg, I'm right now in implementing a setup_perl_environment and stumbled about a tricky problem (that is not only related to perl but also for ruby, python and R). The Problem: Lets assume a perl package (A) requires a xml parser written in C/C++ (Z). (Z) is a dependency that I can import but as far as I can see there is no way to call set_environment_for_install before setup_perl_environment, because setup_perl_environment defines an installation type. The above is fairly difficult to understand - can you provide an actual xml recipe that provides some context? Attached, please see a detailed explanation at the bottom. I would like to discuss that issue to get a few ideas. I can think about these solutions: - hackish solution: I can call install_environment.add_env_shell_file_paths( action_dict[ 'env_shell_file_paths' ] ) inside of the setup_*_environment path and remove it from action type afterwards Again, it's difficult to provide good feedback on the above approach without an example recipe. However, your hackish solution term probably means it is not ideal. ;) :) - import all env.sh variables from every (package) definition. Regardless if set_environment_for_install is set or not. I don't think the above approach would be ideal. It seems that it could fairly easily create conflicting environment variables within a single installation, so the latest value for an environment variable may not be what is expected. What means conflicting ENV vars, I only can imagine multiple set_to that overwrite each other. append_to and prepend_to should be save or? I must admit, I do not understand why set_environment_for_install is actually needed. I think we can assume that if I specify a package name=R_3_0_1 version=3.0.1 repository name=package_r_3_0_1 owner=iuc prior_installation_required=True / /package I want the ENV vars sourced. Hmmm…so you are saying that you want the be able to define the above package tag set inside of an actions tag set and have everything work? Oh no, I mean just have it as package like it is but source the env.sh file for every other package set automatically. So you do not need set_environment_for_install. In the attached example: package name=expat version=2.1.0 repository changeset_revision=8fc96166cddd name=package_expat_2_1 owner=iuc prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package Is not imported with set_environment_for_install so its actually useless (now). But the env.sh needs to be sourced during the setup_perl_environment part. I think this may cause problems because I believe the set_environment_for_install tag set restricts activity to only the time when a dependent repository will be using the defined environment from the required repository in order to compile one or more of it's dependencies. Eliminating this restriction may cause problems after compilation. ALthough I cannot state this as a definite fact. Furthermore, that can solve an other issue: Namely, the need of ENV vars from a package definition in the same file. Lets imagine package P has dependency D and you want to download compile both in one tool_dependencies.xml file. You can either do it in one package definition or you need to split them up in 2 tool_dependencies.xml files, rigth? Maybe we can just assume a strict order in a tool_dependencies.xml file, where every ENV vars are sourced for the following one? Does that make sense? It may make sense, but without an example it's diffiecult to answer this for sure. Can you provide some xml recipes that use your different proposals? Sure, attached. Its quite complicated. - TPP needs libgd to compile C-Code (no problem). - TPP needs some perl libs and perl - setup_perl_environment (at runtime) - no problem until one of these perl packages (here XML-Parser) needs a C library (expat) - I don't see how to source expat during setup_perl_environment - TPP needs perl (at compile time) ... It would be more readable or logical to separate this recipe into two parts: TPP and Perl libraries Something like that: package name=trans_proteomic_pipeline_perl_libs version=4.6.3 . set PERL5LIBS /package package name=trans_proteomic_pipeline version=4.6.3 Here I need the PERL5LIBS /package - I don't see any way to get the PERL5LIBS from the perl libraries into a separate package section which tries to compile TPP.
Re: [galaxy-dev] set_environment_for_install problem, seeking for ideas
Perl complicates things, TPP complicates things greatly. Bjoern, can I ask you if this hypothetical exhibits the same problem and can be used to reason about these things more easily and drive a test implementation. So right now, Galaxy has setup_virtualenv which will build and install Python packages in a virtual environment. However, some Python packages have C library dependencies that could prevent them from being installed. As a specific example - take PyTables (install via pip install tables) - which is a package for managing hierarchical datasets. If you try to install this with pip the way Galaxy will - it will fail if you do not have libhdf5 installed. So at a high-level, it would be nice if the tool shed had a libhdf5 definition and the dependencies file had some mechanism for declaring libhdf5 should be installed before a setup_virtualenv containing tables and its environment configured properly so the pip install succeeds (maybe just LD_LIBRARY_PATH needs to be set). -John On Tue, Nov 5, 2013 at 3:35 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Greg, Hello Bjoern, On Nov 5, 2013, at 12:13 PM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Hi Greg, I'm right now in implementing a setup_perl_environment and stumbled about a tricky problem (that is not only related to perl but also for ruby, python and R). The Problem: Lets assume a perl package (A) requires a xml parser written in C/C++ (Z). (Z) is a dependency that I can import but as far as I can see there is no way to call set_environment_for_install before setup_perl_environment, because setup_perl_environment defines an installation type. The above is fairly difficult to understand - can you provide an actual xml recipe that provides some context? Attached, please see a detailed explanation at the bottom. I would like to discuss that issue to get a few ideas. I can think about these solutions: - hackish solution: I can call install_environment.add_env_shell_file_paths( action_dict[ 'env_shell_file_paths' ] ) inside of the setup_*_environment path and remove it from action type afterwards Again, it's difficult to provide good feedback on the above approach without an example recipe. However, your hackish solution term probably means it is not ideal. ;) :) - import all env.sh variables from every (package) definition. Regardless if set_environment_for_install is set or not. I don't think the above approach would be ideal. It seems that it could fairly easily create conflicting environment variables within a single installation, so the latest value for an environment variable may not be what is expected. What means conflicting ENV vars, I only can imagine multiple set_to that overwrite each other. append_to and prepend_to should be save or? I must admit, I do not understand why set_environment_for_install is actually needed. I think we can assume that if I specify a package name=R_3_0_1 version=3.0.1 repository name=package_r_3_0_1 owner=iuc prior_installation_required=True / /package I want the ENV vars sourced. Hmmm…so you are saying that you want the be able to define the above package tag set inside of an actions tag set and have everything work? Oh no, I mean just have it as package like it is but source the env.sh file for every other package set automatically. So you do not need set_environment_for_install. In the attached example: package name=expat version=2.1.0 repository changeset_revision=8fc96166cddd name=package_expat_2_1 owner=iuc prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package Is not imported with set_environment_for_install so its actually useless (now). But the env.sh needs to be sourced during the setup_perl_environment part. I think this may cause problems because I believe the set_environment_for_install tag set restricts activity to only the time when a dependent repository will be using the defined environment from the required repository in order to compile one or more of it's dependencies. Eliminating this restriction may cause problems after compilation. ALthough I cannot state this as a definite fact. Furthermore, that can solve an other issue: Namely, the need of ENV vars from a package definition in the same file. Lets imagine package P has dependency D and you want to download compile both in one tool_dependencies.xml file. You can either do it in one package definition or you need to split them up in 2 tool_dependencies.xml files, rigth? Maybe we can just assume a strict order in a tool_dependencies.xml file, where every ENV vars are sourced for the following one? Does that make sense? It may make sense, but without an example it's diffiecult to answer this for sure. Can you provide some xml recipes that use your different proposals? Sure, attached. Its quite complicated. - TPP needs libgd to
[galaxy-dev] Installing Galaxy behind an Apache proxy using mod_auth_cas for user auth
Dear all, I would like to set up a local Galaxy instance behind an Apache server with our local CAS for authentication. It would be great if you could give me a hint for the httpd.conf. I have the problem that after authenticating against CAS in the browser, I get following error message and REMOTE_USER doesn't seem to be in the HTTP header for Galaxy (I can see the REMOTE_USER in the access_log of Apache but not any more in paster.log of Galaxy). Access to Galaxy is denied Galaxy is configured to authenticate users via an external method (such as HTTP authentication in Apache), but a username was not provided by the upstream (proxy) server. This is generally due to a misconfiguration in the upstream server. I know that the same question was already asked in the following post but I haven't seen an option to extend the post and I haven't found an answer. http://dev.list.galaxyproject.org/Installing-Galaxy-behind-an-Apache-proxy-using-mod-auth-cas-for-user-auth-tt4660837.html#none Any help is much appreciated. Many thanks, Sandra ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] errors after updating galaxy distribution
Hi, I updated my galaxy distribution with hg pull -u Then tried to re-run galaxy and got an error: $ ./run.sh Traceback (most recent call last): File ./scripts/paster.py, line 33, in module serve.run() File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/serve.py, line 1049, in run invoke(command, command_name, options, args[1:]) File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/serve.py, line 1055, in invoke exit_code = runner.run(args) File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/serve.py, line 220, in run result = self.command() File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/serve.py, line 643, in command app = loadapp( app_spec, name=app_name, relative_to=base, global_conf=vars) File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line 350, in loadapp return loadobj(APP, uri, name=name, **kw) File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line 374, in loadobj global_conf=global_conf) File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line 399, in loadcontext global_conf=global_conf) File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line 423, in _loadconfig return loader.get_context(object_type, name, global_conf) File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line 561, in get_context section) File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line 620, in _context_from_explicit value = import_string(found_expr) File /home/microbeco/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line 125, in import_string return pkg_resources.EntryPoint.parse(x= + s).load(False) File /home/microbeco/galaxy-dist/lib/pkg_resources.py, line 1954, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File /home/microbeco/galaxy-dist/lib/galaxy/web/buildapp.py, line 3, in module from galaxy.webapps.galaxy.buildapp import app_factory File /home/microbeco/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py, line 16, in module import galaxy.model.mapping File /home/microbeco/galaxy-dist/lib/galaxy/model/mapping.py, line 14, in module from sqlalchemy.types import BigInteger ImportError: cannot import name BigInteger I also tried to update my database but got a different error: $ sh manage_db.sh upgrade 114 - 115... Traceback (most recent call last): File ./scripts/manage_db.py, line 62, in module main( repository=repo, url=db_url ) File /home/microbeco/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/shell.py, line 150, in main ret = command_func(**kwargs) File /home/microbeco/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py, line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File /home/microbeco/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/api.py, line 349, in _migrate schema.runchange(ver, change, changeset.step) File /home/microbeco/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/schema.py, line 184, in runchange change.run(self.engine, step) File /home/microbeco/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/script/py.py, line 101, in run func() TypeError: upgrade() takes exactly 1 argument (0 given) How can I fix this? Thanks, Dan ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] latest galaxy-central version
Hi Robert, We are currently working on improving data libraries including the speed enhancements. In the meantime one of our users also reported that he is using for the dataset folders multiple libraries - each having less folder-depth which for him improves performance significantly. Maybe it will be of use to you. --Martin On Tue, Nov 5, 2013 at 5:19 PM, Robert Baertsch robert.baert...@gmail.comwrote: Carl, Thanks for building a great tool. Can you fix the library performance problem? Steps to reproduce: 1. http://su2c-dev.ucsc.edu/library 2. open library TCGA Pancan. It takes more than 20 seconds! -Robert On Nov 5, 2013, at 1:29 PM, Carl Eberhard carlfeberh...@gmail.com wrote: Thanks for the thorough report, Robert. I've added some sanity checking to the panel in default:11240:c271cdb443c6 My only guess as to why the panel javascript is trying to load at all in the library section is a link to the controller/datasets methods (unhide, delete, etc.) that still assume histories are in their own frames. Still investigating. On Tue, Nov 5, 2013 at 12:06 PM, Robert Baertsch baert...@soe.ucsc.eduwrote: Carl, Just to make sure I just deleted my tree and did a fresh checkout and tried a third time. It happened all three times. I'm using a postgres database and this time I started with a fresh config files and just changed the setting below. I didn't see the javascript error in Chrome, perhaps it is some strangeness in Firebug. I've seen strange errors before in Firebug. Attached are two screen grab of the lastest try before and after browser refresh. changeset: 11227:151b7d3b2f1b branch: stable tag: tip parent: 11219:5c789ab4144a user:Carl Eberhard carlfeberh...@gmail.com date:Tue Nov 05 11:33:06 2013 -0500 summary: History panel: fix naming bug in purge async error handling -Robert 29c29 port = 8585 --- #port = 8080 34c34 host = 0.0.0.0 --- #host = 127.0.0.1 96d95 database_connection = postgresql://localhost/medbookgalaxycentral3 542c541 allow_library_path_paste = True --- #allow_library_path_paste = False 599c598 admin_users = baert...@soe.ucsc.edu --- #admin_users = None Screen Shot 2013-11-05 at 8.35.09 AM.png Screen Shot 2013-11-05 at 8.37.04 AM.png On Nov 5, 2013, at 8:21 AM, Carl Eberhard carlfeberh...@gmail.com wrote: Hello, Robert I'm having difficulty reproducing this with a fresh install of galaxy_central:default:11226:c67b9518c1e0. Is this an intermittent error or does it happen reliably each time with the steps above? Is it still the same javascript error you mentioned above? I'll investigate further. On Mon, Nov 4, 2013 at 7:29 PM, Robert Baertsch baert...@soe.ucsc.eduwrote: I updated to the stable release and reproduced the issue. Step to reproduce 1. go to admin 2. Manage data librarieshttp://su2c-dev.ucsc.edu:8383/library_admin/browse_libraries 3. add dataset 4. select Upload files from filesystem paths 5. paste full path to any bam file. 6. leave defaults: auto-detect and copy files into galaxy 7. select role to restrict access 8. click upload to start Screen shows strange formatting and Job is running for Bam file. python /data/galaxy-central/tools/data_source/upload.py /data/galaxy-central /data/galaxy-central/database/tmp/tmpJoasJl /data/galaxy-central/database/tmp/tmpzZO8t1 8877:/data/galaxy-central/database/job_working_directory/004/4548/dataset_8877_files:/data/galaxy-central/database/files/008/dataset_8877.dat If I do a firefox refresh and go back to the library, the formatting is normal. I'm assuming the fix is to just render the page before waiting for the job to complete. -Robert On Nov 4, 2013, at 12:45 PM, Martin Čech mar...@bx.psu.edu wrote: Hello, I have also seen some of these errors while developing libraries. The library code is not in central however it might be related to recent changes to the history panel. Carl Eberhard might now more, adding him to the conversation. --Marten On Mon, Nov 4, 2013 at 2:45 PM, Robert Baertsch baert...@soe.ucsc.eduwrote: It keeps doing posts, and I'm not seeing any new errors. POST http://su2c-dev.ucsc.edu:8383/library_common/library_item_updates 200 OK 121ms When I did a browser refresh, I got the following javascript error: (I am logged in) Galaxy.currUser is undefined on Line 631 in history-panel.js When I opened the data library where the bam file was copying, everything is rendered ok. It seems the browser refresh fixed things. -Robert On Nov 4, 2013, at 11:14 AM, James Taylor ja...@jamestaylor.org wrote: Robert, I'm not sure what is going on here, other than that the javascript that converts buttons into dropdown menus has not fired. Are there any javascript errors? Marten is working on rewriting libraries, and we will be eliminating the progressive loading popupmenus for something much more efficient, but this also might
Re: [galaxy-dev] Question regarding walltime exceeded not being correctly reported via the WebUI
On Tue, Nov 5, 2013 at 11:53 AM, Daniel Patrick Sullivan dansulli...@gmail.com wrote: Hi, John, Thank you for taking the time to help me look into this issue. I have applied the patch you provided and confirmed that it appears to help remediate the problem (when a walltime is exceeded feedback is in fact provided via the Galaxy web UI; it no longer appears that jobs are running indefinitely).One thing I would like to note is that the error that is provided to the user is generic, i.e. the web UI reports An error occurred with this dataset: Job cannot be completed due to a cluster error, please retry it later. So, the fact that a Walltime exceeded error actually occurred is not presented to the user (I am not sure if this is intentional or not). Again, I appreciate you taking the time to verify and patch this issue. I have attached a screenshot of the output for your review. Glad we are making progress - I have committed that previous patch to galaxy-central. Lets see if we cannot improve the user feedback so they know they hit the maximum walltime. Can you try this new patch? The message about the timeout was being built but it was not being logged not set as the error message on the dataset - this should resolve that. I am probably going to be testing Galaxy with Torque 4.2.5 in the coming weeks, I will let you know if I identify any additional problems. Thank you so much have a wonderful day. You too, thanks for working with me on fixing this! -John Dan Sullivan On Tue, Nov 5, 2013 at 8:48 AM, John Chilton chil...@msi.umn.edu wrote: Hey Daniel, Thanks so much for the details problem report, it was very helpful. Reviewing the code there appears to be a bug in the PBS job runner - in some cases pbs_job_state.stop_job is never set but is attempted to be read. I don't have torque so I don't have a great test setup for this problem, any chance you can make the following changes for me and let me know if they work? Between the following two lines: log.error( '(%s/%s) PBS job failed: %s' % ( galaxy_job_id, job_id, JOB_EXIT_STATUS.get( int( status.exit_status ), 'Unknown error: %s' % status.exit_status ) ) ) self.work_queue.put( ( self.fail_job, pbs_job_state ) ) log.error( '(%s/%s) PBS job failed: %s' % ( galaxy_job_id, job_id, JOB_EXIT_STATUS.get( int( status.exit_status ), 'Unknown error: %s' % status.exit_status ) ) ) pbs_job_state.stop_job = False self.work_queue.put( ( self.fail_job, pbs_job_state ) ) And at the top of the file can you add a -11 option to the JOB_EXIT_STATUS to indicate a job timeout. I have attached a patch that would apply against the latest stable - it will probably will work against your branch as well. If you would rather not act as my QC layer, I can try to come up with a way to do some testing on my end :). Thanks again, -John On Mon, Nov 4, 2013 at 10:10 AM, Daniel Patrick Sullivan dansulli...@gmail.com wrote: Hi, Galaxy Developers, I have what I hops is somewhat of a basic question regarding Galaxy's interaction with a pbs job cluster and information reported via the webUI. Basically, in certain situations, the walltime of a specific job is exceeded. This is of course to be expected and all fine and understandeable. My problem is that the information is not being relayed back to the end user via the Galaxy web UI, which causes confusion in our Galaxy user community. Basically the Torque scheduler generates the following message when a walltime is exceeded: 11/04/2013 08:39:45;000d;PBS_Server.30621;Job;163.sctest.cri.uchicago.edu;preparing to send 'a' mail for job 163.sctest.cri.uchicago.edu to s.cri.gal...@crigalaxy-test.uchicago.edu (Job exceeded its walltime limit. Job was aborted 11/04/2013 08:39:45;0009;PBS_Server.30621;Job;163.sctest.cri.uchicago.edu;job exit status -11 handled Now, my problem is that this status -11 return code is not being correctly handled by Galaxy. What happens is that Galaxy throws an exception, specificially: 10.135.217.178 - - [04/Nov/2013:08:39:42 -0500] GET /api/histories/90240358ebde1489 HTTP/1.1 200 - https://crigalaxy-test.uchicago.edu/history; Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0 galaxy.jobs.runners.pbs DEBUG 2013-11-04 08:39:46,137 (2150/163.sctest.cri.uchicago.edu) PBS job state changed from R to C galaxy.jobs.runners.pbs ERROR 2013-11-04 08:39:46,139 (2150/163.sctest.cri.uchicago.edu) PBS job failed: Unknown error: -11 galaxy.jobs.runners ERROR 2013-11-04 08:39:46,139 (unknown) Unhandled exception calling fail_job Traceback (most recent call last): File /group/galaxy_test/galaxy-dist/lib/galaxy/jobs/runners/__init__.py, line 60, in run_next method(arg) File /group/galaxy_test/galaxy-dist/lib/galaxy/jobs/runners/pbs.py, line 561,
