Author: tbooth-guest Date: 2012-11-14 13:55:16 +0000 (Wed, 14 Nov 2012) New Revision: 12534
Added: trunk/packages/qiime/trunk/debian/patches/check_config_file_in_new_location.patch trunk/packages/qiime/trunk/debian/patches/fix_path_for_support_files trunk/packages/qiime/trunk/debian/patches/fix_t_test_bug trunk/packages/qiime/trunk/debian/patches/make_qiime_accept_new_rdp_classifier Modified: trunk/packages/qiime/trunk/debian/changelog trunk/packages/qiime/trunk/debian/control trunk/packages/qiime/trunk/debian/patches/series trunk/packages/qiime/trunk/debian/scripts/qiime trunk/packages/qiime/trunk/debian/scripts/shell/qiime_environment Log: Added some patches to address QIIME bugs. The patches are tested but not pretty, especially the make_qiime_accept_new_rdp_classifier patch which is a hatchet job. Modified: trunk/packages/qiime/trunk/debian/changelog =================================================================== --- trunk/packages/qiime/trunk/debian/changelog 2012-11-14 10:09:03 UTC (rev 12533) +++ trunk/packages/qiime/trunk/debian/changelog 2012-11-14 13:55:16 UTC (rev 12534) @@ -1,3 +1,15 @@ +qiime (1.5.0+repack-2ubuntu10) precise; urgency=low + + * Made QIIME accept RDP classifier >2.2 though the patch is + a total hack. Solves issue reported by MJC. + * Add patch to make QIIME look in the Debian standard location + for king.jar and other support_files + * Added patch to fix alpha diversity bug and tested by running + test_compare_alpha_diversities.py + * QIIME now needs python-cogent 1.5.3 + + -- Tim Booth <[email protected]> Wed, 14 Nov 2012 09:59:24 +0000 + qiime (1.5.0+dfsg-1) UNRELEASED; urgency=low * debian/copyright: - Add Files-Excluded to document what was removed @@ -6,8 +18,14 @@ -- Andreas Tille <[email protected]> Wed, 29 Aug 2012 14:30:02 +0200 -qiime (1.5.0-2ubuntu4) precise; urgency=low +qiime (1.5.0+repack-2ubuntu5) precise; urgency=low + * Tweaked startup script to report the QIIME version + + -- Tim Booth <[email protected]> Thu, 11 Oct 2012 13:49:35 +0100 + +qiime (1.5.0+repack-2ubuntu4) precise; urgency=low + * Remove python26_trim_sff_primers.patch as Qiime now requires 2.7 anyway * Avoid calling dh_auto_clean at all as it triggers a partial @@ -19,7 +37,7 @@ * Added patch to self-test where it looks for default config file * Modified to work with new RDP-classifier package - -- Tim Booth <[email protected]> Tue, 14 Aug 2012 17:33:26 +0100 + -- Tim Booth <[email protected]> Tue, 14 Aug 2012 17:53:13 +0100 qiime (1.5.0-2) unstable; urgency=low Modified: trunk/packages/qiime/trunk/debian/control =================================================================== --- trunk/packages/qiime/trunk/debian/control 2012-11-14 10:09:03 UTC (rev 12533) +++ trunk/packages/qiime/trunk/debian/control 2012-11-14 13:55:16 UTC (rev 12534) @@ -5,7 +5,7 @@ Uploaders: Steffen Moeller <[email protected]>, Tim Booth <[email protected]>, Andreas Tille <[email protected]> DM-Upload-Allowed: yes -Build-Depends: debhelper (>= 9), python-all-dev (>= 2.7), python-cogent ( >= 1.5.1 ), +Build-Depends: debhelper (>= 9), python-all-dev (>= 2.7), python-cogent ( >= 1.5.3 ), python-numpy, python-matplotlib, ghc6, python-sphinx Standards-Version: 3.9.3 Homepage: http://www.qiime.org/ @@ -16,8 +16,7 @@ Package: qiime Architecture: any Depends: ${shlibs:Depends}, ${misc:Depends}, ${python:Depends}, - python-pynast (>= 1.1)|pynast (>= 1.1), python-cogent ( >= 1.5.1 ), - king, python-biom-format + python-pynast (>= 1.1)|pynast (>= 1.1), python-cogent ( >= 1.5.3 ), king, python-biom-format Recommends: blast2 | blast+-legacy, cd-hit, rdp-classifier, chimeraslayer, muscle, infernal, fasttree, ampliconnoise, python-matplotlib, python-numpy, libjs-jquery Suggests: t-coffee, cytoscape Added: trunk/packages/qiime/trunk/debian/patches/check_config_file_in_new_location.patch =================================================================== --- trunk/packages/qiime/trunk/debian/patches/check_config_file_in_new_location.patch (rev 0) +++ trunk/packages/qiime/trunk/debian/patches/check_config_file_in_new_location.patch 2012-11-14 13:55:16 UTC (rev 12534) @@ -0,0 +1,15 @@ +We've moved the default config file, so tell this script to look in the new +location. See the next patch for other path fixes related to support_files + +--- a/scripts/print_qiime_config.py ++++ b/scripts/print_qiime_config.py +@@ -247,8 +247,7 @@ + """local qiime_config has no extra params""" + + qiime_project_dir = get_qiime_project_dir() +- orig_config = parse_qiime_config_file(open(qiime_project_dir + +- '/qiime/support_files/qiime_config')) ++ orig_config = parse_qiime_config_file(open('/etc/qiime/qiime_config')) + + #check the env qiime_config + qiime_config_env_filepath = getenv('QIIME_CONFIG_FP') Added: trunk/packages/qiime/trunk/debian/patches/fix_path_for_support_files =================================================================== --- trunk/packages/qiime/trunk/debian/patches/fix_path_for_support_files (rev 0) +++ trunk/packages/qiime/trunk/debian/patches/fix_path_for_support_files 2012-11-14 13:55:16 UTC (rev 12534) @@ -0,0 +1,88 @@ +Get QIIME to look in the right directory for support_files. +The default location is relative to the script which is +in a different location depending on the python version. +Author: Tim Booth <[email protected]> + +--- +The information above should follow the Patch Tagging Guidelines, please +checkout http://dep.debian.net/deps/dep3/ to learn about the format. Here +are templates for supplementary fields that you might want to add: + +Origin: <vendor|upstream|other>, <url of original patch> +Bug: <url in upstream bugtracker> +Bug-Debian: http://bugs.debian.org/<bugnumber> +Bug-Ubuntu: https://launchpad.net/bugs/<bugnumber> +Forwarded: <no|not-needed|url proving that it has been forwarded> +Reviewed-By: <name and email of someone who approved the patch> +Last-Update: <YYYY-MM-DD> + +--- qiime-1.5.0+repack.orig/scripts/plot_taxa_summary.py ++++ qiime-1.5.0+repack/scripts/plot_taxa_summary.py +@@ -219,8 +219,7 @@ def main(): + pass + + # move javascript file to javascript output directory +- shutil.copyfile(os.path.join(qiime_dir,'qiime','support_files',\ +- 'js/overlib.js'),\ ++ shutil.copyfile('/usr/lib/qiime/support_files/js/overlib.js',\ + os.path.join(javascript_path,'overlib.js')) + + #make css output directory +@@ -231,8 +230,7 @@ def main(): + pass + + # move css file to css output directory +- shutil.copyfile(os.path.join(qiime_dir,'qiime','support_files',\ +- 'css/qiime_style.css'),\ ++ shutil.copyfile('/usr/lib/qiime/support_files/css/qiime_style.css',\ + os.path.join(css_path,'qiime_style.css')) + + # verify all parameters are valid +--- qiime-1.5.0+repack.orig/scripts/make_otu_heatmap_html.py ++++ qiime-1.5.0+repack/scripts/make_otu_heatmap_html.py +@@ -137,7 +137,8 @@ def main(): + + qiime_dir=get_qiime_project_dir() + +- js_path=os.path.join(qiime_dir,'qiime/support_files/js') ++ #js_path=os.path.join(qiime_dir,'qiime/support_files/js') ++ js_path='/usr/lib/qiime/support_files/js' + + shutil.copyfile(os.path.join(js_path,'overlib.js'), os.path.join(js_dir_path,'overlib.js')) + shutil.copyfile(os.path.join(js_path,'otu_count_display.js'), os.path.join(js_dir_path,'otu_count_display.js')) +--- qiime-1.5.0+repack.orig/scripts/compare_3d_plots.py ++++ qiime-1.5.0+repack/scripts/compare_3d_plots.py +@@ -143,7 +143,8 @@ def main(): + + qiime_dir=get_qiime_project_dir() + +- jar_path=os.path.join(qiime_dir,'qiime/support_files/jar/') ++ #jar_path=os.path.join(qiime_dir,'qiime/support_files/jar/') ++ jar_path='/usr/lib/qiime/support_files/jar/' + + data_dir_path = get_random_directory_name(output_dir=dir_path, + return_absolute_path=False) +--- qiime-1.5.0+repack.orig/scripts/make_distance_histograms.py ++++ qiime-1.5.0+repack/scripts/make_distance_histograms.py +@@ -142,7 +142,8 @@ def main(): + for color_info in groups_and_colors: + field_to_colors[color_info[0]]=color_info[1:] + +- qiime_dir = get_qiime_project_dir()+'/qiime/support_files/' ++ #qiime_dir = get_qiime_project_dir()+'/qiime/support_files/' ++ qiime_dir='/usr/lib/qiime/support_files/' + + fields = opts.fields + if fields is not None: +--- qiime-1.5.0+repack.orig/scripts/make_2d_plots.py ++++ qiime-1.5.0+repack/scripts/make_2d_plots.py +@@ -148,7 +148,8 @@ def main(): + + qiime_dir=get_qiime_project_dir() + +- js_path=os.path.join(qiime_dir,'qiime','support_files','js') ++ #js_path=os.path.join(qiime_dir,'qiime','support_files','js') ++ js_path='/usr/lib/qiime/support_files/js' + + if opts.output_dir: + if os.path.exists(opts.output_dir): Added: trunk/packages/qiime/trunk/debian/patches/fix_t_test_bug =================================================================== --- trunk/packages/qiime/trunk/debian/patches/fix_t_test_bug (rev 0) +++ trunk/packages/qiime/trunk/debian/patches/fix_t_test_bug 2012-11-14 13:55:16 UTC (rev 12534) @@ -0,0 +1,565 @@ +Version of the patch found here: +https://github.com/jrrideout/qiime/commit/f9d6f4df9b227d23adf080ac012bd1331176d50c +Extracted by cloning the repo and doing: +git diff-tree -M -p f9d6f4d +Then patch was manually merged and tested by running the test case. +Note this patch introduces a dependency on the newer python-cogent. +Patch can be dropped for 1.6 release. +Author: Tim Booth <[email protected]> + +--- +The information above should follow the Patch Tagging Guidelines, please +checkout http://dep.debian.net/deps/dep3/ to learn about the format. Here +are templates for supplementary fields that you might want to add: + +Origin: <vendor|upstream|other>, <url of original patch> +Bug: <url in upstream bugtracker> +Bug-Debian: http://bugs.debian.org/<bugnumber> +Bug-Ubuntu: https://launchpad.net/bugs/<bugnumber> +Forwarded: <no|not-needed|url proving that it has been forwarded> +Reviewed-By: <name and email of someone who approved the patch> +Last-Update: <YYYY-MM-DD> + +--- qiime-1.5.0+repack.orig/ChangeLog ++++ qiime-1.5.0+repack/ChangeLog +@@ -1,3 +1,7 @@ ++Patch backported from QIIME 1.6.0, dev branch ++================================== ++* compare_alpha_diversity.py now supports both parametric and nonparametric two sample t-tests (nonparametric is the default) with the new optional options -t/--test_type and -n/--num_permutations. Also fixed a bug that used the wrong degrees of freedom in the t-tests, yielding incorrect t statistics and p-values. ++ + QIIME 1.5.0 (8 May 2012) + ================================== + * OTU tables are now stored on disk in the BIOM file format (see http://biom-format.org). The BIOM format webpage describes the motivation for the switch, but briefly it will support interoperability of related tools (e.g., QIIME/MG-RAST/mothur/VAMPS), and is a more efficient representation of data/metadata. The biom-format projects DenseTable and SparseTable objects are now used to represent OTU tables in memory. See the convert_biom.py script in the biom-format project for converting between 'classic' and BIOM formatted OTU tables. +--- qiime-1.5.0+repack.orig/scripts/compare_alpha_diversity.py ++++ qiime-1.5.0+repack/scripts/compare_alpha_diversity.py +@@ -4,7 +4,7 @@ from __future__ import division + + __author__ = "William Van Treuren" + __copyright__ = "Copyright 2011, The QIIME project" +-__credits__ = ["William Van Treuren, Greg Caparaso"] ++__credits__ = ["William Van Treuren", "Greg Caparaso", "Jai Ram Rideout"] + __license__ = "GPL" + __version__ = "1.5.0" + __maintainer__ = "William Van Treuren" +@@ -13,80 +13,117 @@ __status__ = "Release" + + + from qiime.util import parse_command_line_parameters, make_option +-from qiime.compare_alpha_diversity import compare_alpha_diversities,\ +- extract_rarefaction_scores_at_depth ++from qiime.compare_alpha_diversity import (compare_alpha_diversities, ++ extract_rarefaction_scores_at_depth, ++ test_types) + + script_info = {} +-script_info['brief_description'] = """This script compares alpha diversities based on a t_two_sample test""" ++script_info['brief_description'] = """This script compares alpha diversities based on a two sample t-test""" + +-script_info['script_description'] = """This script compares the alpha diversity of entries in a rarefaction file after they have been grouped based on some category found in the mapping file based on a t_two_sample test.""" ++script_info['script_description'] = """ ++This script compares the alpha diversity of entries in a rarefaction file after ++they have been grouped based on some category found in the mapping file based ++on a two sample t-test. The output file contains the ++(Category: (Subcategories): t, prob). By default the two sample t-test will be ++nonparametric (i.e. using Monte Carlo permutations to calculate the p-value), ++though the user has the option to make the test a parametric t-test. ++""" + + script_info['script_usage'] = [] +-script_info['script_usage'].append(("""Explanation:\ Inputs: mapping file lines (lines of a mapping file which associates to each OTU/sample a number of characteristics, given as file path), rarefaction file lines (lines of a rarefaction file that has scores for each OTU/sample based on a certain rarefaction depth, given as a file path), depth (the depth score of the rarefaction file to use), category (the category to compare OTU/samples on), output file path (a path to the output file).""",\ +- """""",\ +- """%prog -i PD_whole_tree.txt -m map.txt -c Treatment -d 100 -o PD_d100.txt""")) +-script_info['output_description']= \ +- """Script generates an output nested dictionary which has as a first +- key:value pair the category passed, and a dictionary which gives the +- t_two_sample score for every possible combination of the values +- under that category in the mapping file, saved as a text file into +- the directory specified by the output path.""" + ++script_info['script_usage'].append(("Comparing alpha diversities", ++"The following command takes the following input: a mapping file (which " ++"associaties each sample with a number of characteristics), alpha diversity " ++"metric (the results of collate_alpha for an alpha diverity metric, like " ++"PD_whole_tree), depth (the rarefaction depth to use for comparison), " ++"category (the category in the mapping file to determine which samples to " ++"compare to each other), and output file path (a path to the output file). A " ++"nonparametric two sample t-test is run to compare the alpha diversities " ++"using the default number of Monte Carlo permutations (999).", ++"%prog -i PD_whole_tree.txt -m mapping.txt -c Treatment -d 100 -o PD_d100.txt")) ++ ++script_info['script_usage'].append(("Parametric t-test", ++"The following command runs a parametric two sample t-test using the " ++"t-distribution instead of Monte Carlo permutations.", ++"%prog -i PD_whole_tree.txt -m mapping.txt -c Treatment -d 100 -o " ++"PD_d100_parametric.txt -t parametric")) ++ ++script_info['output_description']= """ ++Script generates an output nested dictionary which has as a first key:value ++pair the category passed, and a dictionary which gives the t_two_sample score ++for every possible combination of the values under that category in the ++mapping file, saved as a text file into the directory specified by the output ++path. ++""" + +-script_info['required_options']=[\ ++script_info['required_options']=[ + make_option('-i', + '--alpha_diversity_filepath', + action='store', + type='existing_filepath', + dest='alpha_diversity_fp', +- help='path to collated alpha diversity file (as generated by collate_alpha.py) [REQUIRED]'),\ ++ help='path to collated alpha diversity file (as generated by ' ++ 'collate_alpha.py) [REQUIRED]'), + make_option('-m', + '--mapping_filepath', + action='store', + type='existing_filepath', + dest='mapping_fp', +- help='path to the mapping file [REQUIRED]'),\ ++ help='path to the mapping file [REQUIRED]'), + make_option('-c', + '--category', + action='store', + type='string', + dest='category', +- help='category for comparison [REQUIRED]'),\ ++ help='category for comparison [REQUIRED]'), + make_option('-d', + '--depth', + action='store', + type='string', + dest='depth', +- help='depth of rarefaction file to use [REQUIRED]'),\ ++ help='depth of rarefaction file to use [REQUIRED]'), + make_option('-o', + '--output_filepath', + action='store', + type='new_filepath', + dest='output_fp', + help='output file path [REQUIRED]')] ++ ++script_info['optional_options'] = [ ++ make_option('-t', '--test_type', type='choice', choices=test_types, ++ help='the type of test to perform when calculating the p-values. Valid ' ++ 'choices: ' + ', '.join(test_types) + '. If test_type is ' ++ 'nonparametric, Monte Carlo permutations will be used to determine the ' ++ 'p-value. If test_type is parametric, the num_permutations option will ' ++ 'be ignored and the t-distribution will be used instead [default: ' ++ '%default]', default='nonparametric'), ++ make_option('-n', '--num_permutations', type='int', default=999, ++ help='the number of permutations to perform when calculating the ' ++ 'p-value. Must be greater than zero. Only applies if test_type is ' ++ 'nonparametric [default: %default]')] ++ + script_info['version'] = __version__ + + def main(): + option_parser, opts, args = parse_command_line_parameters(**script_info) ++ + rarefaction_lines = open(opts.alpha_diversity_fp, 'U') +- + mapping_lines = open(opts.mapping_fp, 'U') + category = opts.category + depth = int(opts.depth) + output_path = opts.output_fp +- +- +- result = compare_alpha_diversities(rarefaction_lines,\ +- mapping_lines,\ +- category,\ +- depth) ++ ++ result = compare_alpha_diversities(rarefaction_lines, mapping_lines, ++ category, depth, opts.test_type, ++ opts.num_permutations) + outfile = open(output_path, 'w') + outfile.write(str(result)) +- ++ outfile.write('\n') ++ + outfile.close() + rarefaction_lines.close() +- mapping_lines.close() +- +- ++ mapping_lines.close() ++ ++ + if __name__ == "__main__": + main() +--- qiime-1.5.0+repack.orig/qiime/compare_alpha_diversity.py ++++ qiime-1.5.0+repack/qiime/compare_alpha_diversity.py +@@ -4,7 +4,7 @@ from __future__ import division + + __author__ = "William Van Treuren" + __copyright__ = "Copyright 2011, The QIIME project" +-__credits__ = ["William Van Treuren","Greg Caporaso"] ++__credits__ = ["William Van Treuren", "Greg Caporaso", "Jai Ram Rideout"] + __license__ = "GPL" + __version__ = "1.5.0" + __maintainer__ = "William Van Treuren" +@@ -12,8 +12,11 @@ __email__ = "[email protected]" + __status__ = "Release" + + from numpy import array, isnan ++from qiime.format import format_p_value_for_num_iters + from qiime.parse import parse_mapping_file_to_dict, parse_rarefaction +-from cogent.maths.stats.test import t_two_sample ++from cogent.maths.stats.test import mc_t_two_sample, t_two_sample ++ ++test_types = ['parametric', 'nonparametric'] + + def make_value_pairs_from_category(mapping_data, category): + """creates all pairs of unique category values from mapping data +@@ -249,7 +252,8 @@ def convert_SampleIds_to_rarefaction_mtx + + + def compare_alpha_diversities(rarefaction_lines, mapping_lines, +- category, depth): ++ category, depth, test_type='nonparametric', ++ num_permutations=999): + """compares alpha diversities + + inputs: +@@ -262,6 +266,12 @@ def compare_alpha_diversities(rarefactio + category - the category to be compared, is a string + + depth - the depth of the rarefaction_file to use, is an integer ++ ++ test_type - the type of t-test to perform, is a string. Must be either ++ 'parametric' or 'nonparametric' ++ ++ num_permutations - the number of Monte Carlo permutations to use if ++ test_type is 'nonparametric', is an integer + + outputs: + results - a nested dictionary which specifies the category as +@@ -270,6 +280,9 @@ def compare_alpha_diversities(rarefactio + in the specified category + + """ ++ if test_type == 'nonparametric' and num_permutations < 1: ++ raise ValueError("Invalid number of permutations: %d. Must be greater " ++ "than zero." % num_permutations) + + rarefaction_data = parse_rarefaction(rarefaction_lines) + mapping_data = parse_mapping_file_to_dict(mapping_lines)[0] +@@ -290,15 +303,26 @@ def compare_alpha_diversities(rarefactio + results = {category:{}} + + for pair in range(len(SampleId_pairs)): ++ # Must flatten the matrix because t_two_sample only operates on ++ # non-nested sequences (otherwise we'll get the wrong degrees of ++ # freedom). + i=(convert_SampleIds_to_rarefaction_mtx(SampleId_pairs[pair][0], +- reduced_rarefaction_mtx, map_from_Id_to_col)) ++ reduced_rarefaction_mtx, ++ map_from_Id_to_col)).flatten() + + j=(convert_SampleIds_to_rarefaction_mtx(SampleId_pairs[pair][1], +- reduced_rarefaction_mtx, map_from_Id_to_col)) +- ++ reduced_rarefaction_mtx, ++ map_from_Id_to_col)).flatten() ++ ++ if test_type == 'parametric': ++ obs_t, p_val = t_two_sample(i,j) ++ elif test_type == 'nonparametric': ++ obs_t, _, _, p_val = mc_t_two_sample(i,j, ++ permutations=num_permutations) ++ p_val = format_p_value_for_num_iters(p_val, num_permutations) ++ else: ++ raise ValueError("Invalid test type '%s'." % test_type) ++ + results[category][(str(value_pairs[pair][0]), +- str(value_pairs[pair][1]))] =\ +- t_two_sample(i,j) +- ++ str(value_pairs[pair][1]))] = obs_t, p_val + return results +- +--- qiime-1.5.0+repack.orig/tests/test_compare_alpha_diversities.py ++++ qiime-1.5.0+repack/tests/test_compare_alpha_diversities.py +@@ -1,19 +1,16 @@ + #!/usr/bin/env python + # File created on 19 May 2011 + +- + from __future__ import division + + __author__ = "William Van Treuren" + __copyright__ = "Copyright 2011, The QIIME project" +-__credits__ = ["William Van Treuren, Greg Caporaso"] ++__credits__ = ["William Van Treuren", "Greg Caporaso", "Jai Ram Rideout"] + __license__ = "GPL" + __version__ = "1.5.0" + __maintainer__ = "William Van Treuren" + __email__ = "[email protected]" +-__status__ = "Release" +- +- ++__status__ = "Development" + + from cogent.util.unit_test import TestCase,main + from qiime.parse import parse_mapping_file_to_dict, parse_rarefaction +@@ -32,13 +29,13 @@ class TopLevelTests(TestCase): + def setUp(self): + """define data for tests""" + self.rarefaction_file = \ +- ['\tsequences per sample\titeration\t123\t234\t345', +- 'rare10.txt\t10\t0\t1.99181\t5.42877\t2.13996', +- 'rare10.txt\t10\t1\t2.07163\t1.42877\t2.37055', +- 'rare310.txt\t310\t0\t8.83115\t6.42877\t11.00725', +- 'rare310.txt\t310\t1\t10.05242\t9.42877\t8.24474', +- 'rare810.txt\t810\t0\t12.03067\tn/a\t11.58928', +- 'rare910.txt\t910\t1\t12.9862\t2.42877\t11.58642'] ++ ['\tsequences per sample\titeration\t123\t234\t345\t456', ++ 'rare10.txt\t10\t0\t1.99181\t5.42877\t2.13996\t0.002322', ++ 'rare10.txt\t10\t1\t2.07163\t1.42877\t2.37055\t0.01219', ++ 'rare310.txt\t310\t0\t8.83115\t6.42877\t11.00725\t0.18233', ++ 'rare310.txt\t310\t1\t10.05242\t9.42877\t8.24474\t0.99229', ++ 'rare810.txt\t810\t0\t12.03067\tn/a\t11.58928\t0.8993', ++ 'rare910.txt\t910\t1\t12.9862\t2.42877\t11.58642\t1.22563'] + + self.rarefaction_data = parse_rarefaction(self.rarefaction_file) + +@@ -48,7 +45,9 @@ class TopLevelTests(TestCase): + '#Comment Line', + '123\tAAAA\tBBBB\tHigh\t31\tM_ID_123', + '234\tCCCC\tDDDD\tLow\t67\tM_ID_234', +- '345\tAAAA\tFFFF\tMed\t21\tM_ID_345'] ++ '345\tAAAA\tFFFF\tMed\t21\tM_ID_345', ++ '456\tAAAA\tGGGG\tLow\t67\tM_ID_456' ++ ] + + self.mapping_data = \ + parse_mapping_file_to_dict(self.mapping_file)[0] +@@ -59,32 +58,34 @@ class TopLevelTests(TestCase): + self.value_pairs_Treatment = \ + [('CCCC', 'AAAA')] + self.cat_val_Dose = \ +- {'High': ['123'], 'Low': ['234'], 'Med': ['345']} ++ {'High': ['123'], 'Low': ['234', '456'], 'Med': ['345']} + self.cat_val_TTD = \ +- {'21': ['345'], '31': ['123'], '67': ['234']} ++ {'21': ['345'], '31': ['123'], '67': ['234', '456']} + self.cat_val_Treatment = \ +- {'AAAA': ['345', '123'], 'CCCC': ['234']} ++ {'AAAA': ['345', '123', '456'], 'CCCC': ['234']} + self.Id_pairs_Dose = \ +- [(['234'], ['345']), (['234'], ['123']), (['345'], ['123'])] ++ [(['234', '456'], ['345']), (['234', '456'], ['123']), ++ (['345'], ['123'])] + self.Id_pairs_TTD = \ +- [(['234'], ['345']), (['234'], ['123']), (['345'], ['123'])] ++ [(['234', '456'], ['345']), (['234', '456'], ['123']), ++ (['345'], ['123'])] + + self.Id_pairs_Treatment = \ +- [(['234'], ['345', '123'])] ++ [(['234'], ['345', '123', '456'])] + + self.rarefaction_cols_dict = \ +- {'123': 0, '234': 1, '345': 2} ++ {'123': 0, '234': 1, '345': 2, '456':3} + + self.extracted_mtx_10 = \ +- array([[ 1.99181, 5.42877, 2.13996], +- [ 2.07163, 1.42877, 2.37055]]) ++ array([[ 1.99181, 5.42877, 2.13996, 0.002322], ++ [ 2.07163, 1.42877, 2.37055, 0.01219]]) + + self.extracted_mtx_310 = \ +- array([[ 8.83115, 6.42877, 11.00725], +- [ 10.05242, 9.42877, 8.24474]]) ++ array([[ 8.83115, 6.42877, 11.00725, 0.18233], ++ [ 10.05242, 9.42877, 8.24474, 0.99229]]) + + self.extracted_mtx_910 = \ +- array([[ 12.9862 , 2.42877, 11.58642]]) ++ array([[ 12.9862 , 2.42877, 11.58642, 1.22563]]) + + self.sample_pair1 = \ + (['234'], ['345', '123']) +@@ -97,28 +98,26 @@ class TopLevelTests(TestCase): + array([[ 2.13996, 1.99181], + [ 2.37055, 2.07163]]) + +- self.compared_alpha_diversities_TTD = \ +- {'TTD':{('21','31'):(1.8321466933860993,0.20839398129924847), +- ('67', '21'): (0.58578495700890432, 0.61731739324369639), +- ('67', '31'): (0.69838596448703294, 0.55721515283248324)}} +- ++ self.compared_alpha_diversities_TTD = {'TTD': {('67', '21'): ++ (-0.27929839680103463, 0.79386220041241184), ('21', '31'): ++ (1.8321466933860993, 0.20839398129924847), ('67', '31'): ++ (-0.16318504125427058, 0.87828549279958279)}} ++ + def test_make_value_pairs_from_category(self): + """check value pairs returns correct unique pairs for categories + """ +- +- ++ + self.assertEqual( +- self.value_pairs_Dose, +- make_value_pairs_from_category(self.mapping_data, 'Dose')) ++ make_value_pairs_from_category(self.mapping_data, 'Dose'), ++ self.value_pairs_Dose) + + self.assertEqual( +- self.value_pairs_TTD, +- make_value_pairs_from_category(self.mapping_data, 'TTD')) ++ make_value_pairs_from_category(self.mapping_data, 'TTD'), ++ self.value_pairs_TTD) + + self.assertEqual( +- self.value_pairs_Treatment, +- make_value_pairs_from_category(self.mapping_data, +- 'Treatment')) ++ make_value_pairs_from_category(self.mapping_data, 'Treatment'), ++ self.value_pairs_Treatment) + + self.assertRaises( + ValueError, +@@ -128,59 +127,57 @@ class TopLevelTests(TestCase): + """check value pairs reference correct Id pairs""" + + self.assertEqual( +- self.cat_val_Dose, +- make_category_values_Id_dict(self.mapping_data, 'Dose')) ++ make_category_values_Id_dict(self.mapping_data, 'Dose'), ++ self.cat_val_Dose) + + self.assertEqual( +- self.cat_val_TTD, +- make_category_values_Id_dict(self.mapping_data, 'TTD')) ++ make_category_values_Id_dict(self.mapping_data, 'TTD'), ++ self.cat_val_TTD) + + self.assertEqual( +- self.cat_val_Treatment, +- make_category_values_Id_dict(self.mapping_data, +- 'Treatment')) ++ make_category_values_Id_dict(self.mapping_data, 'Treatment'), ++ self.cat_val_Treatment) + + def test_map_category_value_pairs_to_Ids(self): + """check value pairs converted to correct Ids""" + + self.assertEqual( +- self.Id_pairs_Dose, + map_category_value_pairs_to_Ids(self.value_pairs_Dose, +- self.cat_val_Dose)) ++ self.cat_val_Dose), ++ self.Id_pairs_Dose) + + self.assertEqual( +- self.Id_pairs_TTD, + map_category_value_pairs_to_Ids(self.value_pairs_TTD, +- self.cat_val_TTD)) ++ self.cat_val_TTD), ++ self.Id_pairs_TTD) + + self.assertEqual( +- self.Id_pairs_Treatment, +- map_category_value_pairs_to_Ids( +- self.value_pairs_Treatment, +- self.cat_val_Treatment)) ++ map_category_value_pairs_to_Ids(self.value_pairs_Treatment, ++ self.cat_val_Treatment), ++ self.Id_pairs_Treatment) + + def test_make_SampleIds_rarefaction_columns_dict(self): + """ """ + + self.assertEqual( +- self.rarefaction_cols_dict, +- make_SampleIds_rarefaction_columns_dict(self.rarefaction_data)) ++ make_SampleIds_rarefaction_columns_dict(self.rarefaction_data), ++ self.rarefaction_cols_dict) + + def test_extract_rarefaction_scores_at_depth(self): + """check correct errors raised for wrong depths and correct mtx + """ + +- self.assertEqual( +- self.extracted_mtx_10, +- extract_rarefaction_scores_at_depth(10,self.rarefaction_data)) +- +- self.assertEqual( +- self.extracted_mtx_310, +- extract_rarefaction_scores_at_depth(310,self.rarefaction_data)) +- +- self.assertEqual( +- self.extracted_mtx_910, +- extract_rarefaction_scores_at_depth(910,self.rarefaction_data)) ++ self.assertFloatEqual( ++ extract_rarefaction_scores_at_depth(10,self.rarefaction_data), ++ self.extracted_mtx_10) ++ ++ self.assertFloatEqual( ++ extract_rarefaction_scores_at_depth(310,self.rarefaction_data), ++ self.extracted_mtx_310) ++ ++ self.assertFloatEqual( ++ extract_rarefaction_scores_at_depth(910,self.rarefaction_data), ++ self.extracted_mtx_910) + + self.assertRaises( + ValueError, +@@ -215,13 +212,48 @@ class TopLevelTests(TestCase): + self.rarefaction_cols_dict), + self.rarefaction_mtx_for_sample_pair1_1) + +- def test_compare_alpha_diversity(self): +- """test main function properly compares alpha diversities""" ++ def test_compare_alpha_diversities_parametric(self): ++ """test main function properly compares alpha divs (parametric)""" + +- self.assertEqual( +- self.compared_alpha_diversities_TTD, ++ self.assertFloatEqual( + compare_alpha_diversities(self.rarefaction_file, +- self.mapping_file, 'TTD', 10)) ++ self.mapping_file, 'TTD', 10, ++ 'parametric'), ++ self.compared_alpha_diversities_TTD) ++ ++ # Should ignore num_permutations if test_type is parametric. ++ self.assertFloatEqual( ++ compare_alpha_diversities(self.rarefaction_file, ++ self.mapping_file, 'TTD', 10, ++ 'parametric', 0), ++ self.compared_alpha_diversities_TTD) ++ ++ def test_compare_alpha_diversities_nonparametric(self): ++ """test main function properly compares alpha divs (nonparametric)""" ++ ++ obs = compare_alpha_diversities(self.rarefaction_file, ++ self.mapping_file, 'TTD', 10, ++ 'nonparametric') ++ ++ # Since p-values are stochastic, we'll check that they are sane and ++ # that the t statistics are the same as we'd get for a parametric test. ++ for comp, (t, p_val) in obs['TTD'].items(): ++ exp = self.compared_alpha_diversities_TTD['TTD'][comp] ++ self.assertFloatEqual(t, exp[0]) ++ self.assertIsProb(float(p_val)) ++ ++ def test_compare_alpha_diversities_invalid_input(self): ++ """test main function bails on bad input""" ++ ++ # Invalid test_type. ++ self.assertRaises(ValueError, compare_alpha_diversities, ++ self.rarefaction_file, self.mapping_file, 'TTD', 10, ++ 'foo') ++ ++ # Invalid num_permutations. ++ self.assertRaises(ValueError, compare_alpha_diversities, ++ self.rarefaction_file, self.mapping_file, 'TTD', 10, ++ 'nonparametric', 0) + + + if __name__ == "__main__": Added: trunk/packages/qiime/trunk/debian/patches/make_qiime_accept_new_rdp_classifier =================================================================== --- trunk/packages/qiime/trunk/debian/patches/make_qiime_accept_new_rdp_classifier (rev 0) +++ trunk/packages/qiime/trunk/debian/patches/make_qiime_accept_new_rdp_classifier 2012-11-14 13:55:16 UTC (rev 12534) @@ -0,0 +1,72 @@ +This patch twists QIIME's arm to accept running a newer version +of the RDP classifier by setting RDP_JAR_VERSION_OK, which is done +by the QIIME wrapper. +This is a nasty hack and hopefully the patch can be dropped for QIIME 1.6 +--- a/qiime/assign_taxonomy.py ++++ b/qiime/assign_taxonomy.py +@@ -12,7 +12,7 @@ + #import csv + import logging + import re +-from os import system, remove, path, mkdir ++from os import system, remove, path, mkdir, getenv + from os.path import split, splitext + from glob import glob + from itertools import count +@@ -45,9 +45,11 @@ + "http://qiime.org/install/install.html#rdp-install";) + elif "2.2" in rdp_jarpath: + return "rdp22" ++ elif getenv('RDP_JAR_VERSION_OK') is not None and float(getenv('RDP_JAR_VERSION_OK')) >= 2.2: ++ return "rdp22" + else: + raise ValueError( +- "RDP classifier filename does not look like version 2.2. Only " ++ "RDP classifier filename does not look like version 2.2 or higher. Only " + "version 2.2 is supported by QIIME. RDP jar path is:" + " %s" % rdp_jarpath) + +@@ -306,7 +308,7 @@ + """Assign taxon using RDP's naive Bayesian classifier + """ + Name = "RdpTaxonAssigner" +- Application = "RDP classfier, version 2.2" ++ Application = "RDP classfier, version 2.2+" + Citation = "Wang, Q, G. M. Garrity, J. M. Tiedje, and J. R. Cole. 2007. Naive Bayesian Classifier for Rapid Assignment of rRNA Sequences into the New Bacterial Taxonomy. Appl Environ Microbiol. 73(16):5261-7." + Taxonomy = "RDP" + _tracked_properties = ['Application','Citation','Taxonomy'] +--- a/scripts/assign_taxonomy.py ++++ b/scripts/assign_taxonomy.py +@@ -14,7 +14,7 @@ + + from qiime.util import parse_command_line_parameters, get_options_lookup + from qiime.util import make_option, get_rdp_jarpath, load_qiime_config +-from os import system, remove, path, mkdir ++from os import system, remove, path, mkdir, getenv + from os.path import split, splitext + from qiime.assign_taxonomy import ( + BlastTaxonAssigner, RdpTaxonAssigner, RtaxTaxonAssigner, guess_rdp_version) +@@ -205,6 +205,11 @@ + params['training_data_properties_fp'] = opts.training_data_properties_fp + params['max_memory'] = "%sM" % opts.rdp_max_memory + ++ #Allow RDP classifier to be >2.2, as hinted by an env variable ++ if getenv('RDP_JAR_VERSION_OK') is not None: ++ params['real_rdp_version'] = getenv('RDP_JAR_VERSION_OK') ++ ++ + elif assignment_method == 'rtax': + params['id_to_taxonomy_fp'] = opts.id_to_taxonomy_fp + params['reference_sequences_fp'] = opts.reference_seqs_fp +--- a/scripts/make_3d_plots.py ++++ b/scripts/make_3d_plots.py +@@ -397,7 +397,8 @@ + + qiime_dir=get_qiime_project_dir() + +- jar_path=os.path.join(qiime_dir,'qiime/support_files/jar/') ++ #jar_path=os.path.join(qiime_dir,'qiime/support_files/jar/') ++ jar_path='/usr/lib/qiime/support_files/jar/' + + data_dir_path = get_random_directory_name(output_dir=dir_path, + return_absolute_path=False) Modified: trunk/packages/qiime/trunk/debian/patches/series =================================================================== --- trunk/packages/qiime/trunk/debian/patches/series 2012-11-14 10:09:03 UTC (rev 12533) +++ trunk/packages/qiime/trunk/debian/patches/series 2012-11-14 13:55:16 UTC (rev 12534) @@ -1,2 +1,5 @@ check_config_file_in_new_location.patch fix_binary_helper_location.patch +make_qiime_accept_new_rdp_classifier +fix_path_for_support_files +fix_t_test_bug Modified: trunk/packages/qiime/trunk/debian/scripts/qiime =================================================================== --- trunk/packages/qiime/trunk/debian/scripts/qiime 2012-11-14 10:09:03 UTC (rev 12533) +++ trunk/packages/qiime/trunk/debian/scripts/qiime 2012-11-14 13:55:16 UTC (rev 12534) @@ -27,9 +27,14 @@ fi fi +#Determine Qiime version: +QIIME_VERSION=`/usr/lib/qiime/bin/print_qiime_config.py --version 2>/dev/null | awk '{print $NF}'` +QIIME_VERSION=${QIIME_VERSION:-UNKNOWN} + echo """ -Setting up $QSNAME environment to run Qiime commands. +Setting up $QSNAME environment to run QIIME commands. +You have QIIME version $QIIME_VERSION. Type 'help' for more info, 'exit' to return to regular shell. """ Modified: trunk/packages/qiime/trunk/debian/scripts/shell/qiime_environment =================================================================== --- trunk/packages/qiime/trunk/debian/scripts/shell/qiime_environment 2012-11-14 10:09:03 UTC (rev 12533) +++ trunk/packages/qiime/trunk/debian/scripts/shell/qiime_environment 2012-11-14 13:55:16 UTC (rev 12534) @@ -14,6 +14,8 @@ # This was previously set in bio-linux-rdp-classifier but can now live here # as the rdp-classifier package has been normalised. export RDP_JAR_PATH=${RDP_JAR_PATH:-/usr/share/rdp-classifier/rdp_classifier.jar} +#RDP_classifier is very bad at admitting what version it is, so ask DPKG: +export RDP_JAR_VERSION_OK=`dpkg-query -f '${Version}\n' -W rdp-classifier | sed 's/[~+-].*//'` # Mike Cox reports these are wanted for Ampliconnoise export PYRO_LOOKUP_FILE=${PYRO_LOOKUP_FILE:-/usr/share/ampliconnoise/Data/LookUp_E123.dat} _______________________________________________ debian-med-commit mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/debian-med-commit
