This is an automated email from the git hooks/post-receive script. daube-guest pushed a commit to branch master in repository debian-med-benchmarking-spec.git.
commit fca8f8e5d5b1f24fe111be86c3857ffe76da4c50 Author: Kevin Murray <[email protected]> Date: Fri Feb 5 17:17:01 2016 +0100 Presented version w/ Piotr's edits --- benchmarking.md | 48 +++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/benchmarking.md b/benchmarking.md index 28f8f1b..a1addc3 100644 --- a/benchmarking.md +++ b/benchmarking.md @@ -26,6 +26,33 @@ for tool's scientific accuracy. - Report metrics to Debian infrastructure (UDD) and wider community (SEQWiki, bio.tools) +Packages to be tested +--------------------- + +All packages within debian med that: + + - Have EDAM-compatible DebTags + - Have a CWL tool description for each tool in the package + - Should contain EDAM tags per operation + - Potentially one CWL tool file per subtool/operation (e.g. `samtools view` + vs `samtools sort`) + - Are in `main` + +Operation +--------- + + - Service that runs benchmarks when: + - There is a new version of the underlying package in Debian med unstable + - Including any conversion utilities + - There is a change in an applicable script for the calculation of metrics + - There is a change in an applicable benchmark dataset + - There is an applicable transition in progress?? + - Could catch subtle bugs e.g. py3.4 -> py3.5 issues + - Benchmark datasets and scripts for calculation of metrics are automatically + selected from repository/DB based on their and the tool's EDAM tags + - For reproducibility/debugging, a Docker image performing all setup and + preconversion is published along with the benchmark results + Architecture: ------------- @@ -80,27 +107,6 @@ Architecture: be used for the test execution - Could all this run on a new instance of debci? or Jenkins? -Requirements ------------- - - - Have an EDAM-compatible DebTags - - Have a CWL tool description for each tool in the package - - Should contain EDAM tags per operation - - Potentially one CWL tool file per subtool/operation (e.g. `samtools view` - vs `samtools sort`) - - Be in `main` - -Operation ---------- - - - New service that runs benchmarks when: - - There is a new version of the underlying package in Debian med unstable - - Including any conversion utilities - - There is a change in an applicable script for the calculation of metrics - - There is a change in an applicable benchmark dataset - - There is an applicable transition in progress?? - - Could catch subtle bugs e.g. py3.4 -> py3.5 issues - Schema of ideas --------------- -- Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-med/debian-med-benchmarking-spec.git.git _______________________________________________ debian-med-commit mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/debian-med-commit
