This is an automated email from the git hooks/post-receive script. tille pushed a commit to branch master in repository cwltool.
commit fc0ce1f6a6d2d99b2fa479d8d3a51b2be7c25039 Author: Andreas Tille <[email protected]> Date: Fri Dec 22 08:19:37 2017 +0100 New upstream version 1.0.20171221100033 --- MANIFEST.in | 1 + Makefile | 6 +- PKG-INFO | 93 +++++++++++++++++++++------ README.rst | 91 ++++++++++++++++++++------ cwltool.egg-info/PKG-INFO | 93 +++++++++++++++++++++------ cwltool.egg-info/SOURCES.txt | 16 +++++ cwltool/draft2tool.py | 33 ++++++---- cwltool/load_tool.py | 110 +++++++++++++++++++++++-------- cwltool/main.py | 139 ++++++++++++++++++++++++++++------------ cwltool/pack.py | 37 +++++++++-- cwltool/pathmapper.py | 4 +- cwltool/process.py | 13 +++- cwltool/workflow.py | 14 ++-- setup.cfg | 2 +- tests/override/echo-job-ov.yml | 6 ++ tests/override/echo-job-ov2.yml | 7 ++ tests/override/echo-job.yml | 1 + tests/override/echo-wf.cwl | 14 ++++ tests/override/echo.cwl | 19 ++++++ tests/override/ov.yml | 5 ++ tests/override/ov2.yml | 5 ++ tests/override/ov3.yml | 5 ++ tests/test_ext.py | 1 - tests/test_fetch.py | 8 ++- tests/test_override.py | 66 +++++++++++++++++++ tests/test_pack.py | 101 ++++++++++++++++++++++++++--- tests/wf/count-lines1-wf.cwl | 25 ++++++++ tests/wf/formattest-job.json | 7 ++ tests/wf/formattest.cwl | 20 ++++++ tests/wf/parseInt-tool.cwl | 16 +++++ tests/wf/wc-job.json | 6 ++ tests/wf/wc-tool.cwl | 17 +++++ tests/wf/whale.txt | 16 +++++ 33 files changed, 829 insertions(+), 168 deletions(-) diff --git a/MANIFEST.in b/MANIFEST.in index 47e3080..effba2c 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -2,6 +2,7 @@ include gittaggers.py Makefile cwltool.py include tests/* include tests/tmp1/tmp2/tmp3/.gitkeep include tests/wf/* +include tests/override/* include cwltool/schemas/v1.0/*.yml include cwltool/schemas/draft-2/*.yml include cwltool/schemas/draft-3/*.yml diff --git a/Makefile b/Makefile index 2b6a380..bd4a87d 100644 --- a/Makefile +++ b/Makefile @@ -89,7 +89,7 @@ pydocstyle_report.txt: $(PYSOURCES) pydocstyle setup.py $^ > pydocstyle_report.txt 2>&1 || true diff_pydocstyle_report: pydocstyle_report.txt - diff-quality --violations=pep8 $^ + diff-quality --violations=pycodestyle $^ ## autopep8 : fix most Python code indentation and formatting autopep8: $(PYSOURCES) @@ -160,7 +160,7 @@ mypy2: ${PYSOURCES} rm -Rf typeshed/2and3/schema_salad ln -s $(shell python -c 'from __future__ import print_function; import schema_salad; import os.path; print(os.path.dirname(schema_salad.__file__))') \ typeshed/2and3/schema_salad - MYPYPATH=$MYPYPATH:typeshed/2.7:typeshed/2and3 mypy --py2 --disallow-untyped-calls \ + MYPYPATH=$$MYPYPATH:typeshed/2.7:typeshed/2and3 mypy --py2 --disallow-untyped-calls \ --warn-redundant-casts \ cwltool @@ -171,7 +171,7 @@ mypy3: ${PYSOURCES} rm -Rf typeshed/2and3/schema_salad ln -s $(shell python3 -c 'from __future__ import print_function; import schema_salad; import os.path; print(os.path.dirname(schema_salad.__file__))') \ typeshed/2and3/schema_salad - MYPYPATH=$MYPYPATH:typeshed/3:typeshed/2and3 mypy --disallow-untyped-calls \ + MYPYPATH=$$MYPYPATH:typeshed/3:typeshed/2and3 mypy --disallow-untyped-calls \ --warn-redundant-casts \ cwltool diff --git a/PKG-INFO b/PKG-INFO index d3ce41e..694e059 100644 --- a/PKG-INFO +++ b/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cwltool -Version: 1.0.20171107133715 +Version: 1.0.20171221100033 Summary: Common workflow language reference implementation Home-page: https://github.com/common-workflow-language/cwltool Author: Common workflow language working group @@ -41,7 +41,7 @@ Description: ================================================================== virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well source venv/bin/activate # Activate environment before installing `cwltool` - 1. Installing the official package from PyPi (will install "cwltool" package as + Installing the official package from PyPi (will install "cwltool" package as well) .. code:: bash @@ -54,7 +54,7 @@ Description: ================================================================== pip install cwltool - 2. To install from source + Or you can install from source: .. code:: bash @@ -72,9 +72,16 @@ Description: ================================================================== - Running basic tests ``(/tests)``: - We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_ - to run various tests in all supported Python environments. - You can run the test suite by simply running the following in the terminal: + To run the basis tests after installing `cwltool` execute the following: + + .. code:: bash + + pip install pytest mock + py.test --ignore cwltool/schemas/ --pyarg cwltool + + To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments + first downloading the complete code repository (see the ``git clone`` instructions above) and then run + the following in the terminal: ``pip install tox; tox`` List of all environment can be seen using: @@ -116,6 +123,21 @@ Description: ================================================================== .. |Build Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance :target: https://ci.commonwl.org/job/cwltool-conformance/ + Running user-space implementations of Docker + -------------------------------------------- + + Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option. + + Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images): + + For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions). + + Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests: + + .. code:: bash + + cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json + Tool or workflow loading from remote or local locations ------------------------------------------------------- @@ -379,6 +401,50 @@ Description: ================================================================== - `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__ - `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__ + Overriding workflow requirements at load time + --------------------------------------------- + + Sometimes a workflow needs additional requirements to run in a particular + environment or with a particular dataset. To avoid the need to modify the + underlying workflow, cwltool supports requirement "overrides". + + The format of the "overrides" object is a mapping of item identifier (workflow, + workflow step, or command line tool) followed by a list of ProcessRequirements + that should be applied. + + .. code:: yaml + + cwltool:overrides: + echo.cwl: + - class: EnvVarRequirement + envDef: + MESSAGE: override_value + + + Overrides can be specified either on the command line, or as part of the job + input document. Workflow steps are identified using the name of the workflow + file followed by the step name as a document fragment identifier "#id". + Override identifiers are relative to the toplevel workflow document. + + .. code:: bash + + cwltool --overrides overrides.yml my-tool.cwl my-job.yml + + .. code:: yaml + + input_parameter1: value1 + input_parameter2: value2 + cwltool:overrides: + workflow.cwl#step1: + - class: EnvVarRequirement + envDef: + MESSAGE: override_value + + .. code:: bash + + cwltool my-tool.cwl my-job-with-overrides.yml + + CWL Tool Control Flow --------------------- @@ -511,21 +577,6 @@ Description: ================================================================== Handler object for logging. - Running user-space implementations of Docker - -------------------------------------------- - - Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option. - - Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images): - - For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions). - - Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests: - - ``` - cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json - ``` - Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console diff --git a/README.rst b/README.rst index e44935e..f817cf0 100644 --- a/README.rst +++ b/README.rst @@ -31,7 +31,7 @@ It is highly recommended to setup virtual environment before installing `cwltool virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well source venv/bin/activate # Activate environment before installing `cwltool` -1. Installing the official package from PyPi (will install "cwltool" package as +Installing the official package from PyPi (will install "cwltool" package as well) .. code:: bash @@ -44,7 +44,7 @@ If installing alongside another CWL implementation then pip install cwltool -2. To install from source +Or you can install from source: .. code:: bash @@ -62,9 +62,16 @@ Running tests locally - Running basic tests ``(/tests)``: -We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_ -to run various tests in all supported Python environments. -You can run the test suite by simply running the following in the terminal: +To run the basis tests after installing `cwltool` execute the following: + +.. code:: bash + + pip install pytest mock + py.test --ignore cwltool/schemas/ --pyarg cwltool + +To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments +first downloading the complete code repository (see the ``git clone`` instructions above) and then run +the following in the terminal: ``pip install tox; tox`` List of all environment can be seen using: @@ -106,6 +113,21 @@ and ``--tmp-outdir-prefix`` to somewhere under ``/Users``:: .. |Build Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance :target: https://ci.commonwl.org/job/cwltool-conformance/ +Running user-space implementations of Docker +-------------------------------------------- + +Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option. + +Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images): + +For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions). + +Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests: + +.. code:: bash + + cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json + Tool or workflow loading from remote or local locations ------------------------------------------------------- @@ -369,6 +391,50 @@ at the following links: - `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__ - `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__ +Overriding workflow requirements at load time +--------------------------------------------- + +Sometimes a workflow needs additional requirements to run in a particular +environment or with a particular dataset. To avoid the need to modify the +underlying workflow, cwltool supports requirement "overrides". + +The format of the "overrides" object is a mapping of item identifier (workflow, +workflow step, or command line tool) followed by a list of ProcessRequirements +that should be applied. + +.. code:: yaml + + cwltool:overrides: + echo.cwl: + - class: EnvVarRequirement + envDef: + MESSAGE: override_value + + +Overrides can be specified either on the command line, or as part of the job +input document. Workflow steps are identified using the name of the workflow +file followed by the step name as a document fragment identifier "#id". +Override identifiers are relative to the toplevel workflow document. + +.. code:: bash + + cwltool --overrides overrides.yml my-tool.cwl my-job.yml + +.. code:: yaml + + input_parameter1: value1 + input_parameter2: value2 + cwltool:overrides: + workflow.cwl#step1: + - class: EnvVarRequirement + envDef: + MESSAGE: override_value + +.. code:: bash + + cwltool my-tool.cwl my-job-with-overrides.yml + + CWL Tool Control Flow --------------------- @@ -500,18 +566,3 @@ logger_handler logging.Handler Handler object for logging. - -Running user-space implementations of Docker --------------------------------------------- - -Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option. - -Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images): - -For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions). - -Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests: - -``` -cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json -``` diff --git a/cwltool.egg-info/PKG-INFO b/cwltool.egg-info/PKG-INFO index d3ce41e..694e059 100644 --- a/cwltool.egg-info/PKG-INFO +++ b/cwltool.egg-info/PKG-INFO @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: cwltool -Version: 1.0.20171107133715 +Version: 1.0.20171221100033 Summary: Common workflow language reference implementation Home-page: https://github.com/common-workflow-language/cwltool Author: Common workflow language working group @@ -41,7 +41,7 @@ Description: ================================================================== virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well source venv/bin/activate # Activate environment before installing `cwltool` - 1. Installing the official package from PyPi (will install "cwltool" package as + Installing the official package from PyPi (will install "cwltool" package as well) .. code:: bash @@ -54,7 +54,7 @@ Description: ================================================================== pip install cwltool - 2. To install from source + Or you can install from source: .. code:: bash @@ -72,9 +72,16 @@ Description: ================================================================== - Running basic tests ``(/tests)``: - We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_ - to run various tests in all supported Python environments. - You can run the test suite by simply running the following in the terminal: + To run the basis tests after installing `cwltool` execute the following: + + .. code:: bash + + pip install pytest mock + py.test --ignore cwltool/schemas/ --pyarg cwltool + + To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments + first downloading the complete code repository (see the ``git clone`` instructions above) and then run + the following in the terminal: ``pip install tox; tox`` List of all environment can be seen using: @@ -116,6 +123,21 @@ Description: ================================================================== .. |Build Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance :target: https://ci.commonwl.org/job/cwltool-conformance/ + Running user-space implementations of Docker + -------------------------------------------- + + Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option. + + Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images): + + For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions). + + Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests: + + .. code:: bash + + cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json + Tool or workflow loading from remote or local locations ------------------------------------------------------- @@ -379,6 +401,50 @@ Description: ================================================================== - `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__ - `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__ + Overriding workflow requirements at load time + --------------------------------------------- + + Sometimes a workflow needs additional requirements to run in a particular + environment or with a particular dataset. To avoid the need to modify the + underlying workflow, cwltool supports requirement "overrides". + + The format of the "overrides" object is a mapping of item identifier (workflow, + workflow step, or command line tool) followed by a list of ProcessRequirements + that should be applied. + + .. code:: yaml + + cwltool:overrides: + echo.cwl: + - class: EnvVarRequirement + envDef: + MESSAGE: override_value + + + Overrides can be specified either on the command line, or as part of the job + input document. Workflow steps are identified using the name of the workflow + file followed by the step name as a document fragment identifier "#id". + Override identifiers are relative to the toplevel workflow document. + + .. code:: bash + + cwltool --overrides overrides.yml my-tool.cwl my-job.yml + + .. code:: yaml + + input_parameter1: value1 + input_parameter2: value2 + cwltool:overrides: + workflow.cwl#step1: + - class: EnvVarRequirement + envDef: + MESSAGE: override_value + + .. code:: bash + + cwltool my-tool.cwl my-job-with-overrides.yml + + CWL Tool Control Flow --------------------- @@ -511,21 +577,6 @@ Description: ================================================================== Handler object for logging. - Running user-space implementations of Docker - -------------------------------------------- - - Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option. - - Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images): - - For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions). - - Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests: - - ``` - cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json - ``` - Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console diff --git a/cwltool.egg-info/SOURCES.txt b/cwltool.egg-info/SOURCES.txt index f4bebd2..149d8d9 100644 --- a/cwltool.egg-info/SOURCES.txt +++ b/cwltool.egg-info/SOURCES.txt @@ -173,21 +173,33 @@ tests/test_ext.py tests/test_fetch.py tests/test_http_input.py tests/test_js_sandbox.py +tests/test_override.py tests/test_pack.py tests/test_pathmapper.py tests/test_rdfprint.py tests/test_relax_path_checks.py tests/test_toolargparse.py tests/util.py +tests/override/echo-job-ov.yml +tests/override/echo-job-ov2.yml +tests/override/echo-job.yml +tests/override/echo-wf.cwl +tests/override/echo.cwl +tests/override/ov.yml +tests/override/ov2.yml +tests/override/ov3.yml tests/tmp1/tmp2/tmp3/.gitkeep tests/wf/badout1.cwl tests/wf/badout2.cwl tests/wf/badout3.cwl tests/wf/cat.cwl +tests/wf/count-lines1-wf.cwl tests/wf/default_path.cwl tests/wf/echo.cwl tests/wf/empty.ttl tests/wf/expect_packed.cwl +tests/wf/formattest-job.json +tests/wf/formattest.cwl tests/wf/hello-workflow.cwl tests/wf/hello.txt tests/wf/hello_single_tool.cwl @@ -201,6 +213,7 @@ tests/wf/missing_cwlVersion.cwl tests/wf/mut.cwl tests/wf/mut2.cwl tests/wf/mut3.cwl +tests/wf/parseInt-tool.cwl tests/wf/revsort-job.json tests/wf/revsort.cwl tests/wf/revtool.cwl @@ -211,5 +224,8 @@ tests/wf/updatedir_inplace.cwl tests/wf/updateval.cwl tests/wf/updateval.py tests/wf/updateval_inplace.cwl +tests/wf/wc-job.json +tests/wf/wc-tool.cwl tests/wf/wffail.cwl +tests/wf/whale.txt tests/wf/wrong_cwlVersion.cwl \ No newline at end of file diff --git a/cwltool/draft2tool.py b/cwltool/draft2tool.py index bb56692..5d572bb 100644 --- a/cwltool/draft2tool.py +++ b/cwltool/draft2tool.py @@ -1,14 +1,16 @@ from __future__ import absolute_import import copy import hashlib +import locale import json import logging import os import re import shutil import tempfile -from functools import partial -from typing import Any, Callable, Dict, Generator, List, Optional, Set, Text, Union, cast +from functools import partial, cmp_to_key +from typing import (Any, Callable, Dict, Generator, List, Optional, Set, Text, + Union, cast) from six import string_types, u @@ -208,7 +210,7 @@ class CommandLineTool(Process): }) dockerReq = self.requirements[0] if default_container == windows_default_container_id and use_container and onWindows(): - _logger.warning(DEFAULT_CONTAINER_MSG%(windows_default_container_id, windows_default_container_id)) + _logger.warning(DEFAULT_CONTAINER_MSG % (windows_default_container_id, windows_default_container_id)) if dockerReq and use_container: return DockerCommandLineJob() @@ -523,8 +525,8 @@ class CommandLineTool(Process): for i, port in enumerate(ports): def makeWorkflowException(msg): return WorkflowException( - u"Error collecting output for parameter '%s':\n%s" - % (shortname(port["id"]), msg)) + u"Error collecting output for parameter '%s':\n%s" + % (shortname(port["id"]), msg)) with SourceLine(ports, i, makeWorkflowException, debug): fragment = shortname(port["id"]) ret[fragment] = self.collect_output(port, builder, outdir, fs_access, @@ -575,16 +577,25 @@ class CommandLineTool(Process): elif gb == ".": gb = outdir elif gb.startswith("/"): - raise WorkflowException("glob patterns must not start with '/'") + raise WorkflowException( + "glob patterns must not start with '/'") try: prefix = fs_access.glob(outdir) r.extend([{"location": g, - "path": fs_access.join(builder.outdir, g[len(prefix[0])+1:]), + "path": fs_access.join(builder.outdir, + g[len(prefix[0])+1:]), "basename": os.path.basename(g), - "nameroot": os.path.splitext(os.path.basename(g))[0], - "nameext": os.path.splitext(os.path.basename(g))[1], - "class": "File" if fs_access.isfile(g) else "Directory"} - for g in fs_access.glob(fs_access.join(outdir, gb))]) + "nameroot": os.path.splitext( + os.path.basename(g))[0], + "nameext": os.path.splitext( + os.path.basename(g))[1], + "class": "File" if fs_access.isfile(g) + else "Directory"} + for g in sorted(fs_access.glob( + fs_access.join(outdir, gb)), + key=cmp_to_key(cast( + Callable[[Text, Text], + int], locale.strcoll)))]) except (OSError, IOError) as e: _logger.warning(Text(e)) except: diff --git a/cwltool/load_tool.py b/cwltool/load_tool.py index 8f30b90..59fa62e 100644 --- a/cwltool/load_tool.py +++ b/cwltool/load_tool.py @@ -8,7 +8,8 @@ import re import uuid import hashlib import json -from typing import Any, Callable, Dict, List, Text, Tuple, Union, cast +import copy +from typing import Any, Callable, Dict, List, Text, Tuple, Union, cast, Iterable import requests.sessions from six import itervalues, string_types @@ -23,19 +24,65 @@ from six.moves import urllib from . import process, update from .errors import WorkflowException -from .process import Process, shortname +from .process import Process, shortname, get_schema from .update import ALLUPDATES _logger = logging.getLogger("cwltool") jobloaderctx = { u"cwl": "https://w3id.org/cwl/cwl#";, + u"cwltool": "http://commonwl.org/cwltool#";, u"path": {u"@type": u"@id"}, u"location": {u"@type": u"@id"}, u"format": {u"@type": u"@id"}, u"id": u"@id" } + +overrides_ctx = { + u"overrideTarget": {u"@type": u"@id"}, + u"cwltool": "http://commonwl.org/cwltool#";, + u"overrides": { + "@id": "cwltool:overrides", + "mapSubject": "overrideTarget", + "mapPredicate": "override" + }, + u"override": { + "@id": "cwltool:override", + "mapSubject": "class" + } +} # type: Dict[Text, Union[Dict[Any, Any], Text, Iterable[Text]]] + +def resolve_tool_uri(argsworkflow, # type: Text + resolver=None, # type: Callable[[Loader, Union[Text, Dict[Text, Any]]], Text] + fetcher_constructor=None, + # type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher] + document_loader=None # type: Loader +): + # type: (...) -> Tuple[Text, Text] + + uri = None # type: Text + split = urllib.parse.urlsplit(argsworkflow) + # In case of Windows path, urlsplit misjudge Drive letters as scheme, here we are skipping that + if split.scheme and split.scheme in [u'http',u'https',u'file']: + uri = argsworkflow + elif os.path.exists(os.path.abspath(argsworkflow)): + uri = file_uri(str(os.path.abspath(argsworkflow))) + elif resolver: + if document_loader is None: + document_loader = Loader(jobloaderctx, fetcher_constructor=fetcher_constructor) # type: ignore + uri = resolver(document_loader, argsworkflow) + + if uri is None: + raise ValidationException("Not found: '%s'" % argsworkflow) + + if argsworkflow != uri: + _logger.info("Resolved '%s' to '%s'", argsworkflow, uri) + + fileuri = urllib.parse.urldefrag(uri)[0] + return uri, fileuri + + def fetch_document(argsworkflow, # type: Union[Text, Dict[Text, Any]] resolver=None, # type: Callable[[Loader, Union[Text, Dict[Text, Any]]], Text] fetcher_constructor=None @@ -49,22 +96,7 @@ def fetch_document(argsworkflow, # type: Union[Text, Dict[Text, Any]] uri = None # type: Text workflowobj = None # type: CommentedMap if isinstance(argsworkflow, string_types): - split = urllib.parse.urlsplit(argsworkflow) - # In case of Windows path, urlsplit misjudge Drive letters as scheme, here we are skipping that - if split.scheme and split.scheme in [u'http',u'https',u'file']: - uri = argsworkflow - elif os.path.exists(os.path.abspath(argsworkflow)): - uri = file_uri(str(os.path.abspath(argsworkflow))) - elif resolver: - uri = resolver(document_loader, argsworkflow) - - if uri is None: - raise ValidationException("Not found: '%s'" % argsworkflow) - - if argsworkflow != uri: - _logger.info("Resolved '%s' to '%s'", argsworkflow, uri) - - fileuri = urllib.parse.urldefrag(uri)[0] + uri, fileuri = resolve_tool_uri(argsworkflow, resolver=resolver, document_loader=document_loader) workflowobj = document_loader.fetch(fileuri) elif isinstance(argsworkflow, dict): uri = "#" + Text(id(argsworkflow)) @@ -139,8 +171,9 @@ def validate_document(document_loader, # type: Loader strict=True, # type: bool preprocess_only=False, # type: bool fetcher_constructor=None, - skip_schemas=None + skip_schemas=None, # type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher] + overrides=None # type: List[Dict] ): # type: (...) -> Tuple[Loader, Names, Union[Dict[Text, Any], List[Dict[Text, Any]]], Dict[Text, Any], Text] """Validate a CWL document.""" @@ -155,9 +188,15 @@ def validate_document(document_loader, # type: Loader jobobj = None if "cwl:tool" in workflowobj: - jobobj, _ = document_loader.resolve_all(workflowobj, uri) + job_loader = Loader(jobloaderctx, fetcher_constructor=fetcher_constructor) # type: ignore + jobobj, _ = job_loader.resolve_all(workflowobj, uri) uri = urllib.parse.urljoin(uri, workflowobj["https://w3id.org/cwl/cwl#tool";]) del cast(dict, jobobj)["https://w3id.org/cwl/cwl#tool";] + + if "http://commonwl.org/cwltool#overrides"; in jobobj: + overrides.extend(resolve_overrides(jobobj, uri, uri)) + del jobobj["http://commonwl.org/cwltool#overrides";] + workflowobj = fetch_document(uri, fetcher_constructor=fetcher_constructor)[1] fileuri = urllib.parse.urldefrag(uri)[0] @@ -225,6 +264,9 @@ def validate_document(document_loader, # type: Loader if jobobj: metadata[u"cwl:defaults"] = jobobj + if overrides: + metadata[u"cwltool:overrides"] = overrides + return document_loader, avsc_names, processobj, metadata, uri @@ -239,10 +281,13 @@ def make_tool(document_loader, # type: Loader """Make a Python CWL object.""" resolveduri = document_loader.resolve_ref(uri)[0] + processobj = None if isinstance(resolveduri, list): - if len(resolveduri) == 1: - processobj = resolveduri[0] - else: + for obj in resolveduri: + if obj['id'].endswith('#main'): + processobj = obj + break + if not processobj: raise WorkflowException( u"Tool file contains graph of multiple objects, must specify " "one of #%s" % ", #".join( @@ -277,7 +322,8 @@ def load_tool(argsworkflow, # type: Union[Text, Dict[Text, Any]] enable_dev=False, # type: bool strict=True, # type: bool resolver=None, # type: Callable[[Loader, Union[Text, Dict[Text, Any]]], Text] - fetcher_constructor=None # type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher] + fetcher_constructor=None, # type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher] + overrides=None ): # type: (...) -> Process @@ -285,6 +331,20 @@ def load_tool(argsworkflow, # type: Union[Text, Dict[Text, Any]] fetcher_constructor=fetcher_constructor) document_loader, avsc_names, processobj, metadata, uri = validate_document( document_loader, workflowobj, uri, enable_dev=enable_dev, - strict=strict, fetcher_constructor=fetcher_constructor) + strict=strict, fetcher_constructor=fetcher_constructor, + overrides=overrides) return make_tool(document_loader, avsc_names, metadata, uri, makeTool, kwargs if kwargs else {}) + +def resolve_overrides(ov, ov_uri, baseurl): # type: (CommentedMap, Text, Text) -> List[Dict[Text, Any]] + ovloader = Loader(overrides_ctx) + ret, _ = ovloader.resolve_all(ov, baseurl) + if not isinstance(ret, CommentedMap): + raise Exception("Expected CommentedMap, got %s" % type(ret)) + cwl_docloader = get_schema("v1.0")[0] + cwl_docloader.resolve_all(ret, ov_uri) + return ret["overrides"] + +def load_overrides(ov, base_url): # type: (Text, Text) -> List[Dict[Text, Any]] + ovloader = Loader(overrides_ctx) + return resolve_overrides(ovloader.fetch(ov), ov, base_url) diff --git a/cwltool/main.py b/cwltool/main.py index f57cafb..8f58cb1 100755 --- a/cwltool/main.py +++ b/cwltool/main.py @@ -11,7 +11,7 @@ import os import sys import tempfile from typing import (IO, Any, AnyStr, Callable, Dict, List, Sequence, Text, Tuple, - Union, cast) + Union, cast, Mapping, MutableMapping, Iterable) import pkg_resources # part of setuptools import requests @@ -27,7 +27,8 @@ from . import draft2tool, workflow from .builder import Builder from .cwlrdf import printdot, printrdf from .errors import UnsupportedRequirement, WorkflowException -from .load_tool import fetch_document, make_tool, validate_document, jobloaderctx +from .load_tool import (resolve_tool_uri, fetch_document, make_tool, validate_document, + jobloaderctx, resolve_overrides, load_overrides) from .mutation import MutationManager from .pack import pack from .pathmapper import (adjustDirObjs, adjustFileObjs, get_listing, @@ -36,7 +37,9 @@ from .process import (Process, cleanIntermediate, normalizeFilesDirs, relocateOutputs, scandeps, shortname, use_custom_schema, use_standard_schema) from .resolver import ga4gh_tool_registries, tool_resolver -from .software_requirements import DependenciesConfiguration, get_container_from_software_requirements, SOFTWARE_REQUIREMENTS_ENABLED +from .software_requirements import (DependenciesConfiguration, + get_container_from_software_requirements, + SOFTWARE_REQUIREMENTS_ENABLED) from .stdfsaccess import StdFsAccess from .update import ALLUPDATES, UPDATES from .utils import onWindows, windows_default_container_id @@ -238,6 +241,10 @@ def arg_parser(): # type: () -> argparse.ArgumentParser parser.add_argument("--no-read-only", action="store_true", default=False, help="Do not set root directoy in the" " container as read-only", dest="no_read_only") + + parser.add_argument("--overrides", type=str, + default=None, help="Read process requirement overrides from file.") + parser.add_argument("workflow", type=Text, nargs="?", default=None) parser.add_argument("job_order", nargs=argparse.REMAINDER) @@ -509,14 +516,17 @@ def generate_input_template(tool): -def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False, - stdout=sys.stdout, make_fs_access=None, fetcher_constructor=None): - # type: (argparse.Namespace, Process, IO[Any], bool, bool, IO[Any], Callable[[Text], StdFsAccess], Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher]) -> Union[int, Tuple[Dict[Text, Any], Text]] +def load_job_order(args, # type: argparse.Namespace + stdin, # type: IO[Any] + fetcher_constructor, # Fetcher + overrides, # type: List[Dict[Text, Any]] + tool_file_uri # type: Text +): + # type: (...) -> Tuple[Dict[Text, Any], Text, Loader] job_order_object = None _jobloaderctx = jobloaderctx.copy() - _jobloaderctx.update(t.metadata.get("$namespaces", {})) loader = Loader(_jobloaderctx, fetcher_constructor=fetcher_constructor) # type: ignore if len(args.job_order) == 1 and args.job_order[0][0] != "-": @@ -531,14 +541,31 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False, input_basedir = args.basedir if args.basedir else os.getcwd() elif job_order_file: input_basedir = args.basedir if args.basedir else os.path.abspath(os.path.dirname(job_order_file)) - try: - job_order_object, _ = loader.resolve_ref(job_order_file, checklinks=False) - except Exception as e: - _logger.error(Text(e), exc_info=args.debug) - return 1 - toolparser = None - else: + job_order_object, _ = loader.resolve_ref(job_order_file, checklinks=False) + + if job_order_object and "http://commonwl.org/cwltool#overrides"; in job_order_object: + overrides.extend(resolve_overrides(job_order_object, file_uri(job_order_file), tool_file_uri)) + del job_order_object["http://commonwl.org/cwltool#overrides";] + + if not job_order_object: input_basedir = args.basedir if args.basedir else os.getcwd() + + return (job_order_object, input_basedir, loader) + + +def init_job_order(job_order_object, # type: MutableMapping[Text, Any] + args, # type: argparse.Namespace + t, # type: Process + print_input_deps=False, # type: bool + relative_deps=False, # type: bool + stdout=sys.stdout, # type: IO[Any] + make_fs_access=None, # type: Callable[[Text], StdFsAccess] + loader=None, # type: Loader + input_basedir="" # type: Text +): + # (...) -> Tuple[Dict[Text, Any], Text] + + if not job_order_object: namemap = {} # type: Dict[Text, Text] records = [] # type: List[Text] toolparser = generate_parser( @@ -546,7 +573,7 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False, if toolparser: if args.tool_help: toolparser.print_help() - return 0 + exit(0) cmd_line = vars(toolparser.parse_args(args.job_order)) for record_name in records: record = {} @@ -560,9 +587,7 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False, if cmd_line["job_order"]: try: - input_basedir = args.basedir if args.basedir else os.path.abspath( - os.path.dirname(cmd_line["job_order"])) - job_order_object = loader.resolve_ref(cmd_line["job_order"]) + job_order_object = cast(MutableMapping, loader.resolve_ref(cmd_line["job_order"])[0]) except Exception as e: _logger.error(Text(e), exc_info=args.debug) return 1 @@ -590,12 +615,12 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False, toolparser.print_help() _logger.error("") _logger.error("Input object required, use --help for details") - return 1 + exit(1) if print_input_deps: printdeps(job_order_object, loader, stdout, relative_deps, "", - basedir=file_uri(input_basedir + "/")) - return 0 + basedir=file_uri(str(input_basedir) + "/")) + exit(0) def pathToLoc(p): if "location" not in p and "path" in p: @@ -613,8 +638,16 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False, else: return # best effort + ns = {} # type: Dict[Text, Union[Dict[Any, Any], Text, Iterable[Text]]] + ns.update(t.metadata.get("$namespaces", {})) + ld = Loader(ns) + def expand_formats(p): + if "format" in p: + p["format"] = ld.expand_url(p["format"], "") + visit_class(job_order_object, ("File", "Directory"), pathToLoc) - visit_class(job_order_object, ("File"), addSizes) + visit_class(job_order_object, ("File",), addSizes) + visit_class(job_order_object, ("File",), expand_formats) adjustDirObjs(job_order_object, trim_listing) normalizeFilesDirs(job_order_object) @@ -623,7 +656,7 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False, if "id" in job_order_object: del job_order_object["id"] - return (job_order_object, input_basedir) + return job_order_object def makeRelative(base, ob): @@ -637,7 +670,7 @@ def makeRelative(base, ob): def printdeps(obj, document_loader, stdout, relative_deps, uri, basedir=None): - # type: (Dict[Text, Any], Loader, IO[Any], bool, Text, Text) -> None + # type: (Mapping[Text, Any], Loader, IO[Any], bool, Text, Text) -> None deps = {"class": "File", "location": uri} # type: Dict[Text, Any] @@ -699,7 +732,7 @@ def main(argsl=None, # type: List[str] stdout=sys.stdout, # type: IO[Any] stderr=sys.stderr, # type: IO[Any] versionfunc=versionstring, # type: Callable[[], Text] - job_order_object=None, # type: Union[Tuple[Dict[Text, Any], Text], int] + job_order_object=None, # type: MutableMapping[Text, Any] make_fs_access=StdFsAccess, # type: Callable[[Text], StdFsAccess] fetcher_constructor=None, # type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher] resolver=tool_resolver, @@ -757,7 +790,8 @@ def main(argsl=None, # type: List[str] 'enable_ga4gh_tool_registry': False, 'ga4gh_tool_registries': [], 'find_default_container': None, - 'make_template': False + 'make_template': False, + 'overrides': None }): if not hasattr(args, k): setattr(args, k, v) @@ -802,8 +836,26 @@ def main(argsl=None, # type: List[str] else: use_standard_schema("v1.0") + uri, tool_file_uri = resolve_tool_uri(args.workflow, + resolver=resolver, + fetcher_constructor=fetcher_constructor) + + overrides = [] # type: List[Dict[Text, Any]] + + try: + job_order_object, input_basedir, jobloader = load_job_order(args, + stdin, + fetcher_constructor, + overrides, + tool_file_uri) + except Exception as e: + _logger.error(Text(e), exc_info=args.debug) + + if args.overrides: + overrides.extend(load_overrides(file_uri(os.path.abspath(args.overrides)), tool_file_uri)) + try: - document_loader, workflowobj, uri = fetch_document(args.workflow, resolver=resolver, + document_loader, workflowobj, uri = fetch_document(uri, resolver=resolver, fetcher_constructor=fetcher_constructor) if args.print_deps: @@ -815,16 +867,15 @@ def main(argsl=None, # type: List[str] enable_dev=args.enable_dev, strict=args.strict, preprocess_only=args.print_pre or args.pack, fetcher_constructor=fetcher_constructor, - skip_schemas=args.skip_schemas) - - if args.pack: - stdout.write(print_pack(document_loader, processobj, uri, metadata)) - return 0 + skip_schemas=args.skip_schemas, + overrides=overrides) if args.print_pre: stdout.write(json.dumps(processobj, indent=4)) return 0 + overrides.extend(metadata.get("cwltool:overrides", [])) + conf_file = getattr(args, "beta_dependency_resolvers_configuration", None) # Text use_conda_dependencies = getattr(args, "beta_conda_dependencies", None) # Text @@ -836,6 +887,7 @@ def main(argsl=None, # type: List[str] make_tool_kwds["job_script_provider"] = dependencies_configuration make_tool_kwds["find_default_container"] = functools.partial(find_default_container, args) + make_tool_kwds["overrides"] = overrides tool = make_tool(document_loader, avsc_names, metadata, uri, makeTool, make_tool_kwds) @@ -846,6 +898,11 @@ def main(argsl=None, # type: List[str] return 0 if args.validate: + _logger.info("Tool definition is valid") + return 0 + + if args.pack: + stdout.write(print_pack(document_loader, processobj, uri, metadata)) return 0 if args.print_rdf: @@ -893,13 +950,13 @@ def main(argsl=None, # type: List[str] setattr(args, "tmp_outdir_prefix", args.cachedir) try: - if job_order_object is None: - job_order_object = load_job_order(args, tool, stdin, - print_input_deps=args.print_input_deps, - relative_deps=args.relative_deps, - stdout=stdout, - make_fs_access=make_fs_access, - fetcher_constructor=fetcher_constructor) + job_order_object = init_job_order(job_order_object, args, tool, + print_input_deps=args.print_input_deps, + relative_deps=args.relative_deps, + stdout=stdout, + make_fs_access=make_fs_access, + loader=jobloader, + input_basedir=input_basedir) except SystemExit as e: return e.code @@ -907,10 +964,10 @@ def main(argsl=None, # type: List[str] return job_order_object try: - setattr(args, 'basedir', job_order_object[1]) + setattr(args, 'basedir', input_basedir) del args.workflow del args.job_order - (out, status) = executor(tool, job_order_object[0], + (out, status) = executor(tool, job_order_object, makeTool=makeTool, select_resources=selectResources, make_fs_access=make_fs_access, diff --git a/cwltool/pack.py b/cwltool/pack.py index 898f548..dc2c414 100644 --- a/cwltool/pack.py +++ b/cwltool/pack.py @@ -1,9 +1,11 @@ from __future__ import absolute_import import copy +import re from typing import Any, Callable, Dict, List, Set, Text, Union, cast -from schema_salad.ref_resolver import Loader +from schema_salad.ref_resolver import Loader, SubLoader from six.moves import urllib +from ruamel.yaml.comments import CommentedSeq, CommentedMap from .process import shortname, uniquename import six @@ -64,7 +66,12 @@ def replace_refs(d, rewrite, stem, newstem): if v in rewrite: d[s] = rewrite[v] elif v.startswith(stem): - d[s] = newstem + v[len(stem):] + id_ = v[len(stem):] + # prevent appending newstems if tool is already packed + if id_.startswith(newstem.strip("#")): + d[s] = "#" + id_ + else: + d[s] = newstem + id_ replace_refs(v, rewrite, stem, newstem) def import_embed(d, seen): @@ -84,12 +91,25 @@ def import_embed(d, seen): seen.add(this) break - for v in d.values(): - import_embed(v, seen) + for k in sorted(d.keys()): + import_embed(d[k], seen) def pack(document_loader, processobj, uri, metadata): # type: (Loader, Union[Dict[Text, Any], List[Dict[Text, Any]]], Text, Dict[Text, Text]) -> Dict[Text, Any] + + document_loader = SubLoader(document_loader) + document_loader.idx = {} + if isinstance(processobj, dict): + document_loader.idx[processobj["id"]] = CommentedMap(six.iteritems(processobj)) + elif isinstance(processobj, list): + path, frag = urllib.parse.urldefrag(uri) + for po in processobj: + if not frag: + if po["id"].endswith("#main"): + uri = po["id"] + document_loader.idx[po["id"]] = CommentedMap(six.iteritems(po)) + def loadref(b, u): # type: (Text, Text) -> Union[Dict, List, Text] return document_loader.resolve_ref(u, base_url=b)[0] @@ -111,7 +131,8 @@ def pack(document_loader, processobj, uri, metadata): if r == mainuri: rewrite[r] = "#main" elif r.startswith(mainuri) and r[len(mainuri)] in ("#", "/"): - pass + path, frag = urllib.parse.urldefrag(r) + rewrite[r] = "#"+frag else: path, frag = urllib.parse.urldefrag(r) if path == mainpath: @@ -128,10 +149,14 @@ def pack(document_loader, processobj, uri, metadata): packed = {"$graph": [], "cwlVersion": metadata["cwlVersion"] } # type: Dict[Text, Any] + namespaces = metadata.get('$namespaces', None) schemas = set() # type: Set[Text] for r in sorted(runs): dcr, metadata = document_loader.resolve_ref(r) + if isinstance(dcr, CommentedSeq): + dcr = dcr[0] + dcr = cast(CommentedMap, dcr) if not isinstance(dcr, dict): continue for doc in (dcr, metadata): @@ -161,5 +186,7 @@ def pack(document_loader, processobj, uri, metadata): # duplicate 'cwlVersion' inside $graph when there is a single item # because we're printing contents inside '$graph' rather than whole dict packed["$graph"][0]["cwlVersion"] = packed["cwlVersion"] + if namespaces: + packed["$graph"][0]["$namespaces"] = dict(cast(Dict, namespaces)) return packed diff --git a/cwltool/pathmapper.py b/cwltool/pathmapper.py index 7127d23..c3aed1e 100644 --- a/cwltool/pathmapper.py +++ b/cwltool/pathmapper.py @@ -10,7 +10,7 @@ from tempfile import NamedTemporaryFile import requests from cachecontrol import CacheControl from cachecontrol.caches import FileCache -from typing import Any, Callable, Dict, Iterable, List, Set, Text, Tuple, Union +from typing import Any, Callable, Dict, Iterable, List, Set, Text, Tuple, Union, MutableMapping import schema_salad.validate as validate from schema_salad.ref_resolver import uri_file_path @@ -60,7 +60,7 @@ def adjustDirObjs(rec, op): visit_class(rec, ("Directory",), op) def normalizeFilesDirs(job): - # type: (Union[List[Dict[Text, Any]], Dict[Text, Any]]) -> None + # type: (Union[List[Dict[Text, Any]], MutableMapping[Text, Any]]) -> None def addLocation(d): if "location" not in d: if d["class"] == "File" and ("contents" not in d): diff --git a/cwltool/process.py b/cwltool/process.py index bdbee58..e9356f0 100644 --- a/cwltool/process.py +++ b/cwltool/process.py @@ -422,6 +422,15 @@ def avroize_type(field_type, name_prefix=""): avroize_type(field_type["items"], name_prefix) return field_type +def get_overrides(overrides, toolid): # type: (List[Dict[Text, Any]], Text) -> List[Dict[Text, Any]] + req = [] # type: List[Dict[Text, Any]] + if not isinstance(overrides, list): + raise validate.ValidationException("Expected overrides to be a list, but was %s" % type(overrides)) + for ov in overrides: + if ov["overrideTarget"] == toolid: + req.extend(ov["override"]) + return req + class Process(six.with_metaclass(abc.ABCMeta, object)): def __init__(self, toolpath_object, **kwargs): # type: (Dict[Text, Any], **Any) -> None @@ -456,7 +465,9 @@ class Process(six.with_metaclass(abc.ABCMeta, object)): else: self.names = names self.tool = toolpath_object - self.requirements = kwargs.get("requirements", []) + self.tool.get("requirements", []) + self.requirements = (kwargs.get("requirements", []) + + self.tool.get("requirements", []) + + get_overrides(kwargs.get("overrides", []), self.tool["id"])) self.hints = kwargs.get("hints", []) + self.tool.get("hints", []) self.formatgraph = None # type: Graph if "loader" in kwargs: diff --git a/cwltool/workflow.py b/cwltool/workflow.py index 2081430..ea9def2 100644 --- a/cwltool/workflow.py +++ b/cwltool/workflow.py @@ -15,7 +15,7 @@ from schema_salad.sourceline import SourceLine, cmap from . import draft2tool, expression from .errors import WorkflowException from .load_tool import load_tool -from .process import Process, shortname, uniquename +from .process import Process, shortname, uniquename, get_overrides from .utils import aslist import six from six.moves import range @@ -521,6 +521,8 @@ class Workflow(Process): try: self.steps.append(WorkflowStep(step, n, **kwargs)) except validate.ValidationException as v: + if _logger.isEnabledFor(logging.DEBUG): + _logger.exception("Validation failed at") validation_errors.append(v) if validation_errors: @@ -623,8 +625,7 @@ def static_checker(workflow_inputs, workflow_outputs, step_inputs, step_outputs) all_exception_msg = "\n".join(exception_msgs) if warnings: - _logger.warning("Workflow checker warning:") - _logger.warning(all_warning_msg) + _logger.warning("Workflow checker warning:\n%s" % all_warning_msg) if exceptions: raise validate.ValidationException(all_exception_msg) @@ -666,7 +667,9 @@ class WorkflowStep(Process): else: self.id = "#step" + Text(pos) - kwargs["requirements"] = kwargs.get("requirements", []) + toolpath_object.get("requirements", []) + kwargs["requirements"] = (kwargs.get("requirements", []) + + toolpath_object.get("requirements", []) + + get_overrides(kwargs.get("overrides", []), self.id)) kwargs["hints"] = kwargs.get("hints", []) + toolpath_object.get("hints", []) try: @@ -678,7 +681,8 @@ class WorkflowStep(Process): enable_dev=kwargs.get("enable_dev"), strict=kwargs.get("strict"), fetcher_constructor=kwargs.get("fetcher_constructor"), - resolver=kwargs.get("resolver")) + resolver=kwargs.get("resolver"), + overrides=kwargs.get("overrides")) except validate.ValidationException as v: raise WorkflowException( u"Tool definition %s failed validation:\n%s" % diff --git a/setup.cfg b/setup.cfg index 8336837..773cc2f 100644 --- a/setup.cfg +++ b/setup.cfg @@ -13,6 +13,6 @@ addopts = --ignore cwltool/schemas testpaths = tests [egg_info] -tag_build = .20171107133715 +tag_build = .20171221100033 tag_date = 0 diff --git a/tests/override/echo-job-ov.yml b/tests/override/echo-job-ov.yml new file mode 100644 index 0000000..6170fd2 --- /dev/null +++ b/tests/override/echo-job-ov.yml @@ -0,0 +1,6 @@ +m1: zing +cwltool:overrides: + echo.cwl: + - class: EnvVarRequirement + envDef: + MESSAGE: hello3 diff --git a/tests/override/echo-job-ov2.yml b/tests/override/echo-job-ov2.yml new file mode 100644 index 0000000..3e8da10 --- /dev/null +++ b/tests/override/echo-job-ov2.yml @@ -0,0 +1,7 @@ +m1: zing +cwltool:overrides: + echo.cwl: + - class: EnvVarRequirement + envDef: + MESSAGE: hello4 +cwl:tool: echo.cwl diff --git a/tests/override/echo-job.yml b/tests/override/echo-job.yml new file mode 100644 index 0000000..b1ac4c4 --- /dev/null +++ b/tests/override/echo-job.yml @@ -0,0 +1 @@ +m1: zing \ No newline at end of file diff --git a/tests/override/echo-wf.cwl b/tests/override/echo-wf.cwl new file mode 100644 index 0000000..a9f86af --- /dev/null +++ b/tests/override/echo-wf.cwl @@ -0,0 +1,14 @@ +cwlVersion: v1.0 +class: Workflow +inputs: + m1: string +outputs: + out: + type: string + outputSource: step1/out +steps: + step1: + in: + m1: m1 + out: [out] + run: echo.cwl diff --git a/tests/override/echo.cwl b/tests/override/echo.cwl new file mode 100644 index 0000000..1c6c19a --- /dev/null +++ b/tests/override/echo.cwl @@ -0,0 +1,19 @@ +cwlVersion: v1.0 +class: CommandLineTool +requirements: + ShellCommandRequirement: {} +hints: + EnvVarRequirement: + envDef: + MESSAGE: hello1 +inputs: + m1: string +outputs: + - id: out + type: string + outputBinding: + glob: out.txt + loadContents: true + outputEval: $(self[0].contents) +arguments: ["echo", "-n", $(inputs.m1), {shellQuote: false, valueFrom: "$MESSAGE"}] +stdout: out.txt diff --git a/tests/override/ov.yml b/tests/override/ov.yml new file mode 100644 index 0000000..6a2d903 --- /dev/null +++ b/tests/override/ov.yml @@ -0,0 +1,5 @@ +cwltool:overrides: + echo.cwl: + - class: EnvVarRequirement + envDef: + MESSAGE: hello2 diff --git a/tests/override/ov2.yml b/tests/override/ov2.yml new file mode 100644 index 0000000..05c90ba --- /dev/null +++ b/tests/override/ov2.yml @@ -0,0 +1,5 @@ +cwltool:overrides: + "echo-wf.cwl#step1": + - class: EnvVarRequirement + envDef: + MESSAGE: hello5 diff --git a/tests/override/ov3.yml b/tests/override/ov3.yml new file mode 100644 index 0000000..5ee06b1 --- /dev/null +++ b/tests/override/ov3.yml @@ -0,0 +1,5 @@ +cwltool:overrides: + "echo-wf.cwl": + - class: EnvVarRequirement + envDef: + MESSAGE: hello6 diff --git a/tests/test_ext.py b/tests/test_ext.py index 6f690d3..76c8e08 100644 --- a/tests/test_ext.py +++ b/tests/test_ext.py @@ -6,7 +6,6 @@ import unittest import pytest import cwltool.expression as expr -import cwltool.factory import cwltool.pathmapper import cwltool.process import cwltool.workflow diff --git a/tests/test_fetch.py b/tests/test_fetch.py index e30fb47..cebcaf0 100644 --- a/tests/test_fetch.py +++ b/tests/test_fetch.py @@ -27,7 +27,7 @@ inputs: [] outputs: [] """ else: - raise RuntimeError("Not foo.cwl") + raise RuntimeError("Not foo.cwl, was %s" % url) def check_exists(self, url): # type: (unicode) -> bool if url == "baz:bar/foo.cwl": @@ -46,7 +46,11 @@ outputs: [] return urllib.parse.urljoin(base, url) def test_resolver(d, a): - return "baz:bar/" + a + if a.startswith("baz:bar/"): + return a + else: + return "baz:bar/" + a + load_tool("foo.cwl", defaultMakeTool, resolver=test_resolver, fetcher_constructor=TestFetcher) diff --git a/tests/test_override.py b/tests/test_override.py new file mode 100644 index 0000000..d49410c --- /dev/null +++ b/tests/test_override.py @@ -0,0 +1,66 @@ +from __future__ import absolute_import +import unittest + +import cwltool.expression as expr +import cwltool.pathmapper +import cwltool.process +import cwltool.workflow +import pytest +import json +from cwltool.main import main +from cwltool.utils import onWindows +from six import StringIO + +from .util import get_data + + +class TestOverride(unittest.TestCase): + @pytest.mark.skipif(onWindows(), + reason="Instance of Cwltool is used, On windows that invoke a default docker Container") + def test_overrides(self): + sio = StringIO() + + self.assertEquals(main([get_data('tests/override/echo.cwl'), + get_data('tests/override/echo-job.yml')], + stdout=sio), 0) + self.assertEquals({"out": "zing hello1"}, json.loads(sio.getvalue())) + + sio = StringIO() + self.assertEquals(main(["--overrides", get_data('tests/override/ov.yml'), + get_data('tests/override/echo.cwl'), + get_data('tests/override/echo-job.yml')], + stdout=sio), 0) + self.assertEquals({"out": "zing hello2"}, json.loads(sio.getvalue())) + + sio = StringIO() + self.assertEquals(main([get_data('tests/override/echo.cwl'), + get_data('tests/override/echo-job-ov.yml')], + stdout=sio), 0) + self.assertEquals({"out": "zing hello3"}, json.loads(sio.getvalue())) + + sio = StringIO() + self.assertEquals(main([get_data('tests/override/echo-job-ov2.yml')], + stdout=sio), 0) + self.assertEquals({"out": "zing hello4"}, json.loads(sio.getvalue())) + + + sio = StringIO() + self.assertEquals(main(["--overrides", get_data('tests/override/ov.yml'), + get_data('tests/override/echo-wf.cwl'), + get_data('tests/override/echo-job.yml')], + stdout=sio), 0) + self.assertEquals({"out": "zing hello2"}, json.loads(sio.getvalue())) + + sio = StringIO() + self.assertEquals(main(["--overrides", get_data('tests/override/ov2.yml'), + get_data('tests/override/echo-wf.cwl'), + get_data('tests/override/echo-job.yml')], + stdout=sio), 0) + self.assertEquals({"out": "zing hello5"}, json.loads(sio.getvalue())) + + sio = StringIO() + self.assertEquals(main(["--overrides", get_data('tests/override/ov3.yml'), + get_data('tests/override/echo-wf.cwl'), + get_data('tests/override/echo-job.yml')], + stdout=sio), 0) + self.assertEquals({"out": "zing hello6"}, json.loads(sio.getvalue())) diff --git a/tests/test_pack.py b/tests/test_pack.py index 34a14e6..728a1e0 100644 --- a/tests/test_pack.py +++ b/tests/test_pack.py @@ -1,22 +1,28 @@ from __future__ import absolute_import + import json -import os import unittest + +import os from functools import partial +import tempfile + +import pytest +from six import StringIO import cwltool.pack -from cwltool.main import print_pack as print_pack import cwltool.workflow from cwltool.load_tool import fetch_document, validate_document -from cwltool.main import makeRelative +from cwltool.main import makeRelative, main, print_pack from cwltool.pathmapper import adjustDirObjs, adjustFileObjs - +from cwltool.utils import onWindows from .util import get_data class TestPack(unittest.TestCase): + maxDiff = None + def test_pack(self): - self.maxDiff = None document_loader, workflowobj, uri = fetch_document( get_data("tests/wf/revsort.cwl")) @@ -38,13 +44,11 @@ class TestPack(unittest.TestCase): def test_pack_missing_cwlVersion(self): """Test to ensure the generated pack output is not missing the `cwlVersion` in case of single tool workflow and single step workflow""" - # Since diff is longer than 3174 characters - self.maxDiff = None # Testing single tool workflow document_loader, workflowobj, uri = fetch_document( get_data("tests/wf/hello_single_tool.cwl")) - document_loader, avsc_names, processobj, metadata, uri = validate_document( + document_loader, _, processobj, metadata, uri = validate_document( document_loader, workflowobj, uri) # generate pack output dict packed = json.loads(print_pack(document_loader, processobj, uri, metadata)) @@ -54,9 +58,88 @@ class TestPack(unittest.TestCase): # Testing single step workflow document_loader, workflowobj, uri = fetch_document( get_data("tests/wf/hello-workflow.cwl")) - document_loader, avsc_names, processobj, metadata, uri = validate_document( + document_loader, _, processobj, metadata, uri = validate_document( document_loader, workflowobj, uri) # generate pack output dict packed = json.loads(print_pack(document_loader, processobj, uri, metadata)) self.assertEqual('v1.0', packed["cwlVersion"]) + + def test_pack_idempotence_tool(self): + """Test to ensure that pack produces exactly the same document for + an already packed document""" + + # Testing single tool + self._pack_idempotently("tests/wf/hello_single_tool.cwl") + + def test_pack_idempotence_workflow(self): + """Test to ensure that pack produces exactly the same document for + an already packed document""" + + # Testing workflow + self._pack_idempotently("tests/wf/count-lines1-wf.cwl") + + def _pack_idempotently(self, document): + document_loader, workflowobj, uri = fetch_document( + get_data(document)) + document_loader, avsc_names, processobj, metadata, uri = validate_document( + document_loader, workflowobj, uri) + # generate pack output dict + packed = json.loads(print_pack(document_loader, processobj, uri, metadata)) + + document_loader, workflowobj, uri2 = fetch_document(packed) + document_loader, avsc_names, processobj, metadata, uri2 = validate_document( + document_loader, workflowobj, uri) + double_packed = json.loads(print_pack(document_loader, processobj, uri2, metadata)) + self.assertEqual(packed, double_packed) + + @pytest.mark.skipif(onWindows(), + reason="Instance of cwltool is used, on Windows it invokes a default docker container" + "which is not supported on AppVeyor") + def test_packed_workflow_execution(self): + test_wf = "tests/wf/count-lines1-wf.cwl" + test_wf_job = "tests/wf/wc-job.json" + document_loader, workflowobj, uri = fetch_document( + get_data(test_wf)) + document_loader, avsc_names, processobj, metadata, uri = validate_document( + document_loader, workflowobj, uri) + packed = json.loads(print_pack(document_loader, processobj, uri, metadata)) + temp_packed_path = tempfile.mkstemp()[1] + with open(temp_packed_path, 'w') as f: + json.dump(packed, f) + normal_output = StringIO() + packed_output = StringIO() + self.assertEquals(main(['--debug', get_data(temp_packed_path), + get_data(test_wf_job)], + stdout=packed_output), 0) + self.assertEquals(main([get_data(test_wf), + get_data(test_wf_job)], + stdout=normal_output), 0) + self.assertEquals(json.loads(packed_output.getvalue()), json.loads(normal_output.getvalue())) + os.remove(temp_packed_path) + + @pytest.mark.skipif(onWindows(), + reason="Instance of cwltool is used, on Windows it invokes a default docker container" + "which is not supported on AppVeyor") + def test_preserving_namespaces(self): + test_wf = "tests/wf/formattest.cwl" + test_wf_job = "tests/wf/formattest-job.json" + document_loader, workflowobj, uri = fetch_document( + get_data(test_wf)) + document_loader, avsc_names, processobj, metadata, uri = validate_document( + document_loader, workflowobj, uri) + packed = json.loads(print_pack(document_loader, processobj, uri, metadata)) + assert "$namespaces" in packed + temp_packed_path = tempfile.mkstemp()[1] + with open(temp_packed_path, 'w') as f: + json.dump(packed, f) + normal_output = StringIO() + packed_output = StringIO() + self.assertEquals(main(['--debug', get_data(temp_packed_path), + get_data(test_wf_job)], + stdout=packed_output), 0) + self.assertEquals(main([get_data(test_wf), + get_data(test_wf_job)], + stdout=normal_output), 0) + self.assertEquals(json.loads(packed_output.getvalue()), json.loads(normal_output.getvalue())) + os.remove(temp_packed_path) diff --git a/tests/wf/count-lines1-wf.cwl b/tests/wf/count-lines1-wf.cwl new file mode 100644 index 0000000..77cbf3a --- /dev/null +++ b/tests/wf/count-lines1-wf.cwl @@ -0,0 +1,25 @@ +#!/usr/bin/env cwl-runner +class: Workflow +cwlVersion: v1.0 + +inputs: + file1: + type: File + +outputs: + count_output: + type: int + outputSource: step2/output + +steps: + step1: + run: wc-tool.cwl + in: + file1: file1 + out: [output] + + step2: + run: parseInt-tool.cwl + in: + file1: step1/output + out: [output] diff --git a/tests/wf/formattest-job.json b/tests/wf/formattest-job.json new file mode 100644 index 0000000..0ff0240 --- /dev/null +++ b/tests/wf/formattest-job.json @@ -0,0 +1,7 @@ +{ + "input": { + "class": "File", + "location": "whale.txt", + "format": "edam:format_2330" + } +} diff --git a/tests/wf/formattest.cwl b/tests/wf/formattest.cwl new file mode 100644 index 0000000..19168e8 --- /dev/null +++ b/tests/wf/formattest.cwl @@ -0,0 +1,20 @@ +$namespaces: + edam: "http://edamontology.org/"; +cwlVersion: v1.0 +class: CommandLineTool +doc: "Reverse each line using the `rev` command" +inputs: + input: + type: File + inputBinding: {} + format: edam:format_2330 + +outputs: + output: + type: File + outputBinding: + glob: output.txt + format: edam:format_2330 + +baseCommand: rev +stdout: output.txt \ No newline at end of file diff --git a/tests/wf/parseInt-tool.cwl b/tests/wf/parseInt-tool.cwl new file mode 100644 index 0000000..42f166b --- /dev/null +++ b/tests/wf/parseInt-tool.cwl @@ -0,0 +1,16 @@ +#!/usr/bin/env cwl-runner + +class: ExpressionTool +requirements: + - class: InlineJavascriptRequirement +cwlVersion: v1.0 + +inputs: + file1: + type: File + inputBinding: { loadContents: true } + +outputs: + output: int + +expression: "$({'output': parseInt(inputs.file1.contents)})" diff --git a/tests/wf/wc-job.json b/tests/wf/wc-job.json new file mode 100644 index 0000000..598568d --- /dev/null +++ b/tests/wf/wc-job.json @@ -0,0 +1,6 @@ +{ + "file1": { + "class": "File", + "location": "whale.txt" + } +} diff --git a/tests/wf/wc-tool.cwl b/tests/wf/wc-tool.cwl new file mode 100644 index 0000000..1655854 --- /dev/null +++ b/tests/wf/wc-tool.cwl @@ -0,0 +1,17 @@ +#!/usr/bin/env cwl-runner + +class: CommandLineTool +cwlVersion: v1.0 + +inputs: + file1: File + +outputs: + output: + type: File + outputBinding: { glob: output } + +baseCommand: [wc, -l] + +stdin: $(inputs.file1.path) +stdout: output diff --git a/tests/wf/whale.txt b/tests/wf/whale.txt new file mode 100644 index 0000000..425d1ed --- /dev/null +++ b/tests/wf/whale.txt @@ -0,0 +1,16 @@ +Call me Ishmael. Some years ago--never mind how long precisely--having +little or no money in my purse, and nothing particular to interest me on +shore, I thought I would sail about a little and see the watery part of +the world. It is a way I have of driving off the spleen and regulating +the circulation. Whenever I find myself growing grim about the mouth; +whenever it is a damp, drizzly November in my soul; whenever I find +myself involuntarily pausing before coffin warehouses, and bringing up +the rear of every funeral I meet; and especially whenever my hypos get +such an upper hand of me, that it requires a strong moral principle to +prevent me from deliberately stepping into the street, and methodically +knocking people's hats off--then, I account it high time to get to sea +as soon as I can. This is my substitute for pistol and ball. With a +philosophical flourish Cato throws himself upon his sword; I quietly +take to the ship. There is nothing surprising in this. If they but knew +it, almost all men in their degree, some time or other, cherish very +nearly the same feelings towards the ocean with me. -- Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-med/cwltool.git _______________________________________________ debian-med-commit mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/debian-med-commit
