Repository: climate Updated Branches: refs/heads/master 511a7e175 -> 8e69e625f
CLIMATE-582 - Add initial pass at config docs to sphinx build Project: http://git-wip-us.apache.org/repos/asf/climate/repo Commit: http://git-wip-us.apache.org/repos/asf/climate/commit/c1c53afc Tree: http://git-wip-us.apache.org/repos/asf/climate/tree/c1c53afc Diff: http://git-wip-us.apache.org/repos/asf/climate/diff/c1c53afc Branch: refs/heads/master Commit: c1c53afc1875d267dcbeb9f1876f623e7e074688 Parents: 6e73c8a Author: Michael Joyce <[email protected]> Authored: Thu Mar 5 13:58:47 2015 -0800 Committer: Michael Joyce <[email protected]> Committed: Thu Mar 5 13:58:47 2015 -0800 ---------------------------------------------------------------------- docs/source/conf.py | 1 + docs/source/config/config_overview.rst | 69 +++++++++++++++++++ docs/source/config/dataset_information.rst | 89 +++++++++++++++++++++++++ docs/source/config/evaluation_settings.rst | 56 ++++++++++++++++ docs/source/config/metrics_information.rst | 12 ++++ docs/source/config/plots_settings.rst | 44 ++++++++++++ docs/source/index.rst | 1 + 7 files changed, 272 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/conf.py ---------------------------------------------------------------------- diff --git a/docs/source/conf.py b/docs/source/conf.py index 96ad8cb..1095b7b 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -22,6 +22,7 @@ import sys, os sys.path.insert(0, os.path.abspath('../../ocw')) sys.path.insert(0, os.path.abspath('../../ocw/data_source')) sys.path.insert(0, os.path.abspath('../../ocw-ui/backend')) +sys.path.insert(0, os.path.abspath('../../ocw-config-runner')) # -- General configuration ----------------------------------------------------- http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/config_overview.rst ---------------------------------------------------------------------- diff --git a/docs/source/config/config_overview.rst b/docs/source/config/config_overview.rst new file mode 100644 index 0000000..dd985cf --- /dev/null +++ b/docs/source/config/config_overview.rst @@ -0,0 +1,69 @@ +Configuration File Overview +=========================== + +Apache Open Climate Workbench includes tools for creating and reading configuration files. Below is an explanation of the general configuration file structure, and in-depth look at the various configuration options, and explanations of how to use configuration files in an evaluation. + +Getting Started +--------------- + +OCW configuration files are written in `YAML <http://yaml.org/>`_ with type annotations that are supported by the `PyYAML library <http://pyyaml.org/wiki/PyYAMLDocumentation>`_. Let's look at an example configuration file to get started. + +.. code:: + + evaluation: + temporal_time_delta: 365 + spatial_regrid_lats: !!python/tuple [-20, 20, 1] + spatial_regrid_lons: !!python/tuple [-20, 20, 1] + + datasets: + reference: + data_source: local + file_count: 1 + path: /tmp/AFRICA_KNMI-RACMO2.2b_CTL_ERAINT_MM_50km_1989-2008_tasmax.nc + variable: tasmax + + targets: + - data_source: local + file_count: 1 + path: /tmp/AFRICA_UC-WRF311_CTL_ERAINT_MM_50km-rg_1989-2008_tasmax.nc + variable: tasmax + metrics: + - Bias + + plots: + - type: contour + results_indeces: + - !!python/tuple [0, 0] + lats: + range_min: -20 + range_max: 20 + range_step: 1 + lons: + range_min: -20 + range_max: 20 + range_step: 1 + output_name: wrf_bias_compared_to_knmi + optional_args: + gridshape: !!python/tuple [6, 6] + +There are 4 main categories for configuration settings: Evaluation, Datasets, Metrics, and Plots. + +Evaluation Settings +------------------- + +This is where you will set evaluation specific settings such as temporal and spatial bin sizes to use during dataset preparation. Visit the :doc:`Evaluation Settings <evaluation_settings>` page for additional information. + +Dataset Information +------------------- + +The datasets section is where you specify all the datasets to use for an evaluation. You can specify what the reference dataset should be as well as giving a list of target datasets. Visit the :doc:`Dataset Information <dataset_information>` page for additional information. + +Metrics Information +------------------- + +You will need to load some metrics if you want to get anything useful out of your evaluation. Visit the :doc:`Metrics Information <metrics_information>` page to learn how to specify the metrics that should be used in the evaluation. + +Plots Settings +-------------- + +This is where you specify what plots to make after running the evaluation. The :doc:`Plots Settings <plots_settings>` page provides examples for the supported plot types. http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/dataset_information.rst ---------------------------------------------------------------------- diff --git a/docs/source/config/dataset_information.rst b/docs/source/config/dataset_information.rst new file mode 100644 index 0000000..3c710e3 --- /dev/null +++ b/docs/source/config/dataset_information.rst @@ -0,0 +1,89 @@ +Dataset Information +=================== + +Dataset configuration information is passed in the **datasets** section of the configuration file. You can specify one reference dataset and one or more target datasets for your evaluation:: + + datasets: + reference: + data_source: local + file_count: 1 + path: /tmp/AFRICA_KNMI-RACMO2.2b_CTL_ERAINT_MM_50km_1989-2008_tasmax.nc + variable: tasmax + + targets: + - data_source: local + file_count: 1 + path: /tmp/AFRICA_UC-WRF311_CTL_ERAINT_MM_50km-rg_1989-2008_tasmax.nc + variable: tasmax + - data_source: local + file_count: 1 + path: /tmp/AFRICA_UC-WRF311_CTL_ERAINT_MM_50km-rg_1989-2008_tasmax.nc + variable: tasmax + +Each **data_source** module requires datasets to be passed in a slightly different manner. Below is an explanation of the format for each of the supported data sources. + +Local Dataset +------------- +.. code:: + + data_source: local + file_count: 1 + path: /tmp/AFRICA_KNMI-RACMO2.2b_CTL_ERAINT_MM_50km_1989-2008_tasmax.nc + variable: tasmax + +The **path** flag is the location where the dataset is located on your computer. The **variable** flag is the variable that should be pulled out of the NetCDF file once it has been opened. You pass any optional flags that are accepted by :func:`local.load_file` by using the **optional_args** flag:: + + data_source: local + file_count: 1 + path: /tmp/AFRICA_KNMI-RACMO2.2b_CTL_ERAINT_MM_50km_1989-2008_tasmax.nc + variable: tasmax + optional_args: + elevation_index=0, + name='foo' + +.. note:: + + The **file_count** flag is currently not used. It is there to support planned future functionality. However, you still need to specify it! Leave it as 1. + + +RCMED Dataset +------------- + +.. code:: + + data_source: rcmed + dataset_id: 4 + parameter_id: 32 + min_lat: -10 + max_lat: 10 + min_lon: -20 + max_lon: 20 + start_time: 1997-01-01 + end_time: 2000-01-01 + +To load a dataset from the Jet Propulsion Laboratory's RCMED you will need to specify the above flags. The **dataset_id** and **parameter_id** are dataset specific and can be looked up on the `RCMES project website <https://rcmes.jpl.nasa.gov/content/rcmes-and-data>`_. Pretty much any common time format will be accepted for the start and end times. However, just to be safe you should try to stick with something very standard such as `ISO-8601 <http://www.w3.org/TR/NOTE-datetime>`_ formatted time values. You may also pass any optional parameters that are accepted by :func:`rcmed.parameter_dataset` with the **optional_args** flag. + +ESGF Dataset +------------ + +In order to load an ESGF dataset you will need to specify the following parameters in addition to having an ESGF login:: + + data_source: esgf + dataset_id: obs4MIPs.CNES.AVISO.mon.v1|esg-datanode.jpl.nasa.gov + variable: zosStderr + esgf_password: totallynotmypassword + esgf_username: totallynotmyusername + +The **dataset_id** and **variable** flags are ESGF dataset specific. You can locate them through and ESGF nodes search page. You may also pass any optional parameters that are accepted by :func:`esgf.load_dataset` with the **optional_args** flag. + + +OpenDAP Dataset +--------------- + +A dataset can be downloaded from an OpenDAP URL with the following settings:: + + data_source: dap + url: http://test.opendap.org/dap/data/nc/sst.mnmean.nc.gz + variable: sst + +You may also pass any optional parameters that are accepted by :func:`dap.load` with the **optional_args** flag. http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/evaluation_settings.rst ---------------------------------------------------------------------- diff --git a/docs/source/config/evaluation_settings.rst b/docs/source/config/evaluation_settings.rst new file mode 100644 index 0000000..f10973c --- /dev/null +++ b/docs/source/config/evaluation_settings.rst @@ -0,0 +1,56 @@ +Evaluation Settings +=================== + +The evaluation settings section of the configuration file allows you to set attributes that are critical for making adjustments to the loaded datasets prior to an evaluation run. Here is an example evaluation settings section of a configuration file. Below, we'll look at each of the configuration options in detail. + +.. code:: + + evaluation: + temporal_time_delta: 365 + spatial_regrid_lats: !!python/tuple [-20, 20, 1] + spatial_regrid_lons: !!python/tuple [-20, 20, 1] + +Temporal Rebin +-------------- + +It is often necessary to temporally rebin datasets prior to an evaluation. The **temporal_time_delta** flag is where you can set the **temporal_resolution** parameter for :func:`dataset_processor.temporal_rebin`. The value that you pass here is interpreted as the number of days to assign to a :class:`datetime.timedelta` object before running the :func:`dataset_processor.temporal_rebin` function. + +.. note:: + + This attribute is only useful if you use the configuration data to create an :class:`evaluation.Evaluation` object with the :func:`evaluation_creation.generate_evaluation_from_config` config parser function. + +Spatial Regrid +-------------- + +.. note:: + + Some funcitonality here is still in development. Specifically, passing the spatial_regrid_* flags as lists of values. + +If you need to regrid your datasets onto a new lat/lon grid you will need to set the **spatial_regrid_lats** and **spatial_regrid_lons** options. These will be passed to the :func:`dataset_processor.spatial_regrid` function along with each dataset. There are two valid ways to pass these parameters. First, you can pass them as a list of all values:: + + evaluation: + spatial_regrid_lats: [-10, -5, 0, 5, 10] + spatial_regrid_lons: [-10, -5, 0, 5, 10] + +This is generally useful if you only need to pass a few parameters or if the sequence isn't easy to define as a valid **range** in Python. The other option is to pass **range** information as a tuple. This requires you to use `PyYAML's Python Type Annotations <http://pyyaml.org/wiki/PyYAMLDocumentation#YAMLtagsandPythontypes>`_ but provides a far more compact representation:: + + evaluation: + spatial_regrid_lats: !!python/tuple [-20, 20, 1] + spatial_regrid_lons: !!python/tuple [-20, 20, 1] + +Using this style directly maps to a call to :func:`numpy.arange`:: + + # spatial_regrid_lats: !!python/tuple [-20, 20, 1] becomes + lats = numpy.arange(-20, 20, 1) + +Be sure to pay special attention to the end value for your interval. The :func:`numpy.arange` function does not include the end value in the returned interval. + +Subset Bounds +------------- + +In order to subset the datasets down to an area of interest you will need to pass bounds information:: + + evaluation: + subset: [-10, 10, -20, 20, "1997-01-01", "2000-01-01"] + +Here you're passing the bounding lat/lon box with the first 4 values as well as the valid temporal range bounds with the starting and end time values. Pretty much any common time format will be accepted. However, just to be safe you should try to stick with something very standard such as `ISO-8601 <http://www.w3.org/TR/NOTE-datetime>`_ formatted time values. http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/metrics_information.rst ---------------------------------------------------------------------- diff --git a/docs/source/config/metrics_information.rst b/docs/source/config/metrics_information.rst new file mode 100644 index 0000000..1935e87 --- /dev/null +++ b/docs/source/config/metrics_information.rst @@ -0,0 +1,12 @@ +Metrics Information +=================== + +.. note:: + + At the moment, you can only load metrics that are in :mod:`ocw.metrics`. In the future you will also be able to specify user defined metrics here as well. However, as a work around you can define your custom metrics in the :mod:`ocw.metrics` module. + +You can set the metrics you want to use in the evaluation in the **metrics** section of the config. You simply need to supply a list of the metric class names that you want to be used:: + + metrics: + - Bias + - TemporalStdDev http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/plots_settings.rst ---------------------------------------------------------------------- diff --git a/docs/source/config/plots_settings.rst b/docs/source/config/plots_settings.rst new file mode 100644 index 0000000..2959092 --- /dev/null +++ b/docs/source/config/plots_settings.rst @@ -0,0 +1,44 @@ +Plots Settings +============== + +Plotting configuration information is passed in the **plots** section of the configuration file:: + + plots: + - type: contour + results_indeces: + - !!python/tuple [0, 0] + lats: + range_min: -20 + range_max: 20 + range_step: 1 + lons: + range_min: -20 + range_max: 20 + range_step: 1 + output_name: wrf_bias_compared_to_knmi + optional_args: + gridshape: !!python/tuple [6, 6] + +Each type of support plot has a different configuration format expected. Each of these are explained below. Note, most of these will require you to specify what result data you want included in the plots with the **results_indeces** flag. This relates the format that an Evaluation object outputs results in. Check the :class:`evaluation.Evaluation` documentation for more details. + +Contour Maps +------------- + +The contour maps config configures data for OCW's contour plotter :func:`plotting.draw_contour_map`:: + + type: contour + results_indeces: + - !!python/tuple [0, 0] + lats: + range_min: -20 + range_max: 20 + range_step: 1 + lons: + range_min: -20 + range_max: 20 + range_step: 1 + output_name: wrf_bias_compared_to_knmi + optional_args: + gridshape: !!python/tuple [6, 6] + +The **lat** and **lon** parameters are specified as a range of values. Be aware that the **range_max** element is not included in the output range so you may need to adjust it slightly if you want a particular value included. The **output_name** parameter is the name of the resulting output graph. You may also pass any optional parameters that are supported by the :func:`plotting.draw_contour_map` function with the **optional_args** flag. http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/index.rst ---------------------------------------------------------------------- diff --git a/docs/source/index.rst b/docs/source/index.rst index 796e8da..2f5bf63 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -20,6 +20,7 @@ Contents: ocw/utils data_source/data_sources ui-backend/backend.rst + config/config_overview Indices and tables
