[
https://issues.apache.org/jira/browse/CLIMATE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399822#comment-15399822
]
ASF GitHub Bot commented on CLIMATE-825:
----------------------------------------
Github user huikyole commented on the issue:
https://github.com/apache/climate/pull/374
@agoodm This is wonderful. Thank you so much. I have very minor suggestions.
- dataset_loader.py can be located in ocw/data_source folder.
- Would it be possible to have 1) multiple references and 2) reference only
without targets?
1) is very useful when calculating metrics using multiple variables.
2) for example, we should be able to load daily reanalysis data only and
calculate k-means clustering.
> Coalesce data sources into one module
> -------------------------------------
>
> Key: CLIMATE-825
> URL: https://issues.apache.org/jira/browse/CLIMATE-825
> Project: Apache Open Climate Workbench
> Issue Type: Improvement
> Components: data sources
> Affects Versions: 1.0.0
> Reporter: Alex Goodman
> Assignee: Alex Goodman
> Fix For: 1.2.0
>
>
> Kyo and I will be working on overhauling the way data loading is handled in
> the current RCMES workflow. Right now, the user manually specifies the
> sources for each dataset which are currently separated into three categories:
> local files on disk, the RCMES database (RCMED), and the Earth System Grid
> (ESGF). These cases are currently handled in separate modules / function
> calls, but it would be most ideal in the future to create one universal
> function call for all the data loading. An example schematic would be
> something like:
> datasets = load(sources, ...)
> Here datasets would be a list of OCW Dataset objects, sources would be a list
> of source specifications for each requested dataset (eg, 'esgf', 'local', or
> 'rcmed'). Ideally we would also like better support for handling datasets
> spanned by multiple files as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
