[jira] [Commented] (CLIMATE-643) Updating unit conversion function
[ https://issues.apache.org/jira/browse/CLIMATE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648504#comment-14648504 ] ASF GitHub Bot commented on CLIMATE-643: Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/213#issuecomment-126522367 Merged build started. Test Failed. Updating unit conversion function -- Key: CLIMATE-643 URL: https://issues.apache.org/jira/browse/CLIMATE-643 Project: Apache Open Climate Workbench Issue Type: Improvement Components: general, regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 In many cases, observational and model datasets have different units for the same variable. We need to update variable_unit_conversion module is needed in dataset_processor. This new module is based on water_flux_unit_conversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-643) Updating unit conversion function
[ https://issues.apache.org/jira/browse/CLIMATE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648503#comment-14648503 ] ASF GitHub Bot commented on CLIMATE-643: Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/213#issuecomment-126522358 Merged build triggered. Test Failed. Updating unit conversion function -- Key: CLIMATE-643 URL: https://issues.apache.org/jira/browse/CLIMATE-643 Project: Apache Open Climate Workbench Issue Type: Improvement Components: general, regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 In many cases, observational and model datasets have different units for the same variable. We need to update variable_unit_conversion module is needed in dataset_processor. This new module is based on water_flux_unit_conversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-643 - Updating unit conversion funct...
Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/213#issuecomment-126522551 Merged build finished. Test Passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CLIMATE-650) Fix the ensemble calculation
[ https://issues.apache.org/jira/browse/CLIMATE-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648544#comment-14648544 ] ASF GitHub Bot commented on CLIMATE-650: GitHub user huikyole opened a pull request: https://github.com/apache/climate/pull/215 CLIMATE-650 - Fix the ensemble calculation - Fix: ocw.dataset_processor.ensemble now uses numpy.ma.mean instead of numpy.mean to properly handle missing values. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huikyole/climate CLIMATE-650 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/climate/pull/215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #215 commit 9411152efa3eaef538acb96d13657ac7af9b8540 Author: huikyole huiky...@argo.jpl.nasa.gov Date: 2015-07-31T00:15:45Z Fix the ensemble calculation Fix the ensemble calculation Key: CLIMATE-650 URL: https://issues.apache.org/jira/browse/CLIMATE-650 Project: Apache Open Climate Workbench Issue Type: Bug Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 When calculating an ensemble average, we need to use numpy.ma.mean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-650) Fix the ensemble calculation
[ https://issues.apache.org/jira/browse/CLIMATE-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648545#comment-14648545 ] ASF GitHub Bot commented on CLIMATE-650: Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/215#issuecomment-126527630 Merged build triggered. Test Failed. Fix the ensemble calculation Key: CLIMATE-650 URL: https://issues.apache.org/jira/browse/CLIMATE-650 Project: Apache Open Climate Workbench Issue Type: Bug Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 When calculating an ensemble average, we need to use numpy.ma.mean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-650 - Fix the ensemble calculation
Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/215#issuecomment-126527638 Merged build started. Test Failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] climate pull request: CLIMATE-650 - Fix the ensemble calculation
Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/215#issuecomment-126527630 Merged build triggered. Test Failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] climate pull request: CLIMATE-648 - Propagation of missing data in...
GitHub user huikyole opened a pull request: https://github.com/apache/climate/pull/214 CLIMATE-648 - Propagation of missing data information from each dataset A new module ocw.dataset_processor.mask_missing_data has been added. Any missing data flag is propagated into the all datasets in the input 'dataset_array'. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huikyole/climate CLIMATE-648 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/climate/pull/214.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #214 commit fb43681b8143bc244557927fdbca788e17c9dc20 Author: huikyole huiky...@argo.jpl.nasa.gov Date: 2015-07-30T23:52:34Z Propagation of missing data information from each dataset --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] climate pull request: CLIMATE-648 - Propagation of missing data in...
Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/214#issuecomment-126524007 Merged build started. Test Failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CLIMATE-649) Update bias calculation in CLI
Huikyo Lee created CLIMATE-649: -- Summary: Update bias calculation in CLI Key: CLIMATE-649 URL: https://issues.apache.org/jira/browse/CLIMATE-649 Project: Apache Open Climate Workbench Issue Type: Improvement Components: command line interface Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 The bias calculation in CLI does not properly handle missing values. Recently updated modules will address this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-650) Fix the ensemble calculation
[ https://issues.apache.org/jira/browse/CLIMATE-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648547#comment-14648547 ] ASF GitHub Bot commented on CLIMATE-650: Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/215#issuecomment-126527708 Merged build finished. Test Passed. Fix the ensemble calculation Key: CLIMATE-650 URL: https://issues.apache.org/jira/browse/CLIMATE-650 Project: Apache Open Climate Workbench Issue Type: Bug Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 When calculating an ensemble average, we need to use numpy.ma.mean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-643 - Updating unit conversion funct...
GitHub user huikyole opened a pull request: https://github.com/apache/climate/pull/213 CLIMATE-643 - Updating unit conversion function In many cases, observational and model datasets have different units for the same variable. We need to update variable_unit_conversion module is needed in dataset_processor. This new module is based on water_flux_unit_conversion. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huikyole/climate CLIMATE-643 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/climate/pull/213.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #213 commit 3b297dbf29e789f4aec46e5f2f3ea151d9e14372 Author: huikyole huiky...@argo.jpl.nasa.gov Date: 2015-07-30T23:37:34Z Added: ocw.dataset_processor.variable_unit_conversion --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CLIMATE-643) Updating unit conversion function
[ https://issues.apache.org/jira/browse/CLIMATE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648501#comment-14648501 ] ASF GitHub Bot commented on CLIMATE-643: GitHub user huikyole opened a pull request: https://github.com/apache/climate/pull/213 CLIMATE-643 - Updating unit conversion function In many cases, observational and model datasets have different units for the same variable. We need to update variable_unit_conversion module is needed in dataset_processor. This new module is based on water_flux_unit_conversion. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huikyole/climate CLIMATE-643 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/climate/pull/213.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #213 commit 3b297dbf29e789f4aec46e5f2f3ea151d9e14372 Author: huikyole huiky...@argo.jpl.nasa.gov Date: 2015-07-30T23:37:34Z Added: ocw.dataset_processor.variable_unit_conversion Updating unit conversion function -- Key: CLIMATE-643 URL: https://issues.apache.org/jira/browse/CLIMATE-643 Project: Apache Open Climate Workbench Issue Type: Improvement Components: general, regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 In many cases, observational and model datasets have different units for the same variable. We need to update variable_unit_conversion module is needed in dataset_processor. This new module is based on water_flux_unit_conversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-643 - Updating unit conversion funct...
Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/213#issuecomment-126522358 Merged build triggered. Test Failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] climate pull request: CLIMATE-643 - Updating unit conversion funct...
Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/213#issuecomment-126522367 Merged build started. Test Failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CLIMATE-650) Fix the ensemble calculation
Huikyo Lee created CLIMATE-650: -- Summary: Fix the ensemble calculation Key: CLIMATE-650 URL: https://issues.apache.org/jira/browse/CLIMATE-650 Project: Apache Open Climate Workbench Issue Type: Bug Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 When calculating an ensemble average, we need to use numpy.ma.mean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-643) Updating unit conversion function
[ https://issues.apache.org/jira/browse/CLIMATE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648507#comment-14648507 ] ASF GitHub Bot commented on CLIMATE-643: Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/213#issuecomment-126522551 Merged build finished. Test Passed. Updating unit conversion function -- Key: CLIMATE-643 URL: https://issues.apache.org/jira/browse/CLIMATE-643 Project: Apache Open Climate Workbench Issue Type: Improvement Components: general, regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 In many cases, observational and model datasets have different units for the same variable. We need to update variable_unit_conversion module is needed in dataset_processor. This new module is based on water_flux_unit_conversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-650 - Fix the ensemble calculation
GitHub user huikyole opened a pull request: https://github.com/apache/climate/pull/215 CLIMATE-650 - Fix the ensemble calculation - Fix: ocw.dataset_processor.ensemble now uses numpy.ma.mean instead of numpy.mean to properly handle missing values. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huikyole/climate CLIMATE-650 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/climate/pull/215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #215 commit 9411152efa3eaef538acb96d13657ac7af9b8540 Author: huikyole huiky...@argo.jpl.nasa.gov Date: 2015-07-31T00:15:45Z Fix the ensemble calculation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (CLIMATE-643) Updating unit conversion function
[ https://issues.apache.org/jira/browse/CLIMATE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huikyo Lee updated CLIMATE-643: --- Description: In many cases, observational and model datasets have different units for the same variable. We need to update variable_unit_conversion module is needed in dataset_processor. This new module is based on water_flux_unit_conversion. (was: Paul L. suggested some ideas to update examples. For example, knmi_to_cru_full_bias.py needs to be updated with better description. The model to model bias could be replaced by model to observation data bias. The goal is providing 5 examples all based on an actual published papers. Currently, OCW examples generate wrong results when there is missing data in observational datasets. It is important to mask those grid points with missing values in model datasets so that no metrics calculation is done at those grid points. In other words, if any of observation/model dataset has missing value at a grid point, we need to propagate the missing information to the other datasets.) Component/s: regridding/data processing Summary: Updating unit conversion function (was: Updating some of examples) Updating unit conversion function -- Key: CLIMATE-643 URL: https://issues.apache.org/jira/browse/CLIMATE-643 Project: Apache Open Climate Workbench Issue Type: Improvement Components: general, regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 In many cases, observational and model datasets have different units for the same variable. We need to update variable_unit_conversion module is needed in dataset_processor. This new module is based on water_flux_unit_conversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-643) Updating unit conversion function
[ https://issues.apache.org/jira/browse/CLIMATE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648496#comment-14648496 ] ASF GitHub Bot commented on CLIMATE-643: Github user huikyole closed the pull request at: https://github.com/apache/climate/pull/210 Updating unit conversion function -- Key: CLIMATE-643 URL: https://issues.apache.org/jira/browse/CLIMATE-643 Project: Apache Open Climate Workbench Issue Type: Improvement Components: general, regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 In many cases, observational and model datasets have different units for the same variable. We need to update variable_unit_conversion module is needed in dataset_processor. This new module is based on water_flux_unit_conversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-643 - Updating some of examples
Github user huikyole closed the pull request at: https://github.com/apache/climate/pull/210 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CLIMATE-648) Propagation of missing data information from each dataset
Huikyo Lee created CLIMATE-648: -- Summary: Propagation of missing data information from each dataset Key: CLIMATE-648 URL: https://issues.apache.org/jira/browse/CLIMATE-648 Project: Apache Open Climate Workbench Issue Type: Improvement Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 Current examples do not properly share missing data information between observations and models. To fix this, we need a new module in dataset_processor which propagate missing data flag from each dataset to the other datasets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-643 - Updating some of examples
Github user huikyole commented on the pull request: https://github.com/apache/climate/pull/210#issuecomment-126383587 @MJJoyce I understand your concerns. However, updating examples was impossible without adding new modules to the dataset_processor. CLI is also using the code based on examples. I will try to separate these into three tickets. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] climate pull request: CLIMATE-643 - Updating some of examples
Github user MJJoyce commented on the pull request: https://github.com/apache/climate/pull/210#issuecomment-126403057 If the changes build off one another just separate them out into different feature branches and merge/rebase according during the dev work. Otherwise we end up with overly complicated pull requests that address multiple tickets and that just makes reviews and integration more challenging. Small, atomic changes are always easier. If you need help with the branching/merging let me know. I'm more than happy to show you a few easy ways to take care of this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CLIMATE-643) Updating some of examples
[ https://issues.apache.org/jira/browse/CLIMATE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647870#comment-14647870 ] ASF GitHub Bot commented on CLIMATE-643: Github user huikyole commented on the pull request: https://github.com/apache/climate/pull/210#issuecomment-126383587 @MJJoyce I understand your concerns. However, updating examples was impossible without adding new modules to the dataset_processor. CLI is also using the code based on examples. I will try to separate these into three tickets. Updating some of examples - Key: CLIMATE-643 URL: https://issues.apache.org/jira/browse/CLIMATE-643 Project: Apache Open Climate Workbench Issue Type: Improvement Components: general Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 Paul L. suggested some ideas to update examples. For example, knmi_to_cru_full_bias.py needs to be updated with better description. The model to model bias could be replaced by model to observation data bias. The goal is providing 5 examples all based on an actual published papers. Currently, OCW examples generate wrong results when there is missing data in observational datasets. It is important to mask those grid points with missing values in model datasets so that no metrics calculation is done at those grid points. In other words, if any of observation/model dataset has missing value at a grid point, we need to propagate the missing information to the other datasets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-643) Updating some of examples
[ https://issues.apache.org/jira/browse/CLIMATE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647952#comment-14647952 ] ASF GitHub Bot commented on CLIMATE-643: Github user MJJoyce commented on the pull request: https://github.com/apache/climate/pull/210#issuecomment-126403057 If the changes build off one another just separate them out into different feature branches and merge/rebase according during the dev work. Otherwise we end up with overly complicated pull requests that address multiple tickets and that just makes reviews and integration more challenging. Small, atomic changes are always easier. If you need help with the branching/merging let me know. I'm more than happy to show you a few easy ways to take care of this! Updating some of examples - Key: CLIMATE-643 URL: https://issues.apache.org/jira/browse/CLIMATE-643 Project: Apache Open Climate Workbench Issue Type: Improvement Components: general Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 Paul L. suggested some ideas to update examples. For example, knmi_to_cru_full_bias.py needs to be updated with better description. The model to model bias could be replaced by model to observation data bias. The goal is providing 5 examples all based on an actual published papers. Currently, OCW examples generate wrong results when there is missing data in observational datasets. It is important to mask those grid points with missing values in model datasets so that no metrics calculation is done at those grid points. In other words, if any of observation/model dataset has missing value at a grid point, we need to propagate the missing information to the other datasets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-643) Updating some of examples
[ https://issues.apache.org/jira/browse/CLIMATE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647985#comment-14647985 ] ASF GitHub Bot commented on CLIMATE-643: Github user MJJoyce commented on the pull request: https://github.com/apache/climate/pull/210#issuecomment-126409195 To give you an idea of what I'm talking about here @huikyole in case it's confusing. Here's an example of how I would have broken these down into multiple tickets and their relevant changes. New Ticket 1: This [new function](https://github.com/apache/climate/pull/210/files#diff-e41b7ee3c7e3b2339c964f6882ecfaa5R426) would be here. This [variable name update](https://github.com/apache/climate/pull/210/files#diff-e41b7ee3c7e3b2339c964f6882ecfaa5R409) is related so we could include that as well. Tests for the new function would be included in a commit in here as well. New Ticket 2: Missing data [changes](https://github.com/apache/climate/pull/210/files#diff-e41b7ee3c7e3b2339c964f6882ecfaa5R463) would fall in another ticket here. Tests for this new function would be here as well. New Ticket 3: [This is a propagation bug](https://github.com/apache/climate/pull/210/files#diff-e41b7ee3c7e3b2339c964f6882ecfaa5R463) for the existing values and would be its own ticket. New test would be included in here as well so we don't miss this bug in the future! New Ticket 4: Any change in [the CLI app](https://github.com/apache/climate/pull/210/files#diff-9e183b82ea6b20fcfefa86ca818c670eR19) that you have in here would be in another ticket. Would probably need to update tests if appropriate for the CLI (I'm not familiar w/ the code so I'm unsure if that's the case) This ticket: The changes in [examples/knmi_to_cru31_full_bias.py](https://github.com/apache/climate/pull/210/files#diff-a84c6055f2eee6f83c3eac5d00e324d0R28) would be in this ticket since it's the actual update to the example file. --- So, if these tickets build off of each other you would do something to the effect of: * Make feature branch off master for Ticket 1. Make changes, update tests, and make PR * Make feature branch off master for Ticket 2. Make changes, update tests, and make PR. When we merge Ticket 1/2 we will need to rebase this but that should be fine. The changes are obvious enough to make merging and issues easy/trivial. * Make feature branch for Ticket 3 off master. Make changes, write test, and make PR. Will need to rebase here as well when we merge but see above. Should be easy. * Make feature branch for Ticket 4 off master. Etc. * Make feature branch for this ticket. You will probably need to merge some of the stuff in the ticket 1, 2, and 3 feature branches in here so that way you have the changes that you need. When we merge the others into master it's simple enough to rebase again and include those relevant commits. --- Hopefully that helps. If you have questions let me know! Updating some of examples - Key: CLIMATE-643 URL: https://issues.apache.org/jira/browse/CLIMATE-643 Project: Apache Open Climate Workbench Issue Type: Improvement Components: general Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 Paul L. suggested some ideas to update examples. For example, knmi_to_cru_full_bias.py needs to be updated with better description. The model to model bias could be replaced by model to observation data bias. The goal is providing 5 examples all based on an actual published papers. Currently, OCW examples generate wrong results when there is missing data in observational datasets. It is important to mask those grid points with missing values in model datasets so that no metrics calculation is done at those grid points. In other words, if any of observation/model dataset has missing value at a grid point, we need to propagate the missing information to the other datasets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-648) Propagation of missing data information from each dataset
[ https://issues.apache.org/jira/browse/CLIMATE-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648529#comment-14648529 ] ASF GitHub Bot commented on CLIMATE-648: GitHub user huikyole opened a pull request: https://github.com/apache/climate/pull/214 CLIMATE-648 - Propagation of missing data information from each dataset A new module ocw.dataset_processor.mask_missing_data has been added. Any missing data flag is propagated into the all datasets in the input 'dataset_array'. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huikyole/climate CLIMATE-648 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/climate/pull/214.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #214 commit fb43681b8143bc244557927fdbca788e17c9dc20 Author: huikyole huiky...@argo.jpl.nasa.gov Date: 2015-07-30T23:52:34Z Propagation of missing data information from each dataset Propagation of missing data information from each dataset - Key: CLIMATE-648 URL: https://issues.apache.org/jira/browse/CLIMATE-648 Project: Apache Open Climate Workbench Issue Type: Improvement Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 Current examples do not properly share missing data information between observations and models. To fix this, we need a new module in dataset_processor which propagate missing data flag from each dataset to the other datasets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-648 - Propagation of missing data in...
Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/214#issuecomment-126523998 Merged build triggered. Test Failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CLIMATE-648) Propagation of missing data information from each dataset
[ https://issues.apache.org/jira/browse/CLIMATE-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648532#comment-14648532 ] ASF GitHub Bot commented on CLIMATE-648: Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/214#issuecomment-126524007 Merged build started. Test Failed. Propagation of missing data information from each dataset - Key: CLIMATE-648 URL: https://issues.apache.org/jira/browse/CLIMATE-648 Project: Apache Open Climate Workbench Issue Type: Improvement Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 Current examples do not properly share missing data information between observations and models. To fix this, we need a new module in dataset_processor which propagate missing data flag from each dataset to the other datasets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-648 - Propagation of missing data in...
Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/214#issuecomment-126524122 Merged build finished. Test Passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CLIMATE-648) Propagation of missing data information from each dataset
[ https://issues.apache.org/jira/browse/CLIMATE-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648531#comment-14648531 ] ASF GitHub Bot commented on CLIMATE-648: Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/214#issuecomment-126523998 Merged build triggered. Test Failed. Propagation of missing data information from each dataset - Key: CLIMATE-648 URL: https://issues.apache.org/jira/browse/CLIMATE-648 Project: Apache Open Climate Workbench Issue Type: Improvement Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 Current examples do not properly share missing data information between observations and models. To fix this, we need a new module in dataset_processor which propagate missing data flag from each dataset to the other datasets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-648) Propagation of missing data information from each dataset
[ https://issues.apache.org/jira/browse/CLIMATE-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648533#comment-14648533 ] ASF GitHub Bot commented on CLIMATE-648: Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/214#issuecomment-126524122 Merged build finished. Test Passed. Propagation of missing data information from each dataset - Key: CLIMATE-648 URL: https://issues.apache.org/jira/browse/CLIMATE-648 Project: Apache Open Climate Workbench Issue Type: Improvement Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 Current examples do not properly share missing data information between observations and models. To fix this, we need a new module in dataset_processor which propagate missing data flag from each dataset to the other datasets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLIMATE-650) Fix the ensemble calculation
[ https://issues.apache.org/jira/browse/CLIMATE-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648546#comment-14648546 ] ASF GitHub Bot commented on CLIMATE-650: Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/215#issuecomment-126527638 Merged build started. Test Failed. Fix the ensemble calculation Key: CLIMATE-650 URL: https://issues.apache.org/jira/browse/CLIMATE-650 Project: Apache Open Climate Workbench Issue Type: Bug Components: regridding/data processing Affects Versions: 1.0.0 Reporter: Huikyo Lee Assignee: Huikyo Lee Fix For: 1.0.0 When calculating an ensemble average, we need to use numpy.ma.mean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] climate pull request: CLIMATE-650 - Fix the ensemble calculation
Github user OCWJenkins commented on the pull request: https://github.com/apache/climate/pull/215#issuecomment-126527708 Merged build finished. Test Passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] climate pull request: CLIMATE-643 - Updating some of examples
Github user MJJoyce commented on the pull request: https://github.com/apache/climate/pull/210#issuecomment-126409195 To give you an idea of what I'm talking about here @huikyole in case it's confusing. Here's an example of how I would have broken these down into multiple tickets and their relevant changes. New Ticket 1: This [new function](https://github.com/apache/climate/pull/210/files#diff-e41b7ee3c7e3b2339c964f6882ecfaa5R426) would be here. This [variable name update](https://github.com/apache/climate/pull/210/files#diff-e41b7ee3c7e3b2339c964f6882ecfaa5R409) is related so we could include that as well. Tests for the new function would be included in a commit in here as well. New Ticket 2: Missing data [changes](https://github.com/apache/climate/pull/210/files#diff-e41b7ee3c7e3b2339c964f6882ecfaa5R463) would fall in another ticket here. Tests for this new function would be here as well. New Ticket 3: [This is a propagation bug](https://github.com/apache/climate/pull/210/files#diff-e41b7ee3c7e3b2339c964f6882ecfaa5R463) for the existing values and would be its own ticket. New test would be included in here as well so we don't miss this bug in the future! New Ticket 4: Any change in [the CLI app](https://github.com/apache/climate/pull/210/files#diff-9e183b82ea6b20fcfefa86ca818c670eR19) that you have in here would be in another ticket. Would probably need to update tests if appropriate for the CLI (I'm not familiar w/ the code so I'm unsure if that's the case) This ticket: The changes in [examples/knmi_to_cru31_full_bias.py](https://github.com/apache/climate/pull/210/files#diff-a84c6055f2eee6f83c3eac5d00e324d0R28) would be in this ticket since it's the actual update to the example file. --- So, if these tickets build off of each other you would do something to the effect of: * Make feature branch off master for Ticket 1. Make changes, update tests, and make PR * Make feature branch off master for Ticket 2. Make changes, update tests, and make PR. When we merge Ticket 1/2 we will need to rebase this but that should be fine. The changes are obvious enough to make merging and issues easy/trivial. * Make feature branch for Ticket 3 off master. Make changes, write test, and make PR. Will need to rebase here as well when we merge but see above. Should be easy. * Make feature branch for Ticket 4 off master. Etc. * Make feature branch for this ticket. You will probably need to merge some of the stuff in the ticket 1, 2, and 3 feature branches in here so that way you have the changes that you need. When we merge the others into master it's simple enough to rebase again and include those relevant commits. --- Hopefully that helps. If you have questions let me know! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
