[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-02-22 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373965#comment-16373965
 ] 

ASF GitHub Bot commented on CLIMATE-744:


Github user asfgit closed the pull request at:

https://github.com/apache/climate/pull/498


> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-02-13 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363558#comment-16363558
 ] 

ASF GitHub Bot commented on CLIMATE-744:


Github user lewismc commented on the issue:

https://github.com/apache/climate/pull/498
  
@agoodm @huikyole I have scoped the code out and all looks good here. The 
issue is well documented over at JIRA. I think we just need to ensure that this 
patch doesn't break any existing functionality. Any comments?


> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-02-11 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360196#comment-16360196
 ] 

ASF GitHub Bot commented on CLIMATE-744:


GitHub user MichaelArthurAnderson opened a pull request:

https://github.com/apache/climate/pull/498

CLIMATE-744 Cannot load TRMM data from RCMED

CLIMATE-744 Cannot load TRMM data from RCMED

- Break single,  multi year requests to RCMED into several smaller requests 
to avoid single, ultra large, but partial, result sets.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MichaelArthurAnderson/climate CLIMATE-744

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/climate/pull/498.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #498


commit cc3b36c87f7e861bd9797fd90fc2156d1aa7df32
Author: michaelarthuranderson 
Date:   2018-02-12T00:34:08Z

CLIMATE-744 Cannot load TRMM data from RCMED




> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-01-30 Thread Lewis John McGibbney (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345756#comment-16345756
 ] 

Lewis John McGibbney commented on CLIMATE-744:
--

[~huikyole] [~agoodman] any ideas here folks?

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-01-28 Thread Michael Anderson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342554#comment-16342554
 ] 

Michael Anderson commented on CLIMATE-744:
--

Using dataset 10, parameter 37 (CRU / precipitation) rather than TRMM for the 
same duration returns the same amount of data on each iteration:

-rw-r--r--  1 michaelanderson  staff  300778632 Jan 28 06:58 curl.1

-rw-r--r--  1 michaelanderson  staff  300778632 Jan 28 07:08 curl.2

-rw-r--r--  1 michaelanderson  staff  300778632 Jan 28 07:12 curl.3

 

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-01-27 Thread Michael Anderson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342258#comment-16342258
 ] 

Michael Anderson commented on CLIMATE-744:
--

I ran the same curl command on an AWS with 64 CPU, 488 GB RAM and 25GB network:

[ec2-user@ip-172-31-16-17 ~]$ ls -l

total 1950572

-rw-rw-r--. 1 ec2-user ec2-user 675561581 Jan 27 18:41 curl.1

-rw-rw-r--. 1 ec2-user ec2-user 688015459 Jan 27 18:43 curl.2

-rw-rw-r--. 1 ec2-user ec2-user 633800036 Jan 27 18:45 curl.3

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-01-27 Thread Michael Anderson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342257#comment-16342257
 ] 

Michael Anderson commented on CLIMATE-744:
--

I ran the same curl command on an AWS with 16 cpu and 122 GB RAM on RHEL 7.  
Here are the results:

 

[ec2-user@ip-172-31-2-252 ~]$ ls -l

total 1970744

-rw-rw-r--. 1 ec2-user ec2-user 659658707 Jan 27 18:30 curl.1

-rw-rw-r--. 1 ec2-user ec2-user 671670784 Jan 27 18:32 curl.2

-rw-rw-r--. 1 ec2-user ec2-user 686702730 Jan 27 18:34 curl.3

[ec2-user@ip-172-31-2-252 ~]$ 

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-01-27 Thread Michael Anderson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342249#comment-16342249
 ] 

Michael Anderson commented on CLIMATE-744:
--

Ran the same three curl commands on a pretty large AWS host with RHEL 7:

 

[ec2-user@ip-172-31-53-246 ~]$ ls -l

total 1571704

-rw-rw-r--. 1 ec2-user ec2-user 546429902 Jan 27 18:12 curl.1

-rw-rw-r--. 1 ec2-user ec2-user 578373399 Jan 27 18:15 curl.2

-rw-rw-r--. 1 ec2-user ec2-user 484616136 Jan 27 18:17 curl.3

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-01-27 Thread Michael Anderson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342245#comment-16342245
 ] 

Michael Anderson commented on CLIMATE-744:
--

Running this on my mac:

 

curl 
"https://rcmes.jpl.nasa.gov/query-api/query.php?datasetId=3=36=-45.76=42.24=-24.64=60.28=19900101TZ=20071231TZ;
 > curl.1

curl 
"https://rcmes.jpl.nasa.gov/query-api/query.php?datasetId=3=36=-45.76=42.24=-24.64=60.28=19900101TZ=20071231TZ;
 > curl.2

curl 
"https://rcmes.jpl.nasa.gov/query-api/query.php?datasetId=3=36=-45.76=42.24=-24.64=60.28=19900101TZ=20071231TZ;
 > curl.3

ls -l curl.*

-rw-r--r--  1 michaelanderson  staff  419156308 Jan 27 12:48 curl.1

-rw-r--r--  1 michaelanderson  staff  432335472 Jan 27 12:54 curl.2

-rw-r--r--  1 michaelanderson  staff  422500568 Jan 27 12:59 curl.3

 

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-01-27 Thread Michael Anderson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342234#comment-16342234
 ] 

Michael Anderson commented on CLIMATE-744:
--

I changed my local copy of rcmed.py as follows:

chunk = string.read()

while chunk:
     data_string += chunk
     chunk = string.read()

Each run of the program brings back a different amount of data consistent with 
my previous comments.

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-01-27 Thread Michael Anderson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342218#comment-16342218
 ] 

Michael Anderson commented on CLIMATE-744:
--

I changed my local copy of rcmed.py as follows:

string = urlopen(url)
data_string = ''
while True:
     data = string.read(1024)
     if data:
          data_string += data
    else:
         break

Each run of the program brings back a different amount of data consistent with 
my previous comments.

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2018-01-27 Thread Michael Anderson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342217#comment-16342217
 ] 

Michael Anderson commented on CLIMATE-744:
--

[~lewi...@apache.org] any feedback from the RCMED team?

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Michael Anderson
>Priority: Blocker
> Fix For: 1.3.0
>
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CLIMATE-744) Cannot load TRMM data from RCMED

2016-02-04 Thread Huikyo Lee (JIRA) 

[ 
https://issues.apache.org/jira/browse/CLIMATE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133114#comment-15133114
 ] 

Huikyo Lee commented on CLIMATE-744:


Traceback (most recent call last):
  File "run_RCMES.py", line 78, in 
start_time, end_time)
  File "/home/huikyole/climate/ocw/data_source/rcmed.py", line 358, in 
parameter_dataset
values = _reshape_values(values, unique_lats_lons_times)
  File "/home/huikyole/climate/ocw/data_source/rcmed.py", line 106, in 
_reshape_values
values = values.reshape(times_len, lats_len, lons_len)
ValueError: total size of new array must be unchanged

> Cannot load TRMM data from RCMED
> 
>
> Key: CLIMATE-744
> URL: https://issues.apache.org/jira/browse/CLIMATE-744
> Project: Apache Open Climate Workbench
>  Issue Type: Bug
>Reporter: Huikyo Lee
>Assignee: Maziyar Boustani
>Priority: Blocker
>
> For some reasons, rcmed.parameter_dataset causes errors when loading TRMM 
> precipitation data. 
> start_time = 1990-01-01
> end_time = 2007-12-31
> rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, start_time, 
> end_time)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)