[GitHub] Yongyao commented on a change in pull request #7: SDAP-35 Overhaul MUDROD configuration

[GitBox]

Yongyao commented on a change in pull request #7: SDAP-35 Overhaul MUDROD 
configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/7#discussion_r174137301
 
 

 ##
 File path: 
core/src/main/java/org/apache/sdap/mudrod/weblog/structure/ApacheAccessLog.java
 ##
 @@ -15,55 +15,49 @@
 
 import com.google.gson.Gson;
 
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.weblog.pre.CrawlerDetection;
+
 import java.io.IOException;
 import java.io.Serializable;
 import java.text.ParseException;
 import java.text.SimpleDateFormat;
 import java.util.Date;
+import java.util.Properties;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
-import org.apache.sdap.mudrod.weblog.pre.CrawlerDetection;
-
 /**
  * This class represents an Apache access log line. See
  * http://httpd.apache.org/docs/2.2/logs.html for more details.
  */
 public class ApacheAccessLog extends WebLog implements Serializable {
-
-
-  /**
-   * 
-   */
-  private static final long serialVersionUID = 1L;
-
-  public ApacheAccessLog() {
-//default constructor
-  }
-
-  String response;
-  String referer;
-  String browser;
+  String Response;
 
 Review comment:
   These variables were all upper case at the beginning. The previous 
enhancement PR changed it to lower case which can result in a null-pointer 
error, as they correspond to the field names in Elasticsearch which he didn't 
change. Therefore, I changed it back for the time being as these filed names 
are used explicitly at many places.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396957#comment-16396957
 ] 

ASF GitHub Bot commented on SDAP-35:


Yongyao commented on a change in pull request #7: SDAP-35 Overhaul MUDROD 
configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/7#discussion_r174137301
 
 

 ##
 File path: 
core/src/main/java/org/apache/sdap/mudrod/weblog/structure/ApacheAccessLog.java
 ##
 @@ -15,55 +15,49 @@
 
 import com.google.gson.Gson;
 
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.weblog.pre.CrawlerDetection;
+
 import java.io.IOException;
 import java.io.Serializable;
 import java.text.ParseException;
 import java.text.SimpleDateFormat;
 import java.util.Date;
+import java.util.Properties;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
-import org.apache.sdap.mudrod.weblog.pre.CrawlerDetection;
-
 /**
  * This class represents an Apache access log line. See
  * http://httpd.apache.org/docs/2.2/logs.html for more details.
  */
 public class ApacheAccessLog extends WebLog implements Serializable {
-
-
-  /**
-   * 
-   */
-  private static final long serialVersionUID = 1L;
-
-  public ApacheAccessLog() {
-//default constructor
-  }
-
-  String response;
-  String referer;
-  String browser;
+  String Response;
 
 Review comment:
   These variables were all upper case at the beginning. The previous 
enhancement PR changed it to lower case which can result in a null-pointer 
error, as they correspond to the field names in Elasticsearch which he didn't 
change. Therefore, I changed it back for the time being as these filed names 
are used explicitly at many places.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Yongyao commented on issue #7: SDAP-35 Overhaul MUDROD configuration

[GitBox]

Yongyao commented on issue #7: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/7#issuecomment-372672299
 
 
   @lewismc Yes, all make sense expect the one I just replied. I will submit a 
new PR today or tomorrow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396959#comment-16396959
 ] 

ASF GitHub Bot commented on SDAP-35:


Yongyao commented on issue #7: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/7#issuecomment-372672299
 
 
   @lewismc Yes, all make sense expect the one I just replied. I will submit a 
new PR today or tomorrow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] asfgit commented on issue #8: SDAP-35 Overhaul MUDROD configuration

[GitBox]

asfgit commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372693186
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397057#comment-16397057
 ] 

ASF GitHub Bot commented on SDAP-35:


asfgit commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372693186
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397056#comment-16397056
 ] 

ASF GitHub Bot commented on SDAP-35:


Yongyao opened a new pull request #8: SDAP-35 Overhaul MUDROD configuration
URL: https://github.com/apache/incubator-sdap-mudrod/pull/8
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Yongyao opened a new pull request #8: SDAP-35 Overhaul MUDROD configuration

[GitBox]

Yongyao opened a new pull request #8: SDAP-35 Overhaul MUDROD configuration
URL: https://github.com/apache/incubator-sdap-mudrod/pull/8
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration

[GitBox]

Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372693740
 
 
   @lewismc Please check this new PR out. All change requests have been made 
except for the upper case one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397060#comment-16397060
 ] 

ASF GitHub Bot commented on SDAP-35:


Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372693740
 
 
   @lewismc Please check this new PR out. All change requests have been made 
except for the upper case one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SDAP-45) Format all MUDROD code as 2 space indents

[Lewis John McGibbney (JIRA)]

Lewis John McGibbney created SDAP-45:


 Summary: Format all MUDROD code as 2 space indents
 Key: SDAP-45
 URL: https://issues.apache.org/jira/browse/SDAP-45
 Project: Apache Science Data Analytics Platform
  Issue Type: Improvement
Reporter: Lewis John McGibbney


We have been discussing formatting in MUDROD. The convention we are going to 
use is 2 space indents. I will put an Eclipse formatting template together and 
format all code accordingly for a PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-45) Format all MUDROD code as 2 space indents

[Lewis John McGibbney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-45?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney updated SDAP-45:
-
Component/s: mudrod

> Format all MUDROD code as 2 space indents
> -
>
> Key: SDAP-45
> URL: https://issues.apache.org/jira/browse/SDAP-45
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> We have been discussing formatting in MUDROD. The convention we are going to 
> use is 2 space indents. I will put an Eclipse formatting template together 
> and format all code accordingly for a PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration

[GitBox]

Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372721902
 
 
   @lewismc Format changed. I don't have permission to merge the PR. I saw you 
sent out some apache roster a while ago. Is this related to that?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397183#comment-16397183
 ] 

ASF GitHub Bot commented on SDAP-35:


Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372721902
 
 
   @lewismc Format changed. I don't have permission to merge the PR. I saw you 
sent out some apache roster a while ago. Is this related to that?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] lewismc commented on issue #8: SDAP-35 Overhaul MUDROD configuration

[GitBox]

lewismc commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372727170
 
 
   > I don't have permission to merge the PR. I saw you sent out some apache 
roster a while ago. Is this related to that?
   
   See https://gitbox.apache.org/setup/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397197#comment-16397197
 ] 

ASF GitHub Bot commented on SDAP-35:


lewismc commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372727170
 
 
   > I don't have permission to merge the PR. I saw you sent out some apache 
roster a while ago. Is this related to that?
   
   See https://gitbox.apache.org/setup/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SDAP-45) Format all MUDROD code as 2 space indents

[Lewis John McGibbney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-45?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney reassigned SDAP-45:


Assignee: Lewis John McGibbney

> Format all MUDROD code as 2 space indents
> -
>
> Key: SDAP-45
> URL: https://issues.apache.org/jira/browse/SDAP-45
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Major
>
> We have been discussing formatting in MUDROD. The convention we are going to 
> use is 2 space indents. I will put an Eclipse formatting template together 
> and format all code accordingly for a PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration

[GitBox]

Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372731812
 
 
   @lewismc It turns out I don't have an apahe id (https://id.apache.org/). I 
have access to https://issues.apache.org, but it looks like they are not the 
same thing. 
   As for the roaster email, the website ( 
https://whimsy.apache.org/roster/ppmc/sdap) is asking for my pw which I don't 
have either.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397222#comment-16397222
 ] 

ASF GitHub Bot commented on SDAP-35:


Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372731812
 
 
   @lewismc It turns out I don't have an apahe id (https://id.apache.org/). I 
have access to https://issues.apache.org, but it looks like they are not the 
same thing. 
   As for the roaster email, the website ( 
https://whimsy.apache.org/roster/ppmc/sdap) is asking for my pw which I don't 
have either.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SDAP-41) Update NEXUS tile service to apply a mask for start/end time

[Nga Chung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Chung reassigned SDAP-41:
-

Assignee: Nga Chung

> Update NEXUS tile service to apply a mask for start/end time
> 
>
> Key: SDAP-41
> URL: https://issues.apache.org/jira/browse/SDAP-41
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>Reporter: Nga Chung
>Assignee: Nga Chung
>Priority: Major
>
> Update NEXUS tile service to apply a mask for start/end time since we can no 
> longer assume that the tile will only contain data relevant to the requested 
> start/end time. A tile might now partially overlap the requested time frame, 
> so we should mask the irrelevant data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ntquach opened a new pull request #9: SDAP-42 Fix NEXUS tile service find_tiles_by_metadata

[GitBox]

ntquach opened a new pull request #9: SDAP-42 Fix NEXUS tile service 
find_tiles_by_metadata
URL: https://github.com/apache/incubator-sdap-nexus/pull/9
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-42) Fix NEXUS tile service find_tiles_by_metadata

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397255#comment-16397255
 ] 

ASF GitHub Bot commented on SDAP-42:


ntquach opened a new pull request #9: SDAP-42 Fix NEXUS tile service 
find_tiles_by_metadata
URL: https://github.com/apache/incubator-sdap-nexus/pull/9
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix NEXUS tile service find_tiles_by_metadata
> -
>
> Key: SDAP-42
> URL: https://issues.apache.org/jira/browse/SDAP-42
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>Reporter: Nga Chung
>Assignee: Nga Chung
>Priority: Major
>
> find_tiles_by_metadata by metadata implemented in SDAP-37 needs to be fixed 
> because it masks tiles by time before actual tile data was retrieved from 
> cassandra



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] lewismc commented on issue #8: SDAP-35 Overhaul MUDROD configuration

[GitBox]

lewismc commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372742125
 
 
   @Yongyao in id.apache.org you will see a reset password option. Use it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397272#comment-16397272
 ] 

ASF GitHub Bot commented on SDAP-35:


lewismc commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372742125
 
 
   @Yongyao in id.apache.org you will see a reset password option. Use it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SDAP-35) Overhaul MUDROD configuration

[Lewis John McGibbney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney reassigned SDAP-35:


Assignee: (was: Lewis John McGibbney)

> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-35) Overhaul MUDROD configuration

[Lewis John McGibbney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney updated SDAP-35:
-
Assignee: Lewis John McGibbney
  Status: In Progress  (was: To Do)

> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-35) Overhaul MUDROD configuration

[Lewis John McGibbney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney updated SDAP-35:
-
Resolution: Fixed
Status: Done  (was: In Progress)

> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Assignee: Yongyao Jiang
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SDAP-35) Overhaul MUDROD configuration

[Lewis John McGibbney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney reassigned SDAP-35:


Assignee: Yongyao Jiang

> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Assignee: Yongyao Jiang
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] fgreg closed pull request #3: SDAP-30

[GitBox]

fgreg closed pull request #3: SDAP-30
URL: https://github.com/apache/incubator-sdap-ningesterpy/pull/3
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdap/processors/tilereadingprocessor.py 
b/sdap/processors/tilereadingprocessor.py
index cc3afef..a483907 100644
--- a/sdap/processors/tilereadingprocessor.py
+++ b/sdap/processors/tilereadingprocessor.py
@@ -253,11 +253,8 @@ def read_data(self, tile_specifications, file_path, 
output_tile):
 tile.meta_data.add().CopyFrom(
 to_metadata(self.metadata, 
ds[self.metadata][tuple(ordered_slices.values())]))
 
-timevar = ds[self.time]
-# Note assumption is that index of time is start value in 
dimtoslice
-tile.time = 
to_seconds_from_epoch(timevar[dimtoslice[self.time].start],
-  
timeunits=timevar.getncattr('units'),
-  timeoffset=self.time_offset)
+tile.time.CopyFrom(
+
to_shaped_array(numpy.ma.filled(ds[self.time][dimtoslice[self.time]], 
numpy.NaN)))
 
 output_tile.tile.time_series_tile.CopyFrom(tile)
 
diff --git a/tests/datafiles/not_empty_wswm.nc 
b/tests/datafiles/not_empty_wswm.nc
index 772bbcb..ce0ebcc 100644
Binary files a/tests/datafiles/not_empty_wswm.nc and 
b/tests/datafiles/not_empty_wswm.nc differ
diff --git a/tests/tilereadingprocessor_test.py 
b/tests/tilereadingprocessor_test.py
index 2700073..927b436 100644
--- a/tests/tilereadingprocessor_test.py
+++ b/tests/tilereadingprocessor_test.py
@@ -275,7 +275,7 @@ def test_read_not_empty_wswm(self):
 input_tile = nexusproto.NexusTile()
 tile_summary = nexusproto.TileSummary()
 tile_summary.granule = "file:%s" % test_file
-tile_summary.section_spec = "time:0:1,rivid:0:500"
+tile_summary.section_spec = "time:0:5832,rivid:0:1"
 input_tile.summary.CopyFrom(tile_summary)
 
 results = list(wswm_reader.process(input_tile))
@@ -287,23 +287,25 @@ def test_read_not_empty_wswm(self):
 self.assertTrue(nexus_tile.tile.HasField('time_series_tile'))
 
 tile = nexus_tile.tile.time_series_tile
-self.assertEqual(500, from_shaped_array(tile.latitude).size)
-self.assertEqual(500, from_shaped_array(tile.longitude).size)
-self.assertEqual((1, 500), 
from_shaped_array(tile.variable_data).shape)
+self.assertEqual(1, from_shaped_array(tile.latitude).size)
+self.assertEqual(1, from_shaped_array(tile.longitude).size)
+self.assertEqual((5832, 1), 
from_shaped_array(tile.variable_data).shape)
 
 tile1_data = 
np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.variable_data))
-self.assertEqual(500, np.ma.count(tile1_data))
-self.assertAlmostEqual(41.390,
+self.assertEqual(5832, np.ma.count(tile1_data))
+self.assertAlmostEqual(45.837,
np.ma.min(

np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.latitude))),
places=3)
-self.assertAlmostEqual(42.071,
+self.assertAlmostEqual(-122.789,
np.ma.max(
-   
np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.latitude))),
+   
np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.longitude))),
places=3)
 
-self.assertEqual(852098400, results[0].tile.time_series_tile.time)
-self.assertAlmostEqual(0.009,
+tile1_times = from_shaped_array(results[0].tile.time_series_tile.time)
+self.assertEqual(852098400, tile1_times[0])
+self.assertEqual(915073200, tile1_times[-1])
+self.assertAlmostEqual(1.473,

np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.variable_data))[
0, 0],
places=3)


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-30) Tile metadata does not have correct min/max times

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397378#comment-16397378
 ] 

ASF GitHub Bot commented on SDAP-30:


fgreg closed pull request #3: SDAP-30
URL: https://github.com/apache/incubator-sdap-ningesterpy/pull/3
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdap/processors/tilereadingprocessor.py 
b/sdap/processors/tilereadingprocessor.py
index cc3afef..a483907 100644
--- a/sdap/processors/tilereadingprocessor.py
+++ b/sdap/processors/tilereadingprocessor.py
@@ -253,11 +253,8 @@ def read_data(self, tile_specifications, file_path, 
output_tile):
 tile.meta_data.add().CopyFrom(
 to_metadata(self.metadata, 
ds[self.metadata][tuple(ordered_slices.values())]))
 
-timevar = ds[self.time]
-# Note assumption is that index of time is start value in 
dimtoslice
-tile.time = 
to_seconds_from_epoch(timevar[dimtoslice[self.time].start],
-  
timeunits=timevar.getncattr('units'),
-  timeoffset=self.time_offset)
+tile.time.CopyFrom(
+
to_shaped_array(numpy.ma.filled(ds[self.time][dimtoslice[self.time]], 
numpy.NaN)))
 
 output_tile.tile.time_series_tile.CopyFrom(tile)
 
diff --git a/tests/datafiles/not_empty_wswm.nc 
b/tests/datafiles/not_empty_wswm.nc
index 772bbcb..ce0ebcc 100644
Binary files a/tests/datafiles/not_empty_wswm.nc and 
b/tests/datafiles/not_empty_wswm.nc differ
diff --git a/tests/tilereadingprocessor_test.py 
b/tests/tilereadingprocessor_test.py
index 2700073..927b436 100644
--- a/tests/tilereadingprocessor_test.py
+++ b/tests/tilereadingprocessor_test.py
@@ -275,7 +275,7 @@ def test_read_not_empty_wswm(self):
 input_tile = nexusproto.NexusTile()
 tile_summary = nexusproto.TileSummary()
 tile_summary.granule = "file:%s" % test_file
-tile_summary.section_spec = "time:0:1,rivid:0:500"
+tile_summary.section_spec = "time:0:5832,rivid:0:1"
 input_tile.summary.CopyFrom(tile_summary)
 
 results = list(wswm_reader.process(input_tile))
@@ -287,23 +287,25 @@ def test_read_not_empty_wswm(self):
 self.assertTrue(nexus_tile.tile.HasField('time_series_tile'))
 
 tile = nexus_tile.tile.time_series_tile
-self.assertEqual(500, from_shaped_array(tile.latitude).size)
-self.assertEqual(500, from_shaped_array(tile.longitude).size)
-self.assertEqual((1, 500), 
from_shaped_array(tile.variable_data).shape)
+self.assertEqual(1, from_shaped_array(tile.latitude).size)
+self.assertEqual(1, from_shaped_array(tile.longitude).size)
+self.assertEqual((5832, 1), 
from_shaped_array(tile.variable_data).shape)
 
 tile1_data = 
np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.variable_data))
-self.assertEqual(500, np.ma.count(tile1_data))
-self.assertAlmostEqual(41.390,
+self.assertEqual(5832, np.ma.count(tile1_data))
+self.assertAlmostEqual(45.837,
np.ma.min(

np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.latitude))),
places=3)
-self.assertAlmostEqual(42.071,
+self.assertAlmostEqual(-122.789,
np.ma.max(
-   
np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.latitude))),
+   
np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.longitude))),
places=3)
 
-self.assertEqual(852098400, results[0].tile.time_series_tile.time)
-self.assertAlmostEqual(0.009,
+tile1_times = from_shaped_array(results[0].tile.time_series_tile.time)
+self.assertEqual(852098400, tile1_times[0])
+self.assertEqual(915073200, tile1_times[-1])
+self.assertAlmostEqual(1.473,

np.ma.masked_invalid(from_shaped_array(results[0].tile.time_series_tile.variable_data))[
0, 0],
places=3)


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Tile met

[GitHub] fgreg closed pull request #4: SDAP-33 New Processor for adding Global Attributes

[GitBox]

fgreg closed pull request #4: SDAP-33 New Processor for adding Global Attributes
URL: https://github.com/apache/incubator-sdap-ningesterpy/pull/4
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdap/processors/__init__.py b/sdap/processors/__init__.py
index 8a48500..6d4a679 100644
--- a/sdap/processors/__init__.py
+++ b/sdap/processors/__init__.py
@@ -56,6 +56,7 @@ def process_nexus_tile(self, nexus_tile):
 from sdap.processors.emptytilefilter import EmptyTileFilter
 from sdap.processors.kelvintocelsius import KelvinToCelsius
 from sdap.processors.normalizetimebeginningofmonth import 
NormalizeTimeBeginningOfMonth
+from sdap.processors.promotevariabletoglobalattribute import 
PromoteVariableToGlobalAttribute
 from sdap.processors.regrid1x1 import Regrid1x1
 from sdap.processors.subtract180longitude import Subtract180Longitude
 from sdap.processors.tilereadingprocessor import GridReadingProcessor, 
SwathReadingProcessor, TimeSeriesReadingProcessor
@@ -70,6 +71,7 @@ def process_nexus_tile(self, nexus_tile):
 "EmptyTileFilter": EmptyTileFilter,
 "KelvinToCelsius": KelvinToCelsius,
 "NormalizeTimeBeginningOfMonth": NormalizeTimeBeginningOfMonth,
+"PromoteVariableToGlobalAttribute": PromoteVariableToGlobalAttribute,
 "Regrid1x1": Regrid1x1,
 "Subtract180Longitude": Subtract180Longitude,
 "GridReadingProcessor": GridReadingProcessor,
diff --git a/sdap/processors/processorchain.py 
b/sdap/processors/processorchain.py
index 6a68b04..82bad8e 100644
--- a/sdap/processors/processorchain.py
+++ b/sdap/processors/processorchain.py
@@ -14,6 +14,7 @@
 # limitations under the License.
 
 import inspect
+import re
 
 import sdap.processors
 
@@ -51,18 +52,37 @@ def __init__(self, processor_list, *args, **kwargs):
 except KeyError as e:
 raise ProcessorNotFound(processor['name']) from e
 
+processor_config = dict(**processor['config'])
+
 missing_args = []
 for arg in 
inspect.signature(processor_constructor).parameters.keys():
 if arg in ['args', 'kwargs']:
 continue
-if arg not in processor['config']:
+if arg not in processor_config:
 missing_args.append(arg)
 
+# Need to check for list type args
+list_pattern = re.compile('\.\d+$')
+list_args = [k for k in processor_config if list_pattern.search(k)]
+if list_args:
+import itertools
+grouped = itertools.groupby(list_args, key=lambda k: 
k.split('.')[0])
+for group, grouped_args in grouped:
+for list_arg in grouped_args:
+key, idx = list_arg.split('.')
+if group not in processor_config:
+processor_config[group] = []
+
+processor_config[group].insert(int(idx), 
processor_config[list_arg])
+del(processor_config[list_arg])
+
+# Check if the list args satisfied the
+missing_args = list(filter(lambda a: a not in 
processor_config.keys(), missing_args))
 if missing_args:
 raise MissingProcessorArguments(processor['name'], 
missing_args)
 
 if 'config' in processor.keys():
-processor_instance = 
processor_constructor(**processor['config'])
+processor_instance = processor_constructor(**processor_config)
 else:
 processor_instance = processor_constructor()
 
diff --git a/sdap/processors/promotevariabletoglobalattribute.py 
b/sdap/processors/promotevariabletoglobalattribute.py
new file mode 100644
index 000..11fa9d8
--- /dev/null
+++ b/sdap/processors/promotevariabletoglobalattribute.py
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import nexusproto.DataTile_pb2
+from netCDF4 import Dataset
+
+from sdap.processors import NexusTileProcessor
+
+
+class Promo

[jira] [Commented] (SDAP-33) Add new python processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397382#comment-16397382
 ] 

ASF GitHub Bot commented on SDAP-33:


fgreg closed pull request #4: SDAP-33 New Processor for adding Global Attributes
URL: https://github.com/apache/incubator-sdap-ningesterpy/pull/4
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdap/processors/__init__.py b/sdap/processors/__init__.py
index 8a48500..6d4a679 100644
--- a/sdap/processors/__init__.py
+++ b/sdap/processors/__init__.py
@@ -56,6 +56,7 @@ def process_nexus_tile(self, nexus_tile):
 from sdap.processors.emptytilefilter import EmptyTileFilter
 from sdap.processors.kelvintocelsius import KelvinToCelsius
 from sdap.processors.normalizetimebeginningofmonth import 
NormalizeTimeBeginningOfMonth
+from sdap.processors.promotevariabletoglobalattribute import 
PromoteVariableToGlobalAttribute
 from sdap.processors.regrid1x1 import Regrid1x1
 from sdap.processors.subtract180longitude import Subtract180Longitude
 from sdap.processors.tilereadingprocessor import GridReadingProcessor, 
SwathReadingProcessor, TimeSeriesReadingProcessor
@@ -70,6 +71,7 @@ def process_nexus_tile(self, nexus_tile):
 "EmptyTileFilter": EmptyTileFilter,
 "KelvinToCelsius": KelvinToCelsius,
 "NormalizeTimeBeginningOfMonth": NormalizeTimeBeginningOfMonth,
+"PromoteVariableToGlobalAttribute": PromoteVariableToGlobalAttribute,
 "Regrid1x1": Regrid1x1,
 "Subtract180Longitude": Subtract180Longitude,
 "GridReadingProcessor": GridReadingProcessor,
diff --git a/sdap/processors/processorchain.py 
b/sdap/processors/processorchain.py
index 6a68b04..82bad8e 100644
--- a/sdap/processors/processorchain.py
+++ b/sdap/processors/processorchain.py
@@ -14,6 +14,7 @@
 # limitations under the License.
 
 import inspect
+import re
 
 import sdap.processors
 
@@ -51,18 +52,37 @@ def __init__(self, processor_list, *args, **kwargs):
 except KeyError as e:
 raise ProcessorNotFound(processor['name']) from e
 
+processor_config = dict(**processor['config'])
+
 missing_args = []
 for arg in 
inspect.signature(processor_constructor).parameters.keys():
 if arg in ['args', 'kwargs']:
 continue
-if arg not in processor['config']:
+if arg not in processor_config:
 missing_args.append(arg)
 
+# Need to check for list type args
+list_pattern = re.compile('\.\d+$')
+list_args = [k for k in processor_config if list_pattern.search(k)]
+if list_args:
+import itertools
+grouped = itertools.groupby(list_args, key=lambda k: 
k.split('.')[0])
+for group, grouped_args in grouped:
+for list_arg in grouped_args:
+key, idx = list_arg.split('.')
+if group not in processor_config:
+processor_config[group] = []
+
+processor_config[group].insert(int(idx), 
processor_config[list_arg])
+del(processor_config[list_arg])
+
+# Check if the list args satisfied the
+missing_args = list(filter(lambda a: a not in 
processor_config.keys(), missing_args))
 if missing_args:
 raise MissingProcessorArguments(processor['name'], 
missing_args)
 
 if 'config' in processor.keys():
-processor_instance = 
processor_constructor(**processor['config'])
+processor_instance = processor_constructor(**processor_config)
 else:
 processor_instance = processor_constructor()
 
diff --git a/sdap/processors/promotevariabletoglobalattribute.py 
b/sdap/processors/promotevariabletoglobalattribute.py
new file mode 100644
index 000..11fa9d8
--- /dev/null
+++ b/sdap/processors/promotevariabletoglobalattribute.py
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express

[jira] [Updated] (SDAP-31) nexusproto TimeSeriesTile needs to accept ShapedArray for time

[Frank Greguska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-31?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-31:
---
Resolution: Fixed
Status: Done  (was: To Do)

> nexusproto TimeSeriesTile needs to accept ShapedArray for time
> --
>
> Key: SDAP-31
> URL: https://issues.apache.org/jira/browse/SDAP-31
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
> Currently the TimeSeriesTile in nexusproto only accepts an int64 as the time 
> for the tile. It needs to be able to store an array.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-30) Tile metadata does not have correct min/max times

[Frank Greguska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-30?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-30:
---
Resolution: Fixed
Status: Done  (was: To Do)

> Tile metadata does not have correct min/max times
> -
>
> Key: SDAP-30
> URL: https://issues.apache.org/jira/browse/SDAP-30
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Blocker
>
> When ingesting a tile with a time range instead of just 1 timestep, the 
> metadata in SOLR is not reflecting the min/max times correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-32) CassandraProxy time series tiles to support time arrays

[Frank Greguska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-32:
---
Resolution: Fixed
Status: Done  (was: To Do)

> CassandraProxy time series tiles to support time arrays
> ---
>
> Key: SDAP-32
> URL: https://issues.apache.org/jira/browse/SDAP-32
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
> CassandraProxy needs to be updated to deserialize a shaped array from the 
> time attribute in time series tiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-34) ningester to store custom attributes

[Frank Greguska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-34?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-34:
---
Resolution: Not A Problem
Status: Done  (was: To Do)

Ningester already stores any 'global attributes' that are stored in the tile 
summary. No changes needed.

> ningester to store custom attributes
> 
>
> Key: SDAP-34
> URL: https://issues.apache.org/jira/browse/SDAP-34
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
> Ningester should be updated to store custom attributes in the tile summary 
> into the metadata store.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-33) Add new python processor

[Frank Greguska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-33?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-33:
---
Resolution: Fixed
Status: Done  (was: In Progress)

> Add new python processor
> 
>
> Key: SDAP-33
> URL: https://issues.apache.org/jira/browse/SDAP-33
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
> New python processor should be able to add custom tile summary attributes 
> based on configuration values.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration

[GitBox]

Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372772738
 
 
   Thanks. Got it.
   
   On Tue, Mar 13, 2018 at 1:05 PM, Lewis John McGibbney <
   notificati...@github.com> wrote:
   
   > @Yongyao  in id.apache.org you will see a
   > reset password option. Use it.
   >
   > ?
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-35) Overhaul MUDROD configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397441#comment-16397441
 ] 

ASF GitHub Bot commented on SDAP-35:


Yongyao commented on issue #8: SDAP-35 Overhaul MUDROD configuration
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/8#issuecomment-372772738
 
 
   Thanks. Got it.
   
   On Tue, Mar 13, 2018 at 1:05 PM, Lewis John McGibbney <
   notificati...@github.com> wrote:
   
   > @Yongyao  in id.apache.org you will see a
   > reset password option. Use it.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Overhaul MUDROD configuration
> -
>
> Key: SDAP-35
> URL: https://issues.apache.org/jira/browse/SDAP-35
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: mudrod
>Reporter: Lewis John McGibbney
>Assignee: Yongyao Jiang
>Priority: Major
>
> [~Yongyao] please augment the description here with your intended patch as 
> per https://github.com/aist-oceanworks/mudrod/pull/215
> Also, please name your branch and commit message after the issue you create 
> in JIRA. It makes things much easier as we try to improve the quality of our 
> source code review and development workflow. Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ntquach opened a new pull request #10: SDAP-43 Update datainbounds algorithm to handle multiple time steps p?

[GitBox]

ntquach opened a new pull request #10: SDAP-43 Update datainbounds algorithm to 
handle multiple time steps p?
URL: https://github.com/apache/incubator-sdap-nexus/pull/10
 
 
   ?er tile


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-43) Update datainbounds algorithm to handle multiple time steps per tile

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397453#comment-16397453
 ] 

ASF GitHub Bot commented on SDAP-43:


ntquach opened a new pull request #10: SDAP-43 Update datainbounds algorithm to 
handle multiple time steps p…
URL: https://github.com/apache/incubator-sdap-nexus/pull/10
 
 
   …er tile


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update datainbounds algorithm to handle multiple time steps per tile
> 
>
> Key: SDAP-43
> URL: https://issues.apache.org/jira/browse/SDAP-43
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Nga Chung
>Assignee: Nga Chung
>Priority: Major
>
> To date, NEXUS has partitioned data to have 1 time step per tile.  For SWOT 
> we want to have multiple time steps per tile for efficiency of the 
> computations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-41) Update NEXUS tile service to apply a mask for start/end time

[Nga Chung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Chung updated SDAP-41:
--
Status: In Progress  (was: To Do)

> Update NEXUS tile service to apply a mask for start/end time
> 
>
> Key: SDAP-41
> URL: https://issues.apache.org/jira/browse/SDAP-41
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>Reporter: Nga Chung
>Assignee: Nga Chung
>Priority: Major
>
> Update NEXUS tile service to apply a mask for start/end time since we can no 
> longer assume that the tile will only contain data relevant to the requested 
> start/end time. A tile might now partially overlap the requested time frame, 
> so we should mask the irrelevant data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SDAP-46) Job Fails with no bean found exception

[Frank Greguska (JIRA)]

Frank Greguska created SDAP-46:
--

 Summary: Job Fails with no bean found exception
 Key: SDAP-46
 URL: https://issues.apache.org/jira/browse/SDAP-46
 Project: Apache Science Data Analytics Platform
  Issue Type: Bug
Reporter: Frank Greguska


 Error with ningester job while trying to execute on server

Command used:

{{java -Dspring.batch.enabled=false 
-Dspring.profiles.active=deepdata,solr,cassandra 
-Dspring.config.location=/usr/local/sdeploy/sdap/SWOTJob.yml -jar 
~/sdap/incubator-sdap-ningester/build/libs/incubator-sdap-ningester-1.0.0-SNAPSHOT.jar
 
granule=file:/data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc}}

Exception raised:

{{Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: 
No bean named 'pythonChainProcessor' available}}{{ at 
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:687)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1207)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:284)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
 ~[spring-context-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.apache.sdap.ningester.processors.CompositeItemProcessor.afterPropertiesSet(CompositeItemProcessor.java:44)
 ~[classes!/:na]}}{{ at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}

{{ }}

 

Looks like the conditional load is not catching the bean for some reason:

 

{{ BatchConfig#pythonChainProcessor:}}{{      }}

{{Did not match:}}

{{         - @ConditionalOnBean (types: 
org.apache.sdap.ningester.processors.PythonChainProcessor; SearchStrategy: all) 
did not find any beans (OnBeanCondition)}}{{ }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-46) Job Fails with no bean found exception

[Frank Greguska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-46?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-46:
---
Description: 
 Error with ningester job while trying to execute on server

Command used:

{{java -Dspring.batch.enabled=false 
-Dspring.profiles.active=deepdata,solr,cassandra 
-Dspring.config.location=/usr/local/sdeploy/sdap/SWOTJob.yml -jar 
~/sdap/incubator-sdap-ningester/build/libs/incubator-sdap-ningester-1.0.0-SNAPSHOT.jar
 
granule=[file:/data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc|file:///data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc]}}

Exception raised:
{quote}Caused by: 
org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 
'pythonChainProcessor' available

 at 
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:687)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]

 at 
org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1207)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]

 at 
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:284)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]

 at 
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]

 at 
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
 ~[spring-context-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]

 at 
org.apache.sdap.ningester.processors.CompositeItemProcessor.afterPropertiesSet(CompositeItemProcessor.java:44)
 ~[classes!/:na]

 at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]

 at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]

 ... 72 common frames omitted
{quote}
 

Looks like the conditional load is not catching the bean for some reason:

 

{{BatchConfig#pythonChainProcessor:}}

{{Did not match:}}

{{         - @ConditionalOnBean (types: 
org.apache.sdap.ningester.processors.PythonChainProcessor; SearchStrategy: all) 
did not find any beans (OnBeanCondition)}}{{ }}

  was:
 Error with ningester job while trying to execute on server

Command used:

{{java -Dspring.batch.enabled=false 
-Dspring.profiles.active=deepdata,solr,cassandra 
-Dspring.config.location=/usr/local/sdeploy/sdap/SWOTJob.yml -jar 
~/sdap/incubator-sdap-ningester/build/libs/incubator-sdap-ningester-1.0.0-SNAPSHOT.jar
 
granule=file:/data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc}}

Exception raised:

{{Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: 
No bean named 'pythonChainProcessor' available}}{{ at 
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:687)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1207)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:284)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
 ~[spring-context-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.apache.sdap.ningester.processors.CompositeItemProcessor.afterPropertiesSet(CompositeItemProcessor.java:44)
 ~[classes!/:na]}}{{ at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}{{ at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
 ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]}}

{{ }}

 

Looks like the conditional load is not catching the bean for some reason:

 

{{ BatchConfig#pythonChainProcessor:}}{{      }}

{{Did not match:}}

{{         - @ConditionalOnBean (types: 
org.apache.sdap.ningester.processors.PythonChainProcessor; SearchStrategy: all) 
did not find any beans (OnBeanCondition)}}{{ }}


> Job Fails with no bean found exception
> --

[jira] [Updated] (SDAP-46) Job Fails with no bean found exception

[Frank Greguska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-46?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-46:
---
Component/s: nexus

> Job Fails with no bean found exception
> --
>
> Key: SDAP-46
> URL: https://issues.apache.org/jira/browse/SDAP-46
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>  Components: nexus
>Reporter: Frank Greguska
>Priority: Major
>
>  Error with ningester job while trying to execute on server
> Command used:
> {{java -Dspring.batch.enabled=false 
> -Dspring.profiles.active=deepdata,solr,cassandra 
> -Dspring.config.location=/usr/local/sdeploy/sdap/SWOTJob.yml -jar 
> ~/sdap/incubator-sdap-ningester/build/libs/incubator-sdap-ningester-1.0.0-SNAPSHOT.jar
>  
> granule=[file:/data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc|file:///data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc]}}
> Exception raised:
> {quote}Caused by: 
> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean 
> named 'pythonChainProcessor' available
>  at 
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:687)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1207)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:284)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
>  ~[spring-context-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.apache.sdap.ningester.processors.CompositeItemProcessor.afterPropertiesSet(CompositeItemProcessor.java:44)
>  ~[classes!/:na]
>  at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  ... 72 common frames omitted
> {quote}
>  
> Looks like the conditional load is not catching the bean for some reason:
>  
> {{BatchConfig#pythonChainProcessor:}}
> {{Did not match:}}
> {{         - @ConditionalOnBean (types: 
> org.apache.sdap.ningester.processors.PythonChainProcessor; SearchStrategy: 
> all) did not find any beans (OnBeanCondition)}}{{ }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SDAP-46) Job Fails with no bean found exception

[Frank Greguska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-46?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska reassigned SDAP-46:
--

Assignee: Frank Greguska

> Job Fails with no bean found exception
> --
>
> Key: SDAP-46
> URL: https://issues.apache.org/jira/browse/SDAP-46
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
>  Error with ningester job while trying to execute on server
> Command used:
> {{java -Dspring.batch.enabled=false 
> -Dspring.profiles.active=deepdata,solr,cassandra 
> -Dspring.config.location=/usr/local/sdeploy/sdap/SWOTJob.yml -jar 
> ~/sdap/incubator-sdap-ningester/build/libs/incubator-sdap-ningester-1.0.0-SNAPSHOT.jar
>  
> granule=[file:/data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc|file:///data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc]}}
> Exception raised:
> {quote}Caused by: 
> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean 
> named 'pythonChainProcessor' available
>  at 
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:687)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1207)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:284)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
>  ~[spring-context-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.apache.sdap.ningester.processors.CompositeItemProcessor.afterPropertiesSet(CompositeItemProcessor.java:44)
>  ~[classes!/:na]
>  at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  ... 72 common frames omitted
> {quote}
>  
> Looks like the conditional load is not catching the bean for some reason:
>  
> {{BatchConfig#pythonChainProcessor:}}
> {{Did not match:}}
> {{         - @ConditionalOnBean (types: 
> org.apache.sdap.ningester.processors.PythonChainProcessor; SearchStrategy: 
> all) did not find any beans (OnBeanCondition)}}{{ }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SDAP-42) Fix NEXUS tile service find_tiles_by_metadata

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397698#comment-16397698
 ] 

ASF GitHub Bot commented on SDAP-42:


ntquach closed pull request #9: SDAP-42 Fix NEXUS tile service 
find_tiles_by_metadata
URL: https://github.com/apache/incubator-sdap-nexus/pull/9
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/data-access/nexustiles/nexustiles.py 
b/data-access/nexustiles/nexustiles.py
index 6ecb033..1330c03 100644
--- a/data-access/nexustiles/nexustiles.py
+++ b/data-access/nexustiles/nexustiles.py
@@ -166,7 +166,7 @@ def find_tiles_in_polygon(self, bounding_polygon, ds=None, 
start_time=0, end_tim
 @tile_data()
 def find_tiles_by_metadata(self, metadata, ds=None, start_time=0, 
end_time=-1, **kwargs):
 """
-Return list of tiles that matches the specified metadata, start_time, 
end_time.
+Return list of tiles whose metadata matches the specified metadata, 
start_time, end_time.
 :param metadata: List of metadata values to search for tiles e.g 
["river_id_i:1", "granule_s:granule_name"]
 :param ds: The dataset name to search
 :param start_time: The start time to search for tiles
@@ -174,6 +174,20 @@ def find_tiles_by_metadata(self, metadata, ds=None, 
start_time=0, end_time=-1, *
 :return: A list of tiles
 """
 tiles = self._metadatastore.find_all_tiles_by_metadata(metadata, ds, 
start_time, end_time, **kwargs)
+
+return tiles
+
+def get_tiles_by_metadata(self, metadata, ds=None, start_time=0, 
end_time=-1, **kwargs):
+"""
+Return list of tiles that matches the specified metadata, start_time, 
end_time with tile data outside of time
+range properly masked out.
+:param metadata: List of metadata values to search for tiles e.g 
["river_id_i:1", "granule_s:granule_name"]
+:param ds: The dataset name to search
+:param start_time: The start time to search for tiles
+:param end_time: The end time to search for tiles
+:return: A list of tiles
+"""
+tiles = self.find_tiles_by_metadata(metadata, ds, start_time, 
end_time, **kwargs)
 tiles = self.mask_tiles_to_time_range(start_time, end_time, tiles)
 
 return tiles
diff --git a/data-access/tests/nexustiles_test.py 
b/data-access/tests/nexustiles_test.py
index 9b2901e..9f533a8 100644
--- a/data-access/tests/nexustiles_test.py
+++ b/data-access/tests/nexustiles_test.py
@@ -76,6 +76,11 @@ def test_time_series_tile(self):
 for tile in tiles:
 print tile.get_summary()
 
+def test_get_tiles_by_metadata(self):
+tiles = 
self.tile_service.get_tiles_by_metadata(['id:60758e00-5721-3a6e-bf57-78448bb0aeeb'],
+
"MUR-JPL-L4-GLOB-v4.1", 1514764800, 1514764800)
+for tile in tiles:
+print tile.get_summary()
 
 # from nexustiles.model.nexusmodel import get_approximate_value_for_lat_lon
 # import numpy as np


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix NEXUS tile service find_tiles_by_metadata
> -
>
> Key: SDAP-42
> URL: https://issues.apache.org/jira/browse/SDAP-42
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>Reporter: Nga Chung
>Assignee: Nga Chung
>Priority: Major
>
> find_tiles_by_metadata by metadata implemented in SDAP-37 needs to be fixed 
> because it masks tiles by time before actual tile data was retrieved from 
> cassandra



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ntquach closed pull request #9: SDAP-42 Fix NEXUS tile service find_tiles_by_metadata

[GitBox]

ntquach closed pull request #9: SDAP-42 Fix NEXUS tile service 
find_tiles_by_metadata
URL: https://github.com/apache/incubator-sdap-nexus/pull/9
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/data-access/nexustiles/nexustiles.py 
b/data-access/nexustiles/nexustiles.py
index 6ecb033..1330c03 100644
--- a/data-access/nexustiles/nexustiles.py
+++ b/data-access/nexustiles/nexustiles.py
@@ -166,7 +166,7 @@ def find_tiles_in_polygon(self, bounding_polygon, ds=None, 
start_time=0, end_tim
 @tile_data()
 def find_tiles_by_metadata(self, metadata, ds=None, start_time=0, 
end_time=-1, **kwargs):
 """
-Return list of tiles that matches the specified metadata, start_time, 
end_time.
+Return list of tiles whose metadata matches the specified metadata, 
start_time, end_time.
 :param metadata: List of metadata values to search for tiles e.g 
["river_id_i:1", "granule_s:granule_name"]
 :param ds: The dataset name to search
 :param start_time: The start time to search for tiles
@@ -174,6 +174,20 @@ def find_tiles_by_metadata(self, metadata, ds=None, 
start_time=0, end_time=-1, *
 :return: A list of tiles
 """
 tiles = self._metadatastore.find_all_tiles_by_metadata(metadata, ds, 
start_time, end_time, **kwargs)
+
+return tiles
+
+def get_tiles_by_metadata(self, metadata, ds=None, start_time=0, 
end_time=-1, **kwargs):
+"""
+Return list of tiles that matches the specified metadata, start_time, 
end_time with tile data outside of time
+range properly masked out.
+:param metadata: List of metadata values to search for tiles e.g 
["river_id_i:1", "granule_s:granule_name"]
+:param ds: The dataset name to search
+:param start_time: The start time to search for tiles
+:param end_time: The end time to search for tiles
+:return: A list of tiles
+"""
+tiles = self.find_tiles_by_metadata(metadata, ds, start_time, 
end_time, **kwargs)
 tiles = self.mask_tiles_to_time_range(start_time, end_time, tiles)
 
 return tiles
diff --git a/data-access/tests/nexustiles_test.py 
b/data-access/tests/nexustiles_test.py
index 9b2901e..9f533a8 100644
--- a/data-access/tests/nexustiles_test.py
+++ b/data-access/tests/nexustiles_test.py
@@ -76,6 +76,11 @@ def test_time_series_tile(self):
 for tile in tiles:
 print tile.get_summary()
 
+def test_get_tiles_by_metadata(self):
+tiles = 
self.tile_service.get_tiles_by_metadata(['id:60758e00-5721-3a6e-bf57-78448bb0aeeb'],
+
"MUR-JPL-L4-GLOB-v4.1", 1514764800, 1514764800)
+for tile in tiles:
+print tile.get_summary()
 
 # from nexustiles.model.nexusmodel import get_approximate_value_for_lat_lon
 # import numpy as np


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-43) Update datainbounds algorithm to handle multiple time steps per tile

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397702#comment-16397702
 ] 

ASF GitHub Bot commented on SDAP-43:


ntquach closed pull request #10: SDAP-43 Update datainbounds algorithm to 
handle multiple time steps p…
URL: https://github.com/apache/incubator-sdap-nexus/pull/10
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/analysis/webservice/algorithms/DataInBoundsSearch.py 
b/analysis/webservice/algorithms/DataInBoundsSearch.py
index 2942649..80f0416 100644
--- a/analysis/webservice/algorithms/DataInBoundsSearch.py
+++ b/analysis/webservice/algorithms/DataInBoundsSearch.py
@@ -46,7 +46,7 @@ class DataInBoundsSearchHandlerImpl(NexusHandler):
 "name": "Bounding box",
 "type": "comma-delimited float",
 "description": "Minimum (Western) Longitude, Minimum (Southern) 
Latitude, "
-   "Maximum (Eastern) Longitude, Maximum (Northern) 
Latitude. Required"
+   "Maximum (Eastern) Longitude, Maximum (Northern) 
Latitude. Required if 'metadataFilter' not provided"
 },
 "startTime": {
 "name": "Start Time",
@@ -57,6 +57,11 @@ class DataInBoundsSearchHandlerImpl(NexusHandler):
 "name": "End Time",
 "type": "string",
 "description": "Ending time in format -MM-DDTHH:mm:ssZ or 
seconds since EPOCH. Required"
+},
+"metadataFilter": {
+"name": "Metadata Filter",
+"type": "string",
+"description": "Filter in format key:value. Required if 'b' not 
provided"
 }
 }
 singleton = True
@@ -100,27 +105,36 @@ def parse_arguments(self, request):
 request.get_start_datetime().strftime(ISO_8601), 
request.get_end_datetime().strftime(ISO_8601)),
 code=400)
 
+bounding_polygon = metadata_filter = None
 try:
 bounding_polygon = request.get_bounding_polygon()
 except:
-raise NexusProcessingException(
-reason="'b' argument is required. Must be comma-delimited 
float formatted as Minimum (Western) Longitude, Minimum (Southern) Latitude, 
Maximum (Eastern) Longitude, Maximum (Northern) Latitude",
-code=400)
+metadata_filter = request.get_metadata_filter()
+if 0 == len(metadata_filter):
+raise NexusProcessingException(
+reason="'b' or 'metadataFilter' argument is required. 'b' 
must be comma-delimited float formatted "
+   "as Minimum (Western) Longitude, Minimum (Southern) 
Latitude, Maximum (Eastern) Longitude, "
+   "Maximum (Northern) Latitude. 'metadataFilter' must 
be in the form key:value",
+code=400)
 
-return ds, parameter_s, start_time, end_time, bounding_polygon
+return ds, parameter_s, start_time, end_time, bounding_polygon, 
metadata_filter
 
 def calc(self, computeOptions, **args):
-ds, parameter, start_time, end_time, bounding_polygon = 
self.parse_arguments(computeOptions)
+ds, parameter, start_time, end_time, bounding_polygon, metadata_filter 
= self.parse_arguments(computeOptions)
 
 includemeta = computeOptions.get_include_meta()
 
-min_lat = bounding_polygon.bounds[1]
-max_lat = bounding_polygon.bounds[3]
-min_lon = bounding_polygon.bounds[0]
-max_lon = bounding_polygon.bounds[2]
+min_lat = max_lat = min_lon = max_lon = None
+if bounding_polygon:
+min_lat = bounding_polygon.bounds[1]
+max_lat = bounding_polygon.bounds[3]
+min_lon = bounding_polygon.bounds[0]
+max_lon = bounding_polygon.bounds[2]
 
-tiles = self._tile_service.get_tiles_bounded_by_box(min_lat, max_lat, 
min_lon, max_lon, ds, start_time,
-end_time)
+tiles = self._tile_service.get_tiles_bounded_by_box(min_lat, 
max_lat, min_lon, max_lon, ds, start_time,
+end_time)
+else:
+tiles = self._tile_service.get_tiles_by_metadata(metadata_filter, 
ds, start_time, end_time)
 
 data = []
 for tile in tiles:
@@ -148,7 +162,7 @@ def calc(self, computeOptions, **args):
 except (KeyError, IndexError):
 pass
 else:
-pass
+point['variable'] = nexus_point.data_val
 
 data.append({
 'latitude': nexus_point.l

[jira] [Commented] (SDAP-46) Job Fails with no bean found exception

[Frank Greguska (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-46?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397701#comment-16397701
 ] 

Frank Greguska commented on SDAP-46:


Problem was with the order in which the configuration classes were getting 
loaded. Some of the beans in BatchConfig are conditional on beans in AppConfig. 
Therefore, need to guarantee AppConfig gets processed before BatchConfig.

 

Fixed by adding {{@Order}} annotation to AppConfig and BatchConfig.

> Job Fails with no bean found exception
> --
>
> Key: SDAP-46
> URL: https://issues.apache.org/jira/browse/SDAP-46
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
>  Error with ningester job while trying to execute on server
> Command used:
> {{java -Dspring.batch.enabled=false 
> -Dspring.profiles.active=deepdata,solr,cassandra 
> -Dspring.config.location=/usr/local/sdeploy/sdap/SWOTJob.yml -jar 
> ~/sdap/incubator-sdap-ningester/build/libs/incubator-sdap-ningester-1.0.0-SNAPSHOT.jar
>  
> granule=[file:/data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc|file:///data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc]}}
> Exception raised:
> {quote}Caused by: 
> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean 
> named 'pythonChainProcessor' available
>  at 
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:687)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1207)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:284)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
>  ~[spring-context-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.apache.sdap.ningester.processors.CompositeItemProcessor.afterPropertiesSet(CompositeItemProcessor.java:44)
>  ~[classes!/:na]
>  at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  ... 72 common frames omitted
> {quote}
>  
> Looks like the conditional load is not catching the bean for some reason:
>  
> {{BatchConfig#pythonChainProcessor:}}
> {{Did not match:}}
> {{         - @ConditionalOnBean (types: 
> org.apache.sdap.ningester.processors.PythonChainProcessor; SearchStrategy: 
> all) did not find any beans (OnBeanCondition)}}{{ }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ntquach closed pull request #10: SDAP-43 Update datainbounds algorithm to handle multiple time steps p?

[GitBox]

ntquach closed pull request #10: SDAP-43 Update datainbounds algorithm to 
handle multiple time steps p?
URL: https://github.com/apache/incubator-sdap-nexus/pull/10
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/analysis/webservice/algorithms/DataInBoundsSearch.py 
b/analysis/webservice/algorithms/DataInBoundsSearch.py
index 2942649..80f0416 100644
--- a/analysis/webservice/algorithms/DataInBoundsSearch.py
+++ b/analysis/webservice/algorithms/DataInBoundsSearch.py
@@ -46,7 +46,7 @@ class DataInBoundsSearchHandlerImpl(NexusHandler):
 "name": "Bounding box",
 "type": "comma-delimited float",
 "description": "Minimum (Western) Longitude, Minimum (Southern) 
Latitude, "
-   "Maximum (Eastern) Longitude, Maximum (Northern) 
Latitude. Required"
+   "Maximum (Eastern) Longitude, Maximum (Northern) 
Latitude. Required if 'metadataFilter' not provided"
 },
 "startTime": {
 "name": "Start Time",
@@ -57,6 +57,11 @@ class DataInBoundsSearchHandlerImpl(NexusHandler):
 "name": "End Time",
 "type": "string",
 "description": "Ending time in format -MM-DDTHH:mm:ssZ or 
seconds since EPOCH. Required"
+},
+"metadataFilter": {
+"name": "Metadata Filter",
+"type": "string",
+"description": "Filter in format key:value. Required if 'b' not 
provided"
 }
 }
 singleton = True
@@ -100,27 +105,36 @@ def parse_arguments(self, request):
 request.get_start_datetime().strftime(ISO_8601), 
request.get_end_datetime().strftime(ISO_8601)),
 code=400)
 
+bounding_polygon = metadata_filter = None
 try:
 bounding_polygon = request.get_bounding_polygon()
 except:
-raise NexusProcessingException(
-reason="'b' argument is required. Must be comma-delimited 
float formatted as Minimum (Western) Longitude, Minimum (Southern) Latitude, 
Maximum (Eastern) Longitude, Maximum (Northern) Latitude",
-code=400)
+metadata_filter = request.get_metadata_filter()
+if 0 == len(metadata_filter):
+raise NexusProcessingException(
+reason="'b' or 'metadataFilter' argument is required. 'b' 
must be comma-delimited float formatted "
+   "as Minimum (Western) Longitude, Minimum (Southern) 
Latitude, Maximum (Eastern) Longitude, "
+   "Maximum (Northern) Latitude. 'metadataFilter' must 
be in the form key:value",
+code=400)
 
-return ds, parameter_s, start_time, end_time, bounding_polygon
+return ds, parameter_s, start_time, end_time, bounding_polygon, 
metadata_filter
 
 def calc(self, computeOptions, **args):
-ds, parameter, start_time, end_time, bounding_polygon = 
self.parse_arguments(computeOptions)
+ds, parameter, start_time, end_time, bounding_polygon, metadata_filter 
= self.parse_arguments(computeOptions)
 
 includemeta = computeOptions.get_include_meta()
 
-min_lat = bounding_polygon.bounds[1]
-max_lat = bounding_polygon.bounds[3]
-min_lon = bounding_polygon.bounds[0]
-max_lon = bounding_polygon.bounds[2]
+min_lat = max_lat = min_lon = max_lon = None
+if bounding_polygon:
+min_lat = bounding_polygon.bounds[1]
+max_lat = bounding_polygon.bounds[3]
+min_lon = bounding_polygon.bounds[0]
+max_lon = bounding_polygon.bounds[2]
 
-tiles = self._tile_service.get_tiles_bounded_by_box(min_lat, max_lat, 
min_lon, max_lon, ds, start_time,
-end_time)
+tiles = self._tile_service.get_tiles_bounded_by_box(min_lat, 
max_lat, min_lon, max_lon, ds, start_time,
+end_time)
+else:
+tiles = self._tile_service.get_tiles_by_metadata(metadata_filter, 
ds, start_time, end_time)
 
 data = []
 for tile in tiles:
@@ -148,7 +162,7 @@ def calc(self, computeOptions, **args):
 except (KeyError, IndexError):
 pass
 else:
-pass
+point['variable'] = nexus_point.data_val
 
 data.append({
 'latitude': nexus_point.latitude,
diff --git a/analysis/webservice/webmodel.py b/analysis/webservice/webmodel.py
index 3f01eac..e75ac01 100644
--- a/analysis/webservice/webmodel.py
+++ b/analysis/webservice/webmodel.py
@@ -51,6 +51,7 @@ class RequestParameters(object):
  

[jira] [Updated] (SDAP-46) Job Fails with no bean found exception

[Frank Greguska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-46?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-46:
---
Resolution: Fixed
Status: Done  (was: To Do)

Fix added to PR [https://github.com/apache/incubator-sdap-ningester/pull/2]

 

Commit: 
https://github.com/apache/incubator-sdap-ningester/pull/2/commits/a55e72d292630c0faaaf9bf69d40a56edbb877f8

> Job Fails with no bean found exception
> --
>
> Key: SDAP-46
> URL: https://issues.apache.org/jira/browse/SDAP-46
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
>  Error with ningester job while trying to execute on server
> Command used:
> {{java -Dspring.batch.enabled=false 
> -Dspring.profiles.active=deepdata,solr,cassandra 
> -Dspring.config.location=/usr/local/sdeploy/sdap/SWOTJob.yml -jar 
> ~/sdap/incubator-sdap-ningester/build/libs/incubator-sdap-ningester-1.0.0-SNAPSHOT.jar
>  
> granule=[file:/data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc|file:///data/share/datasets/SWOT_Hydro/latest/Qout_WSWM_729days_p0_dtR900s_n1_preonly_20160416.nc]}}
> Exception raised:
> {quote}Caused by: 
> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean 
> named 'pythonChainProcessor' available
>  at 
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:687)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1207)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:284)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
>  ~[spring-context-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.apache.sdap.ningester.processors.CompositeItemProcessor.afterPropertiesSet(CompositeItemProcessor.java:44)
>  ~[classes!/:na]
>  at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
>  ~[spring-beans-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
>  ... 72 common frames omitted
> {quote}
>  
> Looks like the conditional load is not catching the bean for some reason:
>  
> {{BatchConfig#pythonChainProcessor:}}
> {{Did not match:}}
> {{         - @ConditionalOnBean (types: 
> org.apache.sdap.ningester.processors.PythonChainProcessor; SearchStrategy: 
> all) did not find any beans (OnBeanCondition)}}{{ }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-42) Fix NEXUS tile service find_tiles_by_metadata

[Nga Chung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Chung updated SDAP-42:
--
Resolution: Fixed
Status: Done  (was: In Progress)

> Fix NEXUS tile service find_tiles_by_metadata
> -
>
> Key: SDAP-42
> URL: https://issues.apache.org/jira/browse/SDAP-42
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>Reporter: Nga Chung
>Assignee: Nga Chung
>Priority: Major
>
> find_tiles_by_metadata by metadata implemented in SDAP-37 needs to be fixed 
> because it masks tiles by time before actual tile data was retrieved from 
> cassandra



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SDAP-43) Update datainbounds algorithm to handle multiple time steps per tile

[Nga Chung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SDAP-43?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Chung updated SDAP-43:
--
Resolution: Fixed
Status: Done  (was: In Progress)

> Update datainbounds algorithm to handle multiple time steps per tile
> 
>
> Key: SDAP-43
> URL: https://issues.apache.org/jira/browse/SDAP-43
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Nga Chung
>Assignee: Nga Chung
>Priority: Major
>
> To date, NEXUS has partitioned data to have 1 time step per tile.  For SWOT 
> we want to have multiple time steps per tile for efficiency of the 
> computations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SDAP-47) Update NEXUS CLI to support datainbounds algorithm

[Nga Chung (JIRA)]

Nga Chung created SDAP-47:
-

 Summary: Update NEXUS CLI to support datainbounds algorithm
 Key: SDAP-47
 URL: https://issues.apache.org/jira/browse/SDAP-47
 Project: Apache Science Data Analytics Platform
  Issue Type: New Feature
Reporter: Nga Chung


Update the NEXUS CLI to support call to datainbounds algorithm



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ntquach opened a new pull request #11: SDAP-41 Update NEXUS tile service to apply a mask for start/end time

[GitBox]

ntquach opened a new pull request #11: SDAP-41 Update NEXUS tile service to 
apply a mask for start/end time
URL: https://github.com/apache/incubator-sdap-nexus/pull/11
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-41) Update NEXUS tile service to apply a mask for start/end time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397871#comment-16397871
 ] 

ASF GitHub Bot commented on SDAP-41:


ntquach opened a new pull request #11: SDAP-41 Update NEXUS tile service to 
apply a mask for start/end time
URL: https://github.com/apache/incubator-sdap-nexus/pull/11
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update NEXUS tile service to apply a mask for start/end time
> 
>
> Key: SDAP-41
> URL: https://issues.apache.org/jira/browse/SDAP-41
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>Reporter: Nga Chung
>Assignee: Nga Chung
>Priority: Major
>
> Update NEXUS tile service to apply a mask for start/end time since we can no 
> longer assume that the tile will only contain data relevant to the requested 
> start/end time. A tile might now partially overlap the requested time frame, 
> so we should mask the irrelevant data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] fgreg commented on a change in pull request #8: SDAP-40 Update Spark Time Series algorithm to handle multiple time steps per tile

[GitBox]

fgreg commented on a change in pull request #8: SDAP-40 Update Spark Time 
Series algorithm to handle multiple time steps per tile
URL: https://github.com/apache/incubator-sdap-nexus/pull/8#discussion_r174330935
 
 

 ##
 File path: analysis/webservice/algorithms_spark/TimeSeriesSpark.py
 ##
 @@ -178,27 +178,10 @@ def calc(self, request, **args):
 for shortName in ds:
 
 the_time = datetime.now()
-daysinrange = 
self._tile_service.find_days_in_range_asc(bounding_polygon.bounds[1],
-
bounding_polygon.bounds[3],
-
bounding_polygon.bounds[0],
-
bounding_polygon.bounds[2],
-shortName,
-
start_seconds_from_epoch,
-
end_seconds_from_epoch)
-self.log.info("Finding days in range took %s for dataset %s" % 
(str(datetime.now() - the_time), shortName))
-
-ndays = len(daysinrange)
-if ndays == 0:
-raise NoDataException(reason="No data found for selected 
timeframe")
-
-self.log.debug('Found {0} days in range'.format(ndays))
-for i, d in enumerate(daysinrange):
-self.log.debug('{0}, {1}'.format(i, 
datetime.utcfromtimestamp(d)))
-spark_nparts_needed = min(spark_nparts, ndays)
-
-the_time = datetime.now()
-results, meta = spark_driver(daysinrange, bounding_polygon, 
shortName,
- 
spark_nparts_needed=spark_nparts_needed, sc=self._sc)
+results, meta = spark_driver(start_seconds_from_epoch,
 
 Review comment:
   This is good, but I'm thinking we should run a sanity check query to make 
sure there are at least some tiles in the search domain before passing off to 
Spark. Otherwise we incur the cost of spinning up a bunch of spark tasks for no 
reason.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (SDAP-40) Update Spark Time Series algorithm to handle multiple time steps per tile

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SDAP-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397947#comment-16397947
 ] 

ASF GitHub Bot commented on SDAP-40:


fgreg commented on a change in pull request #8: SDAP-40 Update Spark Time 
Series algorithm to handle multiple time steps per tile
URL: https://github.com/apache/incubator-sdap-nexus/pull/8#discussion_r174330935
 
 

 ##
 File path: analysis/webservice/algorithms_spark/TimeSeriesSpark.py
 ##
 @@ -178,27 +178,10 @@ def calc(self, request, **args):
 for shortName in ds:
 
 the_time = datetime.now()
-daysinrange = 
self._tile_service.find_days_in_range_asc(bounding_polygon.bounds[1],
-
bounding_polygon.bounds[3],
-
bounding_polygon.bounds[0],
-
bounding_polygon.bounds[2],
-shortName,
-
start_seconds_from_epoch,
-
end_seconds_from_epoch)
-self.log.info("Finding days in range took %s for dataset %s" % 
(str(datetime.now() - the_time), shortName))
-
-ndays = len(daysinrange)
-if ndays == 0:
-raise NoDataException(reason="No data found for selected 
timeframe")
-
-self.log.debug('Found {0} days in range'.format(ndays))
-for i, d in enumerate(daysinrange):
-self.log.debug('{0}, {1}'.format(i, 
datetime.utcfromtimestamp(d)))
-spark_nparts_needed = min(spark_nparts, ndays)
-
-the_time = datetime.now()
-results, meta = spark_driver(daysinrange, bounding_polygon, 
shortName,
- 
spark_nparts_needed=spark_nparts_needed, sc=self._sc)
+results, meta = spark_driver(start_seconds_from_epoch,
 
 Review comment:
   This is good, but I'm thinking we should run a sanity check query to make 
sure there are at least some tiles in the search domain before passing off to 
Spark. Otherwise we incur the cost of spinning up a bunch of spark tasks for no 
reason.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update Spark Time Series algorithm to handle multiple time steps per tile
> -
>
> Key: SDAP-40
> URL: https://issues.apache.org/jira/browse/SDAP-40
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Joseph Jacob
>Assignee: Joseph Jacob
>Priority: Major
>
> To date, NEXUS has partitioned data to have 1 time step per tile.  For SWOT 
> we want to have multiple time steps per tile for efficiency of the 
> computations.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)