[jira] [Created] (SDAP-33) Add new python processor

2018-03-06 Thread Frank Greguska (JIRA)
Frank Greguska created SDAP-33:
--

 Summary: Add new python processor
 Key: SDAP-33
 URL: https://issues.apache.org/jira/browse/SDAP-33
 Project: Apache Science Data Analytics Platform
  Issue Type: Sub-task
Reporter: Frank Greguska


New python processor should be able to add custom tile summary attributes based 
on configuration values.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (SDAP-33) Add new python processor

2018-03-06 Thread Frank Greguska (JIRA)

 [ 
https://issues.apache.org/jira/browse/SDAP-33?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-33:
---
Status: In Progress  (was: To Do)

> Add new python processor
> 
>
> Key: SDAP-33
> URL: https://issues.apache.org/jira/browse/SDAP-33
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
> New python processor should be able to add custom tile summary attributes 
> based on configuration values.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] fgreg commented on a change in pull request #2: SDAP-28 Ingest SWOT Sample Data Tiled by time

2018-03-06 Thread GitBox
fgreg commented on a change in pull request #2: SDAP-28 Ingest SWOT Sample Data 
Tiled by time
URL: 
https://github.com/apache/incubator-sdap-ningester/pull/2#discussion_r172628732
 
 

 ##
 File path: 
src/main/java/org/apache/sdap/ningester/datatiler/SliceFileByStepSize.java
 ##
 @@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.sdap.ningester.datatiler;
+
+import com.google.common.collect.Sets;
+import ucar.nc2.Dimension;
+import ucar.nc2.dataset.NetcdfDataset;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
 
 Review comment:
   It's IntelliJ automatically collapsing like 5 util imports down to 1. TBH 
I'm not sure how to turn it off.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (SDAP-28) Ingest SWOT Sample Data Tiled by time

2018-03-06 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SDAP-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388354#comment-16388354
 ] 

ASF GitHub Bot commented on SDAP-28:


fgreg commented on a change in pull request #2: SDAP-28 Ingest SWOT Sample Data 
Tiled by time
URL: 
https://github.com/apache/incubator-sdap-ningester/pull/2#discussion_r172628732
 
 

 ##
 File path: 
src/main/java/org/apache/sdap/ningester/datatiler/SliceFileByStepSize.java
 ##
 @@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.sdap.ningester.datatiler;
+
+import com.google.common.collect.Sets;
+import ucar.nc2.Dimension;
+import ucar.nc2.dataset.NetcdfDataset;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
 
 Review comment:
   It's IntelliJ automatically collapsing like 5 util imports down to 1. TBH 
I'm not sure how to turn it off.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Ingest SWOT Sample Data Tiled by time
> -
>
> Key: SDAP-28
> URL: https://issues.apache.org/jira/browse/SDAP-28
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
> We want to try ingesting SWOT sample data tiled by time instead of rivid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (SDAP-32) CassandraProxy time series tiles to support time arrays

2018-03-06 Thread Frank Greguska (JIRA)

 [ 
https://issues.apache.org/jira/browse/SDAP-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-32:
---
Status: To Do  (was: In Progress)

> CassandraProxy time series tiles to support time arrays
> ---
>
> Key: SDAP-32
> URL: https://issues.apache.org/jira/browse/SDAP-32
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
> CassandraProxy needs to be updated to deserialize a shaped array from the 
> time attribute in time series tiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (SDAP-34) ningester to store custom attributes

2018-03-06 Thread Frank Greguska (JIRA)

 [ 
https://issues.apache.org/jira/browse/SDAP-34?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-34:
---
Component/s: nexus

> ningester to store custom attributes
> 
>
> Key: SDAP-34
> URL: https://issues.apache.org/jira/browse/SDAP-34
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Priority: Major
>
> Ningester should be updated to store custom attributes in the tile summary 
> into the metadata store.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (SDAP-33) Add new python processor

2018-03-06 Thread Frank Greguska (JIRA)

 [ 
https://issues.apache.org/jira/browse/SDAP-33?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Greguska updated SDAP-33:
---
Component/s: nexus

> Add new python processor
> 
>
> Key: SDAP-33
> URL: https://issues.apache.org/jira/browse/SDAP-33
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
> New python processor should be able to add custom tile summary attributes 
> based on configuration values.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (SDAP-34) ningester to store custom attributes

2018-03-06 Thread Frank Greguska (JIRA)
Frank Greguska created SDAP-34:
--

 Summary: ningester to store custom attributes
 Key: SDAP-34
 URL: https://issues.apache.org/jira/browse/SDAP-34
 Project: Apache Science Data Analytics Platform
  Issue Type: Sub-task
Reporter: Frank Greguska


Ningester should be updated to store custom attributes in the tile summary into 
the metadata store.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] fgreg opened a new pull request #6: SDAP-32 CassandraProxy time series tiles to support time arrays

2018-03-06 Thread GitBox
fgreg opened a new pull request #6: SDAP-32 CassandraProxy time series tiles to 
support time arrays
URL: https://github.com/apache/incubator-sdap-nexus/pull/6
 
 
   This PR alters the data-access library for time series tiles. It now expects 
time series tiles to have an array for their time attribute instead of a single 
value.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (SDAP-33) Add new python processor

2018-03-06 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SDAP-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388758#comment-16388758
 ] 

ASF GitHub Bot commented on SDAP-33:


fgreg opened a new pull request #4: SDAP-33 New Processor for adding Global 
Attributes
URL: https://github.com/apache/incubator-sdap-ningesterpy/pull/4
 
 
   This PR adds a python processor which can promote the value inside a 
variable into a global attribute on the tile summary. This is to support being 
able to search by rivid for the example SWOT data.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add new python processor
> 
>
> Key: SDAP-33
> URL: https://issues.apache.org/jira/browse/SDAP-33
> Project: Apache Science Data Analytics Platform
>  Issue Type: Sub-task
>  Components: nexus
>Reporter: Frank Greguska
>Assignee: Frank Greguska
>Priority: Major
>
> New python processor should be able to add custom tile summary attributes 
> based on configuration values.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] fgreg opened a new pull request #4: SDAP-33 New Processor for adding Global Attributes

2018-03-06 Thread GitBox
fgreg opened a new pull request #4: SDAP-33 New Processor for adding Global 
Attributes
URL: https://github.com/apache/incubator-sdap-ningesterpy/pull/4
 
 
   This PR adds a python processor which can promote the value inside a 
variable into a global attribute on the tile summary. This is to support being 
able to search by rivid for the example SWOT data.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (SDAP-25) Remove all non-Apache material from MUDROD parent pom.xml

2018-03-06 Thread Lewis John McGibbney (JIRA)

 [ 
https://issues.apache.org/jira/browse/SDAP-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney updated SDAP-25:
-
Summary: Remove all non-Apache material from MUDROD parent pom.xml  (was: 
Remove all non-Apache material from parent pom.xml)

> Remove all non-Apache material from MUDROD parent pom.xml
> -
>
> Key: SDAP-25
> URL: https://issues.apache.org/jira/browse/SDAP-25
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Critical
>
> The SNAPSHOT repository, organization, CI, etc. Maven elements need to be 
> corrected. PR coming up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (SDAP-25) Remove all non-Apache material from MUDROD parent pom.xml

2018-03-06 Thread Lewis John McGibbney (JIRA)

 [ 
https://issues.apache.org/jira/browse/SDAP-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney updated SDAP-25:
-
Resolution: Fixed
Status: Done  (was: In Progress)

> Remove all non-Apache material from MUDROD parent pom.xml
> -
>
> Key: SDAP-25
> URL: https://issues.apache.org/jira/browse/SDAP-25
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Critical
>
> The SNAPSHOT repository, organization, CI, etc. Maven elements need to be 
> corrected. PR coming up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (SDAP-18) Review code contribition from Sigee

2018-03-06 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SDAP-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388936#comment-16388936
 ] 

ASF GitHub Bot commented on SDAP-18:


lewismc opened a new pull request #5: SDAP-18 Review code contribition from 
Sigee
URL: https://github.com/apache/incubator-sdap-mudrod/pull/5
 
 
   This issue supersedes the PR from @sigee at 
https://github.com/aist-oceanworks/mudrod/pull/214 and addresses 
https://issues.apache.org/jira/projects/SDAP/issues/SDAP-18
   
   Thank you @sigee


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Review code contribition from Sigee 
> 
>
> Key: SDAP-18
> URL: https://issues.apache.org/jira/browse/SDAP-18
> Project: Apache Science Data Analytics Platform
>  Issue Type: New Feature
>Reporter: Lewis John McGibbney
>Priority: Major
>
> Until right now I did not see the contribution at 
> [https://github.com/aist-oceanworks/mudrod/pull/214]
> We should review this and merge into master mudrod branch



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] lewismc opened a new pull request #5: SDAP-18 Review code contribition from Sigee

2018-03-06 Thread GitBox
lewismc opened a new pull request #5: SDAP-18 Review code contribition from 
Sigee
URL: https://github.com/apache/incubator-sdap-mudrod/pull/5
 
 
   This issue supersedes the PR from @sigee at 
https://github.com/aist-oceanworks/mudrod/pull/214 and addresses 
https://issues.apache.org/jira/projects/SDAP/issues/SDAP-18
   
   Thank you @sigee


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (SDAP-18) Review code contribition from Sigee

2018-03-06 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SDAP-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388939#comment-16388939
 ] 

ASF GitHub Bot commented on SDAP-18:


lewismc closed pull request #5: SDAP-18 Review code contribition from Sigee
URL: https://github.com/apache/incubator-sdap-mudrod/pull/5
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/RecommendEngine.java
 
b/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/RecommendEngine.java
index 2c829ff..d54a556 100644
--- 
a/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/RecommendEngine.java
+++ 
b/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/RecommendEngine.java
@@ -8,7 +8,7 @@
 import org.apache.sdap.mudrod.recommendation.pre.SessionCooccurence;
 import org.apache.sdap.mudrod.recommendation.process.AbstractBasedSimilarity;
 import org.apache.sdap.mudrod.recommendation.process.VariableBasedSimilarity;
-import org.apache.sdap.mudrod.recommendation.process.sessionBasedCF;
+import org.apache.sdap.mudrod.recommendation.process.SessionBasedCF;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -60,7 +60,7 @@ public void process() {
 DiscoveryStepAbstract cbCF = new VariableBasedSimilarity(this.props, 
this.es, this.spark);
 cbCF.execute();
 
-DiscoveryStepAbstract sbCF = new sessionBasedCF(this.props, this.es, 
this.spark);
+DiscoveryStepAbstract sbCF = new SessionBasedCF(this.props, this.es, 
this.spark);
 sbCF.execute();
 
 endTime = System.currentTimeMillis();
diff --git 
a/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngine.java
 
b/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngine.java
index b672a54..b2cdb9f 100644
--- 
a/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngine.java
+++ 
b/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngine.java
@@ -99,11 +99,11 @@ public void preprocess() {
 
 ArrayList inputList = (ArrayList) 
getFileList(props.getProperty(MudrodConstants.DATA_DIR));
 
-for (int i = 0; i < inputList.size(); i++) {
-  timeSuffix = inputList.get(i);
+for (String anInputList : inputList) {
+  timeSuffix = anInputList;
   props.put(MudrodConstants.TIME_SUFFIX, timeSuffix);
   startTime = System.currentTimeMillis();
-  LOG.info("Processing logs dated {}", inputList.get(i));
+  LOG.info("Processing logs dated {}", anInputList);
 
   DiscoveryStepAbstract im = new ImportLogFile(this.props, this.es, 
this.spark);
   im.execute();
@@ -122,7 +122,7 @@ public void preprocess() {
 
   endTime = System.currentTimeMillis();
 
-  LOG.info("Web log preprocessing for logs dated {} complete. Time elapsed 
{} seconds.", inputList.get(i), (endTime - startTime) / 1000);
+  LOG.info("Web log preprocessing for logs dated {} complete. Time elapsed 
{} seconds.", anInputList, (endTime - startTime) / 1000);
 }
 
 DiscoveryStepAbstract hg = new HistoryGenerator(this.props, this.es, 
this.spark);
@@ -140,8 +140,8 @@ public void preprocess() {
   public void logIngest() {
 LOG.info("Starting Web log ingest.");
 ArrayList inputList = (ArrayList) 
getFileList(props.getProperty(MudrodConstants.DATA_DIR));
-for (int i = 0; i < inputList.size(); i++) {
-  timeSuffix = inputList.get(i);
+for (String anInputList : inputList) {
+  timeSuffix = anInputList;
   props.put("TimeSuffix", timeSuffix);
   DiscoveryStepAbstract im = new ImportLogFile(this.props, this.es, 
this.spark);
   im.execute();
@@ -157,8 +157,8 @@ public void logIngest() {
   public void sessionRestruct() {
 LOG.info("Starting Session reconstruction.");
 ArrayList inputList = (ArrayList) 
getFileList(props.getProperty(MudrodConstants.DATA_DIR));
-for (int i = 0; i < inputList.size(); i++) {
-  timeSuffix = inputList.get(i); // change timeSuffix dynamically
+for (String anInputList : inputList) {
+  timeSuffix = anInputList; // change timeSuffix dynamically
   props.put(MudrodConstants.TIME_SUFFIX, timeSuffix);
   DiscoveryStepAbstract cd = new CrawlerDetection(this.props, this.es, 
this.spark);
   cd.execute();
diff --git a/core/src/main/java/org/apache/sdap/mudrod/driver/ESDriver.java 
b/core/src/main/java/org/apache/sdap/mudrod/driver/ESDriver.java
index 7fe0cb4..54b9128 100644
--- a/core/src/main/java/org/apache/sdap/mudrod/driver/ESDriver.java
+++ b/core/src/main/java/org/apache/sdap/mudrod/driver/ESDriver.java
@@ -168,10 +168,9 @@ public String customAnalyzing(String indexName, String 
analyzer, String str) thr
 if (list == 

[GitHub] asfgit commented on issue #5: SDAP-18 Review code contribition from Sigee

2018-03-06 Thread GitBox
asfgit commented on issue #5: SDAP-18 Review code contribition from Sigee
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/5#issuecomment-371008510
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] lewismc closed pull request #5: SDAP-18 Review code contribition from Sigee

2018-03-06 Thread GitBox
lewismc closed pull request #5: SDAP-18 Review code contribition from Sigee
URL: https://github.com/apache/incubator-sdap-mudrod/pull/5
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/RecommendEngine.java
 
b/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/RecommendEngine.java
index 2c829ff..d54a556 100644
--- 
a/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/RecommendEngine.java
+++ 
b/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/RecommendEngine.java
@@ -8,7 +8,7 @@
 import org.apache.sdap.mudrod.recommendation.pre.SessionCooccurence;
 import org.apache.sdap.mudrod.recommendation.process.AbstractBasedSimilarity;
 import org.apache.sdap.mudrod.recommendation.process.VariableBasedSimilarity;
-import org.apache.sdap.mudrod.recommendation.process.sessionBasedCF;
+import org.apache.sdap.mudrod.recommendation.process.SessionBasedCF;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -60,7 +60,7 @@ public void process() {
 DiscoveryStepAbstract cbCF = new VariableBasedSimilarity(this.props, 
this.es, this.spark);
 cbCF.execute();
 
-DiscoveryStepAbstract sbCF = new sessionBasedCF(this.props, this.es, 
this.spark);
+DiscoveryStepAbstract sbCF = new SessionBasedCF(this.props, this.es, 
this.spark);
 sbCF.execute();
 
 endTime = System.currentTimeMillis();
diff --git 
a/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngine.java
 
b/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngine.java
index b672a54..b2cdb9f 100644
--- 
a/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngine.java
+++ 
b/core/src/main/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngine.java
@@ -99,11 +99,11 @@ public void preprocess() {
 
 ArrayList inputList = (ArrayList) 
getFileList(props.getProperty(MudrodConstants.DATA_DIR));
 
-for (int i = 0; i < inputList.size(); i++) {
-  timeSuffix = inputList.get(i);
+for (String anInputList : inputList) {
+  timeSuffix = anInputList;
   props.put(MudrodConstants.TIME_SUFFIX, timeSuffix);
   startTime = System.currentTimeMillis();
-  LOG.info("Processing logs dated {}", inputList.get(i));
+  LOG.info("Processing logs dated {}", anInputList);
 
   DiscoveryStepAbstract im = new ImportLogFile(this.props, this.es, 
this.spark);
   im.execute();
@@ -122,7 +122,7 @@ public void preprocess() {
 
   endTime = System.currentTimeMillis();
 
-  LOG.info("Web log preprocessing for logs dated {} complete. Time elapsed 
{} seconds.", inputList.get(i), (endTime - startTime) / 1000);
+  LOG.info("Web log preprocessing for logs dated {} complete. Time elapsed 
{} seconds.", anInputList, (endTime - startTime) / 1000);
 }
 
 DiscoveryStepAbstract hg = new HistoryGenerator(this.props, this.es, 
this.spark);
@@ -140,8 +140,8 @@ public void preprocess() {
   public void logIngest() {
 LOG.info("Starting Web log ingest.");
 ArrayList inputList = (ArrayList) 
getFileList(props.getProperty(MudrodConstants.DATA_DIR));
-for (int i = 0; i < inputList.size(); i++) {
-  timeSuffix = inputList.get(i);
+for (String anInputList : inputList) {
+  timeSuffix = anInputList;
   props.put("TimeSuffix", timeSuffix);
   DiscoveryStepAbstract im = new ImportLogFile(this.props, this.es, 
this.spark);
   im.execute();
@@ -157,8 +157,8 @@ public void logIngest() {
   public void sessionRestruct() {
 LOG.info("Starting Session reconstruction.");
 ArrayList inputList = (ArrayList) 
getFileList(props.getProperty(MudrodConstants.DATA_DIR));
-for (int i = 0; i < inputList.size(); i++) {
-  timeSuffix = inputList.get(i); // change timeSuffix dynamically
+for (String anInputList : inputList) {
+  timeSuffix = anInputList; // change timeSuffix dynamically
   props.put(MudrodConstants.TIME_SUFFIX, timeSuffix);
   DiscoveryStepAbstract cd = new CrawlerDetection(this.props, this.es, 
this.spark);
   cd.execute();
diff --git a/core/src/main/java/org/apache/sdap/mudrod/driver/ESDriver.java 
b/core/src/main/java/org/apache/sdap/mudrod/driver/ESDriver.java
index 7fe0cb4..54b9128 100644
--- a/core/src/main/java/org/apache/sdap/mudrod/driver/ESDriver.java
+++ b/core/src/main/java/org/apache/sdap/mudrod/driver/ESDriver.java
@@ -168,10 +168,9 @@ public String customAnalyzing(String indexName, String 
analyzer, String str) thr
 if (list == null) {
   return list;
 }
-int size = list.size();
 List customlist = new ArrayList<>();
-for (int i = 0; i < size; i++) {
-  customlist.add(this.customAnalyzing(indexName, list.get(i)));
+for (String 

[jira] [Updated] (SDAP-18) Review code contribition from Sigee

2018-03-06 Thread Lewis John McGibbney (JIRA)

 [ 
https://issues.apache.org/jira/browse/SDAP-18?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney updated SDAP-18:
-
Resolution: Fixed
Status: Done  (was: In Progress)

> Review code contribition from Sigee 
> 
>
> Key: SDAP-18
> URL: https://issues.apache.org/jira/browse/SDAP-18
> Project: Apache Science Data Analytics Platform
>  Issue Type: New Feature
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Major
>
> Until right now I did not see the contribution at 
> [https://github.com/aist-oceanworks/mudrod/pull/214]
> We should review this and merge into master mudrod branch



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (SDAP-36) Fix failing org.apache.sdap.mudrod.ontology.process.TestLocalOntology

2018-03-06 Thread Lewis John McGibbney (JIRA)

 [ 
https://issues.apache.org/jira/browse/SDAP-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lewis John McGibbney reassigned SDAP-36:


Assignee: Lewis John McGibbney

> Fix failing org.apache.sdap.mudrod.ontology.process.TestLocalOntology
> -
>
> Key: SDAP-36
> URL: https://issues.apache.org/jira/browse/SDAP-36
> Project: Apache Science Data Analytics Platform
>  Issue Type: Task
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Major
>
> The remote file at 
> https://raw.githubusercontent.com/ESIPFed/sweet/master/2.4/sweetAll.owl no 
> longer exists as we changed a bunch of things in SWEET including the 
> availability of any OWL serializations.
> The equivalent file (in Turtle serialization) is located at 
> https://raw.githubusercontent.com/ESIPFed/sweet/master/src/sweetAll.ttl



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (SDAP-37) Update SolrProxy to allow tile search by arbitrary tile metadata

2018-03-06 Thread Nga Chung (JIRA)
Nga Chung created SDAP-37:
-

 Summary: Update SolrProxy to allow tile search by arbitrary tile 
metadata
 Key: SDAP-37
 URL: https://issues.apache.org/jira/browse/SDAP-37
 Project: Apache Science Data Analytics Platform
  Issue Type: New Feature
Reporter: Nga Chung


Besides searching for tiles by spatial and temporal parameters, enable 
searching for tiles by custom tile metadata, such as river id.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)