[GitHub] quintinali commented on issue #35: SDAP-161 MUDROD embedded unit test
quintinali commented on issue #35: SDAP-161 MUDROD embedded unit test URL: https://github.com/apache/incubator-sdap-mudrod/pull/35#issuecomment-444932257 @lewismc I revised the code as you required. Could you please take a look at the unit test cases and please let me know whether they are the test case your team needs? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[ https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711657#comment-16711657 ] ASF GitHub Bot commented on SDAP-161: - quintinali commented on issue #35: SDAP-161 MUDROD embedded unit test URL: https://github.com/apache/incubator-sdap-mudrod/pull/35#issuecomment-444932257 @lewismc I revised the code as you required. Could you please take a look at the unit test cases and please let me know whether they are the test case your team needs? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MUDROD embedded unit test > - > > Key: SDAP-161 > URL: https://issues.apache.org/jira/browse/SDAP-161 > Project: Apache Science Data Analytics Platform > Issue Type: Improvement > Components: mudrod >Affects Versions: 1.0 >Reporter: Yun Li >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538277
##
File path:
core/src/main/java/org/apache/sdap/mudrod/weblog/pre/SessionStatistic.java
##
@@ -229,7 +229,7 @@ public int processSession(ESDriver es, String sessionId)
throws IOException, Int
String[] keywordList = keywords.split(",");
for (String item : items) {
if (!Arrays.asList(keywordList).contains(item)) {
- keywords = keywords + item + ",";
+ keywords = keywords + "," + item + ",";
Review comment:
OK thanks is this included in a test?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538351
##
File path:
core/src/main/java/org/apache/sdap/mudrod/weblog/structure/session/SessionTree.java
##
@@ -205,7 +208,8 @@ public JsonObject treeToJson(SessionNode node) {
RequestUrl requestURL = new RequestUrl();
String viewquery = "";
try {
-String infoStr = requestURL.getSearchInfo(viewnode.getRequest());
+//String infoStr = requestURL.getSearchInfo(viewnode.getRequest());
Review comment:
You need to resolve this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239540151
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
+String clickStreamcsvFile = getTestDataPath() + "/clickStreamMatrix.csv";
+System.out.println(clickStreamcsvFile);
+BufferedReader br = new BufferedReader(new FileReader(clickStreamcsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0.0")) { //
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+System.out.println(map);
+
+Assert.assertE
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test URL: https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541170 ## File path: core/src/test/java/org/apache/sdap/mudrod/main/AbstractElasticsearchIntegrationTest.java ## @@ -0,0 +1,37 @@ +package org.apache.sdap.mudrod.main; + +import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer; +import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer; +import org.elasticsearch.client.Client; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; + +/** + * This is a helper class the starts an embedded elasticsearch server + * for each test. + * + * @author Felix Müller Review comment: ??? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239537668
##
File path: core/src/main/java/org/apache/sdap/mudrod/main/MudrodEngine.java
##
@@ -391,12 +391,12 @@ public static void main(String[] args) {
me.end();
} catch (Exception e) {
HelpFormatter formatter = new HelpFormatter();
- formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", true);
+ formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", options, true);
Review comment:
@quintinali please address this. It should not throw help. It should through
the ```Exception e```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239543206
##
File path:
core/src/test/java/org/apache/sdap/mudrod/weblog/structure/log/GeoIpTest.java
##
@@ -0,0 +1,18 @@
+package org.apache.sdap.mudrod.weblog.structure.log;
+
+import static org.junit.Assert.*;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class GeoIpTest {
+
+ @Test
+ public void testToLocation() {
+ GeoIp ip = new GeoIp();
+ String iptest = "185.10.104.194";
+ Coordinates result = ip.toLocation(iptest);
+ Assert.assertEquals("failed in geoip function!",
"22.283001,114.150002", result.latlon);
+ }
+
+}
Review comment:
core/src/test/resources/Testing_Data_1_3dayLog+Meta+Onto/FTP.201502.w1.gz and
core/src/test/resources/Testing_Data_1_3dayLog+Meta+Onto/WWW.201502.w1.gz
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538523
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
Review comment:
Remove TODO and method if nothing happens here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541079
##
File path:
core/src/test/java/org/apache/sdap/mudrod/driver/PluginConfigurableNode.java
##
@@ -0,0 +1,27 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.driver;
+
+import java.util.Collection;
+
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.node.Node;
+import org.elasticsearch.node.internal.InternalSettingsPreparer;
+import org.elasticsearch.plugins.Plugin;
+
+public class PluginConfigurableNode extends Node {
+ public PluginConfigurableNode(Settings settings, Collection> classpathPlugins) {
+super(InternalSettingsPreparer.prepareEnvironment(settings, null),
classpathPlugins);
Review comment:
What is this code doing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239542355
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/ESDriverTest.java
##
@@ -0,0 +1,312 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.elasticsearch.index.query.MatchAllQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.action.bulk.BulkProcessor;
+import org.elasticsearch.action.update.UpdateRequest;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+public class ESDriverTest extends AbstractElasticsearchIntegrationTest {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ESDriver.class);
+ protected static final String ES_SETTINGS = "elastic_settings.json";
+ protected static final String ES_MAPPINGS = "elastic_mappings.json";
+
+ static ESDriver es = null;
+ static MudrodEngine mudrodEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+mudrodEngine = new MudrodEngine();
+es = new ESDriver(mudrodEngine.loadConfig());
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// es.destroyBulkProcessor();
+// es.close();
+ }
+
+ @Test
+ public void testESDriverProperties() {
+
+Client client = es.getClient();
+assert client != null;
+ }
+
+ @Test
+ public void testCreateBulkProcessor() {
+es.createBulkProcessor();
+BulkProcessor processor = es.getBulkProcessor();
+assert processor != null;
+ }
+
+ @Test
+ public void testDestroyBulkProcessor() {
+es.createBulkProcessor();
+es.destroyBulkProcessor();
+
+BulkProcessor processor = es.getBulkProcessor();
+assert processor == null;
+ }
+
+ @Test
+ public void testPutMapping() {
+
+InputStream settingsStream =
getClass().getClassLoader().getResourceAsStream(ES_SETTINGS);
+InputStream mappingsStream =
getClass().getClassLoader().getResourceAsStream(ES_MAPPINGS);
+
+JSONObject settingsJSON = null;
+JSONObject mappingJSON = null;
+
+Properties props = mudrodEngine.loadConfig();
+try {
+ settingsJSON = new JSONObject(IOUtils.toString(settingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch settings!", e1);
+}
+
+try {
+ mappingJSON = new JSONObject(IOUtils.toString(mappingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch mappings!", e1);
+}
+
+try {
+ if (settingsJSON != null && mappingJSON != null) {
+es.putMapping(props.getProperty(MudrodConstants.ES_INDEX_NAME),
settingsJSON.toString(), mappingJSON.toString());
+ }
+} catch (IOException e) {
+ LOG.error("Error entering Elasticsearch Mappings!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringString() {
+
+String str = "temp";
+try {
+ String res = es.customAnalyzing("mudrod", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedException | ExecutionException e) {
+ LOG.error("Error!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringStringString() {
+String str = "temp";
+
+try {
+ String res = es.customAnalyzing("mudrod", "cody", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedException | ExecutionException e) {
+ LOG.error("Error!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringListOfString() {
+
+List customlist = new ArrayList<>();
+customlist.add("string_a");
+customlist.add("string_b
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239539469
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
Review comment:
Never, ever, ever use path's to your local machine in code. This is a huge
No-go. Remove this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239539163
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,127 @@
+package org.apache.sdap.mudrod.discoveryengine;
+
+import static org.junit.Assert.*;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
Review comment:
The issue is the same here. Change ```testPreprocess_clickStream``` to
```testPreprocessClickStream```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538180
##
File path: core/src/main/java/org/apache/sdap/mudrod/main/MudrodEngine.java
##
@@ -391,12 +391,12 @@ public static void main(String[] args) {
me.end();
} catch (Exception e) {
HelpFormatter formatter = new HelpFormatter();
- formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", true);
+ formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", options, true);
LOG.error("Error whilst parsing command line.", e);
}
}
- private static void loadPathConfig(MudrodEngine me, String dataDir) {
+ public static void loadPathConfig(MudrodEngine me, String dataDir) {
Review comment:
OK thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239539629
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
+String clickStreamcsvFile = getTestDataPath() + "/clickStreamMatrix.csv";
+System.out.println(clickStreamcsvFile);
+BufferedReader br = new BufferedReader(new FileReader(clickStreamcsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
Review comment:
If all of this code is duplicated, then factor it out and provide method
parameters for the input data.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test URL: https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541560 ## File path: core/src/test/java/org/apache/sdap/mudrod/main/AbstractElasticsearchIntegrationTest.java ## @@ -0,0 +1,50 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); you + * may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.sdap.mudrod.main; + +import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer; +import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer; +import org.elasticsearch.client.Client; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; + +/** + * This is a helper class the starts an embedded elasticsearch server for each Review comment: We do not need to start a new server for every test. We need to start one server which runs for the duration of the tests. Between tests, the data within the server should probably be deleted/truncated. This however depends on what tests you are running. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538820
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,127 @@
+package org.apache.sdap.mudrod.discoveryengine;
+
+import static org.junit.Assert.*;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
Review comment:
Additionally, ```testPreprocess_userHistory``` is not how to write Java
methods. Please use camelCase as is shown elsewhere throughout the codebase.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711757#comment-16711757
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541940
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/ESDriverTest.java
##
@@ -0,0 +1,312 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.elasticsearch.index.query.MatchAllQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.action.bulk.BulkProcessor;
+import org.elasticsearch.action.update.UpdateRequest;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+public class ESDriverTest extends AbstractElasticsearchIntegrationTest {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ESDriver.class);
+ protected static final String ES_SETTINGS = "elastic_settings.json";
+ protected static final String ES_MAPPINGS = "elastic_mappings.json";
+
+ static ESDriver es = null;
+ static MudrodEngine mudrodEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+mudrodEngine = new MudrodEngine();
+es = new ESDriver(mudrodEngine.loadConfig());
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// es.destroyBulkProcessor();
+// es.close();
+ }
+
+ @Test
+ public void testESDriverProperties() {
+
+Client client = es.getClient();
+assert client != null;
+ }
+
+ @Test
+ public void testCreateBulkProcessor() {
+es.createBulkProcessor();
+BulkProcessor processor = es.getBulkProcessor();
+assert processor != null;
+ }
+
+ @Test
+ public void testDestroyBulkProcessor() {
+es.createBulkProcessor();
+es.destroyBulkProcessor();
+
+BulkProcessor processor = es.getBulkProcessor();
+assert processor == null;
+ }
+
+ @Test
+ public void testPutMapping() {
+
+InputStream settingsStream =
getClass().getClassLoader().getResourceAsStream(ES_SETTINGS);
+InputStream mappingsStream =
getClass().getClassLoader().getResourceAsStream(ES_MAPPINGS);
+
+JSONObject settingsJSON = null;
+JSONObject mappingJSON = null;
+
+Properties props = mudrodEngine.loadConfig();
+try {
+ settingsJSON = new JSONObject(IOUtils.toString(settingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch settings!", e1);
+}
+
+try {
+ mappingJSON = new JSONObject(IOUtils.toString(mappingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch mappings!", e1);
+}
+
+try {
+ if (settingsJSON != null && mappingJSON != null) {
+es.putMapping(props.getProperty(MudrodConstants.ES_INDEX_NAME),
settingsJSON.toString(), mappingJSON.toString());
+ }
+} catch (IOException e) {
+ LOG.error("Error entering Elasticsearch Mappings!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringString() {
+
+String str = "temp";
+try {
+ String res = es.customAnalyzing("mudrod", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedException | ExecutionException e) {
+ LOG.error("Error!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringStringString() {
+String str = "temp";
+
+try {
+ String res = es.customAnalyzing("mudrod", "cody", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedEx
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711758#comment-16711758
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239542280
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/ESDriverTest.java
##
@@ -0,0 +1,312 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.elasticsearch.index.query.MatchAllQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.action.bulk.BulkProcessor;
+import org.elasticsearch.action.update.UpdateRequest;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+public class ESDriverTest extends AbstractElasticsearchIntegrationTest {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ESDriver.class);
+ protected static final String ES_SETTINGS = "elastic_settings.json";
+ protected static final String ES_MAPPINGS = "elastic_mappings.json";
+
+ static ESDriver es = null;
+ static MudrodEngine mudrodEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+mudrodEngine = new MudrodEngine();
+es = new ESDriver(mudrodEngine.loadConfig());
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// es.destroyBulkProcessor();
+// es.close();
+ }
+
+ @Test
+ public void testESDriverProperties() {
+
+Client client = es.getClient();
+assert client != null;
+ }
+
+ @Test
+ public void testCreateBulkProcessor() {
+es.createBulkProcessor();
+BulkProcessor processor = es.getBulkProcessor();
+assert processor != null;
+ }
+
+ @Test
+ public void testDestroyBulkProcessor() {
+es.createBulkProcessor();
+es.destroyBulkProcessor();
+
+BulkProcessor processor = es.getBulkProcessor();
+assert processor == null;
+ }
+
+ @Test
+ public void testPutMapping() {
+
+InputStream settingsStream =
getClass().getClassLoader().getResourceAsStream(ES_SETTINGS);
+InputStream mappingsStream =
getClass().getClassLoader().getResourceAsStream(ES_MAPPINGS);
+
+JSONObject settingsJSON = null;
+JSONObject mappingJSON = null;
+
+Properties props = mudrodEngine.loadConfig();
+try {
+ settingsJSON = new JSONObject(IOUtils.toString(settingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch settings!", e1);
+}
+
+try {
+ mappingJSON = new JSONObject(IOUtils.toString(mappingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch mappings!", e1);
+}
+
+try {
+ if (settingsJSON != null && mappingJSON != null) {
+es.putMapping(props.getProperty(MudrodConstants.ES_INDEX_NAME),
settingsJSON.toString(), mappingJSON.toString());
+ }
+} catch (IOException e) {
+ LOG.error("Error entering Elasticsearch Mappings!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringString() {
+
+String str = "temp";
+try {
+ String res = es.customAnalyzing("mudrod", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedException | ExecutionException e) {
+ LOG.error("Error!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringStringString() {
+String str = "temp";
+
+try {
+ String res = es.customAnalyzing("mudrod", "cody", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedEx
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711756#comment-16711756
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239540790
##
File path:
core/src/test/java/org/apache/sdap/mudrod/driver/EmbeddedElasticsearchServer.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.driver;
+
+import org.apache.commons.io.FileUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.node.Node;
+import org.elasticsearch.node.NodeValidationException;
+import org.elasticsearch.transport.Netty3Plugin;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+
+/**
+ * embedded elasticsearch server.
+ */
+public class EmbeddedElasticsearchServer {
+
+ private static final String DEFAULT_DATA_DIRECTORY =
"target/elasticsearch-data";
+
+ private Node node;
+ private final String dataDirectory;
+
+ public EmbeddedElasticsearchServer() {
+this(DEFAULT_DATA_DIRECTORY);
+ }
+
+ public EmbeddedElasticsearchServer(String dataDirectory) {
+this.dataDirectory = dataDirectory;
+
+Settings.Builder settingsBuilder = Settings.builder();
+settingsBuilder.put("http.type", "netty3");
+settingsBuilder.put("transport.type", "netty3");
+settingsBuilder.put("cluster.name", "MurdorES").put("http.enabled",
"false").put("path.data", dataDirectory).put("path.home", "/");
+
+Settings settings = settingsBuilder.build();
+Collection plugins = Arrays.asList(Netty3Plugin.class);
+node = null;
+try {
+ node = new PluginConfigurableNode(settings, plugins).start();
+ System.out.println(node.toString());
+} catch (NodeValidationException e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
+}
+
+System.out.println(node.getNodeEnvironment().nodeId());
+ }
+
+ public Client getClient() {
+return node.client();
+ }
+
+ public void shutdown() {
Review comment:
If you are not using the content of this method... remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711748#comment-16711748
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541881
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/ESDriverTest.java
##
@@ -0,0 +1,312 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.elasticsearch.index.query.MatchAllQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.action.bulk.BulkProcessor;
+import org.elasticsearch.action.update.UpdateRequest;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+public class ESDriverTest extends AbstractElasticsearchIntegrationTest {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ESDriver.class);
+ protected static final String ES_SETTINGS = "elastic_settings.json";
+ protected static final String ES_MAPPINGS = "elastic_mappings.json";
+
+ static ESDriver es = null;
+ static MudrodEngine mudrodEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+mudrodEngine = new MudrodEngine();
+es = new ESDriver(mudrodEngine.loadConfig());
+ }
+
+ @AfterClass
Review comment:
Never leave method in your code that are commented out.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711736#comment-16711736
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239537668
##
File path: core/src/main/java/org/apache/sdap/mudrod/main/MudrodEngine.java
##
@@ -391,12 +391,12 @@ public static void main(String[] args) {
me.end();
} catch (Exception e) {
HelpFormatter formatter = new HelpFormatter();
- formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", true);
+ formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", options, true);
Review comment:
@quintinali please address this. It should not throw help. It should through
the ```Exception e```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711738#comment-16711738
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538277
##
File path:
core/src/main/java/org/apache/sdap/mudrod/weblog/pre/SessionStatistic.java
##
@@ -229,7 +229,7 @@ public int processSession(ESDriver es, String sessionId)
throws IOException, Int
String[] keywordList = keywords.split(",");
for (String item : items) {
if (!Arrays.asList(keywordList).contains(item)) {
- keywords = keywords + item + ",";
+ keywords = keywords + "," + item + ",";
Review comment:
OK thanks is this included in a test?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711755#comment-16711755
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239543206
##
File path:
core/src/test/java/org/apache/sdap/mudrod/weblog/structure/log/GeoIpTest.java
##
@@ -0,0 +1,18 @@
+package org.apache.sdap.mudrod.weblog.structure.log;
+
+import static org.junit.Assert.*;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class GeoIpTest {
+
+ @Test
+ public void testToLocation() {
+ GeoIp ip = new GeoIp();
+ String iptest = "185.10.104.194";
+ Coordinates result = ip.toLocation(iptest);
+ Assert.assertEquals("failed in geoip function!",
"22.283001,114.150002", result.latlon);
+ }
+
+}
Review comment:
core/src/test/resources/Testing_Data_1_3dayLog+Meta+Onto/FTP.201502.w1.gz and
core/src/test/resources/Testing_Data_1_3dayLog+Meta+Onto/WWW.201502.w1.gz
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711749#comment-16711749
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239542736
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/TestMudrodEngine.java
##
@@ -23,9 +23,9 @@
import org.junit.Test;
/**
- * Initial test case for {@link org.apache.sdap.mudrod.main.MudrodEngine},
currently no
- * tests are implemented, this is merely in place to get the JaCoCo test
reporting to
- * work.
+ * Initial test case for {@link org.apache.sdap.mudrod.main.MudrodEngine},
+ * currently no tests are implemented, this is merely in place to get the
JaCoCo
Review comment:
If no tests are implemented then this is a TODO and a JIRA issue should be
logged such that someone knows that a test needs to be written.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711745#comment-16711745
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538351
##
File path:
core/src/main/java/org/apache/sdap/mudrod/weblog/structure/session/SessionTree.java
##
@@ -205,7 +208,8 @@ public JsonObject treeToJson(SessionNode node) {
RequestUrl requestURL = new RequestUrl();
String viewquery = "";
try {
-String infoStr = requestURL.getSearchInfo(viewnode.getRequest());
+//String infoStr = requestURL.getSearchInfo(viewnode.getRequest());
Review comment:
You need to resolve this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711741#comment-16711741
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239540151
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
+String clickStreamcsvFile = getTestDataPath() + "/clickStreamMatrix.csv";
+System.out.println(clickStreamcsvFile);
+BufferedReader br = new BufferedReader(new FileReader(clickStreamcsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0.0")) { //
+if
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711739#comment-16711739
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239539469
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
Review comment:
Never, ever, ever use path's to your local machine in code. This is a huge
No-go. Remove this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
>
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711750#comment-16711750
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541715
##
File path:
core/src/test/java/org/apache/sdap/mudrod/main/AbstractElasticsearchIntegrationTest.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+
+/**
+ * This is a helper class the starts an embedded elasticsearch server for each
+ * test.
+ *
+ * @author Felix Müller
+ */
+public abstract class AbstractElasticsearchIntegrationTest {
+
+ private static EmbeddedElasticsearchServer embeddedElasticsearchServer;
+
+ @BeforeClass
+ public static void startEmbeddedElasticsearchServer() {
+embeddedElasticsearchServer = new EmbeddedElasticsearchServer();
+ }
+
+ @AfterClass
+ public static void shutdownEmbeddedElasticsearchServer() {
+// embeddedElasticsearchServer.shutdown();
Review comment:
Why are you not shutting the server down? If you are not doing so then you
need to remove this method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711744#comment-16711744
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239540539
##
File path:
core/src/test/java/org/apache/sdap/mudrod/driver/EmbeddedElasticsearchServer.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.driver;
+
+import org.apache.commons.io.FileUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.node.Node;
+import org.elasticsearch.node.NodeValidationException;
+import org.elasticsearch.transport.Netty3Plugin;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+
+/**
+ * embedded elasticsearch server.
+ */
+public class EmbeddedElasticsearchServer {
+
+ private static final String DEFAULT_DATA_DIRECTORY =
"target/elasticsearch-data";
+
+ private Node node;
+ private final String dataDirectory;
+
+ public EmbeddedElasticsearchServer() {
+this(DEFAULT_DATA_DIRECTORY);
+ }
+
+ public EmbeddedElasticsearchServer(String dataDirectory) {
+this.dataDirectory = dataDirectory;
+
+Settings.Builder settingsBuilder = Settings.builder();
+settingsBuilder.put("http.type", "netty3");
+settingsBuilder.put("transport.type", "netty3");
+settingsBuilder.put("cluster.name", "MurdorES").put("http.enabled",
"false").put("path.data", dataDirectory).put("path.home", "/");
+
+Settings settings = settingsBuilder.build();
+Collection plugins = Arrays.asList(Netty3Plugin.class);
+node = null;
+try {
+ node = new PluginConfigurableNode(settings, plugins).start();
+ System.out.println(node.toString());
+} catch (NodeValidationException e) {
+ // TODO Auto-generated catch block
Review comment:
Remove TODO
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711740#comment-16711740
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538523
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
Review comment:
Remove TODO and method if nothing happens here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711746#comment-16711746
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239539629
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
+String clickStreamcsvFile = getTestDataPath() + "/clickStreamMatrix.csv";
+System.out.println(clickStreamcsvFile);
+BufferedReader br = new BufferedReader(new FileReader(clickStreamcsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
Review comment:
If all of this code is duplicated, then factor it out and provide method
parameters for the input data.
This is an automated message from the Apache Git Service.
To respond to the message, please log on Gi
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711743#comment-16711743
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239542355
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/ESDriverTest.java
##
@@ -0,0 +1,312 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.elasticsearch.index.query.MatchAllQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.action.bulk.BulkProcessor;
+import org.elasticsearch.action.update.UpdateRequest;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+public class ESDriverTest extends AbstractElasticsearchIntegrationTest {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ESDriver.class);
+ protected static final String ES_SETTINGS = "elastic_settings.json";
+ protected static final String ES_MAPPINGS = "elastic_mappings.json";
+
+ static ESDriver es = null;
+ static MudrodEngine mudrodEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+mudrodEngine = new MudrodEngine();
+es = new ESDriver(mudrodEngine.loadConfig());
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// es.destroyBulkProcessor();
+// es.close();
+ }
+
+ @Test
+ public void testESDriverProperties() {
+
+Client client = es.getClient();
+assert client != null;
+ }
+
+ @Test
+ public void testCreateBulkProcessor() {
+es.createBulkProcessor();
+BulkProcessor processor = es.getBulkProcessor();
+assert processor != null;
+ }
+
+ @Test
+ public void testDestroyBulkProcessor() {
+es.createBulkProcessor();
+es.destroyBulkProcessor();
+
+BulkProcessor processor = es.getBulkProcessor();
+assert processor == null;
+ }
+
+ @Test
+ public void testPutMapping() {
+
+InputStream settingsStream =
getClass().getClassLoader().getResourceAsStream(ES_SETTINGS);
+InputStream mappingsStream =
getClass().getClassLoader().getResourceAsStream(ES_MAPPINGS);
+
+JSONObject settingsJSON = null;
+JSONObject mappingJSON = null;
+
+Properties props = mudrodEngine.loadConfig();
+try {
+ settingsJSON = new JSONObject(IOUtils.toString(settingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch settings!", e1);
+}
+
+try {
+ mappingJSON = new JSONObject(IOUtils.toString(mappingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch mappings!", e1);
+}
+
+try {
+ if (settingsJSON != null && mappingJSON != null) {
+es.putMapping(props.getProperty(MudrodConstants.ES_INDEX_NAME),
settingsJSON.toString(), mappingJSON.toString());
+ }
+} catch (IOException e) {
+ LOG.error("Error entering Elasticsearch Mappings!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringString() {
+
+String str = "temp";
+try {
+ String res = es.customAnalyzing("mudrod", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedException | ExecutionException e) {
+ LOG.error("Error!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringStringString() {
+String str = "temp";
+
+try {
+ String res = es.customAnalyzing("mudrod", "cody", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedEx
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711751#comment-16711751
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239539163
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,127 @@
+package org.apache.sdap.mudrod.discoveryengine;
+
+import static org.junit.Assert.*;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
Review comment:
The issue is the same here. Change ```testPreprocess_clickStream``` to
```testPreprocessClickStream```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[ https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711754#comment-16711754 ] ASF GitHub Bot commented on SDAP-161: - lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test URL: https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541560 ## File path: core/src/test/java/org/apache/sdap/mudrod/main/AbstractElasticsearchIntegrationTest.java ## @@ -0,0 +1,50 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); you + * may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.sdap.mudrod.main; + +import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer; +import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer; +import org.elasticsearch.client.Client; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; + +/** + * This is a helper class the starts an embedded elasticsearch server for each Review comment: We do not need to start a new server for every test. We need to start one server which runs for the duration of the tests. Between tests, the data within the server should probably be deleted/truncated. This however depends on what tests you are running. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MUDROD embedded unit test > - > > Key: SDAP-161 > URL: https://issues.apache.org/jira/browse/SDAP-161 > Project: Apache Science Data Analytics Platform > Issue Type: Improvement > Components: mudrod >Affects Versions: 1.0 >Reporter: Yun Li >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[ https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711759#comment-16711759 ] ASF GitHub Bot commented on SDAP-161: - lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test URL: https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541170 ## File path: core/src/test/java/org/apache/sdap/mudrod/main/AbstractElasticsearchIntegrationTest.java ## @@ -0,0 +1,37 @@ +package org.apache.sdap.mudrod.main; + +import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer; +import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer; +import org.elasticsearch.client.Client; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; + +/** + * This is a helper class the starts an embedded elasticsearch server + * for each test. + * + * @author Felix Müller Review comment: ??? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MUDROD embedded unit test > - > > Key: SDAP-161 > URL: https://issues.apache.org/jira/browse/SDAP-161 > Project: Apache Science Data Analytics Platform > Issue Type: Improvement > Components: mudrod >Affects Versions: 1.0 >Reporter: Yun Li >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711747#comment-16711747
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538581
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,127 @@
+package org.apache.sdap.mudrod.discoveryengine;
+
+import static org.junit.Assert.*;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
Review comment:
You need to address this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711742#comment-16711742
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538820
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,127 @@
+package org.apache.sdap.mudrod.discoveryengine;
+
+import static org.junit.Assert.*;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
Review comment:
Additionally, ```testPreprocess_userHistory``` is not how to write Java
methods. Please use camelCase as is shown elsewhere throughout the codebase.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541715
##
File path:
core/src/test/java/org/apache/sdap/mudrod/main/AbstractElasticsearchIntegrationTest.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+
+/**
+ * This is a helper class the starts an embedded elasticsearch server for each
+ * test.
+ *
+ * @author Felix Müller
+ */
+public abstract class AbstractElasticsearchIntegrationTest {
+
+ private static EmbeddedElasticsearchServer embeddedElasticsearchServer;
+
+ @BeforeClass
+ public static void startEmbeddedElasticsearchServer() {
+embeddedElasticsearchServer = new EmbeddedElasticsearchServer();
+ }
+
+ @AfterClass
+ public static void shutdownEmbeddedElasticsearchServer() {
+// embeddedElasticsearchServer.shutdown();
Review comment:
Why are you not shutting the server down? If you are not doing so then you
need to remove this method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541881
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/ESDriverTest.java
##
@@ -0,0 +1,312 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.elasticsearch.index.query.MatchAllQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.action.bulk.BulkProcessor;
+import org.elasticsearch.action.update.UpdateRequest;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+public class ESDriverTest extends AbstractElasticsearchIntegrationTest {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ESDriver.class);
+ protected static final String ES_SETTINGS = "elastic_settings.json";
+ protected static final String ES_MAPPINGS = "elastic_mappings.json";
+
+ static ESDriver es = null;
+ static MudrodEngine mudrodEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+mudrodEngine = new MudrodEngine();
+es = new ESDriver(mudrodEngine.loadConfig());
+ }
+
+ @AfterClass
Review comment:
Never leave method in your code that are commented out.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538581
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,127 @@
+package org.apache.sdap.mudrod.discoveryengine;
+
+import static org.junit.Assert.*;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
Review comment:
You need to address this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239540539
##
File path:
core/src/test/java/org/apache/sdap/mudrod/driver/EmbeddedElasticsearchServer.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.driver;
+
+import org.apache.commons.io.FileUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.node.Node;
+import org.elasticsearch.node.NodeValidationException;
+import org.elasticsearch.transport.Netty3Plugin;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+
+/**
+ * embedded elasticsearch server.
+ */
+public class EmbeddedElasticsearchServer {
+
+ private static final String DEFAULT_DATA_DIRECTORY =
"target/elasticsearch-data";
+
+ private Node node;
+ private final String dataDirectory;
+
+ public EmbeddedElasticsearchServer() {
+this(DEFAULT_DATA_DIRECTORY);
+ }
+
+ public EmbeddedElasticsearchServer(String dataDirectory) {
+this.dataDirectory = dataDirectory;
+
+Settings.Builder settingsBuilder = Settings.builder();
+settingsBuilder.put("http.type", "netty3");
+settingsBuilder.put("transport.type", "netty3");
+settingsBuilder.put("cluster.name", "MurdorES").put("http.enabled",
"false").put("path.data", dataDirectory).put("path.home", "/");
+
+Settings settings = settingsBuilder.build();
+Collection plugins = Arrays.asList(Netty3Plugin.class);
+node = null;
+try {
+ node = new PluginConfigurableNode(settings, plugins).start();
+ System.out.println(node.toString());
+} catch (NodeValidationException e) {
+ // TODO Auto-generated catch block
Review comment:
Remove TODO
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541940
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/ESDriverTest.java
##
@@ -0,0 +1,312 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.elasticsearch.index.query.MatchAllQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.action.bulk.BulkProcessor;
+import org.elasticsearch.action.update.UpdateRequest;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+public class ESDriverTest extends AbstractElasticsearchIntegrationTest {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ESDriver.class);
+ protected static final String ES_SETTINGS = "elastic_settings.json";
+ protected static final String ES_MAPPINGS = "elastic_mappings.json";
+
+ static ESDriver es = null;
+ static MudrodEngine mudrodEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+mudrodEngine = new MudrodEngine();
+es = new ESDriver(mudrodEngine.loadConfig());
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// es.destroyBulkProcessor();
+// es.close();
+ }
+
+ @Test
+ public void testESDriverProperties() {
+
+Client client = es.getClient();
+assert client != null;
+ }
+
+ @Test
+ public void testCreateBulkProcessor() {
+es.createBulkProcessor();
+BulkProcessor processor = es.getBulkProcessor();
+assert processor != null;
+ }
+
+ @Test
+ public void testDestroyBulkProcessor() {
+es.createBulkProcessor();
+es.destroyBulkProcessor();
+
+BulkProcessor processor = es.getBulkProcessor();
+assert processor == null;
+ }
+
+ @Test
+ public void testPutMapping() {
+
+InputStream settingsStream =
getClass().getClassLoader().getResourceAsStream(ES_SETTINGS);
+InputStream mappingsStream =
getClass().getClassLoader().getResourceAsStream(ES_MAPPINGS);
+
+JSONObject settingsJSON = null;
+JSONObject mappingJSON = null;
+
+Properties props = mudrodEngine.loadConfig();
+try {
+ settingsJSON = new JSONObject(IOUtils.toString(settingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch settings!", e1);
+}
+
+try {
+ mappingJSON = new JSONObject(IOUtils.toString(mappingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch mappings!", e1);
+}
+
+try {
+ if (settingsJSON != null && mappingJSON != null) {
+es.putMapping(props.getProperty(MudrodConstants.ES_INDEX_NAME),
settingsJSON.toString(), mappingJSON.toString());
+ }
+} catch (IOException e) {
+ LOG.error("Error entering Elasticsearch Mappings!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringString() {
+
+String str = "temp";
+try {
+ String res = es.customAnalyzing("mudrod", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedException | ExecutionException e) {
+ LOG.error("Error!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringStringString() {
+String str = "temp";
+
+try {
+ String res = es.customAnalyzing("mudrod", "cody", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedException | ExecutionException e) {
+ LOG.error("Error!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringListOfString() {
+
+List customlist = new ArrayList<>();
+customlist.add("string_a");
+customlist.add("string_b
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239542280
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/ESDriverTest.java
##
@@ -0,0 +1,312 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.main;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.EmbeddedElasticsearchServer;
+import org.elasticsearch.client.Client;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.elasticsearch.index.query.MatchAllQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.action.bulk.BulkProcessor;
+import org.elasticsearch.action.update.UpdateRequest;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+public class ESDriverTest extends AbstractElasticsearchIntegrationTest {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ESDriver.class);
+ protected static final String ES_SETTINGS = "elastic_settings.json";
+ protected static final String ES_MAPPINGS = "elastic_mappings.json";
+
+ static ESDriver es = null;
+ static MudrodEngine mudrodEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+mudrodEngine = new MudrodEngine();
+es = new ESDriver(mudrodEngine.loadConfig());
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// es.destroyBulkProcessor();
+// es.close();
+ }
+
+ @Test
+ public void testESDriverProperties() {
+
+Client client = es.getClient();
+assert client != null;
+ }
+
+ @Test
+ public void testCreateBulkProcessor() {
+es.createBulkProcessor();
+BulkProcessor processor = es.getBulkProcessor();
+assert processor != null;
+ }
+
+ @Test
+ public void testDestroyBulkProcessor() {
+es.createBulkProcessor();
+es.destroyBulkProcessor();
+
+BulkProcessor processor = es.getBulkProcessor();
+assert processor == null;
+ }
+
+ @Test
+ public void testPutMapping() {
+
+InputStream settingsStream =
getClass().getClassLoader().getResourceAsStream(ES_SETTINGS);
+InputStream mappingsStream =
getClass().getClassLoader().getResourceAsStream(ES_MAPPINGS);
+
+JSONObject settingsJSON = null;
+JSONObject mappingJSON = null;
+
+Properties props = mudrodEngine.loadConfig();
+try {
+ settingsJSON = new JSONObject(IOUtils.toString(settingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch settings!", e1);
+}
+
+try {
+ mappingJSON = new JSONObject(IOUtils.toString(mappingsStream));
+} catch (JSONException | IOException e1) {
+ LOG.error("Error reading Elasticsearch mappings!", e1);
+}
+
+try {
+ if (settingsJSON != null && mappingJSON != null) {
+es.putMapping(props.getProperty(MudrodConstants.ES_INDEX_NAME),
settingsJSON.toString(), mappingJSON.toString());
+ }
+} catch (IOException e) {
+ LOG.error("Error entering Elasticsearch Mappings!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringString() {
+
+String str = "temp";
+try {
+ String res = es.customAnalyzing("mudrod", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedException | ExecutionException e) {
+ LOG.error("Error!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringStringString() {
+String str = "temp";
+
+try {
+ String res = es.customAnalyzing("mudrod", "cody", str);
+ assert res != "";
+ assert res != null;
+} catch (InterruptedException | ExecutionException e) {
+ LOG.error("Error!", e);
+}
+ }
+
+ @Test
+ public void testCustomAnalyzingStringListOfString() {
+
+List customlist = new ArrayList<>();
+customlist.add("string_a");
+customlist.add("string_b
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239539235
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
Review comment:
Remove comments or else address TODO
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239540790
##
File path:
core/src/test/java/org/apache/sdap/mudrod/driver/EmbeddedElasticsearchServer.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.driver;
+
+import org.apache.commons.io.FileUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.node.Node;
+import org.elasticsearch.node.NodeValidationException;
+import org.elasticsearch.transport.Netty3Plugin;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+
+/**
+ * embedded elasticsearch server.
+ */
+public class EmbeddedElasticsearchServer {
+
+ private static final String DEFAULT_DATA_DIRECTORY =
"target/elasticsearch-data";
+
+ private Node node;
+ private final String dataDirectory;
+
+ public EmbeddedElasticsearchServer() {
+this(DEFAULT_DATA_DIRECTORY);
+ }
+
+ public EmbeddedElasticsearchServer(String dataDirectory) {
+this.dataDirectory = dataDirectory;
+
+Settings.Builder settingsBuilder = Settings.builder();
+settingsBuilder.put("http.type", "netty3");
+settingsBuilder.put("transport.type", "netty3");
+settingsBuilder.put("cluster.name", "MurdorES").put("http.enabled",
"false").put("path.data", dataDirectory).put("path.home", "/");
+
+Settings settings = settingsBuilder.build();
+Collection plugins = Arrays.asList(Netty3Plugin.class);
+node = null;
+try {
+ node = new PluginConfigurableNode(settings, plugins).start();
+ System.out.println(node.toString());
+} catch (NodeValidationException e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
+}
+
+System.out.println(node.getNodeEnvironment().nodeId());
+ }
+
+ public Client getClient() {
+return node.client();
+ }
+
+ public void shutdown() {
Review comment:
If you are not using the content of this method... remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239542736
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/TestMudrodEngine.java
##
@@ -23,9 +23,9 @@
import org.junit.Test;
/**
- * Initial test case for {@link org.apache.sdap.mudrod.main.MudrodEngine},
currently no
- * tests are implemented, this is merely in place to get the JaCoCo test
reporting to
- * work.
+ * Initial test case for {@link org.apache.sdap.mudrod.main.MudrodEngine},
+ * currently no tests are implemented, this is merely in place to get the
JaCoCo
Review comment:
If no tests are implemented then this is a TODO and a JIRA issue should be
logged such that someone knows that a test needs to be written.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711737#comment-16711737
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239538180
##
File path: core/src/main/java/org/apache/sdap/mudrod/main/MudrodEngine.java
##
@@ -391,12 +391,12 @@ public static void main(String[] args) {
me.end();
} catch (Exception e) {
HelpFormatter formatter = new HelpFormatter();
- formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", true);
+ formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", options, true);
LOG.error("Error whilst parsing command line.", e);
}
}
- private static void loadPathConfig(MudrodEngine me, String dataDir) {
+ public static void loadPathConfig(MudrodEngine me, String dataDir) {
Review comment:
OK thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711753#comment-16711753
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239539235
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
Review comment:
Remove comments or else address TODO
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian
[jira] [Commented] (SDAP-161) MUDROD embedded unit test
[
https://issues.apache.org/jira/browse/SDAP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711752#comment-16711752
]
ASF GitHub Bot commented on SDAP-161:
-
lewismc commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239541079
##
File path:
core/src/test/java/org/apache/sdap/mudrod/driver/PluginConfigurableNode.java
##
@@ -0,0 +1,27 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.driver;
+
+import java.util.Collection;
+
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.node.Node;
+import org.elasticsearch.node.internal.InternalSettingsPreparer;
+import org.elasticsearch.plugins.Plugin;
+
+public class PluginConfigurableNode extends Node {
+ public PluginConfigurableNode(Settings settings, Collection> classpathPlugins) {
+super(InternalSettingsPreparer.prepareEnvironment(settings, null),
classpathPlugins);
Review comment:
What is this code doing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MUDROD embedded unit test
> -
>
> Key: SDAP-161
> URL: https://issues.apache.org/jira/browse/SDAP-161
> Project: Apache Science Data Analytics Platform
> Issue Type: Improvement
> Components: mudrod
>Affects Versions: 1.0
>Reporter: Yun Li
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
