[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r240271668
##
File path:
core/src/test/java/org/apache/sdap/mudrod/weblog/structure/log/GeoIpTest.java
##
@@ -0,0 +1,18 @@
+package org.apache.sdap.mudrod.weblog.structure.log;
+
+import static org.junit.Assert.*;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class GeoIpTest {
+
+ @Test
+ public void testToLocation() {
+ GeoIp ip = new GeoIp();
+ String iptest = "185.10.104.194";
+ Coordinates result = ip.toLocation(iptest);
+ Assert.assertEquals("failed in geoip function!",
"22.283001,114.150002", result.latlon);
+ }
+
+}
Review comment:
I remembered that @fgreg has added a function of decompressing zip log file
for MUDROD. And the test code can parse the two gz. file. If you need, I can
depress them for you. Is it necessary?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239940986
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
+String clickStreamcsvFile = getTestDataPath() + "/clickStreamMatrix.csv";
+System.out.println(clickStreamcsvFile);
+BufferedReader br = new BufferedReader(new FileReader(clickStreamcsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
Review comment:
I removed duplicated code
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239940404
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
Review comment:
Solved issue mentioned above
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239940478
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
Review comment:
I deleted the useless code
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239940191
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
Review comment:
Resolve most issued metioned above except the first one. In my computer, the
old code (
formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", true)) comes with an error “The
method printHelp(String, Options) in the type HelpFormatter is not applicable
for the arguments (String, boolean)”
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239939252
##
File path:
core/src/main/java/org/apache/sdap/mudrod/weblog/pre/SessionStatistic.java
##
@@ -229,7 +229,7 @@ public int processSession(ESDriver es, String sessionId)
throws IOException, Int
String[] keywordList = keywords.split(",");
for (String item : items) {
if (!Arrays.asList(keywordList).contains(item)) {
- keywords = keywords + item + ",";
+ keywords = keywords + "," + item + ",";
Review comment:
Yes
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239938525
##
File path: core/src/test/java/org/apache/sdap/mudrod/main/TestMudrodEngine.java
##
@@ -23,9 +23,9 @@
import org.junit.Test;
/**
- * Initial test case for {@link org.apache.sdap.mudrod.main.MudrodEngine},
currently no
- * tests are implemented, this is merely in place to get the JaCoCo test
reporting to
- * work.
+ * Initial test case for {@link org.apache.sdap.mudrod.main.MudrodEngine},
+ * currently no tests are implemented, this is merely in place to get the
JaCoCo
Review comment:
I delete the test class for MudrodEngine and will add it in the future
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239933865
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
+String clickStreamcsvFile = getTestDataPath() + "/clickStreamMatrix.csv";
+System.out.println(clickStreamcsvFile);
+BufferedReader br = new BufferedReader(new FileReader(clickStreamcsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0.0")) { //
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+System.out.println(map);
+
+
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r239931381
##
File path:
core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sdap.mudrod.discoveryengine;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.sdap.mudrod.driver.ESDriver;
+import org.apache.sdap.mudrod.driver.SparkDriver;
+import org.apache.sdap.mudrod.main.AbstractElasticsearchIntegrationTest;
+import org.apache.sdap.mudrod.main.MudrodConstants;
+import org.apache.sdap.mudrod.main.MudrodEngine;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class WeblogDiscoveryEngineTest extends
AbstractElasticsearchIntegrationTest {
+
+ private static WeblogDiscoveryEngine weblogEngine = null;
+
+ @BeforeClass
+ public static void setUp() {
+MudrodEngine mudrodEngine = new MudrodEngine();
+Properties props = mudrodEngine.loadConfig();
+ESDriver es = new ESDriver(props);
+SparkDriver spark = new SparkDriver(props);
+String dataDir = getTestDataPath();
+System.out.println(dataDir);
+props.setProperty(MudrodConstants.DATA_DIR, dataDir);
+MudrodEngine.loadPathConfig(mudrodEngine, dataDir);
+weblogEngine = new WeblogDiscoveryEngine(props, es, spark);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+// TODO
+ }
+
+ private static String getTestDataPath() {
+File resourcesDirectory = new File("src/test/resources/");
+String resourcedir = "/Testing_Data_1_3dayLog+Meta+Onto/";
+String dataDir = resourcesDirectory.getAbsolutePath() + resourcedir;
+return dataDir;
+ }
+
+ @Test
+ public void testPreprocess() throws IOException {
+
+weblogEngine.preprocess();
+testPreprocess_userHistory();
+testPreprocess_clickStream();
+ }
+
+ private void testPreprocess_userHistory() throws IOException {
+// compare user history data
+String userHistorycsvFile = getTestDataPath() + "/userHistoryMatrix.csv";
+BufferedReader br = new BufferedReader(new FileReader(userHistorycsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0")) {
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+
+Assert.assertEquals("failed in history data result!", "195.219.98.7",
String.join(",", map.get("sea surface topography")));
+ }
+
+ private void testPreprocess_clickStream() throws IOException {
+// TODO compare clickStream data
+// String clickStreamcsvFile =
+//
"C:/Users/admin/Documents/GitHub/incubator-sdap-mudrod/core/clickStreamMatrix.csv";
+String clickStreamcsvFile = getTestDataPath() + "/clickStreamMatrix.csv";
+System.out.println(clickStreamcsvFile);
+BufferedReader br = new BufferedReader(new FileReader(clickStreamcsvFile));
+String line = null;
+HashMap> map = new HashMap<>();
+
+int i = 0;
+List header = new LinkedList<>();
+while ((line = br.readLine()) != null) {
+ if (i == 0) {
+String str[] = line.split(",");
+for (String s : str) {
+ header.add(s);
+}
+ } else {
+String str[] = line.split(",");
+for (int j = 1; j < str.length; j++) {
+ if (!str[j].equals("0.0")) { //
+if (!map.containsKey(str[0])) {
+ map.put(str[0], new ArrayList<>());
+}
+map.get(str[0]).add(header.get(j));
+ }
+}
+ }
+ i += 1;
+}
+System.out.println(map);
+
+
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r238318734
##
File path:
core/src/test/java/org/apache/sdap/mudrod/weblog/structure/log/GeoIpTest.java
##
@@ -0,0 +1,18 @@
+package org.apache.sdap.mudrod.weblog.structure.log;
+
+import static org.junit.Assert.*;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class GeoIpTest {
+
+ @Test
+ public void testToLocation() {
+ GeoIp ip = new GeoIp();
+ String iptest = "185.10.104.194";
+ Coordinates result = ip.toLocation(iptest);
+ Assert.assertEquals("failed in geoip function!",
"22.283001,114.150002", result.latlon);
+ }
+
+}
Review comment:
I cannot understand which resource do you mean should be decompressed?
There is not a specific reason for choosing logs from 2015.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r238313448
##
File path: core/src/main/java/org/apache/sdap/mudrod/main/MudrodEngine.java
##
@@ -391,12 +391,12 @@ public static void main(String[] args) {
me.end();
} catch (Exception e) {
HelpFormatter formatter = new HelpFormatter();
- formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", true);
+ formatter.printHelp("MudrodEngine: 'dataDir' argument is mandatory. " +
"User must also provide an ingest method.", options, true);
LOG.error("Error whilst parsing command line.", e);
}
}
- private static void loadPathConfig(MudrodEngine me, String dataDir) {
+ public static void loadPathConfig(MudrodEngine me, String dataDir) {
Review comment:
loadPathConfig is invoked outside MudrodEngine in the test code.
https://github.com/quintinali/incubator-sdap-mudrod/blob/b3ef470a39f8df68ddf80805c6948f870feeb2ce/core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java#L31
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded unit test
quintinali commented on a change in pull request #35: SDAP-161 MUDROD embedded
unit test
URL:
https://github.com/apache/incubator-sdap-mudrod/pull/35#discussion_r238314526
##
File path:
core/src/main/java/org/apache/sdap/mudrod/weblog/pre/SessionStatistic.java
##
@@ -229,7 +229,7 @@ public int processSession(ESDriver es, String sessionId)
throws IOException, Int
String[] keywordList = keywords.split(",");
for (String item : items) {
if (!Arrays.asList(keywordList).contains(item)) {
- keywords = keywords + item + ",";
+ keywords = keywords + "," + item + ",";
Review comment:
https://github.com/quintinali/incubator-sdap-mudrod/blob/b3ef470a39f8df68ddf80805c6948f870feeb2ce/core/src/test/java/org/apache/sdap/mudrod/discoveryengine/WeblogDiscoveryEngineTest.java#L61
In the test case, I found the result generated from the old code is not the
same as expected thus I changed the code
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
