[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r315756318
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,271 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+import org.apache.gora.benchmark.generated.User;
+
+/**
+ * The Class GoraBenchmarkClient
+ *
+ * @author sc306 This class extends the Yahoo! Cloud Service Benchmark
benchmark
+ * {@link #com.yahoo.ycsb.DB DB} class to provide functionality for
+ * {@link #insert(String, String, HashMap) insert},
+ * {@link #read(String, String, Set, HashMap) read},
+ * {@link #scan(String, String, int, Set, Vector) scan} and
+ * {@link #update(String, String, HashMap) update} methods as per
Apache
+ * Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+ private static final String FIELDS[] = User._ALL_FIELDS;
+ private static volatile boolean executed;
+ public static int fieldCount;
+ /** This is only for set to array conversion in {@link read()} method */
+ private String[] DUMMY_ARRAY = new String[0];
+ DataStore dataStore;
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+ User user = new User();
+ private Properties prop;
+
+ public GoraBenchmarkClient() {
+ }
+
+ /***
+ * Initialisation method. This method is called once for each database
+ * instance.
+ */
+ public void init() throws DBException {
+try {
+ // Get YCSB properties
+ prop = getProperties();
+ fieldCount = Integer
+ .parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+ String keyClass = prop.getProperty("key.class", "java.lang.String");
+ String persistentClass = prop.getProperty("persistent.class",
"org.apache.gora.benchmark.generated.User");
+ Properties p = DataStoreFactory.createProps();
+ dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass, p,
new Configuration());
Review comment:
I have updated the code accordingly after our chat.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r315750627
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.benchmark.generated.User;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
+
+ private static final String TABLE = "users";
+ private GoraBenchmarkClient client;
+ private static HashMap DATA_TO_INSERT;
+ private static HashMap DATA_TO_UPDATE;
+ private static final int NUMBER_OF_FIELDS = 10;
+ private GoraBenchmarkUtils bmutils = new GoraBenchmarkUtils();
+
+ /**
+ * Sets the up.
+ *
+ * Setup is executed before each test. Use @BeforeClass if you want to
execute
+ * a code block just once.
+ *
+ * @throws Exception
+ * the exceptionfiles are auto-generated. I have the code to add
the license file accordingly
+ */
+ @Before
+ public void setUp() throws Exception {
+DATA_TO_INSERT = new HashMap<>();
+DATA_TO_UPDATE = new HashMap<>();
+for (int i = 0; i < NUMBER_OF_FIELDS; i++) {
+ DATA_TO_INSERT.put("field" + i, new StringByteIterator("value" + i));
+ DATA_TO_UPDATE.put("field" + i, new StringByteIterator("updated" + i));
+}
+Properties p = new Properties();
+p.setProperty("key.class", "java.lang.String");
+p.setProperty("persistent.class",
"org.apache.gora.benchmark.generated.User");
+p.setProperty(CoreWorkload.FIELD_COUNT_PROPERTY, NUMBER_OF_FIELDS + "");
+client = new GoraBenchmarkClient();
+client.setProperties(p);
+client.init();
+ }
+
+ /**
+ * Clean up.
+ *
+ * @throws Exception
+ * the exception
+ */
+ @After
+ public void cleanUp() throws Exception {
+if (client != null)
+ client.cleanup();
+client = null;
+ }
+
+ /**
+ * Read record.
+ *
+ * @param key
+ * the key
+ * @return the user
+ * @throws GoraException
+ * the gora exception
+ */
+ private User readRecord(String key) throws GoraException {
+User u = client.dataStore.get(key);
+return u;
+ }
+
+ /**
+ * Test client initialisation.
+ */
+ @Test
+ public void testClientInitialisation() {
+assertNotNull(client.dataStore);
+ }
+
+ /**files are auto-generated. I have the code to add the license file
accordingly
+ * Test insert.
+ *
+ * @throws GoraException
+ * the gora exception
+ */
+ @Test
+ public void testInsert() throws GoraException {
+Status result1 = client.insert(TABLE, "key1", DATA_TO_INSERT);
+Status result2 = client.insert(TABLE, "key2", DATA_TO_INSERT);
+Status result3 = client.insert(TABLE, "key3", DATA_TO_INSERT);
+assertEquals(Status.OK, result1);
+assertEquals(Status.OK, result2);
+assertEquals(Status.OK, result3);
+ }
+
+ /**
+ * Test read.
+ */
+ @Test
+ public void testRead() {
+HashMap results = new HashMap<>();
+Set fields = new HashSet<>();// this could be null as well
+// fields.add("field0");
+Status result = client.read(TABLE, "key1", fields, results);
+assertEquals(Status.OK, result);
+assertEquals(DATA_TO_INSERT.size(), results.size());
+assertEquals(DATA_TO_INSERT.get("field0").toString(),
results.get("field0").toString());
+assertEquals(DATA_TO_INSERT.get("field0").toString(), "value0");
+ }
+
+ /**
+ * Test scan.
+ */
+ @Test
+ public void testScan() {
+Vector>
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r315736122
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,428 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Properties;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Comment;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.Node;
+
+import com.yahoo.ycsb.ByteIterator;
+
+/**
+ * The Class GoraBenchmarkUtils has some utilities that dynamically generate
files needed to for gora.
+ * It generate the following files.
+ * a. Database Mapping File
+ * b. Avro Files
+ * c. Data Beans
+ */
+public class GoraBenchmarkUtils {
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE =
"org.apache.gora.benchmark.generated";
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+ private static final Logger LOG =
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r315147828 ## File path: pom.xml ## @@ -826,10 +827,21 @@ 2.2.22 0.2.0 - + + +1.4.2 + + +2.0.0 + 4.10 1.4.2 + + +0.15.0 Review comment: @lewismc and @kamaci, I have now updated this to 0.17.0-SNAPSHOT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r315136504 ## File path: pom.xml ## @@ -826,10 +827,21 @@ 2.2.22 0.2.0 - + + +1.4.2 + + +2.0.0 + 4.10 1.4.2 + + +0.15.0 Review comment: @kamaci, thanks, I will update this accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r315113927 ## File path: pom.xml ## @@ -826,10 +827,21 @@ 2.2.22 0.2.0 - + + +1.4.2 + + +2.0.0 + 4.10 1.4.2 + + +0.15.0 Review comment: All other comments are now addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r315113778 ## File path: pom.xml ## @@ -826,10 +827,21 @@ 2.2.22 0.2.0 - + + +1.4.2 + + +2.0.0 + 4.10 1.4.2 + + +0.15.0 Review comment: > This should be the newest jar version until we get the jar published to Maven Central. The user is required to download this so make it the newest. @lewismc , would you please mind clarifying this for me. From https://github.com/brianfrankcooper/YCSB/releases/ it seems 0.15.0 is the latest stable release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r315010693
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.benchmark.generated.User;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
+
+ private static final String TABLE = "users";
+ private GoraBenchmarkClient client;
+ private static HashMap DATA_TO_INSERT;
+ private static HashMap DATA_TO_UPDATE;
+ private static final int NUMBER_OF_FIELDS = 10;
+ private GoraBenchmarkUtils bmutils = new GoraBenchmarkUtils();
+
+ /**
+ * Sets the up.
+ *
+ * Setup is executed before each test. Use @BeforeClass if you want to
execute
+ * a code block just once.
+ *
+ * @throws Exception
+ * the exceptionfiles are auto-generated. I have the code to add
the license file accordingly
+ */
+ @Before
+ public void setUp() throws Exception {
+DATA_TO_INSERT = new HashMap<>();
+DATA_TO_UPDATE = new HashMap<>();
+for (int i = 0; i < NUMBER_OF_FIELDS; i++) {
+ DATA_TO_INSERT.put("field" + i, new StringByteIterator("value" + i));
+ DATA_TO_UPDATE.put("field" + i, new StringByteIterator("updated" + i));
+}
+Properties p = new Properties();
+p.setProperty("key.class", "java.lang.String");
+p.setProperty("persistent.class",
"org.apache.gora.benchmark.generated.User");
+p.setProperty(CoreWorkload.FIELD_COUNT_PROPERTY, NUMBER_OF_FIELDS + "");
+client = new GoraBenchmarkClient();
+client.setProperties(p);
+client.init();
+ }
+
+ /**
+ * Clean up.
+ *
+ * @throws Exception
+ * the exception
+ */
+ @After
+ public void cleanUp() throws Exception {
+if (client != null)
+ client.cleanup();
+client = null;
+ }
+
+ /**
+ * Read record.
+ *
+ * @param key
+ * the key
+ * @return the user
+ * @throws GoraException
+ * the gora exception
+ */
+ private User readRecord(String key) throws GoraException {
+User u = client.dataStore.get(key);
+return u;
+ }
+
+ /**
+ * Test client initialisation.
+ */
+ @Test
+ public void testClientInitialisation() {
+assertNotNull(client.dataStore);
+ }
+
+ /**files are auto-generated. I have the code to add the license file
accordingly
+ * Test insert.
+ *
+ * @throws GoraException
+ * the gora exception
+ */
+ @Test
+ public void testInsert() throws GoraException {
+Status result1 = client.insert(TABLE, "key1", DATA_TO_INSERT);
+Status result2 = client.insert(TABLE, "key2", DATA_TO_INSERT);
+Status result3 = client.insert(TABLE, "key3", DATA_TO_INSERT);
+assertEquals(Status.OK, result1);
+assertEquals(Status.OK, result2);
+assertEquals(Status.OK, result3);
+ }
+
+ /**
+ * Test read.
+ */
+ @Test
+ public void testRead() {
+HashMap results = new HashMap<>();
+Set fields = new HashSet<>();// this could be null as well
+// fields.add("field0");
+Status result = client.read(TABLE, "key1", fields, results);
+assertEquals(Status.OK, result);
+assertEquals(DATA_TO_INSERT.size(), results.size());
+assertEquals(DATA_TO_INSERT.get("field0").toString(),
results.get("field0").toString());
+assertEquals(DATA_TO_INSERT.get("field0").toString(), "value0");
+ }
+
+ /**
+ * Test scan.
+ */
+ @Test
+ public void testScan() {
+Vector>
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r315010843
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.benchmark.generated.User;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
Review comment:
I have updated the test to auto-provision MongoDB cluster without having to
run a local copy. I have just added a simple method to this. I haven't used the
test-jar or the file you have attached.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r315010693
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.benchmark.generated.User;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
+
+ private static final String TABLE = "users";
+ private GoraBenchmarkClient client;
+ private static HashMap DATA_TO_INSERT;
+ private static HashMap DATA_TO_UPDATE;
+ private static final int NUMBER_OF_FIELDS = 10;
+ private GoraBenchmarkUtils bmutils = new GoraBenchmarkUtils();
+
+ /**
+ * Sets the up.
+ *
+ * Setup is executed before each test. Use @BeforeClass if you want to
execute
+ * a code block just once.
+ *
+ * @throws Exception
+ * the exceptionfiles are auto-generated. I have the code to add
the license file accordingly
+ */
+ @Before
+ public void setUp() throws Exception {
+DATA_TO_INSERT = new HashMap<>();
+DATA_TO_UPDATE = new HashMap<>();
+for (int i = 0; i < NUMBER_OF_FIELDS; i++) {
+ DATA_TO_INSERT.put("field" + i, new StringByteIterator("value" + i));
+ DATA_TO_UPDATE.put("field" + i, new StringByteIterator("updated" + i));
+}
+Properties p = new Properties();
+p.setProperty("key.class", "java.lang.String");
+p.setProperty("persistent.class",
"org.apache.gora.benchmark.generated.User");
+p.setProperty(CoreWorkload.FIELD_COUNT_PROPERTY, NUMBER_OF_FIELDS + "");
+client = new GoraBenchmarkClient();
+client.setProperties(p);
+client.init();
+ }
+
+ /**
+ * Clean up.
+ *
+ * @throws Exception
+ * the exception
+ */
+ @After
+ public void cleanUp() throws Exception {
+if (client != null)
+ client.cleanup();
+client = null;
+ }
+
+ /**
+ * Read record.
+ *
+ * @param key
+ * the key
+ * @return the user
+ * @throws GoraException
+ * the gora exception
+ */
+ private User readRecord(String key) throws GoraException {
+User u = client.dataStore.get(key);
+return u;
+ }
+
+ /**
+ * Test client initialisation.
+ */
+ @Test
+ public void testClientInitialisation() {
+assertNotNull(client.dataStore);
+ }
+
+ /**files are auto-generated. I have the code to add the license file
accordingly
+ * Test insert.
+ *
+ * @throws GoraException
+ * the gora exception
+ */
+ @Test
+ public void testInsert() throws GoraException {
+Status result1 = client.insert(TABLE, "key1", DATA_TO_INSERT);
+Status result2 = client.insert(TABLE, "key2", DATA_TO_INSERT);
+Status result3 = client.insert(TABLE, "key3", DATA_TO_INSERT);
+assertEquals(Status.OK, result1);
+assertEquals(Status.OK, result2);
+assertEquals(Status.OK, result3);
+ }
+
+ /**
+ * Test read.
+ */
+ @Test
+ public void testRead() {
+HashMap results = new HashMap<>();
+Set fields = new HashSet<>();// this could be null as well
+// fields.add("field0");
+Status result = client.read(TABLE, "key1", fields, results);
+assertEquals(Status.OK, result);
+assertEquals(DATA_TO_INSERT.size(), results.size());
+assertEquals(DATA_TO_INSERT.get("field0").toString(),
results.get("field0").toString());
+assertEquals(DATA_TO_INSERT.get("field0").toString(), "value0");
+ }
+
+ /**
+ * Test scan.
+ */
+ @Test
+ public void testScan() {
+Vector>
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r313849429
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.benchmark.generated.User;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
+
+ private static final String TABLE = "users";
+ private GoraBenchmarkClient client;
+ private static HashMap DATA_TO_INSERT;
+ private static HashMap DATA_TO_UPDATE;
+ private static final int NUMBER_OF_FIELDS = 10;
+ private GoraBenchmarkUtils bmutils = new GoraBenchmarkUtils();
+
+ /**
+ * Sets the up.
+ *
+ * Setup is executed before each test. Use @BeforeClass if you want to
execute
+ * a code block just once.
+ *
+ * @throws Exception
+ * the exceptionfiles are auto-generated. I have the code to add
the license file accordingly
+ */
+ @Before
+ public void setUp() throws Exception {
+DATA_TO_INSERT = new HashMap<>();
+DATA_TO_UPDATE = new HashMap<>();
+for (int i = 0; i < NUMBER_OF_FIELDS; i++) {
+ DATA_TO_INSERT.put("field" + i, new StringByteIterator("value" + i));
+ DATA_TO_UPDATE.put("field" + i, new StringByteIterator("updated" + i));
+}
+Properties p = new Properties();
+p.setProperty("key.class", "java.lang.String");
+p.setProperty("persistent.class",
"org.apache.gora.benchmark.generated.User");
+p.setProperty(CoreWorkload.FIELD_COUNT_PROPERTY, NUMBER_OF_FIELDS + "");
+client = new GoraBenchmarkClient();
+client.setProperties(p);
+client.init();
+ }
+
+ /**
+ * Clean up.
+ *
+ * @throws Exception
+ * the exception
+ */
+ @After
+ public void cleanUp() throws Exception {
+if (client != null)
+ client.cleanup();
+client = null;
+ }
+
+ /**
+ * Read record.
+ *
+ * @param key
+ * the key
+ * @return the user
+ * @throws GoraException
+ * the gora exception
+ */
+ private User readRecord(String key) throws GoraException {
+User u = client.dataStore.get(key);
+return u;
+ }
+
+ /**
+ * Test client initialisation.
+ */
+ @Test
+ public void testClientInitialisation() {
+assertNotNull(client.dataStore);
+ }
+
+ /**files are auto-generated. I have the code to add the license file
accordingly
+ * Test insert.
+ *
+ * @throws GoraException
+ * the gora exception
+ */
+ @Test
+ public void testInsert() throws GoraException {
+Status result1 = client.insert(TABLE, "key1", DATA_TO_INSERT);
+Status result2 = client.insert(TABLE, "key2", DATA_TO_INSERT);
+Status result3 = client.insert(TABLE, "key3", DATA_TO_INSERT);
+assertEquals(Status.OK, result1);
+assertEquals(Status.OK, result2);
+assertEquals(Status.OK, result3);
+ }
+
+ /**
+ * Test read.
+ */
+ @Test
+ public void testRead() {
+HashMap results = new HashMap<>();
+Set fields = new HashSet<>();// this could be null as well
+// fields.add("field0");
+Status result = client.read(TABLE, "key1", fields, results);
+assertEquals(Status.OK, result);
+assertEquals(DATA_TO_INSERT.size(), results.size());
+assertEquals(DATA_TO_INSERT.get("field0").toString(),
results.get("field0").toString());
+assertEquals(DATA_TO_INSERT.get("field0").toString(), "value0");
+ }
+
+ /**
+ * Test scan.
+ */
+ @Test
+ public void testScan() {
+Vector>
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r313660749
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,428 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Properties;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Comment;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.Node;
+
+import com.yahoo.ycsb.ByteIterator;
+
+/**
+ * The Class GoraBenchmarkUtils has some utilities that dynamically generate
files needed to for gora.
+ * It generate the following files.
+ * a. Database Mapping File
+ * b. Avro Files
+ * c. Data Beans
+ */
+public class GoraBenchmarkUtils {
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE =
"org.apache.gora.benchmark.generated";
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+ private static final Logger LOG =
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r313592963
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,271 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+import org.apache.gora.benchmark.generated.User;
+
+/**
+ * The Class GoraBenchmarkClient
+ *
+ * @author sc306 This class extends the Yahoo! Cloud Service Benchmark
benchmark
+ * {@link #com.yahoo.ycsb.DB DB} class to provide functionality for
+ * {@link #insert(String, String, HashMap) insert},
+ * {@link #read(String, String, Set, HashMap) read},
+ * {@link #scan(String, String, int, Set, Vector) scan} and
+ * {@link #update(String, String, HashMap) update} methods as per
Apache
+ * Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+ private static final String FIELDS[] = User._ALL_FIELDS;
+ private static volatile boolean executed;
+ public static int fieldCount;
+ /** This is only for set to array conversion in {@link read()} method */
+ private String[] DUMMY_ARRAY = new String[0];
+ DataStore dataStore;
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+ User user = new User();
+ private Properties prop;
+
+ public GoraBenchmarkClient() {
+ }
+
+ /***
+ * Initialisation method. This method is called once for each database
+ * instance.
+ */
+ public void init() throws DBException {
+try {
+ // Get YCSB properties
+ prop = getProperties();
+ fieldCount = Integer
+ .parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+ String keyClass = prop.getProperty("key.class", "java.lang.String");
+ String persistentClass = prop.getProperty("persistent.class",
"org.apache.gora.benchmark.generated.User");
+ Properties p = DataStoreFactory.createProps();
+ dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass, p,
new Configuration());
+ synchronized (GoraBenchmarkClient.class) {
+if (executed)
+ return;
+executed = true;
+goraBenchmarkUtils.generateAvroSchema(fieldCount);
+String dataStoreName = goraBenchmarkUtils.getDataStore(p);
+goraBenchmarkUtils.generateMappingFile(dataStoreName);
+goraBenchmarkUtils.generateDataBeans();
+ }
+} catch (GoraException e) {
+ LOG.info("There is a problem in initialising the DataStore
\n"+e.getMessage());
+}
+ }
+
+ /**
+ * Cleanup any state for this DB.
+ *
+ * It is very important to close the datastore properly, otherwise some data
+ * loss might occur.
+ */
+ public void cleanup() throws DBException {
+synchronized (GoraBenchmarkClient.class) {
Review comment:
I actually forgot to remove that bit of the code. It is not needed. We don't
need a synchronised block in the cleanup method. So I will remove that bit of
the code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r313587768
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,271 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+import org.apache.gora.benchmark.generated.User;
+
+/**
+ * The Class GoraBenchmarkClient
+ *
+ * @author sc306 This class extends the Yahoo! Cloud Service Benchmark
benchmark
+ * {@link #com.yahoo.ycsb.DB DB} class to provide functionality for
+ * {@link #insert(String, String, HashMap) insert},
+ * {@link #read(String, String, Set, HashMap) read},
+ * {@link #scan(String, String, int, Set, Vector) scan} and
+ * {@link #update(String, String, HashMap) update} methods as per
Apache
+ * Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+ private static final String FIELDS[] = User._ALL_FIELDS;
+ private static volatile boolean executed;
+ public static int fieldCount;
+ /** This is only for set to array conversion in {@link read()} method */
+ private String[] DUMMY_ARRAY = new String[0];
+ DataStore dataStore;
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+ User user = new User();
+ private Properties prop;
+
+ public GoraBenchmarkClient() {
+ }
+
+ /***
+ * Initialisation method. This method is called once for each database
+ * instance.
+ */
+ public void init() throws DBException {
+try {
+ // Get YCSB properties
+ prop = getProperties();
+ fieldCount = Integer
+ .parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+ String keyClass = prop.getProperty("key.class", "java.lang.String");
+ String persistentClass = prop.getProperty("persistent.class",
"org.apache.gora.benchmark.generated.User");
+ Properties p = DataStoreFactory.createProps();
+ dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass, p,
new Configuration());
Review comment:
@djkevincr, yes that should be possible. Is there a reason for this? What I
know is, when running the benchmark client the number of threads to start can
be supplied and the implemented init() method of YCSB framework is executed
once by each of those threads. Maybe if you explain more then I can provide a
better explanation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r313558077
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,271 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+import org.apache.gora.benchmark.generated.User;
+
+/**
+ * The Class GoraBenchmarkClient
+ *
+ * @author sc306 This class extends the Yahoo! Cloud Service Benchmark
benchmark
+ * {@link #com.yahoo.ycsb.DB DB} class to provide functionality for
+ * {@link #insert(String, String, HashMap) insert},
+ * {@link #read(String, String, Set, HashMap) read},
+ * {@link #scan(String, String, int, Set, Vector) scan} and
+ * {@link #update(String, String, HashMap) update} methods as per
Apache
+ * Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+ private static final String FIELDS[] = User._ALL_FIELDS;
+ private static volatile boolean executed;
+ public static int fieldCount;
+ /** This is only for set to array conversion in {@link read()} method */
+ private String[] DUMMY_ARRAY = new String[0];
+ DataStore dataStore;
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+ User user = new User();
+ private Properties prop;
+
+ public GoraBenchmarkClient() {
+ }
+
+ /***
+ * Initialisation method. This method is called once for each database
+ * instance.
+ */
+ public void init() throws DBException {
+try {
+ // Get YCSB properties
+ prop = getProperties();
+ fieldCount = Integer
+ .parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+ String keyClass = prop.getProperty("key.class", "java.lang.String");
+ String persistentClass = prop.getProperty("persistent.class",
"org.apache.gora.benchmark.generated.User");
+ Properties p = DataStoreFactory.createProps();
+ dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass, p,
new Configuration());
+ synchronized (GoraBenchmarkClient.class) {
Review comment:
@djkevincr, thanks for the comment. I am not entirely sure if this is the
best approach, but my choice of using class level synchronisation is to avoid
any form of interference from other objects. According to my understanding of
how Gora works, the resources I am creating (mapping files, data beans and Avro
schemas) in the synchronization block should be created once. So I thought the
safest way of doing this would be to use class level synchronization. Also, I
am using a shared variable (private static volatile boolean executed) in the
synchronized block that checks whether the code block has been executed by
other threads.
As for the second part, of your question, can you please elaborate further?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r313558077
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,271 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+import org.apache.gora.benchmark.generated.User;
+
+/**
+ * The Class GoraBenchmarkClient
+ *
+ * @author sc306 This class extends the Yahoo! Cloud Service Benchmark
benchmark
+ * {@link #com.yahoo.ycsb.DB DB} class to provide functionality for
+ * {@link #insert(String, String, HashMap) insert},
+ * {@link #read(String, String, Set, HashMap) read},
+ * {@link #scan(String, String, int, Set, Vector) scan} and
+ * {@link #update(String, String, HashMap) update} methods as per
Apache
+ * Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+ private static final String FIELDS[] = User._ALL_FIELDS;
+ private static volatile boolean executed;
+ public static int fieldCount;
+ /** This is only for set to array conversion in {@link read()} method */
+ private String[] DUMMY_ARRAY = new String[0];
+ DataStore dataStore;
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+ User user = new User();
+ private Properties prop;
+
+ public GoraBenchmarkClient() {
+ }
+
+ /***
+ * Initialisation method. This method is called once for each database
+ * instance.
+ */
+ public void init() throws DBException {
+try {
+ // Get YCSB properties
+ prop = getProperties();
+ fieldCount = Integer
+ .parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+ String keyClass = prop.getProperty("key.class", "java.lang.String");
+ String persistentClass = prop.getProperty("persistent.class",
"org.apache.gora.benchmark.generated.User");
+ Properties p = DataStoreFactory.createProps();
+ dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass, p,
new Configuration());
+ synchronized (GoraBenchmarkClient.class) {
Review comment:
@djkevincr, thanks for the comment. I am not entirely sure if this is the
best approach, but my choice of using class level synchronisation is to avoid
any form of interference from other objects. According to my understanding of
how Gora works, the resources I am creating (mapping files, data beans and avro
schemas) in the synchronization block should be created once. So I thought the
safest way of doing this would be to use class level synchronization. Also, I
am using a shared variable (private static volatile boolean executed) in the
synchronized block that checks whether the code block has been executed by
other threads.
As for the second part, of you question, can you please elaborate further?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r313388763
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,186 @@
+
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+org.apache.gora
+gora
+0.9-SNAPSHOT
+
+ gora-benchmark
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an in-memory
data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+The Apache Software Foundation
+http://www.apache.org/
+
+
+JIRA
+https://issues.apache.org/jira/browse/GORA
+
+
+Jenkins
+https://builds.apache.org/job/Gora-trunk/
+
+
+UTF-8
+
+
+target
+target/classes
+${project.artifactId}-${project.version}
+target/test-classes
+src/test/java
+src/main/java
+
+
+${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+org.codehaus.mojo
+build-helper-maven-plugin
+${build-helper-maven-plugin.version}
+
+
+generate-sources
+
+ add-source
+
+
+
+src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.15.0
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+
+
+
+ org.apache.gora
+ gora-hbase
+
+
+
+ org.apache.gora
+ gora-jcache
+
+
+
+ org.apache.gora
+ gora-couchdb
+
+
+
+ org.apache.gora
+ gora-cassandra
+
+
+
+ org.apache.gora
+ gora-solr
+
+
+
+ org.apache.gora
+ gora-aerospike
+
+
+
+ org.apache.avro
+ avro
+ compile
+
+
+
+ gora-compiler
+ org.apache.gora
+
+
+
+
+ org.slf4j
+ slf4j-log4j12
+
+
+ log4j
+ log4j
+
+
+ javax.jms
+ jms
+
+
+
+
+
+
+ org.apache.hadoop
+ hadoop-client
+ compile
+ true
+
+
+
+ org.fluttercode.datafactory
+ datafactory
+ 0.8
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r313388632 ## File path: gora-benchmark/gora-workloada.csv ## @@ -0,0 +1,251 @@ +records,timems,recordsk,timesec,platform Review comment: I think will be best to not include the CSV files at all. So I have excluded all of them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r312845763 ## File path: gora-benchmark/gora-bench.sh ## @@ -0,0 +1,75 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#Adapted from YCSB's version of ycsb.sh. +CLASSPATH="." Review comment: Like you said, I think for now it is best to keep them separate. I will move the script to GORA_HOME/bin. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r312845330
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,186 @@
+
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+org.apache.gora
+gora
+0.9-SNAPSHOT
+
+ gora-benchmark
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an in-memory
data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+The Apache Software Foundation
+http://www.apache.org/
+
+
+JIRA
+https://issues.apache.org/jira/browse/GORA
+
+
+Jenkins
+https://builds.apache.org/job/Gora-trunk/
+
+
+UTF-8
+
+
+target
+target/classes
+${project.artifactId}-${project.version}
+target/test-classes
+src/test/java
+src/main/java
+
+
+${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+org.codehaus.mojo
+build-helper-maven-plugin
+${build-helper-maven-plugin.version}
+
+
+generate-sources
+
+ add-source
+
+
+
+src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.15.0
Review comment:
Okay, once the maven central issue is addressed then I will handle this
accordingly. Does that sounds good?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r312701363 ## File path: gora-benchmark/README.md ## @@ -0,0 +1,48 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +This guidline outlines the process of compiling and running Gora Benchmark Module + +Prerequisite +Install and setup the required database to test. Currently, the benchmark is tested on MongoDB, HBase and CouchDB. +Install maven + +Set up + +1. Edit the gora.properties file located at src/main/resources/gora.properties and set configurations accordingly. + + a. To test MongoDB set gora.datastore.default=org.apache.gora.mongodb.store.MongoStore + b. To test HBase set gora.datastore.default=org.apache.gora.hbase.store.HBaseStore + c. To test CouchDB set gora.datastore.default=org.apache.gora.couchdb.store.CouchDBStore + + +2. From the module directory i.e. gora-benchmark, run mvn clean install +3. This should run some test and successfully build the module. + If the build fails and complains about maven not able to ycsb-core-verion.jar then download the jar manually and add it to ~/.m2/repository/com/yahoo/ycsb/core/0.1.4/ Review comment: We are now working with YCSB to resolve this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r312690799 ## File path: gora-tutorial/conf/gora.properties ## @@ -66,3 +66,18 @@ gora.datastore.jcache.hazelcast.config=hazelcast.xml #gora.aerospikestore.server.port=3000 #gora.aerospikestore.server.username= #gora.aerospikestore.server.password= + + +# MongoDBStore properties # + + +gora.datastore.autocreateschema=true Review comment: I have updated my branch with the current master. This is now resolved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r312689641 ## File path: gora-tutorial/conf/gora.properties ## @@ -66,3 +66,18 @@ gora.datastore.jcache.hazelcast.config=hazelcast.xml #gora.aerospikestore.server.port=3000 #gora.aerospikestore.server.username= #gora.aerospikestore.server.password= + + +# MongoDBStore properties # + + +gora.datastore.autocreateschema=true Review comment: Not sure how this happened. I have already merged master with my branch thinking that all should be okay. I will go ahead and remove the duplicate lines. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r312689269
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,188 @@
+
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+org.apache.gora
+gora
+0.9-SNAPSHOT
+
+ gora-benchmark
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an in-memory
data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+The Apache Software Foundation
+http://www.apache.org/
+
+
+JIRA
+https://issues.apache.org/jira/browse/GORA
+
+
+Jenkins
+https://builds.apache.org/job/Gora-trunk/
+
+
+UTF-8
+
+
+target
+target/classes
+${project.artifactId}-${project.version}
+target/test-classes
+src/test/java
+src/main/java
+
+
+${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+org.codehaus.mojo
+build-helper-maven-plugin
+${build-helper-maven-plugin.version}
+
+
+generate-sources
+
+ add-source
+
+
+
+src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
Review comment:
I have got some feedback from YCSB maintainers [1]. So basically, they have
not published ycsb in maven central.
[1] https://github.com/brianfrankcooper/YCSB/issues/1340
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r312688778
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,188 @@
+
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+org.apache.gora
+gora
+0.9-SNAPSHOT
+
+ gora-benchmark
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an in-memory
data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+The Apache Software Foundation
+http://www.apache.org/
+
+
+JIRA
+https://issues.apache.org/jira/browse/GORA
+
+
+Jenkins
+https://builds.apache.org/job/Gora-trunk/
+
+
+UTF-8
+
+
+target
+target/classes
+${project.artifactId}-${project.version}
+target/test-classes
+src/test/java
+src/main/java
+
+
+${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+org.codehaus.mojo
+build-helper-maven-plugin
+${build-helper-maven-plugin.version}
+
+
+generate-sources
+
+ add-source
+
+
+
+src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
Review comment:
I understand this isn't the standard approach but what I did was to download
the jar file manually from maven website and place it in
~/.m2/repository/com/yahoo/ycsb/core/0.1.4/. I saw this trick from [1]. Also
[2] and [3] experienced similar problems.
I will log an issue on YCSB GitHub page.
[1] https://stackoverflow.com/questions/22600408/use-maven-to-compile-ycsb
[2] https://github.com/blockchain-jd-com/jdchain/issues/13
[3] https://github.com/brianfrankcooper/YCSB/issues/174
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311635669
##
File path: gora-benchmark/runWorkloads.sh
##
@@ -0,0 +1,119 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Check if number of commanline arguments is right
+threadcount=15
+if [ $# -ne 3 ]
+then
+echo "Error: Usage: $0 hbase|mongodb gora|ycsb workloadfile"
+exit;
+fi
+#Log output file
+outputfile="$3-$2-$1.log"
+insertfactor=500
+if [ $2 = "ycsb" ]
+then
+ table="usertable"
+elif [ $2 == "gora" ]
+then
+ table="users"
+fi
+
+if [ $1 = "hbase" ]
+then
+ #Read operation
+ echo -e "Disabling Users Table" | tee -a $outputfile
+ echo -e "disable '$table'" | hbase shell -n
+ echo -e "Dropping Users Table" | tee -a $outputfile
+ echo -e "drop '$table'" | hbase shell -n
+ echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ fi
+for i in {10..500..10}
+do
+ #Insert only workload, uncomment this block of code. I will improve this
later.
+ #echo -e "Disabling Users Table" | tee -a $outputfile
+ #echo -e "disable '$table'" | hbase shell -n
+ #echo -e "Dropping Users Table" | tee -a $outputfile
+ #echo -e "drop '$table'" | hbase shell -n
+ #echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ #echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+
+ echo -e "Table Setup Completed = Now Running Benchmark
for $i records and operations" | tee -a $outputfile
+ echo -e "Record Count $i" | tee -a $outputfile
+ echo -e "Operation Count $i" | tee -a $outputfile
+ echo -e "Thread Count $threadcount" | tee -a $outputfile
+ echo -e ""
+ if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ ./gora-bench.sh run -threads 15 -s -p readallfields=true -p
measurementtype=timeseries -p timeseries.granularity=2000 -p operationcount=$i
-P workloads/$3 | tee -a $outputfile
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ ../../ycsb/bin/ycsb.sh run hbase20 -s -p columnfamily=info -p
measurementtype=timeseries -p timeseries.granularity=2000 -threads 15 -p
readallfields=true -p operationcount=$i -P workloads/$3 | tee -a $outputfile
+ fi
+ echo "End of
Benchmark" | tee -a $outputfile
+done
+elif [ $1 = "mongodb" ]
+then
+ echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ if [ $2 = "ycsb" ]
+ then
+ mongo < setupycsbmongo.js
+ ../../ycsb/bin/ycsb.sh load mongodb -threads 15 -s -p fieldcount=20
-p recordcount=$insertfactor -p operationcount=$insertfactor -P
workloads/workloada
+ fi
+
+ if [ $2 = "gora" ]
+ then
+ mongo < setupgoramongo.js
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$insertfactor -P workloads/workloada
+ fi
+ echo -e "Table Setup Completed = Now Running
Benchmark for $i records and operations" | tee -a $outputfile
+for i in {10..500..10}
+do
+ #echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ #mongo < setupmongo.js
+ #echo -e "Table Setup Completed
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311631483 ## File path: gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java ## @@ -0,0 +1,453 @@ +package org.apache.gora.benchmark; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.StringWriter; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.HashMap; +import java.util.Iterator; +import java.util.Locale; +import java.util.Properties; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; +import javax.xml.parsers.ParserConfigurationException; +import javax.xml.transform.OutputKeys; +import javax.xml.transform.Transformer; +import javax.xml.transform.TransformerConfigurationException; +import javax.xml.transform.TransformerException; +import javax.xml.transform.TransformerFactory; +import javax.xml.transform.dom.DOMSource; +import javax.xml.transform.stream.StreamResult; + +import org.apache.avro.Schema; +import org.apache.avro.SchemaBuilder; +import org.apache.avro.SchemaBuilder.FieldAssembler; +import org.apache.gora.compiler.GoraCompiler; +import org.apache.gora.compiler.utils.LicenseHeaders; +import org.json.JSONArray; +import org.json.JSONObject; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.w3c.dom.Attr; +import org.w3c.dom.Document; +import org.w3c.dom.Element; + +import com.yahoo.ycsb.ByteIterator; + +// TODO: Auto-generated Javadoc +/** + * The Class GoraBenchmarkUtils. Review comment: Updated the file with some description, the methods in this file are called in the init method of GoraBenchmarkClient. The class is not directly executed. Basically, you can specify how many fields a particular table in a particular datastore should have. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311490797 ## File path: gora-benchmark/src/main/resources/gora-couchdb-mapping.xml ## @@ -0,0 +1,26 @@ + Review comment: Mapping files are auto-generated. I have added the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311491122 ## File path: gora-benchmark/src/main/resources/gora-mongodb-mapping.xml ## @@ -0,0 +1,26 @@ + Review comment: Mapping files are auto-generated. I have added the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311490835 ## File path: gora-benchmark/src/main/resources/gora-hbase-mapping.xml ## @@ -0,0 +1,59 @@ + Review comment: Mapping files are auto-generated. I have added the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311490797 ## File path: gora-benchmark/src/main/resources/gora-couchdb-mapping.xml ## @@ -0,0 +1,26 @@ + Review comment: Mapping files are auto-generated. I have the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311490835 ## File path: gora-benchmark/src/main/resources/gora-hbase-mapping.xml ## @@ -0,0 +1,59 @@ + Review comment: Mapping files are auto-generated. I have the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311490191
##
File path: gora-benchmark/src/main/avro/user.json
##
@@ -0,0 +1,112 @@
+{
+"name": "User",
+"namespace": "generated",
+"type": "record",
+"fields": [
+{
Review comment:
The autogenerated JSON files are all the same. The person.json file is not
being used. I will remove the file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311489884
##
File path: gora-benchmark/src/main/avro/person.json
##
@@ -0,0 +1,18 @@
+ {
+ "type": "record",
+ "name": "Person", "default":null,
+ "namespace": "generated",
+ "fields" : [
+{"name": "userId", "type": ["null","string"], "default":null},
Review comment:
I am not actually using this file anymore, but yes it would be interesting
to test with complex objects. For YCSB it uses a simple User object which can
have **n** number of fields.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311474325
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
Review comment:
Unlike other modules, there is no definition or declaration of MongoDB in
the parent pom file, hence the explicit declaration in my case for now. Maybe I
can create a separate issue for this or should I just go ahead and edit the
parent pom accordingly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311474325
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
Review comment:
Like others, there is no definition or declaration of MongoDB in the parent
pom file, hence the explicit declaration for now. Maybe I can create a separate
issue for this or should I just go ahead and edit the parent pom accordingly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
