[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r315047198 ## File path: gora-benchmark/src/main/resources/gora-couchdb-mapping.xml ## @@ -0,0 +1,26 @@ + Review comment: please add it here then This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r315046925 ## File path: gora-benchmark/src/main/java/org/apache/gora/benchmark/Constants.java ## @@ -0,0 +1,97 @@ +package org.apache.gora.benchmark; Review comment: license header This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r315048322 ## File path: pom.xml ## @@ -826,10 +827,21 @@ 2.2.22 0.2.0 - + + +1.4.2 + + +2.0.0 + 4.10 1.4.2 + + +0.15.0 Review comment: This should be the newest jar version until we get the jar published to Maven Central. The user is required to download this so make it the newest. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r31504
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.benchmark.generated.User;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
Review comment:
OK whatever you want.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r312928379 ## File path: gora-benchmark/pom.xml ## @@ -0,0 +1,186 @@ + + +http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd; + xmlns="http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;> + 4.0.0 + +org.apache.gora +gora +0.9-SNAPSHOT + + gora-benchmark Review comment: I agree This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r312928512
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,186 @@
+
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+org.apache.gora
+gora
+0.9-SNAPSHOT
+
+ gora-benchmark
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an in-memory
data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+The Apache Software Foundation
+http://www.apache.org/
+
+
+JIRA
+https://issues.apache.org/jira/browse/GORA
+
+
+Jenkins
+https://builds.apache.org/job/Gora-trunk/
+
+
+UTF-8
+
+
+target
+target/classes
+${project.artifactId}-${project.version}
+target/test-classes
+src/test/java
+src/main/java
+
+
+${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+org.codehaus.mojo
+build-helper-maven-plugin
+${build-helper-maven-plugin.version}
+
+
+generate-sources
+
+ add-source
+
+
+
+src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.15.0
Review comment:
Yes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r312932313
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+import org.apache.gora.benchmark.generated.User;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.Status;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
Review comment:
OK so the issue I have here is that the benchmasrk unit tests expect a local
MongoDB instance ot be running... which one is not. We need to automate the
provisioning of MongoDB just as is done in the
[GoraMongodbTestDriver.java](https://github.com/apache/gora/blob/master/gora-mongodb/src/test/java/org/apache/gora/mongodb/GoraMongodbTestDriver.java).
Please facilitate this. You may wish to just rely upon that test-jar artifact.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r312928778
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,188 @@
+
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+org.apache.gora
+gora
+0.9-SNAPSHOT
+
+ gora-benchmark
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an in-memory
data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+The Apache Software Foundation
+http://www.apache.org/
+
+
+JIRA
+https://issues.apache.org/jira/browse/GORA
+
+
+Jenkins
+https://builds.apache.org/job/Gora-trunk/
+
+
+UTF-8
+
+
+target
+target/classes
+${project.artifactId}-${project.version}
+target/test-classes
+src/test/java
+src/main/java
+
+
+${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+org.codehaus.mojo
+build-helper-maven-plugin
+${build-helper-maven-plugin.version}
+
+
+generate-sources
+
+ add-source
+
+
+
+src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
Review comment:
We need to help them on that issue. I will work with them to do that.
Hopefully when they release 0.17.0 we can upgrade here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r312680635
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,188 @@
+
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+org.apache.gora
+gora
+0.9-SNAPSHOT
+
+ gora-benchmark
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an in-memory
data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+The Apache Software Foundation
+http://www.apache.org/
+
+
+JIRA
+https://issues.apache.org/jira/browse/GORA
+
+
+Jenkins
+https://builds.apache.org/job/Gora-trunk/
+
+
+UTF-8
+
+
+target
+target/classes
+${project.artifactId}-${project.version}
+target/test-classes
+src/test/java
+src/main/java
+
+
+${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+org.codehaus.mojo
+build-helper-maven-plugin
+${build-helper-maven-plugin.version}
+
+
+generate-sources
+
+ add-source
+
+
+
+src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
Review comment:
If this dependency is not available from Maven central then we need to
address that. How are you currently fetching it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r312680421
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,188 @@
+
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+org.apache.gora
+gora
+0.9-SNAPSHOT
+
+ gora-benchmark
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an in-memory
data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+The Apache Software Foundation
+http://www.apache.org/
+
+
+JIRA
+https://issues.apache.org/jira/browse/GORA
+
+
+Jenkins
+https://builds.apache.org/job/Gora-trunk/
+
+
+UTF-8
+
+
+target
+target/classes
+${project.artifactId}-${project.version}
+target/test-classes
+src/test/java
+src/main/java
+
+
+${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+org.codehaus.mojo
+build-helper-maven-plugin
+${build-helper-maven-plugin.version}
+
+
+generate-sources
+
+ add-source
+
+
+
+src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
Review comment:
This issue is still not resolved. Can you please update it otherwise this
module will break when we release another version. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r312680051 ## File path: gora-benchmark/README.md ## @@ -0,0 +1,48 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +This guidline outlines the process of compiling and running Gora Benchmark Module + +Prerequisite +Install and setup the required database to test. Currently, the benchmark is tested on MongoDB, HBase and CouchDB. +Install maven + +Set up + +1. Edit the gora.properties file located at src/main/resources/gora.properties and set configurations accordingly. + + a. To test MongoDB set gora.datastore.default=org.apache.gora.mongodb.store.MongoStore + b. To test HBase set gora.datastore.default=org.apache.gora.hbase.store.HBaseStore + c. To test CouchDB set gora.datastore.default=org.apache.gora.couchdb.store.CouchDBStore + + +2. From the module directory i.e. gora-benchmark, run mvn clean install +3. This should run some test and successfully build the module. + If the build fails and complains about maven not able to ycsb-core-verion.jar then download the jar manually and add it to ~/.m2/repository/com/yahoo/ycsb/core/0.1.4/ Review comment: Why would this ever be the case? Can you elaborate? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r312680021 ## File path: gora-benchmark/README.md ## @@ -0,0 +1,48 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +This guidline outlines the process of compiling and running Gora Benchmark Module Review comment: guidline --> guideline This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r312680102 ## File path: gora-benchmark/README.md ## @@ -0,0 +1,48 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +This guidline outlines the process of compiling and running Gora Benchmark Module + +Prerequisite +Install and setup the required database to test. Currently, the benchmark is tested on MongoDB, HBase and CouchDB. +Install maven + +Set up + +1. Edit the gora.properties file located at src/main/resources/gora.properties and set configurations accordingly. + + a. To test MongoDB set gora.datastore.default=org.apache.gora.mongodb.store.MongoStore + b. To test HBase set gora.datastore.default=org.apache.gora.hbase.store.HBaseStore + c. To test CouchDB set gora.datastore.default=org.apache.gora.couchdb.store.CouchDBStore + + +2. From the module directory i.e. gora-benchmark, run mvn clean install +3. This should run some test and successfully build the module. + If the build fails and complains about maven not able to ycsb-core-verion.jar then download the jar manually and add it to ~/.m2/repository/com/yahoo/ycsb/core/0.1.4/ Review comment: I experienced exactly this issue ``` [ERROR] Failed to execute goal on project gora-benchmark: Could not resolve dependencies for project org.apache.gora:gora-benchmark:jar:0.9-SNAPSHOT: Could not find artifact com.yahoo.ycsb:core:jar:0.1.4 in central (https://repo.maven.apache.org/maven2) -> [Help 1] [ERROR] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311649100 ## File path: gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java ## @@ -0,0 +1,453 @@ +package org.apache.gora.benchmark; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.StringWriter; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.HashMap; +import java.util.Iterator; +import java.util.Locale; +import java.util.Properties; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; +import javax.xml.parsers.ParserConfigurationException; +import javax.xml.transform.OutputKeys; +import javax.xml.transform.Transformer; +import javax.xml.transform.TransformerConfigurationException; +import javax.xml.transform.TransformerException; +import javax.xml.transform.TransformerFactory; +import javax.xml.transform.dom.DOMSource; +import javax.xml.transform.stream.StreamResult; + +import org.apache.avro.Schema; +import org.apache.avro.SchemaBuilder; +import org.apache.avro.SchemaBuilder.FieldAssembler; +import org.apache.gora.compiler.GoraCompiler; +import org.apache.gora.compiler.utils.LicenseHeaders; +import org.json.JSONArray; +import org.json.JSONObject; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.w3c.dom.Attr; +import org.w3c.dom.Document; +import org.w3c.dom.Element; + +import com.yahoo.ycsb.ByteIterator; + +// TODO: Auto-generated Javadoc +/** + * The Class GoraBenchmarkUtils. Review comment: Thanks for updating with a description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311648826
##
File path: gora-benchmark/src/main/avro/person.json
##
@@ -0,0 +1,18 @@
+ {
+ "type": "record",
+ "name": "Person", "default":null,
+ "namespace": "generated",
+ "fields" : [
+{"name": "userId", "type": ["null","string"], "default":null},
Review comment:
OK no problem. Please remote file. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311648700
##
File path: gora-benchmark/runWorkloads.sh
##
@@ -0,0 +1,119 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Check if number of commanline arguments is right
+threadcount=15
+if [ $# -ne 3 ]
+then
+echo "Error: Usage: $0 hbase|mongodb gora|ycsb workloadfile"
+exit;
+fi
+#Log output file
+outputfile="$3-$2-$1.log"
+insertfactor=500
+if [ $2 = "ycsb" ]
+then
+ table="usertable"
+elif [ $2 == "gora" ]
+then
+ table="users"
+fi
+
+if [ $1 = "hbase" ]
+then
+ #Read operation
+ echo -e "Disabling Users Table" | tee -a $outputfile
+ echo -e "disable '$table'" | hbase shell -n
+ echo -e "Dropping Users Table" | tee -a $outputfile
+ echo -e "drop '$table'" | hbase shell -n
+ echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ fi
+for i in {10..500..10}
+do
+ #Insert only workload, uncomment this block of code. I will improve this
later.
+ #echo -e "Disabling Users Table" | tee -a $outputfile
+ #echo -e "disable '$table'" | hbase shell -n
+ #echo -e "Dropping Users Table" | tee -a $outputfile
+ #echo -e "drop '$table'" | hbase shell -n
+ #echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ #echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+
+ echo -e "Table Setup Completed = Now Running Benchmark
for $i records and operations" | tee -a $outputfile
+ echo -e "Record Count $i" | tee -a $outputfile
+ echo -e "Operation Count $i" | tee -a $outputfile
+ echo -e "Thread Count $threadcount" | tee -a $outputfile
+ echo -e ""
+ if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ ./gora-bench.sh run -threads 15 -s -p readallfields=true -p
measurementtype=timeseries -p timeseries.granularity=2000 -p operationcount=$i
-P workloads/$3 | tee -a $outputfile
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ ../../ycsb/bin/ycsb.sh run hbase20 -s -p columnfamily=info -p
measurementtype=timeseries -p timeseries.granularity=2000 -threads 15 -p
readallfields=true -p operationcount=$i -P workloads/$3 | tee -a $outputfile
+ fi
+ echo "End of
Benchmark" | tee -a $outputfile
+done
+elif [ $1 = "mongodb" ]
+then
+ echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ if [ $2 = "ycsb" ]
+ then
+ mongo < setupycsbmongo.js
+ ../../ycsb/bin/ycsb.sh load mongodb -threads 15 -s -p fieldcount=20
-p recordcount=$insertfactor -p operationcount=$insertfactor -P
workloads/workloada
+ fi
+
+ if [ $2 = "gora" ]
+ then
+ mongo < setupgoramongo.js
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$insertfactor -P workloads/workloada
+ fi
+ echo -e "Table Setup Completed = Now Running
Benchmark for $i records and operations" | tee -a $outputfile
+for i in {10..500..10}
+do
+ #echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ #mongo < setupmongo.js
+ #echo -e "Table Setup Completed
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311648487
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
Review comment:
+1, just go ahead and obtain this from parent pom. Do whatever you need to
do for that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310847083 ## File path: gora-benchmark/workloads/tsworkload_template ## @@ -0,0 +1,283 @@ +# Copyright (c) 2017 YCSB contributors. All rights reserved. Review comment: Remove gora-benchmark/workloads/.workloadb.swp above this should not be available within the pull request This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845256
##
File path: gora-benchmark/runWorkloads.sh
##
@@ -0,0 +1,119 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Check if number of commanline arguments is right
+threadcount=15
+if [ $# -ne 3 ]
+then
+echo "Error: Usage: $0 hbase|mongodb gora|ycsb workloadfile"
+exit;
+fi
+#Log output file
+outputfile="$3-$2-$1.log"
+insertfactor=500
+if [ $2 = "ycsb" ]
+then
+ table="usertable"
+elif [ $2 == "gora" ]
+then
+ table="users"
+fi
+
+if [ $1 = "hbase" ]
+then
+ #Read operation
+ echo -e "Disabling Users Table" | tee -a $outputfile
+ echo -e "disable '$table'" | hbase shell -n
+ echo -e "Dropping Users Table" | tee -a $outputfile
+ echo -e "drop '$table'" | hbase shell -n
+ echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ fi
+for i in {10..500..10}
+do
+ #Insert only workload, uncomment this block of code. I will improve this
later.
+ #echo -e "Disabling Users Table" | tee -a $outputfile
Review comment:
Remove code not being used
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846993
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,201 @@
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
+
+
+ /** The Constant TABLE. */
+ private static final String TABLE = "users";
+
+ /** The client. */
+ private GoraBenchmarkClient client;
+
+ /** The data to insert. */
+ private static HashMap DATA_TO_INSERT;
+
+ /** The data to update. */
+ private static HashMap DATA_TO_UPDATE;
+
+ /** The data stores. */
+ private static List dataStores = new ArrayList();
+
+ /** The Constant NUMBER_OF_FIELDS. */
+ private static final int NUMBER_OF_FIELDS = 10;
+
+ /** The bmutils. */
+ private GoraBenchmarkUtils bmutils = new GoraBenchmarkUtils();
+
+ /**
+ * Sets the up.
+ *
+ * @throws Exception the exception
+ */
+ //Setup is executed before each test. Use @BeforeClass if you want to
execute a code block just once.
+ @Before
+ public void setUp() throws Exception {
+//dataStores.add("mongodb");
+//dataStores.add("hbase");
+//bmutils.generateAvroSchema(NUMBER_OF_FIELDS);
+//for(String dataStore: dataStores) {
+ //bmutils.generateMappingFile(dataStore);
+//}
+//bmutils.generateDataBeans();
+DATA_TO_INSERT = new HashMap<>();
+DATA_TO_UPDATE = new HashMap<>();
+for(int i=0; i < NUMBER_OF_FIELDS; i++) {
+ DATA_TO_INSERT.put("field"+i, new StringByteIterator("value"+i));
+ DATA_TO_UPDATE.put("field"+i, new StringByteIterator("updated"+i));
+}
+Properties p = new Properties();
+p.setProperty("key.class", "java.lang.String");
+p.setProperty("persistent.class", "generated.User");
+p.setProperty(CoreWorkload.FIELD_COUNT_PROPERTY, NUMBER_OF_FIELDS+"");
+client = new GoraBenchmarkClient();
+client.setProperties(p);
+client.init();
+ }
+
+ /**
+ * Clean up.
+ *
+ * @throws Exception the exception
+ */
+ @After
+ public void cleanUp() throws Exception{
+if(client != null)
+ client.cleanup();
+client = null;
+ }
+
+ /**
+ * Read record.
+ *
+ * @param key the key
+ * @return the user
+ * @throws GoraException the gora exception
+ */
+ private User readRecord(String key) throws GoraException {
+User u = client.dataStore.get(key);
+return u;
+ }
+
+ /**
+ * Test client initialisation.
+ */
+ @Test
+ public void testClientInitialisation() {
+assertNotNull(client.dataStore);
+ }
+
+
+ /**
+ * Test insert.
+ *
+ * @throws GoraException the gora exception
+ */
+ @Test
+ public void testInsert() throws GoraException{
+int result1 = client.insert(TABLE, "key1", DATA_TO_INSERT);
+int result2 = client.insert(TABLE, "key2", DATA_TO_INSERT);
+int result3 = client.insert(TABLE, "key3", DATA_TO_INSERT);
+assertEquals(0, result1);
+assertEquals(0, result2);
+assertEquals(0, result3);
+ }
+
+
+ /**
+ * Test read.
+ */
+ @Test
+ public void testRead() {
+HashMap results = new HashMap<>();
+Set fields = new HashSet<>();//this could be null as well
+//fields.add("field0");
+int result = client.read(TABLE, "key1", fields, results);
+assertEquals(0, result);
+assertEquals(DATA_TO_INSERT.size(), results.size());
+assertEquals(DATA_TO_INSERT.get("field0").toString(),
results.get("field0").toString());
+assertEquals(DATA_TO_INSERT.get("field0").toString(), "value0");
+ }
+
+ /**
+ * Test scan.
+ */
+ @Test
+ public void testScan(){
+Vector> results = new Vector>();
+Set fields = new HashSet<>();
+//fields.add("field0");
Review comment:
Remove
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845310
##
File path: gora-benchmark/runWorkloads.sh
##
@@ -0,0 +1,119 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Check if number of commanline arguments is right
+threadcount=15
+if [ $# -ne 3 ]
+then
+echo "Error: Usage: $0 hbase|mongodb gora|ycsb workloadfile"
+exit;
+fi
+#Log output file
+outputfile="$3-$2-$1.log"
+insertfactor=500
+if [ $2 = "ycsb" ]
+then
+ table="usertable"
+elif [ $2 == "gora" ]
+then
+ table="users"
+fi
+
+if [ $1 = "hbase" ]
+then
+ #Read operation
+ echo -e "Disabling Users Table" | tee -a $outputfile
+ echo -e "disable '$table'" | hbase shell -n
+ echo -e "Dropping Users Table" | tee -a $outputfile
+ echo -e "drop '$table'" | hbase shell -n
+ echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ fi
+for i in {10..500..10}
+do
+ #Insert only workload, uncomment this block of code. I will improve this
later.
+ #echo -e "Disabling Users Table" | tee -a $outputfile
+ #echo -e "disable '$table'" | hbase shell -n
+ #echo -e "Dropping Users Table" | tee -a $outputfile
+ #echo -e "drop '$table'" | hbase shell -n
+ #echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ #echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+
+ echo -e "Table Setup Completed = Now Running Benchmark
for $i records and operations" | tee -a $outputfile
+ echo -e "Record Count $i" | tee -a $outputfile
+ echo -e "Operation Count $i" | tee -a $outputfile
+ echo -e "Thread Count $threadcount" | tee -a $outputfile
+ echo -e ""
+ if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ ./gora-bench.sh run -threads 15 -s -p readallfields=true -p
measurementtype=timeseries -p timeseries.granularity=2000 -p operationcount=$i
-P workloads/$3 | tee -a $outputfile
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ ../../ycsb/bin/ycsb.sh run hbase20 -s -p columnfamily=info -p
measurementtype=timeseries -p timeseries.granularity=2000 -threads 15 -p
readallfields=true -p operationcount=$i -P workloads/$3 | tee -a $outputfile
+ fi
+ echo "End of
Benchmark" | tee -a $outputfile
+done
+elif [ $1 = "mongodb" ]
+then
+ echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ if [ $2 = "ycsb" ]
+ then
+ mongo < setupycsbmongo.js
+ ../../ycsb/bin/ycsb.sh load mongodb -threads 15 -s -p fieldcount=20
-p recordcount=$insertfactor -p operationcount=$insertfactor -P
workloads/workloada
+ fi
+
+ if [ $2 = "gora" ]
+ then
+ mongo < setupgoramongo.js
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$insertfactor -P workloads/workloada
+ fi
+ echo -e "Table Setup Completed = Now Running
Benchmark for $i records and operations" | tee -a $outputfile
+for i in {10..500..10}
+do
+ #echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ #mongo < setupmongo.js
+ #echo -e "Table Setup Completed
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845103
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
+
+
+
+ org.apache.gora
+ gora-hbase
+
+
+
+ org.apache.gora
+ gora-jcache
+
+
+
+ org.apache.gora
+ gora-couchdb
+
+
+
+ org.apache.gora
+ gora-cassandra
+
+
+
+ org.apache.gora
+ gora-solr
+
+
+
+ org.apache.gora
+ gora-aerospike
+
+
+
+ org.apache.avro
+ avro
+ compile
+
+
+
+ org.apache.gora
+ gora-core
+ test-jar
+ test
+
+
+
+ gora-compiler
+ org.apache.gora
+
+
+
+
+ org.slf4j
+ slf4j-log4j12
+
+
+ log4j
+ log4j
+
+
+ javax.jms
+ jms
+
+
+
+
+
+
+
+
+ org.apache.hadoop
+ hadoop-client
+ compile
+
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845323
##
File path: gora-benchmark/runWorkloads.sh
##
@@ -0,0 +1,119 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Check if number of commanline arguments is right
+threadcount=15
+if [ $# -ne 3 ]
+then
+echo "Error: Usage: $0 hbase|mongodb gora|ycsb workloadfile"
+exit;
+fi
+#Log output file
+outputfile="$3-$2-$1.log"
+insertfactor=500
+if [ $2 = "ycsb" ]
+then
+ table="usertable"
+elif [ $2 == "gora" ]
+then
+ table="users"
+fi
+
+if [ $1 = "hbase" ]
+then
+ #Read operation
+ echo -e "Disabling Users Table" | tee -a $outputfile
+ echo -e "disable '$table'" | hbase shell -n
+ echo -e "Dropping Users Table" | tee -a $outputfile
+ echo -e "drop '$table'" | hbase shell -n
+ echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ fi
+for i in {10..500..10}
+do
+ #Insert only workload, uncomment this block of code. I will improve this
later.
+ #echo -e "Disabling Users Table" | tee -a $outputfile
+ #echo -e "disable '$table'" | hbase shell -n
+ #echo -e "Dropping Users Table" | tee -a $outputfile
+ #echo -e "drop '$table'" | hbase shell -n
+ #echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ #echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+
+ echo -e "Table Setup Completed = Now Running Benchmark
for $i records and operations" | tee -a $outputfile
+ echo -e "Record Count $i" | tee -a $outputfile
+ echo -e "Operation Count $i" | tee -a $outputfile
+ echo -e "Thread Count $threadcount" | tee -a $outputfile
+ echo -e ""
+ if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ ./gora-bench.sh run -threads 15 -s -p readallfields=true -p
measurementtype=timeseries -p timeseries.granularity=2000 -p operationcount=$i
-P workloads/$3 | tee -a $outputfile
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ ../../ycsb/bin/ycsb.sh run hbase20 -s -p columnfamily=info -p
measurementtype=timeseries -p timeseries.granularity=2000 -threads 15 -p
readallfields=true -p operationcount=$i -P workloads/$3 | tee -a $outputfile
+ fi
+ echo "End of
Benchmark" | tee -a $outputfile
+done
+elif [ $1 = "mongodb" ]
+then
+ echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ if [ $2 = "ycsb" ]
+ then
+ mongo < setupycsbmongo.js
+ ../../ycsb/bin/ycsb.sh load mongodb -threads 15 -s -p fieldcount=20
-p recordcount=$insertfactor -p operationcount=$insertfactor -P
workloads/workloada
+ fi
+
+ if [ $2 = "gora" ]
+ then
+ mongo < setupgoramongo.js
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$insertfactor -P workloads/workloada
+ fi
+ echo -e "Table Setup Completed = Now Running
Benchmark for $i records and operations" | tee -a $outputfile
+for i in {10..500..10}
+do
+ #echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ #mongo < setupmongo.js
+ #echo -e "Table Setup Completed
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310846823 ## File path: gora-benchmark/src/main/resources/gora-hbase-mapping.xml ## @@ -0,0 +1,59 @@ + Review comment: License header This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845293
##
File path: gora-benchmark/runWorkloads.sh
##
@@ -0,0 +1,119 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Check if number of commanline arguments is right
+threadcount=15
+if [ $# -ne 3 ]
+then
+echo "Error: Usage: $0 hbase|mongodb gora|ycsb workloadfile"
+exit;
+fi
+#Log output file
+outputfile="$3-$2-$1.log"
+insertfactor=500
+if [ $2 = "ycsb" ]
+then
+ table="usertable"
+elif [ $2 == "gora" ]
+then
+ table="users"
+fi
+
+if [ $1 = "hbase" ]
+then
+ #Read operation
+ echo -e "Disabling Users Table" | tee -a $outputfile
+ echo -e "disable '$table'" | hbase shell -n
+ echo -e "Dropping Users Table" | tee -a $outputfile
+ echo -e "drop '$table'" | hbase shell -n
+ echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ fi
+for i in {10..500..10}
+do
+ #Insert only workload, uncomment this block of code. I will improve this
later.
+ #echo -e "Disabling Users Table" | tee -a $outputfile
+ #echo -e "disable '$table'" | hbase shell -n
+ #echo -e "Dropping Users Table" | tee -a $outputfile
+ #echo -e "drop '$table'" | hbase shell -n
+ #echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ #echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+
+ echo -e "Table Setup Completed = Now Running Benchmark
for $i records and operations" | tee -a $outputfile
+ echo -e "Record Count $i" | tee -a $outputfile
+ echo -e "Operation Count $i" | tee -a $outputfile
+ echo -e "Thread Count $threadcount" | tee -a $outputfile
+ echo -e ""
+ if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ ./gora-bench.sh run -threads 15 -s -p readallfields=true -p
measurementtype=timeseries -p timeseries.granularity=2000 -p operationcount=$i
-P workloads/$3 | tee -a $outputfile
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ ../../ycsb/bin/ycsb.sh run hbase20 -s -p columnfamily=info -p
measurementtype=timeseries -p timeseries.granularity=2000 -threads 15 -p
readallfields=true -p operationcount=$i -P workloads/$3 | tee -a $outputfile
+ fi
+ echo "End of
Benchmark" | tee -a $outputfile
+done
+elif [ $1 = "mongodb" ]
+then
+ echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ if [ $2 = "ycsb" ]
+ then
+ mongo < setupycsbmongo.js
+ ../../ycsb/bin/ycsb.sh load mongodb -threads 15 -s -p fieldcount=20
-p recordcount=$insertfactor -p operationcount=$insertfactor -P
workloads/workloada
+ fi
+
+ if [ $2 = "gora" ]
+ then
+ mongo < setupgoramongo.js
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$insertfactor -P workloads/workloada
+ fi
+ echo -e "Table Setup Completed = Now Running
Benchmark for $i records and operations" | tee -a $outputfile
+for i in {10..500..10}
+do
+ #echo -e "Dropping $table from MongoDB" | tee -a $outputfile
Review comment:
Remove code not being used
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846412
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
+
+ /**
+ * Checks if is field updatable.
+ *
+ * @param field the field
+ * @param values the values
+ * @return true, if is field updatable
+ */
+ public static boolean isFieldUpdatable(String field, HashMap values) {
+if (values.get(field) == null) {
+ return false;
+}
+return true;
+ }
+
+ /**
+ * Generate avro schema based on the number of fields. supplied when running
the benchmark. These files are json files
+ *
+ * @param numberOfFields the number of fields
+ */
+ public void generateAvroSchema(int numberOfFields) {
+try {
+ File avroFile = new File(AVRO_FULL_PATH);
+ FieldAssembler fieldAssembler =
SchemaBuilder.record(RECORD).namespace(NAMESPACE_VALUE).fields();
+
fieldAssembler.name(USER_ID_VALUE).type().stringType().stringDefault(NULL);
+ for (int i = 0; i < numberOfFields; i++) {
+fieldAssembler.name(FIELD_PREFIX +
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846365
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
Review comment:
Remove whitespace
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846388
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
Review comment:
Remove whitespace
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846477
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
+
+ /**
+ * Checks if is field updatable.
+ *
+ * @param field the field
+ * @param values the values
+ * @return true, if is field updatable
+ */
+ public static boolean isFieldUpdatable(String field, HashMap values) {
+if (values.get(field) == null) {
+ return false;
+}
+return true;
+ }
+
+ /**
+ * Generate avro schema based on the number of fields. supplied when running
the benchmark. These files are json files
+ *
+ * @param numberOfFields the number of fields
+ */
+ public void generateAvroSchema(int numberOfFields) {
+try {
+ File avroFile = new File(AVRO_FULL_PATH);
+ FieldAssembler fieldAssembler =
SchemaBuilder.record(RECORD).namespace(NAMESPACE_VALUE).fields();
+
fieldAssembler.name(USER_ID_VALUE).type().stringType().stringDefault(NULL);
+ for (int i = 0; i < numberOfFields; i++) {
+fieldAssembler.name(FIELD_PREFIX +
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846109
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+
+/**
+ * The Class GoraBenchmarkClient.
+ *
+ * @author sc306
+ * This class extends the Yahoo! Cloud Service Benchmark benchmark {@link
#com.yahoo.ycsb.DB DB} class to provide functionality
+ * for {@link #insert(String, String, HashMap) insert}, {@link #read(String,
String, Set, HashMap) read}, {@link #scan(String, String, int, Set, Vector)
scan}
+ * and {@link #update(String, String, HashMap) update} methods as per Apache
Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+
+ private static final int SUCCESS = 0;
+
+ private static final int FAILED = 1;
+
+ private static final String FIELDS[] = User._ALL_FIELDS;
+
+ private static volatile boolean executed;
+
+ public static int fieldCount;
+
+ /**This is only for set to array conversion in {@link read()} method*/
+ private String[] DUMMY_ARRAY = new String[0];
+
+ DataStore dataStore;
+
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+
+ User user = new User();
+
+ private Properties prop;
+
+ public GoraBenchmarkClient() {}
+
+
+ /***
+ * Initialisation method. This method is called once for each database
instance.
+ */
+ public void init() throws DBException {
+try {
+
+//Get YCSB properties
+prop = getProperties();
+
+fieldCount =
Integer.parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+
+String keyClass = prop.getProperty("key.class", "java.lang.String");
+String persistentClass = prop.getProperty("persistent.class",
"generated.User");
+Properties p = DataStoreFactory.createProps();
+dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass,
p, new Configuration());
+synchronized(GoraBenchmarkClient.class) {
+ if(executed)
+return;
+ executed = true;
+ goraBenchmarkUtils.generateAvroSchema(fieldCount);
+ String dataStoreName = goraBenchmarkUtils.getDataStore(p);
+ goraBenchmarkUtils.generateMappingFile(dataStoreName);
+ goraBenchmarkUtils.generateDataBeans();
+}
+
+
+} catch (GoraException e) {
+ //e.printStackTrace();
+}
+ }
+
+ /**
+ * Cleanup any state for this DB.
+ *
+ * It is very important to close the datastore properly, otherwise some data
+ * loss might occur.
+ */
+ public void cleanup() throws DBException {
+synchronized (GoraBenchmarkClient.class) {
+ if (dataStore != null)
+dataStore.close();
+}
+
+ }
+
+ /**
+ * Delete a record from the database.
+ *
+ * @param table The name of the table to delete the data from
+ * @param key The key of the record to delete.
+ * @return Status of the operation failed or success.
+ */
+ @Override
+ public int delete(String table, String key) {
+try {
+ //LOG.info("Deleting Key "+key+" From Table "+table);
+ dataStore.delete(key);
+
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310846254 ## File path: gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java ## @@ -0,0 +1,453 @@ +package org.apache.gora.benchmark; Review comment: License header This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846121
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+
+/**
+ * The Class GoraBenchmarkClient.
+ *
+ * @author sc306
+ * This class extends the Yahoo! Cloud Service Benchmark benchmark {@link
#com.yahoo.ycsb.DB DB} class to provide functionality
+ * for {@link #insert(String, String, HashMap) insert}, {@link #read(String,
String, Set, HashMap) read}, {@link #scan(String, String, int, Set, Vector)
scan}
+ * and {@link #update(String, String, HashMap) update} methods as per Apache
Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+
+ private static final int SUCCESS = 0;
+
+ private static final int FAILED = 1;
+
+ private static final String FIELDS[] = User._ALL_FIELDS;
+
+ private static volatile boolean executed;
+
+ public static int fieldCount;
+
+ /**This is only for set to array conversion in {@link read()} method*/
+ private String[] DUMMY_ARRAY = new String[0];
+
+ DataStore dataStore;
+
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+
+ User user = new User();
+
+ private Properties prop;
+
+ public GoraBenchmarkClient() {}
+
+
+ /***
+ * Initialisation method. This method is called once for each database
instance.
+ */
+ public void init() throws DBException {
+try {
+
+//Get YCSB properties
+prop = getProperties();
+
+fieldCount =
Integer.parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+
+String keyClass = prop.getProperty("key.class", "java.lang.String");
+String persistentClass = prop.getProperty("persistent.class",
"generated.User");
+Properties p = DataStoreFactory.createProps();
+dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass,
p, new Configuration());
+synchronized(GoraBenchmarkClient.class) {
+ if(executed)
+return;
+ executed = true;
+ goraBenchmarkUtils.generateAvroSchema(fieldCount);
+ String dataStoreName = goraBenchmarkUtils.getDataStore(p);
+ goraBenchmarkUtils.generateMappingFile(dataStoreName);
+ goraBenchmarkUtils.generateDataBeans();
+}
+
+
+} catch (GoraException e) {
+ //e.printStackTrace();
+}
+ }
+
+ /**
+ * Cleanup any state for this DB.
+ *
+ * It is very important to close the datastore properly, otherwise some data
+ * loss might occur.
+ */
+ public void cleanup() throws DBException {
+synchronized (GoraBenchmarkClient.class) {
+ if (dataStore != null)
+dataStore.close();
+}
+
+ }
+
+ /**
+ * Delete a record from the database.
+ *
+ * @param table The name of the table to delete the data from
+ * @param key The key of the record to delete.
+ * @return Status of the operation failed or success.
+ */
+ @Override
+ public int delete(String table, String key) {
+try {
+ //LOG.info("Deleting Key "+key+" From Table "+table);
+ dataStore.delete(key);
+
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846439
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
+
+ /**
+ * Checks if is field updatable.
+ *
+ * @param field the field
+ * @param values the values
+ * @return true, if is field updatable
+ */
+ public static boolean isFieldUpdatable(String field, HashMap values) {
+if (values.get(field) == null) {
+ return false;
+}
+return true;
+ }
+
+ /**
+ * Generate avro schema based on the number of fields. supplied when running
the benchmark. These files are json files
+ *
+ * @param numberOfFields the number of fields
+ */
+ public void generateAvroSchema(int numberOfFields) {
+try {
+ File avroFile = new File(AVRO_FULL_PATH);
+ FieldAssembler fieldAssembler =
SchemaBuilder.record(RECORD).namespace(NAMESPACE_VALUE).fields();
+
fieldAssembler.name(USER_ID_VALUE).type().stringType().stringDefault(NULL);
+ for (int i = 0; i < numberOfFields; i++) {
+fieldAssembler.name(FIELD_PREFIX +
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310847005
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,201 @@
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
+
+
+ /** The Constant TABLE. */
+ private static final String TABLE = "users";
+
+ /** The client. */
+ private GoraBenchmarkClient client;
+
+ /** The data to insert. */
+ private static HashMap DATA_TO_INSERT;
+
+ /** The data to update. */
+ private static HashMap DATA_TO_UPDATE;
+
+ /** The data stores. */
+ private static List dataStores = new ArrayList();
+
+ /** The Constant NUMBER_OF_FIELDS. */
+ private static final int NUMBER_OF_FIELDS = 10;
+
+ /** The bmutils. */
+ private GoraBenchmarkUtils bmutils = new GoraBenchmarkUtils();
+
+ /**
+ * Sets the up.
+ *
+ * @throws Exception the exception
+ */
+ //Setup is executed before each test. Use @BeforeClass if you want to
execute a code block just once.
+ @Before
+ public void setUp() throws Exception {
+//dataStores.add("mongodb");
+//dataStores.add("hbase");
+//bmutils.generateAvroSchema(NUMBER_OF_FIELDS);
+//for(String dataStore: dataStores) {
+ //bmutils.generateMappingFile(dataStore);
+//}
+//bmutils.generateDataBeans();
+DATA_TO_INSERT = new HashMap<>();
+DATA_TO_UPDATE = new HashMap<>();
+for(int i=0; i < NUMBER_OF_FIELDS; i++) {
+ DATA_TO_INSERT.put("field"+i, new StringByteIterator("value"+i));
+ DATA_TO_UPDATE.put("field"+i, new StringByteIterator("updated"+i));
+}
+Properties p = new Properties();
+p.setProperty("key.class", "java.lang.String");
+p.setProperty("persistent.class", "generated.User");
+p.setProperty(CoreWorkload.FIELD_COUNT_PROPERTY, NUMBER_OF_FIELDS+"");
+client = new GoraBenchmarkClient();
+client.setProperties(p);
+client.init();
+ }
+
+ /**
+ * Clean up.
+ *
+ * @throws Exception the exception
+ */
+ @After
+ public void cleanUp() throws Exception{
+if(client != null)
+ client.cleanup();
+client = null;
+ }
+
+ /**
+ * Read record.
+ *
+ * @param key the key
+ * @return the user
+ * @throws GoraException the gora exception
+ */
+ private User readRecord(String key) throws GoraException {
+User u = client.dataStore.get(key);
+return u;
+ }
+
+ /**
+ * Test client initialisation.
+ */
+ @Test
+ public void testClientInitialisation() {
+assertNotNull(client.dataStore);
+ }
+
+
+ /**
+ * Test insert.
+ *
+ * @throws GoraException the gora exception
+ */
+ @Test
+ public void testInsert() throws GoraException{
+int result1 = client.insert(TABLE, "key1", DATA_TO_INSERT);
+int result2 = client.insert(TABLE, "key2", DATA_TO_INSERT);
+int result3 = client.insert(TABLE, "key3", DATA_TO_INSERT);
+assertEquals(0, result1);
+assertEquals(0, result2);
+assertEquals(0, result3);
+ }
+
+
+ /**
+ * Test read.
+ */
+ @Test
+ public void testRead() {
+HashMap results = new HashMap<>();
+Set fields = new HashSet<>();//this could be null as well
+//fields.add("field0");
+int result = client.read(TABLE, "key1", fields, results);
+assertEquals(0, result);
+assertEquals(DATA_TO_INSERT.size(), results.size());
+assertEquals(DATA_TO_INSERT.get("field0").toString(),
results.get("field0").toString());
+assertEquals(DATA_TO_INSERT.get("field0").toString(), "value0");
+ }
+
+ /**
+ * Test scan.
+ */
+ @Test
+ public void testScan(){
+Vector> results = new Vector>();
+Set fields = new HashSet<>();
+//fields.add("field0");
+int result = client.scan(TABLE, "key1", 2, fields, results);
+assertEquals(result,0);
+assertEquals(results.size(),2);
+ }
+
+ /**
+ * Test update.
+ *
+ * @throws GoraException the gora exception
+ */
+ @Test
+ public void testUpdate() throws GoraException{
+int result = client.update(TABLE, "key1", DATA_TO_UPDATE);
+assertEquals(result,0);
+if(result==0) {
+ client.dataStore.flush();
+ User u = readRecord("key1");
+ assertEquals("updated0",
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846220
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+
+/**
+ * The Class GoraBenchmarkClient.
+ *
+ * @author sc306
+ * This class extends the Yahoo! Cloud Service Benchmark benchmark {@link
#com.yahoo.ycsb.DB DB} class to provide functionality
+ * for {@link #insert(String, String, HashMap) insert}, {@link #read(String,
String, Set, HashMap) read}, {@link #scan(String, String, int, Set, Vector)
scan}
+ * and {@link #update(String, String, HashMap) update} methods as per Apache
Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+
+ private static final int SUCCESS = 0;
+
+ private static final int FAILED = 1;
+
+ private static final String FIELDS[] = User._ALL_FIELDS;
+
+ private static volatile boolean executed;
+
+ public static int fieldCount;
+
+ /**This is only for set to array conversion in {@link read()} method*/
+ private String[] DUMMY_ARRAY = new String[0];
+
+ DataStore dataStore;
+
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+
+ User user = new User();
+
+ private Properties prop;
+
+ public GoraBenchmarkClient() {}
+
+
+ /***
+ * Initialisation method. This method is called once for each database
instance.
+ */
+ public void init() throws DBException {
+try {
+
+//Get YCSB properties
+prop = getProperties();
+
+fieldCount =
Integer.parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+
+String keyClass = prop.getProperty("key.class", "java.lang.String");
+String persistentClass = prop.getProperty("persistent.class",
"generated.User");
+Properties p = DataStoreFactory.createProps();
+dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass,
p, new Configuration());
+synchronized(GoraBenchmarkClient.class) {
+ if(executed)
+return;
+ executed = true;
+ goraBenchmarkUtils.generateAvroSchema(fieldCount);
+ String dataStoreName = goraBenchmarkUtils.getDataStore(p);
+ goraBenchmarkUtils.generateMappingFile(dataStoreName);
+ goraBenchmarkUtils.generateDataBeans();
+}
+
+
+} catch (GoraException e) {
+ //e.printStackTrace();
+}
+ }
+
+ /**
+ * Cleanup any state for this DB.
+ *
+ * It is very important to close the datastore properly, otherwise some data
+ * loss might occur.
+ */
+ public void cleanup() throws DBException {
+synchronized (GoraBenchmarkClient.class) {
+ if (dataStore != null)
+dataStore.close();
+}
+
+ }
+
+ /**
+ * Delete a record from the database.
+ *
+ * @param table The name of the table to delete the data from
+ * @param key The key of the record to delete.
+ * @return Status of the operation failed or success.
+ */
+ @Override
+ public int delete(String table, String key) {
+try {
+ //LOG.info("Deleting Key "+key+" From Table "+table);
+ dataStore.delete(key);
+
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310845362 ## File path: gora-benchmark/setupgoramongo.js ## @@ -0,0 +1,2 @@ +use gorabm111 Review comment: Add ALv2.0 license header This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845670
##
File path: gora-benchmark/src/main/avro/user.json
##
@@ -0,0 +1,112 @@
+{
+"name": "User",
+"namespace": "generated",
+"type": "record",
+"fields": [
+{
Review comment:
Please format all of your JSON the same. either flatten it (as in
person.json) or expand it as here. Keep it the same regardless which option you
use.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310844596
##
File path: gora-benchmark/gora-bench.sh
##
@@ -0,0 +1,80 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Adapted from YCSB's version of ycsb.sh.
+CLASSPATH="."
+
+# Attempt to find the available JAVA, if JAVA_HOME not set
+if [ -z "$JAVA_HOME" ]; then
+ JAVA_PATH=$(which java 2>/dev/null)
+ if [ "x$JAVA_PATH" != "x" ]; then
+JAVA_HOME=$(dirname "$(dirname "$JAVA_PATH" 2>/dev/null)")
+ fi
+fi
+
+# If JAVA_HOME still not set, error
+if [ -z "$JAVA_HOME" ]; then
+ echo "[ERROR] Java executable not found. Exiting."
+ exit 1;
+fi
+
+# Determine YCSB command argument
+if [ "load" = "$1" ] ; then
+ YCSB_COMMAND=-load
+ YCSB_CLASS=com.yahoo.ycsb.Client
+ DB_CLASS=org.apache.gora.benchmark.GoraBenchmarkClient
+elif [ "run" = "$1" ] ; then
+ YCSB_COMMAND=-t
+ YCSB_CLASS=com.yahoo.ycsb.Client
+ DB_CLASS=org.apache.gora.benchmark.GoraBenchmarkClient
+else
+ echo "[ERROR] Found unknown command '$1'"
+ echo "[ERROR] Expected one of 'load', 'run', or 'shell'. Exiting."
+ exit 1;
+fi
+
+SAVEIFS=$IFS
+IFS=$(echo -en "\n\b")
+
+# add libs to CLASSPATH
+for f in target/*.jar; do
+ CLASSPATH=${CLASSPATH}:$f;
+done
+
+for f in lib/*.jar; do
+ CLASSPATH=${CLASSPATH}:$f;
+done
+
+IFS=$SAVEIFS
+
+CLASSPATH=${CLASSPATH}:target/classes/
+CLASSPATH=${CLASSPATH}:target/test-classes/
+
+#CLASSPATH=${CLASSPATH}:conf
+#CLASSPATH=${CLASSPATH}:conf
+
+# Get the rest of the arguments
+YCSB_ARGS=$(echo "$@" | cut -d' ' -f2-)
+
+#echo $YCSB_ARGS
Review comment:
Remove
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310846846 ## File path: gora-benchmark/src/main/resources/gora-mongodb-mapping.xml ## @@ -0,0 +1,26 @@ + Review comment: License header This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845901
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+
+/**
+ * The Class GoraBenchmarkClient.
+ *
+ * @author sc306
+ * This class extends the Yahoo! Cloud Service Benchmark benchmark {@link
#com.yahoo.ycsb.DB DB} class to provide functionality
+ * for {@link #insert(String, String, HashMap) insert}, {@link #read(String,
String, Set, HashMap) read}, {@link #scan(String, String, int, Set, Vector)
scan}
+ * and {@link #update(String, String, HashMap) update} methods as per Apache
Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+
+ private static final int SUCCESS = 0;
+
+ private static final int FAILED = 1;
+
+ private static final String FIELDS[] = User._ALL_FIELDS;
+
+ private static volatile boolean executed;
+
+ public static int fieldCount;
+
+ /**This is only for set to array conversion in {@link read()} method*/
+ private String[] DUMMY_ARRAY = new String[0];
+
+ DataStore dataStore;
+
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+
+ User user = new User();
+
+ private Properties prop;
+
+ public GoraBenchmarkClient() {}
+
+
+ /***
+ * Initialisation method. This method is called once for each database
instance.
+ */
+ public void init() throws DBException {
+try {
+
+//Get YCSB properties
+prop = getProperties();
Review comment:
Please format all Java code at 2 space indents.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846020
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+
+/**
+ * The Class GoraBenchmarkClient.
+ *
+ * @author sc306
+ * This class extends the Yahoo! Cloud Service Benchmark benchmark {@link
#com.yahoo.ycsb.DB DB} class to provide functionality
+ * for {@link #insert(String, String, HashMap) insert}, {@link #read(String,
String, Set, HashMap) read}, {@link #scan(String, String, int, Set, Vector)
scan}
+ * and {@link #update(String, String, HashMap) update} methods as per Apache
Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+
+ private static final int SUCCESS = 0;
+
+ private static final int FAILED = 1;
+
+ private static final String FIELDS[] = User._ALL_FIELDS;
+
+ private static volatile boolean executed;
+
+ public static int fieldCount;
+
+ /**This is only for set to array conversion in {@link read()} method*/
+ private String[] DUMMY_ARRAY = new String[0];
+
+ DataStore dataStore;
+
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+
+ User user = new User();
+
+ private Properties prop;
+
+ public GoraBenchmarkClient() {}
+
+
+ /***
+ * Initialisation method. This method is called once for each database
instance.
+ */
+ public void init() throws DBException {
+try {
+
+//Get YCSB properties
+prop = getProperties();
+
+fieldCount =
Integer.parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+
+String keyClass = prop.getProperty("key.class", "java.lang.String");
+String persistentClass = prop.getProperty("persistent.class",
"generated.User");
Review comment:
The default property value will need to change if the generated Java code
package name changes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846086
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+
+/**
+ * The Class GoraBenchmarkClient.
+ *
+ * @author sc306
+ * This class extends the Yahoo! Cloud Service Benchmark benchmark {@link
#com.yahoo.ycsb.DB DB} class to provide functionality
+ * for {@link #insert(String, String, HashMap) insert}, {@link #read(String,
String, Set, HashMap) read}, {@link #scan(String, String, int, Set, Vector)
scan}
+ * and {@link #update(String, String, HashMap) update} methods as per Apache
Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+
+ private static final int SUCCESS = 0;
+
+ private static final int FAILED = 1;
+
+ private static final String FIELDS[] = User._ALL_FIELDS;
+
+ private static volatile boolean executed;
+
+ public static int fieldCount;
+
+ /**This is only for set to array conversion in {@link read()} method*/
+ private String[] DUMMY_ARRAY = new String[0];
+
+ DataStore dataStore;
+
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+
+ User user = new User();
+
+ private Properties prop;
+
+ public GoraBenchmarkClient() {}
+
+
+ /***
+ * Initialisation method. This method is called once for each database
instance.
+ */
+ public void init() throws DBException {
+try {
+
+//Get YCSB properties
+prop = getProperties();
+
+fieldCount =
Integer.parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+
+String keyClass = prop.getProperty("key.class", "java.lang.String");
+String persistentClass = prop.getProperty("persistent.class",
"generated.User");
+Properties p = DataStoreFactory.createProps();
+dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass,
p, new Configuration());
+synchronized(GoraBenchmarkClient.class) {
+ if(executed)
+return;
+ executed = true;
+ goraBenchmarkUtils.generateAvroSchema(fieldCount);
+ String dataStoreName = goraBenchmarkUtils.getDataStore(p);
+ goraBenchmarkUtils.generateMappingFile(dataStoreName);
+ goraBenchmarkUtils.generateDataBeans();
+}
+
+
+} catch (GoraException e) {
+ //e.printStackTrace();
+}
+ }
+
+ /**
+ * Cleanup any state for this DB.
+ *
+ * It is very important to close the datastore properly, otherwise some data
+ * loss might occur.
+ */
+ public void cleanup() throws DBException {
+synchronized (GoraBenchmarkClient.class) {
+ if (dataStore != null)
+dataStore.close();
+}
+
+ }
+
+ /**
+ * Delete a record from the database.
+ *
+ * @param table The name of the table to delete the data from
+ * @param key The key of the record to delete.
+ * @return Status of the operation failed or success.
+ */
+ @Override
+ public int delete(String table, String key) {
+try {
+ //LOG.info("Deleting Key "+key+" From Table "+table);
Review comment:
Delete
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310846321 ## File path: gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java ## @@ -0,0 +1,453 @@ +package org.apache.gora.benchmark; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.StringWriter; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.HashMap; +import java.util.Iterator; +import java.util.Locale; +import java.util.Properties; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; +import javax.xml.parsers.ParserConfigurationException; +import javax.xml.transform.OutputKeys; +import javax.xml.transform.Transformer; +import javax.xml.transform.TransformerConfigurationException; +import javax.xml.transform.TransformerException; +import javax.xml.transform.TransformerFactory; +import javax.xml.transform.dom.DOMSource; +import javax.xml.transform.stream.StreamResult; + +import org.apache.avro.Schema; +import org.apache.avro.SchemaBuilder; +import org.apache.avro.SchemaBuilder.FieldAssembler; +import org.apache.gora.compiler.GoraCompiler; +import org.apache.gora.compiler.utils.LicenseHeaders; +import org.json.JSONArray; +import org.json.JSONObject; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.w3c.dom.Attr; +import org.w3c.dom.Document; +import org.w3c.dom.Element; + +import com.yahoo.ycsb.ByteIterator; + +// TODO: Auto-generated Javadoc +/** + * The Class GoraBenchmarkUtils. Review comment: Please describe what the class does and how it is run. What input is expected and what output is expected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310845373 ## File path: gora-benchmark/setupycsbmongo.js ## @@ -0,0 +1,2 @@ +use ycsb Review comment: Add ALv2.0 license header This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310845781 ## File path: gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java ## @@ -0,0 +1,273 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.gora.benchmark; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Properties; +import java.util.Set; +import java.util.Vector; + +import org.apache.gora.query.Query; +import org.apache.gora.query.Result; +import org.apache.gora.store.DataStore; +import org.apache.gora.store.DataStoreFactory; +import org.apache.gora.util.GoraException; +import org.apache.hadoop.conf.Configuration; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.yahoo.ycsb.ByteIterator; +import com.yahoo.ycsb.DB; +import com.yahoo.ycsb.DBException; +import com.yahoo.ycsb.StringByteIterator; +import com.yahoo.ycsb.workloads.CoreWorkload; + +import generated.User; Review comment: Please rename the package to org.apache.gora.benchmark.generated.User This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845525
##
File path: gora-benchmark/src/main/avro/person.json
##
@@ -0,0 +1,18 @@
+ {
+ "type": "record",
+ "name": "Person", "default":null,
+ "namespace": "generated",
+ "fields" : [
+{"name": "userId", "type": ["null","string"], "default":null},
Review comment:
These are pretty simple objects... I wonder if creating more complex objects
would result in 'better' tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846964
##
File path:
gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java
##
@@ -0,0 +1,201 @@
+package org.apache.gora.benchmark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraClientTest.
+ */
+public class GoraClientTest {
+
+
+ /** The Constant TABLE. */
+ private static final String TABLE = "users";
+
+ /** The client. */
+ private GoraBenchmarkClient client;
+
+ /** The data to insert. */
+ private static HashMap DATA_TO_INSERT;
+
+ /** The data to update. */
+ private static HashMap DATA_TO_UPDATE;
+
+ /** The data stores. */
+ private static List dataStores = new ArrayList();
+
+ /** The Constant NUMBER_OF_FIELDS. */
+ private static final int NUMBER_OF_FIELDS = 10;
+
+ /** The bmutils. */
+ private GoraBenchmarkUtils bmutils = new GoraBenchmarkUtils();
+
+ /**
+ * Sets the up.
+ *
+ * @throws Exception the exception
+ */
+ //Setup is executed before each test. Use @BeforeClass if you want to
execute a code block just once.
+ @Before
+ public void setUp() throws Exception {
+//dataStores.add("mongodb");
+//dataStores.add("hbase");
+//bmutils.generateAvroSchema(NUMBER_OF_FIELDS);
+//for(String dataStore: dataStores) {
Review comment:
Remove all commented-out code
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846426
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
+
+ /**
+ * Checks if is field updatable.
+ *
+ * @param field the field
+ * @param values the values
+ * @return true, if is field updatable
+ */
+ public static boolean isFieldUpdatable(String field, HashMap values) {
+if (values.get(field) == null) {
+ return false;
+}
+return true;
+ }
+
+ /**
+ * Generate avro schema based on the number of fields. supplied when running
the benchmark. These files are json files
+ *
+ * @param numberOfFields the number of fields
+ */
+ public void generateAvroSchema(int numberOfFields) {
+try {
+ File avroFile = new File(AVRO_FULL_PATH);
+ FieldAssembler fieldAssembler =
SchemaBuilder.record(RECORD).namespace(NAMESPACE_VALUE).fields();
+
fieldAssembler.name(USER_ID_VALUE).type().stringType().stringDefault(NULL);
+ for (int i = 0; i < numberOfFields; i++) {
+fieldAssembler.name(FIELD_PREFIX +
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310846910 ## File path: gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java ## @@ -0,0 +1,201 @@ +package org.apache.gora.benchmark; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Properties; +import java.util.Set; +import java.util.Vector; + +import org.apache.gora.util.GoraException; +import org.junit.After; +import org.junit.Before; +import org.junit.Ignore; +import org.junit.Test; + +import com.yahoo.ycsb.ByteIterator; +import com.yahoo.ycsb.StringByteIterator; +import com.yahoo.ycsb.workloads.CoreWorkload; + +import generated.User; + +// TODO: Auto-generated Javadoc Review comment: Remove comment This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310846806 ## File path: gora-benchmark/src/main/resources/gora-couchdb-mapping.xml ## @@ -0,0 +1,26 @@ + Review comment: License header This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310846879 ## File path: gora-benchmark/src/test/java/org/apache/gora/benchmark/GoraClientTest.java ## @@ -0,0 +1,201 @@ +package org.apache.gora.benchmark; Review comment: License header This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846052
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkClient.java
##
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.benchmark;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.Vector;
+
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.StringByteIterator;
+import com.yahoo.ycsb.workloads.CoreWorkload;
+
+import generated.User;
+
+
+/**
+ * The Class GoraBenchmarkClient.
+ *
+ * @author sc306
+ * This class extends the Yahoo! Cloud Service Benchmark benchmark {@link
#com.yahoo.ycsb.DB DB} class to provide functionality
+ * for {@link #insert(String, String, HashMap) insert}, {@link #read(String,
String, Set, HashMap) read}, {@link #scan(String, String, int, Set, Vector)
scan}
+ * and {@link #update(String, String, HashMap) update} methods as per Apache
Gora implementation.
+ */
+public class GoraBenchmarkClient extends DB {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkClient.class);
+
+ private static final int SUCCESS = 0;
+
+ private static final int FAILED = 1;
+
+ private static final String FIELDS[] = User._ALL_FIELDS;
+
+ private static volatile boolean executed;
+
+ public static int fieldCount;
+
+ /**This is only for set to array conversion in {@link read()} method*/
+ private String[] DUMMY_ARRAY = new String[0];
+
+ DataStore dataStore;
+
+ GoraBenchmarkUtils goraBenchmarkUtils = new GoraBenchmarkUtils();
+
+ User user = new User();
+
+ private Properties prop;
+
+ public GoraBenchmarkClient() {}
+
+
+ /***
+ * Initialisation method. This method is called once for each database
instance.
+ */
+ public void init() throws DBException {
+try {
+
+//Get YCSB properties
+prop = getProperties();
+
+fieldCount =
Integer.parseInt(prop.getProperty(CoreWorkload.FIELD_COUNT_PROPERTY,
CoreWorkload.FIELD_COUNT_PROPERTY_DEFAULT));
+
+String keyClass = prop.getProperty("key.class", "java.lang.String");
+String persistentClass = prop.getProperty("persistent.class",
"generated.User");
+Properties p = DataStoreFactory.createProps();
+dataStore = DataStoreFactory.getDataStore(keyClass, persistentClass,
p, new Configuration());
+synchronized(GoraBenchmarkClient.class) {
+ if(executed)
+return;
+ executed = true;
+ goraBenchmarkUtils.generateAvroSchema(fieldCount);
+ String dataStoreName = goraBenchmarkUtils.getDataStore(p);
+ goraBenchmarkUtils.generateMappingFile(dataStoreName);
+ goraBenchmarkUtils.generateDataBeans();
+}
+
+
+} catch (GoraException e) {
+ //e.printStackTrace();
Review comment:
Remove commented out code
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846537
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
+
+ /**
+ * Checks if is field updatable.
+ *
+ * @param field the field
+ * @param values the values
+ * @return true, if is field updatable
+ */
+ public static boolean isFieldUpdatable(String field, HashMap values) {
+if (values.get(field) == null) {
+ return false;
+}
+return true;
+ }
+
+ /**
+ * Generate avro schema based on the number of fields. supplied when running
the benchmark. These files are json files
+ *
+ * @param numberOfFields the number of fields
+ */
+ public void generateAvroSchema(int numberOfFields) {
+try {
+ File avroFile = new File(AVRO_FULL_PATH);
+ FieldAssembler fieldAssembler =
SchemaBuilder.record(RECORD).namespace(NAMESPACE_VALUE).fields();
+
fieldAssembler.name(USER_ID_VALUE).type().stringType().stringDefault(NULL);
+ for (int i = 0; i < numberOfFields; i++) {
+fieldAssembler.name(FIELD_PREFIX +
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846525
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
+
+ /**
+ * Checks if is field updatable.
+ *
+ * @param field the field
+ * @param values the values
+ * @return true, if is field updatable
+ */
+ public static boolean isFieldUpdatable(String field, HashMap values) {
+if (values.get(field) == null) {
+ return false;
+}
+return true;
+ }
+
+ /**
+ * Generate avro schema based on the number of fields. supplied when running
the benchmark. These files are json files
+ *
+ * @param numberOfFields the number of fields
+ */
+ public void generateAvroSchema(int numberOfFields) {
+try {
+ File avroFile = new File(AVRO_FULL_PATH);
+ FieldAssembler fieldAssembler =
SchemaBuilder.record(RECORD).namespace(NAMESPACE_VALUE).fields();
+
fieldAssembler.name(USER_ID_VALUE).type().stringType().stringDefault(NULL);
+ for (int i = 0; i < numberOfFields; i++) {
+fieldAssembler.name(FIELD_PREFIX +
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845147
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
+
+
+
+ org.apache.gora
+ gora-hbase
+
+
+
+ org.apache.gora
+ gora-jcache
+
+
+
+ org.apache.gora
+ gora-couchdb
+
+
+
+ org.apache.gora
+ gora-cassandra
+
+
+
+ org.apache.gora
+ gora-solr
+
+
+
+ org.apache.gora
+ gora-aerospike
+
+
+
+ org.apache.avro
+ avro
+ compile
+
+
+
+ org.apache.gora
+ gora-core
+ test-jar
+ test
+
+
+
+ gora-compiler
+ org.apache.gora
+
+
+
+
+ org.slf4j
+ slf4j-log4j12
+
+
+ log4j
+ log4j
+
+
+ javax.jms
+ jms
+
+
+
+
+
+
+
+
+ org.apache.hadoop
+ hadoop-client
+ compile
+
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846496
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
+
+ /**
+ * Checks if is field updatable.
+ *
+ * @param field the field
+ * @param values the values
+ * @return true, if is field updatable
+ */
+ public static boolean isFieldUpdatable(String field, HashMap values) {
+if (values.get(field) == null) {
+ return false;
+}
+return true;
+ }
+
+ /**
+ * Generate avro schema based on the number of fields. supplied when running
the benchmark. These files are json files
+ *
+ * @param numberOfFields the number of fields
+ */
+ public void generateAvroSchema(int numberOfFields) {
+try {
+ File avroFile = new File(AVRO_FULL_PATH);
+ FieldAssembler fieldAssembler =
SchemaBuilder.record(RECORD).namespace(NAMESPACE_VALUE).fields();
+
fieldAssembler.name(USER_ID_VALUE).type().stringType().stringDefault(NULL);
+ for (int i = 0; i < numberOfFields; i++) {
+fieldAssembler.name(FIELD_PREFIX +
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846552
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
+
+ /**
+ * Checks if is field updatable.
+ *
+ * @param field the field
+ * @param values the values
+ * @return true, if is field updatable
+ */
+ public static boolean isFieldUpdatable(String field, HashMap values) {
+if (values.get(field) == null) {
+ return false;
+}
+return true;
+ }
+
+ /**
+ * Generate avro schema based on the number of fields. supplied when running
the benchmark. These files are json files
+ *
+ * @param numberOfFields the number of fields
+ */
+ public void generateAvroSchema(int numberOfFields) {
+try {
+ File avroFile = new File(AVRO_FULL_PATH);
+ FieldAssembler fieldAssembler =
SchemaBuilder.record(RECORD).namespace(NAMESPACE_VALUE).fields();
+
fieldAssembler.name(USER_ID_VALUE).type().stringType().stringDefault(NULL);
+ for (int i = 0; i < numberOfFields; i++) {
+fieldAssembler.name(FIELD_PREFIX +
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310846562
##
File path:
gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java
##
@@ -0,0 +1,453 @@
+package org.apache.gora.benchmark;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerConfigurationException;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.SchemaBuilder.FieldAssembler;
+import org.apache.gora.compiler.GoraCompiler;
+import org.apache.gora.compiler.utils.LicenseHeaders;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Attr;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import com.yahoo.ycsb.ByteIterator;
+
+// TODO: Auto-generated Javadoc
+/**
+ * The Class GoraBenchmarkUtils.
+ */
+public class GoraBenchmarkUtils {
+
+ /** The Constant AVRO_PATH. */
+ private static final String AVRO_PATH = "src/main/avro";
+
+ /** The Constant AVRO_FILE. */
+ private static final String AVRO_FILE = "user.json";
+
+ /** The field prefix. */
+ private static String FIELD_PREFIX = "field";
+
+ /** The Constant USER_ID_VALUE. */
+ private static final String USER_ID_VALUE = "userId";
+
+ /** The Constant AVRO_FULL_PATH. */
+ private static final String AVRO_FULL_PATH = AVRO_PATH + "/" + AVRO_FILE;
+
+ /** The Constant NULL. */
+ private static final String NULL = "null";
+
+ /** The Constant RECORD. */
+ private static final String RECORD = "User";
+
+ /** The Constant NAMESPACE_VALUE. */
+ private static final String NAMESPACE_VALUE = "generated";
+
+ /** The Constant NAMESPACE_KEY. */
+ private static final String NAMESPACE_KEY = "namespace";
+
+ /** The Constant NAME_KEY. */
+ private static final String NAME_KEY = "name";
+
+ /** The Constant MONGODB. */
+ private static final String MONGODB = "mongodb";
+
+ /** The Constant COUCHDB. */
+ private static final String COUCHDB = "couchdb";
+
+ /** The Constant HBASE. */
+ private static final String HBASE = "hbase";
+
+ /** The Constant KEYCLASS. */
+ private static final String KEYCLASS = "java.lang.String";
+
+ /** The db mapping path. */
+ private static String DB_MAPPING_PATH = "src/main/resources";
+
+ /** The Constant MONGO_MAPPING_FILE. */
+ private static final String MONGO_MAPPING_FILE = "gora-mongodb-mapping.xml";
+
+ /** The Constant HBASE_MAPPING_FILE. */
+ private static final String HBASE_MAPPING_FILE = "gora-hbase-mapping.xml";
+
+ /** The Constant COUCHDB_MAPPING_FILE. */
+ private static final String COUCHDB_MAPPING_FILE =
"gora-couchdb-mapping.xml";
+
+ /** The Constant BEAN_DESTINATION_DIR. */
+ private static final File BEAN_DESTINATION_DIR = new File("src/main/java/");
+
+ /** The Constant DEFAULT_DATA_STORE_KEY. */
+ private static final String DEFAULT_DATA_STORE_KEY =
"gora.datastore.default";
+
+
+ private static final String GORA_ROOT_ELEMENT = "gora-otd";
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GoraBenchmarkUtils.class);
+
+
+
+ /**
+ * Checks if is field updatable.
+ *
+ * @param field the field
+ * @param values the values
+ * @return true, if is field updatable
+ */
+ public static boolean isFieldUpdatable(String field, HashMap values) {
+if (values.get(field) == null) {
+ return false;
+}
+return true;
+ }
+
+ /**
+ * Generate avro schema based on the number of fields. supplied when running
the benchmark. These files are json files
+ *
+ * @param numberOfFields the number of fields
+ */
+ public void generateAvroSchema(int numberOfFields) {
+try {
+ File avroFile = new File(AVRO_FULL_PATH);
+ FieldAssembler fieldAssembler =
SchemaBuilder.record(RECORD).namespace(NAMESPACE_VALUE).fields();
+
fieldAssembler.name(USER_ID_VALUE).type().stringType().stringDefault(NULL);
+ for (int i = 0; i < numberOfFields; i++) {
+fieldAssembler.name(FIELD_PREFIX +
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310845046
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
+
+
+
+ org.apache.gora
+ gora-hbase
+
+
+
+ org.apache.gora
+ gora-jcache
+
+
+
+ org.apache.gora
+ gora-couchdb
+
+
+
+ org.apache.gora
+ gora-cassandra
+
+
+
+ org.apache.gora
+ gora-solr
+
+
+
+ org.apache.gora
+ gora-aerospike
+
+
+
+ org.apache.avro
+ avro
+ compile
+
+
+
+ org.apache.gora
+ gora-core
+ test-jar
+ test
Review comment:
Is this actually needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310844845
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
Review comment:
Remove
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310844584
##
File path: gora-benchmark/gora-bench.sh
##
@@ -0,0 +1,80 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Adapted from YCSB's version of ycsb.sh.
+CLASSPATH="."
+
+# Attempt to find the available JAVA, if JAVA_HOME not set
+if [ -z "$JAVA_HOME" ]; then
+ JAVA_PATH=$(which java 2>/dev/null)
+ if [ "x$JAVA_PATH" != "x" ]; then
+JAVA_HOME=$(dirname "$(dirname "$JAVA_PATH" 2>/dev/null)")
+ fi
+fi
+
+# If JAVA_HOME still not set, error
+if [ -z "$JAVA_HOME" ]; then
+ echo "[ERROR] Java executable not found. Exiting."
+ exit 1;
+fi
+
+# Determine YCSB command argument
+if [ "load" = "$1" ] ; then
+ YCSB_COMMAND=-load
+ YCSB_CLASS=com.yahoo.ycsb.Client
+ DB_CLASS=org.apache.gora.benchmark.GoraBenchmarkClient
+elif [ "run" = "$1" ] ; then
+ YCSB_COMMAND=-t
+ YCSB_CLASS=com.yahoo.ycsb.Client
+ DB_CLASS=org.apache.gora.benchmark.GoraBenchmarkClient
+else
+ echo "[ERROR] Found unknown command '$1'"
+ echo "[ERROR] Expected one of 'load', 'run', or 'shell'. Exiting."
+ exit 1;
+fi
+
+SAVEIFS=$IFS
+IFS=$(echo -en "\n\b")
+
+# add libs to CLASSPATH
+for f in target/*.jar; do
+ CLASSPATH=${CLASSPATH}:$f;
+done
+
+for f in lib/*.jar; do
+ CLASSPATH=${CLASSPATH}:$f;
+done
+
+IFS=$SAVEIFS
+
+CLASSPATH=${CLASSPATH}:target/classes/
+CLASSPATH=${CLASSPATH}:target/test-classes/
+
+#CLASSPATH=${CLASSPATH}:conf
+#CLASSPATH=${CLASSPATH}:conf
Review comment:
Remove both of these
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310844887
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
Review comment:
Remove comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310845212 ## File path: gora-benchmark/runWorkloads.sh ## @@ -0,0 +1,119 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#Check if number of commanline arguments is right Review comment: commandline rather than commanline This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310844688 ## File path: gora-benchmark/pom.xml ## @@ -0,0 +1,197 @@ + Review comment: Include ALv2.0 license header This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310844996
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
Review comment:
Version should not be there but inherited from parent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r310844713 ## File path: gora-benchmark/pom.xml ## @@ -0,0 +1,197 @@ + +http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd; + xmlns="http://maven.apache.org/POM/4.0.0; Review comment: Format file with 2 space indents This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r310844937
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
Review comment:
Right now the dependencies are kinda all over the place. Can you list them
alphabetically?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
