[jira] [Commented] (GORA-320) Hive DataStore
[ https://issues.apache.org/jira/browse/GORA-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902705#comment-16902705 ] Kevin Ratnasekera commented on GORA-320: [~chanakadkb] I just assigned the ticket for you. If you need any help or when you have an update please let us know. This will be a great addition to Apache Gora. > Hive DataStore > -- > > Key: GORA-320 > URL: https://issues.apache.org/jira/browse/GORA-320 > Project: Apache Gora > Issue Type: New Feature > Components: gora-hive >Reporter: Renato Javier Marroquín Mogrovejo >Assignee: Chanaka >Priority: Major > Fix For: 1.0 > > > We could/should also support Hive as a backend. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (GORA-320) Hive DataStore
[ https://issues.apache.org/jira/browse/GORA-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Ratnasekera reassigned GORA-320: -- Assignee: Chanaka > Hive DataStore > -- > > Key: GORA-320 > URL: https://issues.apache.org/jira/browse/GORA-320 > Project: Apache Gora > Issue Type: New Feature > Components: gora-hive >Reporter: Renato Javier Marroquín Mogrovejo >Assignee: Chanaka >Priority: Major > Fix For: 1.0 > > > We could/should also support Hive as a backend. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (GORA-616) Multiple slf4j conflict issue
[ https://issues.apache.org/jira/browse/GORA-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Ratnasekera resolved GORA-616. Resolution: Fixed [~chanakadkb] Thank you for the contribution. Looking forward for your future contributions. > Multiple slf4j conflict issue > -- > > Key: GORA-616 > URL: https://issues.apache.org/jira/browse/GORA-616 > Project: Apache Gora > Issue Type: Improvement > Components: gora-tutorial >Affects Versions: 0.9 >Reporter: Sheriffo Ceesay >Assignee: Chanaka >Priority: Major > Labels: build, test > Fix For: 0.9 > > > Testing the 0.9 release using the gora-tutorial module, invoking the line > below gives you some warnings and error. > > bin/gora logmanager -parse gora-tutorial/src/main/resources/access.log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/home/user/apache-gora-0.9/gora-tutorial/lib/log4j-slf4j-impl-2.11.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/home/user/apache-gora-0.9/gora-tutorial/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/home/user/apache-gora-0.9/gora-tutorial/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ERROR StatusLogger No Log4j 2 configuration file found. Using default > configuration (logging only errors to the console), or user programmatically > provided configurations. Set system property 'log4j2.debug' to show Log4j 2 > internal initialization logging. See > [https://logging.apache.org/log4j/2.x/manual/configuration.html] for > instructions on how to configure Log4j 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (GORA-616) Multiple slf4j conflict issue
[ https://issues.apache.org/jira/browse/GORA-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Ratnasekera updated GORA-616: --- Fix Version/s: (was: 0.8) > Multiple slf4j conflict issue > -- > > Key: GORA-616 > URL: https://issues.apache.org/jira/browse/GORA-616 > Project: Apache Gora > Issue Type: Improvement > Components: gora-tutorial >Affects Versions: 0.9 >Reporter: Sheriffo Ceesay >Assignee: Chanaka >Priority: Major > Labels: build, test > Fix For: 0.9 > > > Testing the 0.9 release using the gora-tutorial module, invoking the line > below gives you some warnings and error. > > bin/gora logmanager -parse gora-tutorial/src/main/resources/access.log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/home/user/apache-gora-0.9/gora-tutorial/lib/log4j-slf4j-impl-2.11.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/home/user/apache-gora-0.9/gora-tutorial/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/home/user/apache-gora-0.9/gora-tutorial/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ERROR StatusLogger No Log4j 2 configuration file found. Using default > configuration (logging only errors to the console), or user programmatically > provided configurations. Set system property 'log4j2.debug' to show Log4j 2 > internal initialization logging. See > [https://logging.apache.org/log4j/2.x/manual/configuration.html] for > instructions on how to configure Log4j 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (GORA-616) Multiple slf4j conflict issue
[ https://issues.apache.org/jira/browse/GORA-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Ratnasekera reassigned GORA-616: -- Assignee: Chanaka > Multiple slf4j conflict issue > -- > > Key: GORA-616 > URL: https://issues.apache.org/jira/browse/GORA-616 > Project: Apache Gora > Issue Type: Improvement > Components: gora-tutorial >Affects Versions: 0.9 >Reporter: Sheriffo Ceesay >Assignee: Chanaka >Priority: Major > Labels: build, test > Fix For: 0.8, 0.9 > > > Testing the 0.9 release using the gora-tutorial module, invoking the line > below gives you some warnings and error. > > bin/gora logmanager -parse gora-tutorial/src/main/resources/access.log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/home/user/apache-gora-0.9/gora-tutorial/lib/log4j-slf4j-impl-2.11.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/home/user/apache-gora-0.9/gora-tutorial/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/home/user/apache-gora-0.9/gora-tutorial/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ERROR StatusLogger No Log4j 2 configuration file found. Using default > configuration (logging only errors to the console), or user programmatically > provided configurations. Set system property 'log4j2.debug' to show Log4j 2 > internal initialization logging. See > [https://logging.apache.org/log4j/2.x/manual/configuration.html] for > instructions on how to configure Log4j 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [gora] sneceesay77 opened a new pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 opened a new pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179 Initial pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 edited a comment on issue #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 edited a comment on issue #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#issuecomment-519293208 @lewismc , please see my latest update to the pull request. I have added a README file detailing how to run the benchmark module. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on issue #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on issue #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#issuecomment-519293208 > Please ping me here @sneceesay77 when the PR updated and you've provided a README of how we can run this. Thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 closed pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 closed pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on issue #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on issue #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#issuecomment-519173439 Please ping me here @sneceesay77 when the PR updated and you've provided a README of how we can run this. Thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311649100 ## File path: gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java ## @@ -0,0 +1,453 @@ +package org.apache.gora.benchmark; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.StringWriter; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.HashMap; +import java.util.Iterator; +import java.util.Locale; +import java.util.Properties; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; +import javax.xml.parsers.ParserConfigurationException; +import javax.xml.transform.OutputKeys; +import javax.xml.transform.Transformer; +import javax.xml.transform.TransformerConfigurationException; +import javax.xml.transform.TransformerException; +import javax.xml.transform.TransformerFactory; +import javax.xml.transform.dom.DOMSource; +import javax.xml.transform.stream.StreamResult; + +import org.apache.avro.Schema; +import org.apache.avro.SchemaBuilder; +import org.apache.avro.SchemaBuilder.FieldAssembler; +import org.apache.gora.compiler.GoraCompiler; +import org.apache.gora.compiler.utils.LicenseHeaders; +import org.json.JSONArray; +import org.json.JSONObject; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.w3c.dom.Attr; +import org.w3c.dom.Document; +import org.w3c.dom.Element; + +import com.yahoo.ycsb.ByteIterator; + +// TODO: Auto-generated Javadoc +/** + * The Class GoraBenchmarkUtils. Review comment: Thanks for updating with a description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311648826
##
File path: gora-benchmark/src/main/avro/person.json
##
@@ -0,0 +1,18 @@
+ {
+ "type": "record",
+ "name": "Person", "default":null,
+ "namespace": "generated",
+ "fields" : [
+{"name": "userId", "type": ["null","string"], "default":null},
Review comment:
OK no problem. Please remote file. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311648700
##
File path: gora-benchmark/runWorkloads.sh
##
@@ -0,0 +1,119 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Check if number of commanline arguments is right
+threadcount=15
+if [ $# -ne 3 ]
+then
+echo "Error: Usage: $0 hbase|mongodb gora|ycsb workloadfile"
+exit;
+fi
+#Log output file
+outputfile="$3-$2-$1.log"
+insertfactor=500
+if [ $2 = "ycsb" ]
+then
+ table="usertable"
+elif [ $2 == "gora" ]
+then
+ table="users"
+fi
+
+if [ $1 = "hbase" ]
+then
+ #Read operation
+ echo -e "Disabling Users Table" | tee -a $outputfile
+ echo -e "disable '$table'" | hbase shell -n
+ echo -e "Dropping Users Table" | tee -a $outputfile
+ echo -e "drop '$table'" | hbase shell -n
+ echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ fi
+for i in {10..500..10}
+do
+ #Insert only workload, uncomment this block of code. I will improve this
later.
+ #echo -e "Disabling Users Table" | tee -a $outputfile
+ #echo -e "disable '$table'" | hbase shell -n
+ #echo -e "Dropping Users Table" | tee -a $outputfile
+ #echo -e "drop '$table'" | hbase shell -n
+ #echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ #echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+
+ echo -e "Table Setup Completed = Now Running Benchmark
for $i records and operations" | tee -a $outputfile
+ echo -e "Record Count $i" | tee -a $outputfile
+ echo -e "Operation Count $i" | tee -a $outputfile
+ echo -e "Thread Count $threadcount" | tee -a $outputfile
+ echo -e ""
+ if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ ./gora-bench.sh run -threads 15 -s -p readallfields=true -p
measurementtype=timeseries -p timeseries.granularity=2000 -p operationcount=$i
-P workloads/$3 | tee -a $outputfile
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ ../../ycsb/bin/ycsb.sh run hbase20 -s -p columnfamily=info -p
measurementtype=timeseries -p timeseries.granularity=2000 -threads 15 -p
readallfields=true -p operationcount=$i -P workloads/$3 | tee -a $outputfile
+ fi
+ echo "End of
Benchmark" | tee -a $outputfile
+done
+elif [ $1 = "mongodb" ]
+then
+ echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ if [ $2 = "ycsb" ]
+ then
+ mongo < setupycsbmongo.js
+ ../../ycsb/bin/ycsb.sh load mongodb -threads 15 -s -p fieldcount=20
-p recordcount=$insertfactor -p operationcount=$insertfactor -P
workloads/workloada
+ fi
+
+ if [ $2 = "gora" ]
+ then
+ mongo < setupgoramongo.js
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$insertfactor -P workloads/workloada
+ fi
+ echo -e "Table Setup Completed = Now Running
Benchmark for $i records and operations" | tee -a $outputfile
+for i in {10..500..10}
+do
+ #echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ #mongo < setupmongo.js
+ #echo -e "Table Setup Completed
[GitHub] [gora] lewismc commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
lewismc commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311648487
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
Review comment:
+1, just go ahead and obtain this from parent pom. Do whatever you need to
do for that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311635669
##
File path: gora-benchmark/runWorkloads.sh
##
@@ -0,0 +1,119 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#Check if number of commanline arguments is right
+threadcount=15
+if [ $# -ne 3 ]
+then
+echo "Error: Usage: $0 hbase|mongodb gora|ycsb workloadfile"
+exit;
+fi
+#Log output file
+outputfile="$3-$2-$1.log"
+insertfactor=500
+if [ $2 = "ycsb" ]
+then
+ table="usertable"
+elif [ $2 == "gora" ]
+then
+ table="users"
+fi
+
+if [ $1 = "hbase" ]
+then
+ #Read operation
+ echo -e "Disabling Users Table" | tee -a $outputfile
+ echo -e "disable '$table'" | hbase shell -n
+ echo -e "Dropping Users Table" | tee -a $outputfile
+ echo -e "drop '$table'" | hbase shell -n
+ echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ fi
+for i in {10..500..10}
+do
+ #Insert only workload, uncomment this block of code. I will improve this
later.
+ #echo -e "Disabling Users Table" | tee -a $outputfile
+ #echo -e "disable '$table'" | hbase shell -n
+ #echo -e "Dropping Users Table" | tee -a $outputfile
+ #echo -e "drop '$table'" | hbase shell -n
+ #echo -e "Creating Users table with 200 regions: 200 Regions is recommended
by YCSB" | tee -a $outputfile
+ #echo -e "create '$table', 'info', {SPLITS => (1..200).map {|i|
\"user#{1000+i*(-1000)/200}\"}}" | hbase shell -n
+
+ echo -e "Table Setup Completed = Now Running Benchmark
for $i records and operations" | tee -a $outputfile
+ echo -e "Record Count $i" | tee -a $outputfile
+ echo -e "Operation Count $i" | tee -a $outputfile
+ echo -e "Thread Count $threadcount" | tee -a $outputfile
+ echo -e ""
+ if [ $2 = "gora" ]
+ then
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$i -P workloads/workloada
+ ./gora-bench.sh run -threads 15 -s -p readallfields=true -p
measurementtype=timeseries -p timeseries.granularity=2000 -p operationcount=$i
-P workloads/$3 | tee -a $outputfile
+ elif [ $2 = "ycsb" ]
+ then
+ ../../ycsb/bin/ycsb.sh load hbase20 -p columnfamily=info -threads 15 -s
-p fieldcount=20 -p recordcount=$insertfactor -p operationcount=$i -P
workloads/workloada
+ ../../ycsb/bin/ycsb.sh run hbase20 -s -p columnfamily=info -p
measurementtype=timeseries -p timeseries.granularity=2000 -threads 15 -p
readallfields=true -p operationcount=$i -P workloads/$3 | tee -a $outputfile
+ fi
+ echo "End of
Benchmark" | tee -a $outputfile
+done
+elif [ $1 = "mongodb" ]
+then
+ echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ if [ $2 = "ycsb" ]
+ then
+ mongo < setupycsbmongo.js
+ ../../ycsb/bin/ycsb.sh load mongodb -threads 15 -s -p fieldcount=20
-p recordcount=$insertfactor -p operationcount=$insertfactor -P
workloads/workloada
+ fi
+
+ if [ $2 = "gora" ]
+ then
+ mongo < setupgoramongo.js
+ ./gora-bench.sh load -threads 15 -s -p fieldcount=20 -p
recordcount=$insertfactor -p operationcount=$insertfactor -P workloads/workloada
+ fi
+ echo -e "Table Setup Completed = Now Running
Benchmark for $i records and operations" | tee -a $outputfile
+for i in {10..500..10}
+do
+ #echo -e "Dropping $table from MongoDB" | tee -a $outputfile
+ #mongo < setupmongo.js
+ #echo -e "Table Setup Completed
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311631483 ## File path: gora-benchmark/src/main/java/org/apache/gora/benchmark/GoraBenchmarkUtils.java ## @@ -0,0 +1,453 @@ +package org.apache.gora.benchmark; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.StringWriter; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.HashMap; +import java.util.Iterator; +import java.util.Locale; +import java.util.Properties; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; +import javax.xml.parsers.ParserConfigurationException; +import javax.xml.transform.OutputKeys; +import javax.xml.transform.Transformer; +import javax.xml.transform.TransformerConfigurationException; +import javax.xml.transform.TransformerException; +import javax.xml.transform.TransformerFactory; +import javax.xml.transform.dom.DOMSource; +import javax.xml.transform.stream.StreamResult; + +import org.apache.avro.Schema; +import org.apache.avro.SchemaBuilder; +import org.apache.avro.SchemaBuilder.FieldAssembler; +import org.apache.gora.compiler.GoraCompiler; +import org.apache.gora.compiler.utils.LicenseHeaders; +import org.json.JSONArray; +import org.json.JSONObject; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.w3c.dom.Attr; +import org.w3c.dom.Document; +import org.w3c.dom.Element; + +import com.yahoo.ycsb.ByteIterator; + +// TODO: Auto-generated Javadoc +/** + * The Class GoraBenchmarkUtils. Review comment: Updated the file with some description, the methods in this file are called in the init method of GoraBenchmarkClient. The class is not directly executed. Basically, you can specify how many fields a particular table in a particular datastore should have. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: Kudu datastore reports
Hi John, Thank you for the Clarification. If the current implementation has no issues with build in those environments, I think there is *no* need to invest time in other approaches Eg:- Docker. Simply just do build test whether tests pass for those operating systems. Regards Kevin On Wed, Aug 7, 2019 at 3:09 AM John Mora wrote: > Hi Kevin. > > KuduTestHarness, theoretically should detect environment through this > plugin "os-maven-plugin" and download the corresponding kudu binaries [1], > and it have worked fine for me. > > Nonetheless, docker is a good idea. I will give a look to testcontainers > and docker. > > [1] > https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing > > Regards, > John > > El lun., 5 ago. 2019 a las 23:58, Kevin Ratnasekera (< > djkevincr1...@gmail.com>) escribió: > > > Hi John, > > > > Can't we spin up Kudu docker [1] instance for testing purposes? We have > > used Test containers [2] some data stores like couch DB. Gora build > should > > work in both Linux and Non Linux environments. Eg:- Windows. Is > classifier > > [3] depend on the environment the build is running? > > > > Kudu is based on C/ C++, so to spin up a server instance, we need check a > > approach like docker, using such approach allow us to avoid these OS, > > dependency related stuff come in to play in builds. > > > > [1] https://hub.docker.com/r/usuresearch/apache-kudu/ > > [2] https://www.testcontainers.org/ > > [3] linux-x86_64 > > > > Regards > > Kevin > > > > On Tue, Aug 6, 2019 at 9:56 AM John Mora wrote: > > > > > Hi Alfonso, > > > > > > Unfortunately, I have not been able to reproduce the issue. Maybe it is > > > related with my Java version (Oracle), I will try with OpenJDK. > > > Some details about my development environment: > > > > > > os.detected.name: linux > > > os.detected.arch: x86_64 > > > os.detected.version: 4.10 > > > os.detected.version.major: 4 > > > os.detected.version.minor: 10 > > > os.detected.release: linuxmint > > > os.detected.release.version: 18.3 > > > os.detected.release.like.linuxmint: true > > > os.detected.release.like.ubuntu: true > > > os.detected.classifier: linux-x86_64 > > > > > > Java > > > java version "1.8.0_171" > > > Java(TM) SE Runtime Environment (build 1.8.0_171-b11) > > > Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode) > > > > > > Maven > > > Apache Maven 3.3.9 > > > Maven home: /usr/share/maven > > > Java version: 1.8.0_171, vendor: Oracle Corporation > > > Java home: /usr/lib/jvm/java-8-oracle/jre > > > Default locale: en_US, platform encoding: UTF-8 > > > OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family: > > > "unix" > > > > > > > > > Best, > > > John. > > > > > > El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (< > > > alfonso.nishik...@gmail.com>) escribió: > > > > > >> Hi, > > >> > > >> I am using now the following pom configuration I got from executing > `mvn > > >> dependency:tree`: > > >> > > >> > > >> org.apache.kudu > > >> kudu-binary > > >> linux-x86_64 > > >> 1.9.0 > > >> test > > >> > > >> > > >> When I execute `mvn clen package` on gora-kudu I find that it spawns > the > > >> following command: > > >> > > >> kudu-master > > >> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal > > >> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data > > >> --block_manager=log --webserver_interface=localhost > > --ipki_ca_key_size=1024 > > >> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535 > > >> --webserver_interface=*127.26.116.190* --webserver_port=0 > --never_fsync > > >> --ipki_server_key_size=1024 --enable_minidumps=false --redact=none > > >> --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 > > >> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs > > >> > > > --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb > > >> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports > > >> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true > > >> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913, > > >> *127.26.116.188*:42253 > > >> > > >> > > >> I highlight the IP addresses because they clearly are not my computer, > > >> and I guess that is why the tests can't connect to the the database. > > >> > > >> Any idea on how to solve this? > > >> > > >> Thank you! > > >> > > >> > > >> Best Regards, > > >> > > >> Alfonso Nishikawa > > >> > > >> > > >> > > >> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (< > > >> alfonso.nishik...@gmail.com>) escribió: > > >> > > >>> Hi, John. > > >>> > > >>> I get a core dump from the binary kudu server when trying to run the > > >>> tests. Didn't find a log file, but will search thoroughly later. > > Happened > > >>> anytime to you? Does it happens to anyone? > > >>> > > >>> I am using Ubuntu 18.04 > > >>> > > >>> Thank you! > > >>> > > >>> Regards, > > >>> > > >>>
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311490797 ## File path: gora-benchmark/src/main/resources/gora-couchdb-mapping.xml ## @@ -0,0 +1,26 @@ + Review comment: Mapping files are auto-generated. I have added the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311491122 ## File path: gora-benchmark/src/main/resources/gora-mongodb-mapping.xml ## @@ -0,0 +1,26 @@ + Review comment: Mapping files are auto-generated. I have added the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311490835 ## File path: gora-benchmark/src/main/resources/gora-hbase-mapping.xml ## @@ -0,0 +1,59 @@ + Review comment: Mapping files are auto-generated. I have added the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311490797 ## File path: gora-benchmark/src/main/resources/gora-couchdb-mapping.xml ## @@ -0,0 +1,26 @@ + Review comment: Mapping files are auto-generated. I have the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r311490835 ## File path: gora-benchmark/src/main/resources/gora-hbase-mapping.xml ## @@ -0,0 +1,59 @@ + Review comment: Mapping files are auto-generated. I have the code to add the license file accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311490191
##
File path: gora-benchmark/src/main/avro/user.json
##
@@ -0,0 +1,112 @@
+{
+"name": "User",
+"namespace": "generated",
+"type": "record",
+"fields": [
+{
Review comment:
The autogenerated JSON files are all the same. The person.json file is not
being used. I will remove the file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311489884
##
File path: gora-benchmark/src/main/avro/person.json
##
@@ -0,0 +1,18 @@
+ {
+ "type": "record",
+ "name": "Person", "default":null,
+ "namespace": "generated",
+ "fields" : [
+{"name": "userId", "type": ["null","string"], "default":null},
Review comment:
I am not actually using this file anymore, but yes it would be interesting
to test with complex objects. For YCSB it uses a simple User object which can
have **n** number of fields.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311474325
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
Review comment:
Unlike other modules, there is no definition or declaration of MongoDB in
the parent pom file, hence the explicit declaration in my case for now. Maybe I
can create a separate issue for this or should I just go ahead and edit the
parent pom accordingly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora
Benchmark initial pull request for review and comments
URL: https://github.com/apache/gora/pull/179#discussion_r311474325
##
File path: gora-benchmark/pom.xml
##
@@ -0,0 +1,197 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+ xmlns="http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+ 4.0.0
+
+ org.apache.gora
+ gora
+ 0.9-SNAPSHOT
+
+ gora-benchmark
+
+ Apache Gora Benchamark Module
+ http://maven.apache.org
+ The Apache Gora open source framework provides an
in-memory data model and
+persistence for big data. Gora supports persisting to column stores, key
value stores,
+document stores and RDBMSs, and analyzing the data with extensive Apache
Hadoop MapReduce
+support.
+ 2010
+
+ The Apache Software Foundation
+ http://www.apache.org/
+
+
+ JIRA
+ https://issues.apache.org/jira/browse/GORA
+
+
+ Jenkins
+ https://builds.apache.org/job/Gora-trunk/
+
+
+
UTF-8
+
+
+ target
+ target/classes
+ ${project.artifactId}-${project.version}
+ target/test-classes
+ src/test/java
+ src/main/java
+
+
+
${project.basedir}/src/test/conf
+
+ **/*
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+
${build-helper-maven-plugin.version}
+
+
+ generate-sources
+
+ add-source
+
+
+
+
src/examples/java
+
+
+
+
+
+
+
+
+
+
+ com.yahoo.ycsb
+ core
+ 0.1.4
+
+
+
+
+
+ org.apache.gora
+ gora-core
+ compile
+
+
+
+
+ org.apache.gora
+ gora-mongodb
+ compile
+ 0.9-SNAPSHOT
Review comment:
Like others, there is no definition or declaration of MongoDB in the parent
pom file, hence the explicit declaration for now. Maybe I can create a separate
issue for this or should I just go ahead and edit the parent pom accordingly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
