[GitHub] [gora] djkevincr commented on a change in pull request #186: [GORA-527] Implement a data store for redis
djkevincr commented on a change in pull request #186: [GORA-527] Implement a data store for redis URL: https://github.com/apache/gora/pull/186#discussion_r316498705 ## File path: pom.xml ## @@ -382,7 +382,7 @@ -posix +gnu Review comment: Please address the comments I have put onto root parent pom. None of them are addressed. You can basically compare things with current master and revert the changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] djkevincr commented on a change in pull request #186: [GORA-527] Implement a data store for redis
djkevincr commented on a change in pull request #186: [GORA-527] Implement a data store for redis URL: https://github.com/apache/gora/pull/186#discussion_r316498705 ## File path: pom.xml ## @@ -382,7 +382,7 @@ -posix +gnu Review comment: Please address the comments I have put onto root parent pom. Non of them are addressed. You can basically compare things with current master and revert the changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] djkevincr commented on a change in pull request #186: [GORA-527] Implement a data store for redis
djkevincr commented on a change in pull request #186: [GORA-527] Implement a
data store for redis
URL: https://github.com/apache/gora/pull/186#discussion_r316496181
##
File path:
gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java
##
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.redis.query;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.impl.ResultBase;
+import org.apache.gora.redis.store.RedisStore;
+import org.apache.gora.store.DataStore;
+
+/**
+ * Redis specific implementation of the {@link org.apache.gora.query.Result}
+ * interface.
+ */
+public class RedisResult extends ResultBase
{
+
+ private Iterator range;
+ private final int size;
+
+ /**
+ * Gets the data store used
+ *
+ * @return
+ */
+ @Override
+ public RedisStore getDataStore() {
+return (RedisStore) super.getDataStore();
+ }
+
+ /**
+ * @param dataStore
+ * @param query
+ * @param rg
+ */
+ public RedisResult(DataStore dataStore, Query query,
Collection rg) {//, Scanner scanner) {
+super(dataStore, query);
+this.size = rg.size();
+this.range = rg.iterator();
+ }
+
+ /**
+ * Gets the items reading progress
+ *
+ * @return
+ * @throws java.io.IOException
+ */
+ @Override
+ public float getProgress() throws IOException {
+if (this.limit != -1) {
+ return (float) this.offset / (float) this.limit;
+} else {
+ return 0;
+}
+ }
+
+ @Override
+ public void close() throws IOException {
+ }
+
+ /**
+ * Gets the next item
+ *
+ * @return
+ * @throws java.io.IOException
+ */
+ @Override
+ protected boolean nextInner() throws IOException {
+if (range == null) {
+ return false;
+}
+boolean next = range.hasNext();
+if (next) {
+ String nextkey = range.next();
+ key = (K) nextkey;
Review comment:
@cuent So if I my understanding is correct. We currently do not have support
for keys other than String right? Clearly the below type cast will end up in
exception if K == java.lang.Long
```
String nextkey = range.next();
key = (K) nextkey;
```
And on the other hand I don't see any usage/import for ScoredSortedList in
current code.
This is perfectly fine, I just wanted to clear all the doubts on where are
we now. Please document all the limitations. Please document these and open
tickets clearly mentioned cases where we further improved with details like
above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] djkevincr commented on a change in pull request #186: [GORA-527] Implement a data store for redis
djkevincr commented on a change in pull request #186: [GORA-527] Implement a
data store for redis
URL: https://github.com/apache/gora/pull/186#discussion_r316492084
##
File path: gora-redis/src/main/java/org/apache/gora/redis/query/RedisQuery.java
##
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.redis.query;
+
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.impl.QueryBase;
+import org.apache.gora.store.DataStore;
+
+/**
+ * Redis specific implementation of the {@link org.apache.gora.query.Query}
interface.
+ */
+public class RedisQuery extends QueryBase {
+
+ /**
+ * Constructor for the query
+ */
+ public RedisQuery() {
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] cuent commented on a change in pull request #186: [GORA-527] Implement a data store for redis
cuent commented on a change in pull request #186: [GORA-527] Implement a data
store for redis
URL: https://github.com/apache/gora/pull/186#discussion_r316485046
##
File path:
gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java
##
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.redis.query;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.impl.ResultBase;
+import org.apache.gora.redis.store.RedisStore;
+import org.apache.gora.store.DataStore;
+
+/**
+ * Redis specific implementation of the {@link org.apache.gora.query.Result}
+ * interface.
+ */
+public class RedisResult extends ResultBase
{
+
+ private Iterator range;
+ private final int size;
+
+ /**
+ * Gets the data store used
+ *
+ * @return
+ */
+ @Override
+ public RedisStore getDataStore() {
+return (RedisStore) super.getDataStore();
+ }
+
+ /**
+ * @param dataStore
+ * @param query
+ * @param rg
+ */
+ public RedisResult(DataStore dataStore, Query query,
Collection rg) {//, Scanner scanner) {
+super(dataStore, query);
+this.size = rg.size();
+this.range = rg.iterator();
+ }
+
+ /**
+ * Gets the items reading progress
+ *
+ * @return
+ * @throws java.io.IOException
+ */
+ @Override
+ public float getProgress() throws IOException {
+if (this.limit != -1) {
+ return (float) this.offset / (float) this.limit;
+} else {
+ return 0;
+}
+ }
+
+ @Override
+ public void close() throws IOException {
+ }
+
+ /**
+ * Gets the next item
+ *
+ * @return
+ * @throws java.io.IOException
+ */
+ @Override
+ protected boolean nextInner() throws IOException {
+if (range == null) {
+ return false;
+}
+boolean next = range.hasNext();
+if (next) {
+ String nextkey = range.next();
+ key = (K) nextkey;
Review comment:
Redis mainly supports two types of search by rank (Necessary for the
implementation of queries in Gora).
Details: https://redis.io/topics/indexes
1) Lexicographical indexes
Using LexSortedSet that only supports searches by String ranges
2) Simple numerical indexes with sorted sets
Using ScoredSortedList, search using a numeric score.
For simplicity, only support for the first type of index was added.
But to improve the code support for the second was added.
Under the following rules:
If the class of the Gora Key is of some type of numerical data (Long, Int,
...) the type of indexes 2) is used.
If the Gora Key class is of another type (String or other) the keys are cast
as String and index 1) is used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] cuent commented on a change in pull request #186: [GORA-527] Implement a data store for redis
cuent commented on a change in pull request #186: [GORA-527] Implement a data
store for redis
URL: https://github.com/apache/gora/pull/186#discussion_r316484889
##
File path: gora-redis/src/main/java/org/apache/gora/redis/query/RedisQuery.java
##
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.redis.query;
+
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.impl.QueryBase;
+import org.apache.gora.store.DataStore;
+
+/**
+ * Redis specific implementation of the {@link org.apache.gora.query.Query}
interface.
+ */
+public class RedisQuery extends QueryBase {
+
+ /**
+ * Constructor for the query
+ */
+ public RedisQuery() {
Review comment:
This constructor is needed, in the map reduce test, reflection is used and
if this constructor is removed, this error is obtained
`java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.NoSuchMethodException: org.apache.gora.redis.query.RedisQuery.
()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
Re: Final Report
Hi Sheriffo, Thanks for your great effort! 1) Could you separate charts for HBase and MongoDB? HBase charts suppress MongoDB ones. 2) Report says that: *"In this work, we have time to include only three gora data stores (MongoDB, HBase and CouchDB)"* However, you have not run this benchmark for CouchDB as far as I know? 3) I don't think there is a need to add commit hashes and messages as Appendix. Especially if we consider that hashes will be changed once the PR merged into the codebase. Kind Regards, Furkan KAMACI On Wed, Aug 21, 2019 at 7:42 PM Sheriffo Ceesay wrote: > All, > > My draft final report is available at > > https://cwiki.apache.org/confluence/display/GORA/Final+Report%3A+%5BGORA-532%5D+Benchmark+Module+For+Apache+Gora > > We have until 26th of this month submit the report. Please let me know if > you have any comments to improve it. > > Meanwhile, I will work on the documentation on how to run the benchmark > module and publish on gora website. > > Thank you. > > **Sheriffo Ceesay** >
Final Report
All, My draft final report is available at https://cwiki.apache.org/confluence/display/GORA/Final+Report%3A+%5BGORA-532%5D+Benchmark+Module+For+Apache+Gora We have until 26th of this month submit the report. Please let me know if you have any comments to improve it. Meanwhile, I will work on the documentation on how to run the benchmark module and publish on gora website. Thank you. **Sheriffo Ceesay**
[GitHub] [gora] djkevincr commented on a change in pull request #178: GORA-485 Apache Kudu datastore for Gora
djkevincr commented on a change in pull request #178: GORA-485 Apache Kudu
datastore for Gora
URL: https://github.com/apache/gora/pull/178#discussion_r316101724
##
File path: pom.xml
##
@@ -1642,16 +1676,16 @@
- org.apache.pig
- pig
- h2
- ${pig.version}
-
-
- org.apache.avro
- avro
-
-
+org.apache.pig
Review comment:
This is Apache Pig related dependency is already there in the parent pom.
Please take an update from master.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] djkevincr commented on a change in pull request #178: GORA-485 Apache Kudu datastore for Gora
djkevincr commented on a change in pull request #178: GORA-485 Apache Kudu datastore for Gora URL: https://github.com/apache/gora/pull/178#discussion_r316100408 ## File path: gora-kudu/pom.xml ## @@ -0,0 +1,189 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + 4.0.0 + + +org.apache.gora +gora +1.0-SNAPSHOT Review comment: Thank you for updating this to latest development iteration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] djkevincr commented on issue #186: [GORA-527] Implement a data store for redis
djkevincr commented on issue #186: [GORA-527] Implement a data store for redis URL: https://github.com/apache/gora/pull/186#issuecomment-523383201 @cuent This is excellent work and thank you for all the hard work/research you put into it. This PR is in great shape and there are several cases which pointed out in my review where you can further improve. Please have look on comments and update the PR. Once you do that, I will go ahead and proceed merging this to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] djkevincr commented on issue #186: [GORA-527] Implement a data store for redis
djkevincr commented on issue #186: [GORA-527] Implement a data store for redis URL: https://github.com/apache/gora/pull/186#issuecomment-523380046 mvn javadoc:aggregate - Its better if you can fix these warnings. ``` [WARNING] Javadoc Warnings [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java:40: warning: no description for @return [WARNING] * @return [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java:61: warning: no description for @return [WARNING] * @return [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java:62: warning: no description for @throws [WARNING] * @throws java.io.IOException [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java:80: warning: no description for @return [WARNING] * @return [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java:81: warning: no description for @throws [WARNING] * @throws java.io.IOException [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java:48: warning: no description for @param [WARNING] * @param dataStore [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java:49: warning: no description for @param [WARNING] * @param query [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/query/RedisResult.java:50: warning: no description for @param [WARNING] * @param rg [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java:99: warning: no description for @param [WARNING] * @param keyClass [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java:100: warning: no description for @param [WARNING] * @param persistentClass [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java:101: warning: no description for @param [WARNING] * @param properties [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java:102: warning: no description for @throws [WARNING] * @throws org.apache.gora.util.GoraException [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java:215: warning: no description for @throws [WARNING] * @throws org.apache.gora.util.GoraException [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java:232: warning: no description for @throws [WARNING] * @throws org.apache.gora.util.GoraException [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java:471: warning: no description for @param [WARNING] * @param query [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java:472: warning: no description for @return [WARNING] * @return [WARNING] ^ [WARNING] /home/djkevincr/apache_member/gora/gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java:473: warning: no description for @throws [WARNING] * @throws org.apache.gora.util.GoraException [WARNING] ^ [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 4.492 s [INFO] Finished at: 2019-08-21T15:05:46+05:30 [INFO] Final Memory: 31M/400M [INFO] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] djkevincr commented on issue #186: [GORA-527] Implement a data store for redis
djkevincr commented on issue #186: [GORA-527] Implement a data store for redis URL: https://github.com/apache/gora/pull/186#issuecomment-523379426 mvn clean install ``` [WARNING] Tests run: 43, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 30.902 s - in org.apache.gora.redis.store.RedisStoreClusterTest [INFO] [INFO] Results: [INFO] [WARNING] Tests run: 130, Failures: 0, Errors: 0, Skipped: 9 [INFO] [INFO] [INFO] --- maven-bundle-plugin:2.5.3:bundle (default-bundle) @ gora-redis --- [INFO] [INFO] --- maven-site-plugin:3.7.1:attach-descriptor (attach-descriptor) @ gora-redis --- [INFO] Skipping because packaging 'bundle' is not pom. [INFO] [INFO] [INFO] --- forbiddenapis:2.0:check (default) @ gora-redis --- [INFO] Scanning for classes to check... [INFO] Reading bundled API signatures: jdk-unsafe-1.8 [INFO] Reading bundled API signatures: jdk-deprecated-1.8 [INFO] Reading bundled API signatures: jdk-system-out [INFO] Loading classes to check... [INFO] Scanning classes for violations... [INFO] Scanned 10 (and 341 related) class file(s) for forbidden API invocations (in 0.15s), 0 error(s). [INFO] [INFO] --- forbiddenapis:2.0:testCheck (default) @ gora-redis --- [INFO] Scanning for classes to check... [INFO] Reading bundled API signatures: jdk-unsafe-1.8 [INFO] Reading bundled API signatures: jdk-deprecated-1.8 [INFO] Reading bundled API signatures: jdk-system-out [INFO] Loading classes to check... [INFO] Scanning classes for violations... [INFO] Scanned 7 (and 284 related) class file(s) for forbidden API invocations (in 0.07s), 0 error(s). [INFO] [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ gora-redis --- [INFO] Installing /home/djkevincr/apache_member/gora/gora-redis/target/gora-redis-0.9-SNAPSHOT.jar to /home/djkevincr/.m2/repository/org/apache/gora/gora-redis/0.9-SNAPSHOT/gora-redis-0.9-SNAPSHOT.jar [INFO] Installing /home/djkevincr/apache_member/gora/gora-redis/pom.xml to /home/djkevincr/.m2/repository/org/apache/gora/gora-redis/0.9-SNAPSHOT/gora-redis-0.9-SNAPSHOT.pom [INFO] [INFO] --- maven-bundle-plugin:2.5.3:install (default-install) @ gora-redis --- [INFO] Installing org/apache/gora/gora-redis/0.9-SNAPSHOT/gora-redis-0.9-SNAPSHOT.jar [INFO] Writing OBR metadata [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:01 min [INFO] Finished at: 2019-08-21T15:02:21+05:30 [INFO] Final Memory: 46M/794M [INFO] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
