[GitHub] [gora] cuent commented on issue #177: [GORA-527] Implement a data store for redis
cuent commented on issue #177: [GORA-527] Implement a data store for redis URL: https://github.com/apache/gora/pull/177#issuecomment-520171565 > @cuent Can you please take update/merge against current master ? I see so many files are added to the PR, Please add only the files where the changes are needed. Once you do that I ll have a look on your PR. I am not sure why those files are being added. In fact, they are showing with 0 changes, maybe it is because of a change on the hash of the branch history. I don't know how to solve that, @djkevincr do you have any idea? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] cuent commented on a change in pull request #177: [GORA-527] Implement a data store for redis
cuent commented on a change in pull request #177: [GORA-527] Implement a data
store for redis
URL: https://github.com/apache/gora/pull/177#discussion_r312712034
##
File path:
gora-redis/src/test/java/org/apache/gora/redis/GoraRedisTestDriver.java
##
@@ -0,0 +1,105 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.redis;
+
+import java.io.IOException;
+import java.time.Duration;
+import org.apache.gora.GoraTestDriver;
+import org.apache.gora.redis.store.RedisStore;
+import org.apache.gora.redis.util.RedisStartupLogWaitStrategy;
+import org.apache.gora.redis.util.ServerMode;
+import org.apache.gora.redis.util.StorageMode;
+import org.testcontainers.containers.FixedHostPortGenericContainer;
+
+/**
+ * Helper class to execute tests in a embedded instance of Redis.
+ *
+ * @author Xavier Sumba
+ */
+public class GoraRedisTestDriver extends GoraTestDriver {
+
+ private static final String DOCKER_CONTAINER_NAME =
"grokzen/redis-cluster:latest";
+ private FixedHostPortGenericContainer redisContainer =
((FixedHostPortGenericContainer) new
FixedHostPortGenericContainer(DOCKER_CONTAINER_NAME)
+ .waitingFor(new RedisStartupLogWaitStrategy())
+ .withStartupTimeout(Duration.ofMinutes(3))
+ .withEnv("STANDALONE", "true")
+ .withEnv("SENTINEL", "true"))
Review comment:
Unfortunately, port configuration for masters and slaves into the container
is static. So, if we do a forward from a port to a random, this reference will
be lost, and as a consequence, the connection won't be possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] cuent commented on a change in pull request #177: [GORA-527] Implement a data store for redis
cuent commented on a change in pull request #177: [GORA-527] Implement a data
store for redis
URL: https://github.com/apache/gora/pull/177#discussion_r312712055
##
File path: gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java
##
@@ -0,0 +1,519 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.redis.store;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.stream.Collectors;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import org.apache.avro.Schema;
+import org.apache.commons.io.IOUtils;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.redis.query.RedisQuery;
+import org.apache.gora.redis.query.RedisResult;
+import org.apache.gora.redis.util.DatumHandler;
+import org.apache.gora.redis.util.ServerMode;
+import org.apache.gora.redis.util.StorageMode;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.GoraException;
+import org.redisson.Redisson;
+import org.redisson.api.RBatch;
+import org.redisson.api.RBucket;
+import org.redisson.api.RBucketAsync;
+import org.redisson.api.RFuture;
+import org.redisson.api.RLexSortedSet;
+import org.redisson.api.RLexSortedSetAsync;
+import org.redisson.api.RList;
+import org.redisson.api.RListAsync;
+import org.redisson.api.RMap;
+import org.redisson.api.RMapAsync;
+import org.redisson.api.RedissonClient;
+import org.redisson.config.Config;
+import org.redisson.config.ReadMode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.NodeList;
+
+/**
+ * Implementation of a Redis data store to be used by gora.
+ *
+ * @param class to be used for the key
+ * @param class to be persisted within the store
+ */
+public class RedisStore extends DataStoreBase {
+
+ //Redis constants
+ private static final String FIELD_SEPARATOR = ".";
+ private static final String WILDCARD = "*";
+ private static final String INDEX = "index";
+ private static final String START_TAG = "{";
+ private static final String END_TAG = "}";
+ private static final String PREFIX = "redis://";
+
+ protected static final String MOCK_PROPERTY = "redis.mock";
Review comment:
yes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] cuent commented on a change in pull request #177: [GORA-527] Implement a data store for redis
cuent commented on a change in pull request #177: [GORA-527] Implement a data
store for redis
URL: https://github.com/apache/gora/pull/177#discussion_r312711903
##
File path: gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java
##
@@ -0,0 +1,519 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.redis.store;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.stream.Collectors;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import org.apache.avro.Schema;
+import org.apache.commons.io.IOUtils;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.redis.query.RedisQuery;
+import org.apache.gora.redis.query.RedisResult;
+import org.apache.gora.redis.util.DatumHandler;
+import org.apache.gora.redis.util.ServerMode;
+import org.apache.gora.redis.util.StorageMode;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.GoraException;
+import org.redisson.Redisson;
+import org.redisson.api.RBatch;
+import org.redisson.api.RBucket;
+import org.redisson.api.RBucketAsync;
+import org.redisson.api.RFuture;
+import org.redisson.api.RLexSortedSet;
+import org.redisson.api.RLexSortedSetAsync;
+import org.redisson.api.RList;
+import org.redisson.api.RListAsync;
+import org.redisson.api.RMap;
+import org.redisson.api.RMapAsync;
+import org.redisson.api.RedissonClient;
+import org.redisson.config.Config;
+import org.redisson.config.ReadMode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.NodeList;
+
+/**
+ * Implementation of a Redis data store to be used by gora.
+ *
+ * @param class to be used for the key
+ * @param class to be persisted within the store
+ */
+public class RedisStore extends DataStoreBase {
+
+ //Redis constants
+ private static final String FIELD_SEPARATOR = ".";
+ private static final String WILDCARD = "*";
+ private static final String INDEX = "index";
+ private static final String START_TAG = "{";
+ private static final String END_TAG = "}";
+ private static final String PREFIX = "redis://";
+
+ protected static final String MOCK_PROPERTY = "redis.mock";
+ protected static final String INSTANCE_NAME_PROPERTY = "redis.instance";
+ protected static final String ZOOKEEPERS_NAME_PROPERTY = "redis.zookeepers";
+ protected static final String USERNAME_PROPERTY = "redis.user";
+ protected static final String PASSWORD_PROPERTY = "redis.password";
+ protected static final String PARSE_MAPPING_FILE_KEY =
"gora.redis.mapping.file";
+ protected static final String DEFAULT_MAPPING_FILE =
"gora-redis-mapping.xml";
+ protected static final String XML_MAPPING_DEFINITION = "gora.mapping";
+ private RedissonClient redisInstance;
+ private RedisMapping mapping;
+ public static final Logger LOG = LoggerFactory.getLogger(RedisStore.class);
+
+ private static final DatumHandler handler = new DatumHandler();
+ private StorageMode mode;
+
+ /**
+ * Initialize the data store by reading the credentials, setting the client's
+ * properties up and reading the mapping file. Initialize is called when then
+ * the call to {@link org.apache.gora.store.DataStoreFactory#createDataStore}
+ * is made.
+ *
+ * @param keyClass
+ * @param persistentClass
+ * @param properties
+ */
+ @Override
+ public void initialize(Class keyClass, Class persistentClass,
Properties properties) throws GoraException {
+try {
+ super.initialize(keyClass, persistentClass, properties);
+
+ InputStream mappingStream;
+ if (properties.containsKey(XML_MAPPING_DEFINITION)) {
+if (LOG.isTraceEnabled()) {
+ LOG.trace(XML_MAPPING_DEFINITION + " = " +
[GitHub] [gora] cuent commented on a change in pull request #177: [GORA-527] Implement a data store for redis
cuent commented on a change in pull request #177: [GORA-527] Implement a data
store for redis
URL: https://github.com/apache/gora/pull/177#discussion_r312711903
##
File path: gora-redis/src/main/java/org/apache/gora/redis/store/RedisStore.java
##
@@ -0,0 +1,519 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.redis.store;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.stream.Collectors;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import org.apache.avro.Schema;
+import org.apache.commons.io.IOUtils;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.redis.query.RedisQuery;
+import org.apache.gora.redis.query.RedisResult;
+import org.apache.gora.redis.util.DatumHandler;
+import org.apache.gora.redis.util.ServerMode;
+import org.apache.gora.redis.util.StorageMode;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.GoraException;
+import org.redisson.Redisson;
+import org.redisson.api.RBatch;
+import org.redisson.api.RBucket;
+import org.redisson.api.RBucketAsync;
+import org.redisson.api.RFuture;
+import org.redisson.api.RLexSortedSet;
+import org.redisson.api.RLexSortedSetAsync;
+import org.redisson.api.RList;
+import org.redisson.api.RListAsync;
+import org.redisson.api.RMap;
+import org.redisson.api.RMapAsync;
+import org.redisson.api.RedissonClient;
+import org.redisson.config.Config;
+import org.redisson.config.ReadMode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.NodeList;
+
+/**
+ * Implementation of a Redis data store to be used by gora.
+ *
+ * @param class to be used for the key
+ * @param class to be persisted within the store
+ */
+public class RedisStore extends DataStoreBase {
+
+ //Redis constants
+ private static final String FIELD_SEPARATOR = ".";
+ private static final String WILDCARD = "*";
+ private static final String INDEX = "index";
+ private static final String START_TAG = "{";
+ private static final String END_TAG = "}";
+ private static final String PREFIX = "redis://";
+
+ protected static final String MOCK_PROPERTY = "redis.mock";
+ protected static final String INSTANCE_NAME_PROPERTY = "redis.instance";
+ protected static final String ZOOKEEPERS_NAME_PROPERTY = "redis.zookeepers";
+ protected static final String USERNAME_PROPERTY = "redis.user";
+ protected static final String PASSWORD_PROPERTY = "redis.password";
+ protected static final String PARSE_MAPPING_FILE_KEY =
"gora.redis.mapping.file";
+ protected static final String DEFAULT_MAPPING_FILE =
"gora-redis-mapping.xml";
+ protected static final String XML_MAPPING_DEFINITION = "gora.mapping";
+ private RedissonClient redisInstance;
+ private RedisMapping mapping;
+ public static final Logger LOG = LoggerFactory.getLogger(RedisStore.class);
+
+ private static final DatumHandler handler = new DatumHandler();
+ private StorageMode mode;
+
+ /**
+ * Initialize the data store by reading the credentials, setting the client's
+ * properties up and reading the mapping file. Initialize is called when then
+ * the call to {@link org.apache.gora.store.DataStoreFactory#createDataStore}
+ * is made.
+ *
+ * @param keyClass
+ * @param persistentClass
+ * @param properties
+ */
+ @Override
+ public void initialize(Class keyClass, Class persistentClass,
Properties properties) throws GoraException {
+try {
+ super.initialize(keyClass, persistentClass, properties);
+
+ InputStream mappingStream;
+ if (properties.containsKey(XML_MAPPING_DEFINITION)) {
+if (LOG.isTraceEnabled()) {
+ LOG.trace(XML_MAPPING_DEFINITION + " = " +
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711875
##
File path: gora-jet/src/main/java/org/apache/gora/jet/JetSource.java
##
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.jet.Traverser;
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.ProcessorSupplier;
+import com.hazelcast.nio.Address;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Result;
+
+import javax.annotation.Nonnull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+import static com.hazelcast.jet.Traversers.traverseIterable;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.IntStream.range;
+
+/**
+ * jet-source implementation.
+ */
+public class JetSource implements
ProcessorMetaSupplier {
+
+ private int totalParallelism;
+ private transient int localParallelism;
+
+ @Override
+ public void init(@Nonnull Context context) {
+totalParallelism = context.totalParallelism();
+localParallelism = context.localParallelism();
+ }
+
+ @Nonnull
+ @Override
+ public Function get(@Nonnull
List addresses) {
+Map map = new HashMap<>();
+for (int i = 0; i < addresses.size(); i++) {
+ // We'll calculate the global index of each processor in the cluster:
+ //globalIndexBase is the first processor index in a certain Jet-Cluster
member
+ int globalIndexBase = localParallelism * i;
+
+ // processorCount will be equal to localParallelism:
+ ProcessorSupplier supplier = processorCount ->
+ range(globalIndexBase, globalIndexBase + processorCount)
+ .mapToObj(globalIndex ->
+ new GoraJetProcessor(getPartionedData(globalIndex))
+ ).collect(toList());
+ map.put(addresses.get(i), supplier);
+}
+return map::get;
+ }
+
+ List> getPartionedData(int globalIndex)
{
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711871
##
File path: gora-jet/src/main/java/org/apache/gora/jet/JetSource.java
##
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.jet.Traverser;
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.ProcessorSupplier;
+import com.hazelcast.nio.Address;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Result;
+
+import javax.annotation.Nonnull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+import static com.hazelcast.jet.Traversers.traverseIterable;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.IntStream.range;
+
+/**
+ * jet-source implementation.
+ */
+public class JetSource implements
ProcessorMetaSupplier {
+
+ private int totalParallelism;
+ private transient int localParallelism;
+
+ @Override
+ public void init(@Nonnull Context context) {
+totalParallelism = context.totalParallelism();
+localParallelism = context.localParallelism();
+ }
+
+ @Nonnull
+ @Override
+ public Function get(@Nonnull
List addresses) {
+Map map = new HashMap<>();
+for (int i = 0; i < addresses.size(); i++) {
+ // We'll calculate the global index of each processor in the cluster:
+ //globalIndexBase is the first processor index in a certain Jet-Cluster
member
+ int globalIndexBase = localParallelism * i;
+
+ // processorCount will be equal to localParallelism:
+ ProcessorSupplier supplier = processorCount ->
+ range(globalIndexBase, globalIndexBase + processorCount)
+ .mapToObj(globalIndex ->
+ new GoraJetProcessor(getPartionedData(globalIndex))
+ ).collect(toList());
+ map.put(addresses.get(i), supplier);
+}
+return map::get;
+ }
+
+ List> getPartionedData(int globalIndex)
{
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711881
##
File path: gora-jet/src/main/java/org/apache/gora/jet/JetSource.java
##
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.jet.Traverser;
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.ProcessorSupplier;
+import com.hazelcast.nio.Address;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Result;
+
+import javax.annotation.Nonnull;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+import static com.hazelcast.jet.Traversers.traverseIterable;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.IntStream.range;
+
+/**
+ * jet-source implementation.
+ */
+public class JetSource implements
ProcessorMetaSupplier {
+
+ private int totalParallelism;
+ private transient int localParallelism;
+
+ @Override
+ public void init(@Nonnull Context context) {
+totalParallelism = context.totalParallelism();
+localParallelism = context.localParallelism();
+ }
+
+ @Nonnull
+ @Override
+ public Function get(@Nonnull
List addresses) {
+Map map = new HashMap<>();
+for (int i = 0; i < addresses.size(); i++) {
+ // We'll calculate the global index of each processor in the cluster:
+ //globalIndexBase is the first processor index in a certain Jet-Cluster
member
+ int globalIndexBase = localParallelism * i;
+
+ // processorCount will be equal to localParallelism:
+ ProcessorSupplier supplier = processorCount ->
+ range(globalIndexBase, globalIndexBase + processorCount)
+ .mapToObj(globalIndex ->
+ new GoraJetProcessor(getPartionedData(globalIndex))
+ ).collect(toList());
+ map.put(addresses.get(i), supplier);
+}
+return map::get;
+ }
+
+ List> getPartionedData(int globalIndex)
{
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711890
##
File path: gora-jet/src/test/java/org/apache/gora/jet/JetTest.java
##
@@ -0,0 +1,149 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.core.IMap;
+import com.hazelcast.jet.Jet;
+import com.hazelcast.jet.JetInstance;
+import com.hazelcast.jet.pipeline.BatchSource;
+import com.hazelcast.jet.pipeline.Pipeline;
+import com.hazelcast.jet.pipeline.Sinks;
+import org.apache.gora.jet.generated.Pageview;
+import org.apache.gora.jet.generated.ResultPageView;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.regex.Pattern;
+
+import static com.hazelcast.jet.Traversers.traverseArray;
+import static com.hazelcast.jet.aggregate.AggregateOperations.counting;
+import static com.hazelcast.jet.function.Functions.wholeItem;
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Test case for jet sink and source connectors.
+ */
+public class JetTest {
+
+ private static DataStore dataStore;
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711885
##
File path: gora-jet/src/test/java/org/apache/gora/jet/JetTest.java
##
@@ -0,0 +1,149 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.core.IMap;
+import com.hazelcast.jet.Jet;
+import com.hazelcast.jet.JetInstance;
+import com.hazelcast.jet.pipeline.BatchSource;
+import com.hazelcast.jet.pipeline.Pipeline;
+import com.hazelcast.jet.pipeline.Sinks;
+import org.apache.gora.jet.generated.Pageview;
+import org.apache.gora.jet.generated.ResultPageView;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.regex.Pattern;
+
+import static com.hazelcast.jet.Traversers.traverseArray;
+import static com.hazelcast.jet.aggregate.AggregateOperations.counting;
+import static com.hazelcast.jet.function.Functions.wholeItem;
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Test case for jet sink and source connectors.
+ */
+public class JetTest {
+
+ private static DataStore dataStore;
+ private static DataStore dataStoreOut;
+ static Query query = null;
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711862
##
File path: gora-jet/src/main/java/org/apache/gora/jet/JetSink.java
##
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.ProcessorSupplier;
+import com.hazelcast.nio.Address;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.util.GoraException;
+
+import javax.annotation.Nonnull;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.IntStream.range;
+
+/**
+ * jet-sink implementation.
+ */
+public class JetSink implements
ProcessorMetaSupplier {
+
+ private transient int localParallelism;
+
+ @Override
+ public void init(@Nonnull Context context) {
+localParallelism = context.localParallelism();
+ }
+
+ @Nonnull
+ @Override
+ public Function get(@Nonnull
List addresses) {
+Map map = new HashMap<>();
+for (int i = 0; i < addresses.size(); i++) {
+ //globalIndexBase is the first processor index in a certain Jet-Cluster
member
+ int globalIndexBase = localParallelism * i;
+
+ // processorCount will be equal to localParallelism:
+ ProcessorSupplier supplier = processorCount ->
+ range(globalIndexBase, globalIndexBase + processorCount)
+ .mapToObj(globalIndex ->
+ new SinkProcessor()
+ ).collect(toList());
+ map.put(addresses.get(i), supplier);
+}
+return map::get;
+ }
+}
+
+class SinkProcessor extends
AbstractProcessor {
+
+ @Override
+ public boolean isCooperative() {
+return false;
+ }
+
+ @Override
+ protected boolean tryProcess(int ordinal, Object item) throws Exception {
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711846
##
File path: gora-jet/src/main/java/org/apache/gora/jet/JetEngine.java
##
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.jet.pipeline.BatchSource;
+import com.hazelcast.jet.pipeline.Sink;
+import com.hazelcast.jet.pipeline.Sinks;
+import com.hazelcast.jet.pipeline.Sources;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.store.DataStore;
+
+/**
+ * Core class which handles Gora - Jet Engine integration.
+ */
+public class JetEngine {
+ public static DataStore dataOutStore;
+ public static DataStore dataInStore;
+ public static Query query;
+
+ public BatchSource>
createDataSource(DataStore dataOutStore) {
+return createDataSource(dataOutStore, dataOutStore.newQuery());
+ }
+
+ public BatchSource>
createDataSource(DataStore dataOutStore,
+
Query query) {
+JetEngine.dataInStore = dataOutStore;
+JetEngine.query = query;
+BatchSource> source =
Sources.batchFromProcessor("gora-jet-source",
+new JetSource());
+return source;
+ }
+
+ public Sink>
createDataSink(DataStore dataOutStore) {
+JetEngine.dataOutStore = dataOutStore;
+Sink> sink =
Sinks.fromProcessor("gora-jet-sink",
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711854
##
File path: gora-jet/src/main/java/org/apache/gora/jet/JetEngine.java
##
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.jet.pipeline.BatchSource;
+import com.hazelcast.jet.pipeline.Sink;
+import com.hazelcast.jet.pipeline.Sinks;
+import com.hazelcast.jet.pipeline.Sources;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.store.DataStore;
+
+/**
+ * Core class which handles Gora - Jet Engine integration.
+ */
+public class JetEngine {
+ public static DataStore dataOutStore;
+ public static DataStore dataInStore;
+ public static Query query;
+
+ public BatchSource>
createDataSource(DataStore dataOutStore) {
+return createDataSource(dataOutStore, dataOutStore.newQuery());
+ }
+
+ public BatchSource>
createDataSource(DataStore dataOutStore,
+
Query query) {
+JetEngine.dataInStore = dataOutStore;
+JetEngine.query = query;
+BatchSource> source =
Sources.batchFromProcessor("gora-jet-source",
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711860
##
File path: gora-jet/src/main/java/org/apache/gora/jet/JetInputOutputFormat.java
##
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import org.apache.gora.persistency.impl.PersistentBase;
+
+/**
+ * Wrapper class which will be used to fetch data from data stores to Gora-
+ * jet-source and to write data into data stores through Gora-jet-sink.
+ */
+public class JetInputOutputFormat {
+ public KeyOut key;
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711838
##
File path: gora-jet/src/main/java/org/apache/gora/jet/JetEngine.java
##
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.jet.pipeline.BatchSource;
+import com.hazelcast.jet.pipeline.Sink;
+import com.hazelcast.jet.pipeline.Sinks;
+import com.hazelcast.jet.pipeline.Sources;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.store.DataStore;
+
+/**
+ * Core class which handles Gora - Jet Engine integration.
+ */
+public class JetEngine {
+ public static DataStore dataOutStore;
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support URL: https://github.com/apache/gora/pull/175#discussion_r312711837 ## File path: gora-jet/pom.xml ## @@ -0,0 +1,132 @@ + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.gora +gora +0.9-SNAPSHOT +../ + + +gora-jet +bundle + +Apache Gora :: Jet +http://gora.apache.org +Jet -> Gora -> Jet Sink and Source connectors +2019 Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast Jet execution engine support
LahiruJayasekara commented on a change in pull request #175: GORA-546 Hazelcast
Jet execution engine support
URL: https://github.com/apache/gora/pull/175#discussion_r312711418
##
File path: gora-jet/src/main/java/org/apache/gora/jet/JetEngine.java
##
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.jet;
+
+import com.hazelcast.jet.pipeline.BatchSource;
+import com.hazelcast.jet.pipeline.Sink;
+import com.hazelcast.jet.pipeline.Sinks;
+import com.hazelcast.jet.pipeline.Sources;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.store.DataStore;
+
+/**
+ * Core class which handles Gora - Jet Engine integration.
+ */
+public class JetEngine {
+ public static DataStore dataOutStore;
+ public static DataStore dataInStore;
+ public static Query query;
+
+ public BatchSource>
createDataSource(DataStore dataOutStore) {
+return createDataSource(dataOutStore, dataOutStore.newQuery());
+ }
+
+ public BatchSource>
createDataSource(DataStore dataOutStore,
Review comment:
actually these methods are the methods exposed to end user to create
jet-source and sink.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [gora] djkevincr commented on issue #177: [GORA-527] Implement a data store for redis
djkevincr commented on issue #177: [GORA-527] Implement a data store for redis URL: https://github.com/apache/gora/pull/177#issuecomment-520156768 @cuent Can you please take update/merge against current master ? I see so many files are added to the PR, Please add only the files where the changes are needed. Once you do that I ll have a look on your PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on issue #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on issue #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#issuecomment-520146033 @lewismc, so what's the way forward regarding the YCSB dependency issue? I have updated my local dependency to the latest version of YCSB i.e. 0.15.0 and updated my code accordingly. I have added these changes to this pull request. Right now the simplest way to go around this without maven is to: 1. Download YCSB from [1] and copy all the jar files in `ycsb-0.15.0/lib` directory to `gora-benchmark/lib`. I have tested it and it is working with both MongoDB and HBase. 2. Copy `ycsb-0.15.0/lib/core-0.15.0.jar` to ~/.m2/repository/com/yahoo/ycsb/core/0.15.0/ 3. mvn clean install [1] https://github.com/brianfrankcooper/YCSB/releases/download/0.15.0/ycsb-0.15.0.tar.gz @djkevincr, I have updated your comments in the latest commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [gora] sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments
sneceesay77 commented on a change in pull request #179: GORA-532: Apache Gora Benchmark initial pull request for review and comments URL: https://github.com/apache/gora/pull/179#discussion_r312701363 ## File path: gora-benchmark/README.md ## @@ -0,0 +1,48 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +This guidline outlines the process of compiling and running Gora Benchmark Module + +Prerequisite +Install and setup the required database to test. Currently, the benchmark is tested on MongoDB, HBase and CouchDB. +Install maven + +Set up + +1. Edit the gora.properties file located at src/main/resources/gora.properties and set configurations accordingly. + + a. To test MongoDB set gora.datastore.default=org.apache.gora.mongodb.store.MongoStore + b. To test HBase set gora.datastore.default=org.apache.gora.hbase.store.HBaseStore + c. To test CouchDB set gora.datastore.default=org.apache.gora.couchdb.store.CouchDBStore + + +2. From the module directory i.e. gora-benchmark, run mvn clean install +3. This should run some test and successfully build the module. + If the build fails and complains about maven not able to ycsb-core-verion.jar then download the jar manually and add it to ~/.m2/repository/com/yahoo/ycsb/core/0.1.4/ Review comment: We are now working with YCSB to resolve this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
