[GitHub] incubator-rya pull request #307: Add rep-synch functionality
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/307 Add rep-synch functionality ## Description >What Changed? The parent can now refresh/synch the child. Added GUI entry point into the rep/synch tool. The functionality is now more thread safe. ### Tests >Coverage? Updated tests ### Links ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @kchilton2 @ejwhite922 @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya repsynch_updatingChild Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/307.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #307 commit 05ab6a745425a2ef9505c1c111eb8222982830d5 Author: Andrew Smith Date: 2018-06-26T16:38:34Z Add rep-synch functionality The parent can now refresh/synch the child. Added GUI entry point into the rep/synch tool. The functionality is now more thread safe. ---
[GitHub] incubator-rya pull request #306: Repsynch updating child
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/306 ---
[GitHub] incubator-rya pull request #306: Repsynch updating child
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/306 Repsynch updating child ## Description >What Changed? Each type of rep/synch is its own runnable now to be more thread safe. A GUI tool was created as an entry point into rep/synch instead of CLI. ### Tests >Coverage? updated the tests. ### Links ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @kchilton2 @pujav65 @ejwhite922 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya repsynch_updatingChild Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/306.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #306 commit 04a4910ee01d924abb3f1ac187a666f3a2433a6f Author: Andrew Smith Date: 2018-06-26T16:38:34Z Fixes made the Merge Tool Conflicts: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBRyaDAO.java extras/rya.export/export.client/src/main/java/org/apache/rya/export/client/MergeDriverClient.java extras/rya.export/export.client/src/main/java/org/apache/rya/export/client/conf/MergeConfigHadoopAdapter.java extras/rya.export/export.client/src/main/java/org/apache/rya/export/client/merge/MemoryTimeMerger.java commit c6dbc8ce21fc33d613076aa148f6a8dbd24f3438 Author: Andrew Smith Date: 2018-09-18T19:21:57Z Add rep-synch functionality The parent can now refresh/synch the child. Added GUI entry point into the rep/synch tool. The functionality is now more thread safe. ---
[GitHub] incubator-rya pull request #304: RYA-501 Change guava Iterators to Collectio...
GitHub user isper3at reopened a pull request: https://github.com/apache/incubator-rya/pull/304 RYA-501 Change guava Iterators to Collections ## Description >What Changed? Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ### Tests >Coverage? N/A ### Links [Jira](https://issues.apache.org/jira/browse/RYA-501) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @ejwhite922 @kchilton2 @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-501_guavaToJava-4.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/304.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #304 commit 93c1d89dafc16d4d50bc4581ec6ada311ca494ae Author: Andrew Smith Date: 2018-09-14T16:37:16Z RYA-501 Change guava Iterators to Collections Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ---
[GitHub] incubator-rya pull request #305: Rya 135 collection name
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/305 Rya 135 collection name ## Description >What Changed? Merging existing content into master. Already reviewed and merged into 3.x You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-135_collectionName Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/305.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #305 commit 942d9d62350815a33f28a22f101cdb5c262c70ed Author: Andrew Smith Date: 2018-09-14T16:33:10Z RYA-135 Hard code triples collection There is no need for the triples collection to be configurable, as only the mongo database name will change the rya instance. commit 010f40de276b2152451a42e1c830a2ad88b94860 Author: Andrew Smith Date: 2018-09-14T21:06:27Z Responded to code review removed all references to mongo collection prefix commit 3593c535ddf387d656cde87e793be7bb8980289e Author: Andrew Smith Date: 2018-09-14T22:19:56Z More code review ---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/300 ---
[GitHub] incubator-rya pull request #302: RYA-501 Change guava Iterators to Collectio...
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/302 ---
[GitHub] incubator-rya issue #304: RYA-501 Change guava Iterators to Collections
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/304 Closing until the #302 is in ---
[GitHub] incubator-rya issue #303: RYA-135 Hard code triples collection
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/303 opening a new PR against RYA master after 3.x https://github.com/apache/incubator-rya/pull/300 is in ---
[GitHub] incubator-rya pull request #303: RYA-135 Hard code triples collection
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/303 ---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/300#discussion_r217849654 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/AbstractMongoIndexer.java --- @@ -73,12 +72,12 @@ protected T storageStrategy; private MongoDbBatchWriter mongoDbBatchWriter; +protected String collectionName; --- End diff -- getCollectionName() would build the collection name with the collection suffix and rya prefix ---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/300#discussion_r217849215 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/AbstractMongoIndexer.java --- @@ -73,12 +72,12 @@ protected T storageStrategy; private MongoDbBatchWriter mongoDbBatchWriter; +protected String collectionName; --- End diff -- yes, the implementations of AbstractMongoIndexer set it in the init() function. I can't have it in the constructor because of the hadoop/reflexive construction ---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/300#discussion_r217847761
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBRdfConfiguration.java
---
@@ -197,22 +195,22 @@ public void setMongoPassword(final String password) {
* @return The name of the Rya instance to connect to. (default: rya)
*/
public String getRyaInstanceName() {
-return get(MONGO_COLLECTION_PREFIX, "rya");
+return get(MONGO_DB_NAME, "rya");
--- End diff --
yeah that's a good point
---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/300#discussion_r217845838
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBRdfConfiguration.java
---
@@ -274,16 +274,17 @@ public boolean getUseAggregationPipeline() {
* on their child subtrees.
* @param value whether to use aggregation pipeline optimization.
*/
-public void setUseAggregationPipeline(boolean value) {
+public void setUseAggregationPipeline(final boolean value) {
setBoolean(USE_AGGREGATION_PIPELINE, value);
}
@Override
public List> getOptimizers() {
-List> optimizers = super.getOptimizers();
+final List> optimizers =
super.getOptimizers();
if (getUseAggregationPipeline()) {
-Class cl = AggregationPipelineQueryOptimizer.class;
+final Class cl = AggregationPipelineQueryOptimizer.class;
@SuppressWarnings("unchecked")
+final
--- End diff --
fixed
---
[GitHub] incubator-rya pull request #304: RYA-501 Change guava Iterators to Collectio...
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/304 RYA-501 Change guava Iterators to Collections ## Description >What Changed? Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ### Tests >Coverage? N/A ### Links [Jira](https://issues.apache.org/jira/browse/RYA-501) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @ejwhite922 @kchilton2 @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-501_guavaToJava-4.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/304.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #304 commit 93c1d89dafc16d4d50bc4581ec6ada311ca494ae Author: Andrew Smith Date: 2018-09-14T16:37:16Z RYA-501 Change guava Iterators to Collections Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ---
[GitHub] incubator-rya issue #301: RYA-501 change from Guava Iterators to Java Collec...
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/301 changed to -> https://github.com/apache/incubator-rya/pull/302 ---
[GitHub] incubator-rya pull request #302: RYA-501 Change guava Iterators to Collectio...
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/302 RYA-501 Change guava Iterators to Collections ## Description >What Changed? Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ### Tests >Coverage? N/A ### Links [Jira](https://issues.apache.org/jira/browse/RYA-501) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @pujav65 @kchilton2 @ejwhite922 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-501_guavaToJava Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/302.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #302 commit 03fd52e89b44fd39ae97490f43d4922b39e7d07d Author: Andrew Smith Date: 2018-09-14T16:37:16Z RYA-501 Change guava Iterators to Collections Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ---
[GitHub] incubator-rya pull request #301: RYA-501 change from Guava Iterators to Java...
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/301 ---
[GitHub] incubator-rya pull request #301: Version fix
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/301 Version fix ## Description >What Changed? Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. ### Tests >Coverage? ### Links ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @pujav65 @kchilton2 @ejwhite922 @jessehatfield You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya versionFix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/301.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #301 commit ef38e2efe45699d9916e3bb0696a8cfb6b6a171d Author: Andrew Smith Date: 2018-09-14T16:37:16Z Version fix Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187739974
--- Diff:
extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/sink/RyaSinkTask.java
---
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.api.sink;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.errors.ConnectException;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.eclipse.rdf4j.model.Statement;
+import org.eclipse.rdf4j.repository.sail.SailRepository;
+import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection;
+import org.eclipse.rdf4j.sail.Sail;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.jcabi.manifests.Manifests;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+
+/**
+ * Handles the common components required to write {@link Statement}s to
Rya.
+ *
+ * Implementations of this class only need to specify functionality that
is specific to the
+ * Rya implementation.
+ */
+@DefaultAnnotation(NonNull.class)
+public abstract class RyaSinkTask extends SinkTask {
+private static final Logger log =
LoggerFactory.getLogger(RyaSinkTask.class);
+
+@Nullable
+private SailRepository sailRepo = null;
+
+@Nullable
+private SailRepositoryConnection conn = null;
+
+/**
+ * Throws an exception if the configured Rya Instance is not already
installed
+ * within the configured database.
+ *
+ * @param taskConfig - The configuration values that were provided to
the task. (not null)
+ * @throws ConnectException The configured Rya Instance is not
installed to the configured database
+ * or we were unable to figure out if it is installed.
+ */
+protected abstract void checkRyaInstanceExists(final Map<String,
String> taskConfig) throws ConnectException;
+
+/**
+ * Creates an initialized {@link Sail} object that may be used to
write {@link Statement}s to the configured
+ * Rya Instance.
+ *
+ * @param taskConfig - Configures how the Sail object will be created.
(not null)
+ * @return The created Sail object.
+ * @throws ConnectException The Sail object could not be made.
+ */
+protected abstract Sail makeSail(final Map<String, String> taskConfig)
throws ConnectException;
+
+@Override
+public String version() {
+return Manifests.exists("Build-Version") ?
Manifests.read("Build-Version"): "UNKNOWN";
+}
+
+@Override
+public void start(final Map<String, String> props) throws
ConnectException {
+requireNonNull(props);
+
+// Ensure the configured Rya Instance is installed within the
configured database.
+checkRyaInstanceExists(props);
+
+// Create the Sail object that is connected to the Rya Instance.
+final Sail sail = makeSail(props);
+sailRepo = new SailRepository( sail );
+conn = sailRepo.getConnection();
+}
+
+@Override
+public void put(final Collection records) {
+requireNonNull(records);
+
+// Return immediately if there are no records to handle.
+if(records.isEmpty()) {
+return;
+}
+
+// If a transaction has not been started yet, then start
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187739885
--- Diff:
extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/sink/RyaSinkTask.java
---
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.api.sink;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.errors.ConnectException;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.eclipse.rdf4j.model.Statement;
+import org.eclipse.rdf4j.repository.sail.SailRepository;
+import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection;
+import org.eclipse.rdf4j.sail.Sail;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.jcabi.manifests.Manifests;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+
+/**
+ * Handles the common components required to write {@link Statement}s to
Rya.
+ *
+ * Implementations of this class only need to specify functionality that
is specific to the
+ * Rya implementation.
+ */
+@DefaultAnnotation(NonNull.class)
--- End diff --
gotcha, thanks for the doc paste dump
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187739762
--- Diff:
extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/StatementsSerializer.java
---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.api;
+
+import java.io.ByteArrayOutputStream;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.eclipse.rdf4j.model.Statement;
+import org.eclipse.rdf4j.rio.RDFWriter;
+import org.eclipse.rdf4j.rio.binary.BinaryRDFWriterFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * A Kafka {@link Serializer} that is able to serialize a set of {@link
Statement}s
+ * using the RDF4J Rio Binary format.
--- End diff --
fair enough, it just seemed like you are delegating the actual
serialization to something else.
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187739297
--- Diff:
extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/ConfigUtils.java
---
@@ -438,6 +438,9 @@ public static boolean getUsePcjUpdaterIndex(final
Configuration conf) {
return Optional.fromNullable(conf.get(FLUO_APP_NAME));
}
+public static void setUseMongo(final Configuration conf, final boolean
useMongo) {
--- End diff --
sure makes sense
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187739254
--- Diff:
dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/AccumuloRdfConfiguration.java
---
@@ -62,14 +62,14 @@ public AccumuloRdfConfiguration() {
super();
}
-public AccumuloRdfConfiguration(Configuration other) {
+public AccumuloRdfConfiguration(final Configuration other) {
super(other);
}
-public AccumuloRdfConfigurationBuilder getBuilder() {
+public static AccumuloRdfConfigurationBuilder getBuilder() {
--- End diff --
ð
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187717175
--- Diff:
extras/kafka.connect/mongo-it/src/test/java/org/apache/rya/kafka/connect/mongo/MongoRyaSinkTaskIT.java
---
@@ -0,0 +1,97 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.mongo;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.kafka.connect.errors.ConnectException;
+import org.apache.rya.api.client.Install.InstallConfiguration;
+import org.apache.rya.api.client.RyaClient;
+import org.apache.rya.api.client.mongo.MongoConnectionDetails;
+import org.apache.rya.api.client.mongo.MongoRyaClientFactory;
+import org.apache.rya.test.mongo.MongoITBase;
+import org.junit.Test;
+
+/**
+ * Integration tests the methods of {@link MongoRyaSinkTask}.
+ */
+public class MongoRyaSinkTaskIT extends MongoITBase {
+
+@Test
+public void instanceExists() throws Exception {
+// Install an instance of Rya.
+final String ryaInstanceName = "rya";
+final MongoConnectionDetails connectionDetails = new
MongoConnectionDetails(
+super.getMongoHostname(),
+super.getMongoPort(),
+Optional.empty(),
+Optional.empty());
+
+final InstallConfiguration installConfig =
InstallConfiguration.builder()
+.setEnableTableHashPrefix(false)
+.setEnableEntityCentricIndex(false)
+.setEnableFreeTextIndex(false)
+.setEnableTemporalIndex(false)
+.setEnablePcjIndex(false)
+.setEnableGeoIndex(false)
+.build();
+
+final RyaClient ryaClient =
MongoRyaClientFactory.build(connectionDetails, super.getMongoClient());
+ryaClient.getInstall().install(ryaInstanceName, installConfig);
+
+// Create the task that will be tested.
+final MongoRyaSinkTask task = new MongoRyaSinkTask();
+
+try {
+// Configure the task to use the embedded Mongo DB instance
for Rya.
+final Map<String, String> config = new HashMap<>();
+config.put(MongoRyaSinkConfig.HOSTNAME,
super.getMongoHostname());
+config.put(MongoRyaSinkConfig.PORT, "" + super.getMongoPort());
+config.put(MongoRyaSinkConfig.RYA_INSTANCE_NAME, "rya");
+
+// This will pass because the Rya instance exists.
+task.start(config);
+} finally {
+task.stop();
+}
+}
+
+@Test(expected = ConnectException.class)
+public void instanceDoesNotExist() throws Exception {
+// Create the task that will be tested.
+final MongoRyaSinkTask task = new MongoRyaSinkTask();
+
+try {
+// Configure the task to use the embedded Mongo DB instance
for Rya.
+final Map<String, String> config = new HashMap<>();
+config.put(MongoRyaSinkConfig.HOSTNAME,
super.getMongoHostname());
+config.put(MongoRyaSinkConfig.PORT, "" + super.getMongoPort());
+config.put(MongoRyaSinkConfig.RYA_INSTANCE_NAME,
"instance-does-not-exist");
+
+// Starting the task will fail because the Rya instance does
not exist.
+task.start(config);
+} finally {
+task.stop();
+}
+}
+
+// TODO show that inserts using visibilities work.
--- End diff --
TODO?
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187717488
--- Diff:
extras/kafka.connect/mongo/src/main/java/org/apache/rya/kafka/connect/mongo/MongoRyaSinkConfig.java
---
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.mongo;
+
+import java.util.Map;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigDef.Importance;
+import org.apache.kafka.common.config.ConfigDef.Type;
+import org.apache.rya.kafka.connect.api.sink.RyaSinkConfig;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * A Kafka Connect configuration that is used to configure {@link
MongoRyaSinkConnector}s and {@link MongoRyaSinkTask}s.
+ */
+@DefaultAnnotation(NonNull.class)
+public class MongoRyaSinkConfig extends RyaSinkConfig {
+
+public static final String HOSTNAME = "mongo.hostname";
+private static final String HOSTNAME_DOC = "The Mongo DB hostname the
Sail connections wlll use.";
--- End diff --
typo: wlll
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187718927
--- Diff:
extras/kafka.connect/mongo/src/main/java/org/apache/rya/kafka/connect/mongo/MongoRyaSinkConnector.java
---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.mongo;
+
+import java.util.Map;
+
+import org.apache.kafka.common.config.AbstractConfig;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.rya.kafka.connect.api.sink.RyaSinkConnector;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+
+/**
+ * A {@link RyaSinkConnector} that uses a Mongo DB Rya backend when
creating tasks.
+ */
+@DefaultAnnotation(NonNull.class)
+public class MongoRyaSinkConnector extends RyaSinkConnector {
+
+@Nullable
+private MongoRyaSinkConfig config = null;
+
+@Override
+public void start(final Map<String, String> props) {
+this.config = new MongoRyaSinkConfig( props );
+}
+
+@Override
+protected AbstractConfig getConfig() {
+if(config == null) {
+throw new IllegalStateException("The configuration has not
been set yet. Invoke start(props) first.");
--- End diff --
same start(Map)
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187698795
--- Diff:
extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/ConfigUtils.java
---
@@ -438,6 +438,9 @@ public static boolean getUsePcjUpdaterIndex(final
Configuration conf) {
return Optional.fromNullable(conf.get(FLUO_APP_NAME));
}
+public static void setUseMongo(final Configuration conf, final boolean
useMongo) {
--- End diff --
can you add this to the constructor of the MongoDbRDFConfiguration
constructor? if we're going to keep using this field, it would make sense for
that to set it.
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187708244
--- Diff:
extras/kafka.connect/accumulo/src/main/java/org/apache/rya/kafka/connect/accumulo/AccumuloRyaSinkConnector.java
---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.accumulo;
+
+import java.util.Map;
+
+import org.apache.kafka.common.config.AbstractConfig;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.rya.kafka.connect.api.sink.RyaSinkConnector;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+
+/**
+ * A {@link RyaSinkConnector} that uses an Accumulo Rya backend when
creating tasks.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloRyaSinkConnector extends RyaSinkConnector {
+
+@Nullable
+private AccumuloRyaSinkConfig config = null;
+
+@Override
+public void start(final Map<String, String> props) {
+this.config = new AccumuloRyaSinkConfig( props );
+}
+
+@Override
+protected AbstractConfig getConfig() {
+if(config == null) {
+throw new IllegalStateException("The configuration has not
been set yet. Invoke start(props) first.");
--- End diff --
usually the doc'd function has the parameters be the type, not the name:
start(Map)
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187708536
--- Diff:
extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/sink/RyaSinkTask.java
---
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.api.sink;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.errors.ConnectException;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.eclipse.rdf4j.model.Statement;
+import org.eclipse.rdf4j.repository.sail.SailRepository;
+import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection;
+import org.eclipse.rdf4j.sail.Sail;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.jcabi.manifests.Manifests;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+
+/**
+ * Handles the common components required to write {@link Statement}s to
Rya.
+ *
+ * Implementations of this class only need to specify functionality that
is specific to the
+ * Rya implementation.
+ */
+@DefaultAnnotation(NonNull.class)
--- End diff --
maybe I'm not clear what this annotation does, but again, all the fields
are declared Nullable
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187698179
--- Diff:
dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/AccumuloRdfConfiguration.java
---
@@ -62,14 +62,14 @@ public AccumuloRdfConfiguration() {
super();
}
-public AccumuloRdfConfiguration(Configuration other) {
+public AccumuloRdfConfiguration(final Configuration other) {
super(other);
}
-public AccumuloRdfConfigurationBuilder getBuilder() {
+public static AccumuloRdfConfigurationBuilder getBuilder() {
--- End diff --
isn't the convention usually builder() for static builder functions?
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187702555
--- Diff:
extras/kafka.connect/accumulo/src/main/java/org/apache/rya/kafka/connect/accumulo/AccumuloRyaSinkTask.java
---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Map;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.client.Instance;
+import org.apache.accumulo.core.client.ZooKeeperInstance;
+import org.apache.accumulo.core.client.security.tokens.PasswordToken;
+import org.apache.kafka.connect.errors.ConnectException;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.RyaClient;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.client.accumulo.AccumuloConnectionDetails;
+import org.apache.rya.api.client.accumulo.AccumuloRyaClientFactory;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.kafka.connect.api.sink.RyaSinkTask;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.eclipse.rdf4j.sail.Sail;
+import org.eclipse.rdf4j.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * A {@link RyaSinkTask} that uses the Accumulo implementation of Rya to
store data.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloRyaSinkTask extends RyaSinkTask {
+
+@Override
+protected void checkRyaInstanceExists(final Map<String, String>
taskConfig) throws ConnectException {
+requireNonNull(taskConfig);
+
+// Parse the configuration object.
+final AccumuloRyaSinkConfig config = new
AccumuloRyaSinkConfig(taskConfig);
+
+// Connect to the instance of Accumulo.
+final Connector connector;
+try {
+final Instance instance = new
ZooKeeperInstance(config.getClusterName(), config.getZookeepers());
+connector = instance.getConnector(config.getUsername(), new
PasswordToken( config.getPassword() ));
+} catch (AccumuloException | AccumuloSecurityException e) {
+throw new ConnectException("Could not create a Connector to
the configured Accumulo instance.", e);
+}
+
+// Use a RyaClient to see if the configured instance exists.
+try {
+final AccumuloConnectionDetails connectionDetails = new
AccumuloConnectionDetails(
+config.getUsername(),
+config.getPassword().toCharArray(),
+config.getClusterName(),
+config.getZookeepers());
+final RyaClient client =
AccumuloRyaClientFactory.build(connectionDetails, connector);
+
+if(!client.getInstanceExists().exists(
config.getRyaInstanceName() )) {
+throw new ConnectException("The Rya Instance named " +
+config.getRyaInstanceName() + " has not been
installed.");
+}
+
+} catch (final RyaClientException e) {
+throw new ConnectException("Unable to determine if the Rya
Instance named " +
+config.getRyaInstanceName() + " has been installed.",
e);
+}
+}
+
+@Override
+protected Sail makeSail(final Map<String, String> taskConfig) throws
ConnectException {
+requireNonNull(taskConfig);
+
+// Par
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187706947
--- Diff:
extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/StatementsSerializer.java
---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.api;
+
+import java.io.ByteArrayOutputStream;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.eclipse.rdf4j.model.Statement;
+import org.eclipse.rdf4j.rio.RDFWriter;
+import org.eclipse.rdf4j.rio.binary.BinaryRDFWriterFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * A Kafka {@link Serializer} that is able to serialize a set of {@link
Statement}s
+ * using the RDF4J Rio Binary format.
--- End diff --
i feel like it might be worthwhile to mention that you use the RDFParser
and RDFWriter here and the Deserializer respectively.
---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187709021
--- Diff:
extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/sink/RyaSinkTask.java
---
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.api.sink;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.errors.ConnectException;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.eclipse.rdf4j.model.Statement;
+import org.eclipse.rdf4j.repository.sail.SailRepository;
+import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection;
+import org.eclipse.rdf4j.sail.Sail;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.jcabi.manifests.Manifests;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+
+/**
+ * Handles the common components required to write {@link Statement}s to
Rya.
+ *
+ * Implementations of this class only need to specify functionality that
is specific to the
+ * Rya implementation.
+ */
+@DefaultAnnotation(NonNull.class)
+public abstract class RyaSinkTask extends SinkTask {
+private static final Logger log =
LoggerFactory.getLogger(RyaSinkTask.class);
+
+@Nullable
+private SailRepository sailRepo = null;
+
+@Nullable
+private SailRepositoryConnection conn = null;
+
+/**
+ * Throws an exception if the configured Rya Instance is not already
installed
+ * within the configured database.
+ *
+ * @param taskConfig - The configuration values that were provided to
the task. (not null)
+ * @throws ConnectException The configured Rya Instance is not
installed to the configured database
+ * or we were unable to figure out if it is installed.
+ */
+protected abstract void checkRyaInstanceExists(final Map<String,
String> taskConfig) throws ConnectException;
+
+/**
+ * Creates an initialized {@link Sail} object that may be used to
write {@link Statement}s to the configured
+ * Rya Instance.
+ *
+ * @param taskConfig - Configures how the Sail object will be created.
(not null)
+ * @return The created Sail object.
+ * @throws ConnectException The Sail object could not be made.
+ */
+protected abstract Sail makeSail(final Map<String, String> taskConfig)
throws ConnectException;
+
+@Override
+public String version() {
+return Manifests.exists("Build-Version") ?
Manifests.read("Build-Version"): "UNKNOWN";
+}
+
+@Override
+public void start(final Map<String, String> props) throws
ConnectException {
+requireNonNull(props);
+
+// Ensure the configured Rya Instance is installed within the
configured database.
+checkRyaInstanceExists(props);
+
+// Create the Sail object that is connected to the Rya Instance.
+final Sail sail = makeSail(props);
+sailRepo = new SailRepository( sail );
+conn = sailRepo.getConnection();
+}
+
+@Override
+public void put(final Collection records) {
+requireNonNull(records);
+
+// Return immediately if there are no records to handle.
+if(records.isEmpty()) {
+return;
+}
+
+// If a transaction has not been started yet, then start
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187699543 --- Diff: extras/kafka.connect/README.md --- @@ -0,0 +1,22 @@ + + +The parent project for all Rya Kafka Connect work. All projects thare are part +of that system must use this project's pom as their parent pom. --- End diff -- typo All projects that* are ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/296#discussion_r187701248
--- Diff:
extras/kafka.connect/accumulo/src/main/java/org/apache/rya/kafka/connect/accumulo/AccumuloRyaSinkConnector.java
---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.kafka.connect.accumulo;
+
+import java.util.Map;
+
+import org.apache.kafka.common.config.AbstractConfig;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.rya.kafka.connect.api.sink.RyaSinkConnector;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+
+/**
+ * A {@link RyaSinkConnector} that uses an Accumulo Rya backend when
creating tasks.
+ */
+@DefaultAnnotation(NonNull.class)
--- End diff --
since there is one field that is marked nullable, is this annotation
usefull?
---
[GitHub] incubator-rya pull request #294: RYA-492 Added language support for Literals
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/294#discussion_r186170248
--- Diff:
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java
---
@@ -194,6 +204,7 @@ public BasicDBObject serializeInternal(final
RyaStatement statement){
.append(OBJECT, statement.getObject().getData())
.append(OBJECT_HASH, hash(statement.getObject().getData()))
.append(OBJECT_TYPE,
statement.getObject().getDataType().toString())
+.append(OBJECT_LANGUAGE, statement.getObject().getLanguage())
--- End diff --
can language be null here?
---
[GitHub] incubator-rya pull request #287: Updating the ExecuteSparqlQuery command to ...
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/287 Updating the ExecuteSparqlQuery command to return BindingSets ## Description >What Changed? Updated the execute sparql interactor to return binding sets instead of a string. ### Tests >Coverage? Updated existing tests ### Links [Jira](https://issues.apache.org/jira/browse/RYA-484) ### Checklist - [ ] Code Review - [ ] Squash Commits People To Reivew @kchilton2 @jdasch @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-484_sparqlQuery Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/287.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #287 commit 8169433be584e6c9ea0cc568621eabde809a2eae Author: Andrew Smith <smith3at@...> Date: 2018-04-06T20:01:43Z Updating the ExecuteSparqlQuery command to return BindingSets ---
[GitHub] incubator-rya pull request #286: RYA-467 update topic cleanup.policy
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/286 RYA-467 update topic cleanup.policy ## Description >What Changed? Added a topic properties builder. Only one property is currently in the builder, but adding others should be easy. ### Tests >Coverage? ### Links [Jira](https://issues.apache.org/jira/browse/RYA-467) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @kchilton2 @ejwhite922 @pujav65 @dlotts @jessehatfield You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-467 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/286.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #286 commit 6565ca4b28c04398c799b5507b0237828a5eb613 Author: Andrew Smith <smith3at@...> Date: 2018-04-02T16:32:40Z RYA-467 update topic cleanup.policy Added a topic properties builder. Only one property is currently in the builder, but adding others should be easy. ---
[GitHub] incubator-rya pull request #285: RYA-469 Added tests for Rya Streams join it...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/285#discussion_r178368019
--- Diff: extras/indexingExample/src/main/java/MongoRyaDirectExample.java
---
@@ -805,6 +819,38 @@ public static void testAddAndDeleteNoContext(final
SailRepositoryConnection conn
Validate.isTrue(resultHandler.getCount() == 0);
}
+public static void testLubmFile(final SailRepositoryConnection conn)
throws MalformedQueryException, RepositoryException,
+UpdateExecutionException, QueryEvaluationException,
TupleQueryResultHandlerException, RDFParseException, IOException {
+
+final String query = LubmQuery.LUBM_QUERY_14.getSparqlQuery();
+//"PREFIX lubm: <" + LUBM_PREFIX + "> \n" +
--- End diff --
remove commented out code
---
[GitHub] incubator-rya pull request #285: RYA-469 Added tests for Rya Streams join it...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/285#discussion_r178368138
--- Diff:
extras/rya.streams/client/src/test/java/org/apache/rya/streams/client/command/RunQueryCommandIT.java
---
@@ -196,4 +203,74 @@ public void run() {
// Show the read results matched the expected ones.
assertEquals(expected, results);
}
+
+@Test
+public void runQueryFromFile() throws Exception {
+// TODO: Maybe test with
org.apache.rya.api.utils.LubmQuery.LUBM_QUERY_1,2,3,etc...
--- End diff --
remove TODO?
---
[GitHub] incubator-rya issue #282: RYA-443 Fixed Rya Streams Query Manager RPM versio...
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/282 rpm -e does not work when you supply a version number. Not sure if I trust that >..> ---
[GitHub] incubator-rya pull request #279: Rya streams base
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/279 Rya streams base ## Description >What Changed? Rya Streams Feature Branch ### Tests >Coverage? Each major feature has test coverate ### Links [Jira](https://issues.apache.org/jira/browse/RYA-460) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @kchilton2 @ejwhite922 @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya Rya-Streams-base Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/279.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #279 commit 28ea2e29a51dc22ba3efb0bdf3aa7fac27648518 Author: Andrew Smith <smith3at@...> Date: 2018-01-16T17:39:34Z RYA-446 Daemonifying Query Manager commit 38c5927768ddeb680a0b3a20155124622d0420c2 Author: Andrew Smith <smith3at@...> Date: 2018-01-18T22:51:40Z RYA-449 Create QueryChangeLogSource commit 01f02535c063361614911a749809afb5cf54b2dc Author: Andrew Smith <smith3at@...> Date: 2018-01-19T20:43:55Z Rya 454 added QueryExecutor interface commit 0be94418dce2229e48cfc760be97d6f90f68fae2 Author: Andrew Smith <smith3at@...> Date: 2018-01-23T20:20:50Z Rya 452 Updated QueryRepository Updated QueryRepository to be a Service Updated InMemoryQueryRepository to be an AbstractScheduledService Added listeners to InMemoryQueryRepository commit fbad2c30c36e1d2df3174bfe95ec68011959689b Author: Andrew Smith <smith3at@...> Date: 2018-01-23T20:44:32Z RYA-455 stopAll queries for a rya instance commit 1dedc48e0a4e311254820ad6662f8fda446cd444 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-23T20:50:17Z RYA-448 Implement JAXB marshalling code for the Query Manager's XML configuration file. commit 2b654b5368faf28f9df73749fdbe87e02ce295e6 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-23T21:17:06Z RYA-450 Implemented a Kafka backed QueryChangeLogSource. commit 3e0a6e89d5cf5f08c5ed5dae14a4f7faf48477e3 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-25T22:19:58Z RYA-446 Create a bin and rpm distribution for the Rya Streams Query Manager application. commit b4b635461cbd0264f1edc2df40e99fa96e7f007b Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-26T17:06:07Z RYA-458 Updating the configuration XML so that you may only specify a single QueryChangeLogSource. commit dc0e803bf8c14ab94c972d43e126588591cf5b91 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-26T20:55:59Z RYA-456 Implement a Single Node implementation of QueryExecutor. commit 33b5d5b1b39b7d3de6c2df1887b0c8dc61ad0fb8 Author: Andrew Smith <smith3at@...> Date: 2018-01-30T19:01:54Z Rya 451 Query manager QueryManager with tests updated InMemoryQueryRepository and its tests commit 69a17be4e66e5dc6ce7ce32f44580f428f18d224 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-30T19:44:58Z RYA-453 Implement the Query Manager's Daemon that controls the application. commit e5ed66854c7f67b6ee6eb4c9c445737bc95568fc Author: Andrew Smith <smith3at@...> Date: 2018-02-01T21:04:48Z RYA-446 Service unit file for systemd commit b4d70758a35cc1c6b9a6c5a93e126b7a644d3526 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-01T23:31:41Z RYA-446 Making the Rya Streams Query Manager run as a service on CentOS 7. commit c7802a2f10b089300d71da9fa1b87e2046efa501 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-03T03:47:59Z RYA-451 Fixing threading issues with the QueryManager class. commit 9a44bc2132a680dec40058741ebe59d4a95fe913 Author: eric.white <eric.white@...> Date: 2018-02-27T20:58:37Z RYA-443 Updated query-manager RPM to use Java based plugin commit d31f4f872115d603c676e4f465a7d42114c044cb Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-14T18:49:52Z RYA-466 Update the Rya Streams Client to stream results to file. commit cd91cd1f1aa0f72b59e952d59552f46b56542488 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-08T22:21:10Z RYA-462 Updated the Kafka topic name for StreamsQueries to include the Rya Instance name. Conflicts: extras/rya.streams/client/src/main/java/org/apache/rya/streams/client/command/StreamResultsCommand.java commit 0b7226e0c27f7f24ae131c8c23121f793321bf25 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-08T22:53:41Z RYA-463 RYA-464 Added an isInsert flag to StreamsQuery to indicate when the results of a query need to be inserted back into Rya. Conflicts: common/rya.api/src/
[GitHub] incubator-rya pull request #275: RYA-466 Update the Rya Streams Client to st...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/275#discussion_r170314969
--- Diff:
common/rya.api/src/main/java/org/apache/rya/api/utils/QueryInvestigator.java ---
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.utils;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.regex.Pattern;
+
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.parser.sparql.SPARQLParser;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * A utility class that is used to glean insight into the structure of
SPARQL queries.
+ */
+@DefaultAnnotation(NonNull.class)
+public class QueryInvestigator {
+
+private static final SPARQLParser PARSER = new SPARQLParser();
+
+private QueryInvestigator() { }
+
+/**
+ * Determines whether a SPARQL command is a CONSTRUCT or not.
+ *
+ * @param sparql - The SPARQL to evaluate. (not null)
+ * @return {@code true} if the provided SPARQL is a CONSTRUCT query;
otherwise {@code false}.
+ * @throws MalformedQueryException The SPARQL is neither a well formed
query or update.
+ */
+public static boolean isConstruct(final String sparql) throws
MalformedQueryException {
+requireNonNull(sparql);
+
+try {
+// Constructs are queries, so try to create a ParsedQuery.
+PARSER.parseQuery(sparql, null);
+
+// Check to see if the SPARQL looks like a CONSTRUCT query.
+return Pattern.matches(".*?construct.*?where.*",
sparql.toLowerCase());
+
+} catch(final MalformedQueryException queryE) {
+try {
+// Maybe it's an update.
+PARSER.parseUpdate(sparql, null);
+
+// It was, so return false.
+return false;
+
+} catch(final MalformedQueryException updateE) {
+// It's not. Actually malformed.
+throw queryE;
+}
+}
+}
+
+/**
+ * Determines whether a SPARQL command is an INSERT with a WHERE
clause or not.
+ *
+ * @param sparql - The SPARQL to evaluate. (not null)
+ * @return {@code true} if the provided SPARQL is an INSERT update;
otherwise {@code false}.
+ * @throws MalformedQueryException The SPARQL is neither a well formed
query or update.
+ */
+public static boolean isInsertWhere(final String sparql) throws
MalformedQueryException {
+requireNonNull(sparql);
+
+try {
+// Inserts are updated, so try to create a ParsedUpdate.
+PARSER.parseUpdate(sparql, null);
+
+// Check to see if the SPARQL looks like an INSERT query.
+return Pattern.matches(".*?insert.*?where.*",
sparql.toLowerCase());
--- End diff --
odd thought, what if the query is something like INSERT { a ?where b }
---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/267#discussion_r163100423
--- Diff:
extras/rya.streams/kafka/src/main/java/org/apache/rya/streams/kafka/KafkaRyaStreamsClientFactory.java
---
@@ -0,0 +1,169 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.streams.kafka;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Properties;
+import java.util.UUID;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.rya.streams.api.RyaStreamsClient;
+import org.apache.rya.streams.api.interactor.defaults.DefaultAddQuery;
+import org.apache.rya.streams.api.interactor.defaults.DefaultDeleteQuery;
+import org.apache.rya.streams.api.interactor.defaults.DefaultGetQuery;
+import org.apache.rya.streams.api.interactor.defaults.DefaultListQueries;
+import org.apache.rya.streams.api.interactor.defaults.DefaultStartQuery;
+import org.apache.rya.streams.api.interactor.defaults.DefaultStopQuery;
+import org.apache.rya.streams.api.queries.InMemoryQueryRepository;
+import org.apache.rya.streams.api.queries.QueryChange;
+import org.apache.rya.streams.api.queries.QueryChangeLog;
+import org.apache.rya.streams.api.queries.QueryRepository;
+import org.apache.rya.streams.kafka.interactor.KafkaGetQueryResultStream;
+import org.apache.rya.streams.kafka.queries.KafkaQueryChangeLog;
+import
org.apache.rya.streams.kafka.serialization.VisibilityBindingSetDeserializer;
+import
org.apache.rya.streams.kafka.serialization.VisibilityStatementDeserializer;
+import
org.apache.rya.streams.kafka.serialization.queries.QueryChangeDeserializer;
+import
org.apache.rya.streams.kafka.serialization.queries.QueryChangeSerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Constructs instances of {@link RyaStreamsClient} that are connected to
a Kafka cluster.
+ */
+@DefaultAnnotation(NonNull.class)
+public final class KafkaRyaStreamsClientFactory {
+private static final Logger log =
LoggerFactory.getLogger(KafkaRyaStreamsClientFactory.class);
+
+/**
+ * Initialize a {@link RyaStreamsClient} that will interact with an
instance of Rya Streams
+ * that is backed by Kafka.
+ *
+ * @param ryaInstance - The name of the Rya Instance the client is
connected to. (not null)
+ * @param kafkaHostname - The hostname of the Kafka Broker.
+ * @param kafkaPort - The port of the Kafka Broker.
+ * @return The initialized commands.
+ */
+public static RyaStreamsClient make(
+final String ryaInstance,
+final String kafkaHostname,
+final int kafkaPort) {
+requireNonNull(kafkaHostname);
+
+// Setup Query Repository used by the Kafka Rya Streams subsystem.
+final Producer queryProducer =
+makeProducer(kafkaHostname, kafkaPort,
StringSerializer.class, QueryChangeSerializer.class);
+final ConsumerqueryConsumer =
+fromStartConsumer(kafkaHostname, kafkaPort,
StringDeserializer.class, QueryChangeDeserializer.class);
+final S
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/267#discussion_r163100365
--- Diff:
extras/indexing/src/test/java/org/apache/rya/api/client/accumulo/AccumuloSetRyaStreamsConfigurationIT.java
---
@@ -0,0 +1,81 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+
+import org.apache.rya.accumulo.AccumuloITBase;
+import org.apache.rya.api.client.Install;
+import org.apache.rya.api.client.Install.InstallConfiguration;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.RyaClient;
+import org.apache.rya.api.instance.RyaDetails.RyaStreamsDetails;
+import org.junit.Test;
+
+/**
+ * Integration tests the methods of {@link
AccumuloSetRyaStreamsConfiguration}.
+ */
+public class AccumuloSetRyaStreamsConfigurationIT extends AccumuloITBase {
+
+@Test(expected = InstanceDoesNotExistException.class)
+public void instanceDoesNotExist() throws Exception {
+final String ryaInstance = getRyaInstanceName();
+final AccumuloConnectionDetails connectionDetails = new
AccumuloConnectionDetails(
+getUsername(),
+getPassword().toCharArray(),
+getInstanceName(),
+getZookeepers());
+final RyaClient ryaClient =
AccumuloRyaClientFactory.build(connectionDetails, getConnector());
+
+// Skip the install step to create error causing situation.
+final RyaStreamsDetails details = new
RyaStreamsDetails("localhost", 6);
+
ryaClient.getSetRyaStreamsConfiguration().setRyaStreamsConfiguration(ryaInstance,
details);
+}
+
+@Test
+public void updatesRyaDetails() throws Exception {
+final String ryaInstance = getRyaInstanceName();
+final AccumuloConnectionDetails connectionDetails = new
AccumuloConnectionDetails(
+getUsername(),
+getPassword().toCharArray(),
+getInstanceName(),
+getZookeepers());
+final RyaClient ryaClient =
AccumuloRyaClientFactory.build(connectionDetails, getConnector());
+
+// Install an instance of Rya.
+final Install installRya = ryaClient.getInstall();
+final InstallConfiguration installConf =
InstallConfiguration.builder()
+.build();
+installRya.install(ryaInstance, installConf);
+
+// Fetch its details and show they do not have any
RyaStreamsDetails.
+com.google.common.base.Optional streamsDetails =
--- End diff --
yeah, still annoying we use 2
---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/267#discussion_r163100285
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoSetRyaStreamsConfiguration.java
---
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.mongo;
+
+import static java.util.Objects.requireNonNull;
+
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.SetRyaStreamsConfiguration;
+import org.apache.rya.api.client.SetRyaStreamsConfigurationBase;
+import org.apache.rya.api.instance.RyaDetailsRepository;
+import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository;
+
+import com.mongodb.MongoClient;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * A MongoDB implementation of {@link SetRyaStreamsConfiguration}.
+ */
+@DefaultAnnotation(NonNull.class)
+public class MongoSetRyaStreamsConfiguration extends
SetRyaStreamsConfigurationBase {
+
+private final MongoClient client;
+
+/**
+ * Constructs an instance of {@link MongoSetRyaStreamsConfiguration}.
+ *
+ * @param instanceExists - The interactor used to verify Rya instances
exist. (not null)
+ * @param client - The MongoDB client used to connect to the Rya
storage. (not null)
+ */
+public MongoSetRyaStreamsConfiguration(
+final InstanceExists instanceExists,
+final MongoClient client) {
+super(instanceExists);
+this.client = requireNonNull(client);
+}
+
+@Override
+protected RyaDetailsRepository getRyaDetailsRepo(final String
ryaInstance) {
+requireNonNull(ryaInstance);
+return new MongoRyaInstanceDetailsRepository(client, ryaInstance);
--- End diff --
fair enough
---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/267#discussion_r163084595
--- Diff:
extras/rya.streams/api/src/main/java/org/apache/rya/streams/api/interactor/defaults/DefaultGetQuery.java
---
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.streams.api.interactor.defaults;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Optional;
+import java.util.UUID;
+
+import org.apache.rya.streams.api.entity.StreamsQuery;
+import org.apache.rya.streams.api.exception.RyaStreamsException;
+import org.apache.rya.streams.api.interactor.GetQuery;
+import org.apache.rya.streams.api.queries.QueryRepository;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Get a {@link StreamsQuery} from Rya Streams.
+ */
+@DefaultAnnotation(NonNull.class)
+public class DefaultGetQuery implements GetQuery {
+private final QueryRepository repository;
+
+public DefaultGetQuery(final QueryRepository repository) {
--- End diff --
doc
---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/267#discussion_r163090091
--- Diff:
extras/shell/src/test/java/org/apache/rya/shell/util/StreamsQueryFormatterTest.java
---
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.shell.util;
+
+import static org.junit.Assert.assertEquals;
+
+import java.util.Set;
+import java.util.UUID;
+
+import org.apache.rya.streams.api.entity.StreamsQuery;
+import org.junit.Test;
+
+import com.google.common.collect.Sets;
+
+/**
+ * Unit tests the methods of {@link StreamsQueryFormatter}.
+ */
+public class StreamsQueryFormatterTest {
+
+@Test
+public void formatQuery() throws Exception {
+// Format the query.
+final StreamsQuery query = new StreamsQuery(
+UUID.fromString("da55cea5-c21c-46a5-ab79-5433eef4efaa"),
+"SELECT * WHERE { ?a ?b ?c . }",
+true);
+final String formatted = StreamsQueryFormatter.format(query);
+
+// Ensure it has the expected format.
+final String expected =
+" Query ID: da55cea5-c21c-46a5-ab79-5433eef4efaa\n" +
+"Is Active: true\n" +
+" SPARQL: select ?a ?b ?c\n" +
+" where {\n" +
+" ?a ?b ?c.\n" +
+" }\n";
+
+assertEquals(expected, formatted);
+}
+
+@Test
+public void formatQueries() throws Exception {
+// Format the queries.
+final Set queries = Sets.newHashSet(
+new StreamsQuery(
+
UUID.fromString("----"),
+"SELECT * WHERE { ?person ?business
. }",
+true),
+new StreamsQuery(
+
UUID.fromString("----"),
+"SELECT * WHERE { ?a ?b ?c . }",
+true),
+new StreamsQuery(
+
UUID.fromString("----"),
+"SELECT * WHERE { ?d ?e ?f . }",
+false));
+
+final String formatted = StreamsQueryFormatter.format(queries);
+System.out.println(formatted);
--- End diff --
SOP?
---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/267#discussion_r163082381
--- Diff:
extras/indexing/src/test/java/org/apache/rya/api/client/accumulo/AccumuloSetRyaStreamsConfigurationIT.java
---
@@ -0,0 +1,81 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+
+import org.apache.rya.accumulo.AccumuloITBase;
+import org.apache.rya.api.client.Install;
+import org.apache.rya.api.client.Install.InstallConfiguration;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.RyaClient;
+import org.apache.rya.api.instance.RyaDetails.RyaStreamsDetails;
+import org.junit.Test;
+
+/**
+ * Integration tests the methods of {@link
AccumuloSetRyaStreamsConfiguration}.
+ */
+public class AccumuloSetRyaStreamsConfigurationIT extends AccumuloITBase {
+
+@Test(expected = InstanceDoesNotExistException.class)
+public void instanceDoesNotExist() throws Exception {
+final String ryaInstance = getRyaInstanceName();
+final AccumuloConnectionDetails connectionDetails = new
AccumuloConnectionDetails(
+getUsername(),
+getPassword().toCharArray(),
+getInstanceName(),
+getZookeepers());
+final RyaClient ryaClient =
AccumuloRyaClientFactory.build(connectionDetails, getConnector());
+
+// Skip the install step to create error causing situation.
+final RyaStreamsDetails details = new
RyaStreamsDetails("localhost", 6);
+
ryaClient.getSetRyaStreamsConfiguration().setRyaStreamsConfiguration(ryaInstance,
details);
+}
+
+@Test
+public void updatesRyaDetails() throws Exception {
+final String ryaInstance = getRyaInstanceName();
+final AccumuloConnectionDetails connectionDetails = new
AccumuloConnectionDetails(
+getUsername(),
+getPassword().toCharArray(),
+getInstanceName(),
+getZookeepers());
+final RyaClient ryaClient =
AccumuloRyaClientFactory.build(connectionDetails, getConnector());
+
+// Install an instance of Rya.
+final Install installRya = ryaClient.getInstall();
+final InstallConfiguration installConf =
InstallConfiguration.builder()
+.build();
+installRya.install(ryaInstance, installConf);
+
+// Fetch its details and show they do not have any
RyaStreamsDetails.
+com.google.common.base.Optional streamsDetails =
--- End diff --
we should really clean up the Optional stuff one day
---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/267#discussion_r163085139
--- Diff:
extras/rya.streams/kafka/src/main/java/org/apache/rya/streams/kafka/KafkaRyaStreamsClientFactory.java
---
@@ -0,0 +1,169 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.streams.kafka;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Properties;
+import java.util.UUID;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.rya.streams.api.RyaStreamsClient;
+import org.apache.rya.streams.api.interactor.defaults.DefaultAddQuery;
+import org.apache.rya.streams.api.interactor.defaults.DefaultDeleteQuery;
+import org.apache.rya.streams.api.interactor.defaults.DefaultGetQuery;
+import org.apache.rya.streams.api.interactor.defaults.DefaultListQueries;
+import org.apache.rya.streams.api.interactor.defaults.DefaultStartQuery;
+import org.apache.rya.streams.api.interactor.defaults.DefaultStopQuery;
+import org.apache.rya.streams.api.queries.InMemoryQueryRepository;
+import org.apache.rya.streams.api.queries.QueryChange;
+import org.apache.rya.streams.api.queries.QueryChangeLog;
+import org.apache.rya.streams.api.queries.QueryRepository;
+import org.apache.rya.streams.kafka.interactor.KafkaGetQueryResultStream;
+import org.apache.rya.streams.kafka.queries.KafkaQueryChangeLog;
+import
org.apache.rya.streams.kafka.serialization.VisibilityBindingSetDeserializer;
+import
org.apache.rya.streams.kafka.serialization.VisibilityStatementDeserializer;
+import
org.apache.rya.streams.kafka.serialization.queries.QueryChangeDeserializer;
+import
org.apache.rya.streams.kafka.serialization.queries.QueryChangeSerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Constructs instances of {@link RyaStreamsClient} that are connected to
a Kafka cluster.
+ */
+@DefaultAnnotation(NonNull.class)
+public final class KafkaRyaStreamsClientFactory {
+private static final Logger log =
LoggerFactory.getLogger(KafkaRyaStreamsClientFactory.class);
+
+/**
+ * Initialize a {@link RyaStreamsClient} that will interact with an
instance of Rya Streams
+ * that is backed by Kafka.
+ *
+ * @param ryaInstance - The name of the Rya Instance the client is
connected to. (not null)
+ * @param kafkaHostname - The hostname of the Kafka Broker.
+ * @param kafkaPort - The port of the Kafka Broker.
+ * @return The initialized commands.
+ */
+public static RyaStreamsClient make(
+final String ryaInstance,
+final String kafkaHostname,
+final int kafkaPort) {
+requireNonNull(kafkaHostname);
+
+// Setup Query Repository used by the Kafka Rya Streams subsystem.
+final Producer queryProducer =
+makeProducer(kafkaHostname, kafkaPort,
StringSerializer.class, QueryChangeSerializer.class);
+final ConsumerqueryConsumer =
+fromStartConsumer(kafkaHostname, kafkaPort,
StringDeserializer.class, QueryChangeDeserializer.class);
+final S
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/267#discussion_r163081707
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloSetRyaStreamsConfiguration.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import org.apache.accumulo.core.client.Connector;
+import
org.apache.rya.accumulo.instance.AccumuloRyaInstanceDetailsRepository;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.SetRyaStreamsConfiguration;
+import org.apache.rya.api.client.SetRyaStreamsConfigurationBase;
+import org.apache.rya.api.instance.RyaDetailsRepository;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * A Accumulo implementation of {@link SetRyaStreamsConfiguration}.
--- End diff --
nit: An*
---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/267#discussion_r163082048
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoSetRyaStreamsConfiguration.java
---
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.mongo;
+
+import static java.util.Objects.requireNonNull;
+
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.SetRyaStreamsConfiguration;
+import org.apache.rya.api.client.SetRyaStreamsConfigurationBase;
+import org.apache.rya.api.instance.RyaDetailsRepository;
+import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository;
+
+import com.mongodb.MongoClient;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * A MongoDB implementation of {@link SetRyaStreamsConfiguration}.
+ */
+@DefaultAnnotation(NonNull.class)
+public class MongoSetRyaStreamsConfiguration extends
SetRyaStreamsConfigurationBase {
+
+private final MongoClient client;
+
+/**
+ * Constructs an instance of {@link MongoSetRyaStreamsConfiguration}.
+ *
+ * @param instanceExists - The interactor used to verify Rya instances
exist. (not null)
+ * @param client - The MongoDB client used to connect to the Rya
storage. (not null)
+ */
+public MongoSetRyaStreamsConfiguration(
+final InstanceExists instanceExists,
+final MongoClient client) {
+super(instanceExists);
+this.client = requireNonNull(client);
+}
+
+@Override
+protected RyaDetailsRepository getRyaDetailsRepo(final String
ryaInstance) {
+requireNonNull(ryaInstance);
+return new MongoRyaInstanceDetailsRepository(client, ryaInstance);
--- End diff --
you could one line this?
---
[GitHub] incubator-rya pull request #265: RYA-442 Implementing the Start and Stop Que...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/265#discussion_r162113248 --- Diff: extras/rya.streams/api/src/main/java/org/apache/rya/streams/api/interactor/defaults/DefaultStartQuery.java --- @@ -0,0 +1,54 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.streams.api.interactor.defaults; + +import static java.util.Objects.requireNonNull; + +import java.util.UUID; + +import org.apache.rya.streams.api.exception.RyaStreamsException; +import org.apache.rya.streams.api.interactor.StartQuery; +import org.apache.rya.streams.api.queries.QueryRepository; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * TODO --- End diff -- doc ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r161103080
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [pcj_ID]_METADATA,
+ * sparql: [sparql query to match results],
+ * varOrders: [varOrder1, VarOrder2, ..., VarOrdern]
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjId: [pcj_ID],
+ * visibilities: [visibilities]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_METADATA_ID = "_id";
+public static final String VAR_ORDER_FIELD = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "rdfType";
+private static final String VISIBILITIES_FIELD = "visibilities";
+private static final String PCJ_ID = "pcjId";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName -
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160834195
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [pcj_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * varOrders: [varOrder1, VarOrder2, ..., VarOrdern]
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjId: [pcj_name],
+ * visibilities: [visibilities]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_FIELD = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "rdfType";
+private static final String VISIBILITIES_FIELD = "visibilities";
+private static final String PCJ_NAME = "pcjId";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName -
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160834100
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [pcj_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * varOrders: [varOrder1, VarOrder2, ..., VarOrdern]
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjId: [pcj_name],
+ * visibilities: [visibilities]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_FIELD = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "rdfType";
+private static final String VISIBILITIES_FIELD = "visibilities";
+private static final String PCJ_NAME = "pcjId";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName -
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160834091
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [pcj_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * varOrders: [varOrder1, VarOrder2, ..., VarOrdern]
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjId: [pcj_name],
+ * visibilities: [visibilities]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_FIELD = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "rdfType";
+private static final String VISIBILITIES_FIELD = "visibilities";
+private static final String PCJ_NAME = "pcjId";
--- End diff --
done
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160833809
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [pcj_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * varOrders: [varOrder1, VarOrder2, ..., VarOrdern]
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjId: [pcj_name],
--- End diff --
done
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160833781
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [pcj_name]_METADATA,
--- End diff --
done
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160813417
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [pcj_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * varOrders: [varOrder1, VarOrder2, ..., VarOrdern]
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [pcj_name],
+ * visibilities: [visibilities]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_FIELD = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "rdfType";
+private static final String VISIBILITIES_FIELD = "visibilities";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName -
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160813316
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjStorage.java
---
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.rya.api.instance.RyaDetails;
+import org.apache.rya.api.instance.RyaDetails.PCJIndexDetails;
+import org.apache.rya.api.instance.RyaDetails.PCJIndexDetails.PCJDetails;
+import
org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException;
+import org.apache.rya.api.instance.RyaDetailsUpdater;
+import
org.apache.rya.api.instance.RyaDetailsUpdater.RyaDetailsMutator.CouldNotApplyMutationException;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PCJIdFactory;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage;
+import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository;
+import org.openrdf.query.BindingSet;
+
+import com.mongodb.MongoClient;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * A mongo backed implementation of {@link PrecomputedJoinStorage}.
+ */
+@DefaultAnnotation(NonNull.class)
+public class MongoPcjStorage implements PrecomputedJoinStorage {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+// Used to update the instance's metadata.
+private final MongoRyaInstanceDetailsRepository ryaDetailsRepo;
+
+private final String ryaInstanceName;
+
+// Factories that are used to create new PCJs.
+private final PCJIdFactory pcjIdFactory = new PCJIdFactory();
+
+private final MongoPcjDocuments pcjDocs;
+
+/**
+ * Constructs an instance of {@link MongoPcjStorage}.
+ *
+ * @param client - The {@link MongoClient} that will be used to
connect to Mongodb. (not null)
+ * @param ryaInstanceName - The name of the RYA instance that will be
accessed. (not null)
+ */
+public MongoPcjStorage(final MongoClient client, final String
ryaInstanceName) {
+requireNonNull(client);
+this.ryaInstanceName = requireNonNull(ryaInstanceName);
+pcjDocs = new MongoPcjDocuments(client, ryaInstanceName);
+ryaDetailsRepo = new MongoRyaInstanceDetailsRepository(client,
ryaInstanceName);
+}
+
+@Override
+public String createPcj(final String sparql) throws
PCJStorageException {
+requireNonNull(sparql);
+
+// Update the Rya Details for this instance to include the new PCJ
+// table.
+final String pcjId = pcjIdFactory.nextId();
+
+try {
+new RyaDetailsUpdater(ryaDetailsRepo).update(originalDetails
-> {
+// Create the new PCJ's details.
+final PCJDetails.Builder newPcjDetails =
PCJDetails.builder().setId(pcjId);
+
+// Add them to the instance's details.
+final RyaDetails.Builder mutated =
RyaDetails.builder(originalDetails);
+mutated.getPCJIndexDetails().addPCJDetails(newPcjDetails);
+return mutated.build();
+});
+} catch (final RyaDetailsRepositoryException |
CouldNotApplyMutationException e) {
+throw new PCJStorageException(String.format("Could not create
a new PCJ for Rya instance '%s' "
++ "because of a problem while updatin
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160813231
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [pcj_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * varOrders: [varOrder1, VarOrder2, ..., VarOrdern]
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [pcj_name],
+ * visibilities: [visibilities]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: [value]
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
--- End diff --
you store the PCJ ID plus metadata
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160813158
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.model.VisibilityBindingSet;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.api.utils.CloseableIterator;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [pcj_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * varOrders: [varOrder1, VarOrder2, ..., VarOrdern]
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
--- End diff --
it lets mongo handle that
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160813037
--- Diff:
extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/pcj/MongoPcjIndexSetProvider.java
---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.mongodb.pcj;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rya.api.instance.RyaDetailsRepository;
+import
org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException;
+import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet;
+import
org.apache.rya.indexing.pcj.matching.provider.AbstractPcjIndexSetProvider;
+import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjDocuments;
+import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage;
+import org.apache.rya.mongodb.StatefulMongoDBRdfConfiguration;
+import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository;
+import org.openrdf.query.MalformedQueryException;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.mongodb.MongoClient;
+
+/**
+ * Implementation of {@link AbstractPcjIndexSetProvider} for MongoDB.
+ */
+public class MongoPcjIndexSetProvider extends AbstractPcjIndexSetProvider {
+/**
+ * Creates a new {@link MongoPcjIndexSetProvider}.
+ * @param conf - The configuration for this provider. (not null)
+ */
+public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration
conf) {
+super(conf);
+}
+
+/**
+ * Creates a new {@link MongoPcjIndexSetProvider}.
+ * @param conf - The configuration for this provider.
+ * @param indices - The predefined indicies on this provider.
+ * @param client - The {@link MongoClient} used to connect to mongo.
+ */
+public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration
conf, final List indices) {
+super(conf, indices);
+}
+
+@Override
+protected List getIndices() throws
PcjIndexSetException {
+try {
+//TODO use the PCJ collection.
--- End diff --
no, removed it
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160811345
--- Diff:
extras/shell/src/main/java/org/apache/rya/shell/util/RyaDetailsFormatter.java
---
@@ -72,30 +72,28 @@ public String format(StorageType storageType, final
RyaDetails details) {
report.append(" Temporal Index:\n");
report.append("Enabled: ").append(
details.getTemporalIndexDetails().isEnabled() ).append("\n");
-if(storageType == StorageType.ACCUMULO) {
-report.append(" PCJ Index:\n");
-final PCJIndexDetails pcjDetails =
details.getPCJIndexDetails();
-report.append("Enabled: ").append( pcjDetails.isEnabled()
).append("\n");
-if(pcjDetails.isEnabled()) {
-if(pcjDetails.getFluoDetails().isPresent()) {
-final String fluoAppName =
pcjDetails.getFluoDetails().get().getUpdateAppName();
-report.append("Fluo App Name:
").append(fluoAppName).append("\n");
-}
+report.append(" PCJ Index:\n");
--- End diff --
that include Join Selectivity?
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160810833
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160810774
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/254#discussion_r160808196 --- Diff: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryNode.java --- @@ -0,0 +1,882 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.mongodb.aggregation; + +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.CONTEXT; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.DOCUMENT_VISIBILITY; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_HASH; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_TYPE; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE_HASH; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.STATEMENT_METADATA; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT_HASH; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.TIMESTAMP; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.NavigableSet; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.ConcurrentSkipListSet; +import java.util.function.Function; + +import org.apache.rya.api.domain.RyaStatement; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.domain.RyaURI; +import org.apache.rya.api.domain.StatementMetadata; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.mongodb.MongoDbRdfConstants; +import org.apache.rya.mongodb.dao.MongoDBStorageStrategy; +import org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy; +import org.apache.rya.mongodb.document.operators.query.ConditionalOperators; +import org.apache.rya.mongodb.document.visibility.DocumentVisibilityAdapter; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.Literal; +import org.openrdf.model.Resource; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.vocabulary.XMLSchema; +import org.openrdf.query.BindingSet; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.algebra.Compare; +import org.openrdf.query.algebra.ExtensionElem; +import org.openrdf.query.algebra.ProjectionElem; +import org.openrdf.query.algebra.ProjectionElemList; +import org.openrdf.query.algebra.StatementPattern; +import org.openrdf.query.algebra.ValueConstant; +import org.openrdf.query.algebra.ValueExpr; +import org.openrdf.query.algebra.Var; +import org.openrdf.query.algebra.evaluation.impl.ExternalSet; + +import com.google.common.base.Preconditions; +import com.google.common.collect.BiMap; +import com.google.common.collect.HashBiMap; +import com.mongodb.BasicDBObject; +import com.mongodb.DBObject; +import com.mongodb.client.MongoCollection; +import com.mongodb.client.model.Aggregates; +import com.mongodb.client.model.BsonField; +import com.mongodb.client.model.Filters; +import com.mongodb.client.model.Projections; + +import info.aduna.iteration.CloseableIteration; + +/** + * Represents a portion of a query tree as MongoDB aggregation pipeline. Should + * be built bottom-up: start with a statement pattern implemented as a $match + * step, then add steps to the pipeline to handle higher levels of the query + * tree. Methods are provided to add certain supported query operations to the + * end of the internal pip
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160762129
--- Diff:
extras/shell/src/main/java/org/apache/rya/shell/util/InstallPrompt.java ---
@@ -91,14 +91,14 @@ public InstallConfiguration
promptInstallConfiguration(final String instanceName
checkState(storageType.isPresent(), "The shell must be
connected to a storage to use the install prompt.");
switch(sharedShellState.getShellState().getStorageType().get()) {
-case ACCUMULO:
-return promptAccumuloConfig(instanceName);
+case ACCUMULO:
+return promptAccumuloConfig(instanceName);
-case MONGO:
-return promptMongoConfig(instanceName);
+case MONGO:
+return promptMongoConfig(instanceName);
-default:
-throw new IllegalStateException("Unsupported storage
type: " + storageType.get());
+default:
+throw new IllegalStateException("Unsupported storage type:
" + storageType.get());
--- End diff --
done
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160762007
--- Diff:
extras/shell/src/main/java/org/apache/rya/shell/RyaAdminCommands.java ---
@@ -320,11 +343,14 @@ public String installWithMongoParameters(
final boolean enableFreeTextIndex,
// TODO RYA-215
-//@CliOption(key = {"enableGeospatialIndex"}, mandatory =
false, help = "Use Geospatial Indexing.", unspecifiedDefaultValue = "false",
specifiedDefaultValue = "true")
-//final boolean enableGeospatialIndex,
+//@CliOption(key = {"enableGeospatialIndex"},
mandatory = false, help = "Use Geospatial Indexing.", unspecifiedDefaultValue =
"false", specifiedDefaultValue = "true")
+//final boolean enableGeospatialIndex,
@CliOption(key = {"enableTemporalIndex"}, mandatory = false,
help = "Use Temporal Indexing.", unspecifiedDefaultValue = "false",
specifiedDefaultValue = "true")
-final boolean enableTemporalIndex) {
+final boolean enableTemporalIndex,
--- End diff --
I went ahead and just reverted it, was faster to just take care of it later
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160761918
--- Diff:
extras/shell/src/main/java/org/apache/rya/shell/RyaAdminCommands.java ---
@@ -104,11 +104,11 @@ public RyaAdminCommands(
INSTALL_CMD})
public boolean areStorageCommandsAvailable() {
switch(state.getShellState().getConnectionState()) {
-case CONNECTED_TO_STORAGE:
-case CONNECTED_TO_INSTANCE:
-return true;
-default:
-return false;
+case CONNECTED_TO_STORAGE:
--- End diff --
done
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160761790
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160761434
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160761412
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160761257
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160761195
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160760999
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160760932
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160760282
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoCreatePCJ.java
---
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.mongo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Set;
+
+import org.apache.rya.api.client.CreatePCJ;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.RyaClientException;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.mongodb.MongoClient;
+
+/**
+ * A Mongo implementation of {@link CreatePCJ}.
+ */
+public class MongoCreatePCJ implements CreatePCJ {
+private static final Logger log =
LoggerFactory.getLogger(MongoCreatePCJ.class);
+
+private final MongoConnectionDetails connectionDetails;
--- End diff --
removed
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160760303
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoDeletePCJ.java
---
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.mongo;
+
+import static java.util.Objects.requireNonNull;
+
+import org.apache.rya.api.client.DeletePCJ;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.RyaClientException;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage;
+
+import com.mongodb.MongoClient;
+
+/**
+ * A Mongo implementation of {@link DeletePCJ}.
+ */
+public class MongoDeletePCJ implements DeletePCJ {
+private final MongoConnectionDetails connectionDetails;
--- End diff --
removed
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160755378
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160755315 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; --- End diff -- The way VisibilityBindingSets were implemented, I think they use accumulo's Authorizations api ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160753606
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160570620
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya issue #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/172 what is RyaDetailsFormatter. this is more of a problem of just stuff floating around that doesn't break when functionality gets added ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160570534
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
+
+private final MongoCollection pcjCollection;
+private static final PcjVarOrderFactory pcjVarOrderFactory = new
ShiftVarOrderFactory();
+
+/**
+ * Creates a new {@link MongoPcjDocuments}.
+ * @param client - The {@link MongoClient} to use to connect to mongo.
+ * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160570420
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
+private static final String AUTHS_FIELD = "auths";
+private static final String PCJ_NAME = "pcjName";
--- End diff --
no, its for the pcj name. the ID is calculated differently
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160570327
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
+
+// pcj results fields
+private static final String BINDING_VALUE = "value";
+private static final String BINDING_TYPE = "uri";
--- End diff --
done
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160570226
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * .
+ * .
+ * .
+ * [binding_varn]: {
+ * uri: [type_uri],
+ * value: value
+ * }
+ * }
+ *
+ *
+ */
+public class MongoPcjDocuments {
+public static final String PCJ_COLLECTION_NAME = "pcjs";
+
+// metadata fields
+public static final String CARDINALITY_FIELD = "cardinality";
+public static final String SPARQL_FIELD = "sparql";
+public static final String PCJ_ID = "_id";
+public static final String VAR_ORDER_ID = "varOrders";
--- End diff --
done
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160570170
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
--- End diff --
nvm. didn't look right
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160570068
--- Diff:
extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java
---
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.storage.mongo;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.apache.rya.indexing.pcj.storage.PcjMetadata;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder;
+import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.MapBindingSet;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.RepositoryException;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.util.JSON;
+
+/**
+ * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows:
+ *
+ *
+ *
+ * - PCJ Metadata Doc -
+ * {
+ * _id: [table_name]_METADATA,
+ * sparql: [sparql query to match results],
+ * cardinality: [number of results]
+ * }
+ *
+ * - PCJ Results Doc -
+ * {
+ * pcjName: [table_name],
+ * auths: [auths]
+ * [binding_var1]: {
+ * uri: [type_uri],
+ * value: value
--- End diff --
well square brackets denotes list/set so really the docs should not
have them
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160569220
--- Diff:
extras/indexing/src/main/java/org/apache/rya/indexing/pcj/matching/provider/AccumuloIndexSetProvider.java
---
@@ -0,0 +1,139 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.matching.provider;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import
org.apache.rya.accumulo.instance.AccumuloRyaInstanceDetailsRepository;
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.api.instance.RyaDetailsRepository;
+import
org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException;
+import org.apache.rya.indexing.accumulo.ConfigUtils;
+import org.apache.rya.indexing.external.tupleSet.AccumuloIndexSet;
+import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet;
+import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.accumulo.AccumuloPcjStorage;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjTableNameFactory;
+import org.apache.rya.indexing.pcj.storage.accumulo.PcjTables;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.sail.SailException;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/**
+ * Implementation of {@link AbstractPcjIndexSetProvider} for Accumulo.
+ * This provider uses either user specified Accumulo configuration
information or user a specified
+ * List of ExternalTupleSets to populate an internal cache of
ExternalTupleSets. If Accumulo configuration
+ * is provided, the provider connects to an instance of RyaDetails and
populates the cache with
+ * PCJs registered in RyaDetails.
+ */
+public class AccumuloIndexSetProvider extends AbstractPcjIndexSetProvider {
+private static final Logger log =
Logger.getLogger(AccumuloIndexSetProvider.class);
+
+public AccumuloIndexSetProvider(final Configuration conf) {
+super(conf);
+}
+
+public AccumuloIndexSetProvider(final Configuration conf, final
List indices) {
+super(conf, indices);
+}
+
+@Override
+protected List getIndices() throws
PcjIndexSetException {
+requireNonNull(conf);
+try {
+final String tablePrefix =
requireNonNull(conf.get(RdfCloudTripleStoreConfiguration.CONF_TBL_PREFIX));
+final Connector conn =
requireNonNull(ConfigUtils.getConnector(conf));
+List tables = null;
+
+if (conf instanceof RdfCloudTripleStoreConfiguration) {
+tables = ((RdfCloudTripleStoreConfiguration)
conf).getPcjTables();
+}
+// this maps associates pcj table name with pcj sparql query
+final Map<String, String> indexTables =
Maps.newLinkedHashMap();
+
+try(final PrecomputedJoinStorage storage = new
AccumuloPcjStorage(conn, tablePrefix)) {
+final PcjTableNameFactory pcjFactory = new
PcjTableNameFactory();
+
+final boolean tablesProvided = tables != null &&
!tables.isEmpty();
+
+if (tablesProvided) {
--- End diff --
I'm not touching accumulo code. this is just a repackage
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160569137
--- Diff:
extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/pcj/MongoPcjIndexSetProvider.java
---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.mongodb.pcj;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rya.api.instance.RyaDetailsRepository;
+import
org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException;
+import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet;
+import
org.apache.rya.indexing.pcj.matching.provider.AbstractPcjIndexSetProvider;
+import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjDocuments;
+import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage;
+import org.apache.rya.mongodb.StatefulMongoDBRdfConfiguration;
+import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository;
+import org.openrdf.query.MalformedQueryException;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.mongodb.MongoClient;
+
+/**
+ * Implementation of {@link AbstractPcjIndexSetProvider} for MongoDB.
+ */
+public class MongoPcjIndexSetProvider extends AbstractPcjIndexSetProvider {
+/**
+ * Creates a new {@link MongoPcjIndexSetProvider}.
+ * @param conf - The configuration for this provider. (not null)
+ */
+public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration
conf) {
+super(conf);
+}
+
+/**
+ * Creates a new {@link MongoPcjIndexSetProvider}.
+ * @param conf - The configuration for this provider.
+ * @param indices - The predefined indicies on this provider.
+ * @param client - The {@link MongoClient} used to connect to mongo.
+ */
+public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration
conf, final List indices) {
+super(conf, indices);
+}
+
+@Override
+protected List getIndices() throws
PcjIndexSetException {
+requireNonNull(conf);
+try {
+final StatefulMongoDBRdfConfiguration mongoConf =
(StatefulMongoDBRdfConfiguration) conf;
+final MongoClient client = mongoConf.getMongoClient();
+final MongoPcjDocuments pcjDocs = new
MongoPcjDocuments(client, mongoConf.getRyaInstanceName());
+List tables = null;
--- End diff --
there should really only be the PCJ collection. I may take a look at just
cleaning this up to use the PCJ collection
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160562616
--- Diff:
extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/pcj/MongoPcjIndexSetProvider.java
---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.mongodb.pcj;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rya.api.instance.RyaDetailsRepository;
+import
org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException;
+import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet;
+import
org.apache.rya.indexing.pcj.matching.provider.AbstractPcjIndexSetProvider;
+import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjDocuments;
+import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage;
+import org.apache.rya.mongodb.StatefulMongoDBRdfConfiguration;
+import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository;
+import org.openrdf.query.MalformedQueryException;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.mongodb.MongoClient;
+
+/**
+ * Implementation of {@link AbstractPcjIndexSetProvider} for MongoDB.
+ */
+public class MongoPcjIndexSetProvider extends AbstractPcjIndexSetProvider {
+/**
+ * Creates a new {@link MongoPcjIndexSetProvider}.
+ * @param conf - The configuration for this provider. (not null)
+ */
+public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration
conf) {
+super(conf);
+}
+
+/**
+ * Creates a new {@link MongoPcjIndexSetProvider}.
+ * @param conf - The configuration for this provider.
+ * @param indices - The predefined indicies on this provider.
+ * @param client - The {@link MongoClient} used to connect to mongo.
+ */
+public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration
conf, final List indices) {
+super(conf, indices);
+}
+
+@Override
+protected List getIndices() throws
PcjIndexSetException {
+requireNonNull(conf);
--- End diff --
probably should
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160535354
--- Diff:
extras/indexing/src/main/java/org/apache/rya/indexing/pcj/matching/provider/AbstractPcjIndexSetProvider.java
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.matching.provider;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import org.apache.rya.api.instance.RyaDetails;
+import
org.apache.rya.indexing.IndexPlanValidator.IndexedExecutionPlanGenerator;
+import
org.apache.rya.indexing.IndexPlanValidator.ValidIndexCombinationGenerator;
+import org.apache.rya.indexing.external.matching.ExternalSetProvider;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet;
+import org.apache.rya.indexing.pcj.matching.PCJOptimizerUtilities;
+import org.apache.rya.indexing.pcj.matching.PCJToSegmentConverter;
+import org.openrdf.query.algebra.TupleExpr;
+
+import com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Abstraction of {@link ExternalSetProvider} that provides {@link
ExternalTupleSet}s.
+ * Implementations of this use either user specified configuration
information or user a specified
+ * List of ExternalTupleSets to populate an internal cache of
ExternalTupleSets. If a configuration
+ * is provided, the provider connects to an instance of RyaDetails and
populates the cache with
+ * PCJs registered in RyaDetails.
+ */
+public abstract class AbstractPcjIndexSetProvider implements
ExternalSetProvider {
+protected static final Logger log =
Logger.getLogger(AbstractPcjIndexSetProvider.class);
+protected static final PCJToSegmentConverter converter = new
PCJToSegmentConverter();
+protected List indexCache;
+protected final Configuration conf;
+protected boolean init = false;
+
+/**
+ * Creates a new {@link AbstractPcjIndexSetProvider} based on
configuration only.
+ * @param conf - The {@link Configuration} used to connect to {@link
RyaDetails}.
+ */
+public AbstractPcjIndexSetProvider(final Configuration conf) {
+requireNonNull(conf);
+this.conf = conf;
+}
+
+/**
+ * Creates a new {@link AbstractPcjIndexSetProvider} based user
provided {@link ExternalTupleSet}s.
+ * @param conf - The {@link Configuration} used to connect to {@link
RyaDetails}.
+ * @param indices - The {@link ExternalTupleSet}s to populate the
internal cache.
+ */
+public AbstractPcjIndexSetProvider(final Configuration conf, final
List indices) {
+requireNonNull(conf);
+this.conf = conf;
+indexCache = indices;
+init = true;
+}
+
+
+/**
+ *
+ * @param indices
+ */
+@VisibleForTesting
+public void setIndices(final List indices) {
--- End diff --
a lot of the tests would need to be repackaged to allow that
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160534831
--- Diff:
extras/indexing/src/main/java/org/apache/rya/indexing/pcj/matching/provider/AbstractPcjIndexSetProvider.java
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.pcj.matching.provider;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import org.apache.rya.api.instance.RyaDetails;
+import
org.apache.rya.indexing.IndexPlanValidator.IndexedExecutionPlanGenerator;
+import
org.apache.rya.indexing.IndexPlanValidator.ValidIndexCombinationGenerator;
+import org.apache.rya.indexing.external.matching.ExternalSetProvider;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet;
+import org.apache.rya.indexing.pcj.matching.PCJOptimizerUtilities;
+import org.apache.rya.indexing.pcj.matching.PCJToSegmentConverter;
+import org.openrdf.query.algebra.TupleExpr;
+
+import com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Abstraction of {@link ExternalSetProvider} that provides {@link
ExternalTupleSet}s.
+ * Implementations of this use either user specified configuration
information or user a specified
+ * List of ExternalTupleSets to populate an internal cache of
ExternalTupleSets. If a configuration
+ * is provided, the provider connects to an instance of RyaDetails and
populates the cache with
+ * PCJs registered in RyaDetails.
+ */
+public abstract class AbstractPcjIndexSetProvider implements
ExternalSetProvider {
+protected static final Logger log =
Logger.getLogger(AbstractPcjIndexSetProvider.class);
+protected static final PCJToSegmentConverter converter = new
PCJToSegmentConverter();
+protected List indexCache;
+protected final Configuration conf;
+protected boolean init = false;
+
+/**
+ * Creates a new {@link AbstractPcjIndexSetProvider} based on
configuration only.
+ * @param conf - The {@link Configuration} used to connect to {@link
RyaDetails}.
+ */
+public AbstractPcjIndexSetProvider(final Configuration conf) {
+requireNonNull(conf);
--- End diff --
a lot of these look like I did them, its just a repackage
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160534291
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoInstall.java
---
@@ -191,26 +192,22 @@ private static MongoDBRdfConfiguration makeRyaConfig(
final MongoDBRdfConfiguration conf =
connectionDetails.build(ryaDetails.getRyaInstanceName());
-// The Mongo implementation of Rya does not currently support PCJs.
-if(ryaDetails.getPCJIndexDetails().isEnabled()) {
-log.warn("The install configuration says to enable PCJ
indexing, but Mongo RYA does not support that " +
-"feature. Ignoring this configuration.");
-}
-conf.set(ConfigUtils.USE_PCJ, "false");
+conf.setBoolean(ConfigUtils.USE_PCJ,
ryaDetails.getPCJIndexDetails().isEnabled());
// Mongo does not support entity indexing.
if(ryaDetails.getEntityCentricIndexDetails().isEnabled()) {
log.warn("The install configuration says to enable Entity
Centric indexing, but Mongo RYA does not support " +
"that feature. Ignoring this configuration.");
}
-conf.set(ConfigUtils.USE_ENTITY, "false");
+//TODO mongo now has an entity index, just needs CLI support.
--- End diff --
discussed in person
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160534095
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoDeletePCJ.java
---
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.mongo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Optional;
+
+import org.apache.rya.api.client.DeletePCJ;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.RyaClientException;
+import
org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException;
+import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Lists;
+import com.mongodb.MongoClient;
+import com.mongodb.MongoCredential;
+import com.mongodb.ServerAddress;
+
+/**
+ * A Mongo implementation of {@link DeletePCJ}.
+ */
+public class MongoDeletePCJ implements DeletePCJ {
+private static final Logger log =
LoggerFactory.getLogger(MongoDeletePCJ.class);
+
+private final MongoConnectionDetails connectionDetails;
+private final InstanceExists instanceExists;
+
+/**
+ * Constructs an instance of {@link MongoDeletePCJ}.
+ *
+ * @param connectionDetails - Details to connect to the server. (not
null)
+ * @param instanceExists - The interactor used to check if a Rya
instance exists. (not null)
+ */
+public MongoDeletePCJ(
+final MongoConnectionDetails connectionDetails,
+final MongoInstanceExists instanceExists) {
+this.connectionDetails = requireNonNull(connectionDetails);
+this.instanceExists = requireNonNull(instanceExists);
+}
+
+@Override
+public void deletePCJ(final String ryaInstanceName, final String
pcjId) throws InstanceDoesNotExistException, RyaClientException {
+requireNonNull(ryaInstanceName);
+requireNonNull(pcjId);
+
+// Ensure the Rya Instance exists.
+if (!instanceExists.exists(ryaInstanceName)) {
+throw new InstanceDoesNotExistException(String.format("There
is no Rya instance named '%s'.", ryaInstanceName));
+}
+
+final Optional username = connectionDetails.getUsername();
+final MongoCredential cred;
+final MongoClient client;
+
+final ServerAddress addr = new
ServerAddress(connectionDetails.getHostname(), connectionDetails.getPort());
+if(username.isPresent()) {
+cred = MongoCredential.createCredential(username.get(),
ryaInstanceName, connectionDetails.getPassword().get());
+client = new MongoClient(addr, Lists.newArrayList(cred));
+} else {
+client = new MongoClient(addr);
+}
+
+
+//pcjStore doesn't need to be closed since it doesn't need to
release any resources.
+@SuppressWarnings("resource")
+final MongoPcjStorage pcjStore = new MongoPcjStorage(client,
ryaInstanceName);
--- End diff --
done
---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/172#discussion_r160534139
--- Diff:
extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoInstall.java
---
@@ -147,22 +147,23 @@ private RyaDetails initializeRyaDetails(
final InstallConfiguration installConfig) throws
AlreadyInitializedException, RyaDetailsRepositoryException {
final RyaDetailsRepository detailsRepo = new
MongoRyaInstanceDetailsRepository(adminClient, instanceName);
-// Build the PCJ Index details. [not supported in mongo]
-final PCJIndexDetails.Builder pcjDetailsBuilder =
PCJIndexDetails.builder().setEnabled(false);
+if(installConfig.getFluoPcjAppName().isPresent()) {
+ log.warn("Mongo does not have fluo support, use ignoring the
configured fluo application name: " + installConfig.getFluoPcjAppName().get());
+}
+
+// Build the PCJ Index details.
+final PCJIndexDetails.Builder pcjDetailsBuilder =
PCJIndexDetails.builder()
+.setEnabled(installConfig.isPcjIndexEnabled());
final RyaDetails details = RyaDetails.builder()
// General Metadata
.setRyaInstanceName(instanceName).setRyaVersion(getVersion())
-// FIXME RYA-215 .setGeoIndexDetails(new
GeoIndexDetails(installConfig.isGeoIndexEnabled()))
-
// Secondary Index Values
+// FIXME RYA-215 .setGeoIndexDetails(new
GeoIndexDetails(installConfig.isGeoIndexEnabled()))
.setTemporalIndexDetails(new
TemporalIndexDetails(installConfig.isTemporalIndexEnabled()))
.setFreeTextDetails(new
FreeTextIndexDetails(installConfig.isFreeTextIndexEnabled()))//
-
-// Entity centric indexing is not supported in Mongo DB.
-.setEntityCentricIndexDetails(new
EntityCentricIndexDetails(false))
-
+.setEntityCentricIndexDetails(new
EntityCentricIndexDetails(installConfig.isEntityCentrixIndexEnabled()))
--- End diff --
discussed in person
---
