[GitHub] incubator-rya pull request #307: Add rep-synch functionality
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/307 Add rep-synch functionality ## Description >What Changed? The parent can now refresh/synch the child. Added GUI entry point into the rep/synch tool. The functionality is now more thread safe. ### Tests >Coverage? Updated tests ### Links ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @kchilton2 @ejwhite922 @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya repsynch_updatingChild Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/307.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #307 commit 05ab6a745425a2ef9505c1c111eb8222982830d5 Author: Andrew Smith Date: 2018-06-26T16:38:34Z Add rep-synch functionality The parent can now refresh/synch the child. Added GUI entry point into the rep/synch tool. The functionality is now more thread safe. ---
[GitHub] incubator-rya pull request #306: Repsynch updating child
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/306 ---
[GitHub] incubator-rya pull request #306: Repsynch updating child
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/306 Repsynch updating child ## Description >What Changed? Each type of rep/synch is its own runnable now to be more thread safe. A GUI tool was created as an entry point into rep/synch instead of CLI. ### Tests >Coverage? updated the tests. ### Links ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @kchilton2 @pujav65 @ejwhite922 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya repsynch_updatingChild Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/306.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #306 commit 04a4910ee01d924abb3f1ac187a666f3a2433a6f Author: Andrew Smith Date: 2018-06-26T16:38:34Z Fixes made the Merge Tool Conflicts: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBRyaDAO.java extras/rya.export/export.client/src/main/java/org/apache/rya/export/client/MergeDriverClient.java extras/rya.export/export.client/src/main/java/org/apache/rya/export/client/conf/MergeConfigHadoopAdapter.java extras/rya.export/export.client/src/main/java/org/apache/rya/export/client/merge/MemoryTimeMerger.java commit c6dbc8ce21fc33d613076aa148f6a8dbd24f3438 Author: Andrew Smith Date: 2018-09-18T19:21:57Z Add rep-synch functionality The parent can now refresh/synch the child. Added GUI entry point into the rep/synch tool. The functionality is now more thread safe. ---
[GitHub] incubator-rya pull request #304: RYA-501 Change guava Iterators to Collectio...
GitHub user isper3at reopened a pull request: https://github.com/apache/incubator-rya/pull/304 RYA-501 Change guava Iterators to Collections ## Description >What Changed? Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ### Tests >Coverage? N/A ### Links [Jira](https://issues.apache.org/jira/browse/RYA-501) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @ejwhite922 @kchilton2 @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-501_guavaToJava-4.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/304.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #304 commit 93c1d89dafc16d4d50bc4581ec6ada311ca494ae Author: Andrew Smith Date: 2018-09-14T16:37:16Z RYA-501 Change guava Iterators to Collections Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ---
[GitHub] incubator-rya pull request #305: Rya 135 collection name
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/305 Rya 135 collection name ## Description >What Changed? Merging existing content into master. Already reviewed and merged into 3.x You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-135_collectionName Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/305.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #305 commit 942d9d62350815a33f28a22f101cdb5c262c70ed Author: Andrew Smith Date: 2018-09-14T16:33:10Z RYA-135 Hard code triples collection There is no need for the triples collection to be configurable, as only the mongo database name will change the rya instance. commit 010f40de276b2152451a42e1c830a2ad88b94860 Author: Andrew Smith Date: 2018-09-14T21:06:27Z Responded to code review removed all references to mongo collection prefix commit 3593c535ddf387d656cde87e793be7bb8980289e Author: Andrew Smith Date: 2018-09-14T22:19:56Z More code review ---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/300 ---
[GitHub] incubator-rya pull request #302: RYA-501 Change guava Iterators to Collectio...
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/302 ---
[GitHub] incubator-rya issue #304: RYA-501 Change guava Iterators to Collections
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/304 Closing until the #302 is in ---
[GitHub] incubator-rya issue #303: RYA-135 Hard code triples collection
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/303 opening a new PR against RYA master after 3.x https://github.com/apache/incubator-rya/pull/300 is in ---
[GitHub] incubator-rya pull request #303: RYA-135 Hard code triples collection
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/303 ---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/300#discussion_r217849654 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/AbstractMongoIndexer.java --- @@ -73,12 +72,12 @@ protected T storageStrategy; private MongoDbBatchWriter mongoDbBatchWriter; +protected String collectionName; --- End diff -- getCollectionName() would build the collection name with the collection suffix and rya prefix ---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/300#discussion_r217849215 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/AbstractMongoIndexer.java --- @@ -73,12 +72,12 @@ protected T storageStrategy; private MongoDbBatchWriter mongoDbBatchWriter; +protected String collectionName; --- End diff -- yes, the implementations of AbstractMongoIndexer set it in the init() function. I can't have it in the constructor because of the hadoop/reflexive construction ---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/300#discussion_r217847761 --- Diff: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBRdfConfiguration.java --- @@ -197,22 +195,22 @@ public void setMongoPassword(final String password) { * @return The name of the Rya instance to connect to. (default: rya) */ public String getRyaInstanceName() { -return get(MONGO_COLLECTION_PREFIX, "rya"); +return get(MONGO_DB_NAME, "rya"); --- End diff -- yeah that's a good point ---
[GitHub] incubator-rya pull request #300: RYA-135 Hard code triples collection
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/300#discussion_r217845838 --- Diff: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/MongoDBRdfConfiguration.java --- @@ -274,16 +274,17 @@ public boolean getUseAggregationPipeline() { * on their child subtrees. * @param value whether to use aggregation pipeline optimization. */ -public void setUseAggregationPipeline(boolean value) { +public void setUseAggregationPipeline(final boolean value) { setBoolean(USE_AGGREGATION_PIPELINE, value); } @Override public List> getOptimizers() { -List> optimizers = super.getOptimizers(); +final List> optimizers = super.getOptimizers(); if (getUseAggregationPipeline()) { -Class cl = AggregationPipelineQueryOptimizer.class; +final Class cl = AggregationPipelineQueryOptimizer.class; @SuppressWarnings("unchecked") +final --- End diff -- fixed ---
[GitHub] incubator-rya pull request #304: RYA-501 Change guava Iterators to Collectio...
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/304 RYA-501 Change guava Iterators to Collections ## Description >What Changed? Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ### Tests >Coverage? N/A ### Links [Jira](https://issues.apache.org/jira/browse/RYA-501) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @ejwhite922 @kchilton2 @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-501_guavaToJava-4.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/304.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #304 commit 93c1d89dafc16d4d50bc4581ec6ada311ca494ae Author: Andrew Smith Date: 2018-09-14T16:37:16Z RYA-501 Change guava Iterators to Collections Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ---
[GitHub] incubator-rya issue #301: RYA-501 change from Guava Iterators to Java Collec...
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/301 changed to -> https://github.com/apache/incubator-rya/pull/302 ---
[GitHub] incubator-rya pull request #302: RYA-501 Change guava Iterators to Collectio...
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/302 RYA-501 Change guava Iterators to Collections ## Description >What Changed? Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ### Tests >Coverage? N/A ### Links [Jira](https://issues.apache.org/jira/browse/RYA-501) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @pujav65 @kchilton2 @ejwhite922 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-501_guavaToJava Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/302.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #302 commit 03fd52e89b44fd39ae97490f43d4922b39e7d07d Author: Andrew Smith Date: 2018-09-14T16:37:16Z RYA-501 Change guava Iterators to Collections Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. Using Java's Collections.emptyIterator() instead. ---
[GitHub] incubator-rya pull request #301: RYA-501 change from Guava Iterators to Java...
Github user isper3at closed the pull request at: https://github.com/apache/incubator-rya/pull/301 ---
[GitHub] incubator-rya pull request #301: Version fix
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/301 Version fix ## Description >What Changed? Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. ### Tests >Coverage? ### Links ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @pujav65 @kchilton2 @ejwhite922 @jessehatfield You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya versionFix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/301.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #301 commit ef38e2efe45699d9916e3bb0696a8cfb6b6a171d Author: Andrew Smith Date: 2018-09-14T16:37:16Z Version fix Google changed the Iterators object emptyIterator() visibility. This can cause versioning issues with anything depending on a newer version of guava. ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187739974 --- Diff: extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/sink/RyaSinkTask.java --- @@ -0,0 +1,144 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.api.sink; + +import static java.util.Objects.requireNonNull; + +import java.util.Collection; +import java.util.Map; +import java.util.Set; + +import org.apache.kafka.clients.consumer.OffsetAndMetadata; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.connect.errors.ConnectException; +import org.apache.kafka.connect.sink.SinkRecord; +import org.apache.kafka.connect.sink.SinkTask; +import org.eclipse.rdf4j.model.Statement; +import org.eclipse.rdf4j.repository.sail.SailRepository; +import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection; +import org.eclipse.rdf4j.sail.Sail; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.jcabi.manifests.Manifests; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; +import edu.umd.cs.findbugs.annotations.Nullable; + +/** + * Handles the common components required to write {@link Statement}s to Rya. + * + * Implementations of this class only need to specify functionality that is specific to the + * Rya implementation. + */ +@DefaultAnnotation(NonNull.class) +public abstract class RyaSinkTask extends SinkTask { +private static final Logger log = LoggerFactory.getLogger(RyaSinkTask.class); + +@Nullable +private SailRepository sailRepo = null; + +@Nullable +private SailRepositoryConnection conn = null; + +/** + * Throws an exception if the configured Rya Instance is not already installed + * within the configured database. + * + * @param taskConfig - The configuration values that were provided to the task. (not null) + * @throws ConnectException The configured Rya Instance is not installed to the configured database + * or we were unable to figure out if it is installed. + */ +protected abstract void checkRyaInstanceExists(final Map<String, String> taskConfig) throws ConnectException; + +/** + * Creates an initialized {@link Sail} object that may be used to write {@link Statement}s to the configured + * Rya Instance. + * + * @param taskConfig - Configures how the Sail object will be created. (not null) + * @return The created Sail object. + * @throws ConnectException The Sail object could not be made. + */ +protected abstract Sail makeSail(final Map<String, String> taskConfig) throws ConnectException; + +@Override +public String version() { +return Manifests.exists("Build-Version") ? Manifests.read("Build-Version"): "UNKNOWN"; +} + +@Override +public void start(final Map<String, String> props) throws ConnectException { +requireNonNull(props); + +// Ensure the configured Rya Instance is installed within the configured database. +checkRyaInstanceExists(props); + +// Create the Sail object that is connected to the Rya Instance. +final Sail sail = makeSail(props); +sailRepo = new SailRepository( sail ); +conn = sailRepo.getConnection(); +} + +@Override +public void put(final Collection records) { +requireNonNull(records); + +// Return immediately if there are no records to handle. +if(records.isEmpty()) { +return; +} + +// If a transaction has not been started yet, then start
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187739885 --- Diff: extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/sink/RyaSinkTask.java --- @@ -0,0 +1,144 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.api.sink; + +import static java.util.Objects.requireNonNull; + +import java.util.Collection; +import java.util.Map; +import java.util.Set; + +import org.apache.kafka.clients.consumer.OffsetAndMetadata; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.connect.errors.ConnectException; +import org.apache.kafka.connect.sink.SinkRecord; +import org.apache.kafka.connect.sink.SinkTask; +import org.eclipse.rdf4j.model.Statement; +import org.eclipse.rdf4j.repository.sail.SailRepository; +import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection; +import org.eclipse.rdf4j.sail.Sail; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.jcabi.manifests.Manifests; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; +import edu.umd.cs.findbugs.annotations.Nullable; + +/** + * Handles the common components required to write {@link Statement}s to Rya. + * + * Implementations of this class only need to specify functionality that is specific to the + * Rya implementation. + */ +@DefaultAnnotation(NonNull.class) --- End diff -- gotcha, thanks for the doc paste dump ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187739762 --- Diff: extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/StatementsSerializer.java --- @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.api; + +import java.io.ByteArrayOutputStream; +import java.util.Map; +import java.util.Set; + +import org.apache.kafka.common.serialization.Serializer; +import org.eclipse.rdf4j.model.Statement; +import org.eclipse.rdf4j.rio.RDFWriter; +import org.eclipse.rdf4j.rio.binary.BinaryRDFWriterFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * A Kafka {@link Serializer} that is able to serialize a set of {@link Statement}s + * using the RDF4J Rio Binary format. --- End diff -- fair enough, it just seemed like you are delegating the actual serialization to something else. ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187739297 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/ConfigUtils.java --- @@ -438,6 +438,9 @@ public static boolean getUsePcjUpdaterIndex(final Configuration conf) { return Optional.fromNullable(conf.get(FLUO_APP_NAME)); } +public static void setUseMongo(final Configuration conf, final boolean useMongo) { --- End diff -- sure makes sense ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187739254 --- Diff: dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/AccumuloRdfConfiguration.java --- @@ -62,14 +62,14 @@ public AccumuloRdfConfiguration() { super(); } -public AccumuloRdfConfiguration(Configuration other) { +public AccumuloRdfConfiguration(final Configuration other) { super(other); } -public AccumuloRdfConfigurationBuilder getBuilder() { +public static AccumuloRdfConfigurationBuilder getBuilder() { --- End diff -- ð ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187717175 --- Diff: extras/kafka.connect/mongo-it/src/test/java/org/apache/rya/kafka/connect/mongo/MongoRyaSinkTaskIT.java --- @@ -0,0 +1,97 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.mongo; + +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +import org.apache.kafka.connect.errors.ConnectException; +import org.apache.rya.api.client.Install.InstallConfiguration; +import org.apache.rya.api.client.RyaClient; +import org.apache.rya.api.client.mongo.MongoConnectionDetails; +import org.apache.rya.api.client.mongo.MongoRyaClientFactory; +import org.apache.rya.test.mongo.MongoITBase; +import org.junit.Test; + +/** + * Integration tests the methods of {@link MongoRyaSinkTask}. + */ +public class MongoRyaSinkTaskIT extends MongoITBase { + +@Test +public void instanceExists() throws Exception { +// Install an instance of Rya. +final String ryaInstanceName = "rya"; +final MongoConnectionDetails connectionDetails = new MongoConnectionDetails( +super.getMongoHostname(), +super.getMongoPort(), +Optional.empty(), +Optional.empty()); + +final InstallConfiguration installConfig = InstallConfiguration.builder() +.setEnableTableHashPrefix(false) +.setEnableEntityCentricIndex(false) +.setEnableFreeTextIndex(false) +.setEnableTemporalIndex(false) +.setEnablePcjIndex(false) +.setEnableGeoIndex(false) +.build(); + +final RyaClient ryaClient = MongoRyaClientFactory.build(connectionDetails, super.getMongoClient()); +ryaClient.getInstall().install(ryaInstanceName, installConfig); + +// Create the task that will be tested. +final MongoRyaSinkTask task = new MongoRyaSinkTask(); + +try { +// Configure the task to use the embedded Mongo DB instance for Rya. +final Map<String, String> config = new HashMap<>(); +config.put(MongoRyaSinkConfig.HOSTNAME, super.getMongoHostname()); +config.put(MongoRyaSinkConfig.PORT, "" + super.getMongoPort()); +config.put(MongoRyaSinkConfig.RYA_INSTANCE_NAME, "rya"); + +// This will pass because the Rya instance exists. +task.start(config); +} finally { +task.stop(); +} +} + +@Test(expected = ConnectException.class) +public void instanceDoesNotExist() throws Exception { +// Create the task that will be tested. +final MongoRyaSinkTask task = new MongoRyaSinkTask(); + +try { +// Configure the task to use the embedded Mongo DB instance for Rya. +final Map<String, String> config = new HashMap<>(); +config.put(MongoRyaSinkConfig.HOSTNAME, super.getMongoHostname()); +config.put(MongoRyaSinkConfig.PORT, "" + super.getMongoPort()); +config.put(MongoRyaSinkConfig.RYA_INSTANCE_NAME, "instance-does-not-exist"); + +// Starting the task will fail because the Rya instance does not exist. +task.start(config); +} finally { +task.stop(); +} +} + +// TODO show that inserts using visibilities work. --- End diff -- TODO? ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187717488 --- Diff: extras/kafka.connect/mongo/src/main/java/org/apache/rya/kafka/connect/mongo/MongoRyaSinkConfig.java --- @@ -0,0 +1,94 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.mongo; + +import java.util.Map; + +import org.apache.kafka.common.config.ConfigDef; +import org.apache.kafka.common.config.ConfigDef.Importance; +import org.apache.kafka.common.config.ConfigDef.Type; +import org.apache.rya.kafka.connect.api.sink.RyaSinkConfig; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * A Kafka Connect configuration that is used to configure {@link MongoRyaSinkConnector}s and {@link MongoRyaSinkTask}s. + */ +@DefaultAnnotation(NonNull.class) +public class MongoRyaSinkConfig extends RyaSinkConfig { + +public static final String HOSTNAME = "mongo.hostname"; +private static final String HOSTNAME_DOC = "The Mongo DB hostname the Sail connections wlll use."; --- End diff -- typo: wlll ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187718927 --- Diff: extras/kafka.connect/mongo/src/main/java/org/apache/rya/kafka/connect/mongo/MongoRyaSinkConnector.java --- @@ -0,0 +1,63 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.mongo; + +import java.util.Map; + +import org.apache.kafka.common.config.AbstractConfig; +import org.apache.kafka.common.config.ConfigDef; +import org.apache.kafka.connect.connector.Task; +import org.apache.rya.kafka.connect.api.sink.RyaSinkConnector; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; +import edu.umd.cs.findbugs.annotations.Nullable; + +/** + * A {@link RyaSinkConnector} that uses a Mongo DB Rya backend when creating tasks. + */ +@DefaultAnnotation(NonNull.class) +public class MongoRyaSinkConnector extends RyaSinkConnector { + +@Nullable +private MongoRyaSinkConfig config = null; + +@Override +public void start(final Map<String, String> props) { +this.config = new MongoRyaSinkConfig( props ); +} + +@Override +protected AbstractConfig getConfig() { +if(config == null) { +throw new IllegalStateException("The configuration has not been set yet. Invoke start(props) first."); --- End diff -- same start(Map) ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187698795 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/ConfigUtils.java --- @@ -438,6 +438,9 @@ public static boolean getUsePcjUpdaterIndex(final Configuration conf) { return Optional.fromNullable(conf.get(FLUO_APP_NAME)); } +public static void setUseMongo(final Configuration conf, final boolean useMongo) { --- End diff -- can you add this to the constructor of the MongoDbRDFConfiguration constructor? if we're going to keep using this field, it would make sense for that to set it. ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187708244 --- Diff: extras/kafka.connect/accumulo/src/main/java/org/apache/rya/kafka/connect/accumulo/AccumuloRyaSinkConnector.java --- @@ -0,0 +1,63 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.accumulo; + +import java.util.Map; + +import org.apache.kafka.common.config.AbstractConfig; +import org.apache.kafka.common.config.ConfigDef; +import org.apache.kafka.connect.connector.Task; +import org.apache.rya.kafka.connect.api.sink.RyaSinkConnector; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; +import edu.umd.cs.findbugs.annotations.Nullable; + +/** + * A {@link RyaSinkConnector} that uses an Accumulo Rya backend when creating tasks. + */ +@DefaultAnnotation(NonNull.class) +public class AccumuloRyaSinkConnector extends RyaSinkConnector { + +@Nullable +private AccumuloRyaSinkConfig config = null; + +@Override +public void start(final Map<String, String> props) { +this.config = new AccumuloRyaSinkConfig( props ); +} + +@Override +protected AbstractConfig getConfig() { +if(config == null) { +throw new IllegalStateException("The configuration has not been set yet. Invoke start(props) first."); --- End diff -- usually the doc'd function has the parameters be the type, not the name: start(Map) ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187708536 --- Diff: extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/sink/RyaSinkTask.java --- @@ -0,0 +1,144 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.api.sink; + +import static java.util.Objects.requireNonNull; + +import java.util.Collection; +import java.util.Map; +import java.util.Set; + +import org.apache.kafka.clients.consumer.OffsetAndMetadata; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.connect.errors.ConnectException; +import org.apache.kafka.connect.sink.SinkRecord; +import org.apache.kafka.connect.sink.SinkTask; +import org.eclipse.rdf4j.model.Statement; +import org.eclipse.rdf4j.repository.sail.SailRepository; +import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection; +import org.eclipse.rdf4j.sail.Sail; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.jcabi.manifests.Manifests; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; +import edu.umd.cs.findbugs.annotations.Nullable; + +/** + * Handles the common components required to write {@link Statement}s to Rya. + * + * Implementations of this class only need to specify functionality that is specific to the + * Rya implementation. + */ +@DefaultAnnotation(NonNull.class) --- End diff -- maybe I'm not clear what this annotation does, but again, all the fields are declared Nullable ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187698179 --- Diff: dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/AccumuloRdfConfiguration.java --- @@ -62,14 +62,14 @@ public AccumuloRdfConfiguration() { super(); } -public AccumuloRdfConfiguration(Configuration other) { +public AccumuloRdfConfiguration(final Configuration other) { super(other); } -public AccumuloRdfConfigurationBuilder getBuilder() { +public static AccumuloRdfConfigurationBuilder getBuilder() { --- End diff -- isn't the convention usually builder() for static builder functions? ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187702555 --- Diff: extras/kafka.connect/accumulo/src/main/java/org/apache/rya/kafka/connect/accumulo/AccumuloRyaSinkTask.java --- @@ -0,0 +1,111 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.accumulo; + +import static java.util.Objects.requireNonNull; + +import java.util.Map; + +import org.apache.accumulo.core.client.AccumuloException; +import org.apache.accumulo.core.client.AccumuloSecurityException; +import org.apache.accumulo.core.client.Connector; +import org.apache.accumulo.core.client.Instance; +import org.apache.accumulo.core.client.ZooKeeperInstance; +import org.apache.accumulo.core.client.security.tokens.PasswordToken; +import org.apache.kafka.connect.errors.ConnectException; +import org.apache.rya.accumulo.AccumuloRdfConfiguration; +import org.apache.rya.api.client.RyaClient; +import org.apache.rya.api.client.RyaClientException; +import org.apache.rya.api.client.accumulo.AccumuloConnectionDetails; +import org.apache.rya.api.client.accumulo.AccumuloRyaClientFactory; +import org.apache.rya.api.persist.RyaDAOException; +import org.apache.rya.kafka.connect.api.sink.RyaSinkTask; +import org.apache.rya.rdftriplestore.inference.InferenceEngineException; +import org.apache.rya.sail.config.RyaSailFactory; +import org.eclipse.rdf4j.sail.Sail; +import org.eclipse.rdf4j.sail.SailException; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * A {@link RyaSinkTask} that uses the Accumulo implementation of Rya to store data. + */ +@DefaultAnnotation(NonNull.class) +public class AccumuloRyaSinkTask extends RyaSinkTask { + +@Override +protected void checkRyaInstanceExists(final Map<String, String> taskConfig) throws ConnectException { +requireNonNull(taskConfig); + +// Parse the configuration object. +final AccumuloRyaSinkConfig config = new AccumuloRyaSinkConfig(taskConfig); + +// Connect to the instance of Accumulo. +final Connector connector; +try { +final Instance instance = new ZooKeeperInstance(config.getClusterName(), config.getZookeepers()); +connector = instance.getConnector(config.getUsername(), new PasswordToken( config.getPassword() )); +} catch (AccumuloException | AccumuloSecurityException e) { +throw new ConnectException("Could not create a Connector to the configured Accumulo instance.", e); +} + +// Use a RyaClient to see if the configured instance exists. +try { +final AccumuloConnectionDetails connectionDetails = new AccumuloConnectionDetails( +config.getUsername(), +config.getPassword().toCharArray(), +config.getClusterName(), +config.getZookeepers()); +final RyaClient client = AccumuloRyaClientFactory.build(connectionDetails, connector); + +if(!client.getInstanceExists().exists( config.getRyaInstanceName() )) { +throw new ConnectException("The Rya Instance named " + +config.getRyaInstanceName() + " has not been installed."); +} + +} catch (final RyaClientException e) { +throw new ConnectException("Unable to determine if the Rya Instance named " + +config.getRyaInstanceName() + " has been installed.", e); +} +} + +@Override +protected Sail makeSail(final Map<String, String> taskConfig) throws ConnectException { +requireNonNull(taskConfig); + +// Par
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187706947 --- Diff: extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/StatementsSerializer.java --- @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.api; + +import java.io.ByteArrayOutputStream; +import java.util.Map; +import java.util.Set; + +import org.apache.kafka.common.serialization.Serializer; +import org.eclipse.rdf4j.model.Statement; +import org.eclipse.rdf4j.rio.RDFWriter; +import org.eclipse.rdf4j.rio.binary.BinaryRDFWriterFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * A Kafka {@link Serializer} that is able to serialize a set of {@link Statement}s + * using the RDF4J Rio Binary format. --- End diff -- i feel like it might be worthwhile to mention that you use the RDFParser and RDFWriter here and the Deserializer respectively. ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187709021 --- Diff: extras/kafka.connect/api/src/main/java/org/apache/rya/kafka/connect/api/sink/RyaSinkTask.java --- @@ -0,0 +1,144 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.api.sink; + +import static java.util.Objects.requireNonNull; + +import java.util.Collection; +import java.util.Map; +import java.util.Set; + +import org.apache.kafka.clients.consumer.OffsetAndMetadata; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.connect.errors.ConnectException; +import org.apache.kafka.connect.sink.SinkRecord; +import org.apache.kafka.connect.sink.SinkTask; +import org.eclipse.rdf4j.model.Statement; +import org.eclipse.rdf4j.repository.sail.SailRepository; +import org.eclipse.rdf4j.repository.sail.SailRepositoryConnection; +import org.eclipse.rdf4j.sail.Sail; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.jcabi.manifests.Manifests; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; +import edu.umd.cs.findbugs.annotations.Nullable; + +/** + * Handles the common components required to write {@link Statement}s to Rya. + * + * Implementations of this class only need to specify functionality that is specific to the + * Rya implementation. + */ +@DefaultAnnotation(NonNull.class) +public abstract class RyaSinkTask extends SinkTask { +private static final Logger log = LoggerFactory.getLogger(RyaSinkTask.class); + +@Nullable +private SailRepository sailRepo = null; + +@Nullable +private SailRepositoryConnection conn = null; + +/** + * Throws an exception if the configured Rya Instance is not already installed + * within the configured database. + * + * @param taskConfig - The configuration values that were provided to the task. (not null) + * @throws ConnectException The configured Rya Instance is not installed to the configured database + * or we were unable to figure out if it is installed. + */ +protected abstract void checkRyaInstanceExists(final Map<String, String> taskConfig) throws ConnectException; + +/** + * Creates an initialized {@link Sail} object that may be used to write {@link Statement}s to the configured + * Rya Instance. + * + * @param taskConfig - Configures how the Sail object will be created. (not null) + * @return The created Sail object. + * @throws ConnectException The Sail object could not be made. + */ +protected abstract Sail makeSail(final Map<String, String> taskConfig) throws ConnectException; + +@Override +public String version() { +return Manifests.exists("Build-Version") ? Manifests.read("Build-Version"): "UNKNOWN"; +} + +@Override +public void start(final Map<String, String> props) throws ConnectException { +requireNonNull(props); + +// Ensure the configured Rya Instance is installed within the configured database. +checkRyaInstanceExists(props); + +// Create the Sail object that is connected to the Rya Instance. +final Sail sail = makeSail(props); +sailRepo = new SailRepository( sail ); +conn = sailRepo.getConnection(); +} + +@Override +public void put(final Collection records) { +requireNonNull(records); + +// Return immediately if there are no records to handle. +if(records.isEmpty()) { +return; +} + +// If a transaction has not been started yet, then start
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187699543 --- Diff: extras/kafka.connect/README.md --- @@ -0,0 +1,22 @@ + + +The parent project for all Rya Kafka Connect work. All projects thare are part +of that system must use this project's pom as their parent pom. --- End diff -- typo All projects that* are ---
[GitHub] incubator-rya pull request #296: RYA-487 Kafka Connect Sinks
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/296#discussion_r187701248 --- Diff: extras/kafka.connect/accumulo/src/main/java/org/apache/rya/kafka/connect/accumulo/AccumuloRyaSinkConnector.java --- @@ -0,0 +1,63 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.kafka.connect.accumulo; + +import java.util.Map; + +import org.apache.kafka.common.config.AbstractConfig; +import org.apache.kafka.common.config.ConfigDef; +import org.apache.kafka.connect.connector.Task; +import org.apache.rya.kafka.connect.api.sink.RyaSinkConnector; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; +import edu.umd.cs.findbugs.annotations.Nullable; + +/** + * A {@link RyaSinkConnector} that uses an Accumulo Rya backend when creating tasks. + */ +@DefaultAnnotation(NonNull.class) --- End diff -- since there is one field that is marked nullable, is this annotation usefull? ---
[GitHub] incubator-rya pull request #294: RYA-492 Added language support for Literals
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/294#discussion_r186170248 --- Diff: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java --- @@ -194,6 +204,7 @@ public BasicDBObject serializeInternal(final RyaStatement statement){ .append(OBJECT, statement.getObject().getData()) .append(OBJECT_HASH, hash(statement.getObject().getData())) .append(OBJECT_TYPE, statement.getObject().getDataType().toString()) +.append(OBJECT_LANGUAGE, statement.getObject().getLanguage()) --- End diff -- can language be null here? ---
[GitHub] incubator-rya pull request #287: Updating the ExecuteSparqlQuery command to ...
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/287 Updating the ExecuteSparqlQuery command to return BindingSets ## Description >What Changed? Updated the execute sparql interactor to return binding sets instead of a string. ### Tests >Coverage? Updated existing tests ### Links [Jira](https://issues.apache.org/jira/browse/RYA-484) ### Checklist - [ ] Code Review - [ ] Squash Commits People To Reivew @kchilton2 @jdasch @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-484_sparqlQuery Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/287.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #287 commit 8169433be584e6c9ea0cc568621eabde809a2eae Author: Andrew Smith <smith3at@...> Date: 2018-04-06T20:01:43Z Updating the ExecuteSparqlQuery command to return BindingSets ---
[GitHub] incubator-rya pull request #286: RYA-467 update topic cleanup.policy
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/286 RYA-467 update topic cleanup.policy ## Description >What Changed? Added a topic properties builder. Only one property is currently in the builder, but adding others should be easy. ### Tests >Coverage? ### Links [Jira](https://issues.apache.org/jira/browse/RYA-467) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @kchilton2 @ejwhite922 @pujav65 @dlotts @jessehatfield You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya RYA-467 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/286.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #286 commit 6565ca4b28c04398c799b5507b0237828a5eb613 Author: Andrew Smith <smith3at@...> Date: 2018-04-02T16:32:40Z RYA-467 update topic cleanup.policy Added a topic properties builder. Only one property is currently in the builder, but adding others should be easy. ---
[GitHub] incubator-rya pull request #285: RYA-469 Added tests for Rya Streams join it...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/285#discussion_r178368019 --- Diff: extras/indexingExample/src/main/java/MongoRyaDirectExample.java --- @@ -805,6 +819,38 @@ public static void testAddAndDeleteNoContext(final SailRepositoryConnection conn Validate.isTrue(resultHandler.getCount() == 0); } +public static void testLubmFile(final SailRepositoryConnection conn) throws MalformedQueryException, RepositoryException, +UpdateExecutionException, QueryEvaluationException, TupleQueryResultHandlerException, RDFParseException, IOException { + +final String query = LubmQuery.LUBM_QUERY_14.getSparqlQuery(); +//"PREFIX lubm: <" + LUBM_PREFIX + "> \n" + --- End diff -- remove commented out code ---
[GitHub] incubator-rya pull request #285: RYA-469 Added tests for Rya Streams join it...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/285#discussion_r178368138 --- Diff: extras/rya.streams/client/src/test/java/org/apache/rya/streams/client/command/RunQueryCommandIT.java --- @@ -196,4 +203,74 @@ public void run() { // Show the read results matched the expected ones. assertEquals(expected, results); } + +@Test +public void runQueryFromFile() throws Exception { +// TODO: Maybe test with org.apache.rya.api.utils.LubmQuery.LUBM_QUERY_1,2,3,etc... --- End diff -- remove TODO? ---
[GitHub] incubator-rya issue #282: RYA-443 Fixed Rya Streams Query Manager RPM versio...
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/282 rpm -e does not work when you supply a version number. Not sure if I trust that >..> ---
[GitHub] incubator-rya pull request #279: Rya streams base
GitHub user isper3at opened a pull request: https://github.com/apache/incubator-rya/pull/279 Rya streams base ## Description >What Changed? Rya Streams Feature Branch ### Tests >Coverage? Each major feature has test coverate ### Links [Jira](https://issues.apache.org/jira/browse/RYA-460) ### Checklist - [ ] Code Review - [x] Squash Commits People To Reivew @kchilton2 @ejwhite922 @pujav65 You can merge this pull request into a Git repository by running: $ git pull https://github.com/isper3at/incubator-rya Rya-Streams-base Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rya/pull/279.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #279 commit 28ea2e29a51dc22ba3efb0bdf3aa7fac27648518 Author: Andrew Smith <smith3at@...> Date: 2018-01-16T17:39:34Z RYA-446 Daemonifying Query Manager commit 38c5927768ddeb680a0b3a20155124622d0420c2 Author: Andrew Smith <smith3at@...> Date: 2018-01-18T22:51:40Z RYA-449 Create QueryChangeLogSource commit 01f02535c063361614911a749809afb5cf54b2dc Author: Andrew Smith <smith3at@...> Date: 2018-01-19T20:43:55Z Rya 454 added QueryExecutor interface commit 0be94418dce2229e48cfc760be97d6f90f68fae2 Author: Andrew Smith <smith3at@...> Date: 2018-01-23T20:20:50Z Rya 452 Updated QueryRepository Updated QueryRepository to be a Service Updated InMemoryQueryRepository to be an AbstractScheduledService Added listeners to InMemoryQueryRepository commit fbad2c30c36e1d2df3174bfe95ec68011959689b Author: Andrew Smith <smith3at@...> Date: 2018-01-23T20:44:32Z RYA-455 stopAll queries for a rya instance commit 1dedc48e0a4e311254820ad6662f8fda446cd444 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-23T20:50:17Z RYA-448 Implement JAXB marshalling code for the Query Manager's XML configuration file. commit 2b654b5368faf28f9df73749fdbe87e02ce295e6 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-23T21:17:06Z RYA-450 Implemented a Kafka backed QueryChangeLogSource. commit 3e0a6e89d5cf5f08c5ed5dae14a4f7faf48477e3 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-25T22:19:58Z RYA-446 Create a bin and rpm distribution for the Rya Streams Query Manager application. commit b4b635461cbd0264f1edc2df40e99fa96e7f007b Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-26T17:06:07Z RYA-458 Updating the configuration XML so that you may only specify a single QueryChangeLogSource. commit dc0e803bf8c14ab94c972d43e126588591cf5b91 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-26T20:55:59Z RYA-456 Implement a Single Node implementation of QueryExecutor. commit 33b5d5b1b39b7d3de6c2df1887b0c8dc61ad0fb8 Author: Andrew Smith <smith3at@...> Date: 2018-01-30T19:01:54Z Rya 451 Query manager QueryManager with tests updated InMemoryQueryRepository and its tests commit 69a17be4e66e5dc6ce7ce32f44580f428f18d224 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-01-30T19:44:58Z RYA-453 Implement the Query Manager's Daemon that controls the application. commit e5ed66854c7f67b6ee6eb4c9c445737bc95568fc Author: Andrew Smith <smith3at@...> Date: 2018-02-01T21:04:48Z RYA-446 Service unit file for systemd commit b4d70758a35cc1c6b9a6c5a93e126b7a644d3526 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-01T23:31:41Z RYA-446 Making the Rya Streams Query Manager run as a service on CentOS 7. commit c7802a2f10b089300d71da9fa1b87e2046efa501 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-03T03:47:59Z RYA-451 Fixing threading issues with the QueryManager class. commit 9a44bc2132a680dec40058741ebe59d4a95fe913 Author: eric.white <eric.white@...> Date: 2018-02-27T20:58:37Z RYA-443 Updated query-manager RPM to use Java based plugin commit d31f4f872115d603c676e4f465a7d42114c044cb Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-14T18:49:52Z RYA-466 Update the Rya Streams Client to stream results to file. commit cd91cd1f1aa0f72b59e952d59552f46b56542488 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-08T22:21:10Z RYA-462 Updated the Kafka topic name for StreamsQueries to include the Rya Instance name. Conflicts: extras/rya.streams/client/src/main/java/org/apache/rya/streams/client/command/StreamResultsCommand.java commit 0b7226e0c27f7f24ae131c8c23121f793321bf25 Author: kchilton2 <kevin.e.chilton@...> Date: 2018-02-08T22:53:41Z RYA-463 RYA-464 Added an isInsert flag to StreamsQuery to indicate when the results of a query need to be inserted back into Rya. Conflicts: common/rya.api/src/
[GitHub] incubator-rya pull request #275: RYA-466 Update the Rya Streams Client to st...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/275#discussion_r170314969 --- Diff: common/rya.api/src/main/java/org/apache/rya/api/utils/QueryInvestigator.java --- @@ -0,0 +1,104 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.api.utils; + +import static java.util.Objects.requireNonNull; + +import java.util.regex.Pattern; + +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.parser.sparql.SPARQLParser; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * A utility class that is used to glean insight into the structure of SPARQL queries. + */ +@DefaultAnnotation(NonNull.class) +public class QueryInvestigator { + +private static final SPARQLParser PARSER = new SPARQLParser(); + +private QueryInvestigator() { } + +/** + * Determines whether a SPARQL command is a CONSTRUCT or not. + * + * @param sparql - The SPARQL to evaluate. (not null) + * @return {@code true} if the provided SPARQL is a CONSTRUCT query; otherwise {@code false}. + * @throws MalformedQueryException The SPARQL is neither a well formed query or update. + */ +public static boolean isConstruct(final String sparql) throws MalformedQueryException { +requireNonNull(sparql); + +try { +// Constructs are queries, so try to create a ParsedQuery. +PARSER.parseQuery(sparql, null); + +// Check to see if the SPARQL looks like a CONSTRUCT query. +return Pattern.matches(".*?construct.*?where.*", sparql.toLowerCase()); + +} catch(final MalformedQueryException queryE) { +try { +// Maybe it's an update. +PARSER.parseUpdate(sparql, null); + +// It was, so return false. +return false; + +} catch(final MalformedQueryException updateE) { +// It's not. Actually malformed. +throw queryE; +} +} +} + +/** + * Determines whether a SPARQL command is an INSERT with a WHERE clause or not. + * + * @param sparql - The SPARQL to evaluate. (not null) + * @return {@code true} if the provided SPARQL is an INSERT update; otherwise {@code false}. + * @throws MalformedQueryException The SPARQL is neither a well formed query or update. + */ +public static boolean isInsertWhere(final String sparql) throws MalformedQueryException { +requireNonNull(sparql); + +try { +// Inserts are updated, so try to create a ParsedUpdate. +PARSER.parseUpdate(sparql, null); + +// Check to see if the SPARQL looks like an INSERT query. +return Pattern.matches(".*?insert.*?where.*", sparql.toLowerCase()); --- End diff -- odd thought, what if the query is something like INSERT { a ?where b } ---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/267#discussion_r163100423 --- Diff: extras/rya.streams/kafka/src/main/java/org/apache/rya/streams/kafka/KafkaRyaStreamsClientFactory.java --- @@ -0,0 +1,169 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.streams.kafka; + +import static java.util.Objects.requireNonNull; + +import java.util.Properties; +import java.util.UUID; + +import org.apache.kafka.clients.CommonClientConfigs; +import org.apache.kafka.clients.consumer.Consumer; +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.Producer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.common.serialization.Deserializer; +import org.apache.kafka.common.serialization.Serializer; +import org.apache.kafka.common.serialization.StringDeserializer; +import org.apache.kafka.common.serialization.StringSerializer; +import org.apache.rya.streams.api.RyaStreamsClient; +import org.apache.rya.streams.api.interactor.defaults.DefaultAddQuery; +import org.apache.rya.streams.api.interactor.defaults.DefaultDeleteQuery; +import org.apache.rya.streams.api.interactor.defaults.DefaultGetQuery; +import org.apache.rya.streams.api.interactor.defaults.DefaultListQueries; +import org.apache.rya.streams.api.interactor.defaults.DefaultStartQuery; +import org.apache.rya.streams.api.interactor.defaults.DefaultStopQuery; +import org.apache.rya.streams.api.queries.InMemoryQueryRepository; +import org.apache.rya.streams.api.queries.QueryChange; +import org.apache.rya.streams.api.queries.QueryChangeLog; +import org.apache.rya.streams.api.queries.QueryRepository; +import org.apache.rya.streams.kafka.interactor.KafkaGetQueryResultStream; +import org.apache.rya.streams.kafka.queries.KafkaQueryChangeLog; +import org.apache.rya.streams.kafka.serialization.VisibilityBindingSetDeserializer; +import org.apache.rya.streams.kafka.serialization.VisibilityStatementDeserializer; +import org.apache.rya.streams.kafka.serialization.queries.QueryChangeDeserializer; +import org.apache.rya.streams.kafka.serialization.queries.QueryChangeSerializer; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * Constructs instances of {@link RyaStreamsClient} that are connected to a Kafka cluster. + */ +@DefaultAnnotation(NonNull.class) +public final class KafkaRyaStreamsClientFactory { +private static final Logger log = LoggerFactory.getLogger(KafkaRyaStreamsClientFactory.class); + +/** + * Initialize a {@link RyaStreamsClient} that will interact with an instance of Rya Streams + * that is backed by Kafka. + * + * @param ryaInstance - The name of the Rya Instance the client is connected to. (not null) + * @param kafkaHostname - The hostname of the Kafka Broker. + * @param kafkaPort - The port of the Kafka Broker. + * @return The initialized commands. + */ +public static RyaStreamsClient make( +final String ryaInstance, +final String kafkaHostname, +final int kafkaPort) { +requireNonNull(kafkaHostname); + +// Setup Query Repository used by the Kafka Rya Streams subsystem. +final Producer queryProducer = +makeProducer(kafkaHostname, kafkaPort, StringSerializer.class, QueryChangeSerializer.class); +final ConsumerqueryConsumer = +fromStartConsumer(kafkaHostname, kafkaPort, StringDeserializer.class, QueryChangeDeserializer.class); +final S
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/267#discussion_r163100365 --- Diff: extras/indexing/src/test/java/org/apache/rya/api/client/accumulo/AccumuloSetRyaStreamsConfigurationIT.java --- @@ -0,0 +1,81 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.api.client.accumulo; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + +import org.apache.rya.accumulo.AccumuloITBase; +import org.apache.rya.api.client.Install; +import org.apache.rya.api.client.Install.InstallConfiguration; +import org.apache.rya.api.client.InstanceDoesNotExistException; +import org.apache.rya.api.client.RyaClient; +import org.apache.rya.api.instance.RyaDetails.RyaStreamsDetails; +import org.junit.Test; + +/** + * Integration tests the methods of {@link AccumuloSetRyaStreamsConfiguration}. + */ +public class AccumuloSetRyaStreamsConfigurationIT extends AccumuloITBase { + +@Test(expected = InstanceDoesNotExistException.class) +public void instanceDoesNotExist() throws Exception { +final String ryaInstance = getRyaInstanceName(); +final AccumuloConnectionDetails connectionDetails = new AccumuloConnectionDetails( +getUsername(), +getPassword().toCharArray(), +getInstanceName(), +getZookeepers()); +final RyaClient ryaClient = AccumuloRyaClientFactory.build(connectionDetails, getConnector()); + +// Skip the install step to create error causing situation. +final RyaStreamsDetails details = new RyaStreamsDetails("localhost", 6); + ryaClient.getSetRyaStreamsConfiguration().setRyaStreamsConfiguration(ryaInstance, details); +} + +@Test +public void updatesRyaDetails() throws Exception { +final String ryaInstance = getRyaInstanceName(); +final AccumuloConnectionDetails connectionDetails = new AccumuloConnectionDetails( +getUsername(), +getPassword().toCharArray(), +getInstanceName(), +getZookeepers()); +final RyaClient ryaClient = AccumuloRyaClientFactory.build(connectionDetails, getConnector()); + +// Install an instance of Rya. +final Install installRya = ryaClient.getInstall(); +final InstallConfiguration installConf = InstallConfiguration.builder() +.build(); +installRya.install(ryaInstance, installConf); + +// Fetch its details and show they do not have any RyaStreamsDetails. +com.google.common.base.Optional streamsDetails = --- End diff -- yeah, still annoying we use 2 ---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/267#discussion_r163100285 --- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoSetRyaStreamsConfiguration.java --- @@ -0,0 +1,60 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.api.client.mongo; + +import static java.util.Objects.requireNonNull; + +import org.apache.rya.api.client.InstanceExists; +import org.apache.rya.api.client.SetRyaStreamsConfiguration; +import org.apache.rya.api.client.SetRyaStreamsConfigurationBase; +import org.apache.rya.api.instance.RyaDetailsRepository; +import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository; + +import com.mongodb.MongoClient; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * A MongoDB implementation of {@link SetRyaStreamsConfiguration}. + */ +@DefaultAnnotation(NonNull.class) +public class MongoSetRyaStreamsConfiguration extends SetRyaStreamsConfigurationBase { + +private final MongoClient client; + +/** + * Constructs an instance of {@link MongoSetRyaStreamsConfiguration}. + * + * @param instanceExists - The interactor used to verify Rya instances exist. (not null) + * @param client - The MongoDB client used to connect to the Rya storage. (not null) + */ +public MongoSetRyaStreamsConfiguration( +final InstanceExists instanceExists, +final MongoClient client) { +super(instanceExists); +this.client = requireNonNull(client); +} + +@Override +protected RyaDetailsRepository getRyaDetailsRepo(final String ryaInstance) { +requireNonNull(ryaInstance); +return new MongoRyaInstanceDetailsRepository(client, ryaInstance); --- End diff -- fair enough ---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/267#discussion_r163084595 --- Diff: extras/rya.streams/api/src/main/java/org/apache/rya/streams/api/interactor/defaults/DefaultGetQuery.java --- @@ -0,0 +1,50 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.streams.api.interactor.defaults; + +import static java.util.Objects.requireNonNull; + +import java.util.Optional; +import java.util.UUID; + +import org.apache.rya.streams.api.entity.StreamsQuery; +import org.apache.rya.streams.api.exception.RyaStreamsException; +import org.apache.rya.streams.api.interactor.GetQuery; +import org.apache.rya.streams.api.queries.QueryRepository; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * Get a {@link StreamsQuery} from Rya Streams. + */ +@DefaultAnnotation(NonNull.class) +public class DefaultGetQuery implements GetQuery { +private final QueryRepository repository; + +public DefaultGetQuery(final QueryRepository repository) { --- End diff -- doc ---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/267#discussion_r163090091 --- Diff: extras/shell/src/test/java/org/apache/rya/shell/util/StreamsQueryFormatterTest.java --- @@ -0,0 +1,103 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.shell.util; + +import static org.junit.Assert.assertEquals; + +import java.util.Set; +import java.util.UUID; + +import org.apache.rya.streams.api.entity.StreamsQuery; +import org.junit.Test; + +import com.google.common.collect.Sets; + +/** + * Unit tests the methods of {@link StreamsQueryFormatter}. + */ +public class StreamsQueryFormatterTest { + +@Test +public void formatQuery() throws Exception { +// Format the query. +final StreamsQuery query = new StreamsQuery( +UUID.fromString("da55cea5-c21c-46a5-ab79-5433eef4efaa"), +"SELECT * WHERE { ?a ?b ?c . }", +true); +final String formatted = StreamsQueryFormatter.format(query); + +// Ensure it has the expected format. +final String expected = +" Query ID: da55cea5-c21c-46a5-ab79-5433eef4efaa\n" + +"Is Active: true\n" + +" SPARQL: select ?a ?b ?c\n" + +" where {\n" + +" ?a ?b ?c.\n" + +" }\n"; + +assertEquals(expected, formatted); +} + +@Test +public void formatQueries() throws Exception { +// Format the queries. +final Set queries = Sets.newHashSet( +new StreamsQuery( + UUID.fromString("----"), +"SELECT * WHERE { ?person ?business . }", +true), +new StreamsQuery( + UUID.fromString("----"), +"SELECT * WHERE { ?a ?b ?c . }", +true), +new StreamsQuery( + UUID.fromString("----"), +"SELECT * WHERE { ?d ?e ?f . }", +false)); + +final String formatted = StreamsQueryFormatter.format(queries); +System.out.println(formatted); --- End diff -- SOP? ---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/267#discussion_r163082381 --- Diff: extras/indexing/src/test/java/org/apache/rya/api/client/accumulo/AccumuloSetRyaStreamsConfigurationIT.java --- @@ -0,0 +1,81 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.api.client.accumulo; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + +import org.apache.rya.accumulo.AccumuloITBase; +import org.apache.rya.api.client.Install; +import org.apache.rya.api.client.Install.InstallConfiguration; +import org.apache.rya.api.client.InstanceDoesNotExistException; +import org.apache.rya.api.client.RyaClient; +import org.apache.rya.api.instance.RyaDetails.RyaStreamsDetails; +import org.junit.Test; + +/** + * Integration tests the methods of {@link AccumuloSetRyaStreamsConfiguration}. + */ +public class AccumuloSetRyaStreamsConfigurationIT extends AccumuloITBase { + +@Test(expected = InstanceDoesNotExistException.class) +public void instanceDoesNotExist() throws Exception { +final String ryaInstance = getRyaInstanceName(); +final AccumuloConnectionDetails connectionDetails = new AccumuloConnectionDetails( +getUsername(), +getPassword().toCharArray(), +getInstanceName(), +getZookeepers()); +final RyaClient ryaClient = AccumuloRyaClientFactory.build(connectionDetails, getConnector()); + +// Skip the install step to create error causing situation. +final RyaStreamsDetails details = new RyaStreamsDetails("localhost", 6); + ryaClient.getSetRyaStreamsConfiguration().setRyaStreamsConfiguration(ryaInstance, details); +} + +@Test +public void updatesRyaDetails() throws Exception { +final String ryaInstance = getRyaInstanceName(); +final AccumuloConnectionDetails connectionDetails = new AccumuloConnectionDetails( +getUsername(), +getPassword().toCharArray(), +getInstanceName(), +getZookeepers()); +final RyaClient ryaClient = AccumuloRyaClientFactory.build(connectionDetails, getConnector()); + +// Install an instance of Rya. +final Install installRya = ryaClient.getInstall(); +final InstallConfiguration installConf = InstallConfiguration.builder() +.build(); +installRya.install(ryaInstance, installConf); + +// Fetch its details and show they do not have any RyaStreamsDetails. +com.google.common.base.Optional streamsDetails = --- End diff -- we should really clean up the Optional stuff one day ---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/267#discussion_r163085139 --- Diff: extras/rya.streams/kafka/src/main/java/org/apache/rya/streams/kafka/KafkaRyaStreamsClientFactory.java --- @@ -0,0 +1,169 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.streams.kafka; + +import static java.util.Objects.requireNonNull; + +import java.util.Properties; +import java.util.UUID; + +import org.apache.kafka.clients.CommonClientConfigs; +import org.apache.kafka.clients.consumer.Consumer; +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.Producer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.common.serialization.Deserializer; +import org.apache.kafka.common.serialization.Serializer; +import org.apache.kafka.common.serialization.StringDeserializer; +import org.apache.kafka.common.serialization.StringSerializer; +import org.apache.rya.streams.api.RyaStreamsClient; +import org.apache.rya.streams.api.interactor.defaults.DefaultAddQuery; +import org.apache.rya.streams.api.interactor.defaults.DefaultDeleteQuery; +import org.apache.rya.streams.api.interactor.defaults.DefaultGetQuery; +import org.apache.rya.streams.api.interactor.defaults.DefaultListQueries; +import org.apache.rya.streams.api.interactor.defaults.DefaultStartQuery; +import org.apache.rya.streams.api.interactor.defaults.DefaultStopQuery; +import org.apache.rya.streams.api.queries.InMemoryQueryRepository; +import org.apache.rya.streams.api.queries.QueryChange; +import org.apache.rya.streams.api.queries.QueryChangeLog; +import org.apache.rya.streams.api.queries.QueryRepository; +import org.apache.rya.streams.kafka.interactor.KafkaGetQueryResultStream; +import org.apache.rya.streams.kafka.queries.KafkaQueryChangeLog; +import org.apache.rya.streams.kafka.serialization.VisibilityBindingSetDeserializer; +import org.apache.rya.streams.kafka.serialization.VisibilityStatementDeserializer; +import org.apache.rya.streams.kafka.serialization.queries.QueryChangeDeserializer; +import org.apache.rya.streams.kafka.serialization.queries.QueryChangeSerializer; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * Constructs instances of {@link RyaStreamsClient} that are connected to a Kafka cluster. + */ +@DefaultAnnotation(NonNull.class) +public final class KafkaRyaStreamsClientFactory { +private static final Logger log = LoggerFactory.getLogger(KafkaRyaStreamsClientFactory.class); + +/** + * Initialize a {@link RyaStreamsClient} that will interact with an instance of Rya Streams + * that is backed by Kafka. + * + * @param ryaInstance - The name of the Rya Instance the client is connected to. (not null) + * @param kafkaHostname - The hostname of the Kafka Broker. + * @param kafkaPort - The port of the Kafka Broker. + * @return The initialized commands. + */ +public static RyaStreamsClient make( +final String ryaInstance, +final String kafkaHostname, +final int kafkaPort) { +requireNonNull(kafkaHostname); + +// Setup Query Repository used by the Kafka Rya Streams subsystem. +final Producer queryProducer = +makeProducer(kafkaHostname, kafkaPort, StringSerializer.class, QueryChangeSerializer.class); +final ConsumerqueryConsumer = +fromStartConsumer(kafkaHostname, kafkaPort, StringDeserializer.class, QueryChangeDeserializer.class); +final S
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/267#discussion_r163081707 --- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloSetRyaStreamsConfiguration.java --- @@ -0,0 +1,59 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.api.client.accumulo; + +import static java.util.Objects.requireNonNull; + +import org.apache.accumulo.core.client.Connector; +import org.apache.rya.accumulo.instance.AccumuloRyaInstanceDetailsRepository; +import org.apache.rya.api.client.InstanceExists; +import org.apache.rya.api.client.SetRyaStreamsConfiguration; +import org.apache.rya.api.client.SetRyaStreamsConfigurationBase; +import org.apache.rya.api.instance.RyaDetailsRepository; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * A Accumulo implementation of {@link SetRyaStreamsConfiguration}. --- End diff -- nit: An* ---
[GitHub] incubator-rya pull request #267: RYA-440 Added commands to Rya Shell used to...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/267#discussion_r163082048 --- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoSetRyaStreamsConfiguration.java --- @@ -0,0 +1,60 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.api.client.mongo; + +import static java.util.Objects.requireNonNull; + +import org.apache.rya.api.client.InstanceExists; +import org.apache.rya.api.client.SetRyaStreamsConfiguration; +import org.apache.rya.api.client.SetRyaStreamsConfigurationBase; +import org.apache.rya.api.instance.RyaDetailsRepository; +import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository; + +import com.mongodb.MongoClient; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * A MongoDB implementation of {@link SetRyaStreamsConfiguration}. + */ +@DefaultAnnotation(NonNull.class) +public class MongoSetRyaStreamsConfiguration extends SetRyaStreamsConfigurationBase { + +private final MongoClient client; + +/** + * Constructs an instance of {@link MongoSetRyaStreamsConfiguration}. + * + * @param instanceExists - The interactor used to verify Rya instances exist. (not null) + * @param client - The MongoDB client used to connect to the Rya storage. (not null) + */ +public MongoSetRyaStreamsConfiguration( +final InstanceExists instanceExists, +final MongoClient client) { +super(instanceExists); +this.client = requireNonNull(client); +} + +@Override +protected RyaDetailsRepository getRyaDetailsRepo(final String ryaInstance) { +requireNonNull(ryaInstance); +return new MongoRyaInstanceDetailsRepository(client, ryaInstance); --- End diff -- you could one line this? ---
[GitHub] incubator-rya pull request #265: RYA-442 Implementing the Start and Stop Que...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/265#discussion_r162113248 --- Diff: extras/rya.streams/api/src/main/java/org/apache/rya/streams/api/interactor/defaults/DefaultStartQuery.java --- @@ -0,0 +1,54 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.streams.api.interactor.defaults; + +import static java.util.Objects.requireNonNull; + +import java.util.UUID; + +import org.apache.rya.streams.api.exception.RyaStreamsException; +import org.apache.rya.streams.api.interactor.StartQuery; +import org.apache.rya.streams.api.queries.QueryRepository; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * TODO --- End diff -- doc ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r161103080 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,446 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [pcj_ID]_METADATA, + * sparql: [sparql query to match results], + * varOrders: [varOrder1, VarOrder2, ..., VarOrdern] + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjId: [pcj_ID], + * visibilities: [visibilities] + * [binding_var1]: { + * uri: [type_uri], + * value: [value] + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: [value] + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_METADATA_ID = "_id"; +public static final String VAR_ORDER_FIELD = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "rdfType"; +private static final String VISIBILITIES_FIELD = "visibilities"; +private static final String PCJ_ID = "pcjId"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName -
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160834195 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,466 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [pcj_name]_METADATA, + * sparql: [sparql query to match results], + * varOrders: [varOrder1, VarOrder2, ..., VarOrdern] + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjId: [pcj_name], + * visibilities: [visibilities] + * [binding_var1]: { + * uri: [type_uri], + * value: [value] + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: [value] + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_FIELD = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "rdfType"; +private static final String VISIBILITIES_FIELD = "visibilities"; +private static final String PCJ_NAME = "pcjId"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName -
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160834100 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,466 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [pcj_name]_METADATA, + * sparql: [sparql query to match results], + * varOrders: [varOrder1, VarOrder2, ..., VarOrdern] + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjId: [pcj_name], + * visibilities: [visibilities] + * [binding_var1]: { + * uri: [type_uri], + * value: [value] + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: [value] + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_FIELD = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "rdfType"; +private static final String VISIBILITIES_FIELD = "visibilities"; +private static final String PCJ_NAME = "pcjId"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName -
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160834091 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,466 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [pcj_name]_METADATA, + * sparql: [sparql query to match results], + * varOrders: [varOrder1, VarOrder2, ..., VarOrdern] + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjId: [pcj_name], + * visibilities: [visibilities] + * [binding_var1]: { + * uri: [type_uri], + * value: [value] + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: [value] + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_FIELD = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "rdfType"; +private static final String VISIBILITIES_FIELD = "visibilities"; +private static final String PCJ_NAME = "pcjId"; --- End diff -- done ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160833809 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,466 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [pcj_name]_METADATA, + * sparql: [sparql query to match results], + * varOrders: [varOrder1, VarOrder2, ..., VarOrdern] + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjId: [pcj_name], --- End diff -- done ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160833781 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,466 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [pcj_name]_METADATA, --- End diff -- done ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160813417 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,451 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [pcj_name]_METADATA, + * sparql: [sparql query to match results], + * varOrders: [varOrder1, VarOrder2, ..., VarOrdern] + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [pcj_name], + * visibilities: [visibilities] + * [binding_var1]: { + * uri: [type_uri], + * value: [value] + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: [value] + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_FIELD = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "rdfType"; +private static final String VISIBILITIES_FIELD = "visibilities"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName -
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160813316 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjStorage.java --- @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; + +import org.apache.rya.api.instance.RyaDetails; +import org.apache.rya.api.instance.RyaDetails.PCJIndexDetails; +import org.apache.rya.api.instance.RyaDetails.PCJIndexDetails.PCJDetails; +import org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException; +import org.apache.rya.api.instance.RyaDetailsUpdater; +import org.apache.rya.api.instance.RyaDetailsUpdater.RyaDetailsMutator.CouldNotApplyMutationException; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PCJIdFactory; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage; +import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository; +import org.openrdf.query.BindingSet; + +import com.mongodb.MongoClient; + +import edu.umd.cs.findbugs.annotations.DefaultAnnotation; +import edu.umd.cs.findbugs.annotations.NonNull; + +/** + * A mongo backed implementation of {@link PrecomputedJoinStorage}. + */ +@DefaultAnnotation(NonNull.class) +public class MongoPcjStorage implements PrecomputedJoinStorage { +public static final String PCJ_COLLECTION_NAME = "pcjs"; +// Used to update the instance's metadata. +private final MongoRyaInstanceDetailsRepository ryaDetailsRepo; + +private final String ryaInstanceName; + +// Factories that are used to create new PCJs. +private final PCJIdFactory pcjIdFactory = new PCJIdFactory(); + +private final MongoPcjDocuments pcjDocs; + +/** + * Constructs an instance of {@link MongoPcjStorage}. + * + * @param client - The {@link MongoClient} that will be used to connect to Mongodb. (not null) + * @param ryaInstanceName - The name of the RYA instance that will be accessed. (not null) + */ +public MongoPcjStorage(final MongoClient client, final String ryaInstanceName) { +requireNonNull(client); +this.ryaInstanceName = requireNonNull(ryaInstanceName); +pcjDocs = new MongoPcjDocuments(client, ryaInstanceName); +ryaDetailsRepo = new MongoRyaInstanceDetailsRepository(client, ryaInstanceName); +} + +@Override +public String createPcj(final String sparql) throws PCJStorageException { +requireNonNull(sparql); + +// Update the Rya Details for this instance to include the new PCJ +// table. +final String pcjId = pcjIdFactory.nextId(); + +try { +new RyaDetailsUpdater(ryaDetailsRepo).update(originalDetails -> { +// Create the new PCJ's details. +final PCJDetails.Builder newPcjDetails = PCJDetails.builder().setId(pcjId); + +// Add them to the instance's details. +final RyaDetails.Builder mutated = RyaDetails.builder(originalDetails); +mutated.getPCJIndexDetails().addPCJDetails(newPcjDetails); +return mutated.build(); +}); +} catch (final RyaDetailsRepositoryException | CouldNotApplyMutationException e) { +throw new PCJStorageException(String.format("Could not create a new PCJ for Rya instance '%s' " ++ "because of a problem while updatin
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160813231 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,451 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [pcj_name]_METADATA, + * sparql: [sparql query to match results], + * varOrders: [varOrder1, VarOrder2, ..., VarOrdern] + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [pcj_name], + * visibilities: [visibilities] + * [binding_var1]: { + * uri: [type_uri], + * value: [value] + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: [value] + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; --- End diff -- you store the PCJ ID plus metadata ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160813158 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,451 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.model.VisibilityBindingSet; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.api.utils.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [pcj_name]_METADATA, + * sparql: [sparql query to match results], + * varOrders: [varOrder1, VarOrder2, ..., VarOrdern] + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - --- End diff -- it lets mongo handle that ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160813037 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/pcj/MongoPcjIndexSetProvider.java --- @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.mongodb.pcj; + +import java.util.List; +import java.util.Map; + +import org.apache.rya.api.instance.RyaDetailsRepository; +import org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException; +import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet; +import org.apache.rya.indexing.pcj.matching.provider.AbstractPcjIndexSetProvider; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjDocuments; +import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage; +import org.apache.rya.mongodb.StatefulMongoDBRdfConfiguration; +import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository; +import org.openrdf.query.MalformedQueryException; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import com.mongodb.MongoClient; + +/** + * Implementation of {@link AbstractPcjIndexSetProvider} for MongoDB. + */ +public class MongoPcjIndexSetProvider extends AbstractPcjIndexSetProvider { +/** + * Creates a new {@link MongoPcjIndexSetProvider}. + * @param conf - The configuration for this provider. (not null) + */ +public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration conf) { +super(conf); +} + +/** + * Creates a new {@link MongoPcjIndexSetProvider}. + * @param conf - The configuration for this provider. + * @param indices - The predefined indicies on this provider. + * @param client - The {@link MongoClient} used to connect to mongo. + */ +public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration conf, final List indices) { +super(conf, indices); +} + +@Override +protected List getIndices() throws PcjIndexSetException { +try { +//TODO use the PCJ collection. --- End diff -- no, removed it ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160811345 --- Diff: extras/shell/src/main/java/org/apache/rya/shell/util/RyaDetailsFormatter.java --- @@ -72,30 +72,28 @@ public String format(StorageType storageType, final RyaDetails details) { report.append(" Temporal Index:\n"); report.append("Enabled: ").append( details.getTemporalIndexDetails().isEnabled() ).append("\n"); -if(storageType == StorageType.ACCUMULO) { -report.append(" PCJ Index:\n"); -final PCJIndexDetails pcjDetails = details.getPCJIndexDetails(); -report.append("Enabled: ").append( pcjDetails.isEnabled() ).append("\n"); -if(pcjDetails.isEnabled()) { -if(pcjDetails.getFluoDetails().isPresent()) { -final String fluoAppName = pcjDetails.getFluoDetails().get().getUpdateAppName(); -report.append("Fluo App Name: ").append(fluoAppName).append("\n"); -} +report.append(" PCJ Index:\n"); --- End diff -- that include Join Selectivity? ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160810833 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160810774 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/254#discussion_r160808196 --- Diff: dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryNode.java --- @@ -0,0 +1,882 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.mongodb.aggregation; + +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.CONTEXT; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.DOCUMENT_VISIBILITY; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_HASH; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_TYPE; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE_HASH; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.STATEMENT_METADATA; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT_HASH; +import static org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.TIMESTAMP; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.NavigableSet; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.ConcurrentSkipListSet; +import java.util.function.Function; + +import org.apache.rya.api.domain.RyaStatement; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.domain.RyaURI; +import org.apache.rya.api.domain.StatementMetadata; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.mongodb.MongoDbRdfConstants; +import org.apache.rya.mongodb.dao.MongoDBStorageStrategy; +import org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy; +import org.apache.rya.mongodb.document.operators.query.ConditionalOperators; +import org.apache.rya.mongodb.document.visibility.DocumentVisibilityAdapter; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.Literal; +import org.openrdf.model.Resource; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.vocabulary.XMLSchema; +import org.openrdf.query.BindingSet; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.algebra.Compare; +import org.openrdf.query.algebra.ExtensionElem; +import org.openrdf.query.algebra.ProjectionElem; +import org.openrdf.query.algebra.ProjectionElemList; +import org.openrdf.query.algebra.StatementPattern; +import org.openrdf.query.algebra.ValueConstant; +import org.openrdf.query.algebra.ValueExpr; +import org.openrdf.query.algebra.Var; +import org.openrdf.query.algebra.evaluation.impl.ExternalSet; + +import com.google.common.base.Preconditions; +import com.google.common.collect.BiMap; +import com.google.common.collect.HashBiMap; +import com.mongodb.BasicDBObject; +import com.mongodb.DBObject; +import com.mongodb.client.MongoCollection; +import com.mongodb.client.model.Aggregates; +import com.mongodb.client.model.BsonField; +import com.mongodb.client.model.Filters; +import com.mongodb.client.model.Projections; + +import info.aduna.iteration.CloseableIteration; + +/** + * Represents a portion of a query tree as MongoDB aggregation pipeline. Should + * be built bottom-up: start with a statement pattern implemented as a $match + * step, then add steps to the pipeline to handle higher levels of the query + * tree. Methods are provided to add certain supported query operations to the + * end of the internal pip
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160762129 --- Diff: extras/shell/src/main/java/org/apache/rya/shell/util/InstallPrompt.java --- @@ -91,14 +91,14 @@ public InstallConfiguration promptInstallConfiguration(final String instanceName checkState(storageType.isPresent(), "The shell must be connected to a storage to use the install prompt."); switch(sharedShellState.getShellState().getStorageType().get()) { -case ACCUMULO: -return promptAccumuloConfig(instanceName); +case ACCUMULO: +return promptAccumuloConfig(instanceName); -case MONGO: -return promptMongoConfig(instanceName); +case MONGO: +return promptMongoConfig(instanceName); -default: -throw new IllegalStateException("Unsupported storage type: " + storageType.get()); +default: +throw new IllegalStateException("Unsupported storage type: " + storageType.get()); --- End diff -- done ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160762007 --- Diff: extras/shell/src/main/java/org/apache/rya/shell/RyaAdminCommands.java --- @@ -320,11 +343,14 @@ public String installWithMongoParameters( final boolean enableFreeTextIndex, // TODO RYA-215 -//@CliOption(key = {"enableGeospatialIndex"}, mandatory = false, help = "Use Geospatial Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true") -//final boolean enableGeospatialIndex, +//@CliOption(key = {"enableGeospatialIndex"}, mandatory = false, help = "Use Geospatial Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true") +//final boolean enableGeospatialIndex, @CliOption(key = {"enableTemporalIndex"}, mandatory = false, help = "Use Temporal Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true") -final boolean enableTemporalIndex) { +final boolean enableTemporalIndex, --- End diff -- I went ahead and just reverted it, was faster to just take care of it later ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160761918 --- Diff: extras/shell/src/main/java/org/apache/rya/shell/RyaAdminCommands.java --- @@ -104,11 +104,11 @@ public RyaAdminCommands( INSTALL_CMD}) public boolean areStorageCommandsAvailable() { switch(state.getShellState().getConnectionState()) { -case CONNECTED_TO_STORAGE: -case CONNECTED_TO_INSTANCE: -return true; -default: -return false; +case CONNECTED_TO_STORAGE: --- End diff -- done ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160761790 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160761434 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160761412 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160761257 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160761195 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160760999 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160760932 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160760282 --- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoCreatePCJ.java --- @@ -0,0 +1,85 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.api.client.mongo; + +import static java.util.Objects.requireNonNull; + +import java.util.Set; + +import org.apache.rya.api.client.CreatePCJ; +import org.apache.rya.api.client.InstanceDoesNotExistException; +import org.apache.rya.api.client.InstanceExists; +import org.apache.rya.api.client.RyaClientException; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.collect.Sets; +import com.mongodb.MongoClient; + +/** + * A Mongo implementation of {@link CreatePCJ}. + */ +public class MongoCreatePCJ implements CreatePCJ { +private static final Logger log = LoggerFactory.getLogger(MongoCreatePCJ.class); + +private final MongoConnectionDetails connectionDetails; --- End diff -- removed ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160760303 --- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoDeletePCJ.java --- @@ -0,0 +1,72 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.api.client.mongo; + +import static java.util.Objects.requireNonNull; + +import org.apache.rya.api.client.DeletePCJ; +import org.apache.rya.api.client.InstanceDoesNotExistException; +import org.apache.rya.api.client.InstanceExists; +import org.apache.rya.api.client.RyaClientException; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage; + +import com.mongodb.MongoClient; + +/** + * A Mongo implementation of {@link DeletePCJ}. + */ +public class MongoDeletePCJ implements DeletePCJ { +private final MongoConnectionDetails connectionDetails; --- End diff -- removed ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160755378 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160755315 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; --- End diff -- The way VisibilityBindingSets were implemented, I think they use accumulo's Authorizations api ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160753606 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160570620 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya issue #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on the issue: https://github.com/apache/incubator-rya/pull/172 what is RyaDetailsFormatter. this is more of a problem of just stuff floating around that doesn't break when functionality gets added ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160570534 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; + +private final MongoCollection pcjCollection; +private static final PcjVarOrderFactory pcjVarOrderFactory = new ShiftVarOrderFactory(); + +/** + * Creates a new {@link MongoPcjDocuments}. + * @param client - The {@link MongoClient} to use to connect to mongo. + * @param ryaInstanceName - The rya instance to conn
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160570420 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; +private static final String AUTHS_FIELD = "auths"; +private static final String PCJ_NAME = "pcjName"; --- End diff -- no, its for the pcj name. the ID is calculated differently ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160570327 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; + +// pcj results fields +private static final String BINDING_VALUE = "value"; +private static final String BINDING_TYPE = "uri"; --- End diff -- done ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160570226 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value + * } + * . + * . + * . + * [binding_varn]: { + * uri: [type_uri], + * value: value + * } + * } + * + * + */ +public class MongoPcjDocuments { +public static final String PCJ_COLLECTION_NAME = "pcjs"; + +// metadata fields +public static final String CARDINALITY_FIELD = "cardinality"; +public static final String SPARQL_FIELD = "sparql"; +public static final String PCJ_ID = "_id"; +public static final String VAR_ORDER_ID = "varOrders"; --- End diff -- done ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160570170 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value --- End diff -- nvm. didn't look right ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160570068 --- Diff: extras/rya.indexing.pcj/src/main/java/org/apache/rya/indexing/pcj/storage/mongo/MongoPcjDocuments.java --- @@ -0,0 +1,438 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.storage.mongo; + +import static com.google.common.base.Preconditions.checkNotNull; +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Set; + +import org.apache.accumulo.core.security.Authorizations; +import org.apache.rya.api.domain.RyaType; +import org.apache.rya.api.resolver.RdfToRyaConversions; +import org.apache.rya.api.resolver.RyaToRdfConversions; +import org.apache.rya.indexing.pcj.storage.PcjMetadata; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.CloseableIterator; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.ShiftVarOrderFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.VariableOrder; +import org.apache.rya.indexing.pcj.storage.accumulo.VisibilityBindingSet; +import org.bson.Document; +import org.bson.conversions.Bson; +import org.openrdf.model.URI; +import org.openrdf.model.Value; +import org.openrdf.model.impl.URIImpl; +import org.openrdf.query.BindingSet; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.query.QueryLanguage; +import org.openrdf.query.TupleQuery; +import org.openrdf.query.TupleQueryResult; +import org.openrdf.query.impl.MapBindingSet; +import org.openrdf.repository.RepositoryConnection; +import org.openrdf.repository.RepositoryException; + +import com.mongodb.MongoClient; +import com.mongodb.client.FindIterable; +import com.mongodb.client.MongoCollection; +import com.mongodb.util.JSON; + +/** + * Creates and modifies PCJs in MongoDB. PCJ's are stored as follows: + * + * + * + * - PCJ Metadata Doc - + * { + * _id: [table_name]_METADATA, + * sparql: [sparql query to match results], + * cardinality: [number of results] + * } + * + * - PCJ Results Doc - + * { + * pcjName: [table_name], + * auths: [auths] + * [binding_var1]: { + * uri: [type_uri], + * value: value --- End diff -- well square brackets denotes list/set so really the docs should not have them ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160569220 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/pcj/matching/provider/AccumuloIndexSetProvider.java --- @@ -0,0 +1,139 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.matching.provider; + +import static java.util.Objects.requireNonNull; + +import java.util.List; +import java.util.Map; + +import org.apache.accumulo.core.client.AccumuloException; +import org.apache.accumulo.core.client.AccumuloSecurityException; +import org.apache.accumulo.core.client.Connector; +import org.apache.accumulo.core.client.TableNotFoundException; +import org.apache.hadoop.conf.Configuration; +import org.apache.log4j.Logger; +import org.apache.rya.accumulo.instance.AccumuloRyaInstanceDetailsRepository; +import org.apache.rya.api.RdfCloudTripleStoreConfiguration; +import org.apache.rya.api.instance.RyaDetailsRepository; +import org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException; +import org.apache.rya.indexing.accumulo.ConfigUtils; +import org.apache.rya.indexing.external.tupleSet.AccumuloIndexSet; +import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.accumulo.AccumuloPcjStorage; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjTableNameFactory; +import org.apache.rya.indexing.pcj.storage.accumulo.PcjTables; +import org.openrdf.query.MalformedQueryException; +import org.openrdf.query.QueryEvaluationException; +import org.openrdf.sail.SailException; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; + +/** + * Implementation of {@link AbstractPcjIndexSetProvider} for Accumulo. + * This provider uses either user specified Accumulo configuration information or user a specified + * List of ExternalTupleSets to populate an internal cache of ExternalTupleSets. If Accumulo configuration + * is provided, the provider connects to an instance of RyaDetails and populates the cache with + * PCJs registered in RyaDetails. + */ +public class AccumuloIndexSetProvider extends AbstractPcjIndexSetProvider { +private static final Logger log = Logger.getLogger(AccumuloIndexSetProvider.class); + +public AccumuloIndexSetProvider(final Configuration conf) { +super(conf); +} + +public AccumuloIndexSetProvider(final Configuration conf, final List indices) { +super(conf, indices); +} + +@Override +protected List getIndices() throws PcjIndexSetException { +requireNonNull(conf); +try { +final String tablePrefix = requireNonNull(conf.get(RdfCloudTripleStoreConfiguration.CONF_TBL_PREFIX)); +final Connector conn = requireNonNull(ConfigUtils.getConnector(conf)); +List tables = null; + +if (conf instanceof RdfCloudTripleStoreConfiguration) { +tables = ((RdfCloudTripleStoreConfiguration) conf).getPcjTables(); +} +// this maps associates pcj table name with pcj sparql query +final Map<String, String> indexTables = Maps.newLinkedHashMap(); + +try(final PrecomputedJoinStorage storage = new AccumuloPcjStorage(conn, tablePrefix)) { +final PcjTableNameFactory pcjFactory = new PcjTableNameFactory(); + +final boolean tablesProvided = tables != null && !tables.isEmpty(); + +if (tablesProvided) { --- End diff -- I'm not touching accumulo code. this is just a repackage ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160569137 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/pcj/MongoPcjIndexSetProvider.java --- @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.mongodb.pcj; + +import static java.util.Objects.requireNonNull; + +import java.util.List; +import java.util.Map; + +import org.apache.rya.api.instance.RyaDetailsRepository; +import org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException; +import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet; +import org.apache.rya.indexing.pcj.matching.provider.AbstractPcjIndexSetProvider; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjDocuments; +import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage; +import org.apache.rya.mongodb.StatefulMongoDBRdfConfiguration; +import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository; +import org.openrdf.query.MalformedQueryException; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import com.mongodb.MongoClient; + +/** + * Implementation of {@link AbstractPcjIndexSetProvider} for MongoDB. + */ +public class MongoPcjIndexSetProvider extends AbstractPcjIndexSetProvider { +/** + * Creates a new {@link MongoPcjIndexSetProvider}. + * @param conf - The configuration for this provider. (not null) + */ +public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration conf) { +super(conf); +} + +/** + * Creates a new {@link MongoPcjIndexSetProvider}. + * @param conf - The configuration for this provider. + * @param indices - The predefined indicies on this provider. + * @param client - The {@link MongoClient} used to connect to mongo. + */ +public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration conf, final List indices) { +super(conf, indices); +} + +@Override +protected List getIndices() throws PcjIndexSetException { +requireNonNull(conf); +try { +final StatefulMongoDBRdfConfiguration mongoConf = (StatefulMongoDBRdfConfiguration) conf; +final MongoClient client = mongoConf.getMongoClient(); +final MongoPcjDocuments pcjDocs = new MongoPcjDocuments(client, mongoConf.getRyaInstanceName()); +List tables = null; --- End diff -- there should really only be the PCJ collection. I may take a look at just cleaning this up to use the PCJ collection ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160562616 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/mongodb/pcj/MongoPcjIndexSetProvider.java --- @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.mongodb.pcj; + +import static java.util.Objects.requireNonNull; + +import java.util.List; +import java.util.Map; + +import org.apache.rya.api.instance.RyaDetailsRepository; +import org.apache.rya.api.instance.RyaDetailsRepository.RyaDetailsRepositoryException; +import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet; +import org.apache.rya.indexing.pcj.matching.provider.AbstractPcjIndexSetProvider; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjDocuments; +import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage; +import org.apache.rya.mongodb.StatefulMongoDBRdfConfiguration; +import org.apache.rya.mongodb.instance.MongoRyaInstanceDetailsRepository; +import org.openrdf.query.MalformedQueryException; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import com.mongodb.MongoClient; + +/** + * Implementation of {@link AbstractPcjIndexSetProvider} for MongoDB. + */ +public class MongoPcjIndexSetProvider extends AbstractPcjIndexSetProvider { +/** + * Creates a new {@link MongoPcjIndexSetProvider}. + * @param conf - The configuration for this provider. (not null) + */ +public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration conf) { +super(conf); +} + +/** + * Creates a new {@link MongoPcjIndexSetProvider}. + * @param conf - The configuration for this provider. + * @param indices - The predefined indicies on this provider. + * @param client - The {@link MongoClient} used to connect to mongo. + */ +public MongoPcjIndexSetProvider(final StatefulMongoDBRdfConfiguration conf, final List indices) { +super(conf, indices); +} + +@Override +protected List getIndices() throws PcjIndexSetException { +requireNonNull(conf); --- End diff -- probably should ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160535354 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/pcj/matching/provider/AbstractPcjIndexSetProvider.java --- @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.matching.provider; + +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; + +import org.apache.hadoop.conf.Configuration; +import org.apache.log4j.Logger; +import org.apache.rya.api.instance.RyaDetails; +import org.apache.rya.indexing.IndexPlanValidator.IndexedExecutionPlanGenerator; +import org.apache.rya.indexing.IndexPlanValidator.ValidIndexCombinationGenerator; +import org.apache.rya.indexing.external.matching.ExternalSetProvider; +import org.apache.rya.indexing.external.matching.QuerySegment; +import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet; +import org.apache.rya.indexing.pcj.matching.PCJOptimizerUtilities; +import org.apache.rya.indexing.pcj.matching.PCJToSegmentConverter; +import org.openrdf.query.algebra.TupleExpr; + +import com.google.common.annotations.VisibleForTesting; + +/** + * Abstraction of {@link ExternalSetProvider} that provides {@link ExternalTupleSet}s. + * Implementations of this use either user specified configuration information or user a specified + * List of ExternalTupleSets to populate an internal cache of ExternalTupleSets. If a configuration + * is provided, the provider connects to an instance of RyaDetails and populates the cache with + * PCJs registered in RyaDetails. + */ +public abstract class AbstractPcjIndexSetProvider implements ExternalSetProvider { +protected static final Logger log = Logger.getLogger(AbstractPcjIndexSetProvider.class); +protected static final PCJToSegmentConverter converter = new PCJToSegmentConverter(); +protected List indexCache; +protected final Configuration conf; +protected boolean init = false; + +/** + * Creates a new {@link AbstractPcjIndexSetProvider} based on configuration only. + * @param conf - The {@link Configuration} used to connect to {@link RyaDetails}. + */ +public AbstractPcjIndexSetProvider(final Configuration conf) { +requireNonNull(conf); +this.conf = conf; +} + +/** + * Creates a new {@link AbstractPcjIndexSetProvider} based user provided {@link ExternalTupleSet}s. + * @param conf - The {@link Configuration} used to connect to {@link RyaDetails}. + * @param indices - The {@link ExternalTupleSet}s to populate the internal cache. + */ +public AbstractPcjIndexSetProvider(final Configuration conf, final List indices) { +requireNonNull(conf); +this.conf = conf; +indexCache = indices; +init = true; +} + + +/** + * + * @param indices + */ +@VisibleForTesting +public void setIndices(final List indices) { --- End diff -- a lot of the tests would need to be repackaged to allow that ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160534831 --- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/pcj/matching/provider/AbstractPcjIndexSetProvider.java --- @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.indexing.pcj.matching.provider; + +import static java.util.Objects.requireNonNull; + +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; + +import org.apache.hadoop.conf.Configuration; +import org.apache.log4j.Logger; +import org.apache.rya.api.instance.RyaDetails; +import org.apache.rya.indexing.IndexPlanValidator.IndexedExecutionPlanGenerator; +import org.apache.rya.indexing.IndexPlanValidator.ValidIndexCombinationGenerator; +import org.apache.rya.indexing.external.matching.ExternalSetProvider; +import org.apache.rya.indexing.external.matching.QuerySegment; +import org.apache.rya.indexing.external.tupleSet.ExternalTupleSet; +import org.apache.rya.indexing.pcj.matching.PCJOptimizerUtilities; +import org.apache.rya.indexing.pcj.matching.PCJToSegmentConverter; +import org.openrdf.query.algebra.TupleExpr; + +import com.google.common.annotations.VisibleForTesting; + +/** + * Abstraction of {@link ExternalSetProvider} that provides {@link ExternalTupleSet}s. + * Implementations of this use either user specified configuration information or user a specified + * List of ExternalTupleSets to populate an internal cache of ExternalTupleSets. If a configuration + * is provided, the provider connects to an instance of RyaDetails and populates the cache with + * PCJs registered in RyaDetails. + */ +public abstract class AbstractPcjIndexSetProvider implements ExternalSetProvider { +protected static final Logger log = Logger.getLogger(AbstractPcjIndexSetProvider.class); +protected static final PCJToSegmentConverter converter = new PCJToSegmentConverter(); +protected List indexCache; +protected final Configuration conf; +protected boolean init = false; + +/** + * Creates a new {@link AbstractPcjIndexSetProvider} based on configuration only. + * @param conf - The {@link Configuration} used to connect to {@link RyaDetails}. + */ +public AbstractPcjIndexSetProvider(final Configuration conf) { +requireNonNull(conf); --- End diff -- a lot of these look like I did them, its just a repackage ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160534291 --- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoInstall.java --- @@ -191,26 +192,22 @@ private static MongoDBRdfConfiguration makeRyaConfig( final MongoDBRdfConfiguration conf = connectionDetails.build(ryaDetails.getRyaInstanceName()); -// The Mongo implementation of Rya does not currently support PCJs. -if(ryaDetails.getPCJIndexDetails().isEnabled()) { -log.warn("The install configuration says to enable PCJ indexing, but Mongo RYA does not support that " + -"feature. Ignoring this configuration."); -} -conf.set(ConfigUtils.USE_PCJ, "false"); +conf.setBoolean(ConfigUtils.USE_PCJ, ryaDetails.getPCJIndexDetails().isEnabled()); // Mongo does not support entity indexing. if(ryaDetails.getEntityCentricIndexDetails().isEnabled()) { log.warn("The install configuration says to enable Entity Centric indexing, but Mongo RYA does not support " + "that feature. Ignoring this configuration."); } -conf.set(ConfigUtils.USE_ENTITY, "false"); +//TODO mongo now has an entity index, just needs CLI support. --- End diff -- discussed in person ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160534095 --- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoDeletePCJ.java --- @@ -0,0 +1,96 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.rya.api.client.mongo; + +import static java.util.Objects.requireNonNull; + +import java.util.Optional; + +import org.apache.rya.api.client.DeletePCJ; +import org.apache.rya.api.client.InstanceDoesNotExistException; +import org.apache.rya.api.client.InstanceExists; +import org.apache.rya.api.client.RyaClientException; +import org.apache.rya.indexing.pcj.storage.PrecomputedJoinStorage.PCJStorageException; +import org.apache.rya.indexing.pcj.storage.mongo.MongoPcjStorage; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.collect.Lists; +import com.mongodb.MongoClient; +import com.mongodb.MongoCredential; +import com.mongodb.ServerAddress; + +/** + * A Mongo implementation of {@link DeletePCJ}. + */ +public class MongoDeletePCJ implements DeletePCJ { +private static final Logger log = LoggerFactory.getLogger(MongoDeletePCJ.class); + +private final MongoConnectionDetails connectionDetails; +private final InstanceExists instanceExists; + +/** + * Constructs an instance of {@link MongoDeletePCJ}. + * + * @param connectionDetails - Details to connect to the server. (not null) + * @param instanceExists - The interactor used to check if a Rya instance exists. (not null) + */ +public MongoDeletePCJ( +final MongoConnectionDetails connectionDetails, +final MongoInstanceExists instanceExists) { +this.connectionDetails = requireNonNull(connectionDetails); +this.instanceExists = requireNonNull(instanceExists); +} + +@Override +public void deletePCJ(final String ryaInstanceName, final String pcjId) throws InstanceDoesNotExistException, RyaClientException { +requireNonNull(ryaInstanceName); +requireNonNull(pcjId); + +// Ensure the Rya Instance exists. +if (!instanceExists.exists(ryaInstanceName)) { +throw new InstanceDoesNotExistException(String.format("There is no Rya instance named '%s'.", ryaInstanceName)); +} + +final Optional username = connectionDetails.getUsername(); +final MongoCredential cred; +final MongoClient client; + +final ServerAddress addr = new ServerAddress(connectionDetails.getHostname(), connectionDetails.getPort()); +if(username.isPresent()) { +cred = MongoCredential.createCredential(username.get(), ryaInstanceName, connectionDetails.getPassword().get()); +client = new MongoClient(addr, Lists.newArrayList(cred)); +} else { +client = new MongoClient(addr); +} + + +//pcjStore doesn't need to be closed since it doesn't need to release any resources. +@SuppressWarnings("resource") +final MongoPcjStorage pcjStore = new MongoPcjStorage(client, ryaInstanceName); --- End diff -- done ---
[GitHub] incubator-rya pull request #172: RYA-303 Mongo PCJ Support
Github user isper3at commented on a diff in the pull request: https://github.com/apache/incubator-rya/pull/172#discussion_r160534139 --- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/mongo/MongoInstall.java --- @@ -147,22 +147,23 @@ private RyaDetails initializeRyaDetails( final InstallConfiguration installConfig) throws AlreadyInitializedException, RyaDetailsRepositoryException { final RyaDetailsRepository detailsRepo = new MongoRyaInstanceDetailsRepository(adminClient, instanceName); -// Build the PCJ Index details. [not supported in mongo] -final PCJIndexDetails.Builder pcjDetailsBuilder = PCJIndexDetails.builder().setEnabled(false); +if(installConfig.getFluoPcjAppName().isPresent()) { + log.warn("Mongo does not have fluo support, use ignoring the configured fluo application name: " + installConfig.getFluoPcjAppName().get()); +} + +// Build the PCJ Index details. +final PCJIndexDetails.Builder pcjDetailsBuilder = PCJIndexDetails.builder() +.setEnabled(installConfig.isPcjIndexEnabled()); final RyaDetails details = RyaDetails.builder() // General Metadata .setRyaInstanceName(instanceName).setRyaVersion(getVersion()) -// FIXME RYA-215 .setGeoIndexDetails(new GeoIndexDetails(installConfig.isGeoIndexEnabled())) - // Secondary Index Values +// FIXME RYA-215 .setGeoIndexDetails(new GeoIndexDetails(installConfig.isGeoIndexEnabled())) .setTemporalIndexDetails(new TemporalIndexDetails(installConfig.isTemporalIndexEnabled())) .setFreeTextDetails(new FreeTextIndexDetails(installConfig.isFreeTextIndexEnabled()))// - -// Entity centric indexing is not supported in Mongo DB. -.setEntityCentricIndexDetails(new EntityCentricIndexDetails(false)) - +.setEntityCentricIndexDetails(new EntityCentricIndexDetails(installConfig.isEntityCentrixIndexEnabled())) --- End diff -- discussed in person ---