[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r160811757
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryOptimizer.java
 ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.rya.mongodb.StatefulMongoDBRdfConfiguration;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.Dataset;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.evaluation.QueryOptimizer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MongoDB-specific query optimizer that replaces part or all of a SPARQL 
query
+ * tree with a MongoDB aggregation pipeline.
+ * 
+ * Transforms query trees using {@link SparqlToPipelineTransformVisitor}. 
If
+ * possible, this visitor will replace portions of the query tree, or the 
entire
+ * query, with an equivalent aggregation pipeline (contained in an
+ * {@link AggregationPipelineQueryNode}), thereby allowing query logic to 
be
+ * evaluated by the MongoDB server rather than by the client.
+ */
+public class AggregationPipelineQueryOptimizer implements QueryOptimizer, 
Configurable {
+private Configuration conf;
+private Logger logger = LoggerFactory.getLogger(getClass());
+
+@Override
+public void optimize(TupleExpr tupleExpr, Dataset dataset, BindingSet 
bindings) {
+if (conf instanceof StatefulMongoDBRdfConfiguration) {
+StatefulMongoDBRdfConfiguration mongoConf = 
(StatefulMongoDBRdfConfiguration) conf;
+SparqlToPipelineTransformVisitor pipelineVisitor = new 
SparqlToPipelineTransformVisitor(mongoConf);
+try {
+tupleExpr.visit(pipelineVisitor);
+} catch (Exception e) {
+logger.error("Error attempting to transform query using 
the aggregation pipeline", e);
+}
+}
+}
+
+@Override
+public void setConf(Configuration conf) {
+this.conf = conf;
--- End diff --

Agreed, fixed.


---

[GitHub] incubator-rya pull request #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/255#discussion_r160794365
  
--- Diff: 
extras/rya.forwardchain/src/main/java/org/apache/rya/forwardchain/strategy/RoundRobinStrategy.java
 ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.forwardchain.strategy;
+
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+
+import org.apache.log4j.Logger;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.StatementMetadata;
+import org.apache.rya.forwardchain.ForwardChainConstants;
+import org.apache.rya.forwardchain.ForwardChainException;
+import org.apache.rya.forwardchain.rule.Rule;
+import org.apache.rya.forwardchain.rule.Ruleset;
+import org.openrdf.model.vocabulary.XMLSchema;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * A simple {@link AbstractForwardChainStrategy} that iterates over every
+ * relevant rule once, and repeats until no rules are relevant.
+ * 
+ * Initially, all rules are considered relevant. Iteration 1 executes each 
rule
+ * once.
+ * 
+ * When a rule produces inferences, all rules that are marked as that 
rule's
+ * successors according to the {@link Ruleset} are triggered as potentially
+ * relevant for future execution. If a triggered rule is scheduled to be
+ * executed during the current iteration, nothing changes. If a triggered 
rule
+ * has already been executed once during the current iteration, or was not
+ * activated for the current iteration at all, it is flagged to be executed
+ * during the next iteration.
+ * 
+ * When an iteration concludes, a new iteration begins with the relevant 
set of
+ * rules having been determined during the previous iteration. If there 
are no
+ * such rules, forward chaining ends.
+ * 
+ * Within each iteration, rules are processed such that a rule which may 
trigger
+ * many other rules is given priority over a rule that may be triggered by 
many
+ * other rules.
+ * 
+ * The observation that one rule may trigger another is based on the
+ * relationships between triple patterns produced and consumed by the 
rules in
+ * general, not based on any triples that were actually generated. 
Therefore,
+ * there may be false positives but not false negatives: Rules triggered 
by the
+ * current rule may or may not produce more triples in response, but any 
rule
+ * that could produce triples in response will be triggered.
+ * 
+ * The procedure for executing the individual rules is governed by the
+ * {@link RuleExecutionStrategy}. This class uses the strategy's reported 
counts
+ * to determine whether or not a rule has produced inferences.
+ */
+public class RoundRobinStrategy extends AbstractForwardChainStrategy {
+private static final Logger logger = 
Logger.getLogger(RoundRobinStrategy.class);
+
+private final AbstractRuleExecutionStrategy ruleStrategy;
+private int iteration;
+private Ruleset ruleset;
+private Set activeNow;
+private Set activeNextIteration;
+private long inferencesThisIteration;
+private boolean initialized = false;
--- End diff --

Done.


---

[GitHub] incubator-rya pull request #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/255#discussion_r160794284
  
--- Diff: 
extras/rya.forwardchain/src/main/java/org/apache/rya/forwardchain/batch/ForwardChainSpinTool.java
 ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.forwardchain.batch;
+
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.forwardchain.ForwardChainException;
+import org.apache.rya.forwardchain.rule.Ruleset;
+import org.apache.rya.forwardchain.rule.SpinConstructRule;
+
+/**
+ * {@link Tool} to load SPIN Construct rules from a Rya data store, then 
apply
+ * those rules to the same store using forward-chaining inference
+ * (materialization), adding triples back to Rya until no more information 
can
+ * be derived.
+ */
+public class ForwardChainSpinTool extends AbstractForwardChainTool {
+private Ruleset ruleset;
+
+/**
+ * Constructor that takes in an {@link 
RdfCloudTripleStoreConfiguration}.
+ * @param conf Configuration object containing Rya connection 
information.
+ */
+public ForwardChainSpinTool(RdfCloudTripleStoreConfiguration conf) {
+setConf(conf);
+}
+
+/**
+ * Default constructor that does not take in a configuration object. 
Rya
+ * connection details should be provided via an
+ * RdfCloudTripleStoreConfiguration, either using
+ * {@link AbstractForwardChainTool#setConf} or a {@link ToolRunner}.
+ */
+public ForwardChainSpinTool() { }
+
+/**
+ * Load SPIN Construct rules from Rya.
+ * @return A set of construct query rules.
+ * @throws ForwardChainException if loading rules from Rya fails.
+ */
+@Override
+protected Ruleset getRuleset() throws ForwardChainException {
+if (ruleset == null) {
+ruleset = SpinConstructRule.loadSpinRules(getConf());
+}
+return ruleset;
+}
+
+public static void main(String[] args) throws Exception {
+long start = System.currentTimeMillis();
+ForwardChainSpinTool tool = new ForwardChainSpinTool();
+ToolRunner.run(tool, args);
+long end = System.currentTimeMillis();
+double seconds = (end - start) / 1000.0;
+long inferences = tool.getNumInferences();
+long rules = tool.getRuleset().getRules().size();
+System.out.println(String.format("ForwardChainSpinTool: %d rules, 
%d inferences, %.3f seconds",
+rules, inferences, seconds));
+//Properties props = new Properties();
--- End diff --

Done.


---

[GitHub] incubator-rya pull request #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/255#discussion_r160794337
  
--- Diff: 
extras/rya.forwardchain/src/test/java/org/apache/rya/forwardchain/batch/MongoSpinIT.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.forwardchain.batch;
+
+import java.io.BufferedReader;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.log4j.Level;
+import org.apache.log4j.Logger;
+import org.apache.rya.indexing.mongodb.MongoIndexingConfiguration;
+import 
org.apache.rya.indexing.mongodb.MongoIndexingConfiguration.MongoDBIndexingConfigBuilder;
+import org.apache.rya.mongodb.EmbeddedMongoFactory;
+import org.apache.rya.mongodb.MongoDBRdfConfiguration;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.impl.ListBindingSet;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.rio.RDFFormat;
+import org.openrdf.rio.Rio;
+
+import com.google.common.io.Resources;
+import com.mongodb.MongoClient;
+import com.mongodb.ServerAddress;
+
+public class MongoSpinIT {
+private static ValueFactory VF = ValueFactoryImpl.getInstance();
--- End diff --

Done.


---

[GitHub] incubator-rya pull request #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/255#discussion_r160793851
  
--- Diff: 
extras/rya.forwardchain/src/main/java/org/apache/rya/forwardchain/rule/Ruleset.java
 ---
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.forwardchain.rule;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.log4j.Logger;
+import org.openrdf.query.algebra.StatementPattern;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Represents a set of forward-chaining {@link Rule}s and their 
relationships.
+ */
+public class Ruleset {
+private final Set rules;
+private final Map<Rule, Set> successors;
+private final Map<Rule, Set> predecessors;
+
+private final Logger logger = Logger.getLogger(this.getClass());
+
+/**
+ * Constructor. Takes in a set of rules and determines their 
dependencies.
+ * @param rules The complete set of rules to process; should not be 
null.
+ */
+public Ruleset(Collection rules) {
+Preconditions.checkNotNull(rules);
+this.rules = new HashSet<>();
+for (Rule rule : rules) {
+if (rule != null) {
+this.rules.add(rule);
+}
+}
+successors = new ConcurrentHashMap<>();
+predecessors = new ConcurrentHashMap<>();
+// Build the dependency graph of all the rules, in both directions
+for (Rule rule : rules) {
+successors.put(rule, new HashSet<>());
+predecessors.put(rule, new HashSet<>());
+}
+for (Rule rule1 : rules) {
+for (Rule rule2 : rules) {
+if (canTrigger(rule1, rule2)) {
--- End diff --

Yep, that can happen. (Example from the tests is 
LUBM/suborganization-transitivity: `CONSTRUCT { ?this lubm:subOrganizationOf 
?parent } WHERE { ?this lubm:subOrganizationOf ?child . ?chlid 
lubm:subOrganizationOf ?parent }`). This means whenever such a rule produces 
results, we'll want to re-run it later to see if those results create further 
results. But we check for duplicates and make sure only to do this if the 
inferred triples are actually new, so it will terminate at some point. (If the 
rule could generate new entities, it could go on indefinitely -- that's why any 
rule that constructs bnodes is not supported.)


---

[GitHub] incubator-rya issue #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on the issue:

https://github.com/apache/incubator-rya/pull/255
  
Note that the RYA-417 work sits on top of RYA-416 -- the first two commits 
here are identical to PR https://github.com/apache/incubator-rya/pull/254 . 
That means most of the aggregation pipeline logic, so if there are changes that 
need to be made there it might be better to address them as part of that PR and 
then I can fold them into this one once resolved.


---

[GitHub] incubator-rya pull request #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/255#discussion_r160554313
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/SparqlToPipelineTransformVisitor.java
 ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import java.util.Arrays;
+
+import org.apache.rya.mongodb.StatefulMongoDBRdfConfiguration;
+import org.bson.Document;
+import org.openrdf.query.algebra.Distinct;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.Filter;
+import org.openrdf.query.algebra.Join;
+import org.openrdf.query.algebra.MultiProjection;
+import org.openrdf.query.algebra.Projection;
+import org.openrdf.query.algebra.Reduced;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.helpers.QueryModelVisitorBase;
+
+import com.google.common.base.Preconditions;
+import com.mongodb.MongoClient;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+
+/**
+ * Visitor that transforms a SPARQL query tree by replacing as much of the 
tree
+ * as possible with one or more {@code AggregationPipelineQueryNode}s.
+ * 
+ * Each {@link AggregationPipelineQueryNode} contains a MongoDB aggregation
+ * pipeline which is equivalent to the replaced portion of the original 
query.
+ * Evaluating this node executes the pipeline and converts the results into
+ * query solutions. If only part of the query was transformed, the 
remaining
+ * query logic (higher up in the query tree) can be applied to those
+ * intermediate solutions as normal.
+ * 
+ * In general, processes the tree in bottom-up order: A leaf node
+ * ({@link StatementPattern}) is replaced with a pipeline that matches the
+ * corresponding statements. Then, if the parent node's semantics are 
supported
+ * by the visitor, stages are appended to the pipeline and the subtree at 
the
+ * parent node is replaced with the extended pipeline. This continues up 
the
+ * tree until reaching a node that cannot be transformed, in which case 
that
+ * node's child is now a single {@code AggregationPipelineQueryNode} (a 
leaf
+ * node) instead of the previous subtree, or until the entire tree has been
+ * subsumed into a single pipeline node.
+ * 
+ * Nodes which are transformed into pipeline stages:
+ * 
+ * A {@code StatementPattern} node forms the beginning of each 
pipeline.
+ * Single-argument operations {@link Projection}, {@link 
MultiProjection},
+ * {@link Extension}, {@link Distinct}, and {@link Reduced} will be 
transformed
+ * into pipeline stages whenever the child {@link TupleExpr} represents a
+ * pipeline.
+ * A {@link Filter} operation will be appended to the pipeline when its
+ * child {@code TupleExpr} represents a pipeline and the filter condition 
is a
+ * type of {@link ValueExpr} understood by {@code 
AggregationPipelineQueryNode}.
+ * A {@link Join} operation will be appended to the pipeline when one 
child
+ * is a {@code StatementPattern} and the other is an
+ * {@code AggregationPipelineQueryNode}.
+ * 
+ */
+public class SparqlToPipelineTransformVisitor extends 
QueryModelVisitorBase {
--- End diff --

It doesn't strictly have to be executed first, though it may be better. For 
example, if the tree is `Join(, 
AggregationPipelineQueryNode))`, then the join iterator will get an iterator of 
results from the complex thing on the left, then for each result, execute the 
pipeline -- likely not optimal.

The only logic to try to group pipeline-amenable nodes/subqueries together 
is in this visitor; there's no restructuring done at a higher level to make it 
work better (except to the extent that ordinary query planning steps may happen 
to help). For example, this visitor can turn `Join(Join(Join(S

[GitHub] incubator-rya pull request #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/255#discussion_r160550347
  
--- Diff: 
extras/rya.forwardchain/src/main/java/org/apache/rya/forwardchain/rule/AntecedentVisitor.java
 ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.forwardchain.rule;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.helpers.QueryModelVisitorBase;
+
+/**
+ * Query visitor that identifies all triple patterns represented as
+ * {@link StatementPattern}s in a query, which therefore represent triples
+ * that could potentially contribute to a solution. Considers only the 
statement
+ * patterns themselves, i.e. the leaves of the query tree, and does not 
consider
+ * other constraints that may restrict the set of triples that may be 
relevant.
+ * This means relying on this analysis to determine whether a fact can be 
part
+ * of a solution can yield false positives, but not false negatives.
+ */
+class AntecedentVisitor extends QueryModelVisitorBase {
--- End diff --

Oh, I think I missed that -- looks like the only difference is that it 
skips over filter conditions. I think we don't want to skip filter conditions, 
here, though -- if the rule is `CONSTRUCT { stuff } WHERE { ?x a :T . FILTER 
EXISTS {?x :p :A} }` then the rule should be triggered by either `:x a :T` or 
`:x :p :A` .


---

[GitHub] incubator-rya pull request #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/255#discussion_r160549312
  
--- Diff: 
dao/mongodb.rya/src/test/java/org/apache/rya/mongodb/aggregation/SparqlToPipelineTransformVisitorTest.java
 ---
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.bson.Document;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.openrdf.model.URI;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.model.vocabulary.OWL;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.Join;
+import org.openrdf.query.algebra.MultiProjection;
+import org.openrdf.query.algebra.Not;
+import org.openrdf.query.algebra.Projection;
+import org.openrdf.query.algebra.ProjectionElem;
+import org.openrdf.query.algebra.ProjectionElemList;
+import org.openrdf.query.algebra.QueryRoot;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.ValueConstant;
+import org.openrdf.query.algebra.Var;
+
+import com.google.common.collect.Sets;
+import com.mongodb.MongoNamespace;
+import com.mongodb.client.MongoCollection;
+
+public class SparqlToPipelineTransformVisitorTest {
+
+private static final ValueFactory VF = ValueFactoryImpl.getInstance();
+
+private static final String LUBM = "urn:lubm";
+private static final URI UNDERGRAD = VF.createURI(LUBM, 
"UndergraduateStudent");
+private static final URI PROFESSOR = VF.createURI(LUBM, "Professor");
+private static final URI COURSE = VF.createURI(LUBM, "Course");
+private static final URI TAKES = VF.createURI(LUBM, "takesCourse");
+private static final URI TEACHES = VF.createURI(LUBM, "teachesCourse");
+
+private static Var constant(URI value) {
+return new Var(value.stringValue(), value);
+}
+
+MongoCollection collection;
+
+@Before
+@SuppressWarnings("unchecked")
+public void setUp() {
+collection = Mockito.mock(MongoCollection.class);
+Mockito.when(collection.getNamespace()).thenReturn(new 
MongoNamespace("db", "collection"));
+}
+
+@Test
+public void testStatementPattern() throws Exception {
+QueryRoot query = new QueryRoot(new StatementPattern(
+new Var("x"), constant(RDF.TYPE), constant(UNDERGRAD)));
+SparqlToPipelineTransformVisitor visitor = new 
SparqlToPipelineTransformVisitor(collection);
+query.visit(visitor);
+Assert.assertTrue(query.getArg() instanceof 
AggregationPipelineQueryNode);
+AggregationPipelineQueryNode pipelineNode = 
(AggregationPipelineQueryNode) query.getArg();
+Assert.assertEquals(Sets.newHashSet("x"), 
pipelineNode.getAssuredBindingNames());
+}
+
+@Test
+public void testJoin() throws Exception {
+QueryRoot query = new QueryRoot(new Join(
+new StatementPattern(new Var("x"), constant(RDF.TYPE), 
constant(UNDERGRAD)),
+new StatementPattern(new Var("x"), constant(TAKES), new 
Var("course";
+SparqlToPipelineTransformVisitor visitor = new 
SparqlToPipelineTransformVisitor(collection);
+query.visit(visitor);
+Assert.assertTrue(query.getArg() instanceof 
AggregationPipelineQueryNode);
+AggregationPipelineQueryNode pipe

[GitHub] incubator-rya pull request #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/255#discussion_r160545261
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryNode.java
 ---
@@ -0,0 +1,862 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.CONTEXT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.DOCUMENT_VISIBILITY;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_TYPE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.STATEMENT_METADATA;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.TIMESTAMP;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.Function;
+
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.api.domain.StatementMetadata;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.mongodb.MongoDbRdfConstants;
+import org.apache.rya.mongodb.dao.MongoDBStorageStrategy;
+import org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy;
+import 
org.apache.rya.mongodb.document.operators.query.ConditionalOperators;
+import 
org.apache.rya.mongodb.document.visibility.DocumentVisibilityAdapter;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.Compare;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.ProjectionElem;
+import org.openrdf.query.algebra.ProjectionElemList;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.ValueConstant;
+import org.openrdf.query.algebra.ValueExpr;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+
+import com.google.common.base.Objects;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.BiMap;
+import com.google.common.collect.HashBiMap;
+import com.mongodb.BasicDBObject;
+import com.mongodb.DBObject;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.Aggregates;
+import com.mongodb.client.model.BsonField;
+import com.mongodb.client.model.Filters;
+import com.mongodb.client.model.Projections;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * Represents a portion of a query tree as MongoDB aggregation pipeline. 
Should
+ * be built bottom-up: start with a statement pattern implemented as a 
$match
+ * step, then add steps to the pipeline to handle higher levels of the 
query
+ * tree. Methods are provided to add certain suppor

[GitHub] incubator-rya pull request #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/255#discussion_r160524082
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryNode.java
 ---
@@ -0,0 +1,862 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.CONTEXT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.DOCUMENT_VISIBILITY;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_TYPE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.STATEMENT_METADATA;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.TIMESTAMP;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.Function;
+
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.api.domain.StatementMetadata;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.mongodb.MongoDbRdfConstants;
+import org.apache.rya.mongodb.dao.MongoDBStorageStrategy;
+import org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy;
+import 
org.apache.rya.mongodb.document.operators.query.ConditionalOperators;
+import 
org.apache.rya.mongodb.document.visibility.DocumentVisibilityAdapter;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.Compare;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.ProjectionElem;
+import org.openrdf.query.algebra.ProjectionElemList;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.ValueConstant;
+import org.openrdf.query.algebra.ValueExpr;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+
+import com.google.common.base.Objects;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.BiMap;
+import com.google.common.collect.HashBiMap;
+import com.mongodb.BasicDBObject;
+import com.mongodb.DBObject;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.Aggregates;
+import com.mongodb.client.model.BsonField;
+import com.mongodb.client.model.Filters;
+import com.mongodb.client.model.Projections;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * Represents a portion of a query tree as MongoDB aggregation pipeline. 
Should
+ * be built bottom-up: start with a statement pattern implemented as a 
$match
+ * step, then add steps to the pipeline to handle higher levels of the 
query
+ * tree. Methods are provided to add certain suppor

[GitHub] incubator-rya issue #255: RYA-417 Forward-chaining batch rules engine

[jessehatfield]

Github user jessehatfield commented on the issue:

https://github.com/apache/incubator-rya/pull/255
  
asfbot build


---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159950551
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryNode.java
 ---
@@ -0,0 +1,882 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.CONTEXT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.DOCUMENT_VISIBILITY;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_TYPE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.STATEMENT_METADATA;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.TIMESTAMP;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.Function;
+
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.api.domain.StatementMetadata;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.mongodb.MongoDbRdfConstants;
+import org.apache.rya.mongodb.dao.MongoDBStorageStrategy;
+import org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy;
+import 
org.apache.rya.mongodb.document.operators.query.ConditionalOperators;
+import 
org.apache.rya.mongodb.document.visibility.DocumentVisibilityAdapter;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.Compare;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.ProjectionElem;
+import org.openrdf.query.algebra.ProjectionElemList;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.ValueConstant;
+import org.openrdf.query.algebra.ValueExpr;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.BiMap;
+import com.google.common.collect.HashBiMap;
+import com.mongodb.BasicDBObject;
+import com.mongodb.DBObject;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.Aggregates;
+import com.mongodb.client.model.BsonField;
+import com.mongodb.client.model.Filters;
+import com.mongodb.client.model.Projections;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * Represents a portion of a query tree as MongoDB aggregation pipeline. 
Should
+ * be built bottom-up: start with a statement pattern implemented as a 
$match
+ * step, then add steps to the pipeline to handle higher levels of the 
query
+ * tree. Methods are provided to add certain supported query operations to 
the
+ * end of the internal pip

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159932088
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryNode.java
 ---
@@ -0,0 +1,882 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.CONTEXT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.DOCUMENT_VISIBILITY;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_TYPE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.STATEMENT_METADATA;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.TIMESTAMP;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.Function;
+
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.api.domain.StatementMetadata;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.mongodb.MongoDbRdfConstants;
+import org.apache.rya.mongodb.dao.MongoDBStorageStrategy;
+import org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy;
+import 
org.apache.rya.mongodb.document.operators.query.ConditionalOperators;
+import 
org.apache.rya.mongodb.document.visibility.DocumentVisibilityAdapter;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.Compare;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.ProjectionElem;
+import org.openrdf.query.algebra.ProjectionElemList;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.ValueConstant;
+import org.openrdf.query.algebra.ValueExpr;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.BiMap;
+import com.google.common.collect.HashBiMap;
+import com.mongodb.BasicDBObject;
+import com.mongodb.DBObject;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.Aggregates;
+import com.mongodb.client.model.BsonField;
+import com.mongodb.client.model.Filters;
+import com.mongodb.client.model.Projections;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * Represents a portion of a query tree as MongoDB aggregation pipeline. 
Should
+ * be built bottom-up: start with a statement pattern implemented as a 
$match
+ * step, then add steps to the pipeline to handle higher levels of the 
query
+ * tree. Methods are provided to add certain supported query operations to 
the
+ * end of the internal pip

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159949217
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/SparqlToPipelineTransformVisitor.java
 ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import java.util.Arrays;
+
+import org.apache.rya.mongodb.MongoConnectorFactory;
+import org.apache.rya.mongodb.MongoDBRdfConfiguration;
+import org.bson.Document;
+import org.openrdf.query.algebra.Distinct;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.Filter;
+import org.openrdf.query.algebra.Join;
+import org.openrdf.query.algebra.MultiProjection;
+import org.openrdf.query.algebra.Projection;
+import org.openrdf.query.algebra.Reduced;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.helpers.QueryModelVisitorBase;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+
+public class SparqlToPipelineTransformVisitor extends 
QueryModelVisitorBase {
+private MongoCollection inputCollection;
+
+public SparqlToPipelineTransformVisitor(MongoCollection 
inputCollection) {
+this.inputCollection = inputCollection;
--- End diff --

Shouldn't be, check added.


---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159949017
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/PipelineResultIteration.java
 ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import java.util.Map;
+
+import org.bson.Document;
+import org.openrdf.model.Value;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.Binding;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.evaluation.QueryBindingSet;
+
+import com.mongodb.client.AggregateIterable;
+import com.mongodb.client.MongoCursor;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * An iterator that converts the documents resulting from an
+ * {@link AggregationPipelineQueryNode} into {@link BindingSet}s.
+ */
+public class PipelineResultIteration implements 
CloseableIteration<BindingSet, QueryEvaluationException> {
+private static final int BATCH_SIZE = 1000;
+private static final ValueFactory VF = ValueFactoryImpl.getInstance();
+
+private final MongoCursor cursor;
+private final Map<String, String> varToOriginalName;
+private final BindingSet bindings;
+private BindingSet nextSolution = null;
+
+/**
+ * Constructor.
+ * @param aggIter Iterator of documents in 
AggregationPipelineQueryNode's
+ *  intermediate solution representation.
+ * @param varToOriginalName A mapping from field names in the pipeline
+ *  result documents to equivalent variable names in the original 
query.
+ *  Where an entry does not exist for a field, the field name and 
variable
+ *  name are assumed to be the same.
+ * @param bindings A partial solution. May be empty.
+ */
+public PipelineResultIteration(AggregateIterable aggIter,
+Map<String, String> varToOriginalName,
+BindingSet bindings) {
+aggIter.batchSize(BATCH_SIZE);
+this.cursor = aggIter.iterator();
+this.varToOriginalName = varToOriginalName;
+this.bindings = bindings;
+lookahead();
--- End diff --

Removed.


---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159949072
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/PipelineResultIteration.java
 ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import java.util.Map;
+
+import org.bson.Document;
+import org.openrdf.model.Value;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.Binding;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.evaluation.QueryBindingSet;
+
+import com.mongodb.client.AggregateIterable;
+import com.mongodb.client.MongoCursor;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * An iterator that converts the documents resulting from an
+ * {@link AggregationPipelineQueryNode} into {@link BindingSet}s.
+ */
+public class PipelineResultIteration implements 
CloseableIteration<BindingSet, QueryEvaluationException> {
+private static final int BATCH_SIZE = 1000;
+private static final ValueFactory VF = ValueFactoryImpl.getInstance();
+
+private final MongoCursor cursor;
+private final Map<String, String> varToOriginalName;
+private final BindingSet bindings;
+private BindingSet nextSolution = null;
+
+/**
+ * Constructor.
+ * @param aggIter Iterator of documents in 
AggregationPipelineQueryNode's
+ *  intermediate solution representation.
+ * @param varToOriginalName A mapping from field names in the pipeline
+ *  result documents to equivalent variable names in the original 
query.
+ *  Where an entry does not exist for a field, the field name and 
variable
+ *  name are assumed to be the same.
+ * @param bindings A partial solution. May be empty.
+ */
+public PipelineResultIteration(AggregateIterable aggIter,
+Map<String, String> varToOriginalName,
+BindingSet bindings) {
+aggIter.batchSize(BATCH_SIZE);
+this.cursor = aggIter.iterator();
+this.varToOriginalName = varToOriginalName;
+this.bindings = bindings;
+lookahead();
+}
+
+private void lookahead() {
+while (nextSolution == null && cursor.hasNext()) {
+nextSolution = docToBindingSet(cursor.next());
+}
+}
+
+@Override
+public boolean hasNext() throws QueryEvaluationException {
+lookahead();
+return nextSolution != null;
+}
+
+@Override
+public BindingSet next() throws QueryEvaluationException {
+lookahead();
+BindingSet solution = nextSolution;
+nextSolution = null;
+return solution;
+}
+
+@Override
+public void remove() throws QueryEvaluationException {
--- End diff --

Agreed, done.


---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159949935
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/document/util/DocumentVisibilityUtil.java
 ---
@@ -328,4 +328,22 @@ public static boolean doesUserHaveDocumentAccess(final 
Authorizations authorizat
 }
 return list.toArray(new Object[0]);
 }
+
+/**
+ * Converts a {@link List} into an array of {@link Object}s.
--- End diff --

Removed this method anyway, turned out to be unnecessary. (I had some 
incorrect way of converting a Document to a DBObject; if done properly, the 
existing methods are sufficient.)


---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159925007
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryNode.java
 ---
@@ -0,0 +1,882 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.CONTEXT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.DOCUMENT_VISIBILITY;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_TYPE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.STATEMENT_METADATA;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.TIMESTAMP;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.Function;
+
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.api.domain.StatementMetadata;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.mongodb.MongoDbRdfConstants;
+import org.apache.rya.mongodb.dao.MongoDBStorageStrategy;
+import org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy;
+import 
org.apache.rya.mongodb.document.operators.query.ConditionalOperators;
+import 
org.apache.rya.mongodb.document.visibility.DocumentVisibilityAdapter;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.Compare;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.ProjectionElem;
+import org.openrdf.query.algebra.ProjectionElemList;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.ValueConstant;
+import org.openrdf.query.algebra.ValueExpr;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.BiMap;
+import com.google.common.collect.HashBiMap;
+import com.mongodb.BasicDBObject;
+import com.mongodb.DBObject;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.Aggregates;
+import com.mongodb.client.model.BsonField;
+import com.mongodb.client.model.Filters;
+import com.mongodb.client.model.Projections;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * Represents a portion of a query tree as MongoDB aggregation pipeline. 
Should
+ * be built bottom-up: start with a statement pattern implemented as a 
$match
+ * step, then add steps to the pipeline to handle higher levels of the 
query
+ * tree. Methods are provided to add certain supported query operations to 
the
+ * end of the internal pip

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159950589
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryNode.java
 ---
@@ -0,0 +1,882 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.CONTEXT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.DOCUMENT_VISIBILITY;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_TYPE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.STATEMENT_METADATA;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.TIMESTAMP;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.Function;
+
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.api.domain.StatementMetadata;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.mongodb.MongoDbRdfConstants;
+import org.apache.rya.mongodb.dao.MongoDBStorageStrategy;
+import org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy;
+import 
org.apache.rya.mongodb.document.operators.query.ConditionalOperators;
+import 
org.apache.rya.mongodb.document.visibility.DocumentVisibilityAdapter;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.Compare;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.ProjectionElem;
+import org.openrdf.query.algebra.ProjectionElemList;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.ValueConstant;
+import org.openrdf.query.algebra.ValueExpr;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.BiMap;
+import com.google.common.collect.HashBiMap;
+import com.mongodb.BasicDBObject;
+import com.mongodb.DBObject;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.Aggregates;
+import com.mongodb.client.model.BsonField;
+import com.mongodb.client.model.Filters;
+import com.mongodb.client.model.Projections;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * Represents a portion of a query tree as MongoDB aggregation pipeline. 
Should
+ * be built bottom-up: start with a statement pattern implemented as a 
$match
+ * step, then add steps to the pipeline to handle higher levels of the 
query
+ * tree. Methods are provided to add certain supported query operations to 
the
+ * end of the internal pip

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159949267
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/dao/SimpleMongoDBStorageStrategy.java
 ---
@@ -63,6 +63,10 @@
 public static final String STATEMENT_METADATA = "statementMetadata";
 public static final String DOCUMENT_VISIBILITY = "documentVisibility";
 
+public static String hash(String value) {
--- End diff --

Added.


---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159949139
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/SparqlToPipelineTransformVisitor.java
 ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import java.util.Arrays;
+
+import org.apache.rya.mongodb.MongoConnectorFactory;
+import org.apache.rya.mongodb.MongoDBRdfConfiguration;
+import org.bson.Document;
+import org.openrdf.query.algebra.Distinct;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.Filter;
+import org.openrdf.query.algebra.Join;
+import org.openrdf.query.algebra.MultiProjection;
+import org.openrdf.query.algebra.Projection;
+import org.openrdf.query.algebra.Reduced;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.helpers.QueryModelVisitorBase;
+
+import com.mongodb.MongoClient;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+
+public class SparqlToPipelineTransformVisitor extends 
QueryModelVisitorBase {
--- End diff --

Added.


---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159948908
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryOptimizer.java
 ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import org.apache.rya.mongodb.MongoDBRdfConfiguration;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.Dataset;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.evaluation.QueryOptimizer;
+
+public class AggregationPipelineQueryOptimizer implements QueryOptimizer, 
Configurable {
--- End diff --

Added.


---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159943074
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/PipelineResultIteration.java
 ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import java.util.Map;
+
+import org.bson.Document;
+import org.openrdf.model.Value;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.Binding;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.evaluation.QueryBindingSet;
+
+import com.mongodb.client.AggregateIterable;
+import com.mongodb.client.MongoCursor;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * An iterator that converts the documents resulting from an
+ * {@link AggregationPipelineQueryNode} into {@link BindingSet}s.
+ */
+public class PipelineResultIteration implements 
CloseableIteration<BindingSet, QueryEvaluationException> {
+private static final int BATCH_SIZE = 1000;
+private static final ValueFactory VF = ValueFactoryImpl.getInstance();
+
+private final MongoCursor cursor;
+private final Map<String, String> varToOriginalName;
+private final BindingSet bindings;
+private BindingSet nextSolution = null;
+
+/**
+ * Constructor.
+ * @param aggIter Iterator of documents in 
AggregationPipelineQueryNode's
+ *  intermediate solution representation.
+ * @param varToOriginalName A mapping from field names in the pipeline
+ *  result documents to equivalent variable names in the original 
query.
+ *  Where an entry does not exist for a field, the field name and 
variable
+ *  name are assumed to be the same.
+ * @param bindings A partial solution. May be empty.
+ */
+public PipelineResultIteration(AggregateIterable aggIter,
+Map<String, String> varToOriginalName,
+BindingSet bindings) {
+aggIter.batchSize(BATCH_SIZE);
+this.cursor = aggIter.iterator();
+this.varToOriginalName = varToOriginalName;
+this.bindings = bindings;
+lookahead();
+}
+
+private void lookahead() {
+while (nextSolution == null && cursor.hasNext()) {
+nextSolution = docToBindingSet(cursor.next());
+}
+}
+
+@Override
+public boolean hasNext() throws QueryEvaluationException {
+lookahead();
+return nextSolution != null;
+}
+
+@Override
+public BindingSet next() throws QueryEvaluationException {
+lookahead();
+BindingSet solution = nextSolution;
+nextSolution = null;
+return solution;
+}
+
+@Override
+public void remove() throws QueryEvaluationException {
+lookahead();
+nextSolution = null;
+}
+
+@Override
+public void close() throws QueryEvaluationException {
+cursor.close();
+}
+
+private QueryBindingSet docToBindingSet(Document result) {
+QueryBindingSet bindingSet = new QueryBindingSet(bindings);
+Document valueSet = 
result.get(AggregationPipelineQueryNode.VALUES, Document.class);
+Document typeSet = result.get(AggregationPipelineQueryNode.TYPES, 
Document.class);
+if (valueSet != null) {
+for (Map.Entry<String, Object> entry : valueSet.entrySet()) {
+String fieldName = entry.getKey();
+String valueString = entry.getValue().toString();
+String typeString = t

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159948871
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryOptimizer.java
 ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
--- End diff --

Changed, though it looks like we're inconsistent on this.


---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159950342
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/AggregationPipelineQueryNode.java
 ---
@@ -0,0 +1,882 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.CONTEXT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.DOCUMENT_VISIBILITY;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.OBJECT_TYPE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.PREDICATE_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.STATEMENT_METADATA;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.SUBJECT_HASH;
+import static 
org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy.TIMESTAMP;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.Function;
+
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.api.domain.StatementMetadata;
+import org.apache.rya.api.resolver.RdfToRyaConversions;
+import org.apache.rya.mongodb.MongoDbRdfConstants;
+import org.apache.rya.mongodb.dao.MongoDBStorageStrategy;
+import org.apache.rya.mongodb.dao.SimpleMongoDBStorageStrategy;
+import 
org.apache.rya.mongodb.document.operators.query.ConditionalOperators;
+import 
org.apache.rya.mongodb.document.visibility.DocumentVisibilityAdapter;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.openrdf.model.Literal;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.Compare;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.ProjectionElem;
+import org.openrdf.query.algebra.ProjectionElemList;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.ValueConstant;
+import org.openrdf.query.algebra.ValueExpr;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.BiMap;
+import com.google.common.collect.HashBiMap;
+import com.mongodb.BasicDBObject;
+import com.mongodb.DBObject;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.Aggregates;
+import com.mongodb.client.model.BsonField;
+import com.mongodb.client.model.Filters;
+import com.mongodb.client.model.Projections;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * Represents a portion of a query tree as MongoDB aggregation pipeline. 
Should
+ * be built bottom-up: start with a statement pattern implemented as a 
$match
+ * step, then add steps to the pipeline to handle higher levels of the 
query
+ * tree. Methods are provided to add certain supported query operations to 
the
+ * end of the internal pip

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/254#discussion_r159948988
  
--- Diff: 
dao/mongodb.rya/src/main/java/org/apache/rya/mongodb/aggregation/PipelineResultIteration.java
 ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.mongodb.aggregation;
+
+import java.util.Map;
+
+import org.bson.Document;
+import org.openrdf.model.Value;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.model.vocabulary.XMLSchema;
+import org.openrdf.query.Binding;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.evaluation.QueryBindingSet;
+
+import com.mongodb.client.AggregateIterable;
+import com.mongodb.client.MongoCursor;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * An iterator that converts the documents resulting from an
+ * {@link AggregationPipelineQueryNode} into {@link BindingSet}s.
+ */
+public class PipelineResultIteration implements 
CloseableIteration<BindingSet, QueryEvaluationException> {
+private static final int BATCH_SIZE = 1000;
+private static final ValueFactory VF = ValueFactoryImpl.getInstance();
+
+private final MongoCursor cursor;
+private final Map<String, String> varToOriginalName;
+private final BindingSet bindings;
+private BindingSet nextSolution = null;
+
+/**
+ * Constructor.
+ * @param aggIter Iterator of documents in 
AggregationPipelineQueryNode's
+ *  intermediate solution representation.
+ * @param varToOriginalName A mapping from field names in the pipeline
+ *  result documents to equivalent variable names in the original 
query.
+ *  Where an entry does not exist for a field, the field name and 
variable
+ *  name are assumed to be the same.
+ * @param bindings A partial solution. May be empty.
+ */
+public PipelineResultIteration(AggregateIterable aggIter,
+Map<String, String> varToOriginalName,
+BindingSet bindings) {
+aggIter.batchSize(BATCH_SIZE);
+this.cursor = aggIter.iterator();
--- End diff --

Done.


---

[GitHub] incubator-rya pull request #255: [WIP] RYA-417 Forward-chaining batch rules ...

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/255

[WIP] RYA-417 Forward-chaining batch rules engine


## Description
New project for handling general rules in a batch forward-chaining context, 
with specific implementations for simple SPIN construct rules. Tool reads rules 
from the current data store, then repeatedly applies rules, inserting derived 
triples back to the data store, until no new triples can be derived. Includes 
Mongo-specific implementation which executes rules using the aggregation 
pipeline. Depends on #254 for this reason.

Does not handle rules built from templates with variable substitution 
(templates with no variables are supported just like explicit rules). Does not 
support limits on the number of times a rule might apply, as long as the 
derived information is new. Skips any rule that would produce bnodes so that 
this behavior can't cause an infinite loop.

### Tests
Tests verify that construct queries are correctly parsed, and correctly 
analyzed to determine which rules can trigger which other rules. Integration 
test verifies that the batch tool produces correct derived triples, with both 
Mongo-specific and generic SAIL execution strategies.

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-417)
Depends on PR #254

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
[Add those who should review this]


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-417-forward-chain-spin

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/255.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #255


commit fab624ddda1b50f42ccd3347920668dc56245fc8
Author: Jesse Hatfield <jesse.hatfield@...>
Date:   2017-12-21T22:32:47Z

RYA-416 A new query node type to represent a MongoDB aggregation pipeline 
whose results can be converted to binding sets, and tools for optionally 
transforming some SPARQL expressions into such a node.

commit 80e27d59f6d1a8d1deac33a87a3ac02f44909aa3
Author: Jesse Hatfield <jesse.hatfield@...>
Date:   2017-12-22T17:02:33Z

RYA-417 Batch forward-chaining rules engine




---

[GitHub] incubator-rya pull request #254: RYA-416 Optionally invoke aggregation pipel...

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/254

RYA-416 Optionally invoke aggregation pipeline to execute MongoDB queries

[WIP] Provides tools for converting some or all of a SPARQL query to an 
aggregation pipeline.

## Description
>What Changed?
[Brief Description of what changed]

### Tests
>Coverage?

[Description of what tests were written]

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-NUMBER)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
[Add those who should review this]


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-416-aggregation-pipeline

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/254.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #254


commit 3eed2f92cc534768a6627a8c348eeef2c820c0e9
Author: Jesse Hatfield <jesse.hatfield@...>
Date:   2017-12-21T22:32:47Z

A new query node type to represent a MongoDB aggregation pipeline whose 
results can be converted to binding sets, and tools for optionally transforming 
some SPARQL expressions into such a node.




---

[GitHub] incubator-rya pull request #218: RYA-301 owl:ReflexiveProperty inference

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/218

RYA-301 owl:ReflexiveProperty inference


## Description
1. Added owl:ReflexiveProperty inference: If :hasFamilyMember is reflexive, 
then `?x :hasFamilyMember ?y` is true for `?x==?y`. Inference engine now loads 
reflexive properties like it does transitive and symmetric properties, and a 
visitor expands triples involving reflexive properties to match both when the 
triple actually exists or when the subject and object are the same.

2. Minor cleanup to wrap up latest inference changes: added/fixed 
configuration constants to enable/disable those inference visitors that were 
missing them or using the wrong one; fixed packaging for two test classes.
### Tests
Test rewriting done by ReflexivePropertyVisitor; test that InferenceEngine 
correctly determines property types; test queries involving reflexive 
properties.

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-301)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
@meiercaleb @pujav65 @ejwhite922 @isper3at 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-301-reflexiveproperty-inference

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/218.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #218


commit 86b5124015de895a451f2385be1f638c2c95f2ed
Author: Jesse Hatfield <jesse.hatfi...@parsons.com>
Date:   2017-08-28T22:46:02Z

RYA-301 owl:ReflexiveProperty inference




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r135311804
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -805,6 +843,72 @@ public void handleStatement(final Statement statement) 
throws RDFHandlerExceptio
 }
 
 /**
+ * For a given type, return any properties such that some owl:hasSelf
+ * restrictions implies that properties of this type have the value of
+ * themselves for this type.
+ *
+ * This takes into account type hierarchy, where children of a type 
that
+ * have this property are also assumed to have the property.
+ * 
+ * @param type
+ *The type (URI or bnode) to check against the known
+ *restrictions
+ * @return For each relevant property, a set of values such that 
whenever a
--- End diff --

Description above is good but this is still the old `@return`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r135155805
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -762,6 +800,57 @@ public void handleStatement(final Statement statement) 
throws RDFHandlerExceptio
 }
 
 /**
+ * For a given type, return any properties and values such that 
owl:hasSelf restrictions on
+ * those properties could imply this type. No matter how many 
restrictions are returned, each
+ * one is considered individually sufficient: if a resource has the 
property and the value, then
+ * it belongs to the provided type. Takes type hierarchy into account, 
so the value may imply a
+ * subtype which in turn implies the provided type.
+ * @param type The type (URI or bnode) to check against the known 
restrictions
+ * @return For each relevant property, a set of values such that 
whenever a resource has that
+ *  value for that property, it is implied to belong to the type.
+ */
+public Set getHasSelfImplyingType(final Resource type){
+// return properties that imply this type if reflexive
+final Set properties = new HashSet<>();
+properties.addAll(hasSelfByType.get(type));
+//findParent gets all subclasses, add self.
+if (type instanceof URI) {
+for (final URI subtype : findParents(subClassOfGraph, (URI) 
type)) {
+properties.addAll(hasSelfByType.get(subtype));
+}
+}
+
+// make map hasSelfByType[]
+return properties;
+}
+
+/**
+ * For a given property, return any types such that some owl:hasSelf 
restriction implies that members
+ * of the type have the value of themselves for this property.
+ *
+ * This takes into account type hierarchy, where children of a type 
that have
+ * this property are also assumed to have the property.
+ * @param property The property whose owl:hasSelf restrictions to 
return
+ * @return A set of types that possess the implied property.
+ */
+public Set getHasSelfImplyingProperty(final URI property) {
+// return types that imply this type if reflexive
+final Set types = new HashSet<>();
+final Set baseTypes = hasSelfByProperty.get(property);
--- End diff --

Same issue as with getHasSelfImplyingType -- check that baseTypes isn't 
null before calling addAll and iterating through it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r135154955
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -762,6 +800,57 @@ public void handleStatement(final Statement statement) 
throws RDFHandlerExceptio
 }
 
 /**
+ * For a given type, return any properties and values such that 
owl:hasSelf restrictions on
+ * those properties could imply this type. No matter how many 
restrictions are returned, each
+ * one is considered individually sufficient: if a resource has the 
property and the value, then
+ * it belongs to the provided type. Takes type hierarchy into account, 
so the value may imply a
+ * subtype which in turn implies the provided type.
+ * @param type The type (URI or bnode) to check against the known 
restrictions
+ * @return For each relevant property, a set of values such that 
whenever a resource has that
+ *  value for that property, it is implied to belong to the type.
+ */
+public Set getHasSelfImplyingType(final Resource type){
+// return properties that imply this type if reflexive
+final Set properties = new HashSet<>();
+properties.addAll(hasSelfByType.get(type));
--- End diff --

This line can cause an NPE if there is no hasSelf property associated with 
the type. Looks like that's why unrelated tests can fail if the HasSelfVisitor 
gets a chance to visit them -- any statement pattern like "?x rdf:type :T" will 
end up here, so it needs to check whether hasSelf logic applies before 
proceeding.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r135156081
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -1022,4 +1060,44 @@ public void setSchedule(boolean schedule) {
 }
 return implications;
 }
-}
+
+/**
+ * For a given type, return any properties and values such that 
owl:hasSelf restrictions on
+ * those properties could imply this type. No matter how many 
restrictions are returned, each
+ * one is considered individually sufficient: if a resource has the 
property and the value, then
+ * it belongs to the provided type. Takes type hierarchy into account, 
so the value may imply a
+ * subtype which in turn implies the provided type.
+ * @param type The type (URI or bnode) to check against the known 
restrictions
+ * @return For each relevant property, a set of values such that 
whenever a resource has that
+ *  value for that property, it is implied to belong to the type.
+ */
--- End diff --

This javadoc didn't get updated like the other method's did


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r135155573
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -762,6 +800,57 @@ public void handleStatement(final Statement statement) 
throws RDFHandlerExceptio
 }
 
 /**
+ * For a given type, return any properties and values such that 
owl:hasSelf restrictions on
+ * those properties could imply this type. No matter how many 
restrictions are returned, each
+ * one is considered individually sufficient: if a resource has the 
property and the value, then
+ * it belongs to the provided type. Takes type hierarchy into account, 
so the value may imply a
+ * subtype which in turn implies the provided type.
+ * @param type The type (URI or bnode) to check against the known 
restrictions
+ * @return For each relevant property, a set of values such that 
whenever a resource has that
+ *  value for that property, it is implied to belong to the type.
+ */
+public Set getHasSelfImplyingType(final Resource type){
+// return properties that imply this type if reflexive
+final Set properties = new HashSet<>();
+properties.addAll(hasSelfByType.get(type));
+//findParent gets all subclasses, add self.
+if (type instanceof URI) {
+for (final URI subtype : findParents(subClassOfGraph, (URI) 
type)) {
+properties.addAll(hasSelfByType.get(subtype));
--- End diff --

Same issue here -- should only addAll if we're sure that 
hasSelfByType.get(subtype) returned non-null


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya issue #217: RYA-294 owl:someValuesFrom inference

[jessehatfield]

Github user jessehatfield commented on the issue:

https://github.com/apache/incubator-rya/pull/217
  
Added option to disable someValuesFrom inference. Added null checks to 
underlying methods, documented behavior in calling methods.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #215: RYA-300 Added owl:oneOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/215#discussion_r135078753
  
--- Diff: 
sail/src/test/java/org/apache/rya/rdftriplestore/inference/InferenceIT.java ---
@@ -400,4 +400,97 @@ public void testIntersectionOfQuery() throws Exception 
{
 expectedMen.add(new ListBindingSet(varNames, 
vf.createURI("urn:Bob")));
 Assert.assertEquals(expectedMen, new HashSet<>(solutions));
 }
+
+@Test
+public void testOneOfQuery() throws Exception {
+final String ontology = "INSERT DATA { GRAPH <http://updated/test> 
{\n"
++ "   owl:oneOf _:bnodeS1 . \n"
++ "  _:bnodeS1 rdf:first  . \n"
++ "  _:bnodeS1 rdf:rest _:bnodeS2 . \n"
++ "  _:bnodeS2 rdf:first  . \n"
++ "  _:bnodeS2 rdf:rest _:bnodeS3 . \n"
++ "  _:bnodeS3 rdf:first  . \n"
++ "  _:bnodeS3 rdf:rest _:bnodeS4 . \n"
++ "  _:bnodeS4 rdf:first  . \n"
++ "  _:bnodeS4 rdf:rest rdf:nil . \n"
++ "   owl:oneOf _:bnodeR1 . \n"
++ "  _:bnodeR1 rdf:first  . \n"
++ "  _:bnodeR1 rdf:rest _:bnodeR2 . \n"
++ "  _:bnodeR2 rdf:first  . \n"
++ "  _:bnodeR2 rdf:rest _:bnodeR3 . \n"
++ "  _:bnodeR3 rdf:first  . \n"
++ "  _:bnodeR3 rdf:rest _:bnodeR4 . \n"
++ "  _:bnodeR4 rdf:first  . \n"
++ "  _:bnodeR4 rdf:rest _:bnodeR5 . \n"
++ "  _:bnodeR5 rdf:first  . \n"
++ "  _:bnodeR5 rdf:rest _:bnodeR6 . \n"
++ "  _:bnodeR6 rdf:first  . \n"
++ "  _:bnodeR6 rdf:rest _:bnodeR7 . \n"
++ "  _:bnodeR7 rdf:first  . \n"
++ "  _:bnodeR7 rdf:rest _:bnodeR8 . \n"
++ "  _:bnodeR8 rdf:first  . \n"
++ "  _:bnodeR8 rdf:rest _:bnodeR9 . \n"
++ "  _:bnodeR9 rdf:first  . \n"
++ "  _:bnodeR9 rdf:rest _:bnodeR10 . \n"
++ "  _:bnodeR10 rdf:first  . \n"
++ "  _:bnodeR10 rdf:rest _:bnodeR11 . \n"
++ "  _:bnodeR11 rdf:first  . \n"
++ "  _:bnodeR11 rdf:rest _:bnodeR12 . \n"
++ "  _:bnodeR12 rdf:first  . \n"
++ "  _:bnodeR12 rdf:rest _:bnodeR13 . \n"
++ "  _:bnodeR13 rdf:first  . \n"
++ "  _:bnodeR13 rdf:rest rdf:nil . \n"
++ "   owl:equivalentClass [\n"
++ "owl:onProperty  ; owl:range 
 ;\n"
++ "owl:onProperty  ; owl:range 
 ;\n"
++ "  ] . \n"
--- End diff --

Same comment as with the example regarding equivalent class statement


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #215: RYA-300 Added owl:oneOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/215#discussion_r135076533
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/OneOfVisitor.java ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.rdftriplestore.inference;
+
+import java.util.LinkedHashSet;
+import java.util.Set;
+
+import org.apache.log4j.Logger;
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.algebra.BindingSetAssignment;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.QueryBindingSet;
+
+/**
+ * Visitor for handling owl:oneOf inferencing on a node.
+ */
+public class OneOfVisitor extends AbstractInferVisitor {
+private static final Logger log = Logger.getLogger(OneOfVisitor.class);
+
+/**
+ * Creates a new instance of {@link OneOfVisitor}.
+ * @param conf the {@link RdfCloudeTripleStoreConfiguration}.
+ * @param inferenceEngine the {@link InferenceEngine}.
+ */
+public OneOfVisitor(final RdfCloudTripleStoreConfiguration conf, final 
InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = conf.isInferOneOf();
+}
+
+@Override
+protected void meetSP(final StatementPattern node) throws Exception {
+final Var subVar = node.getSubjectVar();
+final Var predVar = node.getPredicateVar();
+final Var objVar = node.getObjectVar();
+final Var conVar = node.getContextVar();
+if (predVar != null && objVar != null && objVar.getValue() != null 
&& RDF.TYPE.equals(predVar.getValue()) && !EXPANDED.equals(conVar)) {
+final URI object = (URI) objVar.getValue();
--- End diff --

Should this be Resource instead of URI, and should we verify the type in 
the if condition?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #215: RYA-300 Added owl:oneOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/215#discussion_r135080995
  
--- Diff: 
sail/src/test/java/org/apache/rya/rdftriplestore/inference/OneOfVisitorTest.java
 ---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.rdftriplestore.inference;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.junit.Test;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.Binding;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.algebra.BindingSetAssignment;
+import org.openrdf.query.algebra.Projection;
+import org.openrdf.query.algebra.ProjectionElem;
+import org.openrdf.query.algebra.ProjectionElemList;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.QueryBindingSet;
+
+import com.beust.jcommander.internal.Lists;
+import com.google.common.collect.Sets;
+
+/**
+ * Tests the methods of {@link OneOfVisitor}.
+ */
+public class OneOfVisitorTest {
+private final AccumuloRdfConfiguration conf = new 
AccumuloRdfConfiguration();
+private static final ValueFactory VF = new ValueFactoryImpl();
+
+private static final URI SUITS = VF.createURI("urn:Suits");
+private static final URI RANKS = VF.createURI("urn:Ranks");
+
+// Definition #1: :Suits owl:oneOf(:Clubs, :Diamonds, :Hearts, :Spades)
+private static final URI CLUBS = VF.createURI("urn:Clubs");
+private static final URI DIAMONDS = VF.createURI("urn:Diamonds");
+private static final URI HEARTS = VF.createURI("urn:Hearts");
+private static final URI SPADES = VF.createURI("urn:Spades");
+
+// Definition #2: :Ranks owl:oneOf(:Ace, :2, :3, :4, :5, :6, :7, :8, 
:9, :10, :Jack, :Queen, :King)
+private static final URI ACE = VF.createURI("urn:Ace");
+private static final URI TWO = VF.createURI("urn:2");
+private static final URI THREE = VF.createURI("urn:3");
+private static final URI FOUR = VF.createURI("urn:4");
+private static final URI FIVE = VF.createURI("urn:5");
+private static final URI SIX = VF.createURI("urn:6");
+private static final URI SEVEN = VF.createURI("urn:7");
+private static final URI EIGHT = VF.createURI("urn:8");
+private static final URI NINE = VF.createURI("urn:9");
+private static final URI TEN = VF.createURI("urn:10");
+private static final URI JACK = VF.createURI("urn:Jack");
+private static final URI QUEEN = VF.createURI("urn:Queen");
+private static final URI KING = VF.createURI("urn:King");
+
+private static final Set CARD_SUIT_ENUMERATION =
+Sets.newLinkedHashSet(
+Lists.newArrayList(CLUBS, DIAMONDS, HEARTS, SPADES)
+);
+private static final Set CARD_RANK_ENUMERATION =
+Sets.newLinkedHashSet(
+Lists.newArrayList(
+ACE, TWO, THREE, FOUR, FIVE, SIX, SEVEN, EIGHT, NINE, TEN,
+JACK, QUEEN, KING
+)
+);
+
+@Test
+public void testOne

[GitHub] incubator-rya pull request #217: RYA-294 owl:someValuesFrom inference

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/217

RYA-294 owl:someValuesFrom inference


## Description
Inference applies owl:someValuesFrom semantics for queries including 
statement patterns of the form "?x rdf:type :DefinedClass".

An owl:someValuesFrom property restriction is an existential class 
expression: it defines type T1 to be the set of individuals, such that, for a 
given property p and type T2, there is at least one value of type T2 for that 
property, i.e. for any subject belonging to T1 there exists a triple whose 
predicate is p and whose object belongs to T2. Therefore, if an individual is 
known to belong to the value type (T2), then any individual having it as a 
value for property p by definition belongs to T1. This is similar to 
rdfs:domain except that it only applies when the object of the triple belongs 
to the appropriate class expression. It is the converse of owl:allValuesFrom, 
in which the subject's type and the predicate are used to infer the object's 
type.

(It's also theoretically true that if something belongs to the set defined 
by owl:someValuesFrom, then there must exist some individual which is the value 
and belongs to that type. But we don't have a direct way to use that 
implication to answer queries, so it is ignored for now.)

InferenceEngine stores someValuesFrom information nearly identically to 
allValuesFrom information, except that it can be accessed by the type of the 
restriction rather than the type of the value. Some edits to allValuesFrom 
logic for consistency/simplicity/reuse.

SomeValuesFromVisitor processes statement patterns of the form "?x rdf:type 
:T1", and if :T2 is the type of an owl:someValuesFrom restriction according to 
the inference engine, it replaces the statement pattern with a union: of 1) the 
same statement pattern; and 2) a subquery of the form "?y rdf:type :T2. ?x :p 
?y." for the appropriate (restriction type, property) pairs.

RdfCloudTripleStoreConnection adds this to its list of visitors to call on 
a query.

Added an example to MongoRyaDirectExample, using a slightly modified piece 
of the LUBM example schema (can't use the exact formulation from LUBM because 
it relies on composing intersection and property restriction logic, where Rya 
will not presently apply both).

### Tests
Unit test to verify that InferenceEngine stores and returns the schema; 
unit test to verify that type queries are rewritten appropriately; integration 
test to verify correct results for sample ontology+instances+query.

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-294)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
@meiercaleb @ejwhite922 @isper3at @pujav65 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-294-someValuesFrom-inference

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/217.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #217


commit 870bc57f909cb76ed9603ad2a27ff9824e4be0c7
Author: Jesse Hatfield <jesse.hatfi...@parsons.com>
Date:   2017-08-24T01:14:48Z

RYA-294 owl:someValuesFrom inference




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134635678
  
--- Diff: 
sail/src/test/java/org/apache/rya/rdftriplestore/inference/InferenceIT.java ---
@@ -324,15 +325,64 @@ public void testUnionQuery() throws Exception {
 inferenceEngine.refreshGraph();
 conn.prepareUpdate(QueryLanguage.SPARQL, instances).execute();
 conn.prepareTupleQuery(QueryLanguage.SPARQL, 
query).evaluate(resultHandler);
-Set expected = new HashSet<>();
+final Set expected = new HashSet<>();
 expected.add(vf.createURI("urn:Bob"));
 expected.add(vf.createURI("urn:Carol"));
 expected.add(vf.createURI("urn:Eve"));
-Set returned = new HashSet<>();
-for (BindingSet bs : solutions) {
+final Set returned = new HashSet<>();
+for (final BindingSet bs : solutions) {
 returned.add(bs.getBinding("x").getValue());
 }
 Assert.assertEquals(expected, returned);
 Assert.assertEquals(expected.size(), solutions.size());
 }
+
+@Test
+public void testHasSelfQuery() throws Exception {
+final String ontology = "INSERT DATA { GRAPH <http://updated/test> 
{\n"
++ "   owl:onProperty  ; 
owl:hasSelf \"true\" . \n"
++ "}}";
+final String instances = "INSERT DATA { GRAPH 
<http://updated/test> {\n"
++ "   a  . \n"
++ " . \n"
++ "}}";
+conn.prepareUpdate(QueryLanguage.SPARQL, ontology).execute();
+conn.prepareUpdate(QueryLanguage.SPARQL, instances).execute();
+inferenceEngine.refreshGraph();
+
+String query = "SELECT ?who ?self { GRAPH <http://updated/test> { 
?self  ?who } } \n";
+conn.prepareTupleQuery(QueryLanguage.SPARQL, 
query).evaluate(resultHandler);
+final Set expected = new HashSet();
+final List varNames = new LinkedList<>();
+varNames.add("who");
+varNames.add("self");
+expected.add(new ListBindingSet(varNames, 
vf.createURI("urn:Alice"), vf.createURI("urn:Alice")));
+expected.add(new ListBindingSet(varNames, 
vf.createURI("urn:Narcissus"), vf.createURI("urn:Narcissus")));
+Assert.assertEquals(expected, new HashSet<>(solutions));
+
+query = "SELECT ?self { GRAPH <http://updated/test> {  
 ?self } } \n";
+conn.prepareTupleQuery(QueryLanguage.SPARQL, 
query).evaluate(resultHandler);
+expected.clear();
+varNames.clear();
+varNames.add("self");
+expected.add(new ListBindingSet(varNames, 
vf.createURI("urn:Alice")));
+Assert.assertEquals(expected, new HashSet<>(solutions));
+
+query = "SELECT ?who { GRAPH <http://updated/test> { ?who 
  } } \n";
--- End diff --

Was this case supposed to be Alice instead of Narcissus? That would test 
the hasSelf reasoning when the object is a constant (which I think should catch 
the issues I see there), whereas this version is just matching the direct 
triple (urn:Narcissus urn:loves urn:Narcissus).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134620096
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasSelfVisitor.java 
---
@@ -0,0 +1,81 @@
+package org.apache.rya.rdftriplestore.inference;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+
+/**
+ * Expands the query tree to account for any relevant has-self class 
expressions
+ * in the ontology known to the {@link InferenceEngine}.
+ *
+ * Only operates on {@link StatementPattern} nodes, and only those 
including a
+ * defined type or defined predicate which is relevant to a has-self 
expression
+ * in the ontology. When applicable, replaces the node with one or more
+ * {@link InferUnion}s, one of whose leaves is the original 
StatementPattern.
+ *
+ * A has-self restriction defines the set of resources that are connected 
to
+ * themselves by a property. If the ontology states that a type is a 
resource
+ * that has a self referencing property, then the inference engine should:
+ * 1. Rewrite queries of the from ?x rdf:type :T to find all resources
+ * matching ?x :P ?x (as well as anything explicitly stated to be of type 
:T)
+ * 
+ * 2. Rewrite queries of the from :CONST :P ?o or ?subj :P :CONST to 
match
+ * :CONST if :CONST is known to have the type :T
+ * 
+ */
+public class HasSelfVisitor extends AbstractInferVisitor {
+private static final Var TYPE_VAR = new Var("p", RDF.TYPE);
+
+/**
+ * Creates a new {@link HasSelfVisitor}, which is enabled by default.
+ * @param conf The {@link RdfCloudTripleStoreConfiguration}.
+ * @param inferenceEngine The InferenceEngine containing the relevant 
ontology.
+ */
+public HasSelfVisitor(final RdfCloudTripleStoreConfiguration conf, 
final InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = conf.hasSelf();
+}
+
+@Override
+protected void meetSP(final StatementPattern node) throws Exception {
+final URI pred = (URI) node.getPredicateVar().getValue();
+final Var obj = node.getObjectVar();
+//if originalSP like (?s rdf:type :C1):  require that C1 is 
defined, i.e. not a variable
+// node <- originalSP
+final StatementPattern clone = node.clone();
+if(pred.equals(RDF.TYPE) && obj.isConstant()) {
+//for property in getHasSelfImplyingType(C1):
+for (final URI property : 
inferenceEngine.getHasSelfImplyingType((URI) obj.getValue())) {
+//node <- InferUnion(node, StatementPattern(?s, property, 
?s)).
+final InferUnion union = new InferUnion(clone, new 
StatementPattern(clone.getSubjectVar(), new Var("p", property), 
clone.getSubjectVar()));
--- End diff --

Similar comment to TYPE_VAR, might rename from "p" to something like the 
property's actual URI.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134617888
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasSelfVisitor.java 
---
@@ -0,0 +1,81 @@
+package org.apache.rya.rdftriplestore.inference;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+
+/**
+ * Expands the query tree to account for any relevant has-self class 
expressions
+ * in the ontology known to the {@link InferenceEngine}.
+ *
+ * Only operates on {@link StatementPattern} nodes, and only those 
including a
+ * defined type or defined predicate which is relevant to a has-self 
expression
+ * in the ontology. When applicable, replaces the node with one or more
+ * {@link InferUnion}s, one of whose leaves is the original 
StatementPattern.
+ *
+ * A has-self restriction defines the set of resources that are connected 
to
+ * themselves by a property. If the ontology states that a type is a 
resource
+ * that has a self referencing property, then the inference engine should:
+ * 1. Rewrite queries of the from ?x rdf:type :T to find all resources
+ * matching ?x :P ?x (as well as anything explicitly stated to be of type 
:T)
+ * 
+ * 2. Rewrite queries of the from :CONST :P ?o or ?subj :P :CONST to 
match
+ * :CONST if :CONST is known to have the type :T
+ * 
+ */
+public class HasSelfVisitor extends AbstractInferVisitor {
+private static final Var TYPE_VAR = new Var("p", RDF.TYPE);
+
+/**
+ * Creates a new {@link HasSelfVisitor}, which is enabled by default.
+ * @param conf The {@link RdfCloudTripleStoreConfiguration}.
+ * @param inferenceEngine The InferenceEngine containing the relevant 
ontology.
+ */
+public HasSelfVisitor(final RdfCloudTripleStoreConfiguration conf, 
final InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = conf.hasSelf();
+}
+
+@Override
+protected void meetSP(final StatementPattern node) throws Exception {
+final URI pred = (URI) node.getPredicateVar().getValue();
+final Var obj = node.getObjectVar();
+//if originalSP like (?s rdf:type :C1):  require that C1 is 
defined, i.e. not a variable
+// node <- originalSP
+final StatementPattern clone = node.clone();
+if(pred.equals(RDF.TYPE) && obj.isConstant()) {
--- End diff --

pred could be null here if the predicate is a variable. We need to make 
sure we skip over that (make it RDF.TYPE.equals(pred), explicitly check for 
null, or maybe require node.getPredicateVar().isConstant())


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134629463
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasSelfVisitor.java 
---
@@ -0,0 +1,81 @@
+package org.apache.rya.rdftriplestore.inference;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+
+/**
+ * Expands the query tree to account for any relevant has-self class 
expressions
+ * in the ontology known to the {@link InferenceEngine}.
+ *
+ * Only operates on {@link StatementPattern} nodes, and only those 
including a
+ * defined type or defined predicate which is relevant to a has-self 
expression
+ * in the ontology. When applicable, replaces the node with one or more
+ * {@link InferUnion}s, one of whose leaves is the original 
StatementPattern.
+ *
+ * A has-self restriction defines the set of resources that are connected 
to
+ * themselves by a property. If the ontology states that a type is a 
resource
+ * that has a self referencing property, then the inference engine should:
+ * 1. Rewrite queries of the from ?x rdf:type :T to find all resources
+ * matching ?x :P ?x (as well as anything explicitly stated to be of type 
:T)
+ * 
+ * 2. Rewrite queries of the from :CONST :P ?o or ?subj :P :CONST to 
match
+ * :CONST if :CONST is known to have the type :T
+ * 
+ */
+public class HasSelfVisitor extends AbstractInferVisitor {
+private static final Var TYPE_VAR = new Var("p", RDF.TYPE);
+
+/**
+ * Creates a new {@link HasSelfVisitor}, which is enabled by default.
+ * @param conf The {@link RdfCloudTripleStoreConfiguration}.
+ * @param inferenceEngine The InferenceEngine containing the relevant 
ontology.
+ */
+public HasSelfVisitor(final RdfCloudTripleStoreConfiguration conf, 
final InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = conf.hasSelf();
+}
+
+@Override
+protected void meetSP(final StatementPattern node) throws Exception {
+final URI pred = (URI) node.getPredicateVar().getValue();
+final Var obj = node.getObjectVar();
+//if originalSP like (?s rdf:type :C1):  require that C1 is 
defined, i.e. not a variable
+// node <- originalSP
+final StatementPattern clone = node.clone();
+if(pred.equals(RDF.TYPE) && obj.isConstant()) {
+//for property in getHasSelfImplyingType(C1):
+for (final URI property : 
inferenceEngine.getHasSelfImplyingType((URI) obj.getValue())) {
+//node <- InferUnion(node, StatementPattern(?s, property, 
?s)).
+final InferUnion union = new InferUnion(clone, new 
StatementPattern(clone.getSubjectVar(), new Var("p", property), 
clone.getSubjectVar()));
+//originalSP.replaceWith(node)
+node.replaceWith(union);
+}
+//else if originalSP like (s :p o):  where p is not a variable and 
at least one of s and o are variables
+} else if (node.getPredicateVar().isConstant()
+   && (!node.getSubjectVar().isConstant() ||
+   !node.getObjectVar().isConstant())) {
+//for type in getHasSelfImplyingProperty(p):
+for (final Resource type : 
inferenceEngine.getHasSelfImplyingProperty(pred)) {
+final Extension extension;
+if(obj.isConstant()) { // subject is the variable
+//Extension(StatementPattern(o, rdf:type, type), 
ExtensionElem(o, "s"))
+extension = new Extension(new StatementPattern(obj, 
TYPE_VAR, obj),
+new ExtensionElem(obj, "o"));
--- End diff --

ExtensionElem(obj, "o") should be ExtensionElem(obj, 
node.getSubjectVar().getName())
(mirrors else clause -- if there is a triple <obj, rdf:type, :HasSelfType> 
then there is a solution where ?subject=obj)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134629734
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasSelfVisitor.java 
---
@@ -0,0 +1,81 @@
+package org.apache.rya.rdftriplestore.inference;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+
+/**
+ * Expands the query tree to account for any relevant has-self class 
expressions
+ * in the ontology known to the {@link InferenceEngine}.
+ *
+ * Only operates on {@link StatementPattern} nodes, and only those 
including a
+ * defined type or defined predicate which is relevant to a has-self 
expression
+ * in the ontology. When applicable, replaces the node with one or more
+ * {@link InferUnion}s, one of whose leaves is the original 
StatementPattern.
+ *
+ * A has-self restriction defines the set of resources that are connected 
to
+ * themselves by a property. If the ontology states that a type is a 
resource
+ * that has a self referencing property, then the inference engine should:
+ * 1. Rewrite queries of the from ?x rdf:type :T to find all resources
+ * matching ?x :P ?x (as well as anything explicitly stated to be of type 
:T)
+ * 
+ * 2. Rewrite queries of the from :CONST :P ?o or ?subj :P :CONST to 
match
+ * :CONST if :CONST is known to have the type :T
+ * 
+ */
+public class HasSelfVisitor extends AbstractInferVisitor {
+private static final Var TYPE_VAR = new Var("p", RDF.TYPE);
+
+/**
+ * Creates a new {@link HasSelfVisitor}, which is enabled by default.
+ * @param conf The {@link RdfCloudTripleStoreConfiguration}.
+ * @param inferenceEngine The InferenceEngine containing the relevant 
ontology.
+ */
+public HasSelfVisitor(final RdfCloudTripleStoreConfiguration conf, 
final InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = conf.hasSelf();
+}
+
+@Override
+protected void meetSP(final StatementPattern node) throws Exception {
+final URI pred = (URI) node.getPredicateVar().getValue();
+final Var obj = node.getObjectVar();
+//if originalSP like (?s rdf:type :C1):  require that C1 is 
defined, i.e. not a variable
+// node <- originalSP
+final StatementPattern clone = node.clone();
+if(pred.equals(RDF.TYPE) && obj.isConstant()) {
+//for property in getHasSelfImplyingType(C1):
+for (final URI property : 
inferenceEngine.getHasSelfImplyingType((URI) obj.getValue())) {
+//node <- InferUnion(node, StatementPattern(?s, property, 
?s)).
+final InferUnion union = new InferUnion(clone, new 
StatementPattern(clone.getSubjectVar(), new Var("p", property), 
clone.getSubjectVar()));
+//originalSP.replaceWith(node)
+node.replaceWith(union);
+}
+//else if originalSP like (s :p o):  where p is not a variable and 
at least one of s and o are variables
+} else if (node.getPredicateVar().isConstant()
+   && (!node.getSubjectVar().isConstant() ||
+   !node.getObjectVar().isConstant())) {
+//for type in getHasSelfImplyingProperty(p):
+for (final Resource type : 
inferenceEngine.getHasSelfImplyingProperty(pred)) {
+final Extension extension;
+if(obj.isConstant()) { // subject is the variable
+//Extension(StatementPattern(o, rdf:type, type), 
ExtensionElem(o, "s"))
+extension = new Extension(new StatementPattern(obj, 
TYPE_VAR, obj),
+new ExtensionElem(obj, "o"));
+} else { //o is a variable and s may either be defined or 
a variable
+//Extension(StatementPattern(s, rdf:type, type), 
ExtensionElem(s, "o"))
+extension = new Extension(new 
StatementPattern(node.getSubjectVar(), TYPE_VAR, new Var("o", type)),
--- End diff --

Same Var name comment as above; might change name from "o" to 
type.stringValue() or something


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134630848
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -1022,4 +1060,44 @@ public void setSchedule(boolean schedule) {
 }
 return implications;
 }
-}
+
+/**
+ * For a given type, return any properties and values such that 
owl:hasSelf restrictions on
+ * those properties could imply this type. No matter how many 
restrictions are returned, each
+ * one is considered individually sufficient: if a resource has the 
property and the value, then
+ * it belongs to the provided type. Takes type hierarchy into account, 
so the value may imply a
+ * subtype which in turn implies the provided type.
+ * @param type The type (URI or bnode) to check against the known 
restrictions
+ * @return For each relevant property, a set of values such that 
whenever a resource has that
+ *  value for that property, it is implied to belong to the type.
+ */
--- End diff --

Edit doc to apply to hasSelf as opposed to hasValue -- in particular, 
`@return` needs to be fixed, and "if a resource has the property and the value" 
should state the reflexivity condition instead


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134629624
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasSelfVisitor.java 
---
@@ -0,0 +1,81 @@
+package org.apache.rya.rdftriplestore.inference;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+
+/**
+ * Expands the query tree to account for any relevant has-self class 
expressions
+ * in the ontology known to the {@link InferenceEngine}.
+ *
+ * Only operates on {@link StatementPattern} nodes, and only those 
including a
+ * defined type or defined predicate which is relevant to a has-self 
expression
+ * in the ontology. When applicable, replaces the node with one or more
+ * {@link InferUnion}s, one of whose leaves is the original 
StatementPattern.
+ *
+ * A has-self restriction defines the set of resources that are connected 
to
+ * themselves by a property. If the ontology states that a type is a 
resource
+ * that has a self referencing property, then the inference engine should:
+ * 1. Rewrite queries of the from ?x rdf:type :T to find all resources
+ * matching ?x :P ?x (as well as anything explicitly stated to be of type 
:T)
+ * 
+ * 2. Rewrite queries of the from :CONST :P ?o or ?subj :P :CONST to 
match
+ * :CONST if :CONST is known to have the type :T
+ * 
+ */
+public class HasSelfVisitor extends AbstractInferVisitor {
+private static final Var TYPE_VAR = new Var("p", RDF.TYPE);
+
+/**
+ * Creates a new {@link HasSelfVisitor}, which is enabled by default.
+ * @param conf The {@link RdfCloudTripleStoreConfiguration}.
+ * @param inferenceEngine The InferenceEngine containing the relevant 
ontology.
+ */
+public HasSelfVisitor(final RdfCloudTripleStoreConfiguration conf, 
final InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = conf.hasSelf();
+}
+
+@Override
+protected void meetSP(final StatementPattern node) throws Exception {
+final URI pred = (URI) node.getPredicateVar().getValue();
+final Var obj = node.getObjectVar();
+//if originalSP like (?s rdf:type :C1):  require that C1 is 
defined, i.e. not a variable
+// node <- originalSP
+final StatementPattern clone = node.clone();
+if(pred.equals(RDF.TYPE) && obj.isConstant()) {
+//for property in getHasSelfImplyingType(C1):
+for (final URI property : 
inferenceEngine.getHasSelfImplyingType((URI) obj.getValue())) {
+//node <- InferUnion(node, StatementPattern(?s, property, 
?s)).
+final InferUnion union = new InferUnion(clone, new 
StatementPattern(clone.getSubjectVar(), new Var("p", property), 
clone.getSubjectVar()));
+//originalSP.replaceWith(node)
+node.replaceWith(union);
+}
+//else if originalSP like (s :p o):  where p is not a variable and 
at least one of s and o are variables
+} else if (node.getPredicateVar().isConstant()
+   && (!node.getSubjectVar().isConstant() ||
+   !node.getObjectVar().isConstant())) {
+//for type in getHasSelfImplyingProperty(p):
+for (final Resource type : 
inferenceEngine.getHasSelfImplyingProperty(pred)) {
+final Extension extension;
+if(obj.isConstant()) { // subject is the variable
+//Extension(StatementPattern(o, rdf:type, type), 
ExtensionElem(o, "s"))
+extension = new Extension(new StatementPattern(obj, 
TYPE_VAR, obj),
--- End diff --

Object of statement pattern should be a new variable set to `type` 
(mirroring the else clause below, which is correct)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134630981
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -1022,4 +1060,44 @@ public void setSchedule(boolean schedule) {
 }
 return implications;
 }
-}
+
+/**
+ * For a given type, return any properties and values such that 
owl:hasSelf restrictions on
+ * those properties could imply this type. No matter how many 
restrictions are returned, each
+ * one is considered individually sufficient: if a resource has the 
property and the value, then
+ * it belongs to the provided type. Takes type hierarchy into account, 
so the value may imply a
+ * subtype which in turn implies the provided type.
+ * @param type The type (URI or bnode) to check against the known 
restrictions
+ * @return For each relevant property, a set of values such that 
whenever a resource has that
+ *  value for that property, it is implied to belong to the type.
+ */
+public Set getHasSelfImplyingType(final Resource type){
+// return properties that imply this type if reflexive
+final Set properties = new HashSet<>();
+properties.addAll(hasSelfByType.get(type));
+//findParent gets all subclasses, add self.
+if (type instanceof URI) {
+for (final URI subtype : findParents(subClassOfGraph, (URI) 
type)) {
+properties.addAll(hasSelfByType.get(type));
+}
+}
+
+// make map hasSelfByType[]
+return properties;
+}
+
+/**
+ * For a given property, return any types and values such that some 
owl:hasSelf restriction
+ * states that members of the type are implied to have the associated 
specific value(s) for
+ * this property. Takes class hierarchy into account, which means one 
type may imply multiple
+ * values by way of supertypes with their own restrictions. Does not 
consider the property
+ * hierarchy, so only restrictions that directly reference the given 
property (using
+ * owl:onProperty) are relevant.
+ * @param property The property whose owl:hasSelf restrictions to 
return
+ * @return A mapping from type (URIs or bnodes) to the set of any 
values that belonging to that
+ *  type implies.
+ */
+public Set getHasSelfImplyingProperty(final URI property) { 
// return types that imply this property to be reflexive
--- End diff --

Edit doc to apply to hasSelf as opposed to hasValue -- in particular, 
`@return` and references to "specific value(s)"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134618623
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasSelfVisitor.java 
---
@@ -0,0 +1,81 @@
+package org.apache.rya.rdftriplestore.inference;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.Extension;
+import org.openrdf.query.algebra.ExtensionElem;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+
+/**
+ * Expands the query tree to account for any relevant has-self class 
expressions
+ * in the ontology known to the {@link InferenceEngine}.
+ *
+ * Only operates on {@link StatementPattern} nodes, and only those 
including a
+ * defined type or defined predicate which is relevant to a has-self 
expression
+ * in the ontology. When applicable, replaces the node with one or more
+ * {@link InferUnion}s, one of whose leaves is the original 
StatementPattern.
+ *
+ * A has-self restriction defines the set of resources that are connected 
to
+ * themselves by a property. If the ontology states that a type is a 
resource
+ * that has a self referencing property, then the inference engine should:
+ * 1. Rewrite queries of the from ?x rdf:type :T to find all resources
+ * matching ?x :P ?x (as well as anything explicitly stated to be of type 
:T)
+ * 
+ * 2. Rewrite queries of the from :CONST :P ?o or ?subj :P :CONST to 
match
+ * :CONST if :CONST is known to have the type :T
+ * 
+ */
+public class HasSelfVisitor extends AbstractInferVisitor {
+private static final Var TYPE_VAR = new Var("p", RDF.TYPE);
+
+/**
+ * Creates a new {@link HasSelfVisitor}, which is enabled by default.
+ * @param conf The {@link RdfCloudTripleStoreConfiguration}.
+ * @param inferenceEngine The InferenceEngine containing the relevant 
ontology.
+ */
+public HasSelfVisitor(final RdfCloudTripleStoreConfiguration conf, 
final InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = conf.hasSelf();
+}
+
+@Override
+protected void meetSP(final StatementPattern node) throws Exception {
+final URI pred = (URI) node.getPredicateVar().getValue();
+final Var obj = node.getObjectVar();
+//if originalSP like (?s rdf:type :C1):  require that C1 is 
defined, i.e. not a variable
+// node <- originalSP
+final StatementPattern clone = node.clone();
+if(pred.equals(RDF.TYPE) && obj.isConstant()) {
+//for property in getHasSelfImplyingType(C1):
+for (final URI property : 
inferenceEngine.getHasSelfImplyingType((URI) obj.getValue())) {
--- End diff --

Might want to check that obj.getValue() is a URI before casting. It'll 
probably always be one but I think it theoretically could be a Resource or 
Literal? (even if '?x rdf:type "string literal"' isn't going to be a very 
useful query)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134632048
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -1022,4 +1060,44 @@ public void setSchedule(boolean schedule) {
 }
 return implications;
 }
-}
+
+/**
+ * For a given type, return any properties and values such that 
owl:hasSelf restrictions on
+ * those properties could imply this type. No matter how many 
restrictions are returned, each
+ * one is considered individually sufficient: if a resource has the 
property and the value, then
+ * it belongs to the provided type. Takes type hierarchy into account, 
so the value may imply a
+ * subtype which in turn implies the provided type.
+ * @param type The type (URI or bnode) to check against the known 
restrictions
+ * @return For each relevant property, a set of values such that 
whenever a resource has that
+ *  value for that property, it is implied to belong to the type.
+ */
+public Set getHasSelfImplyingType(final Resource type){
+// return properties that imply this type if reflexive
+final Set properties = new HashSet<>();
+properties.addAll(hasSelfByType.get(type));
+//findParent gets all subclasses, add self.
+if (type instanceof URI) {
+for (final URI subtype : findParents(subClassOfGraph, (URI) 
type)) {
+properties.addAll(hasSelfByType.get(type));
+}
+}
+
+// make map hasSelfByType[]
+return properties;
+}
+
+/**
+ * For a given property, return any types and values such that some 
owl:hasSelf restriction
+ * states that members of the type are implied to have the associated 
specific value(s) for
+ * this property. Takes class hierarchy into account, which means one 
type may imply multiple
+ * values by way of supertypes with their own restrictions. Does not 
consider the property
+ * hierarchy, so only restrictions that directly reference the given 
property (using
+ * owl:onProperty) are relevant.
+ * @param property The property whose owl:hasSelf restrictions to 
return
+ * @return A mapping from type (URIs or bnodes) to the set of any 
values that belonging to that
+ *  type implies.
+ */
+public Set getHasSelfImplyingProperty(final URI property) { 
// return types that imply this property to be reflexive
+   return hasSelfByProperty.get(property);
--- End diff --

Looks like it doesn't take class hierarchy into account -- if there's a 
type T that represents a hasSelf condition on this property, and there's a type 
T2 which is a subclass of T, then only T will be added to the result set. 
Should either change the doc to reflect that, or loop through the types 
returned by hasSelfByProperty.get(property) and add their subtypes too. 
(Meaning T2 will be in the result set, since T2 implies T and therefore implies 
the hasSelf condition.) I'd lean toward the latter since we are already taking 
subclass/superclass relationships into account in the other hasSelf method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #209: Rya 296 hasSelf

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/209#discussion_r134632781
  
--- Diff: 
sail/src/test/java/org/apache/rya/rdftriplestore/inference/InferenceEngineTest.java
 ---
@@ -365,4 +365,20 @@ public void testUnionOf() throws Exception {
 Assert.assertEquals(expectedA, subClassesA);
 Assert.assertEquals(expectedB, subClassesB);
 }
+
+@Test
+public void testHasSelfGivenProperty() throws Exception {
--- End diff --

Could rename to just testHasSelf; was about to suggest another test before 
I realized this one tests both (given property, return types) and (given type, 
return properties)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r134537523
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -723,29 +724,38 @@ public void handleStatement(final Statement 
statement) throws RDFHandlerExceptio
 }
 }
 }
-
-final List typeStatements = new ArrayList<>();
-ryaDaoQueryWrapper.queryAll(type, OWL.INTERSECTIONOF, null, 
new RDFHandlerBase() {
-@Override
-public void handleStatement(final Statement statement) 
throws RDFHandlerException {
-typeStatements.add(statement);
-}
-});
+for (final Set intersection : intersectionList) {
+addIntersection(intersection, type);
+}
+}
+for (final Entry<Resource, List<Set>> entry : 
intersectionsProp.entrySet()) {
+final Resource type = entry.getKey();
+final List<Set> intersectionList = entry.getValue();
 
 final Set superClasses = getSuperClasses((URI) type);
+for (final URI superClass : superClasses) {
+// Add intersections to super classes if applicable.
+// IF:
+// :A intersectionOf[:B, :C]
+// AND
+// :A subclassOf :D
+// Then we can infer:
+// intersectionOf[:B, :C] subclassOf :D
+for (final Set intersection : intersectionList) {
+addIntersection(intersection, superClass);
+}
+}
+// Check if other keys have any of the same intersections and 
infer
+// the same subclass logic to them that we know from the 
current
+// type. Propagating up through all the superclasses.
 for (final Set intersection : intersectionList) {
-addIntersection(intersection, type);
-for (final URI superClass : superClasses) {
-// Add intersections to super classes if applicable.
-// IF:
-// :A intersectionOf[:B, :C]
-// AND
-// :A subclassOf :D
-// Then we can infer:
-// intersectionOf[:B, :C] subclassOf :D
-for (final Statement statement : typeStatements) {
-final Resource intersectionOfBnode = (Resource) 
statement.getObject();
-addSubClassOf(intersectionOfBnode, superClass);
+final Set otherKeys = 
Sets.newHashSet(intersectionsProp.keySet());
+otherKeys.remove(type);
+for (final Resource otherKey : otherKeys) {
+if 
(intersectionsProp.get(otherKey).contains(intersection)) {
+for (final URI superClass : superClasses) {
--- End diff --

I think if we get here we've found an equivalence between type and 
otherKey, not just a subclass relationship (if I'm following correctly, 
intersectionsProp has the direct definitions, as opposed to the indirect 
implications we're also going through, so at this point we have type and 
otherKey having the same definition).  If that's right, then instead of looping 
through the superclasses and adding them explicitly, we could do both 
"addSubClassOf(otherKey, type) ; addSubClassOf(type, otherKey)", and leave it 
up to whatever needs the superclass logic whether to traverse the graph 
recursively for the indirect superclasses.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r134081316
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -416,22 +425,131 @@ private void 
refreshHasValueRestrictions(Map<Resource, URI> restrictions) throws
 }
 }
 
-private static Vertex getVertex(Graph graph, Object id) {
-Iterator it = graph.vertices(id.toString());
+private void refreshIntersectionOf() throws QueryEvaluationException {
+final Map<Resource, List<Set>> intersectionsProp = new 
HashMap<>();
+
+// First query for all the owl:intersectionOf's.
+// If we have the following intersectionOf:
+// :A owl:intersectionOf[:B, :C]
+// It will be represented by triples following a pattern similar 
to:
+// <:A> owl:intersectionOf _:bnode1 .
+//  _:bnode1 rdf:first <:B> .
+//  _:bnode1 rdf:rest _:bnode2 .
+// _:bnode2 rdf:first <:C> .
+// _:bnode2 rdf:rest rdf:nil .
+ryaDaoQueryWrapper.queryAll(null, OWL.INTERSECTIONOF, null, new 
RyaDaoStatementIterHandler() {
+@Override
+public void handleStatementIter(final Statement st1) throws 
Exception {
+final Resource type = st1.getSubject();
+// head will point to a type that is part of the 
intersection.
+URI head = (URI) st1.getObject();
+if (!intersectionsProp.containsKey(type)) {
+intersectionsProp.put(type, new 
ArrayList<Set>());
+}
+final Set intersection = new HashSet<>();
+// Go through and find all bnodes that are part of the 
defined
+// intersection.
+while (!RDF.NIL.equals(head)) {
+// rdf.first will point to a type item that is in the
+// intersection.
+ryaDaoQueryWrapper.queryFirst(head, RDF.FIRST, null, 
new RyaDaoStatementIterHandler() {
+@Override
+public void handleStatementIter(final Statement 
st2) throws Exception{
+// The object found in the query represents a 
type
+// that should be included in the intersection.
+final URI obj2 = (URI) st2.getObject();
+intersection.add(obj2);
+}
+});
+final List headHolder = new ArrayList<>(1);
+// rdf.rest will point to the next bnode that's part 
of the
+// intersection.
+ryaDaoQueryWrapper.queryFirst(head, RDF.REST, null, 
new RyaDaoStatementIterHandler() {
+@Override
+public void handleStatementIter(final Statement 
st3) throws Exception {
+// This object is the next bnode head to look 
for.
+final URI obj3 = (URI) st3.getObject();
+headHolder.add(obj3);
+}
+});
+// As long as we get a new head there are more bnodes 
that
+// are part of the intersection. Keep going until we 
reach
+// rdf.nil.
+if (!headHolder.isEmpty()) {
+head = headHolder.get(0);
+} else {
+head = RDF.NIL;
+}
+}
+// Add this intersection for this type. There may be more
+// intersections for this type so each type has a list of
+// intersection sets.
+intersectionsProp.get(type).add(intersection);
+}
+});
+
+for (final Map.Entry<Resource, List<Set>> entry : 
intersectionsProp.entrySet()) {
+final Resource type = entry.getKey();
+final List<Set> intersectionList = entry.getValue();
+final Set otherTypes = new HashSet<>();
+// Combine all of a type's intersections together.
+for (final Set intersection : intersectionList) {
+otherTypes.addAll(intersection);
+}
+for (final Resource other : otherTypes) {
+// :A intersectionOf[:B, :C] implies that
+// :A subclassOf :B
+// :A su

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r134081075
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -621,22 +631,189 @@ private void 
refreshHasValueRestrictions(Map<Resource, URI> restrictions) throws
 }
 }
 
-private static Vertex getVertex(Graph graph, Object id) {
-Iterator it = graph.vertices(id.toString());
+private void refreshIntersectionOf() throws QueryEvaluationException {
+final Map<Resource, List<Set>> intersectionsProp = new 
HashMap<>();
+
+// First query for all the owl:intersectionOf's.
+// If we have the following intersectionOf:
+// :A owl:intersectionOf[:B, :C]
+// It will be represented by triples following a pattern similar 
to:
+// <:A> owl:intersectionOf _:bnode1 .
+//  _:bnode1 rdf:first <:B> .
+//  _:bnode1 rdf:rest _:bnode2 .
+// _:bnode2 rdf:first <:C> .
+// _:bnode2 rdf:rest rdf:nil .
+ryaDaoQueryWrapper.queryAll(null, OWL.INTERSECTIONOF, null, new 
RDFHandlerBase() {
+@Override
+public void handleStatement(final Statement statement) throws 
RDFHandlerException {
+final Resource type = statement.getSubject();
+// head will point to a type that is part of the 
intersection.
+final URI head = (URI) statement.getObject();
+if (!intersectionsProp.containsKey(type)) {
+intersectionsProp.put(type, new 
ArrayList<Set>());
+}
+
+// head should point to a list of items that forms the
+// intersection.
+try {
+final Set intersection = new 
LinkedHashSet<>(getList(head));
+if (!intersection.isEmpty()) {
+// Add this intersection for this type. There may 
be more
+// intersections for this type so each type has a 
list of
+// intersection sets.
+intersectionsProp.get(type).add(intersection);
+}
+} catch (final QueryEvaluationException e) {
+throw new RDFHandlerException("Error getting 
intersection list.", e);
+}
+}
+});
+
+intersections.clear();
+for (final Entry<Resource, List<Set>> entry : 
intersectionsProp.entrySet()) {
+final Resource type = entry.getKey();
+final List<Set> intersectionList = entry.getValue();
+final Set otherTypes = new HashSet<>();
+// Combine all of a type's intersections together.
+for (final Set intersection : intersectionList) {
+otherTypes.addAll(intersection);
+}
+for (final Resource other : otherTypes) {
+// :A intersectionOf[:B, :C] implies that
+// :A subclassOf :B
+// :A subclassOf :C
+// So add each type that's part of the intersection to the
+// subClassOf graph.
+addSubClassOf(type, other);
+for (final Set intersection : intersectionList) {
+if (!intersection.contains(other)) {
+addIntersection(intersection, other);
+}
+}
+}
+
+final List typeStatements = new ArrayList<>();
+ryaDaoQueryWrapper.queryAll(type, OWL.INTERSECTIONOF, null, 
new RDFHandlerBase() {
+@Override
+public void handleStatement(final Statement statement) 
throws RDFHandlerException {
+typeStatements.add(statement);
+}
+});
+
+final Set superClasses = getSuperClasses((URI) type);
+for (final Set intersection : intersectionList) {
+addIntersection(intersection, type);
+for (final URI superClass : superClasses) {
+// Add intersections to super classes if applicable.
+// IF:
+// :A intersectionOf[:B, :C]
+// AND
+// :A subclassOf :D
+// Then we can infer:
+// intersectionOf[:B, :C] subclassOf :D
+for (final Statement statement : typeStatements) {
   

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r133323192
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -467,126 +585,180 @@ private static void addParents(Vertex v, Set 
parents) {
 });
 }
 
-public boolean isSymmetricProperty(URI prop) {
+public boolean isSymmetricProperty(final URI prop) {
 return (symmetricPropertySet != null) && 
symmetricPropertySet.contains(prop);
 }
 
-public URI findInverseOf(URI prop) {
+public URI findInverseOf(final URI prop) {
 return (inverseOfMap != null) ? inverseOfMap.get(prop) : (null);
 }
 
-public boolean isTransitiveProperty(URI prop) {
+public boolean isTransitiveProperty(final URI prop) {
 return (transitivePropertySet != null) && 
transitivePropertySet.contains(prop);
 }
 
 /**
  * TODO: This chaining can be slow at query execution. the other 
option is to perform this in the query itself, but that will be constrained to 
how many levels we decide to go
  */
-public Set findTransitiveProperty(Resource subj, URI prop, 
Value obj, Resource... contxts) throws InferenceEngineException {
+public Set findTransitiveProperty(final Resource subj, 
final URI prop, final Value obj, final Resource... contxts) throws 
InferenceEngineException {
 if (transitivePropertySet.contains(prop)) {
-Set sts = new HashSet();
-boolean goUp = subj == null;
+final Set sts = new HashSet<>();
+final boolean goUp = subj == null;
 chainTransitiveProperty(subj, prop, obj, (goUp) ? (obj) : 
(subj), sts, goUp, contxts);
 return sts;
-} else
+} else {
 return null;
+}
 }
 
 /**
  * TODO: This chaining can be slow at query execution. the other 
option is to perform this in the query itself, but that will be constrained to 
how many levels we decide to go
  */
-public Set findSameAs(Resource value, Resource... contxts) 
throws InferenceEngineException{
-   Set sameAs = new HashSet();
-   sameAs.add(value);
-   findSameAsChaining(value, sameAs, contxts);
-   return sameAs;
+public Set findSameAs(final Resource value, final 
Resource... contxts) throws InferenceEngineException{
+final Set sameAs = new HashSet();
+sameAs.add(value);
+findSameAsChaining(value, sameAs, contxts);
+return sameAs;
+}
+
+public CloseableIteration<Statement, QueryEvaluationException> 
queryDao(final Resource subject, final URI predicate, final Value object, final 
Resource... contexts) throws QueryEvaluationException {
+return RyaDAOHelper.query(ryaDAO, subject, predicate, object, 
conf, contexts);
 }
 
 /**
  * TODO: This chaining can be slow at query execution. the other 
option is to perform this in the query itself, but that will be constrained to 
how many levels we decide to go
  */
-public void findSameAsChaining(Resource subj, Set 
currentSameAs, Resource[] contxts) throws InferenceEngineException{
+public void findSameAsChaining(final Resource subj, final 
Set currentSameAs, final Resource[] contxts) throws 
InferenceEngineException{
+CloseableIteration<Statement, QueryEvaluationException> subjIter = 
null;
+CloseableIteration<Statement, QueryEvaluationException> objIter = 
null;
 try {
-   CloseableIteration<Statement, QueryEvaluationException> 
subjIter = RyaDAOHelper.query(ryaDAO, subj, OWL.SAMEAS, null, conf, contxts);
-   while (subjIter.hasNext()){
-   Statement st = subjIter.next();
-   if (!currentSameAs.contains(st.getObject())){
-   Resource castedObj = (Resource) 
st.getObject();
-   currentSameAs.add(castedObj);
-   findSameAsChaining(castedObj, 
currentSameAs, contxts);
-   }
-   }
-   subjIter.close();
-   CloseableIteration<Statement, QueryEvaluationException> 
objIter = RyaDAOHelper.query(ryaDAO, null, OWL.SAMEAS, subj, conf, contxts);
-   while (objIter.hasNext()){
-   Statement st = objIter.next();
-   if (!currentSameAs.contains(st.getSubject())){
-   Resource sameAsSubj = st.getSubject();
-   

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r133319932
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -416,22 +425,131 @@ private void 
refreshHasValueRestrictions(Map<Resource, URI> restrictions) throws
 }
 }
 
-private static Vertex getVertex(Graph graph, Object id) {
-Iterator it = graph.vertices(id.toString());
+private void refreshIntersectionOf() throws QueryEvaluationException {
+final Map<Resource, List<Set>> intersectionsProp = new 
HashMap<>();
+
+// First query for all the owl:intersectionOf's.
+// If we have the following intersectionOf:
+// :A owl:intersectionOf[:B, :C]
+// It will be represented by triples following a pattern similar 
to:
+// <:A> owl:intersectionOf _:bnode1 .
+//  _:bnode1 rdf:first <:B> .
+//  _:bnode1 rdf:rest _:bnode2 .
+// _:bnode2 rdf:first <:C> .
+// _:bnode2 rdf:rest rdf:nil .
+ryaDaoQueryWrapper.queryAll(null, OWL.INTERSECTIONOF, null, new 
RyaDaoStatementIterHandler() {
+@Override
+public void handleStatementIter(final Statement st1) throws 
Exception {
+final Resource type = st1.getSubject();
+// head will point to a type that is part of the 
intersection.
+URI head = (URI) st1.getObject();
+if (!intersectionsProp.containsKey(type)) {
+intersectionsProp.put(type, new 
ArrayList<Set>());
+}
+final Set intersection = new HashSet<>();
+// Go through and find all bnodes that are part of the 
defined
+// intersection.
+while (!RDF.NIL.equals(head)) {
+// rdf.first will point to a type item that is in the
+// intersection.
+ryaDaoQueryWrapper.queryFirst(head, RDF.FIRST, null, 
new RyaDaoStatementIterHandler() {
+@Override
+public void handleStatementIter(final Statement 
st2) throws Exception{
+// The object found in the query represents a 
type
+// that should be included in the intersection.
+final URI obj2 = (URI) st2.getObject();
+intersection.add(obj2);
+}
+});
+final List headHolder = new ArrayList<>(1);
+// rdf.rest will point to the next bnode that's part 
of the
+// intersection.
+ryaDaoQueryWrapper.queryFirst(head, RDF.REST, null, 
new RyaDaoStatementIterHandler() {
+@Override
+public void handleStatementIter(final Statement 
st3) throws Exception {
+// This object is the next bnode head to look 
for.
+final URI obj3 = (URI) st3.getObject();
+headHolder.add(obj3);
+}
+});
+// As long as we get a new head there are more bnodes 
that
+// are part of the intersection. Keep going until we 
reach
+// rdf.nil.
+if (!headHolder.isEmpty()) {
+head = headHolder.get(0);
+} else {
+head = RDF.NIL;
+}
+}
+// Add this intersection for this type. There may be more
+// intersections for this type so each type has a list of
+// intersection sets.
+intersectionsProp.get(type).add(intersection);
+}
+});
+
+for (final Map.Entry<Resource, List<Set>> entry : 
intersectionsProp.entrySet()) {
+final Resource type = entry.getKey();
+final List<Set> intersectionList = entry.getValue();
+final Set otherTypes = new HashSet<>();
+// Combine all of a type's intersections together.
+for (final Set intersection : intersectionList) {
+otherTypes.addAll(intersection);
+}
+for (final Resource other : otherTypes) {
+// :A intersectionOf[:B, :C] implies that
+// :A subclassOf :B
+// :A su

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r133311778
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -220,163 +229,163 @@ public void refreshGraph() throws 
InferenceEngineException {
 }
 }
 inverseOfMap = invProp;
-
-ValueFactory vf = ValueFactoryImpl.getInstance();
-iter = RyaDAOHelper.query(ryaDAO, null, 
-   
vf.createURI("http://www.w3.org/2002/07/owl#propertyChainAxiom;),
-   null, conf);
-Map<URI,URI> propertyChainPropertiesToBNodes = new 
HashMap<URI, URI>();
-propertyChainPropertyToChain = new HashMap<URI, List>();
+
+final ValueFactory vf = ValueFactoryImpl.getInstance();
+iter = RyaDAOHelper.query(ryaDAO, null,
+
vf.createURI("http://www.w3.org/2002/07/owl#propertyChainAxiom;),
+null, conf);
+final Map<URI,URI> propertyChainPropertiesToBNodes = new 
HashMap<>();
+propertyChainPropertyToChain = new HashMap<>();
 try {
-   while (iter.hasNext()){
-   Statement st = iter.next();
-   
propertyChainPropertiesToBNodes.put((URI)st.getSubject(), (URI)st.getObject());
-   }
+while (iter.hasNext()){
+final Statement st = iter.next();
+
propertyChainPropertiesToBNodes.put((URI)st.getSubject(), (URI)st.getObject());
+}
 } finally {
 if (iter != null) {
 iter.close();
 }
 }
 // now for each property chain bNode, get the indexed list of 
properties associated with that chain
-for (URI propertyChainProperty : 
propertyChainPropertiesToBNodes.keySet()){
-   URI bNode = 
propertyChainPropertiesToBNodes.get(propertyChainProperty);
-   // query for the list of indexed properties
-   iter = RyaDAOHelper.query(ryaDAO, bNode, 
vf.createURI("http://www.w3.org/2000/10/swap/list#index;),
-   null, conf);
-   TreeMap<Integer, URI> orderedProperties = new 
TreeMap<Integer, URI>();
-   // TODO refactor this.  Wish I could execute sparql
-   try {
-   while (iter.hasNext()){
- Statement st = iter.next();
- String indexedElement = 
st.getObject().stringValue();
- System.out.println(indexedElement);
- CloseableIteration<Statement, 
QueryEvaluationException>  iter2 = RyaDAOHelper.query(ryaDAO, 
vf.createURI(st.getObject().stringValue()), RDF.FIRST,
-   null, conf);
- String integerValue = "";
- Value anonPropNode = null;
- Value propURI = null;
- if (iter2 != null){
- while (iter2.hasNext()){
- Statement iter2Statement = 
iter2.next();
- integerValue = 
iter2Statement.getObject().stringValue();
- break;
- }
- iter2.close();
- }
- iter2 = RyaDAOHelper.query(ryaDAO, 
vf.createURI(st.getObject().stringValue()), RDF.REST,
-   null, conf);
- if (iter2 != null){
- while (iter2.hasNext()){
- Statement iter2Statement = 
iter2.next();
- anonPropNode = 
iter2Statement.getObject();
- break;
- }
- iter2.close();
- if (anonPropNode != null){
- iter2 = 
RyaDAOHelper.query(ryaDAO, vf.createURI(anonPropNode.stringValue()), RDF.FIRST,
-   null, conf);
- while (iter2.hasNext()){
-

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r133310947
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -220,163 +229,163 @@ public void refreshGraph() throws 
InferenceEngineException {
 }
 }
 inverseOfMap = invProp;
-
-ValueFactory vf = ValueFactoryImpl.getInstance();
-iter = RyaDAOHelper.query(ryaDAO, null, 
-   
vf.createURI("http://www.w3.org/2002/07/owl#propertyChainAxiom;),
-   null, conf);
-Map<URI,URI> propertyChainPropertiesToBNodes = new 
HashMap<URI, URI>();
-propertyChainPropertyToChain = new HashMap<URI, List>();
+
+final ValueFactory vf = ValueFactoryImpl.getInstance();
+iter = RyaDAOHelper.query(ryaDAO, null,
+
vf.createURI("http://www.w3.org/2002/07/owl#propertyChainAxiom;),
+null, conf);
+final Map<URI,URI> propertyChainPropertiesToBNodes = new 
HashMap<>();
+propertyChainPropertyToChain = new HashMap<>();
 try {
-   while (iter.hasNext()){
-   Statement st = iter.next();
-   
propertyChainPropertiesToBNodes.put((URI)st.getSubject(), (URI)st.getObject());
-   }
+while (iter.hasNext()){
+final Statement st = iter.next();
+
propertyChainPropertiesToBNodes.put((URI)st.getSubject(), (URI)st.getObject());
+}
 } finally {
 if (iter != null) {
 iter.close();
 }
 }
 // now for each property chain bNode, get the indexed list of 
properties associated with that chain
-for (URI propertyChainProperty : 
propertyChainPropertiesToBNodes.keySet()){
-   URI bNode = 
propertyChainPropertiesToBNodes.get(propertyChainProperty);
-   // query for the list of indexed properties
-   iter = RyaDAOHelper.query(ryaDAO, bNode, 
vf.createURI("http://www.w3.org/2000/10/swap/list#index;),
-   null, conf);
-   TreeMap<Integer, URI> orderedProperties = new 
TreeMap<Integer, URI>();
-   // TODO refactor this.  Wish I could execute sparql
-   try {
-   while (iter.hasNext()){
- Statement st = iter.next();
- String indexedElement = 
st.getObject().stringValue();
- System.out.println(indexedElement);
- CloseableIteration<Statement, 
QueryEvaluationException>  iter2 = RyaDAOHelper.query(ryaDAO, 
vf.createURI(st.getObject().stringValue()), RDF.FIRST,
-   null, conf);
- String integerValue = "";
- Value anonPropNode = null;
- Value propURI = null;
- if (iter2 != null){
- while (iter2.hasNext()){
- Statement iter2Statement = 
iter2.next();
- integerValue = 
iter2Statement.getObject().stringValue();
- break;
- }
- iter2.close();
- }
- iter2 = RyaDAOHelper.query(ryaDAO, 
vf.createURI(st.getObject().stringValue()), RDF.REST,
-   null, conf);
- if (iter2 != null){
- while (iter2.hasNext()){
- Statement iter2Statement = 
iter2.next();
- anonPropNode = 
iter2Statement.getObject();
- break;
- }
- iter2.close();
- if (anonPropNode != null){
- iter2 = 
RyaDAOHelper.query(ryaDAO, vf.createURI(anonPropNode.stringValue()), RDF.FIRST,
-   null, conf);
- while (iter2.hasNext()){
-

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r133329139
  
--- Diff: 
common/rya.api/src/main/java/org/apache/rya/api/persist/utils/RyaDaoQueryWrapper.java
 ---
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.persist.utils;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.api.domain.RyaStatement;
+import org.apache.rya.api.persist.RyaDAO;
+import org.apache.rya.api.resolver.RyaToRdfConversions;
+import org.openrdf.model.Resource;
+import org.openrdf.model.Statement;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.query.QueryEvaluationException;
+
+import info.aduna.iteration.CloseableIteration;
+
+/**
+ * Wraps Rya DAO queries into a simpler interface that just passes in the
+ * statement to query for and a handler for dealing with each statement in 
the
+ * query result. This handles iterating over the query, throwing any 
exceptions,
+ * and closing the query iterator when done. The same wrapper can be 
re-used
+ * for multiple queries.
+ */
+public class RyaDaoQueryWrapper {
+private final RyaDAO ryaDao;
+private final RdfCloudTripleStoreConfiguration conf;
+
+/**
+ * Creates a new instance of {@link RyaDaoQueryWrapper}.
+ * @param ryaDao the {@link RyaDAO}. (not {@code null})
+ * @param conf the {@link RdfCloudTripleStoreConfiguration}.
+ * (not {@code null})
+ */
+public RyaDaoQueryWrapper(final RyaDAO ryaDao, final 
RdfCloudTripleStoreConfiguration conf) {
+this.ryaDao = checkNotNull(ryaDao);
+this.conf = checkNotNull(conf);
+}
+
+/**
+ * Creates a new instance of {@link RyaDaoQueryWrapper}.
+ * @param ryaDao the {@link RyaDAO}. (not {@code null})
+ */
+public RyaDaoQueryWrapper(final RyaDAO ryaDao) {
+this(checkNotNull(ryaDao), ryaDao.getConf());
+}
+
+/**
+ * Handles all results of a query. Closes the query iterator when done.
+ * @param subject the subject {@link Resource} to query for.
+ * @param predicate the predicate {@link URI} to query for.
+ * @param object the object {@link Value} to query for.
+ * @param ryaDaoStatementIterHandler the {@link 
RyaDaoStatementIterHandler}
+ * to use for handling each statement returned. (not {@code null})
+ * @param contexts the context {@link Resource}s to query for.
+ * @throws QueryEvaluationException
+ */
+public void queryAll(final Resource subject, final URI predicate, 
final Value object, final RyaDaoStatementIterHandler 
ryaDaoStatementIterHandler, final Resource... contexts) throws 
QueryEvaluationException {
--- End diff --

Would it make sense to use the org.openrdf.rio.RDFHandler interface (and 
RDFHandlerBase) instead of a new handler class? That would let someone directly 
pass in something like an RDFWriter, if that were ever useful for something.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r133312263
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -220,163 +229,163 @@ public void refreshGraph() throws 
InferenceEngineException {
 }
 }
 inverseOfMap = invProp;
-
-ValueFactory vf = ValueFactoryImpl.getInstance();
-iter = RyaDAOHelper.query(ryaDAO, null, 
-   
vf.createURI("http://www.w3.org/2002/07/owl#propertyChainAxiom;),
-   null, conf);
-Map<URI,URI> propertyChainPropertiesToBNodes = new 
HashMap<URI, URI>();
-propertyChainPropertyToChain = new HashMap<URI, List>();
+
+final ValueFactory vf = ValueFactoryImpl.getInstance();
+iter = RyaDAOHelper.query(ryaDAO, null,
+
vf.createURI("http://www.w3.org/2002/07/owl#propertyChainAxiom;),
+null, conf);
+final Map<URI,URI> propertyChainPropertiesToBNodes = new 
HashMap<>();
+propertyChainPropertyToChain = new HashMap<>();
 try {
-   while (iter.hasNext()){
-   Statement st = iter.next();
-   
propertyChainPropertiesToBNodes.put((URI)st.getSubject(), (URI)st.getObject());
-   }
+while (iter.hasNext()){
+final Statement st = iter.next();
+
propertyChainPropertiesToBNodes.put((URI)st.getSubject(), (URI)st.getObject());
+}
 } finally {
 if (iter != null) {
 iter.close();
 }
 }
 // now for each property chain bNode, get the indexed list of 
properties associated with that chain
-for (URI propertyChainProperty : 
propertyChainPropertiesToBNodes.keySet()){
-   URI bNode = 
propertyChainPropertiesToBNodes.get(propertyChainProperty);
-   // query for the list of indexed properties
-   iter = RyaDAOHelper.query(ryaDAO, bNode, 
vf.createURI("http://www.w3.org/2000/10/swap/list#index;),
-   null, conf);
-   TreeMap<Integer, URI> orderedProperties = new 
TreeMap<Integer, URI>();
-   // TODO refactor this.  Wish I could execute sparql
-   try {
-   while (iter.hasNext()){
- Statement st = iter.next();
- String indexedElement = 
st.getObject().stringValue();
- System.out.println(indexedElement);
- CloseableIteration<Statement, 
QueryEvaluationException>  iter2 = RyaDAOHelper.query(ryaDAO, 
vf.createURI(st.getObject().stringValue()), RDF.FIRST,
-   null, conf);
- String integerValue = "";
- Value anonPropNode = null;
- Value propURI = null;
- if (iter2 != null){
- while (iter2.hasNext()){
- Statement iter2Statement = 
iter2.next();
- integerValue = 
iter2Statement.getObject().stringValue();
- break;
- }
- iter2.close();
- }
- iter2 = RyaDAOHelper.query(ryaDAO, 
vf.createURI(st.getObject().stringValue()), RDF.REST,
-   null, conf);
- if (iter2 != null){
- while (iter2.hasNext()){
- Statement iter2Statement = 
iter2.next();
- anonPropNode = 
iter2Statement.getObject();
- break;
- }
- iter2.close();
- if (anonPropNode != null){
- iter2 = 
RyaDAOHelper.query(ryaDAO, vf.createURI(anonPropNode.stringValue()), RDF.FIRST,
-   null, conf);
- while (iter2.hasNext()){
-

[GitHub] incubator-rya pull request #206: RYA-292 Added owl:intersectionOf inference.

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/206#discussion_r133321220
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -416,22 +425,131 @@ private void 
refreshHasValueRestrictions(Map<Resource, URI> restrictions) throws
 }
 }
 
-private static Vertex getVertex(Graph graph, Object id) {
-Iterator it = graph.vertices(id.toString());
+private void refreshIntersectionOf() throws QueryEvaluationException {
+final Map<Resource, List<Set>> intersectionsProp = new 
HashMap<>();
+
+// First query for all the owl:intersectionOf's.
+// If we have the following intersectionOf:
+// :A owl:intersectionOf[:B, :C]
+// It will be represented by triples following a pattern similar 
to:
+// <:A> owl:intersectionOf _:bnode1 .
+//  _:bnode1 rdf:first <:B> .
+//  _:bnode1 rdf:rest _:bnode2 .
+// _:bnode2 rdf:first <:C> .
+// _:bnode2 rdf:rest rdf:nil .
+ryaDaoQueryWrapper.queryAll(null, OWL.INTERSECTIONOF, null, new 
RyaDaoStatementIterHandler() {
+@Override
+public void handleStatementIter(final Statement st1) throws 
Exception {
+final Resource type = st1.getSubject();
+// head will point to a type that is part of the 
intersection.
+URI head = (URI) st1.getObject();
+if (!intersectionsProp.containsKey(type)) {
+intersectionsProp.put(type, new 
ArrayList<Set>());
+}
+final Set intersection = new HashSet<>();
+// Go through and find all bnodes that are part of the 
defined
+// intersection.
+while (!RDF.NIL.equals(head)) {
+// rdf.first will point to a type item that is in the
+// intersection.
+ryaDaoQueryWrapper.queryFirst(head, RDF.FIRST, null, 
new RyaDaoStatementIterHandler() {
+@Override
+public void handleStatementIter(final Statement 
st2) throws Exception{
+// The object found in the query represents a 
type
+// that should be included in the intersection.
+final URI obj2 = (URI) st2.getObject();
+intersection.add(obj2);
+}
+});
+final List headHolder = new ArrayList<>(1);
+// rdf.rest will point to the next bnode that's part 
of the
+// intersection.
+ryaDaoQueryWrapper.queryFirst(head, RDF.REST, null, 
new RyaDaoStatementIterHandler() {
+@Override
+public void handleStatementIter(final Statement 
st3) throws Exception {
+// This object is the next bnode head to look 
for.
+final URI obj3 = (URI) st3.getObject();
+headHolder.add(obj3);
+}
+});
+// As long as we get a new head there are more bnodes 
that
+// are part of the intersection. Keep going until we 
reach
+// rdf.nil.
+if (!headHolder.isEmpty()) {
+head = headHolder.get(0);
+} else {
+head = RDF.NIL;
+}
+}
+// Add this intersection for this type. There may be more
+// intersections for this type so each type has a list of
+// intersection sets.
+intersectionsProp.get(type).add(intersection);
+}
+});
+
+for (final Map.Entry<Resource, List<Set>> entry : 
intersectionsProp.entrySet()) {
+final Resource type = entry.getKey();
+final List<Set> intersectionList = entry.getValue();
+final Set otherTypes = new HashSet<>();
+// Combine all of a type's intersections together.
+for (final Set intersection : intersectionList) {
+otherTypes.addAll(intersection);
+}
+for (final Resource other : otherTypes) {
+// :A intersectionOf[:B, :C] implies that
+// :A subclassOf :B
+// :A su

[GitHub] incubator-rya issue #201: RYA-295 owl:allValuesFrom inference

[jessehatfield]

Github user jessehatfield commented on the issue:

https://github.com/apache/incubator-rya/pull/201
  
Fixed a typo in a log message and a redundant line in the example; updated 
with latest changes to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya issue #197: RYA-298, RYA-299 Domain/range inference.

[jessehatfield]

Github user jessehatfield commented on the issue:

https://github.com/apache/incubator-rya/pull/197
  
Updated. At some point it would be nice to make a pass over the inference 
engine for thread-safety across both the old and new logic, maybe once the 
currently-pending extensions are added.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #180: RYA-293 Added owl:unionOf inference

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/180#discussion_r132230928
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -142,6 +143,53 @@ public void refreshGraph() throws 
InferenceEngineException {
 }
 }
 
+// Add unions to the subclass graph: if c owl:unionOf LIST(c1, 
c2, ... cn), then any
+// instances of c1, c2, ... or cn are also instances of c, 
meaning c is a superclass
+// of all the rest.
+// (In principle, an instance of c is likewise implied to be 
at least one of the other
+// types, but this fact is ignored for now to avoid 
nondeterministic reasoning.)
+iter = RyaDAOHelper.query(ryaDAO, null, OWL.UNIONOF, null, 
conf);
+try {
+while (iter.hasNext()) {
+Statement st = iter.next();
+Value unionType = st.getSubject();
+// Traverse the list of types constituting the union
+Value current = st.getObject();
+while (current instanceof Resource && 
!RDF.NIL.equals(current)) {
+Resource listNode = (Resource) current;
+CloseableIteration<Statement, 
QueryEvaluationException> listIter = RyaDAOHelper.query(ryaDAO,
+listNode, RDF.FIRST, null, conf);
+try {
+if (listIter.hasNext()) {
+Statement firstStatement = listIter.next();
+if (firstStatement.getObject() instanceof 
Resource) {
+Resource subclass = (Resource) 
firstStatement.getObject();
+Statement subclassStatement = 
vf.createStatement(subclass, RDFS.SUBCLASSOF, unionType);
+addStatementEdge(graph, 
RDFS.SUBCLASSOF.stringValue(), subclassStatement);
+}
+}
+} finally {
+listIter.close();
+}
+listIter = RyaDAOHelper.query(ryaDAO, listNode, 
RDF.REST, null, conf);
+try {
+if (listIter.hasNext()) {
+current = listIter.next().getObject();
--- End diff --

Yep, a union is given as a linked list so we just walk down adding subclass 
statements until we get to a node with no rdf:rest or with rdf:rest equal to 
rdf:nil. If the list is poorly-formed or someone tries to express the union 
using the wrong collection type ([describes unionOf 
expression](https://www.w3.org/TR/owl2-rdf-based-semantics/#Semantic_Conditions_for_Boolean_Connectives)
 | [gives interpretation of 
sequence](https://www.w3.org/TR/owl2-rdf-based-semantics/#Semantic_Conditions)) 
then it won't work. In most cases that just means the intended union isn't 
fully represented, but I suppose if it were somehow a cyclical list, we'd end 
up in an infinite loop.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya issue #180: RYA-293 Added owl:unionOf inference

[jessehatfield]

Github user jessehatfield commented on the issue:

https://github.com/apache/incubator-rya/pull/180
  
So this is a case where a few different schema terms (rdfs:subClassOf, 
owl:unionOf, and eventually owl:equivalentClass) end up being represented by 
just one term internally (subclass/superclass relationships). We could 
plausibly organize by either; my intuition is to use the internal 
representation, since the internal graph being complete and accurate can matter 
to the other rules. That would mean pulling the logic introduced here, plus the 
logic introduced in [https://github.com/apache/incubator-rya/pull/184](PR 184) 
, plus the logic for rdfs:subClassOf (which 184 incidentally simplifies to a 
method call anyway) into some "refreshSubClassGraph" method. Thoughts on that 
approach? Would probably do the same with subPropertyOfGraph for consistency 
(though it's simpler because there's no equivalent to union).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #201: RYA-295 owl:allValuesFrom inference

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/201

RYA-295 owl:allValuesFrom inference


## Description
Inference applies owl:allValuesFrom semantics for queries including 
statement patterns of the form "?x rdf:type :DefinedClass".

An owl:allValuesFrom property restriction is a universal class expression: 
it defines type T1 to be the set of individuals such that, for a given property 
p and type T2, all values of that property (i.e. all objects of triples where 
the predicate is p and the subject belongs to T1) have type T2. Therefore, if 
an individual is known to belong to the universal class expression (T1), then 
it can be inferred that all of its values for the property belong to the value 
type (T2). This is similar to rdfs:range except that it only applies when the 
subject of the triple belongs to the appropriate class expression.

InferenceEngine, at refresh time, stores information about 
owl:allValuesFrom restrictions (universal class expressions). These definitions 
can then be accessed by the value type: getAllValuesFromByValueType() returns every (restriction type, property) pair.

AllValuesFromVisitor processes statement patterns of the form "?x rdf:type 
:T2", and if :T2 is the value type for any owl:allValuesFrom restriction 
according to the inference engine, it replaces the statement pattern with a 
union: of 1) that same statement pattern (in case the type is explicitly 
asserted or can be inferred by some other rule); and 2) a subquery of the form 
"?y \:p ?x . ?y rdf:type :T1" for the appropriate (restriction type, property) 
pair.

RdfCloudTripleStoreConnection calls the visitor along with the other 
inference logic. Because the original statement pattern is preserved as one 
branch of the union, other visitors can still apply if there are other ways to 
derive the type.

Added a simple example of a query that relies on this inference to 
MongoRyaDirectExample.

### Tests
Unit test to verify that InferenceEngine stores and returns the schema; 
unit test to verify that AllValuesFromVisitor rewrites queries of the proper 
form; integration test to verify correct results for sample 
ontology+instances+query relying on allValuesFrom inference.

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-295)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
@ejwhite922 
@isper3at 
@meiercaleb 
@pujav65 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-295-allValuesFrom-inference

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #201


commit 55eae37e2f4aaef2f8cd41c735241c2ae85b88af
Author: Jesse Hatfield <jesse.hatfi...@parsons.com>
Date:   2017-08-08T19:58:10Z

RYA-295 owl:allValuesFrom inference




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #197: RYA-298, RYA-299 Domain/range inference.

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/197

RYA-298, RYA-299 Domain/range inference.


## Description
InferenceEngine: Added a method to query for and infer domain/range schema, 
incorporating subclass, subproperty, and inverse property information to 
compute the closure. Extended graph traversal methods to allow traversal in 
each direction (findParents has the same behavior as before, but is joined by 
findChildren, both of which call findConnections with a direction parameter).

DomainRangeVisitor: New class rewrites queries for members of a specific 
defined type to check for domain and range implications as well. Applies to 
StatementPatterns of the form , where Class is not a variable, and 
produces (if domain and/or range exist for that type) a union which includes 
the original pattern as well.

RdfCloudTripleStoreConnection: Call the DomainRangeVisitor if inference is 
enabled. Called at the beginning of the sequence of visitors because the 
original statement is preserved as one branch of the union, so this won't stop 
other inference rules from being applied in that branch. However, as with the 
other visitors, the inferred branches of the unions won't be expanded. This 
means we won't infer the type by applying multiple rules at once (regardless of 
order of visitor calls), but we will have a union of alternative ways of 
inferring the type.

### Tests
Added tests to: Verify that InferenceEngine loads and infers expected 
schema (taking into account class hierarchy, property graph, and inverse 
properties); verify that the visitor produces the expected query tree; and 
verify that a type query returns the expected results given an ontology.

### Links
[RYA-298](https://issues.apache.org/jira/browse/RYA-298)
[RYA-299](https://issues.apache.org/jira/browse/RYA-299)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
@meiercaleb 
@ejwhite922 
@pujav65
@amihalik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-298-299-domain-range-inference

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #197


commit 36d8f752da4a3c5e8d7ebbbcb790ec0117c790ea
Author: Jesse Hatfield <jesse.hatfi...@parsons.com>
Date:   2017-08-04T17:05:45Z

RYA-298, RYA-299 Domain/range inference.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #184: RYA-297 Added owl:equivalentClass inference

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/184

RYA-297 Added owl:equivalentClass inference

If A and B are equivalent classes, then A is a subclass of B and B is a 
subclass of A. Handled the same way as owl:equivalentProperty.


## Description
Modified InferenceEngine to add equivalentClass information to the subclass 
graph. (Semantics: for two classes to be equivalent means that they represent 
the same set, while for one class to be a subclass of another means that it 
represents a subset. Therefore to say that two classes are equivalent is 
identical to saying that each is a subclass of the other.) Pulled the query 
logic into a helper function because the same basic pattern is used for 
subclass, subproperty, equivalent property, and now equivalent class relations.

Because subClassOf inference is already supported, this means no new 
visitor is required.

Slightly edited one line in SubClassOfVisitor to avoid adding the same 
subclass relation to the FixedStatementPattern twice, which would otherwise be 
possible: equivalent classes are mutual subclasses of each other, creating a 
cycle in the subclass graph. and allowing findParents to return a set 
containing the original node. Therefore, instead of always adding a statement 
corresponding to the original type, just add the original type to the set 
returned by findParents, which will do nothing if it was already included. This 
is identical to the logic in SubPropertyOfVisitor.

### Tests
Added two tests to InferenceEngineTest, one for classes and one for 
properties, to verify that the inference engine builds the expected 
subclass/subproperty graph given an ontology involving sub- and equivalent- 
relations. Added a test to InferenceIT that inserts schema and instance data, 
then runs a query that depends on the application of subclass and equivalent 
class inference.

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-297)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
@meiercaleb 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-297-equivalentClass-inference

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/184.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #184


commit df49b04386c6e0ccb92757f31f1865c913eb85d3
Author: Jesse Hatfield <jesse.hatfi...@parsons.com>
Date:   2017-07-25T21:08:31Z

RYA-297 Added owl:equivalentClass inference

If A and B are equivalent classes, then A is a subclass of B and B is a 
subclass of A. Handled the same way as owl:equivalentProperty.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #174: RYA-291 Added owl:hasValue inference

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/174#discussion_r127121558
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasValueVisitor.java 
---
@@ -0,0 +1,96 @@
+package org.apache.rya.rdftriplestore.inference;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.api.utils.NullableStatementImpl;
+import org.apache.rya.rdftriplestore.utils.FixedStatementPattern;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.OWL;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.Var;
+
+public class HasValueVisitor extends AbstractInferVisitor {
+public HasValueVisitor(RdfCloudTripleStoreConfiguration conf, 
InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = true;
+}
+
+@Override
+protected void meetSP(StatementPattern node) throws Exception {
+final Var subjVar = node.getSubjectVar();
+final Var predVar = node.getPredicateVar();
+final Var objVar = node.getObjectVar();
+// We can reason over two types of statement patterns:
+// { ?var rdf:type :Restriction } and { ?var :property ?value }
+// Both require defined predicate
--- End diff --

I wouldn't describe it as an imposition; the semantics of the hasValue 
expression are: "If you have p1=v1, then you're an A (and vice versa)". So if 
the query is "Get all the A's," then we want to return anything with p1=v1 
(objVar here is A, and we ask the inference engine for any predicate/value 
pairs that entail membership in A). And the semantics of the subclass 
relationship are: "If you're an A, then you're also a B," so if the query is 
instead "Get all the B's," then we again want to return anything with p1=v1, in 
addition to anything that turns out to be a B for some other reason (objVar 
here is B, and we ask the inference engine for any predicate/value pairs that 
entail membership in B, including those that entail membership in A because 
membership in A entails membership in B).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #174: RYA-291 Added owl:hasValue inference

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/174#discussion_r127032327
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -583,4 +625,69 @@ public boolean isSchedule() {
 public void setSchedule(boolean schedule) {
 this.schedule = schedule;
 }
+
+/**
+ * For a given type, return any properties and values such that 
owl:hasValue restrictions on
+ * those properties could imply this type. No matter how many 
restrictions are returned, each
+ * one is considered individually sufficient: if a resource has the 
property and the value, then
+ * it belongs to the provided type. Takes type hierarchy into account, 
so the value may imply a
--- End diff --

If A were a superclass of B and C, rather than a subclass, then we would 
consider that individual to be of type A. This method is going to return all 
the hasValue restrictions on all of a type's subclasses, and since belonging to 
one subclass is sufficient to determine membership in the superclass, meeting 
one of those restrictions is also sufficient to determine membership in the 
superclass. Also keep in mind that hasValue property restrictions don't 
*require* any data to be there; they actually imply it: "x is a member of B <-> 
x has prop1=val1". So in your example, if an individual is known to be of type 
A, then it must also have types B and C, and it must also have prop1=val1 and 
prop2=val2. If those triples aren't in the data, that's OK, we can infer them 
(though that would be the other rewriting case and handled by the other 
method). The clearest  statement of this in the standards is probably 
[here](https://www.w3.org/TR/owl2-profiles/#Reasoning_in_OWL_2_RL_and_RDF_Graphs_
 using_Rules), specifically cls-hv1 and cls-hv2 in table 6. (This list of rules 
is incomplete since it only covers OWL RL languages, but the terms it does 
cover have their standard meaning.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #174: RYA-291 Added owl:hasValue inference

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/174#discussion_r127032610
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/InferenceEngine.java 
---
@@ -360,7 +362,47 @@ public void refreshGraph() throws 
InferenceEngineException {
  }
}
 }
-
+
+// Get a set of all property restrictions of any type
+iter = RyaDAOHelper.query(ryaDAO, null, OWL.ONPROPERTY, null, 
conf);
--- End diff --

Agreed, I'll try and clean this up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #174: RYA-291 Added owl:hasValue inference

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/174#discussion_r127020227
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasValueVisitor.java 
---
@@ -0,0 +1,96 @@
+package org.apache.rya.rdftriplestore.inference;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.api.utils.NullableStatementImpl;
+import org.apache.rya.rdftriplestore.utils.FixedStatementPattern;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.OWL;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.Var;
+
+public class HasValueVisitor extends AbstractInferVisitor {
+public HasValueVisitor(RdfCloudTripleStoreConfiguration conf, 
InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = true;
+}
+
+@Override
+protected void meetSP(StatementPattern node) throws Exception {
+final Var subjVar = node.getSubjectVar();
+final Var predVar = node.getPredicateVar();
+final Var objVar = node.getObjectVar();
+// We can reason over two types of statement patterns:
+// { ?var rdf:type :Restriction } and { ?var :property ?value }
+// Both require defined predicate
+if (predVar != null && predVar.getValue() != null) {
+final URI predURI = (URI) predVar.getValue();
+if (RDF.TYPE.equals(predURI) && objVar != null && 
objVar.getValue() != null
+&& objVar.getValue() instanceof Resource) {
+// If the predicate is rdf:type and the type is specified, 
check whether it can be
+// inferred using any hasValue restriction(s)
+final Resource objType = (Resource) objVar.getValue();
+final Map<URI, Set> sufficientValues = 
inferenceEngine.getHasValueByType(objType);
+if (sufficientValues.size() > 0) {
+final Var valueVar = new Var("v-" + UUID.randomUUID());
+TupleExpr currentNode = node.clone();
+for (URI property : sufficientValues.keySet()) {
+final Var propVar = new Var(property.toString(), 
property);
+final TupleExpr valueSP = new 
DoNotExpandSP(subjVar, propVar, valueVar);
+final FixedStatementPattern relevantValues = new 
FixedStatementPattern(objVar, propVar, valueVar);
+for (Value value : sufficientValues.get(property)) 
{
+relevantValues.statements.add(new 
NullableStatementImpl(objType, property, value));
+}
+currentNode = new InferUnion(currentNode, new 
InferJoin(relevantValues, valueSP));
--- End diff --

So the way I understand it, something like ":C1 owl:hasValue :v1 ; 
owl:onProperty :p1" means: "C1 is the set of individuals who have value v1 for 
property p1." Therefore, anything with p1=v1 is by definition a member of C1. 
If we have n such classes C1...Cn, and they all have "C(i) rdfs:subClassOf 
:Superclass1", then we can say: "The set of individuals with value v1 for 
property p1 are a subset of Superclass1; the set of individuals with value v2 
for property p2 are a subset of Superclass1; ... ; the set of individuals with 
value vn for property pn are a subset of Superclass1." Therefore we can say 
that anything with p1=v1 is by definition a member of Superclass1 (regardless 
o

[GitHub] incubator-rya pull request #174: RYA-291 Added owl:hasValue inference

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/174#discussion_r127020889
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasValueVisitor.java 
---
@@ -0,0 +1,96 @@
+package org.apache.rya.rdftriplestore.inference;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.api.utils.NullableStatementImpl;
+import org.apache.rya.rdftriplestore.utils.FixedStatementPattern;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.OWL;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.Var;
+
+public class HasValueVisitor extends AbstractInferVisitor {
+public HasValueVisitor(RdfCloudTripleStoreConfiguration conf, 
InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = true;
+}
+
+@Override
+protected void meetSP(StatementPattern node) throws Exception {
+final Var subjVar = node.getSubjectVar();
+final Var predVar = node.getPredicateVar();
+final Var objVar = node.getObjectVar();
+// We can reason over two types of statement patterns:
+// { ?var rdf:type :Restriction } and { ?var :property ?value }
+// Both require defined predicate
+if (predVar != null && predVar.getValue() != null) {
+final URI predURI = (URI) predVar.getValue();
+if (RDF.TYPE.equals(predURI) && objVar != null && 
objVar.getValue() != null
+&& objVar.getValue() instanceof Resource) {
+// If the predicate is rdf:type and the type is specified, 
check whether it can be
+// inferred using any hasValue restriction(s)
+final Resource objType = (Resource) objVar.getValue();
+final Map<URI, Set> sufficientValues = 
inferenceEngine.getHasValueByType(objType);
--- End diff --

Yep. Any predicate/value combination that would imply that the subject 
belongs to that type.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #174: RYA-291 Added owl:hasValue inference

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/174#discussion_r127015051
  
--- Diff: 
sail/src/main/java/org/apache/rya/rdftriplestore/inference/HasValueVisitor.java 
---
@@ -0,0 +1,96 @@
+package org.apache.rya.rdftriplestore.inference;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import org.apache.rya.api.RdfCloudTripleStoreConfiguration;
+import org.apache.rya.api.utils.NullableStatementImpl;
+import org.apache.rya.rdftriplestore.utils.FixedStatementPattern;
+import org.openrdf.model.Resource;
+import org.openrdf.model.URI;
+import org.openrdf.model.Value;
+import org.openrdf.model.vocabulary.OWL;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.Var;
+
+public class HasValueVisitor extends AbstractInferVisitor {
+public HasValueVisitor(RdfCloudTripleStoreConfiguration conf, 
InferenceEngine inferenceEngine) {
+super(conf, inferenceEngine);
+include = true;
+}
+
+@Override
+protected void meetSP(StatementPattern node) throws Exception {
+final Var subjVar = node.getSubjectVar();
+final Var predVar = node.getPredicateVar();
+final Var objVar = node.getObjectVar();
+// We can reason over two types of statement patterns:
+// { ?var rdf:type :Restriction } and { ?var :property ?value }
+// Both require defined predicate
+if (predVar != null && predVar.getValue() != null) {
+final URI predURI = (URI) predVar.getValue();
+if (RDF.TYPE.equals(predURI) && objVar != null && 
objVar.getValue() != null
+&& objVar.getValue() instanceof Resource) {
+// If the predicate is rdf:type and the type is specified, 
check whether it can be
+// inferred using any hasValue restriction(s)
+final Resource objType = (Resource) objVar.getValue();
+final Map<URI, Set> sufficientValues = 
inferenceEngine.getHasValueByType(objType);
+if (sufficientValues.size() > 0) {
+final Var valueVar = new Var("v-" + UUID.randomUUID());
+TupleExpr currentNode = node.clone();
+for (URI property : sufficientValues.keySet()) {
+final Var propVar = new Var(property.toString(), 
property);
+final TupleExpr valueSP = new 
DoNotExpandSP(subjVar, propVar, valueVar);
+final FixedStatementPattern relevantValues = new 
FixedStatementPattern(objVar, propVar, valueVar);
+for (Value value : sufficientValues.get(property)) 
{
+relevantValues.statements.add(new 
NullableStatementImpl(objType, property, value));
+}
+currentNode = new InferUnion(currentNode, new 
InferJoin(relevantValues, valueSP));
+}
+node.replaceWith(currentNode);
+}
+}
+else {
+// If the predicate has some hasValue restriction 
associated with it, then finding
+// that the object belongs to the appropriate type implies 
a value.
+final Map<Resource, Set> impliedValues = 
inferenceEngine.getHasValueByProperty(predURI);
--- End diff --

All classes with such a restriction, and the values associated with them, 
and we're trying to match objVar to the value. At this point, the check 
RDF.TYPE.equals(predURI) failed, which means there isn't any reason to expect 
objVar to be a class. Instead, we're looking at

[GitHub] incubator-rya pull request #174: RYA-291 Added owl:hasValue inference

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/174

RYA-291 Added owl:hasValue inference

Given a type associated with a hasValue property restriction: 1) expand 
queries
for members of the type to also check for anything with the value; and 2) 
expand
queries for values of that property to check for instances of the type.


## Description
1. Inference engine now stores information about owl:hasValue property 
restrictions, which relate a property, a value, and a type, such that members 
of the type are defined to have that value for that property. 
(https://www.w3.org/TR/2012/REC-owl2-syntax-20121211/#Individual_Value_Restriction)
 These type/property/value associations can be retrieved by type or by property.
2. HasValueVisitor rewrites queries according to the hasValue restrictions, 
expanding statement patterns referencing either the type or the property. Class 
hierarchy is considered, so if something is declared to belong to a subclass of 
such a type, then its value can be inferred; and if something has the 
property/value associated with such a type, then its membership in that type 
and any superclasses can be inferred. The resulting query tree is a union of 
the hasValue logic and the original statement pattern. Other inference visitors 
may still transform the original statement pattern.

### Tests
InferenceEngineTest verifies that the correct schema is extracted from the 
ontology triples; HasValueVisitorTest verifies that query trees are expanded as 
expected; and InferenceIT verifies expected query results given ontology and 
instance triples.

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-291)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
@meiercaleb 
@ejwhite922 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-291-hasValue-inference

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/174.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #174


commit c0cc9596673904f9b9f8dea7be84de75066f73a5
Author: Jesse Hatfield <jesse.hatfi...@parsons.com>
Date:   2017-06-26T18:06:06Z

RYA-291 Added owl:hasValue inference

Given a type associated with a hasValue property restriction: 1) expand 
queries
for members of the type to also check for anything with the value; and 2) 
expand
queries for values of that property to check for instances of the type.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #173: RYA-304 Fixed rya.merger integration tests

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/173#discussion_r123583040
  
--- Diff: 
extras/rya.merger/src/test/java/org/apache/rya/accumulo/mr/merge/RulesetCopyIT.java
 ---
@@ -144,32 +146,33 @@ private static RyaType literal(final String lit, 
final URI type) {
 
 @BeforeClass
 public static void setUpPerClass() throws Exception {
+DemoUtilities.setupLogging(LoggingDetail.LIGHT);
 accumuloDualInstanceDriver = new 
AccumuloDualInstanceDriver(IS_MOCK, true, true, false, false);
 accumuloDualInstanceDriver.setUpInstances();
 }
 
 @Before
 public void setUpPerTest() throws Exception {
-parentConfig = accumuloDualInstanceDriver.getParentConfig();
-childConfig = accumuloDualInstanceDriver.getChildConfig();
-accumuloDualInstanceDriver.setUpTables();
 accumuloDualInstanceDriver.setUpConfigs();
+accumuloDualInstanceDriver.setUpTables();
 accumuloDualInstanceDriver.setUpDaos();
+parentConfig = accumuloDualInstanceDriver.getParentConfig();
+childConfig = accumuloDualInstanceDriver.getChildConfig();
 parentDao = accumuloDualInstanceDriver.getParentDao();
 }
 
 @After
 public void tearDownPerTest() throws Exception {
 log.info("tearDownPerTest(): tearing down now.");
-accumuloDualInstanceDriver.tearDownDaos();
 accumuloDualInstanceDriver.tearDownTables();
+accumuloDualInstanceDriver.tearDownDaos();
+if (rulesetTool != null) {
+rulesetTool.shutdown();
+}
 }
 
 @AfterClass
 public static void tearDownPerClass() throws Exception {
-if (rulesetTool != null) {
-rulesetTool.shutdown();
-}
 accumuloDualInstanceDriver.tearDown();
--- End diff --

Oh, I suppose there are two parts: The AccumuloDualInstanceDriver is 
initialized once per class, and in turn initializes the parent MAC. But the 
child MAC is instead initialized when the tool is run (via 
CopyTool.createChildInstance), which is once per test. Calling 
rulesetTool.shutdown() will close the child instance (but not the parent). 
Previously, this shutdown call was only being made once per class, so the extra 
child MACs weren't being shut down between tests. This PR fixes that by moving 
that call into the per-test shutdown. This leaves the parent MAC open between 
tests (hence the minimum 5 processes), but there's only ever one parent MAC 
(the one that was initialized by the AccumuloDualInstanceDriver) and it does 
get shut down at the end, so we don't have these extraneous processes. So I 
believe this PR solves the problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #173: RYA-304 Fixed rya.merger integration tests

[jessehatfield]

Github user jessehatfield commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/173#discussion_r123563097
  
--- Diff: 
extras/rya.merger/src/test/java/org/apache/rya/accumulo/mr/merge/RulesetCopyIT.java
 ---
@@ -144,32 +146,33 @@ private static RyaType literal(final String lit, 
final URI type) {
 
 @BeforeClass
 public static void setUpPerClass() throws Exception {
+DemoUtilities.setupLogging(LoggingDetail.LIGHT);
 accumuloDualInstanceDriver = new 
AccumuloDualInstanceDriver(IS_MOCK, true, true, false, false);
 accumuloDualInstanceDriver.setUpInstances();
 }
 
 @Before
 public void setUpPerTest() throws Exception {
-parentConfig = accumuloDualInstanceDriver.getParentConfig();
-childConfig = accumuloDualInstanceDriver.getChildConfig();
-accumuloDualInstanceDriver.setUpTables();
 accumuloDualInstanceDriver.setUpConfigs();
+accumuloDualInstanceDriver.setUpTables();
 accumuloDualInstanceDriver.setUpDaos();
+parentConfig = accumuloDualInstanceDriver.getParentConfig();
+childConfig = accumuloDualInstanceDriver.getChildConfig();
 parentDao = accumuloDualInstanceDriver.getParentDao();
 }
 
 @After
 public void tearDownPerTest() throws Exception {
 log.info("tearDownPerTest(): tearing down now.");
-accumuloDualInstanceDriver.tearDownDaos();
 accumuloDualInstanceDriver.tearDownTables();
+accumuloDualInstanceDriver.tearDownDaos();
+if (rulesetTool != null) {
+rulesetTool.shutdown();
+}
 }
 
 @AfterClass
 public static void tearDownPerClass() throws Exception {
-if (rulesetTool != null) {
-rulesetTool.shutdown();
-}
 accumuloDualInstanceDriver.tearDown();
--- End diff --

I can verify that it this helps. Ran tests from the command line ("mvn 
verify"), monitored the number of org.apache.accumulo.start.Main processes 
running: seems to vary between 11 (during tests) down to 5 (between tests), 
where previously it continued to climb up to a max of 31 for me. And the change 
makes sense: the tool gets initialized in each test (initialization happens in 
runRulesetCopyTest, which each test calls), so it should be torn down after 
each test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #55: RYA-76 Consolidated MapReduce API and applic...

[jessehatfield]

Github user jessehatfield closed the pull request at:

https://github.com/apache/incubator-rya/pull/55


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #50: Included project rya.reasoning.

[jessehatfield]

Github user jessehatfield closed the pull request at:

https://github.com/apache/incubator-rya/pull/50


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request: Added a deserializeStatement method fo...

[jessehatfield]

GitHub user jessehatfield opened a pull request:

https://github.com/apache/incubator-rya/pull/45

Added a deserializeStatement method for the EntityCentricIndex

Convert a Key and Value from the entity table into a RyaStatement.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jessehatfield/incubator-rya 
RYA-76-entitycentricindex-deserialize

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/45.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #45


commit 361748d5ed8071423e901f43633c290b0f9dfccd
Author: Jesse Hatfield <jesse.hatfi...@parsons.com>
Date:   2016-05-27T15:00:31Z

Added a deserializeStatement method for the EntityCentricIndex




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---