[GitHub] incubator-rya pull request #135: Rya 144

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-rya/pull/135


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r105975873
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/EntityIndexOptimizer.java
 ---
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Type;
+import org.apache.rya.indexing.entity.query.EntityQueryNode;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+import org.apache.rya.indexing.entity.storage.TypeStorage;
+import org.apache.rya.indexing.entity.update.mongo.MongoEntityIndexer;
+import 
org.apache.rya.indexing.external.matching.AbstractExternalSetOptimizer;
+import org.apache.rya.indexing.external.matching.ExternalSetMatcher;
+import org.apache.rya.indexing.external.matching.ExternalSetProvider;
+import org.apache.rya.indexing.external.matching.QueryNodeListRater;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.apache.rya.indexing.external.matching.TopOfQueryFilterRelocator;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.Dataset;
+import org.openrdf.query.algebra.TupleExpr;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+
+/**
+ * Optimizes a query over {@link Entity}s.
+ */
+public class EntityIndexOptimizer extends 
AbstractExternalSetOptimizer implements Configurable {
+private static final Logger log = 
Logger.getLogger(EntityIndexOptimizer.class);
+private static final EntityExternalSetMatcherFactory MATCHER_FACTORY = 
new EntityExternalSetMatcherFactory();
+
+private final MongoEntityIndexer indexer;
+private EntityIndexSetProvider provider;
+private Configuration conf;
+
+private EntityStorage entityStorage;
+private TypeStorage typeStorage;
+
+/**
+ * Creates a new {@link EntityIndexOptimizer}.
+ * @param typeStorage - The mechanism for access the various {@link 
Type}s. (not null)
+ * @param entityStorage - The mechanism for access to the various 
{@link Entity}s. (not null)
+ */
+public EntityIndexOptimizer() {
+indexer = new MongoEntityIndexer();
+
+//There is no rater for entity query nodes.
+useOptimal = false;
+}
+
+@Override
+public void setConf(final Configuration conf) {
+this.conf = conf;
+indexer.setConf(conf);
+
+typeStorage = indexer.getTypeStorage(conf);
+entityStorage = indexer.getEntityStorage(conf);
+
+provider = new EntityIndexSetProvider(typeStorage, entityStorage);
+}
+
+@Override
+public Configuration getConf() {
+return conf;
+}
+
+@Override
+public void optimize(TupleExpr tupleExpr, final Dataset dataset, final 
BindingSet bindings) {
+checkNotNull(tupleExpr);
+checkNotNull(indexer);
+
+// first standardize query by pulling all filters to top of query 
if
+// they exist using TopOfQueryFilterRelocator
+tupleExpr = TopOfQueryFilterRelocator.moveFiltersToTop(tupleExpr);
+super.optimize(tupleExpr, null, null);
+}
+
+@VisibleForTesting
+public EntityStorage getEntityStorage() {
--- End diff --

left here per our talk


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at 

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r105973476
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/EntityIndexOptimizer.java
 ---
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Type;
+import org.apache.rya.indexing.entity.query.EntityQueryNode;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+import org.apache.rya.indexing.entity.storage.TypeStorage;
+import org.apache.rya.indexing.entity.update.mongo.MongoEntityIndexer;
+import 
org.apache.rya.indexing.external.matching.AbstractExternalSetOptimizer;
+import org.apache.rya.indexing.external.matching.ExternalSetMatcher;
+import org.apache.rya.indexing.external.matching.ExternalSetProvider;
+import org.apache.rya.indexing.external.matching.QueryNodeListRater;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.apache.rya.indexing.external.matching.TopOfQueryFilterRelocator;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.Dataset;
+import org.openrdf.query.algebra.TupleExpr;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+
+/**
+ * Optimizes a query over {@link Entity}s.
+ */
+public class EntityIndexOptimizer extends 
AbstractExternalSetOptimizer implements Configurable {
+private static final Logger log = 
Logger.getLogger(EntityIndexOptimizer.class);
+private static final EntityExternalSetMatcherFactory MATCHER_FACTORY = 
new EntityExternalSetMatcherFactory();
+
+private final MongoEntityIndexer indexer;
+private EntityIndexSetProvider provider;
+private Configuration conf;
+
+private EntityStorage entityStorage;
+private TypeStorage typeStorage;
+
+/**
+ * Creates a new {@link EntityIndexOptimizer}.
+ * @param typeStorage - The mechanism for access the various {@link 
Type}s. (not null)
--- End diff --

fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r105973218
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -204,35 +241,116 @@ private static void 
verifyAllPredicatesPartOfType(Type type, Collection 
evaluate(Collection bindingSets) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final Collection bindingSets) throws 
QueryEvaluationException {
 // TODO Auto-generated method stub
 return null;
 }
 
 @Override
-public CloseableIteration 
evaluate(BindingSet bindingSet) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final BindingSet bindingSet) throws QueryEvaluationException {
 requireNonNull(bindingSet);
-
 // ... ok, so if the subject needs to be filled in, then we need 
to see if the subject variable is in the binding set.
 // if it is, fetch that value and then fetch the entity for the 
subject.
+final MapBindingSet resultSet = new MapBindingSet();
+try {
+final Optional optEntity;
+final String subj;
+if(subjectIsConstant) {
+// if it isn't, fetch the entity for the constant?
+subj = subjectConstant.get();
+optEntity = entities.get(new 
RyaURI(subjectConstant.get()));
+} else {
+subj = subjectVar.get();
+if(bindingSet.getBindingNames().contains(subj)) {
+final Binding binding = bindingSet.getBinding(subj);
+optEntity = entities.get(new 
RyaURI(binding.getValue().toString()));
+} else {
+// RETURN AN EMPTY ITERATION IF IT CAN NOT FILL IT IN!
+return new EmptyIteration();
+}
+}
 
-// if it isn't, fetch the entity for the constant?
-
+if(optEntity.isPresent()) {
+// for all variables in the OBJECT portion of the SPs, 
fill 'em in using the entity that is stored in the index.
+final Entity entity = optEntity.get();
+final ImmutableMap entityProps = 
entity.getProperties().get(type.getId());
+//ensure properties match and only add properties that are 
in the statement patterns to the binding set
+for(final RyaURI key : objectVariables.keySet()) {
+if(entityProps.containsKey(key)) {
+//the statement pattern has this property, check 
to see if the values match(if constant)
+final Var object = objectVariables.get(key);
+final RyaType entityType = 
entityProps.get(key).getValue();
+final Value entityVal = 
ValueFactoryImpl.getInstance().createLiteral(entityType.getData(), 
entityType.getDataType());
+if(object.isConstant()) {
+final Value objectVal = object.getValue();
+if(objectVal.equals(entityVal)) {
+//add to binding set
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+//entity doesn't hold to statement patterns
+}
+}
+} else {
+return new EmptyIteration();
+}
+} catch (final EntityStorageException e) {
+throw new QueryEvaluationException("Failed to evaluate the 
binding set", e);
+}
+// x = alice's SSN
+// y = blue  <-- how do i know this is for urn:eye property? FROM 
THE STATEMENT PATTERN. look for the ?y in the SP, and that has the property 
name in it.
+bindingSet.forEach(new Consumer() {
+@Override
+public void accept(final Binding binding) {
+resultSet.addBinding(binding);
+}
+});
+final List list = new ArrayList<>();
+list.add(resultSet);
+return new CollectionIteration<>(list);
+}
 
-// RETURN AN EMPTY ITERATION IF IT CAN NOT FILL IT IN!
+/**
+ * @return - The 

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r105971116
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -204,35 +241,116 @@ private static void 
verifyAllPredicatesPartOfType(Type type, Collection 
evaluate(Collection bindingSets) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final Collection bindingSets) throws 
QueryEvaluationException {
 // TODO Auto-generated method stub
 return null;
 }
 
 @Override
-public CloseableIteration 
evaluate(BindingSet bindingSet) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final BindingSet bindingSet) throws QueryEvaluationException {
 requireNonNull(bindingSet);
-
 // ... ok, so if the subject needs to be filled in, then we need 
to see if the subject variable is in the binding set.
 // if it is, fetch that value and then fetch the entity for the 
subject.
+final MapBindingSet resultSet = new MapBindingSet();
+try {
+final Optional optEntity;
+final String subj;
+if(subjectIsConstant) {
+// if it isn't, fetch the entity for the constant?
+subj = subjectConstant.get();
+optEntity = entities.get(new 
RyaURI(subjectConstant.get()));
+} else {
+subj = subjectVar.get();
+if(bindingSet.getBindingNames().contains(subj)) {
+final Binding binding = bindingSet.getBinding(subj);
+optEntity = entities.get(new 
RyaURI(binding.getValue().toString()));
+} else {
+// RETURN AN EMPTY ITERATION IF IT CAN NOT FILL IT IN!
+return new EmptyIteration();
+}
+}
 
-// if it isn't, fetch the entity for the constant?
-
+if(optEntity.isPresent()) {
+// for all variables in the OBJECT portion of the SPs, 
fill 'em in using the entity that is stored in the index.
+final Entity entity = optEntity.get();
+final ImmutableMap entityProps = 
entity.getProperties().get(type.getId());
+//ensure properties match and only add properties that are 
in the statement patterns to the binding set
+for(final RyaURI key : objectVariables.keySet()) {
+if(entityProps.containsKey(key)) {
+//the statement pattern has this property, check 
to see if the values match(if constant)
+final Var object = objectVariables.get(key);
+final RyaType entityType = 
entityProps.get(key).getValue();
+final Value entityVal = 
ValueFactoryImpl.getInstance().createLiteral(entityType.getData(), 
entityType.getDataType());
+if(object.isConstant()) {
+final Value objectVal = object.getValue();
+if(objectVal.equals(entityVal)) {
+//add to binding set
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+//entity doesn't hold to statement patterns
+}
+}
+} else {
+return new EmptyIteration();
+}
+} catch (final EntityStorageException e) {
+throw new QueryEvaluationException("Failed to evaluate the 
binding set", e);
+}
+// x = alice's SSN
+// y = blue  <-- how do i know this is for urn:eye property? FROM 
THE STATEMENT PATTERN. look for the ?y in the SP, and that has the property 
name in it.
--- End diff --

doc no longer applies, removed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r105970120
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -105,8 +126,24 @@ public EntityQueryNode(final Type type, final 
Collection patte
 // TODO Also, map each variable that is in an Object spot each 
variable can be mapped to a property name as well
--- End diff --

yes, removed it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r105969903
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -62,11 +78,14 @@
 private final Optional subjectConstant;
 private final Optional subjectVar;
 
+//since and EntityQueryNode exists in a single segment, all binding 
names are garunteed to be assured.
+private final Set bindingNames;
+
 // Information about the objects of the patterns.
 
 // XXX what does this map? property name -> binding variable?
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r105969648
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -20,31 +20,47 @@
 
 import static java.util.Objects.requireNonNull;
 
+import java.util.ArrayList;
 import java.util.Collection;
+import java.util.HashSet;
 import java.util.Iterator;
+import java.util.List;
 import java.util.Optional;
+import java.util.Set;
+import java.util.function.Consumer;
 
-import javax.annotation.ParametersAreNonnullByDefault;
-
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Property;
 import org.apache.rya.indexing.entity.model.Type;
 import org.apache.rya.indexing.entity.storage.EntityStorage;
+import 
org.apache.rya.indexing.entity.storage.EntityStorage.EntityStorageException;
+import org.apache.rya.rdftriplestore.evaluation.ExternalBatchingIterator;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.ValueFactoryImpl;
 import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.Binding;
 import org.openrdf.query.BindingSet;
 import org.openrdf.query.QueryEvaluationException;
 import org.openrdf.query.algebra.StatementPattern;
 import org.openrdf.query.algebra.Var;
 import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+import org.openrdf.query.algebra.evaluation.iterator.CollectionIteration;
+import org.openrdf.query.impl.MapBindingSet;
 
 import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableMap.Builder;
 
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
 import info.aduna.iteration.CloseableIteration;
-import mvm.rya.api.domain.RyaURI;
-import mvm.rya.rdftriplestore.evaluation.ExternalBatchingIterator;
+import info.aduna.iteration.EmptyIteration;
 
 /**
  * TODO impl, test, doc
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[kchilton2]

Github user kchilton2 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r104004058
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -204,35 +241,116 @@ private static void 
verifyAllPredicatesPartOfType(Type type, Collection 
evaluate(Collection bindingSets) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final Collection bindingSets) throws 
QueryEvaluationException {
 // TODO Auto-generated method stub
 return null;
 }
 
 @Override
-public CloseableIteration 
evaluate(BindingSet bindingSet) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final BindingSet bindingSet) throws QueryEvaluationException {
 requireNonNull(bindingSet);
-
 // ... ok, so if the subject needs to be filled in, then we need 
to see if the subject variable is in the binding set.
 // if it is, fetch that value and then fetch the entity for the 
subject.
+final MapBindingSet resultSet = new MapBindingSet();
+try {
+final Optional optEntity;
+final String subj;
+if(subjectIsConstant) {
+// if it isn't, fetch the entity for the constant?
+subj = subjectConstant.get();
+optEntity = entities.get(new 
RyaURI(subjectConstant.get()));
+} else {
+subj = subjectVar.get();
+if(bindingSet.getBindingNames().contains(subj)) {
+final Binding binding = bindingSet.getBinding(subj);
+optEntity = entities.get(new 
RyaURI(binding.getValue().toString()));
+} else {
+// RETURN AN EMPTY ITERATION IF IT CAN NOT FILL IT IN!
+return new EmptyIteration();
+}
+}
 
-// if it isn't, fetch the entity for the constant?
-
+if(optEntity.isPresent()) {
+// for all variables in the OBJECT portion of the SPs, 
fill 'em in using the entity that is stored in the index.
+final Entity entity = optEntity.get();
+final ImmutableMap entityProps = 
entity.getProperties().get(type.getId());
+//ensure properties match and only add properties that are 
in the statement patterns to the binding set
+for(final RyaURI key : objectVariables.keySet()) {
+if(entityProps.containsKey(key)) {
+//the statement pattern has this property, check 
to see if the values match(if constant)
+final Var object = objectVariables.get(key);
+final RyaType entityType = 
entityProps.get(key).getValue();
+final Value entityVal = 
ValueFactoryImpl.getInstance().createLiteral(entityType.getData(), 
entityType.getDataType());
+if(object.isConstant()) {
+final Value objectVal = object.getValue();
+if(objectVal.equals(entityVal)) {
+//add to binding set
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+//entity doesn't hold to statement patterns
+}
+}
+} else {
+return new EmptyIteration();
+}
+} catch (final EntityStorageException e) {
+throw new QueryEvaluationException("Failed to evaluate the 
binding set", e);
+}
+// x = alice's SSN
+// y = blue  <-- how do i know this is for urn:eye property? FROM 
THE STATEMENT PATTERN. look for the ?y in the SP, and that has the property 
name in it.
+bindingSet.forEach(new Consumer() {
+@Override
+public void accept(final Binding binding) {
+resultSet.addBinding(binding);
+}
+});
+final List list = new ArrayList<>();
+list.add(resultSet);
+return new CollectionIteration<>(list);
+}
 
-// RETURN AN EMPTY ITERATION IF IT CAN NOT FILL IT IN!
+/**
+ * @return - 

[GitHub] incubator-rya pull request #135: Rya 144

[kchilton2]

Github user kchilton2 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r104003321
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -20,31 +20,47 @@
 
 import static java.util.Objects.requireNonNull;
 
+import java.util.ArrayList;
 import java.util.Collection;
+import java.util.HashSet;
 import java.util.Iterator;
+import java.util.List;
 import java.util.Optional;
+import java.util.Set;
+import java.util.function.Consumer;
 
-import javax.annotation.ParametersAreNonnullByDefault;
-
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Property;
 import org.apache.rya.indexing.entity.model.Type;
 import org.apache.rya.indexing.entity.storage.EntityStorage;
+import 
org.apache.rya.indexing.entity.storage.EntityStorage.EntityStorageException;
+import org.apache.rya.rdftriplestore.evaluation.ExternalBatchingIterator;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.ValueFactoryImpl;
 import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.Binding;
 import org.openrdf.query.BindingSet;
 import org.openrdf.query.QueryEvaluationException;
 import org.openrdf.query.algebra.StatementPattern;
 import org.openrdf.query.algebra.Var;
 import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+import org.openrdf.query.algebra.evaluation.iterator.CollectionIteration;
+import org.openrdf.query.impl.MapBindingSet;
 
 import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableMap.Builder;
 
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
 import info.aduna.iteration.CloseableIteration;
-import mvm.rya.api.domain.RyaURI;
-import mvm.rya.rdftriplestore.evaluation.ExternalBatchingIterator;
+import info.aduna.iteration.EmptyIteration;
 
 /**
  * TODO impl, test, doc
--- End diff --

Missing docs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[kchilton2]

Github user kchilton2 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r104004702
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -204,35 +241,116 @@ private static void 
verifyAllPredicatesPartOfType(Type type, Collection 
evaluate(Collection bindingSets) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final Collection bindingSets) throws 
QueryEvaluationException {
 // TODO Auto-generated method stub
 return null;
 }
 
 @Override
-public CloseableIteration 
evaluate(BindingSet bindingSet) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final BindingSet bindingSet) throws QueryEvaluationException {
 requireNonNull(bindingSet);
-
 // ... ok, so if the subject needs to be filled in, then we need 
to see if the subject variable is in the binding set.
 // if it is, fetch that value and then fetch the entity for the 
subject.
+final MapBindingSet resultSet = new MapBindingSet();
+try {
+final Optional optEntity;
+final String subj;
+if(subjectIsConstant) {
+// if it isn't, fetch the entity for the constant?
+subj = subjectConstant.get();
+optEntity = entities.get(new 
RyaURI(subjectConstant.get()));
+} else {
+subj = subjectVar.get();
+if(bindingSet.getBindingNames().contains(subj)) {
+final Binding binding = bindingSet.getBinding(subj);
+optEntity = entities.get(new 
RyaURI(binding.getValue().toString()));
+} else {
+// RETURN AN EMPTY ITERATION IF IT CAN NOT FILL IT IN!
+return new EmptyIteration();
+}
+}
 
-// if it isn't, fetch the entity for the constant?
-
+if(optEntity.isPresent()) {
+// for all variables in the OBJECT portion of the SPs, 
fill 'em in using the entity that is stored in the index.
+final Entity entity = optEntity.get();
+final ImmutableMap entityProps = 
entity.getProperties().get(type.getId());
+//ensure properties match and only add properties that are 
in the statement patterns to the binding set
+for(final RyaURI key : objectVariables.keySet()) {
+if(entityProps.containsKey(key)) {
+//the statement pattern has this property, check 
to see if the values match(if constant)
+final Var object = objectVariables.get(key);
+final RyaType entityType = 
entityProps.get(key).getValue();
+final Value entityVal = 
ValueFactoryImpl.getInstance().createLiteral(entityType.getData(), 
entityType.getDataType());
+if(object.isConstant()) {
+final Value objectVal = object.getValue();
+if(objectVal.equals(entityVal)) {
+//add to binding set
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+//entity doesn't hold to statement patterns
--- End diff --

No point in having an empty else.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[kchilton2]

Github user kchilton2 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r104005397
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/EntityIndexOptimizer.java
 ---
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.Logger;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Type;
+import org.apache.rya.indexing.entity.query.EntityQueryNode;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+import org.apache.rya.indexing.entity.storage.TypeStorage;
+import org.apache.rya.indexing.entity.update.mongo.MongoEntityIndexer;
+import 
org.apache.rya.indexing.external.matching.AbstractExternalSetOptimizer;
+import org.apache.rya.indexing.external.matching.ExternalSetMatcher;
+import org.apache.rya.indexing.external.matching.ExternalSetProvider;
+import org.apache.rya.indexing.external.matching.QueryNodeListRater;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.apache.rya.indexing.external.matching.TopOfQueryFilterRelocator;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.Dataset;
+import org.openrdf.query.algebra.TupleExpr;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+
+/**
+ * Optimizes a query over {@link Entity}s.
+ */
+public class EntityIndexOptimizer extends 
AbstractExternalSetOptimizer implements Configurable {
+private static final Logger log = 
Logger.getLogger(EntityIndexOptimizer.class);
+private static final EntityExternalSetMatcherFactory MATCHER_FACTORY = 
new EntityExternalSetMatcherFactory();
+
+private final MongoEntityIndexer indexer;
+private EntityIndexSetProvider provider;
+private Configuration conf;
+
+private EntityStorage entityStorage;
+private TypeStorage typeStorage;
+
+/**
+ * Creates a new {@link EntityIndexOptimizer}.
+ * @param typeStorage - The mechanism for access the various {@link 
Type}s. (not null)
+ * @param entityStorage - The mechanism for access to the various 
{@link Entity}s. (not null)
+ */
+public EntityIndexOptimizer() {
+indexer = new MongoEntityIndexer();
+
+//There is no rater for entity query nodes.
+useOptimal = false;
+}
+
+@Override
+public void setConf(final Configuration conf) {
+this.conf = conf;
+indexer.setConf(conf);
+
+typeStorage = indexer.getTypeStorage(conf);
+entityStorage = indexer.getEntityStorage(conf);
+
+provider = new EntityIndexSetProvider(typeStorage, entityStorage);
+}
+
+@Override
+public Configuration getConf() {
+return conf;
+}
+
+@Override
+public void optimize(TupleExpr tupleExpr, final Dataset dataset, final 
BindingSet bindings) {
+checkNotNull(tupleExpr);
+checkNotNull(indexer);
+
+// first standardize query by pulling all filters to top of query 
if
+// they exist using TopOfQueryFilterRelocator
+tupleExpr = TopOfQueryFilterRelocator.moveFiltersToTop(tupleExpr);
+super.optimize(tupleExpr, null, null);
+}
+
+@VisibleForTesting
+public EntityStorage getEntityStorage() {
--- End diff --

This methods that only exist for testing are a smell. You should be able to 
push a mock or the object you need access to within your tests into this class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project 

[GitHub] incubator-rya pull request #135: Rya 144

[kchilton2]

Github user kchilton2 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r104003478
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -105,8 +126,24 @@ public EntityQueryNode(final Type type, final 
Collection patte
 // TODO Also, map each variable that is in an Object spot each 
variable can be mapped to a property name as well
--- End diff --

Has this TODO been addressed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[kchilton2]

Github user kchilton2 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r104003847
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -204,35 +241,116 @@ private static void 
verifyAllPredicatesPartOfType(Type type, Collection 
evaluate(Collection bindingSets) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final Collection bindingSets) throws 
QueryEvaluationException {
 // TODO Auto-generated method stub
 return null;
 }
 
 @Override
-public CloseableIteration 
evaluate(BindingSet bindingSet) throws QueryEvaluationException {
+public CloseableIteration 
evaluate(final BindingSet bindingSet) throws QueryEvaluationException {
 requireNonNull(bindingSet);
-
 // ... ok, so if the subject needs to be filled in, then we need 
to see if the subject variable is in the binding set.
 // if it is, fetch that value and then fetch the entity for the 
subject.
+final MapBindingSet resultSet = new MapBindingSet();
+try {
+final Optional optEntity;
+final String subj;
+if(subjectIsConstant) {
+// if it isn't, fetch the entity for the constant?
+subj = subjectConstant.get();
+optEntity = entities.get(new 
RyaURI(subjectConstant.get()));
+} else {
+subj = subjectVar.get();
+if(bindingSet.getBindingNames().contains(subj)) {
+final Binding binding = bindingSet.getBinding(subj);
+optEntity = entities.get(new 
RyaURI(binding.getValue().toString()));
+} else {
+// RETURN AN EMPTY ITERATION IF IT CAN NOT FILL IT IN!
+return new EmptyIteration();
+}
+}
 
-// if it isn't, fetch the entity for the constant?
-
+if(optEntity.isPresent()) {
+// for all variables in the OBJECT portion of the SPs, 
fill 'em in using the entity that is stored in the index.
+final Entity entity = optEntity.get();
+final ImmutableMap entityProps = 
entity.getProperties().get(type.getId());
+//ensure properties match and only add properties that are 
in the statement patterns to the binding set
+for(final RyaURI key : objectVariables.keySet()) {
+if(entityProps.containsKey(key)) {
+//the statement pattern has this property, check 
to see if the values match(if constant)
+final Var object = objectVariables.get(key);
+final RyaType entityType = 
entityProps.get(key).getValue();
+final Value entityVal = 
ValueFactoryImpl.getInstance().createLiteral(entityType.getData(), 
entityType.getDataType());
+if(object.isConstant()) {
+final Value objectVal = object.getValue();
+if(objectVal.equals(entityVal)) {
+//add to binding set
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+resultSet.addBinding(object.getName(), 
entityVal);
+}
+} else {
+//entity doesn't hold to statement patterns
+}
+}
+} else {
+return new EmptyIteration();
+}
+} catch (final EntityStorageException e) {
+throw new QueryEvaluationException("Failed to evaluate the 
binding set", e);
+}
+// x = alice's SSN
+// y = blue  <-- how do i know this is for urn:eye property? FROM 
THE STATEMENT PATTERN. look for the ?y in the SP, and that has the property 
name in it.
--- End diff --

Cleanup docs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[kchilton2]

Github user kchilton2 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r104003396
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -62,11 +78,14 @@
 private final Optional subjectConstant;
 private final Optional subjectVar;
 
+//since and EntityQueryNode exists in a single segment, all binding 
names are garunteed to be assured.
+private final Set bindingNames;
+
 // Information about the objects of the patterns.
 
 // XXX what does this map? property name -> binding variable?
--- End diff --

Cleanup docs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[pujav65]

Github user pujav65 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r97357559
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/storage/CloseableIterator.java
 ---
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.storage;
+
+import java.io.Closeable;
+import java.util.Iterator;
+
+/**
+ * An {@link Iterator} that is also {@link Closeable}.
+ *
+ * @param  - The type of object that will be iterated over.
+ */
+public interface CloseableIterator extends Iterator, Closeable { }
--- End diff --

i'm not good with merging this until this is either removed or renamed to 
not be the same name as the mango implementation.  it reminds me of places 
where you see custom lists (like arraylist), which i would like to avoid 
introducing if we can.
If you won't refactor to use the mango implementation, then rename to 
something other than CloseableIterator.  Once that's done I think this is good 
to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r95199111
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/update/EntityIndexer.java
 ---
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.update;
+
+import org.apache.rya.api.persist.index.RyaSecondaryIndexer;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+
+/**
+ * Updates the {@link Entity}s that are in a {@link EntityStorage} when new
+ * {@link RyaStatement}s are added/removed from the Rya instance.
+ */
+public interface EntityIndexer extends RyaSecondaryIndexer {
--- End diff --

moved up the get storage functions.  the interface makes it way easier to 
test/mock.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r95198684
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/storage/CloseableIterator.java
 ---
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.storage;
+
+import java.io.Closeable;
+import java.util.Iterator;
+
+/**
+ * An {@link Iterator} that is also {@link Closeable}.
+ *
+ * @param  - The type of object that will be iterated over.
+ */
+public interface CloseableIterator extends Iterator, Closeable { }
--- End diff --

I think the mango version has functionality we don't necessarily want


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[pujav65]

Github user pujav65 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r95188019
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/storage/CloseableIterator.java
 ---
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.storage;
+
+import java.io.Closeable;
+import java.util.Iterator;
+
+/**
+ * An {@link Iterator} that is also {@link Closeable}.
+ *
+ * @param  - The type of object that will be iterated over.
+ */
+public interface CloseableIterator extends Iterator, Closeable { }
--- End diff --

ok, can you please remove it and then refactor places its used to use the 
mango version?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[pujav65]

Github user pujav65 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r95188564
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/update/EntityIndexer.java
 ---
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.update;
+
+import org.apache.rya.api.persist.index.RyaSecondaryIndexer;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+
+/**
+ * Updates the {@link Entity}s that are in a {@link EntityStorage} when new
+ * {@link RyaStatement}s are added/removed from the Rya instance.
+ */
+public interface EntityIndexer extends RyaSecondaryIndexer {
--- End diff --

i would just remove this then -- either that or pull up some methods that 
indicate its intent, i'm not really sure what having an empty interface buys us 
since its not even particularly well documented.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r95018453
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/update/EntityIndexer.java
 ---
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.update;
+
+import org.apache.rya.api.persist.index.RyaSecondaryIndexer;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+
+/**
+ * Updates the {@link Entity}s that are in a {@link EntityStorage} when new
+ * {@link RyaStatement}s are added/removed from the Rya instance.
+ */
+public interface EntityIndexer extends RyaSecondaryIndexer {
--- End diff --

Again, not mine. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r95018110
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -0,0 +1,356 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.query;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Consumer;
+
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Property;
+import org.apache.rya.indexing.entity.model.Type;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+import 
org.apache.rya.indexing.entity.storage.EntityStorage.EntityStorageException;
+import org.apache.rya.rdftriplestore.evaluation.ExternalBatchingIterator;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.Binding;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+import org.openrdf.query.algebra.evaluation.iterator.CollectionIteration;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableMap.Builder;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import info.aduna.iteration.CloseableIteration;
+import info.aduna.iteration.EmptyIteration;
+
+/**
+ * TODO impl, test, doc
+ */
+@DefaultAnnotation(NonNull.class)
+public class EntityQueryNode extends ExternalSet implements 
ExternalBatchingIterator {
+
+/**
+ * The RyaURI that when used as the Predicate of a Statement Pattern 
indicates the Type of the Entities.
+ */
+private static final RyaURI TYPE_ID_URI = new 
RyaURI(RDF.TYPE.toString());
+
+// Provided at construction time.
+private final Type type;
+private final Collection patterns;
+private final EntityStorage entities;
+
+// Information about the subject of the patterns.
+private final boolean subjectIsConstant;
+private final Optional subjectConstant;
+private final Optional subjectVar;
+
+//since and EntityQueryNode exists in a single segment, all binding 
names are garunteed to be assured.
+private final Set bindingNames;
+
+// Information about the objects of the patterns.
+
+// XXX what does this map? property name -> binding variable?
+// for any property of the entity that has a variable, have to fill it 
in?
+private final ImmutableMap objectVariables;
+
+
+/**
+ * Constructs an instance of {@link EntityQueryNode}.
+ *
+ * @param type - The type of {@link Entity} this node matches. (not 
null)
+ * @param patterns - The query StatementPatterns that are solved using 
an
+ *   Entity of the Type. (not null)
+ * @param entities - The {@link EntityStorage} that will be searched 
to match
+ *   {@link BindingSet}s when evaluating a query. (not null)
+ */
+public EntityQueryNode(final Type type, final 
Collection patterns, final EntityStorage entities) throws 
IllegalStateException {
+this.type = requireNonNull(type);
+this.patterns = requireNonNull(patterns);
+this.entities = requireNonNull(entities);
+
 

[GitHub] incubator-rya pull request #135: Rya 144

[pujav65]

Github user pujav65 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r95007185
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/query/EntityQueryNode.java
 ---
@@ -0,0 +1,356 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.query;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Consumer;
+
+import org.apache.rya.api.domain.RyaType;
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Property;
+import org.apache.rya.indexing.entity.model.Type;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+import 
org.apache.rya.indexing.entity.storage.EntityStorage.EntityStorageException;
+import org.apache.rya.rdftriplestore.evaluation.ExternalBatchingIterator;
+import org.openrdf.model.Value;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.Binding;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+import org.openrdf.query.algebra.evaluation.impl.ExternalSet;
+import org.openrdf.query.algebra.evaluation.iterator.CollectionIteration;
+import org.openrdf.query.impl.MapBindingSet;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableMap.Builder;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import info.aduna.iteration.CloseableIteration;
+import info.aduna.iteration.EmptyIteration;
+
+/**
+ * TODO impl, test, doc
+ */
+@DefaultAnnotation(NonNull.class)
+public class EntityQueryNode extends ExternalSet implements 
ExternalBatchingIterator {
+
+/**
+ * The RyaURI that when used as the Predicate of a Statement Pattern 
indicates the Type of the Entities.
+ */
+private static final RyaURI TYPE_ID_URI = new 
RyaURI(RDF.TYPE.toString());
+
+// Provided at construction time.
+private final Type type;
+private final Collection patterns;
+private final EntityStorage entities;
+
+// Information about the subject of the patterns.
+private final boolean subjectIsConstant;
+private final Optional subjectConstant;
+private final Optional subjectVar;
+
+//since and EntityQueryNode exists in a single segment, all binding 
names are garunteed to be assured.
+private final Set bindingNames;
+
+// Information about the objects of the patterns.
+
+// XXX what does this map? property name -> binding variable?
+// for any property of the entity that has a variable, have to fill it 
in?
+private final ImmutableMap objectVariables;
+
+
+/**
+ * Constructs an instance of {@link EntityQueryNode}.
+ *
+ * @param type - The type of {@link Entity} this node matches. (not 
null)
+ * @param patterns - The query StatementPatterns that are solved using 
an
+ *   Entity of the Type. (not null)
+ * @param entities - The {@link EntityStorage} that will be searched 
to match
+ *   {@link BindingSet}s when evaluating a query. (not null)
+ */
+public EntityQueryNode(final Type type, final 
Collection patterns, final EntityStorage entities) throws 
IllegalStateException {
+this.type = requireNonNull(type);
+this.patterns = requireNonNull(patterns);
+this.entities = requireNonNull(entities);
+
  

[GitHub] incubator-rya pull request #135: Rya 144

[pujav65]

Github user pujav65 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r95009003
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/update/EntityIndexer.java
 ---
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.update;
+
+import org.apache.rya.api.persist.index.RyaSecondaryIndexer;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+
+/**
+ * Updates the {@link Entity}s that are in a {@link EntityStorage} when new
+ * {@link RyaStatement}s are added/removed from the Rya instance.
+ */
+public interface EntityIndexer extends RyaSecondaryIndexer {
--- End diff --

why doesn't this have any methods?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[pujav65]

Github user pujav65 commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r95007650
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/storage/CloseableIterator.java
 ---
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.indexing.entity.storage;
+
+import java.io.Closeable;
+import java.util.Iterator;
+
+/**
+ * An {@link Iterator} that is also {@link Closeable}.
+ *
+ * @param  - The type of object that will be iterated over.
+ */
+public interface CloseableIterator extends Iterator, Closeable { }
--- End diff --

why did we re-implement mango's closeableiterator
(org.calrissian.mango.collect.closeableiterator)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r94358355
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/EntityIndexSetProvider.java
 ---
@@ -0,0 +1,136 @@
+package org.apache.rya.indexing.entity;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Type;
+import org.apache.rya.indexing.entity.query.EntityQueryNode;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+import org.apache.rya.indexing.entity.storage.TypeStorage;
+import 
org.apache.rya.indexing.entity.storage.TypeStorage.TypeStorageException;
+import org.apache.rya.indexing.external.matching.ExternalSetProvider;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.QueryModelNode;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+
+import com.google.common.collect.HashMultimap;
+import com.google.common.collect.Multimap;
+
+/**
+ * Provides {@link EntityQueryNodes}s.
+ */
+public class EntityIndexSetProvider implements 
ExternalSetProvider {
+private Multimap typeMap;
+private Map subjectTypeMap;
+private final TypeStorage typeStorage;
+private final EntityStorage entityStorage;
+
+/**
+ * Creates a new {@link EntityIndexSetProvider}.
+ * @param typeStorage - The mechanism for access the various {@link 
Type}s. (not null)
+ * @param entityStorage - The mechanism for access to the various 
{@link Entity}s. (not null)
+ */
+public EntityIndexSetProvider(final TypeStorage typeStorage, final 
EntityStorage entityStorage) {
+this.typeStorage = requireNonNull(typeStorage);
+this.entityStorage = requireNonNull(entityStorage);
+}
+
+private Var getTypeSubject(final Type type) {
+//we just need the first pattern since all the patterns in this 
type map are the same subject
+final StatementPattern pattern = 
typeMap.get(type).iterator().next();
+return pattern.getSubjectVar();
+}
+
+private RyaURI getPredURI(final StatementPattern pattern) {
+final Var pred = pattern.getPredicateVar();
+return new RyaURI(pred.getValue().stringValue());
+}
+
+@Override
+public List getExternalSets(final 
QuerySegment node) {
+typeMap = HashMultimap.create();
+subjectTypeMap = new HashMap<>();
+
+//discover entities
+final List unused = new ArrayList<>();
+for (final QueryModelNode pattern : node.getOrderedNodes()) {
+if(pattern instanceof StatementPattern) {
+discoverEntities((StatementPattern) pattern, unused);
+}
+}
+
+final List nodes = new ArrayList<>();
+for(final Type type : typeMap.keySet()) {
+//replace all nodes in the tupleExpr of the collection of 
statement patterns with this node.
+final EntityQueryNode entity = new EntityQueryNode(type, 
typeMap.get(type), entityStorage);
+nodes.add(entity);
+}
+return nodes;
+}
+
+private void discoverEntities(final StatementPattern pattern, final 
List unmatched) {
+final Var subj = pattern.getSubjectVar();
+final String subjStr = subj.getName();
+final RyaURI predURI = getPredURI(pattern);
+//check to see if current node is type
+if(new URIImpl(predURI.getData()).equals(RDF.TYPE)) {
+final Var obj = pattern.getObjectVar();
+final RyaURI objURI = new RyaURI(obj.getValue().stringValue());
+try {
+final Optional optType = typeStorage.get(objURI);
+//if is type, fetch type add to subject -> type map
+if(optType.isPresent()) {
+final Type type = optType.get();
+typeMap.put(type, pattern);
+subjectTypeMap.put(subjStr, type);
+//check unmatched properties, add matches
+for(final StatementPattern propertyPattern : 
unmatched) {
+//store sp's as uri?
+final RyaURI 

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

Github user isper3at commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r94358316
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/EntityToSegmentConverter.java
 ---
@@ -0,0 +1,109 @@
+package org.apache.rya.indexing.entity;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.rya.indexing.entity.query.EntityQueryNode;
+import org.apache.rya.indexing.external.matching.ExternalSetConverter;
+import org.apache.rya.indexing.external.matching.JoinSegment;
+import org.apache.rya.indexing.external.matching.OptionalJoinSegment;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.openrdf.query.algebra.Filter;
+import org.openrdf.query.algebra.Join;
+import org.openrdf.query.algebra.LeftJoin;
+import org.openrdf.query.algebra.QueryModelNode;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.ValueExpr;
+import org.openrdf.query.algebra.helpers.QueryModelVisitorBase;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Implementation of {@link ExternalSetConverter} to convert {@link 
EntityQueryNode}s
+ * to {@link QuerySegment}s.
+ *
+ */
+public class EntityToSegmentConverter implements 
ExternalSetConverter {
+
+private static final EntityToOptionalJoinSegment optional = new 
EntityToOptionalJoinSegment();
+private static final EntityToJoinSegment join = new 
EntityToJoinSegment();
+
+
+@Override
+public QuerySegment setToSegment(final 
EntityQueryNode set) {
+Preconditions.checkNotNull(set);
+//if 
(PCJOptimizerUtilities.tupleContainsLeftJoins(set.getTupleExpr())) {
+return optional.getSegment(set);
+   // } else {
+  //  return join.getSegment(set);
+//}
+}
+
+/**
+ * This class extracts the {@link JoinSegment} from the {@link 
TupleExpr} of
+ * specified PCJ.
+ *
+ */
+static class EntityToJoinSegment extends 
QueryModelVisitorBase {
+
--- End diff --

This may be an artifact of what I did before and after your refactor.  if 
its isolated not used code maven is happy with it.  same below


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[meiercaleb]

Github user meiercaleb commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r94344553
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/EntityIndexOptimizer.java
 ---
@@ -0,0 +1,73 @@
+package org.apache.rya.indexing.entity;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.log4j.Logger;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Type;
+import org.apache.rya.indexing.entity.query.EntityQueryNode;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+import org.apache.rya.indexing.entity.storage.TypeStorage;
+import 
org.apache.rya.indexing.external.matching.AbstractExternalSetOptimizer;
+import org.apache.rya.indexing.external.matching.ExternalSetMatcher;
+import org.apache.rya.indexing.external.matching.ExternalSetProvider;
+import org.apache.rya.indexing.external.matching.QueryNodeListRater;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.apache.rya.indexing.external.matching.TopOfQueryFilterRelocator;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.Dataset;
+import org.openrdf.query.algebra.TupleExpr;
+
+import com.google.common.base.Optional;
+
+/**
+ * Optimizes a query over {@link Entity}s.
+ */
+public class EntityIndexOptimizer extends 
AbstractExternalSetOptimizer {
+private static final Logger log = 
Logger.getLogger(EntityIndexOptimizer.class);
+private static final EntityExternalSetMatcherFactory MATCHER_FACTORY = 
new EntityExternalSetMatcherFactory();
+private final EntityIndexSetProvider provider;
+
+/**
+ * Creates a new {@link EntityIndexOptimizer}.
+ * @param typeStorage - The mechanism for access the various {@link 
Type}s. (not null)
+ * @param entityStorage - The mechanism for access to the various 
{@link Entity}s. (not null)
+ */
+public EntityIndexOptimizer(final TypeStorage typeStorage, final 
EntityStorage entityStorage) {
--- End diff --

Unfortunately, you need to be able to create the Optimizer from  a 
Configuration object.  See RdfCloudTripleStoreConnection 316-335.  This is 
where your optimizer will be used.  You also need to add setters and getters to 
the MongoIndexingConfiguration object to configure Rya to add your optimizer as 
an additional optimizer.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[meiercaleb]

Github user meiercaleb commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r94295897
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/EntityToSegmentConverter.java
 ---
@@ -0,0 +1,109 @@
+package org.apache.rya.indexing.entity;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.rya.indexing.entity.query.EntityQueryNode;
+import org.apache.rya.indexing.external.matching.ExternalSetConverter;
+import org.apache.rya.indexing.external.matching.JoinSegment;
+import org.apache.rya.indexing.external.matching.OptionalJoinSegment;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.openrdf.query.algebra.Filter;
+import org.openrdf.query.algebra.Join;
+import org.openrdf.query.algebra.LeftJoin;
+import org.openrdf.query.algebra.QueryModelNode;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.ValueExpr;
+import org.openrdf.query.algebra.helpers.QueryModelVisitorBase;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Implementation of {@link ExternalSetConverter} to convert {@link 
EntityQueryNode}s
+ * to {@link QuerySegment}s.
+ *
+ */
+public class EntityToSegmentConverter implements 
ExternalSetConverter {
+
+private static final EntityToOptionalJoinSegment optional = new 
EntityToOptionalJoinSegment();
+private static final EntityToJoinSegment join = new 
EntityToJoinSegment();
+
+
+@Override
+public QuerySegment setToSegment(final 
EntityQueryNode set) {
+Preconditions.checkNotNull(set);
+//if 
(PCJOptimizerUtilities.tupleContainsLeftJoins(set.getTupleExpr())) {
+return optional.getSegment(set);
+   // } else {
+  //  return join.getSegment(set);
+//}
+}
+
+/**
+ * This class extracts the {@link JoinSegment} from the {@link 
TupleExpr} of
+ * specified PCJ.
+ *
+ */
+static class EntityToJoinSegment extends 
QueryModelVisitorBase {
+
--- End diff --

There is no tree structure to your EntityQueryNode.  Why are you using a 
visitor here?  Probably just a copy and paste issue.  Migrate the code in lines 
57 - 59 into the method starting at 36.   Or just have a getSegment method.  
This code does not need to be contained inside two distinct visitor classes 
(you're not even using one of them!)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #135: Rya 144

[meiercaleb]

Github user meiercaleb commented on a diff in the pull request:

https://github.com/apache/incubator-rya/pull/135#discussion_r94294942
  
--- Diff: 
extras/indexing/src/main/java/org/apache/rya/indexing/entity/EntityIndexSetProvider.java
 ---
@@ -0,0 +1,136 @@
+package org.apache.rya.indexing.entity;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.rya.api.domain.RyaURI;
+import org.apache.rya.indexing.entity.model.Entity;
+import org.apache.rya.indexing.entity.model.Type;
+import org.apache.rya.indexing.entity.query.EntityQueryNode;
+import org.apache.rya.indexing.entity.storage.EntityStorage;
+import org.apache.rya.indexing.entity.storage.TypeStorage;
+import 
org.apache.rya.indexing.entity.storage.TypeStorage.TypeStorageException;
+import org.apache.rya.indexing.external.matching.ExternalSetProvider;
+import org.apache.rya.indexing.external.matching.QuerySegment;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.algebra.QueryModelNode;
+import org.openrdf.query.algebra.StatementPattern;
+import org.openrdf.query.algebra.Var;
+
+import com.google.common.collect.HashMultimap;
+import com.google.common.collect.Multimap;
+
+/**
+ * Provides {@link EntityQueryNodes}s.
+ */
+public class EntityIndexSetProvider implements 
ExternalSetProvider {
+private Multimap typeMap;
+private Map subjectTypeMap;
+private final TypeStorage typeStorage;
+private final EntityStorage entityStorage;
+
+/**
+ * Creates a new {@link EntityIndexSetProvider}.
+ * @param typeStorage - The mechanism for access the various {@link 
Type}s. (not null)
+ * @param entityStorage - The mechanism for access to the various 
{@link Entity}s. (not null)
+ */
+public EntityIndexSetProvider(final TypeStorage typeStorage, final 
EntityStorage entityStorage) {
+this.typeStorage = requireNonNull(typeStorage);
+this.entityStorage = requireNonNull(entityStorage);
+}
+
+private Var getTypeSubject(final Type type) {
+//we just need the first pattern since all the patterns in this 
type map are the same subject
+final StatementPattern pattern = 
typeMap.get(type).iterator().next();
+return pattern.getSubjectVar();
+}
+
+private RyaURI getPredURI(final StatementPattern pattern) {
+final Var pred = pattern.getPredicateVar();
+return new RyaURI(pred.getValue().stringValue());
+}
+
+@Override
+public List getExternalSets(final 
QuerySegment node) {
+typeMap = HashMultimap.create();
+subjectTypeMap = new HashMap<>();
+
+//discover entities
+final List unused = new ArrayList<>();
+for (final QueryModelNode pattern : node.getOrderedNodes()) {
+if(pattern instanceof StatementPattern) {
+discoverEntities((StatementPattern) pattern, unused);
+}
+}
+
+final List nodes = new ArrayList<>();
+for(final Type type : typeMap.keySet()) {
+//replace all nodes in the tupleExpr of the collection of 
statement patterns with this node.
+final EntityQueryNode entity = new EntityQueryNode(type, 
typeMap.get(type), entityStorage);
+nodes.add(entity);
+}
+return nodes;
+}
+
+private void discoverEntities(final StatementPattern pattern, final 
List unmatched) {
+final Var subj = pattern.getSubjectVar();
+final String subjStr = subj.getName();
+final RyaURI predURI = getPredURI(pattern);
+//check to see if current node is type
+if(new URIImpl(predURI.getData()).equals(RDF.TYPE)) {
+final Var obj = pattern.getObjectVar();
+final RyaURI objURI = new RyaURI(obj.getValue().stringValue());
+try {
+final Optional optType = typeStorage.get(objURI);
+//if is type, fetch type add to subject -> type map
+if(optType.isPresent()) {
+final Type type = optType.get();
+typeMap.put(type, pattern);
+subjectTypeMap.put(subjStr, type);
+//check unmatched properties, add matches
+for(final StatementPattern propertyPattern : 
unmatched) {
+//store sp's as uri?
+final RyaURI 

[GitHub] incubator-rya pull request #135: Rya 144

[isper3at]

GitHub user isper3at opened a pull request:

https://github.com/apache/incubator-rya/pull/135

Rya 144

## Description
>What Changed?
-Finished kchilton2's mongo based entity secondary indexer
-updated to use meiercaleb's new query segment framework
-added optimizer for the entity indexer

### Tests
>Coverage?
Added integration tests for the indexer
Note: no assertions are performed

### Links
[Jira](https://issues.apache.org/jira/browse/RYA-144)

### Checklist
- [ ] Code Review
- [ ] Squash Commits

 People To Reivew
@meiercaleb 
@pujav65 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/isper3at/incubator-rya RYA-144

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rya/pull/135.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #135


commit 2f71ae5ab8a519114a9fd3f9128f77b0374fc299
Author: Kevin Chilton 
Date:   2016-09-16T22:22:54Z

RYA-144 Mongo DB Entity index.

commit 198f4189fafaae3e23d1b1e5f0914bba217f5919
Author: isper3at 
Date:   2016-11-02T15:36:51Z

[WIP]RYA-144: Added optimizer

commit 381c25e1f74bd49ba336d181eb981fc8ebed6f94
Author: Caleb Meier 
Date:   2016-12-13T20:26:29Z

RYA-144-ExternalSetMatchingFramework

commit e485858e3544f4cd630b3eacb75f56848325e748
Author: isper3at 
Date:   2016-12-19T19:24:25Z

[WIP] repackage mvm -> org.apache

commit bd9b581f7511772c4a26ae4e222e1d2dea465716
Author: isper3at 
Date:   2016-12-30T20:43:00Z

Added tests for the entity optimizer and updated to use new segment 
framework




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---