[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla closed the pull request at: https://github.com/apache/lucene-solr/pull/416 --- - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213543737
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -124,10 +124,11 @@ public void testParentFilterLimitJSON() throws
Exception {
assertJQ(req("q", "type_s:donut",
"sort", "id asc",
-"fl", "id, type_s, toppings, _nest_path_, [child limit=1]",
+"fl", "id, type_s, lonely, lonelyGrandChild, test_s, test2_s,
_nest_path_, [child limit=1]",
--- End diff --
Leaving this as a TODO for another day sounds like a decent option.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213541579
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -124,10 +124,11 @@ public void testParentFilterLimitJSON() throws
Exception {
assertJQ(req("q", "type_s:donut",
"sort", "id asc",
-"fl", "id, type_s, toppings, _nest_path_, [child limit=1]",
+"fl", "id, type_s, lonely, lonelyGrandChild, test_s, test2_s,
_nest_path_, [child limit=1]",
--- End diff --
To my point I wrote in JIRA: It's sad that when I see this I have no idea
if it's right/wrong without having to go look at indexSampleData then think
about it. No? (this isn't a critique of you in particular; lots of tests
including some I've written look like the current tests here). imagine one
doc with some nested docs, all of which only have their ID. Since they only
have their ID, it's not a lot of literal text in JSON. The BeforeClass
unmatched docs cold use negative IDs to easily know who's who. Any way if you
would rather leave this as a "TODO" for another day then I understand.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213540255
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -123,6 +124,16 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
// Do we need to do anything with this doc (either ancestor or
matched the child query)
if (isAncestor || childDocSet == null ||
childDocSet.exists(docId)) {
+
+ if(limit != -1) {
+if(!isAncestor) {
+ if(matches == limit) {
+continue;
+ }
+ ++matches;
--- End diff --
I think matches should be incremented if it's in childDocSet (includes
childDocSet being null). Wether it's an ancestor or not doesn't matter I
think. You could pull out a new variable isInChildDocSet. Or I suppose simply
consider all a match; what I see what you just did as I write this; that's fine
too.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213321225
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -109,9 +109,14 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
// Loop each child ID up to the parent (exclusive).
for (int docId = calcDocIdToIterateFrom(lastChildId, rootDocId);
docId < rootDocId; ++docId) {
-// get the path. (note will default to ANON_CHILD_KEY if not in
schema or oddly blank)
+// get the path. (note will default to ANON_CHILD_KEY if schema
is not nested or empty string if blank)
String fullDocPath = getPathByDocId(docId - segBaseId,
segPathDocValues);
+if(isNestedSchema && !fullDocPath.contains(transformedDocPath)) {
+ // is not a descendant of the transformed doc, return fast.
+ return;
--- End diff --
Added another query to
[TestChildDocumentHierarchy#testNonRootChildren](https://github.com/apache/lucene-solr/pull/416/files#diff-9fe0ab006f82be5c6a07d5bb6dbc6da0R243).
This test failed before I changed the return to continue(previous commit),
and passes using the latest.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213319927
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -99,6 +96,9 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
// we'll need this soon...
final SortedDocValues segPathDocValues =
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+ // passing a different SortedDocValues obj since the child documents
which come after are of smaller docIDs,
+ // and the iterator can not be reversed.
+ final String transformedDocPath = getPathByDocId(segRootId,
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME));
--- End diff --
Sure thing.
Done :-)
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213306038
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -109,9 +109,14 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
// Loop each child ID up to the parent (exclusive).
for (int docId = calcDocIdToIterateFrom(lastChildId, rootDocId);
docId < rootDocId; ++docId) {
-// get the path. (note will default to ANON_CHILD_KEY if not in
schema or oddly blank)
+// get the path. (note will default to ANON_CHILD_KEY if schema
is not nested or empty string if blank)
String fullDocPath = getPathByDocId(docId - segBaseId,
segPathDocValues);
+if(isNestedSchema && !fullDocPath.contains(transformedDocPath)) {
+ // is not a descendant of the transformed doc, return fast.
+ return;
--- End diff --
Yep, you're right.
I'll investigate further to see why a test did not fail because of this.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213295374
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -99,6 +96,9 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
// we'll need this soon...
final SortedDocValues segPathDocValues =
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+ // passing a different SortedDocValues obj since the child documents
which come after are of smaller docIDs,
+ // and the iterator can not be reversed.
+ final String transformedDocPath = getPathByDocId(segRootId,
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME));
--- End diff --
Can you call this rootDocPath since we refer to this doc as the "root doc"
elsewhere here? I can see why you chose this name. You could add a comment
that the "root doc" is the input doc we are adding information to, and is
usually but not necessarily the root of the block of documents (i.e. the root
doc may itself be a child doc of another doc).
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213291939
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.ReaderUtil;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.util.BitSet;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class ChildDocTransformer extends DocTransformer {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String ANON_CHILD_KEY = "_childDocuments_";
+
+ private final String name;
+ private final BitSetProducer parentsFilter;
+ private final DocSet childDocSet;
+ private final int limit;
+ private final boolean isNestedSchema;
+
+ private final SolrReturnFields childReturnFields = new
SolrReturnFields();
+
+ ChildDocTransformer(String name, BitSetProducer parentsFilter,
+ DocSet childDocSet, boolean isNestedSchema, int
limit) {
+this.name = name;
+this.parentsFilter = parentsFilter;
+this.childDocSet = childDocSet;
+this.limit = limit;
+this.isNestedSchema = isNestedSchema;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+// note: this algorithm works if both if we have have _nest_path_ and
also if we don't!
+
+try {
+
+ // lookup what the *previous* rootDocId is, and figure which segment
this is
+ final SolrIndexSearcher searcher = context.getSearcher();
+ final List leaves =
searcher.getIndexReader().leaves();
+ final int seg = ReaderUtil.subIndex(rootDocId, leaves);
+ final LeafReaderContext leafReaderContext = leaves.get(seg);
+ final int segBaseId = leafReaderContext.docBase;
+ final int segRootId = rootDocId - segBaseId;
+ final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
+
+ final int segPrevRootId = segRootId==0? -1:
segParentsBitSet.prevSetBit(segRootId - 1); // can return -1 and that's okay
+
+ if(segPrevRootId == (segRootId - 1)) {
+// doc has no children, return fast
+return;
+ }
+
+ // we'll need this soon...
+ final SortedDocValues segPathDocValues =
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+ // passing a different SortedDocValues obj since the child documents
which come after are of smaller docIDs,
+ // and the iterator can not be reversed.
+ final String transformedDocPath = getPathByDocId(segRootId,
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME));
+
+
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213292329
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -109,9 +109,14 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
// Loop each child ID up to the parent (exclusive).
for (int docId = calcDocIdToIterateFrom(lastChildId, rootDocId);
docId < rootDocId; ++docId) {
-// get the path. (note will default to ANON_CHILD_KEY if not in
schema or oddly blank)
+// get the path. (note will default to ANON_CHILD_KEY if schema
is not nested or empty string if blank)
String fullDocPath = getPathByDocId(docId - segBaseId,
segPathDocValues);
+if(isNestedSchema && !fullDocPath.contains(transformedDocPath)) {
+ // is not a descendant of the transformed doc, return fast.
+ return;
--- End diff --
woah; shouldn't this be "continue"? We should have a test that would fail
on this bug. All it would take would be an additional child doc that is not
underneath the input root/transformed doc but follows it (as provided on
input). Some of the first docs we iterate over here might not descend from
rootDocId
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213270075
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.ReaderUtil;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.util.BitSet;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class ChildDocTransformer extends DocTransformer {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String ANON_CHILD_KEY = "_childDocuments_";
+
+ private final String name;
+ private final BitSetProducer parentsFilter;
+ private final DocSet childDocSet;
+ private final int limit;
+ private final boolean isNestedSchema;
+
+ private final SolrReturnFields childReturnFields = new
SolrReturnFields();
+
+ ChildDocTransformer(String name, BitSetProducer parentsFilter,
+ DocSet childDocSet, boolean isNestedSchema, int
limit) {
+this.name = name;
+this.parentsFilter = parentsFilter;
+this.childDocSet = childDocSet;
+this.limit = limit;
+this.isNestedSchema = isNestedSchema;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+// note: this algorithm works if both if we have have _nest_path_ and
also if we don't!
+
+try {
+
+ // lookup what the *previous* rootDocId is, and figure which segment
this is
+ final SolrIndexSearcher searcher = context.getSearcher();
+ final List leaves =
searcher.getIndexReader().leaves();
+ final int seg = ReaderUtil.subIndex(rootDocId, leaves);
+ final LeafReaderContext leafReaderContext = leaves.get(seg);
+ final int segBaseId = leafReaderContext.docBase;
+ final int segRootId = rootDocId - segBaseId;
+ final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
+
+ final int segPrevRootId = segRootId==0? -1:
segParentsBitSet.prevSetBit(segRootId - 1); // can return -1 and that's okay
+
+ if(segPrevRootId == (segRootId - 1)) {
+// doc has no children, return fast
+return;
+ }
+
+ // we'll need this soon...
+ final SortedDocValues segPathDocValues =
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+ // passing a different SortedDocValues obj since the child documents
which come after are of smaller docIDs,
+ // and the iterator can not be reversed.
+ final String transformedDocPath = getPathByDocId(segRootId,
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME));
+
+
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213202931
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -238,6 +238,17 @@ public void testSingularChildFilterJSON() throws
Exception {
tests);
}
+ @Test
+ public void testNonRootChildren() throws Exception {
+indexSampleData(numberOfDocsPerNestedTest);
+assertJQ(req("q", "test_s:testing",
+"sort", "id asc",
+"fl", "*,[child
childFilter='lonely/lonelyGrandChild/test2_s:secondTest'
parentFilter='_nest_path_:\"lonely/\"']",
--- End diff --
Perhaps I could try and remove the parentFilter requirement all together
for hierarchy based queries?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r212993235
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -92,6 +97,18 @@ private void testChildDoctransformerXML() {
assertQ(req("q", "*:*", "fq", "subject:\"parentDocument\" ",
"fl", "subject,[child parentFilter=\"subject:parentDocument\"
childFilter=\"title:foo\"]"), test2);
+try(SolrQueryRequest req = req("q", "*:*", "fq",
"subject:\"parentDocument\" ",
--- End diff --
Please include a comment explaining what it is you're testing here (i.e.
what is the point of this particular test); it's non-obvious to me.
This test tests manually... but it could be done more concisely using the
XPath style assertions done elsewhere in this file and most tests. For example
if you want to test the number of child documents, it'd be something like this:
"count(/response/result/doc[1]/doc)=2" and include a numFound check.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r213001520
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -238,6 +238,17 @@ public void testSingularChildFilterJSON() throws
Exception {
tests);
}
+ @Test
+ public void testNonRootChildren() throws Exception {
+indexSampleData(numberOfDocsPerNestedTest);
+assertJQ(req("q", "test_s:testing",
+"sort", "id asc",
+"fl", "*,[child
childFilter='lonely/lonelyGrandChild/test2_s:secondTest'
parentFilter='_nest_path_:\"lonely/\"']",
--- End diff --
Aha; I see you're requiring that a custom parentFilter be present in order
to use the childDocFilter on non-root docs. I suppose that's fine; it
something that could be enhanced in the future to avoid that inconvenience &
limitation(*). Since this is now required, the transformer should throw an
error to the user if the doc to be transformed isn't in that filter. For
example: "The document to be transformed must be matched by the parentFilter
of the child transformer" with BAD_REQUEST flag as it's user-error. And add a
test for this.
(*) An example of it being a limitation is this: Say the child docs are
all a "comment" in nature; and thus are recursive. The top query "q" might
match certain comments of interest. And we want all children of those comments
returned hierarchically. The parentFilter would end up having to match all
comments, but that would prevent returning child comments and thus blocking the
whole idea altogether :-(
To "fix" this limitation, we'd not constrain transformed docs to those in
the parentFilter, and we wouldn't insist on any special parentFilter. In the
loop that builds the hierarchy, when fetching the path, we'd need to skip over
the current doc if it doesn't descend from that of the doc to be transformed.
Seems pretty straight-forward.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211599253
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.ReaderUtil;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.util.BitSet;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class ChildDocTransformer extends DocTransformer {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String ANON_CHILD_KEY = "_childDocuments_";
+
+ private final String name;
+ private final BitSetProducer parentsFilter;
+ private final DocSet childDocSet;
+ private final int limit;
+
+ private final SolrReturnFields childReturnFields = new
SolrReturnFields();
+
+ ChildDocTransformer(String name, BitSetProducer parentsFilter,
+ DocSet childDocSet, int limit) {
+this.name = name;
+this.parentsFilter = parentsFilter;
+this.childDocSet = childDocSet;
+this.limit = limit;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+// note: this algorithm works if both if we have have _nest_path_ and
also if we don't!
+
+try {
+
+ // lookup what the *previous* rootDocId is, and figure which segment
this is
+ final SolrIndexSearcher searcher = context.getSearcher();
+ final List leaves =
searcher.getIndexReader().leaves();
+ final int seg = ReaderUtil.subIndex(rootDocId, leaves);
+ final LeafReaderContext leafReaderContext = leaves.get(seg);
+ final int segBaseId = leafReaderContext.docBase;
+ final int segRootId = rootDocId - segBaseId;
+ final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
+ final int segPrevRootId = segRootId==0? -1:
segParentsBitSet.prevSetBit(segRootId - 1); // can return -1 and that's okay
+
+ if(segPrevRootId == (segRootId - 1)) {
+// doc has no children, return fast
+return;
+ }
+
+ // we'll need this soon...
+ final SortedDocValues segPathDocValues =
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+
+ // the key in the Map is the document's ancestors key(one above the
parent), while the key in the intermediate
+ // MultiMap is the direct child document's key(of the parent
document)
+ Map>
pendingParentPathsToChildren = new HashMap<>();
+
+ IndexSchema schema = searcher.getSc
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211595662
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.ReaderUtil;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.util.BitSet;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class ChildDocTransformer extends DocTransformer {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String ANON_CHILD_KEY = "_childDocuments_";
+
+ private final String name;
+ private final BitSetProducer parentsFilter;
+ private final DocSet childDocSet;
+ private final int limit;
+
+ private final SolrReturnFields childReturnFields = new
SolrReturnFields();
+
+ ChildDocTransformer(String name, BitSetProducer parentsFilter,
+ DocSet childDocSet, int limit) {
+this.name = name;
+this.parentsFilter = parentsFilter;
+this.childDocSet = childDocSet;
+this.limit = limit;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+// note: this algorithm works if both if we have have _nest_path_ and
also if we don't!
+
+try {
+
+ // lookup what the *previous* rootDocId is, and figure which segment
this is
+ final SolrIndexSearcher searcher = context.getSearcher();
+ final List leaves =
searcher.getIndexReader().leaves();
+ final int seg = ReaderUtil.subIndex(rootDocId, leaves);
+ final LeafReaderContext leafReaderContext = leaves.get(seg);
+ final int segBaseId = leafReaderContext.docBase;
+ final int segRootId = rootDocId - segBaseId;
+ final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
+ final int segPrevRootId = segRootId==0? -1:
segParentsBitSet.prevSetBit(segRootId - 1); // can return -1 and that's okay
+
+ if(segPrevRootId == (segRootId - 1)) {
+// doc has no children, return fast
+return;
+ }
+
+ // we'll need this soon...
+ final SortedDocValues segPathDocValues =
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+
+ // the key in the Map is the document's ancestors key(one above the
parent), while the key in the intermediate
+ // MultiMap is the direct child document's key(of the parent
document)
+ Map>
pendingParentPathsToChildren = new HashMap<>();
+
+ IndexSchema schema = searcher.getSch
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211595043
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response.transform;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+import com.google.common.collect.Iterables;
+import org.apache.lucene.index.IndexableField;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.BasicResultContext;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestChildDocTransformerHierarchy extends SolrTestCaseJ4 {
+
+ private static AtomicInteger counter = new AtomicInteger();
+ private static final String[] types = {"donut", "cake"};
+ private static final String[] ingredients = {"flour", "cocoa",
"vanilla"};
+ private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
+ private static final String[] names = {"Yaz", "Jazz", "Costa"};
+ private static final String[] fieldsToRemove = {"_nest_parent_",
"_nest_path_", "_root_"};
+ private static final int sumOfDocsPerNestedDocument = 8;
+ private static final int numberOfDocsPerNestedTest = 10;
+ private static int randomDocTopId = 0;
+ private static String fqToExcludeNoneTestedDocs; // filter documents
that were created for random segments to ensure the transformer works with
multiple segments.
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+initCore("solrconfig-update-processor-chains.xml", "schema-nest.xml");
// use "nest" schema
+final boolean useSegments = random().nextBoolean();
+if(useSegments) {
+ // create random segments
+ final int numOfDocs = 10;
+ for(int i = 0; i < numOfDocs; ++i) {
+updateJ(generateDocHierarchy(i), params("update.chain", "nested"));
+if(random().nextBoolean()) {
+ assertU(commit());
+}
+ }
+ assertU(commit());
+ randomDocTopId = counter.get();
--- End diff --
I think this line should be at the end of this method, executed always. I
can see it works where it is now, executed conditionally, but I think it'd be
clearer if not done in a condition.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211594271
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response.transform;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+import com.google.common.collect.Iterables;
+import org.apache.lucene.index.IndexableField;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.BasicResultContext;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestChildDocTransformerHierarchy extends SolrTestCaseJ4 {
+
+ private static AtomicInteger counter = new AtomicInteger();
+ private static final String[] types = {"donut", "cake"};
+ private static final String[] ingredients = {"flour", "cocoa",
"vanilla"};
+ private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
+ private static final String[] names = {"Yaz", "Jazz", "Costa"};
+ private static final String[] fieldsToRemove = {"_nest_parent_",
"_nest_path_", "_root_"};
+ private static final int sumOfDocsPerNestedDocument = 8;
+ private static final int numberOfDocsPerNestedTest = 10;
+ private static int randomDocTopId = 0;
+ private static String fqToExcludeNoneTestedDocs; // filter documents
that were created for random segments to ensure the transformer works with
multiple segments.
--- End diff --
None -> Non
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211594079
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response.transform;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+import com.google.common.collect.Iterables;
+import org.apache.lucene.index.IndexableField;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.BasicResultContext;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestChildDocTransformerHierarchy extends SolrTestCaseJ4 {
+
+ private static AtomicInteger counter = new AtomicInteger();
+ private static final String[] types = {"donut", "cake"};
+ private static final String[] ingredients = {"flour", "cocoa",
"vanilla"};
+ private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
+ private static final String[] names = {"Yaz", "Jazz", "Costa"};
+ private static final String[] fieldsToRemove = {"_nest_parent_",
"_nest_path_", "_root_"};
+ private static final int sumOfDocsPerNestedDocument = 8;
+ private static final int numberOfDocsPerNestedTest = 10;
+ private static int randomDocTopId = 0;
--- End diff --
I suggest rename to "firstTestedDocId". And rename "counter" to
"idCounter".
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211519256
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -264,7 +309,7 @@ private static Object cleanIndexableField(Object field)
{
}
private static String grandChildDocTemplate(int id) {
-int docNum = id / 8; // the index of docs sent to solr in the
AddUpdateCommand. e.g. first doc is 0
+int docNum = (id / sumOfDocsPerNestedDocument) %
numberOfDocsPerNestedTest; // the index of docs sent to solr in the
AddUpdateCommand. e.g. first doc is 0
--- End diff --
I tried simplifying the calculation a bit, [here's a
link](https://github.com/apache/lucene-solr/pull/416/files#diff-9fe0ab006f82be5c6a07d5bb6dbc6da0R299).
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211505124
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -264,7 +309,7 @@ private static Object cleanIndexableField(Object field)
{
}
private static String grandChildDocTemplate(int id) {
-int docNum = id / 8; // the index of docs sent to solr in the
AddUpdateCommand. e.g. first doc is 0
+int docNum = (id / sumOfDocsPerNestedDocument) %
numberOfDocsPerNestedTest; // the index of docs sent to solr in the
AddUpdateCommand. e.g. first doc is 0
--- End diff --
the modulo is added to filter the docs in the other segments, and then
calculate the i that was passed, when constructing the nested document. This
then ensures the child transformer did not fail. If we can be satisfied by only
testing the ids, which does not seem as bullet-proof to me, this could be
removed, and only the ids will be tested.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211495503
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -40,22 +43,52 @@
private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
private static final String[] names = {"Yaz", "Jazz", "Costa"};
private static final String[] fieldsToRemove = {"_nest_parent_",
"_nest_path_", "_root_"};
+ private static final int sumOfDocsPerNestedDocument = 8;
+ private static final int numberOfDocsPerNestedTest = 10;
+ private static boolean useSegments;
+ private static int randomDocTopId = 0;
+ private static String filterOtherSegments;
@BeforeClass
public static void beforeClass() throws Exception {
initCore("solrconfig-update-processor-chains.xml", "schema-nest.xml");
// use "nest" schema
+useSegments = random().nextBoolean();
+if(useSegments) {
+ final int numOfDocs = 10;
+ for(int i = 0; i < numOfDocs; ++i) {
+updateJ(generateDocHierarchy(i), params("update.chain", "nested"));
+if(random().nextBoolean()) {
+ assertU(commit());
+}
+ }
+ assertU(commit());
+ randomDocTopId = counter.get();
+ filterOtherSegments = "{!frange l=" + randomDocTopId + "
incl=false}idInt";
+} else {
+ filterOtherSegments = "*:*";
+}
}
@After
public void after() throws Exception {
-clearIndex();
+if (!useSegments) {
--- End diff --
So I could simply use delQ using the filter,
great :-).
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211315461
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -40,22 +43,52 @@
private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
private static final String[] names = {"Yaz", "Jazz", "Costa"};
private static final String[] fieldsToRemove = {"_nest_parent_",
"_nest_path_", "_root_"};
+ private static final int sumOfDocsPerNestedDocument = 8;
+ private static final int numberOfDocsPerNestedTest = 10;
+ private static boolean useSegments;
+ private static int randomDocTopId = 0;
+ private static String filterOtherSegments;
@BeforeClass
public static void beforeClass() throws Exception {
initCore("solrconfig-update-processor-chains.xml", "schema-nest.xml");
// use "nest" schema
+useSegments = random().nextBoolean();
+if(useSegments) {
+ final int numOfDocs = 10;
+ for(int i = 0; i < numOfDocs; ++i) {
+updateJ(generateDocHierarchy(i), params("update.chain", "nested"));
+if(random().nextBoolean()) {
+ assertU(commit());
+}
+ }
+ assertU(commit());
+ randomDocTopId = counter.get();
+ filterOtherSegments = "{!frange l=" + randomDocTopId + "
incl=false}idInt";
+} else {
+ filterOtherSegments = "*:*";
+}
}
@After
public void after() throws Exception {
-clearIndex();
+if (!useSegments) {
--- End diff --
You could keep this if you want (use BeforeClass but have delete query in
after()). But if you do, it could be simplified: Don't conditionally call
"clearIndex"; simply always do this delete. And there's a simpler overloaded
version -- `delQ(queryhere)`
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211258577
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -40,22 +43,52 @@
private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
private static final String[] names = {"Yaz", "Jazz", "Costa"};
private static final String[] fieldsToRemove = {"_nest_parent_",
"_nest_path_", "_root_"};
+ private static final int sumOfDocsPerNestedDocument = 8;
--- End diff --
I understand we need a filter query here that can be referenced by the
test, but I'm a bit dubious on all these other ones. You may very well have
written the test in such a way that they are necessary at the moment, but lets
consider how to simplify.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211255381
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -40,22 +43,52 @@
private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
private static final String[] names = {"Yaz", "Jazz", "Costa"};
private static final String[] fieldsToRemove = {"_nest_parent_",
"_nest_path_", "_root_"};
+ private static final int sumOfDocsPerNestedDocument = 8;
+ private static final int numberOfDocsPerNestedTest = 10;
+ private static boolean useSegments;
+ private static int randomDocTopId = 0;
+ private static String filterOtherSegments;
@BeforeClass
public static void beforeClass() throws Exception {
initCore("solrconfig-update-processor-chains.xml", "schema-nest.xml");
// use "nest" schema
+useSegments = random().nextBoolean();
+if(useSegments) {
+ final int numOfDocs = 10;
+ for(int i = 0; i < numOfDocs; ++i) {
+updateJ(generateDocHierarchy(i), params("update.chain", "nested"));
+if(random().nextBoolean()) {
+ assertU(commit());
+}
+ }
+ assertU(commit());
+ randomDocTopId = counter.get();
+ filterOtherSegments = "{!frange l=" + randomDocTopId + "
incl=false}idInt";
+} else {
+ filterOtherSegments = "*:*";
+}
}
@After
public void after() throws Exception {
-clearIndex();
+if (!useSegments) {
--- End diff --
oh my suggestion of beforeClass didn't consider that you might have to do
this partial deletion. To make this simpler, move the new document addition
stuff in beforeClass into a before() (and make pertinent fields non-static),
then you can revert this after() to as it was.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211257783
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -40,22 +43,52 @@
private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
private static final String[] names = {"Yaz", "Jazz", "Costa"};
private static final String[] fieldsToRemove = {"_nest_parent_",
"_nest_path_", "_root_"};
+ private static final int sumOfDocsPerNestedDocument = 8;
+ private static final int numberOfDocsPerNestedTest = 10;
+ private static boolean useSegments;
+ private static int randomDocTopId = 0;
+ private static String filterOtherSegments;
--- End diff --
minor point: rename `filterOtherSegments` to `fqToExcludeNontestedDocs` and
add a comment explaining *why* we even have non-tested docs -- it's to perturb
the Lucene segments a bit to ensure the transformer works with and without docs
in other segments
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request: https://github.com/apache/lucene-solr/pull/416#discussion_r211259141 --- Diff: solr/core/src/test-files/solr/collection1/conf/schema-nest.xml --- @@ -20,6 +20,9 @@ + --- End diff -- minor: rename to `id_i` to follow typical naming conventions --- - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211256948
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -264,7 +309,7 @@ private static Object cleanIndexableField(Object field)
{
}
private static String grandChildDocTemplate(int id) {
-int docNum = id / 8; // the index of docs sent to solr in the
AddUpdateCommand. e.g. first doc is 0
+int docNum = (id / sumOfDocsPerNestedDocument) %
numberOfDocsPerNestedTest; // the index of docs sent to solr in the
AddUpdateCommand. e.g. first doc is 0
--- End diff --
This is looking kinda complicated now (same for fullNestedDocTemplate).
Does it really matter how the type_s value is chosen exactly? I don't know; I
confess to have glossed over this aspect of the test; I don't get the point. I
wonder if whatever the test is trying to truly test here might be simplified to
go about it in some other means.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r211108109
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -87,7 +87,12 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
final int segBaseId = leafReaderContext.docBase;
final int segRootId = rootDocId - segBaseId;
final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
- final int segPrevRootId = segParentsBitSet.prevSetBit(segRootId -
1); // can return -1 and that's okay
+ final int segPrevRootId = rootDocId==0? -1:
segParentsBitSet.prevSetBit(segRootId - 1); // can return -1 and that's okay
+
+ if(segPrevRootId == (rootDocId - 1)) {
--- End diff --
Yes,
I have added this conditional to the test.
Let's hope I did not misread your intentions :).
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210600024
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -87,7 +87,12 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
final int segBaseId = leafReaderContext.docBase;
final int segRootId = rootDocId - segBaseId;
final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
- final int segPrevRootId = segParentsBitSet.prevSetBit(segRootId -
1); // can return -1 and that's okay
+ final int segPrevRootId = rootDocId==0? -1:
segParentsBitSet.prevSetBit(segRootId - 1); // can return -1 and that's okay
+
+ if(segPrevRootId == (rootDocId - 1)) {
--- End diff --
Ooooh, good catch.
Lets enhance the tests in this file a bit to help us give confidence that
we're using docIDs correctly (and help avoid future enhancers/modifiers from
introducing similar bugs). Here's what I propose: in the @BeforeClass, if
random().nextBoolean(), add some nested docs -- using one of your existing
nested document adding methods. And randomly do a commit() to flush the
segment. Later in the test methods we need to add a filter query that will
exclude those docs. One way to do this is to ensure these first docs have some
field we can exclude. Another way might be knowing the maximum uniqueKey ID
you can query by prior to the test starting, and then adding a filter query
with a range saying the uniqueKey must be at least this value. Make sense?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210482225
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -87,7 +87,12 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
final int segBaseId = leafReaderContext.docBase;
final int segRootId = rootDocId - segBaseId;
final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
- final int segPrevRootId = segParentsBitSet.prevSetBit(segRootId -
1); // can return -1 and that's okay
+ final int segPrevRootId = rootDocId==0? -1:
segParentsBitSet.prevSetBit(segRootId - 1); // can return -1 and that's okay
+
+ if(segPrevRootId == (rootDocId - 1)) {
--- End diff --
If segRootId is 0, segParentsBitSet.prevSetBit(-1) throws an assertion
error, since the index has to be >= 0.
I will fix line 92
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210478700
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -87,7 +87,12 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
final int segBaseId = leafReaderContext.docBase;
final int segRootId = rootDocId - segBaseId;
final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
- final int segPrevRootId = segParentsBitSet.prevSetBit(segRootId -
1); // can return -1 and that's okay
+ final int segPrevRootId = rootDocId==0? -1:
segParentsBitSet.prevSetBit(segRootId - 1); // can return -1 and that's okay
+
+ if(segPrevRootId == (rootDocId - 1)) {
--- End diff --
You altered line 90 in response to my comment. I'm referring to line 92 --
`if(segPrevRootId == (rootDocId - 1))`, where my comment is.
Interesting though line 90 is different from the line of code I
committed to the feature branch. That line on the feature branch is:
`final int segPrevRootId = segParentsBitSet.prevSetBit(segRootId - 1); //
can return -1 and that's okay`
Notice there is no conditional, but there is in your version in this PR.
BitSet.prevSetBit will return -1 if given a 0 input (so says it's
documentation). That's what I was trying to say in my comment on the line of
code. Why did you change it?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210475699
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -291,4 +302,15 @@ private static String generateDocHierarchy(int i) {
"}\n" +
"}";
}
+
+ private static String IndexWoChildDocs() {
--- End diff --
Oops how embarrassing, guess I was in a hurry :(.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210322853
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -291,4 +302,15 @@ private static String generateDocHierarchy(int i) {
"}\n" +
"}";
}
+
+ private static String IndexWoChildDocs() {
--- End diff --
First, never start a method name with an uppercase letter (at least in
Java). But secondly, I suggest inlining this to the point of use since you're
only using it in one place. I know this is a stylistic point but I find it
harder to read code that refers to a bunch of other things in other places that
I have to go read -- it's disjointed; disrupts reading flow. Of course a good
deal of that is normal (calling string.whatever) but here it's only used for
this test.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210322181
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -87,7 +87,12 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
final int segBaseId = leafReaderContext.docBase;
final int segRootId = rootDocId - segBaseId;
final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
- final int segPrevRootId = segParentsBitSet.prevSetBit(segRootId -
1); // can return -1 and that's okay
+ final int segPrevRootId = rootDocId==0? -1:
segParentsBitSet.prevSetBit(segRootId - 1); // can return -1 and that's okay
+
+ if(segPrevRootId == (rootDocId - 1)) {
--- End diff --
you are comparing a segment local ID with a global ID which is incorrect.
You should refer to segRootId. This is why I'm particular about using "seg"
nomenclature in a body of code that deals with both segment and global IDs --
it makes it at least easier to identify such an error. It's difficult to get
tests to detect this; we'd need to commit some docs up front to cause more
segments to be created than many tests will do.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210309845
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -227,27 +225,28 @@ private static String getPathByDocId(int segDocId,
SortedDocValues segPathDocVal
/**
*
- * @param segDocBaseId base docID of the segment
- * @param RootId docID if the current root document
- * @param lastDescendantId lowest docID of the root document's descendant
- * @return the docID to loop and to not surpass limit of descendants to
match specified by query
+ * @param RootDocId docID if the current root document
+ * @param lowestChildDocId lowest docID of the root document's descendant
+ * @return the docID to loop and not surpass limit of descendants to
match specified by query
*/
- private int calcLimitIndex(int segDocBaseId, int RootId, int
lastDescendantId) {
-int i = segDocBaseId + RootId - 1; // the child document with the
highest docID
-final int prevSegRootId = segDocBaseId + lastDescendantId;
-assert prevSegRootId < i; // previous rootId should be smaller then
current RootId
+ private int calcDocIdToIterateFrom(int lowestChildDocId, int RootDocId) {
+assert lowestChildDocId < RootDocId; // first childDocId should be
smaller then current RootId
--- End diff --
Yes it would,
I'll add a test to ensure its behaviour.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210306003
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -227,27 +225,28 @@ private static String getPathByDocId(int segDocId,
SortedDocValues segPathDocVal
/**
*
- * @param segDocBaseId base docID of the segment
- * @param RootId docID if the current root document
- * @param lastDescendantId lowest docID of the root document's descendant
- * @return the docID to loop and to not surpass limit of descendants to
match specified by query
+ * @param RootDocId docID if the current root document
+ * @param lowestChildDocId lowest docID of the root document's descendant
+ * @return the docID to loop and not surpass limit of descendants to
match specified by query
*/
- private int calcLimitIndex(int segDocBaseId, int RootId, int
lastDescendantId) {
-int i = segDocBaseId + RootId - 1; // the child document with the
highest docID
-final int prevSegRootId = segDocBaseId + lastDescendantId;
-assert prevSegRootId < i; // previous rootId should be smaller then
current RootId
+ private int calcDocIdToIterateFrom(int lowestChildDocId, int RootDocId) {
+assert lowestChildDocId < RootDocId; // first childDocId should be
smaller then current RootId
--- End diff --
if the root doc has no children, this assertion would fail, right? Hmm;
should be tested we don't blow up/fail.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210294082
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -224,9 +225,29 @@ private static String getPathByDocId(int segDocId,
SortedDocValues segPathDocVal
return segPathDocValues.binaryValue().utf8ToString();
}
- private static String getSolrFieldString(Object fieldVal, FieldType
fieldType) {
-return fieldVal instanceof IndexableField
-? fieldType.toExternal((IndexableField)fieldVal)
-: fieldVal.toString();
+ /**
+ *
+ * @param segDocBaseId base docID of the segment
+ * @param RootId docID if the current root document
+ * @param lastDescendantId lowest docID of the root document's descendant
+ * @return the docID to loop and to not surpass limit of descendants to
match specified by query
+ */
+ private int calcLimitIndex(int segDocBaseId, int RootId, int
lastDescendantId) {
+int i = segDocBaseId + RootId - 1; // the child document with the
highest docID
--- End diff --
Changed and noted :)
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210291464
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -81,8 +81,8 @@ private void testChildDoctransformerXML() {
String test3[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='5'" };
+"/response/result/doc[1]/doc[1]/str[@name='id']='5'" ,
--- End diff --
The next sentence is a proposal for a different way to limit the number of
child documents.
Incase the current logic is sufficient, then we have no need to worry about
it.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210290075
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -81,8 +81,8 @@ private void testChildDoctransformerXML() {
String test3[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='5'" };
+"/response/result/doc[1]/doc[1]/str[@name='id']='5'" ,
--- End diff --
"If we have 3 child docs" -- the explanation in that sentence is
perfectly clear, I think. The next sentence confused me, but whatever.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210288829
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -81,8 +81,8 @@ private void testChildDoctransformerXML() {
String test3[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='5'" };
+"/response/result/doc[1]/doc[1]/str[@name='id']='5'" ,
--- End diff --
I guess I was not specific enough beforehand.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210287119
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -81,8 +81,8 @@ private void testChildDoctransformerXML() {
String test3[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='5'" };
+"/response/result/doc[1]/doc[1]/str[@name='id']='5'" ,
--- End diff --
since we initialize the document using from the highest index to exhaust
the limit if possible, it means some documents may be skipped. If we have 3
child docs: 1 , 2, 3, and the limit is set to "2", only doc 2 and 3 will be
added if we initialise the index to the one that will exhaust the limit.
Another way is simply to count the number of matching docs so far, and continue
onto the next root doc if reached(return from the transform method).
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210287034
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -81,8 +81,8 @@ private void testChildDoctransformerXML() {
String test3[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='5'" };
+"/response/result/doc[1]/doc[1]/str[@name='id']='5'" ,
--- End diff --
Again, the order hasn't changed ;-)
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210285891
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -81,8 +81,8 @@ private void testChildDoctransformerXML() {
String test3[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='5'" };
+"/response/result/doc[1]/doc[1]/str[@name='id']='5'" ,
--- End diff --
Yes,
I meant the order in which the documents are counted.
My bad.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210280357
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -81,8 +81,8 @@ private void testChildDoctransformerXML() {
String test3[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='5'" };
+"/response/result/doc[1]/doc[1]/str[@name='id']='5'" ,
--- End diff --
I suspect you are simply using the "sort" word inappropriately. "sort" is
related to ordering, but we didn't change the order. We did change the
"window" or "offset", if you will, into the docs to examine. Ordering/sort
hasn't changed.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210273819
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.ReaderUtil;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.util.BitSet;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class ChildDocTransformer extends DocTransformer {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String ANON_CHILD_KEY = "_childDocuments_";
+
+ private final String name;
+ private final BitSetProducer parentsFilter;
+ private final DocSet childDocSet;
+ private final int limit;
+
+ private final SolrReturnFields childReturnFields = new
SolrReturnFields();
+
+ ChildDocTransformer(String name, BitSetProducer parentsFilter,
+ DocSet childDocSet, int limit) {
+this.name = name;
+this.parentsFilter = parentsFilter;
+this.childDocSet = childDocSet;
+this.limit = limit;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+// note: this algorithm works if both if we have have _nest_path_ and
also if we don't!
+
+try {
+
+ // lookup what the *previous* rootDocId is, and figure which segment
this is
+ final SolrIndexSearcher searcher = context.getSearcher();
+ final List leaves =
searcher.getIndexReader().leaves();
+ final int seg = ReaderUtil.subIndex(rootDocId, leaves);
+ final LeafReaderContext leafReaderContext = leaves.get(seg);
+ final int segBaseId = leafReaderContext.docBase;
+ final int segRootId = rootDocId - segBaseId;
+ final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
+ final int segPrevRootId = segParentsBitSet.prevSetBit(segRootId -
1); // can return -1 and that's okay
+
+ // we'll need this soon...
+ final SortedDocValues segPathDocValues =
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+
+ // the key in the Map is the document's ancestors key(one above the
parent), while the key in the intermediate
+ // MultiMap is the direct child document's key(of the parent
document)
+ Map>
pendingParentPathsToChildren = new HashMap<>();
+
+ IndexSchema schema = searcher.getSchema();
+ SolrDocumentFetcher docFetcher = searcher
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210274031
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -224,9 +225,29 @@ private static String getPathByDocId(int segDocId,
SortedDocValues segPathDocVal
return segPathDocValues.binaryValue().utf8ToString();
}
- private static String getSolrFieldString(Object fieldVal, FieldType
fieldType) {
-return fieldVal instanceof IndexableField
-? fieldType.toExternal((IndexableField)fieldVal)
-: fieldVal.toString();
+ /**
+ *
+ * @param segDocBaseId base docID of the segment
+ * @param RootId docID if the current root document
+ * @param lastDescendantId lowest docID of the root document's descendant
+ * @return the docID to loop and to not surpass limit of descendants to
match specified by query
+ */
+ private int calcLimitIndex(int segDocBaseId, int RootId, int
lastDescendantId) {
+int i = segDocBaseId + RootId - 1; // the child document with the
highest docID
--- End diff --
I strongly suggest avoiding 'i' as a variable name unless it's a loop index
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210276463
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -83,6 +84,57 @@ public void testParentFilterJSON() throws Exception {
tests);
}
+ @Test
+ public void testParentFilterLimitJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
--- End diff --
why define these tests up front? The vast majority of "assertJQ" or
similar calls I've seen in Solr's tests put them inline at the method call,
which I think makes the most sense since it's together with the query. And can
the length be checked here? I think that's a key element of this test's
purpose :-) BTW if you use the XML based assertions, you have a richer
language to work with -- XPath. The json-like assertJQ is some home-grown
thing in this project that supposedly is easier for people to understand due to
the json-like nature (industry shifts from XML to JSON) but it's limited in
capability. I'm not sure if assertJQ can assert an array length but you could
investigate.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210271399
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -81,8 +81,8 @@ private void testChildDoctransformerXML() {
String test3[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='5'" };
+"/response/result/doc[1]/doc[1]/str[@name='id']='5'" ,
--- End diff --
How could choosing the index based on the "limit" change the document
ordering? My understanding is that child documents placed onto the parent via
this transformer are in docID order, which is the same order they were in as
provided to Solr.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210263030
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.ReaderUtil;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.util.BitSet;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.search.DocSet;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class ChildDocTransformer extends DocTransformer {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String ANON_CHILD_KEY = "_childDocuments_";
+
+ private final String name;
+ private final BitSetProducer parentsFilter;
+ private final DocSet childDocSet;
+ private final int limit;
+
+ private final SolrReturnFields childReturnFields = new
SolrReturnFields();
+
+ ChildDocTransformer(String name, BitSetProducer parentsFilter,
+ DocSet childDocSet, int limit) {
+this.name = name;
+this.parentsFilter = parentsFilter;
+this.childDocSet = childDocSet;
+this.limit = limit;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+// note: this algorithm works if both if we have have _nest_path_ and
also if we don't!
+
+try {
+
+ // lookup what the *previous* rootDocId is, and figure which segment
this is
+ final SolrIndexSearcher searcher = context.getSearcher();
+ final List leaves =
searcher.getIndexReader().leaves();
+ final int seg = ReaderUtil.subIndex(rootDocId, leaves);
+ final LeafReaderContext leafReaderContext = leaves.get(seg);
+ final int segBaseId = leafReaderContext.docBase;
+ final int segRootId = rootDocId - segBaseId;
+ final BitSet segParentsBitSet =
parentsFilter.getBitSet(leafReaderContext);
+ final int segPrevRootId = segParentsBitSet.prevSetBit(segRootId -
1); // can return -1 and that's okay
+
+ // we'll need this soon...
+ final SortedDocValues segPathDocValues =
DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+
+ // the key in the Map is the document's ancestors key(one above the
parent), while the key in the intermediate
+ // MultiMap is the direct child document's key(of the parent
document)
+ Map>
pendingParentPathsToChildren = new HashMap<>();
+
+ IndexSchema schema = searcher.getSchema();
+ SolrDocumentFetcher docFetcher = searche
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r210262386
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -81,8 +81,8 @@ private void testChildDoctransformerXML() {
String test3[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='5'" };
+"/response/result/doc[1]/doc[1]/str[@name='id']='5'" ,
--- End diff --
This had to be changed since we initialise the for loop in
ChildDocTransformer using the limit param, so the sort is docID descending now.
Thought it is important to point it out, and perhaps this should also be
documented.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request: https://github.com/apache/lucene-solr/pull/416#discussion_r209862802 --- Diff: solr/core/src/test-files/solr/collection1/conf/schema15.xml --- @@ -567,7 +567,17 @@ - + + + + --- End diff -- Sure thing --- - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r209070327
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
---
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class ChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+ private boolean hasPaths;
+ private boolean anonChildDoc;
+
+ public ChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final Query
childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+this.hasPaths = req.getSchema().hasExplicitField(NEST_PATH_FIELD_NAME);
+this.anonChildDoc =
req.getParams().getBool(CommonParams.ANONYMOUS_CHILD_DOCS, false);
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to perform a
ChildBlockJoinQuery
+return new String[] { idField.getName() };
+ }
+
+ public boolean hasPaths() {
+return hasPaths;
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList chi
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r209069776
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformerFactory.java
---
@@ -70,109 +73,59 @@ public DocTransformer create(String field, SolrParams
params, SolrQueryRequest r
}
String parentFilter = params.get( "parentFilter" );
-if( parentFilter == null ) {
- throw new SolrException( ErrorCode.BAD_REQUEST, "Parent filter
should be sent as parentFilter=filterCondition" );
-}
-
-String childFilter = params.get( "childFilter" );
-int limit = params.getInt( "limit", 10 );
-
BitSetProducer parentsFilter = null;
-try {
- Query parentFilterQuery = QParser.getParser( parentFilter,
req).getQuery();
- //TODO shouldn't we try to use the Solr filter cache, and then
ideally implement
- // BitSetProducer over that?
- // DocSet parentDocSet =
req.getSearcher().getDocSet(parentFilterQuery);
- // then return BitSetProducer with custom BitSet impl accessing the
docSet
- parentsFilter = new QueryBitSetProducer(parentFilterQuery);
-} catch (SyntaxError syntaxError) {
- throw new SolrException( ErrorCode.BAD_REQUEST, "Failed to create
correct parent filter query" );
-}
-
-Query childFilterQuery = null;
-if(childFilter != null) {
+boolean buildHierarchy = params.getBool("hierarchy", false);
+if( parentFilter == null) {
+ if(!buildHierarchy) {
+throw new SolrException( ErrorCode.BAD_REQUEST, "Parent filter
should be sent as parentFilter=filterCondition" );
+ }
+ parentsFilter = new QueryBitSetProducer(rootFilter);
+} else {
try {
-childFilterQuery = QParser.getParser( childFilter, req).getQuery();
+Query parentFilterQuery = QParser.getParser(parentFilter,
req).getQuery();
+//TODO shouldn't we try to use the Solr filter cache, and then
ideally implement
+// BitSetProducer over that?
+// DocSet parentDocSet =
req.getSearcher().getDocSet(parentFilterQuery);
+// then return BitSetProducer with custom BitSet impl accessing
the docSet
+parentsFilter = new QueryBitSetProducer(parentFilterQuery);
} catch (SyntaxError syntaxError) {
-throw new SolrException( ErrorCode.BAD_REQUEST, "Failed to create
correct child filter query" );
+throw new SolrException( ErrorCode.BAD_REQUEST, "Failed to create
correct parent filter query" );
}
}
-return new ChildDocTransformer( field, parentsFilter, uniqueKeyField,
req.getSchema(), childFilterQuery, limit);
- }
-}
-
-class ChildDocTransformer extends DocTransformer {
- private final String name;
- private final SchemaField idField;
- private final IndexSchema schema;
- private BitSetProducer parentsFilter;
- private Query childFilterQuery;
- private int limit;
-
- public ChildDocTransformer( String name, final BitSetProducer
parentsFilter,
- final SchemaField idField, IndexSchema
schema,
- final Query childFilterQuery, int limit) {
-this.name = name;
-this.idField = idField;
-this.schema = schema;
-this.parentsFilter = parentsFilter;
-this.childFilterQuery = childFilterQuery;
-this.limit = limit;
- }
+String childFilter = params.get( "childFilter" );
+int limit = params.getInt( "limit", 10 );
- @Override
- public String getName() {
-return name;
- }
-
- @Override
- public String[] getExtraRequestFields() {
-// we always need the idField (of the parent) in order to fill out
it's children
-return new String[] { idField.getName() };
+if(buildHierarchy) {
+ if(childFilter != null) {
+childFilter = buildHierarchyChildFilterString(childFilter);
+return new ChildDocTransformer(field, parentsFilter, req,
+getChildQuery(childFilter, req), limit);
+ }
+ return new ChildDocTransformer(field, parentsFilter, req, null,
limit);
+}
+return new ChildDocTransformer( field, parentsFilter, req,
+childFilter==null? null: getChildQuery(childFilter, req), limit);
}
- @Override
- public void transform(SolrDocument doc, int docid) {
-
-FieldType idFt = idField.getType();
-Object parentIdField = doc.getFirstValue(idField.getName());
-
-String parentIdExt = parentIdField instanceof IndexableField
- ? idFt.toExternal((IndexableField)parentIdField)
- : parentIdField.toString();
-
+ private stati
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r209066014
--- Diff:
solr/core/src/test/org/apache/solr/update/TestNestedUpdateProcessor.java ---
@@ -107,8 +107,8 @@ public void testDeeplyNestedURPGrandChild() throws
Exception {
};
indexSampleData(jDoc);
-assertJQ(req("q", IndexSchema.NEST_PATH_FIELD_NAME + ":*/grandChild#*",
-"fl","*",
+assertJQ(req("q", IndexSchema.NEST_PATH_FIELD_NAME + ":*/grandChild",
+"fl","*, _nest_path_",
--- End diff --
This change (and others here) to a test of an URP that isn't modified in
this issue underscores my point made previously having the test for that URP be
more of a unit test of what the URP produces (test the SolrInputDocument), and
_not_ executing queries. I'm not saying it was wrong to make this change in
this issue but just want you to reflect on the ramifications of these choices.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request: https://github.com/apache/lucene-solr/pull/416#discussion_r209071109 --- Diff: solr/core/src/test-files/solr/collection1/conf/schema15.xml --- @@ -567,7 +567,17 @@ - + + + + --- End diff -- Can you please provide an example input String (here in GH) that has multiple levels and comment out it looks like when it's done? I know how to read regexps but I need to stare at them a bit to figure them out, so lets make it easier to read/review. --- - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r209067568
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response.transform;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+import com.google.common.collect.Iterables;
+import org.apache.lucene.index.IndexableField;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.BasicResultContext;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestChildDocTransformerHierarchy extends SolrTestCaseJ4 {
--- End diff --
I'm glad you have split out this test. I want to further create a new
"schema-nest.xml" for our tests that explicitly deal with this new nested
document stuff. This test will use it, as well as TestNestedUpdateProcessor.
schema15.xml can be reverted to as it was. TestChildDocTransformer can
continue to use the legacy schema and as-such we can observe that our additions
to the underlying code don't disturb anyone who is using it that is unaware of
the new nested stuff.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r209070656
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformerFactory.java
---
@@ -61,6 +57,13 @@
*/
public class ChildDocTransformerFactory extends TransformerFactory {
+ public static final String PATH_SEP_CHAR = "/";
--- End diff --
I don't see why these are declared as Strings and not `char`. It's clumsy
later to do `charAt(0)`
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r209068211
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -242,10 +242,10 @@ private void testChildDocNonStoredDVFields() throws
Exception {
"fl", "*,[child parentFilter=\"subject:parentDocument\"]"), test1);
assertJQ(req("q", "*:*", "fq", "subject:\"parentDocument\" ",
-"fl", "subject,[child parentFilter=\"subject:parentDocument\"
childFilter=\"title:foo\"]"), test2);
+"fl", "id,_childDocuments_,subject,intDvoDefault,[child
parentFilter=\"subject:parentDocument\" childFilter=\"title:foo\"]"), test2);
--- End diff --
I'm really not a fan of {{anonChildDocs}} flag; I regret I conjured up the
idea. If we have "nest" schema fields, the user wants nested documents
(including field/label association), if the schema doesn't it ought to work as
it used to. I think this is straight-forward to reason about and document.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r208932604
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -242,10 +242,10 @@ private void testChildDocNonStoredDVFields() throws
Exception {
"fl", "*,[child parentFilter=\"subject:parentDocument\"]"), test1);
assertJQ(req("q", "*:*", "fq", "subject:\"parentDocument\" ",
-"fl", "subject,[child parentFilter=\"subject:parentDocument\"
childFilter=\"title:foo\"]"), test2);
+"fl", "id,_childDocuments_,subject,intDvoDefault,[child
parentFilter=\"subject:parentDocument\" childFilter=\"title:foo\"]"), test2);
--- End diff --
Of course, one day this will probably get removed. Right now there is a way
to provide the old XML format.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r208928799
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -242,10 +242,10 @@ private void testChildDocNonStoredDVFields() throws
Exception {
"fl", "*,[child parentFilter=\"subject:parentDocument\"]"), test1);
assertJQ(req("q", "*:*", "fq", "subject:\"parentDocument\" ",
-"fl", "subject,[child parentFilter=\"subject:parentDocument\"
childFilter=\"title:foo\"]"), test2);
+"fl", "id,_childDocuments_,subject,intDvoDefault,[child
parentFilter=\"subject:parentDocument\" childFilter=\"title:foo\"]"), test2);
--- End diff --
I think either the user hasn't yet started using the new key'ed/labeled
style of child documents, or they have updated completely. It's a migration to
a new way you either do or don't do (and perhaps one day will not have a
choice).
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r208813471
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -242,10 +242,10 @@ private void testChildDocNonStoredDVFields() throws
Exception {
"fl", "*,[child parentFilter=\"subject:parentDocument\"]"), test1);
assertJQ(req("q", "*:*", "fq", "subject:\"parentDocument\" ",
-"fl", "subject,[child parentFilter=\"subject:parentDocument\"
childFilter=\"title:foo\"]"), test2);
+"fl", "id,_childDocuments_,subject,intDvoDefault,[child
parentFilter=\"subject:parentDocument\" childFilter=\"title:foo\"]"), test2);
--- End diff --
I was thinking about what you said, and perhaps we could add the old format
using if the user did not index the field meta-data. I prefer having a flag,
since the user can query separate two collections and get the same output if
desired, instead of having to deal with two different XML formats.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r208813169
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -242,10 +242,10 @@ private void testChildDocNonStoredDVFields() throws
Exception {
"fl", "*,[child parentFilter=\"subject:parentDocument\"]"), test1);
assertJQ(req("q", "*:*", "fq", "subject:\"parentDocument\" ",
-"fl", "subject,[child parentFilter=\"subject:parentDocument\"
childFilter=\"title:foo\"]"), test2);
+"fl", "id,_childDocuments_,subject,intDvoDefault,[child
parentFilter=\"subject:parentDocument\" childFilter=\"title:foo\"]"), test2);
--- End diff --
I had to add a small addition and indeed, the behaviour was as expected.
I added a test to ensure there is backwards compatibility with the old XML
format
[here](https://github.com/apache/lucene-solr/blob/2573b89e465de615623084145cab7d17e9cb8a07/solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java#L100).
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r208807039
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -242,10 +242,10 @@ private void testChildDocNonStoredDVFields() throws
Exception {
"fl", "*,[child parentFilter=\"subject:parentDocument\"]"), test1);
assertJQ(req("q", "*:*", "fq", "subject:\"parentDocument\" ",
-"fl", "subject,[child parentFilter=\"subject:parentDocument\"
childFilter=\"title:foo\"]"), test2);
+"fl", "id,_childDocuments_,subject,intDvoDefault,[child
parentFilter=\"subject:parentDocument\" childFilter=\"title:foo\"]"), test2);
--- End diff --
This is what I would expect would happen. I am going to try it to check
this works as expected. Would not want to suddenly change this for everyone
:worried:
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r208799313
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -242,10 +242,10 @@ private void testChildDocNonStoredDVFields() throws
Exception {
"fl", "*,[child parentFilter=\"subject:parentDocument\"]"), test1);
assertJQ(req("q", "*:*", "fq", "subject:\"parentDocument\" ",
-"fl", "subject,[child parentFilter=\"subject:parentDocument\"
childFilter=\"title:foo\"]"), test2);
+"fl", "id,_childDocuments_,subject,intDvoDefault,[child
parentFilter=\"subject:parentDocument\" childFilter=\"title:foo\"]"), test2);
--- End diff --
ah; interesting. It's logical. Is this only needed for anonymous child
docs (thus \_childDocuments\_ or any/all possible relationship names that
aren't necessarily just at the root level but anywhere in the hierarchy?
Perhaps this is where that "anonChildDocs" ought to come into play again for
backwards-compatibility sake? Well perhaps not... someone who is using
anonymous child docs today will not have the nested field metadata and thus the
old logic will kick in and ensure child documents are added as it was; right?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r208507249
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformerHierarchy.java
---
@@ -32,7 +32,7 @@
import org.junit.BeforeClass;
import org.junit.Test;
-public class TestDeeplyNestedChildDocTransformer extends SolrTestCaseJ4 {
+public class TestChildDocTransformerHierarchy extends SolrTestCaseJ4 {
--- End diff --
These "hierarchy" tests are in a separate class since they require a
different schema and configuration to be loaded upon start-up.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r208507038
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -242,10 +242,10 @@ private void testChildDocNonStoredDVFields() throws
Exception {
"fl", "*,[child parentFilter=\"subject:parentDocument\"]"), test1);
assertJQ(req("q", "*:*", "fq", "subject:\"parentDocument\" ",
-"fl", "subject,[child parentFilter=\"subject:parentDocument\"
childFilter=\"title:foo\"]"), test2);
+"fl", "id,_childDocuments_,subject,intDvoDefault,[child
parentFilter=\"subject:parentDocument\" childFilter=\"title:foo\"]"), test2);
--- End diff --
Another major difference is that now since child documents are stored as
fields, the user needs to explicitly add them to the list of return fields. I
have no opinion regarding this issue, but this might pose a problem to some use
cases. Perhaps documenting this would be enough?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r208506508
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestChildDocTransformer.java
---
@@ -66,34 +66,34 @@ public void testAllParams() throws Exception {
private void testChildDoctransformerXML() {
String test1[] = new String[] {
"//*[@numFound='1']",
-"/response/result/doc[1]/doc[1]/str[@name='id']='2'" ,
-"/response/result/doc[1]/doc[2]/str[@name='id']='3'" ,
-"/response/result/doc[1]/doc[3]/str[@name='id']='4'" ,
-"/response/result/doc[1]/doc[4]/str[@name='id']='5'" ,
-"/response/result/doc[1]/doc[5]/str[@name='id']='6'" ,
-"/response/result/doc[1]/doc[6]/str[@name='id']='7'"};
+
"/response/result/doc[1]/arr[@name='_childDocuments_']/doc[1]/str[@name='id']='2'"
,
--- End diff --
One major difference is the way nested XML is returned, since now it has a
key it belongs to.
Thought it would make the review process easier for you if I pointed these
differences out.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r206901995
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestDeeplyNestedChildDocTransformer.java
---
@@ -168,35 +172,57 @@ private static String id() {
return "" + counter.incrementAndGet();
}
+ private static void cleanSolrDocumentFields(SolrDocument input) {
+for(Map.Entry field: input) {
+ Object val = field.getValue();
+ if(val instanceof Collection) {
+Object newVals = ((Collection) val).stream().map((item) ->
(cleanIndexableField(item)))
+.collect(Collectors.toList());
+input.setField(field.getKey(), newVals);
+continue;
+ } else {
+input.setField(field.getKey(),
cleanIndexableField(field.getValue()));
+ }
+}
+ }
+
+ private static Object cleanIndexableField(Object field) {
+if(field instanceof IndexableField) {
+ return ((IndexableField) field).stringValue();
+} else if(field instanceof SolrDocument) {
+ cleanSolrDocumentFields((SolrDocument) field);
+}
+return field;
+ }
+
private static String grandChildDocTemplate(int id) {
int docNum = id / 8; // the index of docs sent to solr in the
AddUpdateCommand. e.g. first doc is 0
-return
"SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS, type_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS], " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">, " +
-
"toppings=[SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
type_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ id + ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">, " +
-
"ingredients=[SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS], " +
-
"_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ (id + 3) + ">,
_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">}]}, " +
-
"SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
type_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ id + ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">, " +
-
"ingredients=[SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ (id + 5)+ ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">}, " +
-
"SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ (id + 5) + ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">}]}]}";
+return "SolrDocument{id="+ id + ", type_s=[" + types[docNum %
types.length] + "], name_s=[" + names[docNum % names.length] + "], " +
--- End diff --
Keeping one ID is fine; we certainly don't need additional ones. Maybe
consider using letters or names for IDs instead of incrementing counters.
Anything to help make reading a doc/child structure more readily apparent.
Anything to reduce string interpolation here is also a win IMO.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r206892630
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
--- End diff --
Yes; it can do both easily enough I think? A separate method could take
over for the existing/legacy case.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r206846562
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
--- End diff --
?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r206184346
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
--- End diff --
Would the transformer need to support the old method of adding
childDocuments to the _childDocuments_ field?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r206034965
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestDeeplyNestedChildDocTransformer.java
---
@@ -168,35 +172,57 @@ private static String id() {
return "" + counter.incrementAndGet();
}
+ private static void cleanSolrDocumentFields(SolrDocument input) {
+for(Map.Entry field: input) {
+ Object val = field.getValue();
+ if(val instanceof Collection) {
+Object newVals = ((Collection) val).stream().map((item) ->
(cleanIndexableField(item)))
+.collect(Collectors.toList());
+input.setField(field.getKey(), newVals);
+continue;
+ } else {
+input.setField(field.getKey(),
cleanIndexableField(field.getValue()));
+ }
+}
+ }
+
+ private static Object cleanIndexableField(Object field) {
+if(field instanceof IndexableField) {
+ return ((IndexableField) field).stringValue();
+} else if(field instanceof SolrDocument) {
+ cleanSolrDocumentFields((SolrDocument) field);
+}
+return field;
+ }
+
private static String grandChildDocTemplate(int id) {
int docNum = id / 8; // the index of docs sent to solr in the
AddUpdateCommand. e.g. first doc is 0
-return
"SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS, type_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS], " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">, " +
-
"toppings=[SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
type_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ id + ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">, " +
-
"ingredients=[SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS], " +
-
"_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ (id + 3) + ">,
_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">}]}, " +
-
"SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
type_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ id + ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">, " +
-
"ingredients=[SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ (id + 5)+ ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">}, " +
-
"SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ (id + 5) + ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">}]}]}";
+return "SolrDocument{id="+ id + ", type_s=[" + types[docNum %
types.length] + "], name_s=[" + names[docNum % names.length] + "], " +
--- End diff --
Perhaps we should only leave ID?
I would prefer to have one unique key to make sure the documents are for
sure placed under the right parent. Hopefully that will clean most of the noise.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r206012956
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestDeeplyNestedChildDocTransformer.java
---
@@ -168,35 +172,57 @@ private static String id() {
return "" + counter.incrementAndGet();
}
+ private static void cleanSolrDocumentFields(SolrDocument input) {
+for(Map.Entry field: input) {
+ Object val = field.getValue();
+ if(val instanceof Collection) {
+Object newVals = ((Collection) val).stream().map((item) ->
(cleanIndexableField(item)))
+.collect(Collectors.toList());
+input.setField(field.getKey(), newVals);
+continue;
+ } else {
+input.setField(field.getKey(),
cleanIndexableField(field.getValue()));
+ }
+}
+ }
+
+ private static Object cleanIndexableField(Object field) {
+if(field instanceof IndexableField) {
+ return ((IndexableField) field).stringValue();
+} else if(field instanceof SolrDocument) {
+ cleanSolrDocumentFields((SolrDocument) field);
+}
+return field;
+ }
+
private static String grandChildDocTemplate(int id) {
int docNum = id / 8; // the index of docs sent to solr in the
AddUpdateCommand. e.g. first doc is 0
-return
"SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS, type_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS], " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">, " +
-
"toppings=[SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
type_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ id + ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">, " +
-
"ingredients=[SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS], " +
-
"_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ (id + 3) + ">,
_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">}]}, " +
-
"SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
type_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ id + ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">, " +
-
"ingredients=[SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ (id + 5)+ ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">}, " +
-
"SolrDocument{id=stored,indexed,tokenized,omitNorms,indexOptions=DOCS,
name_s=[stored,indexed,tokenized,omitNorms,indexOptions=DOCS],
_nest_parent_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_nest_parent_:"
+ (id + 5) + ">, " +
-
"_root_=stored,indexed,tokenized,omitNorms,indexOptions=DOCS<_root_:" + id +
">}]}]}";
+return "SolrDocument{id="+ id + ", type_s=[" + types[docNum %
types.length] + "], name_s=[" + names[docNum % names.length] + "], " +
--- End diff --
It's a shame to have all this embedded ID calculation business. During
cleaning can they be removed (both "id" and nest parent and root) and we still
have enough distinguishing characteristics of the docs to know which is which?
Seems that way. It adds a lot of noise.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r206011853
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
--- End diff --
Oh right; I forgot we needed an ID to do the child doc query limited by
this parent ID. Please add a comment. I don't think we should bother with
root. I guess ChildDocTransformer might break if the id field is not stored
but did have docValues. That's a shame; it deserves a TODO.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r206011552
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
--- End diff --
Replace/update ChildDocTransformer, I think.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205965045
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
--- End diff --
Perhaps since _root_ is added to every document, we could use that field
instead of the ID field?
This is just a thought that popped into my head.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205962085
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestDeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response.transform;
+
+import java.util.Iterator;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.Iterables;
+import org.apache.lucene.document.StoredField;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.BasicResultContext;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestDeeplyNestedChildDocTransformer extends SolrTestCaseJ4 {
+
+ private static AtomicInteger counter = new AtomicInteger();
+ private static final char PATH_SEP_CHAR = '/';
+ private static final String[] types = {"donut", "cake"};
+ private static final String[] ingredients = {"flour", "cocoa",
"vanilla"};
+ private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
+ private static final String[] names = {"Yaz", "Jazz", "Costa"};
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+initCore("solrconfig-update-processor-chains.xml", "schema15.xml");
+ }
+
+ @After
+ public void after() throws Exception {
+assertU(delQ("*:*"));
+assertU(commit());
+counter.set(0); // reset id counter
+ }
+
+ @Test
+ public void testParentFilterJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
+"/response/docs/[0]/type_s==[donut]",
+"/response/docs/[0]/toppings/[0]/type_s==[Regular]",
+"/response/docs/[0]/toppings/[1]/type_s==[Chocolate]",
+"/response/docs/[0]/toppings/[0]/ingredients/[0]/name_s==[cocoa]",
+"/response/docs/[0]/toppings/[1]/ingredients/[1]/name_s==[cocoa]",
+"/response/docs/[0]/lonely/test_s==[testing]",
+"/response/docs/[0]/lonely/lonelyGrandChild/test2_s==[secondTest]",
+};
+
+try(SolrQueryRequest req = req("q", "type_s:donut", "sort", "id asc",
"fl", "*, _nest_path_, [child hierarchy=true]")) {
+ BasicResultContext res = (BasicResultContext)
h.queryAndResponse("/select", req).getResponse();
+ Iterator docsStreamer = res.getProcessedDocuments();
+ while (docsStreamer.hasNext()) {
+SolrDocument doc = docsStreamer.next();
+int currDocId = Integer.parseInt(((StoredField)
doc.getFirstValue("id")).stringValue());
+assertEquals("queried docs are not equal to expected output for
id: " + currDocId, fullNestedDocTemplate(currDocId), doc.toString());
+ }
+}
+
+assertJQ(req("q", "type_s:donut",
+"sort", "id asc",
+"fl", "*, _nest_path_, [child hierarchy=true]"),
+tests);
+ }
+
+ @Test
+ public void testExactPath() throws Exception {
+indexSampleData(2);
+String[] tests = {
+"/response/numFound==4",
+"/response/docs/[0]/_nest_path_=='toppings#0'",
+"/response/docs/[1]/_nest_path_=='toppings#0'",
+"/response/docs/[2]/_nest_path_=='toppings#1'",
+"/response/docs/[3]/_nest_path_=='toppings#1'",
+};
+
+assertJQ(req("q", "_nest_path_:*toppings/",
+"sort", "_nest_path_ asc",
+"fl", "*, _nest_path_"),
+tests);
+
+assertJQ(req("q", "+_nest_path_:\"toppings/\"",
+"sort", "_nest_path_ asc",
+"fl", "*, _nest_path_"),
+tests);
+ }
+
+ @Test
+ public void testChildFilterJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
+
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205961339
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
--- End diff --
FieldType idFt = idField.getType();
String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
Doesn't this mean we need the root document's ID?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205960639
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPa
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205960593
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
--- End diff --
Should I just delete the old transformer and make this one the new
ChildDocTransformer?
Or should I give it a new name like ChildDocHierarchyTransformer?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205851891
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestDeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response.transform;
+
+import java.util.Iterator;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.Iterables;
+import org.apache.lucene.document.StoredField;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.BasicResultContext;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestDeeplyNestedChildDocTransformer extends SolrTestCaseJ4 {
+
+ private static AtomicInteger counter = new AtomicInteger();
+ private static final char PATH_SEP_CHAR = '/';
+ private static final String[] types = {"donut", "cake"};
+ private static final String[] ingredients = {"flour", "cocoa",
"vanilla"};
+ private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
+ private static final String[] names = {"Yaz", "Jazz", "Costa"};
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+initCore("solrconfig-update-processor-chains.xml", "schema15.xml");
+ }
+
+ @After
+ public void after() throws Exception {
+assertU(delQ("*:*"));
+assertU(commit());
+counter.set(0); // reset id counter
+ }
+
+ @Test
+ public void testParentFilterJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
+"/response/docs/[0]/type_s==[donut]",
+"/response/docs/[0]/toppings/[0]/type_s==[Regular]",
+"/response/docs/[0]/toppings/[1]/type_s==[Chocolate]",
+"/response/docs/[0]/toppings/[0]/ingredients/[0]/name_s==[cocoa]",
+"/response/docs/[0]/toppings/[1]/ingredients/[1]/name_s==[cocoa]",
+"/response/docs/[0]/lonely/test_s==[testing]",
+"/response/docs/[0]/lonely/lonelyGrandChild/test2_s==[secondTest]",
+};
+
+try(SolrQueryRequest req = req("q", "type_s:donut", "sort", "id asc",
"fl", "*, _nest_path_, [child hierarchy=true]")) {
+ BasicResultContext res = (BasicResultContext)
h.queryAndResponse("/select", req).getResponse();
+ Iterator docsStreamer = res.getProcessedDocuments();
+ while (docsStreamer.hasNext()) {
+SolrDocument doc = docsStreamer.next();
+int currDocId = Integer.parseInt(((StoredField)
doc.getFirstValue("id")).stringValue());
+assertEquals("queried docs are not equal to expected output for
id: " + currDocId, fullNestedDocTemplate(currDocId), doc.toString());
+ }
+}
+
+assertJQ(req("q", "type_s:donut",
+"sort", "id asc",
+"fl", "*, _nest_path_, [child hierarchy=true]"),
+tests);
+ }
+
+ @Test
+ public void testExactPath() throws Exception {
+indexSampleData(2);
+String[] tests = {
+"/response/numFound==4",
+"/response/docs/[0]/_nest_path_=='toppings#0'",
+"/response/docs/[1]/_nest_path_=='toppings#0'",
+"/response/docs/[2]/_nest_path_=='toppings#1'",
+"/response/docs/[3]/_nest_path_=='toppings#1'",
+};
+
+assertJQ(req("q", "_nest_path_:*toppings/",
+"sort", "_nest_path_ asc",
+"fl", "*, _nest_path_"),
+tests);
+
+assertJQ(req("q", "+_nest_path_:\"toppings/\"",
+"sort", "_nest_path_ asc",
+"fl", "*, _nest_path_"),
+tests);
+ }
+
+ @Test
+ public void testChildFilterJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
+
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205468112
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPat
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205488727
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestDeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response.transform;
+
+import java.util.Iterator;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.Iterables;
+import org.apache.lucene.document.StoredField;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.BasicResultContext;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestDeeplyNestedChildDocTransformer extends SolrTestCaseJ4 {
+
+ private static AtomicInteger counter = new AtomicInteger();
+ private static final char PATH_SEP_CHAR = '/';
+ private static final String[] types = {"donut", "cake"};
+ private static final String[] ingredients = {"flour", "cocoa",
"vanilla"};
+ private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
+ private static final String[] names = {"Yaz", "Jazz", "Costa"};
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+initCore("solrconfig-update-processor-chains.xml", "schema15.xml");
+ }
+
+ @After
+ public void after() throws Exception {
+assertU(delQ("*:*"));
+assertU(commit());
+counter.set(0); // reset id counter
+ }
+
+ @Test
+ public void testParentFilterJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
+"/response/docs/[0]/type_s==[donut]",
+"/response/docs/[0]/toppings/[0]/type_s==[Regular]",
+"/response/docs/[0]/toppings/[1]/type_s==[Chocolate]",
+"/response/docs/[0]/toppings/[0]/ingredients/[0]/name_s==[cocoa]",
+"/response/docs/[0]/toppings/[1]/ingredients/[1]/name_s==[cocoa]",
+"/response/docs/[0]/lonely/test_s==[testing]",
+"/response/docs/[0]/lonely/lonelyGrandChild/test2_s==[secondTest]",
+};
+
+try(SolrQueryRequest req = req("q", "type_s:donut", "sort", "id asc",
"fl", "*, _nest_path_, [child hierarchy=true]")) {
+ BasicResultContext res = (BasicResultContext)
h.queryAndResponse("/select", req).getResponse();
+ Iterator docsStreamer = res.getProcessedDocuments();
+ while (docsStreamer.hasNext()) {
+SolrDocument doc = docsStreamer.next();
+int currDocId = Integer.parseInt(((StoredField)
doc.getFirstValue("id")).stringValue());
+assertEquals("queried docs are not equal to expected output for
id: " + currDocId, fullNestedDocTemplate(currDocId), doc.toString());
+ }
+}
+
+assertJQ(req("q", "type_s:donut",
+"sort", "id asc",
+"fl", "*, _nest_path_, [child hierarchy=true]"),
+tests);
+ }
+
+ @Test
+ public void testExactPath() throws Exception {
+indexSampleData(2);
+String[] tests = {
+"/response/numFound==4",
+"/response/docs/[0]/_nest_path_=='toppings#0'",
+"/response/docs/[1]/_nest_path_=='toppings#0'",
+"/response/docs/[2]/_nest_path_=='toppings#1'",
+"/response/docs/[3]/_nest_path_=='toppings#1'",
+};
+
+assertJQ(req("q", "_nest_path_:*toppings/",
+"sort", "_nest_path_ asc",
+"fl", "*, _nest_path_"),
+tests);
+
+assertJQ(req("q", "+_nest_path_:\"toppings/\"",
+"sort", "_nest_path_ asc",
+"fl", "*, _nest_path_"),
+tests);
+ }
+
+ @Test
+ public void testChildFilterJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
+
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205477055
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPat
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205466774
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
--- End diff --
As with our URP, lets forgo the "Deeply" terminology. I hope this will
simply be how any nested docs in the future are done rather than making a
distinction.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205475640
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPat
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205475272
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPat
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205480076
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPat
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205482972
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestDeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response.transform;
+
+import java.util.Iterator;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.Iterables;
+import org.apache.lucene.document.StoredField;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.BasicResultContext;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestDeeplyNestedChildDocTransformer extends SolrTestCaseJ4 {
+
+ private static AtomicInteger counter = new AtomicInteger();
+ private static final char PATH_SEP_CHAR = '/';
+ private static final String[] types = {"donut", "cake"};
+ private static final String[] ingredients = {"flour", "cocoa",
"vanilla"};
+ private static final Iterator ingredientsCycler =
Iterables.cycle(ingredients).iterator();
+ private static final String[] names = {"Yaz", "Jazz", "Costa"};
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+initCore("solrconfig-update-processor-chains.xml", "schema15.xml");
+ }
+
+ @After
+ public void after() throws Exception {
+assertU(delQ("*:*"));
+assertU(commit());
+counter.set(0); // reset id counter
+ }
+
+ @Test
+ public void testParentFilterJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
+"/response/docs/[0]/type_s==[donut]",
+"/response/docs/[0]/toppings/[0]/type_s==[Regular]",
+"/response/docs/[0]/toppings/[1]/type_s==[Chocolate]",
+"/response/docs/[0]/toppings/[0]/ingredients/[0]/name_s==[cocoa]",
+"/response/docs/[0]/toppings/[1]/ingredients/[1]/name_s==[cocoa]",
+"/response/docs/[0]/lonely/test_s==[testing]",
+"/response/docs/[0]/lonely/lonelyGrandChild/test2_s==[secondTest]",
+};
+
+try(SolrQueryRequest req = req("q", "type_s:donut", "sort", "id asc",
"fl", "*, _nest_path_, [child hierarchy=true]")) {
+ BasicResultContext res = (BasicResultContext)
h.queryAndResponse("/select", req).getResponse();
+ Iterator docsStreamer = res.getProcessedDocuments();
+ while (docsStreamer.hasNext()) {
+SolrDocument doc = docsStreamer.next();
+int currDocId = Integer.parseInt(((StoredField)
doc.getFirstValue("id")).stringValue());
+assertEquals("queried docs are not equal to expected output for
id: " + currDocId, fullNestedDocTemplate(currDocId), doc.toString());
+ }
+}
+
+assertJQ(req("q", "type_s:donut",
+"sort", "id asc",
+"fl", "*, _nest_path_, [child hierarchy=true]"),
+tests);
+ }
+
+ @Test
+ public void testExactPath() throws Exception {
+indexSampleData(2);
+String[] tests = {
+"/response/numFound==4",
+"/response/docs/[0]/_nest_path_=='toppings#0'",
+"/response/docs/[1]/_nest_path_=='toppings#0'",
+"/response/docs/[2]/_nest_path_=='toppings#1'",
+"/response/docs/[3]/_nest_path_=='toppings#1'",
+};
+
+assertJQ(req("q", "_nest_path_:*toppings/",
+"sort", "_nest_path_ asc",
+"fl", "*, _nest_path_"),
+tests);
+
+assertJQ(req("q", "+_nest_path_:\"toppings/\"",
+"sort", "_nest_path_ asc",
+"fl", "*, _nest_path_"),
+tests);
+ }
+
+ @Test
+ public void testChildFilterJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
+
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205478036
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPat
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205468966
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPat
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205474348
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPat
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205463666
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformerFactory.java
---
@@ -61,6 +71,12 @@
*/
public class ChildDocTransformerFactory extends TransformerFactory {
+ public static final String PATH_SEP_CHAR = "/";
+ public static final String NUM_SEP_CHAR = "#";
+ private static final BooleanQuery rootFilter = new BooleanQuery.Builder()
+ .add(new BooleanClause(new MatchAllDocsQuery(),
BooleanClause.Occur.MUST))
+ .add(new BooleanClause(new WildcardQuery(new
Term(NEST_PATH_FIELD_NAME, new BytesRef("*"))),
BooleanClause.Occur.MUST_NOT)).build();
--- End diff --
Remember again to use DocValuesExistsQuery
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205467807
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
--- End diff --
Oh? I didn't know we cared at all what the ID is.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205477315
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.response.transform;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.join.BitSetProducer;
+import org.apache.lucene.search.join.ToChildBlockJoinQuery;
+import org.apache.lucene.util.BytesRef;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.DocsStreamer;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrDocumentFetcher;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.search.SolrReturnFields;
+
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.NUM_SEP_CHAR;
+import static
org.apache.solr.response.transform.ChildDocTransformerFactory.PATH_SEP_CHAR;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+
+class DeeplyNestedChildDocTransformer extends DocTransformer {
+
+ private final String name;
+ protected final SchemaField idField;
+ protected final SolrQueryRequest req;
+ protected final IndexSchema schema;
+ private BitSetProducer parentsFilter;
+ protected int limit;
+ private final static Sort docKeySort = new Sort(new SortField(null,
SortField.Type.DOC, false));
+ private Query childFilterQuery;
+
+ public DeeplyNestedChildDocTransformer(String name, final BitSetProducer
parentsFilter,
+ final SolrQueryRequest req, final
Query childFilterQuery, int limit) {
+this.name = name;
+this.schema = req.getSchema();
+this.idField = this.schema.getUniqueKeyField();
+this.req = req;
+this.parentsFilter = parentsFilter;
+this.limit = limit;
+this.childFilterQuery = childFilterQuery;
+ }
+
+ @Override
+ public String getName() {
+return name;
+ }
+
+ @Override
+ public String[] getExtraRequestFields() {
+// we always need the idField (of the parent) in order to fill out
it's children
+return new String[] { idField.getName() };
+ }
+
+ @Override
+ public void transform(SolrDocument rootDoc, int rootDocId) {
+
+FieldType idFt = idField.getType();
+
+String rootIdExt =
getSolrFieldString(rootDoc.getFirstValue(idField.getName()), idFt);
+
+try {
+ Query parentQuery = idFt.getFieldQuery(null, idField, rootIdExt);
+ Query query = new ToChildBlockJoinQuery(parentQuery, parentsFilter);
+ SolrIndexSearcher searcher = context.getSearcher();
+ DocList children = searcher.getDocList(query, childFilterQuery,
docKeySort, 0, limit);
+ long segAndId = searcher.lookupId(new BytesRef(rootIdExt));
+ final int seg = (int) (segAndId >> 32);
+ final LeafReaderContext leafReaderContext =
searcher.getIndexReader().leaves().get(seg);
+ final SortedDocValues segPat
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205465998
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformerFactory.java
---
@@ -91,15 +100,37 @@ public DocTransformer create(String field, SolrParams
params, SolrQueryRequest r
Query childFilterQuery = null;
if(childFilter != null) {
- try {
-childFilterQuery = QParser.getParser( childFilter, req).getQuery();
- } catch (SyntaxError syntaxError) {
-throw new SolrException( ErrorCode.BAD_REQUEST, "Failed to create
correct child filter query" );
+ if(buildHierarchy) {
+childFilter = buildHierarchyChildFilterString(childFilter);
+return new DeeplyNestedChildDocTransformer(field, parentsFilter,
req,
+getChildQuery(childFilter, req), limit);
}
+ childFilterQuery = getChildQuery(childFilter, req);
+} else if(buildHierarchy) {
+ return new DeeplyNestedChildDocTransformer(field, parentsFilter,
req, null, limit);
}
return new ChildDocTransformer( field, parentsFilter, uniqueKeyField,
req.getSchema(), childFilterQuery, limit);
}
+
+ private static Query getChildQuery(String childFilter, SolrQueryRequest
req) {
+try {
+ return QParser.getParser( childFilter, req).getQuery();
+} catch (SyntaxError syntaxError) {
+ throw new SolrException( ErrorCode.BAD_REQUEST, "Failed to create
correct child filter query" );
+}
+ }
+
+ protected static String buildHierarchyChildFilterString(String
queryString) {
--- End diff --
Remember to provide input/output example. I think this is where the
PathHierarchyTokenizer might come into play... and our discussions on the JIRA
issue about that hierarchy. Can we table this for now and do in a follow-up
issue? (i.e. have no special syntax right now). I'm just concerned the scope
of this may be bigger than limited to this doc transformer since presumably
users will want to do join queries using this syntax as well. And this touches
on how we index this; which is kinda a bigger discussion than all the stuff
going on already in this issue. And this'll need to be documented in the Solr
Ref Guide well.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205465327
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformerFactory.java
---
@@ -70,36 +86,62 @@ public DocTransformer create(String field, SolrParams
params, SolrQueryRequest r
}
String parentFilter = params.get( "parentFilter" );
-if( parentFilter == null ) {
- throw new SolrException( ErrorCode.BAD_REQUEST, "Parent filter
should be sent as parentFilter=filterCondition" );
+BitSetProducer parentsFilter = null;
+boolean buildHierarchy = params.getBool("hierarchy", false);
+if( parentFilter == null) {
+ if(!buildHierarchy) {
+throw new SolrException( ErrorCode.BAD_REQUEST, "Parent filter
should be sent as parentFilter=filterCondition" );
+ }
+ parentsFilter = new QueryBitSetProducer(rootFilter);
+} else {
+ try {
+Query parentFilterQuery = QParser.getParser(parentFilter,
req).getQuery();
+//TODO shouldn't we try to use the Solr filter cache, and then
ideally implement
+// BitSetProducer over that?
+// DocSet parentDocSet =
req.getSearcher().getDocSet(parentFilterQuery);
+// then return BitSetProducer with custom BitSet impl accessing
the docSet
+parentsFilter = new QueryBitSetProducer(parentFilterQuery);
+ } catch (SyntaxError syntaxError) {
+throw new SolrException( ErrorCode.BAD_REQUEST, "Failed to create
correct parent filter query" );
+ }
}
String childFilter = params.get( "childFilter" );
int limit = params.getInt( "limit", 10 );
-BitSetProducer parentsFilter = null;
-try {
- Query parentFilterQuery = QParser.getParser( parentFilter,
req).getQuery();
- //TODO shouldn't we try to use the Solr filter cache, and then
ideally implement
- // BitSetProducer over that?
- // DocSet parentDocSet =
req.getSearcher().getDocSet(parentFilterQuery);
- // then return BitSetProducer with custom BitSet impl accessing the
docSet
- parentsFilter = new QueryBitSetProducer(parentFilterQuery);
-} catch (SyntaxError syntaxError) {
- throw new SolrException( ErrorCode.BAD_REQUEST, "Failed to create
correct parent filter query" );
-}
-
Query childFilterQuery = null;
if(childFilter != null) {
--- End diff --
The code flow from here to the end of the method looks very awkward to me.
I think the top "if" condition should test for buildHierarchy that we the
nested and non-nested cases are clearly separated. Do you think that would be
clear?
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user moshebla commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205403130
--- Diff:
solr/core/src/test/org/apache/solr/response/transform/TestDeeplyNestedChildDocTransformer.java
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.response.transform;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestDeeplyNestedChildDocTransformer extends SolrTestCaseJ4 {
+
+ private static AtomicInteger counter = new AtomicInteger();
+ private static final char PATH_SEP_CHAR = '/';
+ private static final String[] types = {"donut", "cake"};
+ private static final String[] ingredients = {"flour", "cocoa",
"vanilla"};
+ private static final String[] names = {"Yaz", "Jazz", "Costa"};
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+initCore("solrconfig-update-processor-chains.xml", "schema15.xml");
+ }
+
+ @After
+ public void after() throws Exception {
+assertU(delQ("*:*"));
+assertU(commit());
+ }
+
+ @Test
+ public void testParentFilterJSON() throws Exception {
+indexSampleData(10);
+String[] tests = new String[] {
--- End diff --
I have just updated this test, hopefully it is a lot better now.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request #416: WIP: SOLR-12519
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/416#discussion_r205126064
--- Diff:
solr/core/src/java/org/apache/solr/response/transform/DeeplyNestedChildDocTransformer.java
---
@@ -132,54 +126,49 @@ public void transform(SolrDocument rootDoc, int
rootDocId) {
// load the doc
SolrDocument doc =
DocsStreamer.convertLuceneDocToSolrDoc(docFetcher.doc(docId),
schema, new SolrReturnFields());
-doc.setField(NEST_PATH_FIELD_NAME, fullDocPath);
if (shouldDecorateWithDVs) {
docFetcher.decorateDocValueFields(doc, docId,
dvFieldsToReturn);
}
// get parent path
// put into pending
String parentDocPath = lookupParentPath(fullDocPath);
-pendingParentPathsToChildren.put(parentDocPath, doc); //
multimap add (won't replace)
-// if this path has pending child docs, add them.
-if (isAncestor) {
- addChildrenToParent(doc,
pendingParentPathsToChildren.get(fullDocPath));
- pendingParentPathsToChildren.removeAll(fullDocPath); // no
longer pending
+if(isAncestor) {
+ // if this path has pending child docs, add them.
+ addChildrenToParent(doc,
pendingParentPathsToChildren.remove(fullDocPath)); // no longer pending
}
+pendingParentPathsToChildren.computeIfAbsent(parentDocPath, x
-> ArrayListMultimap.create())
+.put(trimIfSingleDoc(getLastPath(fullDocPath)), doc); //
multimap add (won't replace)
}
}
// only children of parent remain
assert pendingParentPathsToChildren.keySet().size() == 1;
-addChildrenToParent(rootDoc,
pendingParentPathsToChildren.get(null));
+addChildrenToParent(rootDoc,
pendingParentPathsToChildren.remove(null));
}
} catch (IOException e) {
rootDoc.put(getName(), "Could not fetch child Documents");
}
}
- void addChildToParent(SolrDocument parent, SolrDocument child, String
label) {
-// lookup leaf key for these children using path
-// depending on the label, add to the parent at the right key/label
-// TODO: unfortunately this is the 2nd time we grab the paths for
these docs. resolve how?
-String trimmedPath = trimSuffixFromPaths(getLastPath(label));
-if (!parent.containsKey(trimmedPath) && (label.contains(NUM_SEP_CHAR)
&& !label.endsWith(NUM_SEP_CHAR))) {
- List list = new ArrayList<>();
- parent.setField(trimmedPath, list);
+ void addChildrenToParent(SolrDocument parent, Multimap children) {
+for(String childLabel: children.keySet()) {
--- End diff --
Ah, I see (I didn't look at Multimap's iteration options when I wrote
that). Your code here is good.
---
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
