dsmiley commented on code in PR #5328:
URL: https://github.com/apache/solr/pull/5328#discussion_r4630227280
##########
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java:
##########
@@ -355,6 +358,10 @@ public void process(ResponseBuilder rb) throws IOException
{
}
docFetcher.decorateDocValueFields(
doc, docid, docFetcher.getNonStoredDVsUsedAsStored(),
reuseDvIters);
+
+ docFetcher.decorateWithRawVectorFields(
Review Comment:
if `decorateDocValueFields` doesn't have `with` then let's not put it here.
##########
solr/core/src/java/org/apache/solr/schema/FieldProperties.java:
##########
@@ -73,7 +74,8 @@ public abstract class FieldProperties {
"termPayloads",
"useDocValuesAsStored",
"large",
- "uninvertible"
+ "uninvertible",
+ "useRawVectorsAsStored"
Review Comment:
very very minor but let's sequence this after useDocValuesAsStored
##########
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java:
##########
@@ -905,6 +917,17 @@ private static SolrInputDocument
toSolrInputDocument(SolrDocument doc, IndexSche
return out;
}
+ /**
+ * Returns whether a schema field should be kept when rebuilding a {@link
SolrInputDocument} from
+ * the index. For instance, the value must be recoverable from doc values, a
stored field, or
+ * Lucene's vector values when {@code useRawVectorsAsStored} is enabled.
Copy-field targets are
+ * excluded because indexing regenerates them from their source fields.
+ */
+ private static boolean shouldKeepFieldInInputDocument(SchemaField field,
IndexSchema schema) {
+ return (field.hasDocValues() || field.stored() ||
field.useRawVectorsAsStored())
Review Comment:
I sense a `field.isRetrievable` would be useful... if we can find at least 2
callers
##########
solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java:
##########
@@ -769,8 +788,62 @@ public Set<String> getNonStoredDVsWithoutCopyTargets() {
return nonStoredDVsWithoutCopyTargets;
}
+ public Set<String> calcRawVectorFieldsForReturn(ReturnFields returnFields) {
Review Comment:
the DV equivalent is private; why is this public?
##########
solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc:
##########
@@ -621,6 +673,8 @@ So you should query a multivalued vector fields following
the same syntax:
In terms of rendering the results, you need the child transformer if you want
to output them flat (you can choose to only return the best vector per result
or all vectors):
+The same transformer syntax applies when the vector field uses
`useRawVectorsAsStored="true"` and `stored="false"`.
+
Review Comment:
would that be in doubt? (seems like an obvious/distracting statement)
##########
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java:
##########
@@ -358,45 +358,56 @@ private void addFlatMultiValuedVectorsToParent(
VectorEncoding encoding) {
for (String multiValuedVectorField : multiValuedVectorFields) {
List<SolrDocument> solrDocuments = children.get(multiValuedVectorField);
+ if (solrDocuments == null || solrDocuments.isEmpty()) {
+ continue;
+ }
+
List<List<Number>> multiValuedVectors = new
ArrayList<>(solrDocuments.size());
- for (SolrDocument singleVector : solrDocuments) {
- List<Number> extractedVectors;
- switch (encoding) {
- case FLOAT32:
- extractedVectors =
-
this.extractFloatVector(singleVector.getFieldValues(multiValuedVectorField));
- break;
- case BYTE:
- extractedVectors =
-
this.extractByteVector(singleVector.getFieldValues(multiValuedVectorField));
- break;
- default:
- throw new SolrException(
- SolrException.ErrorCode.BAD_REQUEST, "Unsupported vector
encoding: " + encoding);
+ for (SolrDocument singleVectorDocument : solrDocuments) {
+ var values =
singleVectorDocument.getFieldValues(multiValuedVectorField);
+ if (values == null || values.isEmpty()) {
+ continue;
}
- multiValuedVectors.add(extractedVectors);
+
+ multiValuedVectors.add(extractVector(encoding, values));
}
parent.setField(multiValuedVectorField, multiValuedVectors);
}
}
+ private List<Number> extractVector(VectorEncoding encoding,
Collection<Object> values) {
+ return switch (encoding) {
+ case FLOAT32 -> extractFloatVector(values);
+ case BYTE -> extractByteVector(values);
+ default ->
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, "Unsupported vector
encoding: " + encoding);
+ };
+ }
+
private List<Number> extractFloatVector(Collection<Object> fieldValues) {
- List<Number> vector = new ArrayList<>(fieldValues.size());
- for (Object fieldValue : fieldValues) {
- StoredField storedVectorValue = (StoredField) fieldValue;
- vector.add(storedVectorValue.numericValue());
- }
- return vector;
+ return fieldValues.stream()
+ // If field values are Number instances that means they come from the
vector values
Review Comment:
versus what else?
##########
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java:
##########
@@ -614,6 +621,7 @@ private static SolrDocument fetchSolrDoc(
SolrIndexSearcher searcher, int docId, ReturnFields returnFields) throws
IOException {
final SolrDocumentFetcher docFetcher = searcher.getDocFetcher();
final SolrDocument solrDoc = docFetcher.solrDoc(docId, (SolrReturnFields)
returnFields);
+ removeCopyFieldTargets(solrDoc, searcher.getSchema());
Review Comment:
Sounds like an independent issue (bug? or optimization?) that isn't actually
related to this one.
##########
solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java:
##########
@@ -769,8 +788,62 @@ public Set<String> getNonStoredDVsWithoutCopyTargets() {
return nonStoredDVsWithoutCopyTargets;
}
+ public Set<String> calcRawVectorFieldsForReturn(ReturnFields returnFields) {
+ if (derivedStoredVectorFields.isEmpty()) {
+ return Set.of();
+ }
+
+ if (returnFields.returnsAllFields()) {
+ return new HashSet<>(derivedStoredVectorFields);
Review Comment:
weird to see a couple lines above return an immutable Set and here you do a
copy. this should consistently return either a mutable or immutable Set. An
alternative direction would be for the caller to make no such assumption and
create a mutable version that it can manipulate.
##########
solr/core/src/java/org/apache/solr/update/processor/AtomicUpdateDocumentMerger.java:
##########
@@ -155,6 +163,18 @@ private SolrInputDocument mergeDocHavingSameId(
final SolrInputDocument fromDoc, SolrInputDocument toDoc) {
for (SolrInputField sif : fromDoc.values()) {
Object val = sif.getValue();
+ SchemaField field = schema.getFieldOrNull(sif.getName());
Review Comment:
agreed; feels out of scope
##########
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java:
##########
@@ -905,6 +917,17 @@ private static SolrInputDocument
toSolrInputDocument(SolrDocument doc, IndexSche
return out;
}
+ /**
+ * Returns whether a schema field should be kept when rebuilding a {@link
SolrInputDocument} from
+ * the index. For instance, the value must be recoverable from doc values, a
stored field, or
+ * Lucene's vector values when {@code useRawVectorsAsStored} is enabled.
Copy-field targets are
+ * excluded because indexing regenerates them from their source fields.
+ */
+ private static boolean shouldKeepFieldInInputDocument(SchemaField field,
IndexSchema schema) {
+ return (field.hasDocValues() || field.stored() ||
field.useRawVectorsAsStored())
Review Comment:
I think there's probably more to this logic that should be there. a
non-stored field with docValues, should probably be considered `false` if
useDocValuesAsStored` is false. Encoding the nuance and non-trivial rules here
would be better encapsulated into the field/fieldtype.
##########
solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java:
##########
@@ -769,8 +788,62 @@ public Set<String> getNonStoredDVsWithoutCopyTargets() {
return nonStoredDVsWithoutCopyTargets;
}
+ public Set<String> calcRawVectorFieldsForReturn(ReturnFields returnFields) {
+ if (derivedStoredVectorFields.isEmpty()) {
+ return Set.of();
Review Comment:
the DV equivalent of this method returns a mutable Set because it
potentially manipulates it. If I'm right (check me please), this should
consistently return either a mutable or immutable Set. An alternative
direction would be for the caller to make no such assumption and create a
mutable version that it can manipulate.
##########
solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java:
##########
@@ -769,8 +788,62 @@ public Set<String> getNonStoredDVsWithoutCopyTargets() {
return nonStoredDVsWithoutCopyTargets;
}
+ public Set<String> calcRawVectorFieldsForReturn(ReturnFields returnFields) {
+ if (derivedStoredVectorFields.isEmpty()) {
+ return Set.of();
+ }
+
+ if (returnFields.returnsAllFields()) {
+ return new HashSet<>(derivedStoredVectorFields);
+ }
+
+ if (returnFields.hasPatternMatching()) {
+ return
derivedStoredVectorFields.stream().filter(returnFields::loadsField).collect(toSet());
+ } else {
+ return ofNullable(returnFields.getLuceneFieldNames())
+ .map(fields ->
fields.stream().filter(derivedStoredVectorFields::contains))
+ .orElseGet(() ->
derivedStoredVectorFields.stream().filter(returnFields::loadsField))
+ .collect(Collectors.toSet());
+ }
+ }
+
+ public void decorateWithRawVectorFields(
+ SolrDocument sdoc, int luceneDocId, Set<String> vectorFieldNames) throws
IOException {
Review Comment:
IMO `sdoc` is clearer, and a comment could say *what* we are "decorating"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]