dsmiley commented on code in PR #5328:
URL: https://github.com/apache/solr/pull/5328#discussion_r4630227280


##########
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java:
##########
@@ -355,6 +358,10 @@ public void process(ResponseBuilder rb) throws IOException 
{
         }
         docFetcher.decorateDocValueFields(
             doc, docid, docFetcher.getNonStoredDVsUsedAsStored(), 
reuseDvIters);
+
+        docFetcher.decorateWithRawVectorFields(

Review Comment:
   if `decorateDocValueFields` doesn't have `with` then let's not put it here.



##########
solr/core/src/java/org/apache/solr/schema/FieldProperties.java:
##########
@@ -73,7 +74,8 @@ public abstract class FieldProperties {
     "termPayloads",
     "useDocValuesAsStored",
     "large",
-    "uninvertible"
+    "uninvertible",
+    "useRawVectorsAsStored"

Review Comment:
   very very minor but let's sequence this after useDocValuesAsStored



##########
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java:
##########
@@ -905,6 +917,17 @@ private static SolrInputDocument 
toSolrInputDocument(SolrDocument doc, IndexSche
     return out;
   }
 
+  /**
+   * Returns whether a schema field should be kept when rebuilding a {@link 
SolrInputDocument} from
+   * the index. For instance, the value must be recoverable from doc values, a 
stored field, or
+   * Lucene's vector values when {@code useRawVectorsAsStored} is enabled. 
Copy-field targets are
+   * excluded because indexing regenerates them from their source fields.
+   */
+  private static boolean shouldKeepFieldInInputDocument(SchemaField field, 
IndexSchema schema) {
+    return (field.hasDocValues() || field.stored() || 
field.useRawVectorsAsStored())

Review Comment:
   I sense a `field.isRetrievable` would be useful... if we can find at least 2 
callers



##########
solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java:
##########
@@ -769,8 +788,62 @@ public Set<String> getNonStoredDVsWithoutCopyTargets() {
     return nonStoredDVsWithoutCopyTargets;
   }
 
+  public Set<String> calcRawVectorFieldsForReturn(ReturnFields returnFields) {

Review Comment:
   the DV equivalent is private; why is this public?



##########
solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc:
##########
@@ -621,6 +673,8 @@ So you should query a multivalued vector fields following 
the same syntax:
 
 In terms of rendering the results, you need the child transformer if you want 
to output them flat (you can choose to only return the best vector per result 
or all vectors):
 
+The same transformer syntax applies when the vector field uses 
`useRawVectorsAsStored="true"` and `stored="false"`.
+

Review Comment:
   would that be in doubt?  (seems like an obvious/distracting statement)



##########
solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java:
##########
@@ -358,45 +358,56 @@ private void addFlatMultiValuedVectorsToParent(
       VectorEncoding encoding) {
     for (String multiValuedVectorField : multiValuedVectorFields) {
       List<SolrDocument> solrDocuments = children.get(multiValuedVectorField);
+      if (solrDocuments == null || solrDocuments.isEmpty()) {
+        continue;
+      }
+
       List<List<Number>> multiValuedVectors = new 
ArrayList<>(solrDocuments.size());
-      for (SolrDocument singleVector : solrDocuments) {
-        List<Number> extractedVectors;
-        switch (encoding) {
-          case FLOAT32:
-            extractedVectors =
-                
this.extractFloatVector(singleVector.getFieldValues(multiValuedVectorField));
-            break;
-          case BYTE:
-            extractedVectors =
-                
this.extractByteVector(singleVector.getFieldValues(multiValuedVectorField));
-            break;
-          default:
-            throw new SolrException(
-                SolrException.ErrorCode.BAD_REQUEST, "Unsupported vector 
encoding: " + encoding);
+      for (SolrDocument singleVectorDocument : solrDocuments) {
+        var values = 
singleVectorDocument.getFieldValues(multiValuedVectorField);
+        if (values == null || values.isEmpty()) {
+          continue;
         }
-        multiValuedVectors.add(extractedVectors);
+
+        multiValuedVectors.add(extractVector(encoding, values));
       }
       parent.setField(multiValuedVectorField, multiValuedVectors);
     }
   }
 
+  private List<Number> extractVector(VectorEncoding encoding, 
Collection<Object> values) {
+    return switch (encoding) {
+      case FLOAT32 -> extractFloatVector(values);
+      case BYTE -> extractByteVector(values);
+      default ->
+          throw new SolrException(
+              SolrException.ErrorCode.SERVER_ERROR, "Unsupported vector 
encoding: " + encoding);
+    };
+  }
+
   private List<Number> extractFloatVector(Collection<Object> fieldValues) {
-    List<Number> vector = new ArrayList<>(fieldValues.size());
-    for (Object fieldValue : fieldValues) {
-      StoredField storedVectorValue = (StoredField) fieldValue;
-      vector.add(storedVectorValue.numericValue());
-    }
-    return vector;
+    return fieldValues.stream()
+        // If field values are Number instances that means they come from the 
vector values

Review Comment:
   versus what else?



##########
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java:
##########
@@ -614,6 +621,7 @@ private static SolrDocument fetchSolrDoc(
       SolrIndexSearcher searcher, int docId, ReturnFields returnFields) throws 
IOException {
     final SolrDocumentFetcher docFetcher = searcher.getDocFetcher();
     final SolrDocument solrDoc = docFetcher.solrDoc(docId, (SolrReturnFields) 
returnFields);
+    removeCopyFieldTargets(solrDoc, searcher.getSchema());

Review Comment:
   Sounds like an independent issue (bug? or optimization?) that isn't actually 
related to this one.



##########
solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java:
##########
@@ -769,8 +788,62 @@ public Set<String> getNonStoredDVsWithoutCopyTargets() {
     return nonStoredDVsWithoutCopyTargets;
   }
 
+  public Set<String> calcRawVectorFieldsForReturn(ReturnFields returnFields) {
+    if (derivedStoredVectorFields.isEmpty()) {
+      return Set.of();
+    }
+
+    if (returnFields.returnsAllFields()) {
+      return new HashSet<>(derivedStoredVectorFields);

Review Comment:
   weird to see a couple lines above return an immutable Set and here you do a 
copy.  this should consistently return either a mutable or immutable Set.  An 
alternative direction would be for the caller to make no such assumption and 
create a mutable version that it can manipulate.



##########
solr/core/src/java/org/apache/solr/update/processor/AtomicUpdateDocumentMerger.java:
##########
@@ -155,6 +163,18 @@ private SolrInputDocument mergeDocHavingSameId(
       final SolrInputDocument fromDoc, SolrInputDocument toDoc) {
     for (SolrInputField sif : fromDoc.values()) {
       Object val = sif.getValue();
+      SchemaField field = schema.getFieldOrNull(sif.getName());

Review Comment:
   agreed; feels out of scope



##########
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java:
##########
@@ -905,6 +917,17 @@ private static SolrInputDocument 
toSolrInputDocument(SolrDocument doc, IndexSche
     return out;
   }
 
+  /**
+   * Returns whether a schema field should be kept when rebuilding a {@link 
SolrInputDocument} from
+   * the index. For instance, the value must be recoverable from doc values, a 
stored field, or
+   * Lucene's vector values when {@code useRawVectorsAsStored} is enabled. 
Copy-field targets are
+   * excluded because indexing regenerates them from their source fields.
+   */
+  private static boolean shouldKeepFieldInInputDocument(SchemaField field, 
IndexSchema schema) {
+    return (field.hasDocValues() || field.stored() || 
field.useRawVectorsAsStored())

Review Comment:
   I think there's probably more to this logic that should be there.  a 
non-stored field with docValues, should probably be considered `false` if 
useDocValuesAsStored` is false.  Encoding the nuance and non-trivial rules here 
would be better encapsulated into the field/fieldtype.



##########
solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java:
##########
@@ -769,8 +788,62 @@ public Set<String> getNonStoredDVsWithoutCopyTargets() {
     return nonStoredDVsWithoutCopyTargets;
   }
 
+  public Set<String> calcRawVectorFieldsForReturn(ReturnFields returnFields) {
+    if (derivedStoredVectorFields.isEmpty()) {
+      return Set.of();

Review Comment:
   the DV equivalent of this method returns a mutable Set because it 
potentially manipulates it.  If I'm right (check me please), this should 
consistently return either a mutable or immutable Set.  An alternative 
direction would be for the caller to make no such assumption and create a 
mutable version that it can manipulate.



##########
solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java:
##########
@@ -769,8 +788,62 @@ public Set<String> getNonStoredDVsWithoutCopyTargets() {
     return nonStoredDVsWithoutCopyTargets;
   }
 
+  public Set<String> calcRawVectorFieldsForReturn(ReturnFields returnFields) {
+    if (derivedStoredVectorFields.isEmpty()) {
+      return Set.of();
+    }
+
+    if (returnFields.returnsAllFields()) {
+      return new HashSet<>(derivedStoredVectorFields);
+    }
+
+    if (returnFields.hasPatternMatching()) {
+      return 
derivedStoredVectorFields.stream().filter(returnFields::loadsField).collect(toSet());
+    } else {
+      return ofNullable(returnFields.getLuceneFieldNames())
+          .map(fields -> 
fields.stream().filter(derivedStoredVectorFields::contains))
+          .orElseGet(() -> 
derivedStoredVectorFields.stream().filter(returnFields::loadsField))
+          .collect(Collectors.toSet());
+    }
+  }
+
+  public void decorateWithRawVectorFields(
+      SolrDocument sdoc, int luceneDocId, Set<String> vectorFieldNames) throws 
IOException {

Review Comment:
   IMO `sdoc` is clearer, and a comment could say *what* we are "decorating"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to