[
https://issues.apache.org/jira/browse/SOLR-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15032230#comment-15032230
]
Shalin Shekhar Mangar commented on SOLR-8220:
---------------------------------------------
bq. On a different note, if we are going to tackle only the non-stored
docValues fields for now in this issue, does it now make sense to do this,
performance wise, at the DocTransformer instead of the SolrIndexSearcher?
I don't think there's any difference performance-wise. Changes to DocStreamer
should be enough as it is called only for writing the response and not the
entire result-set.
bq. At this point the question that remains; should we move forward with these
patches and move logic for retrieving dv fields to SolrIndexSearcher, leaving
out *, *_foo and other optimizations for now? i.e. retrieve fields by name, if
they exist in dv, but are not stored.
+1 let's create a patch to retrieve fields by name, if they exist in dv, but
are not stored. I also like Yonik's idea of bumping the schema version to have
fl=* return all fields (stored + non-stored docvalues) in 5.x and to include
both by default in trunk (6.x). So +1 to that as well.
bq. a very common case these days is that the entire index fits in memory.
I propose a middle ground. Let's use Lucene's spinning disk utility method and
prefer docvalues if we detect a SSD and fallback to reading from stored fields
otherwise. Let's discuss this optimization in SOLR-8344 and keep the two issues
separate.
> Read field from docValues for non stored fields
> -----------------------------------------------
>
> Key: SOLR-8220
> URL: https://issues.apache.org/jira/browse/SOLR-8220
> Project: Solr
> Issue Type: Improvement
> Reporter: Keith Laban
> Attachments: SOLR-8220-ishan.patch, SOLR-8220-ishan.patch,
> SOLR-8220-ishan.patch, SOLR-8220-ishan.patch, SOLR-8220.patch,
> SOLR-8220.patch, SOLR-8220.patch, SOLR-8220.patch, SOLR-8220.patch,
> SOLR-8220.patch, SOLR-8220.patch
>
>
> Many times a value will be both stored="true" and docValues="true" which
> requires redundant data to be stored on disk. Since reading from docValues is
> both efficient and a common practice (facets, analytics, streaming, etc),
> reading values from docValues when a stored version of the field does not
> exist would be a valuable disk usage optimization.
> The only caveat with this that I can see would be for multiValued fields as
> they would always be returned sorted in the docValues approach. I believe
> this is a fair compromise.
> I've done a rough implementation for this as a field transform, but I think
> it should live closer to where stored fields are loaded in the
> SolrIndexSearcher.
> Two open questions/observations:
> 1) There doesn't seem to be a standard way to read values for docValues,
> facets, analytics, streaming, etc, all seem to be doing their own ways,
> perhaps some of this logic should be centralized.
> 2) What will the API behavior be? (Below is my proposed implementation)
> Parameters for fl:
> - fl="docValueField"
> -- return field from docValue if the field is not stored and in docValues,
> if the field is stored return it from stored fields
> - fl="*"
> -- return only stored fields
> - fl="+"
> -- return stored fields and docValue fields
> 2a - would be easiest implementation and might be sufficient for a first
> pass. 2b - is current behavior
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
