[
https://issues.apache.org/jira/browse/SOLR-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132147#comment-18132147
]
Nick Shanin commented on SOLR-3473:
-----------------------------------
>From code reading against apache/main b73f03976fa,
>SignatureUpdateProcessorFactory now rejects the unsafe combinations in
>SolrCloud: overwriteDupes needs the signature field to be the uniqueKey
>(solr/core/src/java/org/apache/solr/update/processor/SignatureUpdateProcessorFactory.java:95-102).
> The safe setups described in the latest comment are the ones the code allows.
>[~markus17], if you see the unsafe case still succeed on a supported release,
>reply and I will reopen it. Otherwise I plan to close it as not a bug.
🤖 AI-generated 🤖
> Distributed deduplication broken when using non-uniqueKey for signatureField
> ----------------------------------------------------------------------------
>
> Key: SOLR-3473
> URL: https://issues.apache.org/jira/browse/SOLR-3473
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud, update
> Affects Versions: 4.0-ALPHA
> Reporter: Markus Jelsma
> Priority: Major
> Fix For: 4.9, 6.0
>
> Attachments: SOLR-3473-trunk-2.patch, SOLR-3473.patch, SOLR-3473.patch
>
>
> The current state of things (as of 8.8) is that
> SignatureUpdateProcessorFactory *CAN* be safely used in in SolrCloud for two
> possible usecases:
> * For de-duplication:
> ** the signatureField _MUST_ be the uniqueKey field *AND* the processor
> _MUST_ be configured to run prior to DistributedUpdateProcessor
> * Solely for generating signatures, w/o de-duplication
> ** overwriteDupes _MUST_ be set to false ... any signatureField may be used,
> and it may run at any point in the processor chain
> If you attempt to use SignatureUpdateProcessorFactory for de-duplication w/ a
> non-uniqueKey signature field, one of two failure situations are likely to
> arise:
> * in a multi-shard collection, documents with identical signatureField
> values will not be removed from any shard (leader) other then the one the
> document is routed to (by it's id)
> * even in a single-shard collection, with multiple replicas, documents with
> identical signatureField values will *only* be deleted on the 'leader' and
> not on any other replicas, because the leader does not propogate the
> {{AddUpdateCommand.updateTerm}} computed by the
> SignatureUpdateProcessorFactory to each of it's shards
> {panel:title=original bug report}
> Solr's deduplication via the SignatureUpdateProcessor is broken for
> distributed updates on SolrCloud.
> Mark Miller:
> {quote}
> Looking again at the SignatureUpdateProcessor code, I think that indeed this
> won't currently work with distrib updates. Could you file a JIRA issue for
> that? The problem is that we convert update commands into solr documents -
> and that can cause a loss of info if an update proc modifies the update
> command.
> I think the reason that you see a multiple values error when you try the
> other order is because of the lack of a document clone (the other issue I
> mentioned a few emails back). Addressing that won't solve your issue though -
> we have to come up with a way to propagate the currently lost info on the
> update command.
> {quote}
> Please see the ML thread for the full discussion:
> http://lucene.472066.n3.nabble.com/SolrCloud-deduplication-td3984657.html
> {panel}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
