[ 
https://issues.apache.org/jira/browse/SOLR-8889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris M. Hostetter updated SOLR-8889:
-------------------------------------
    Attachment: SOLR-8889.patch
      Assignee: Chris M. Hostetter
        Status: Open  (was: Open)

My coworker Dan Fox and I have recently been investigating this issue.

To address some of the previous comments: I don't think, in general, that a 
"broadcast" solution is a good idea for the reasons alluded to in the linked 
issue. At a fundemental level: if a collection uses a router field, then it's 
reasonable to expect that a client sending updates should know that that and 
take responsibility for it – not just when adding documents, but when deleting 
documents as well.

That leaves us with the question of how to "fix" the current behavior when 
clients _do_ send routing info w/deleteById, and if we can prevent bugs like 
this from cropping up again in the future?
----
It seems like there are 3 key problems:

*1)* If an arbitrary solr node receives a deleteById w/ explicit "routing" 
information, that information is (at least in some code paths) “lost” when 
trying to decide what shard leader to forward the request to.

*2)* If CloudSolrClient is used to generate an UpdateRequest that includes a 
deleteById command with "routing" info, that info may be (at least in some code 
paths) “ignored” when deciding what shard leader to send that request – and 
(more significantly) can be completely “lost” when the request is actually sent 
to a solr node (meaning that even if #1 is fixed, the info won't be there for 
solr to do the right thing).

*3)* There does not appear to be any existing logic to “fail hard” in Solr if 
an update (add or delete) for a collection using {{router.field}} is processed 
w/o the field (or explicit {{_route_}}  param). Instead the update is silently 
sent to the wrong shard. (there might be a check/failure situation for this in 
the add/update case, but I don’t see it offhand, and certainly not in the 
deleteById case)

While Dan tracked down and fixed #1 and #2, I focused on testing, and spent 
some time looking into what it would take to "fix" #3 in some way – perhaps 
along the lines of Yonik's comment in SOLR-5890 suggesting a "strict" mode when 
using {{router.field}}) – because that seems like it would be the most robust 
way we can be "future proof" against any bugs (like #1 & #2 above) in 
solrj/solr-core code silently sending updates to the wrong shard leaders.

Unfortunately it doesn’t seem very viable to fix #3 w/o a LOT of serious 
changes to the DocRouter API. The general categories of problems I’m seeing are:
 * Sloppy/Loose API Contract
 ** some methods are documented to return “null” if there is not enough details 
to determine the correct Slice for an update, but some impls throw exceptions 
instead (some of the Update Chain processing code actually expects 
{{DocRouter}} to throw exceptions in spite of the API documentation
 ** some methods take in a “route” argument in addition to an "id" argument ... 
but not all methods
 ** some callers act aware of when a “route” has been specified by the user, 
and pass that the {{DocRouter}} methods as the "id" argument in place of the 
document's uniqueKey _instead_ of passing it where the “route” arg is suppose 
to be passed – so the DocRouter impls never really know whether the “id” they 
have been given is from the uniqueKey field or from the {{router.field}}
 ** Some code in Solr's Update Chain assumes a {{CompositeIdRouter}} is in use, 
and directly calls internal (but public) methods from the impl (which are not 
part of the DocRouter API) that don’t have access to the {{router.field}} let 
alone know if the input is a uniqueKey value or a {{router.field}} value (and 
worse: some of this code silently does nothing if some other DocRouter is used)
 * Overly Permissive Functional Design:
 ** IIUC it actually seems like {{router.field}} was designed with the 
expectation that if an "add" command includes a SolrInputDocument that doesn’t 
include a value in the router field, then the uniqueKey should be used as a 
callback to decide which shard to send it to. (ie: documents aren't required to 
have a value in the {{router.field}}

Ultimately i think the entire DocRouter API (and it’s usage in Solr) should be 
reconsidered, with a (strawman) goal of adding a {{strict}} option that (when 
true for a collection, and in some future version of solr it should default to 
true) would cause a DocRouter with {{router.field}} set to throw an Exception 
anytime any {{DocRouter}} caller calls any routing related method w/o 
specifying a value for the {{router.field}}. Unfortunately this seems like it 
would take more time/effort then I have to focus on it right now, so I plan to 
spin it off into it's own Improvement jira.

In the meantime, I'm attaching a patch that includes the following:
 * More robust testing of collections that have a {{router.field}}  and the 
client has sent deleteById commands with routing information
 ** via either the {{UpdateRequest.deleteById(...)}} or 
{{UpdateRequest.withRoute(...)}} methods
 * Fix for for #1 & #2 above to make these tests pass
 * Additional fixes & tests written by Yuki Yano and attached in SOLR-12694
 ** That problem also affects {{router.field}} (not just "implicit" router)
 ** the SOLR-12694.patch was heavily re-written to work with the "new" cloud 
based version of {{FullSolrCloudDistribCmdsTest}}

I think this patch is pretty solid and ready for commit ... but feedback always 
welcome.

> SolrCloud deleteById is broken when router.field is set
> -------------------------------------------------------
>
>                 Key: SOLR-8889
>                 URL: https://issues.apache.org/jira/browse/SOLR-8889
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud, SolrJ
>            Reporter: David Smiley
>            Assignee: Chris M. Hostetter
>            Priority: Major
>         Attachments: SOLR-8889.patch, SOLR_8889_investigation.patch
>
>
> If you set router.field on your collection to shard by something other than 
> the ID, then deleting documents by ID fails some of the time (how much 
> depends on how sharded the collection is).  I suspect that it'd work if the 
> IDs provided when deleting by ID were prefixed using the composite key syntax 
> -- "routekey!id" though I didn't check.  This is terrible.  Internally Solr 
> should broadcast to all the shards if there is no composite key prefix.
> Some affected code is UpdateRequest.getRoutes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to