[
https://issues.apache.org/jira/browse/SOLR-8889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris M. Hostetter updated SOLR-8889:
-------------------------------------
Attachment: SOLR-8889.patch
Assignee: Chris M. Hostetter
Status: Open (was: Open)
My coworker Dan Fox and I have recently been investigating this issue.
To address some of the previous comments: I don't think, in general, that a
"broadcast" solution is a good idea for the reasons alluded to in the linked
issue. At a fundemental level: if a collection uses a router field, then it's
reasonable to expect that a client sending updates should know that that and
take responsibility for it – not just when adding documents, but when deleting
documents as well.
That leaves us with the question of how to "fix" the current behavior when
clients _do_ send routing info w/deleteById, and if we can prevent bugs like
this from cropping up again in the future?
----
It seems like there are 3 key problems:
*1)* If an arbitrary solr node receives a deleteById w/ explicit "routing"
information, that information is (at least in some code paths) “lost” when
trying to decide what shard leader to forward the request to.
*2)* If CloudSolrClient is used to generate an UpdateRequest that includes a
deleteById command with "routing" info, that info may be (at least in some code
paths) “ignored” when deciding what shard leader to send that request – and
(more significantly) can be completely “lost” when the request is actually sent
to a solr node (meaning that even if #1 is fixed, the info won't be there for
solr to do the right thing).
*3)* There does not appear to be any existing logic to “fail hard” in Solr if
an update (add or delete) for a collection using {{router.field}} is processed
w/o the field (or explicit {{_route_}} param). Instead the update is silently
sent to the wrong shard. (there might be a check/failure situation for this in
the add/update case, but I don’t see it offhand, and certainly not in the
deleteById case)
While Dan tracked down and fixed #1 and #2, I focused on testing, and spent
some time looking into what it would take to "fix" #3 in some way – perhaps
along the lines of Yonik's comment in SOLR-5890 suggesting a "strict" mode when
using {{router.field}}) – because that seems like it would be the most robust
way we can be "future proof" against any bugs (like #1 & #2 above) in
solrj/solr-core code silently sending updates to the wrong shard leaders.
Unfortunately it doesn’t seem very viable to fix #3 w/o a LOT of serious
changes to the DocRouter API. The general categories of problems I’m seeing are:
* Sloppy/Loose API Contract
** some methods are documented to return “null” if there is not enough details
to determine the correct Slice for an update, but some impls throw exceptions
instead (some of the Update Chain processing code actually expects
{{DocRouter}} to throw exceptions in spite of the API documentation
** some methods take in a “route” argument in addition to an "id" argument ...
but not all methods
** some callers act aware of when a “route” has been specified by the user,
and pass that the {{DocRouter}} methods as the "id" argument in place of the
document's uniqueKey _instead_ of passing it where the “route” arg is suppose
to be passed – so the DocRouter impls never really know whether the “id” they
have been given is from the uniqueKey field or from the {{router.field}}
** Some code in Solr's Update Chain assumes a {{CompositeIdRouter}} is in use,
and directly calls internal (but public) methods from the impl (which are not
part of the DocRouter API) that don’t have access to the {{router.field}} let
alone know if the input is a uniqueKey value or a {{router.field}} value (and
worse: some of this code silently does nothing if some other DocRouter is used)
* Overly Permissive Functional Design:
** IIUC it actually seems like {{router.field}} was designed with the
expectation that if an "add" command includes a SolrInputDocument that doesn’t
include a value in the router field, then the uniqueKey should be used as a
callback to decide which shard to send it to. (ie: documents aren't required to
have a value in the {{router.field}}
Ultimately i think the entire DocRouter API (and it’s usage in Solr) should be
reconsidered, with a (strawman) goal of adding a {{strict}} option that (when
true for a collection, and in some future version of solr it should default to
true) would cause a DocRouter with {{router.field}} set to throw an Exception
anytime any {{DocRouter}} caller calls any routing related method w/o
specifying a value for the {{router.field}}. Unfortunately this seems like it
would take more time/effort then I have to focus on it right now, so I plan to
spin it off into it's own Improvement jira.
In the meantime, I'm attaching a patch that includes the following:
* More robust testing of collections that have a {{router.field}} and the
client has sent deleteById commands with routing information
** via either the {{UpdateRequest.deleteById(...)}} or
{{UpdateRequest.withRoute(...)}} methods
* Fix for for #1 & #2 above to make these tests pass
* Additional fixes & tests written by Yuki Yano and attached in SOLR-12694
** That problem also affects {{router.field}} (not just "implicit" router)
** the SOLR-12694.patch was heavily re-written to work with the "new" cloud
based version of {{FullSolrCloudDistribCmdsTest}}
I think this patch is pretty solid and ready for commit ... but feedback always
welcome.
> SolrCloud deleteById is broken when router.field is set
> -------------------------------------------------------
>
> Key: SOLR-8889
> URL: https://issues.apache.org/jira/browse/SOLR-8889
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud, SolrJ
> Reporter: David Smiley
> Assignee: Chris M. Hostetter
> Priority: Major
> Attachments: SOLR-8889.patch, SOLR_8889_investigation.patch
>
>
> If you set router.field on your collection to shard by something other than
> the ID, then deleting documents by ID fails some of the time (how much
> depends on how sharded the collection is). I suspect that it'd work if the
> IDs provided when deleting by ID were prefixed using the composite key syntax
> -- "routekey!id" though I didn't check. This is terrible. Internally Solr
> should broadcast to all the shards if there is no composite key prefix.
> Some affected code is UpdateRequest.getRoutes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]