[
https://issues.apache.org/jira/browse/SOLR-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266272#comment-13266272
]
Hoss Man commented on SOLR-2822:
--------------------------------
Yonik suggested that this issue might be a good place for me to start learning
more about the internals of Solr Cloud. First some background on what i
started thinking about (archived for posterity) but have since dismissed as a
bad idea...
{panel:title=My Initial Bad Idea That I Think Sucks| borderStyle=dashed|
borderColor=#ccc| titleBGColor=#F7D6C1| bgColor=#F7E6D1}
my first impression when yonik mentioned the problem was along the lines of the
initial bug summary...
bq. perhaps by using a processor chain dedicated to sub-updates
...at first glance, this seemed like it would be an easy way to deal with the
problem in a way that would be very easy to understand when looking at your
chains in the solrconfig.xml, and wouldn't involve any sort of "magic" that
might be hard for users to understand about how distributed updates work...
* some chain(s) (probably the default) would end with
DistributedUpdateProcessorFactory, which would be configured with a
"{{distrib.chain}}" init param value having some value {{$foo}}
* some other chain would be named {{$foo}}, and would end with
RunUpdateProcessorFactory
* when DistributedUpdateProcessorFactory was triggered, it would forward the
request to all of the nodes (including itself) using the configured
{{distrib.chain}}
But as i started looking at the code a bit more, i realized that was a
trivially naive understanding of what's needed here -- mainly because i
completely forgot about that fact that there are two types of forwarding:
* some arbitrary node gets the initial request from a client, and forwards to
the leader
* the leader does version checking/adding on the command/docs, and forwards to
*all* the nodes
This is when i started thinking that maybe there should be _three_ named
chains, and started understanding Jan's comment above about choosing which
logic should happen "on the leader node".
But the more i thought about the more the more the idea of specifying certain
logic needed to be run on the leader node made less sense. I can think of lots
of reasons for why you would care about whether an UpdateProcessor ran on one
node or N nodes (mainly the trade off between an expensive computation you
might only wnat to do once, vs processors that will add a lot of fields to the
request so you might want to defer until after distribution to minimize data
over the wire) but i couldn't come up with any compelling reason why, if you
have a processor you only want to execute _once_ you would care if it happens
on the first node randomly selected by the client, or if you really wanted to
ensure it happened on the leader -- particularly since leader
election/promotion can happen to any node at any time, so you couldn't even use
the leader logic to try and take advantage of some local resources on a
particular machine.
But even if we go back to the "two chain" model i suggested above, because of
the leader logic, the second chain would need to start with some
UpdateProcessor that handled the "version" & "forward to all nodes" work
currently handled by DistributedUpdateProcessor when "isLeader" is true. Which
means people editing their configs would need to not only remember to put some
sort of "SendDistributedUpdateProcessorFactory" at the end of the chains they
expect clients to use, but also some sort of
"RecieveDistributedUpdateProcessorFactory" at the beginning of the chain that
"SendDistributedUpdateProcessorFactory" would be pointing at (or maybe both are
just instances of DistributedUpdateProcessorFactory and we make it smart enough
to know when it's the start vs end of the chain) ... which starts sounding
really tedious and easy to screw up -- not to mention it doesn't play very
nicely with the idea of a hardcoded default chain users get for free if they
have none in their config; and it thoroughly complicates the story of someone
who starts with the example (no declared chains) in cloud mode and then wants
to customize a little, say by adding SignatureUpdateProcessor -- they wouldn't
be able to just configure a chain starting with SignatureUpdateProcessor,
they'd need to add two, and be careful about how they start and end.
{panel}
i'm all for transparency and eliminating "magic" by having things spelled out
in the configs, but even for me this idea was starting to seem like a serious
pain in the ass.
Which leads me to the idea i do think has merit...
{panel:title=My Current Idea That I Don't Think Sucks| borderStyle=dashed|
borderColor=#ccc| titleBGColor=#D6F7C1| bgColor=#E6F7D1}
I think in an ideal world...
* Solr Cloud users should be able to just declare a straightforwrd chain of
processors and not *need* to worry about which nodes they run on (ie: any chain
they used pre-cloud should still work)
* if a user *wants* to care what parts of the chain run once vs on every node,
it should be simple to indicate that decision by putting some "marker"
processor (ie: DistributedUpdateProcessorFactory) in the chain denoting when
distribution should happen.
* If you are using cloud mode, and don't specify where the distributed update
logic happens, then Solr should pick for you -- either "first" before any other
processors, or "last" just before RunUpdateProcessor ... I don't have an
opinion which is better, i'm sure other people who have been experimenting with
Solr Cloud for a while can tell me.
(I'm usually the person opposed to doing things magically behind the scenes
like this, but "Solr Cloud" is becoming a central enough concept in Solr, with
many components doing special things if "zkEnabled", that I think moving
forward it's "ok" to treat distributed updating as the norm in cloud mode and
optimize for it.)
The most straight forward way i can think of to do this would be:
* if Solr is in "cloud mode" then when SolrCore is initializing the chains
from solrconfig.xml, any chain that doesn't include
DistributedUpdateProcessorFactory should have it added automatically
(alternatively: maybe we only add it if RunUpdateProcessor is in the chain, so
anyone using chains w/o RunUpdateProcessor -- for some weird bizare purpose we
can't imagine -- won't be surprised by DistributedUpdateProcessorFactory
getting added magically)
* DistributedUpdateProcessor should add some param when forwarding to any
other node indicating that the request is being distributed
* when an update request is received, if it has this special param in it, then
any processor in the chain prior to DistributedUpdateProcessorFactory is skipped
This idea is very similar to part of what Jan suggested in his comment above...
bq. The Distrib processor could set some state info on the request to the next
node so that chain processing could continue where it left off. E.g.
&update.chain.nextproc=<name/id-of-next-proc>. This would require introduction
of named processor instances.
...but what i'm thinking of wouldn't require named processors and would be
specific to distributed updates (but wouldn't *precluded* named processors and
more enhanced logic down the road if someone wanted it).
I think this would be fairly feasible just by making some small modifications
to DistributedUpdateProcessor (to add the new special param when forwarding)
and UpdateRequestProcessorChain (to inject the
DistributedUpdateProcessorFactory if cloud mode, and to skip up to the
DistributedUpdateProcessorFactory if the param is set). I do however still
think we should generalize somewhat:
* DistributedUpdateProcessorFactory should be made to implement some marker
interface with no methods (ie: DistributedUpdateMarker)
* UpdateRequestProcessorChain.init should scan for instances of
DistributedUpdateMarker in the chain (instead of looking explicitly for
DistributedUpdateProcessorFactory) when deciding whether to inject a new
DistributedUpdateProcessorFactory into the chain
* UpdateRequestProcessorChain.createProcessor should scan for instances of
DistributedUpdateMarker in the chain (instead of looking explicitly for
DistributedUpdateProcessorFactory) when "skipping ahead" if the special param
is found in the request
...that way advanced users can implement their own distributed update processor
implementing that interface and register it explicitly in their chain if they
are so inclined, or implement a NoOp update processor implementing that
interface if they want to bipass the magic completley.
As a possible optimization/simplification to what gets sent over the wire, the
new param we add that UpdateRequestProcessorChain would start looking for to
"skip ahead" in the chain could replace the existing "leader" boolean param
DistributedUpdateProcessor currently uses (aka: {{SEEN_LEADER}}) by having an
enum style param (perhaps called "{{update.distrib}}" ?)...
* {{none}} - default if unset, means no distribution has happend
* {{toleader}} - means the request is being sent to the leader
* {{fromleader}} - means the leader is sending the request to all nodes
UpdateRequestProcessorChain would only care if the value is not "{{none}}", in
which case it would skip ahead to the DistributedUpdateMarker in the chain.
DistributedUpdateProcessorFactory would care if the value is "{{toleader}}" or
"{{fromleader}}" in which case it's logic would toggle in the same way it does
currently for SEEN_LEADER.
{panel}
So that's my idea. As i mentioned, this is really the first time i've looked
into any of the "Solr Cloud" internals, and it's very possible i may still be
missing something major. So if you spot any holes in this idea, please let me
know -- otherwise i'll try to take a stab at a patch in the next few days.
> don't run update processors twice
> ---------------------------------
>
> Key: SOLR-2822
> URL: https://issues.apache.org/jira/browse/SOLR-2822
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud, update
> Reporter: Yonik Seeley
> Fix For: 4.0
>
>
> An update will first go through processors until it gets to the point where
> it is forwarded to the leader (or forwarded to replicas if already on the
> leader).
> We need a way to skip over the processors that were already run (perhaps by
> using a processor chain dedicated to sub-updates?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
