[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518924#comment-16518924
]
Gus Heck commented on SOLR-11654:
-
Took the symptoms were quite odd... waiting for ever... and failing, sometimes
not finding shards. Turned out our waitCol() method needed the number of shards
passed in and then an assert needed it too.
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch, SOLR-11654.patch, SOLR-11654.patch,
> SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518348#comment-16518348
]
Gus Heck commented on SOLR-11654:
-
I'll look tonight
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch, SOLR-11654.patch, SOLR-11654.patch,
> SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518222#comment-16518222
]
David Smiley commented on SOLR-11654:
-
This new test takes a long time and I don't want it to take so long on average.
Randomization can help here so I chose shards & replicas randomly with the
high end being the numbers you chose. The patch shows this (and other
changes). However this test failed on me on waitFor the collections after the
doc is added. Try seed -Dtests.seed=613B69692267C615 Can you investigate
[~gus_heck]? The # shards & replicas should have no bearing on the test up to
this point.
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch, SOLR-11654.patch, SOLR-11654.patch,
> SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517404#comment-16517404
]
Gus Heck commented on SOLR-11654:
-
The other additions were syntactic sugar, so whatever folks think there is
fine. It just seemed to me that varargs was likely just something nobody
bothered to add yet, though certainly it's great to discuss all of this. For
the method that you pasted above I agree. It would be a good idea to make a
copy of the params and note that in the javadoc for clarity. I'm not so sure
about setting the type to SolrParams however since we then have to add error
logic to account for the fact that seParams takes a ModifiableSolrParams
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch, SOLR-11654.patch, SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517065#comment-16517065
]
David Smiley commented on SOLR-11654:
-
It's not a great ease to add this to SolrClient admittedly (IMO)... and it's
easy enough to add a utility method on this test which does this equivalent
logic. I was about to point out that UpdateRequest's setter methods return
"this" which can make usage even more compact but this isn't done uniformly
here (which is sloppy) – setParams doesn't.
Any way I guess we'll leave SolrClient alone. Thanks for your input Varun.
Yes it's a shame there are so many darned overloaded methods... I think it's a
large part due to the optional "collection" parameter which like doubles the
methods! I've been bitten several times writing SolrJ code that doesn't use
the right overloaded version (forgot to specify collection). I think for 8.0,
*either* all SolrClient methods without "collection" can be removed in favor of
insisting you use the overloaded variant accepting a collection, *or*
SolrClient itself could be locked down to one collection at the time you create
it *or* have a CollectionSolrClient interface retrieved from a
SolrClient.withCollection(collection) in which all the operations that require
a SolrClient are on that interface and not SolrClient proper. Several ideas to
consider.
Gus I'll move the logic out of SolrClient and commit later. I ran the tests
the other day via beasting + precommit and it passed.
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch, SOLR-11654.patch, SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517023#comment-16517023
]
Varun Thacker commented on SOLR-11654:
--
Wow! I was surprised when I saw 10 add methods already in SolrClient . How do
our users even use our APIs :/
Maybe food for another Jira : If we need 10+ add methods and the fact that the
UpdateRequest is public meaning they could do this by stiching it together
themselves anyways doesn't sound right at all. Happy to discuss and contribute
in the right Jira if others feel the same way.
But for the scope of this Jira maybe it's not the worst thing adding a couple
of more methods as well? I guess if we just use UpdateRequest we wouldn't need
this but that depends on Gus and you on how much ease does this add to docs vs
directly using UpdateRequest ?
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch, SOLR-11654.patch, SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515888#comment-16515888
]
David Smiley commented on SOLR-11654:
-
Thanks Gus! This is looking great; I really like your
TrackingUpdateProcessorFactory in particular. Very useful. I have some local
super minor tweaks but I want to mention a few things:
* removed SolrTestCaseJ4.adocs which you added but wasn't used. Besides, I
don't like these methods on SolrTestCaseJ4 that return XML strings.
* removed SolrClient.add(collection,SolrInputDocument...) because I don't think
an API should have both array/varargs form and Collection form (and besides,
Arrays.asList() is not that verbose).
* I did leave just one method you added on SolrClient...
{code:java}
/**
* Adds a collection of documents, specifying max time before they become
committed and additional
* parameters for the update.
*
* @param collection the Solr collection to add documents to
* @param docs the collection of documents
* @param commitWithinMs max time (in ms) before a commit will happen
* @param params any parameters to be sent with this update request.
*
* @return an {@link org.apache.solr.client.solrj.response.UpdateResponse}
from the server
*
* @throws IOException if there is a communication error with the
server
* @throws SolrServerException if there is an error on the server
*
* @since Solr 5.1
*/
public UpdateResponse add(String collection, Collection
docs, int commitWithinMs, ModifiableSolrParams params) throws
SolrServerException, IOException {
UpdateRequest req = new UpdateRequest();
if (params != null) {
req.setParams(params);
}
req.add(docs);
req.setCommitWithin(commitWithinMs);
return req.process(this, collection);
}
{code}
Since I think peer review is especially important on the public API of
SolrClient, I'd like input from [~gerlowskija] or [~varunthacker] that the
above new method is okay. What the above method saves is manual creation of an
UpdateRequest as used above. I know it's not often to use params on an update.
There is one thing I think that should be changed from Gus's proposal -- I
think setParams should copy the input to a new ModifiableSolrParams, and I
think the declared param type of the method here should simply be SolrParams.
I think it's trappy that UpdateRequest takes a mutable SolrParams and then
modifies it.
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch, SOLR-11654.patch, SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511785#comment-16511785
]
Gus Heck commented on SOLR-11654:
-
It looks like the lack of a slice name might cause some problems if a retry
actually happens (seems to be needed to look up the leader). Looking around
DistributedUpdateProcessor is the only class building one of these in
production code besides us and it simply passes in the name of the slice (DURP
line 1509) so I added that in this latest patch
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch, SOLR-11654.patch, SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511776#comment-16511776
]
Gus Heck commented on SOLR-11654:
-
New patch with TrackingUpdateProcessorFactory class (for test use only) and
unit test utilizing it to test that docs are now routed to the shard leader.
Just realized I didn't yet investigate the null for RetryNode... I'll look at
that and add another patch if it seems to worry me
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch, SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502540#comment-16502540
]
David Smiley commented on SOLR-11654:
-
RE your change to TimeRoutedAliasUpdateProcessorTest ... I'd rather the test
not bother doing a cleanup of the conifgsets added. Since the current test
asserts the number of configSets, it can be modified to be less stringent or
not care; it's not important.
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502530#comment-16502530
]
David Smiley commented on SOLR-11654:
-
Looking at your patch. getLeaderNode I noticed doesn't pass along "slice" to
the constructor of RetryNode (last param). It probably should do so for
routeDocToSlice, but lookupShardLeadersOfCollections I guess null (what's
happening today). I'm not completely clear on the implications yet.
Hey [~shalinmangar] or anyone who knows SolrCloud well, do you know if there
are tests for ensuring the doc routing goes right to the correct shard leader
instead of being arbitrary? I suppose it's fundamental to DURP but in
CloudSolrClient AFAIK this is an optimization so perhaps there's a test for
that? We're not sure how to test this since it's an internal routing thing.
See Gus's most recent comment above. I directed Gus at
TrackingShardHandlerFactory but didn't realize it's a search-only tracker not
for updates (ouch).
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11654) TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard
[
https://issues.apache.org/jira/browse/SOLR-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496119#comment-16496119
]
Gus Heck commented on SOLR-11654:
-
This has been dangling a bit too long, so attaching what I have so far. I feel
pretty good that I've got code in place that selects an appropriate shard, and
things seem to not break, but trying to write a test for this code has been a
mess... I've been lost in the weeds chasing the notion that
TrackingShardHandlerFactory could be used to track which shards requests were
sent to, but based on everything I can find updates never touch shardHandlers
so that's a dead end. The in-code documentation on ShardRequest and
ShardHandler, and ShardHandlerFactory and pretty much everything having
anything to do with shard requests is virtually non existent. HttpShardHandler
has the seemingly odd property that many of the request making methods are on
the factory, not the object produced by the factory... In any case I think
something analogous to TrackingShardHandlerFactory for tracking updates is
required to properly test this, probably a custom URP, though it needs to be
configured after DistributedUpdateProcessor to ensure it isn't skipped on the
sub-requests.
> TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to
> the ideal shard
>
>
> Key: SOLR-11654
> URL: https://issues.apache.org/jira/browse/SOLR-11654
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Priority: Major
> Attachments: SOLR-11654.patch
>
>
> {{TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection}} looks up the
> Shard/Slice to talk to for the given collection. It currently picks the
> first active Shard/Slice but it has a TODO to route to the ideal one based on
> the router configuration of the target collection. There is similar code in
> CloudSolrClient & DistributedUpdateProcessor that should probably be
> refactored/standardized so that we don't have to repeat this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
