[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651172#comment-13651172 ] Mark Miller commented on SOLR-3755: --- bq. I'll revert the change to the preRegister method signature and find another way. I'm trying to look at this now. I'm not sure how to go about solving in an 'easy' way. Currently, you have to start buffering those updates *before* publishing, but I want it to so that you publish as DOWN *before* creating the SolrCore - but you need the SolrCore to start buffering. I don't see the 'easy' fix unfortunately. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Assignee: Shalin Shekhar Mangar Fix For: 4.3, 5.0 Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651192#comment-13651192 ] Mark Miller commented on SOLR-3755: --- bq. I don't see the 'easy' fix unfortunately. Okay, I think I found it - doing this stuff in the bottom of the SolrCore constructor rather than preRegister seems to work so far. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Assignee: Shalin Shekhar Mangar Fix For: 4.3, 5.0 Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651267#comment-13651267 ] Shalin Shekhar Mangar commented on SOLR-3755: - Yeah, that'll work. We have an issue open to track this: SOLR-4745 shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Assignee: Shalin Shekhar Mangar Fix For: 4.3, 5.0 Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637505#comment-13637505 ] Shalin Shekhar Mangar commented on SOLR-3755: - I haven't seen the test failure due to extra document after increasing read timeout values in the test. Now that 4.3 is about to release with this feature, I'm going to mark this issue as resolved. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Assignee: Shalin Shekhar Mangar Fix For: 4.3, 5.0 Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13631384#comment-13631384
]
Shalin Shekhar Mangar commented on SOLR-3755:
-
bq. Anshum suggested over chat that we should think about combining
ShardSplitTest and ChaosMonkeyShardSplit tests into one to avoid code
duplication. I'll try to see if we can do that.
I've changed ChaosMonkeyShardSplitTest to extend ShardSplitTest so that we can
share most of the code. The ChaosMonkey test is not completely correct and I
intend to improve it.
bq. The original change around this made preRegister start taking a core rather
than a core descriptor. I'd like to work that out so it doesn't need to be the
case.
I'll revert the change to the preRegister method signature and find another way.
I've found two kinds of test failures of (ChaosMonkey)ShardSplitTest.
The first is because of the following sequence of events:
# A doc addition fails (because of the kill leader jetty command), client
throws an exception and therefore the docCount variable is not incremented
inside the index thread.
# However, the doc addition is recorded in the update logs (of the proxy node?)
and replayed on the new leader so in reality, the doc does get added to the
shard
# Split happens and we assert on docCounts being equal in the server which
fails because the server has the document that we have not counted.
This happens mostly with Lucene-Solr-Tests-4.x-Java6 builds. The bug is in the
tests and not in the split code. Following is the stack trace:
{code}
[junit4:junit4] 1 ERROR - 2013-04-14 14:24:27.697;
org.apache.solr.cloud.ChaosMonkeyShardSplitTest$1; Exception while adding doc
[junit4:junit4] 1 org.apache.solr.client.solrj.SolrServerException: No live
SolrServers available to handle this
request:[http://127.0.0.1:34203/h/y/collection1,
http://127.0.0.1:34304/h/y/collection1, http://127.0.0.1:34311/h/y/collection1,
http://127.0.0.1:34270/h/y/collection1]
[junit4:junit4] 1at
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:333)
[junit4:junit4] 1at
org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:306)
[junit4:junit4] 1at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
[junit4:junit4] 1at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.indexDoc(AbstractFullDistribZkTestBase.java:561)
[junit4:junit4] 1at
org.apache.solr.cloud.ChaosMonkeyShardSplitTest.indexr(ChaosMonkeyShardSplitTest.java:434)
[junit4:junit4] 1at
org.apache.solr.cloud.ChaosMonkeyShardSplitTest$1.run(ChaosMonkeyShardSplitTest.java:158)
[junit4:junit4] 1 Caused by: org.apache.solr.common.SolrException: Server at
http://127.0.0.1:34311/h/y/collection1 returned non ok status:503,
message:Service Unavailable
[junit4:junit4] 1at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
[junit4:junit4] 1at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] 1at
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:264)
[junit4:junit4] 1... 5 more
{code}
Perhaps we should check the exception message and continue to count such a
document?
The second kind of test failures are where a document add fails due to version
conflict. This exception is always seen just after the updateshardstate is
called to switch the shard states. Following is the relevant log:
{code}
[junit4:junit4] 1 INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state invoked
for collection: collection1
[junit4:junit4] 1 INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state shard1
to inactive
[junit4:junit4] 1 INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state shard1_0
to active
[junit4:junit4] 1 INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state shard1_1
to active
[junit4:junit4] 1 INFO - 2013-04-14 19:05:26.873;
org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=
path=/update params={wt=javabinversion=2} {add=[169 (1432319507166134272)]} 0 2
[junit4:junit4] 1 INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1 INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1 INFO -
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13631405#comment-13631405 ] Mark Miller commented on SOLR-3755: --- bq. I'll revert the change to the preRegister method signature and find another way. I'm happy to help on this - it might be easier to just create a new issue rather than reverting, and work on getting it nicer from there, up to you though. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Assignee: Shalin Shekhar Mangar Fix For: 4.3, 5.0 Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13631490#comment-13631490 ] Anshum Gupta commented on SOLR-3755: bq. This happens mostly with Lucene-Solr-Tests-4.x-Java6 builds. Is this true for all the exceptions or just the one that follows this line? I wasn't able to reproduce this on my system running Java7. Also, are these consistent failures? shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Assignee: Shalin Shekhar Mangar Fix For: 4.3, 5.0 Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13631511#comment-13631511 ] Shalin Shekhar Mangar commented on SOLR-3755: - bq. Is this true for all the exceptions or just the one that follows this line? I wasn't able to reproduce this on my system running Java7. The error with the failing add doc happens with Java6 -- haven't seen it with any other version. I've seen the version conflict exception on java7 and java8. bq. Also, are these consistent failures? Yes but only on jenkins! I've had ec2 boxes running these tests all night and I haven't seen a failure in over 500 runs. These failures are very environment and timing dependent. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Assignee: Shalin Shekhar Mangar Fix For: 4.3, 5.0 Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13629010#comment-13629010 ] Mark Miller commented on SOLR-3755: --- bq. Set update log to buffering mode before it is published (fixes bug with extra doc count on sub-shard) Regarding those changes - I'd really like to find another way to do that. The original change around this made preRegister start taking a core rather than a core descriptor. I'd like to work that out so it doesn't need to be the case. That is where the core will find out some of it's properties (shard id, core node name, perhaps more in the future). It would be nice if the core init code had access to this information - so it would be nice if we could call preRegister (or some refactored version) before actually creating the SolrCore. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Assignee: Shalin Shekhar Mangar Fix For: 4.3, 5.0 Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626753#comment-13626753 ] Shalin Shekhar Mangar commented on SOLR-3755: - Committed three changes: # Set update log to buffering mode before it is published (fixes bug with extra doc count on sub-shard) # Use deleteIndex=true while unloading sub-shard cores (if a sub-shard in construction state already exists at the start of the splitshard operation) # Made ChaosMonkeyShardSplitTest consistent with ShardSplitTest -- Use correct router and replica count, assert sub-shards are active, parent shards are inactive etc Anshum suggested over chat that we should think about combining ShardSplitTest and ChaosMonkeyShardSplit tests into one to avoid code duplication. I'll try to see if we can do that. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Assignee: Shalin Shekhar Mangar Fix For: 4.3, 5.0 Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625013#comment-13625013 ] Anshum Gupta commented on SOLR-3755: All of the above mentioned issues (and more) are now fixed. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623537#comment-13623537 ] Anshum Gupta commented on SOLR-3755: I've run into a few more issues while trying to improve the error handling/reporting. 1. Splitting an existing sub-shard gets stuck up. The new sub-sub shards stay in construction state forever. 2. The replicas are generally always created on the same node. (Debugging/fixing that) shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623541#comment-13623541 ] Shalin Shekhar Mangar commented on SOLR-3755: - bq. The sub shard cores are created while the sub shard is in construction state therefore their cloud descriptor keeps construction as the shard state. If the sub shard leader goes down after the shard state has been changed to active, it sets the shard state to construction once again while publishing itself as down. I've fixed it in the git branch. Although I don't like the fix very much. In the git branch, I'm using the shardState and shardRange fields in CloudDescriptor for a one-time usage. They are set to null once the new sub shard core is registered (and the new sub shard is created in zk). Maybe shardState and shardRange should be a core property instead? shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622208#comment-13622208 ] Shalin Shekhar Mangar commented on SOLR-3755: - I ran into another bug. Adding mutable state in cloud descriptor (like shard state and range) is a bad idea. The sub shard cores are created while the sub shard is in construction state therefore their cloud descriptor keeps construction as the shard state. If the sub shard leader goes down after the shard state has been changed to active, it sets the shard state to construction once again while publishing itself as down. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621060#comment-13621060 ] Shalin Shekhar Mangar commented on SOLR-3755: - I'd like to commit the patch to 4x and trunk soon. We can then work on improving the features and tests via the regular route. If there are no objections, I'll commit it tomorrow. bq. It's very common for these types of tests to be sensitive to the exact env (hardware, OS, etc). A lot of times it's some timing issue. Yeah, I'm still trying to reproduce the issue. I'll try to find a solution before I commit. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620192#comment-13620192
]
Yonik Seeley commented on SOLR-3755:
Nice that this is on a git branch - no stale patches, and you can see the full
history!
Does anyone know an easy way to generate a diff?
I did the following:
{code}
git clone https://github.com/shalinmangar/lucene-solr.git lusolr_shardsplitting
cd lusolr_shardsplitting
git remote add upstream git://git.apache.org/lucene-solr.git
git diff remotes/upstream/trunk remotes/origin/trunk
{code}
But this does a diff with the current state of the trunk vs the branch. Any
tips from the git wizards out there?
shard splitting
---
Key: SOLR-3755
URL: https://issues.apache.org/jira/browse/SOLR-3755
Project: Solr
Issue Type: New Feature
Components: SolrCloud
Reporter: Yonik Seeley
Attachments: SOLR-3755-combined.patch,
SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch,
SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch,
SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch,
SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch
We can currently easily add replicas to handle increases in query volume, but
we should also add a way to add additional shards dynamically by splitting
existing shards.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620207#comment-13620207 ] Mark Miller commented on SOLR-3755: --- AFAIK it's somewhat annoying - usually it involves doing a squash commit on a tmp branch and diffing with that if you want it nicely in one file/chunk. Otherwise git format-patch can go back n commits and make a diff for each one and you'd have to stitch them together. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620208#comment-13620208 ] Mark Miller commented on SOLR-3755: --- bq. Was trying to look into it but strangely, I haven't run into it over 15 consecutive runs. It's very common for these types of tests to be sensitive to the exact env (hardware, OS, etc). A lot of times it's some timing issue. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620215#comment-13620215 ] Anshum Gupta commented on SOLR-3755: You'd need to do a git merge and then compare it with the current branch. git fetch upstream git merge upstream/trunk git diff --no-prefix upstream/trunk This should show the diff. For now, I've just merged the current state of the trunk with this branch. Getting the diff now should be straight forward. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620224#comment-13620224 ] Anshum Gupta commented on SOLR-3755: Mark, you're right, it seems like a timing issue. I don't think even Shalin has been able to to recreate it too often under the same environment. Not even with the same seed. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618464#comment-13618464 ] Anshum Gupta commented on SOLR-3755: Was trying to look into it but strangely, I haven't run into it over 15 consecutive runs. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618065#comment-13618065 ] Shalin Shekhar Mangar commented on SOLR-3755: - Okay, the test still fails sometimes. I'm looking into it. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13615249#comment-13615249 ] Mark Miller commented on SOLR-3755: --- Hope to take a look at what you guys have been up to again soon. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13614251#comment-13614251 ] Anshum Gupta commented on SOLR-3755: There'are more changes on the branch, including a ChaosMonkey test for the feature. Any feedback on the design/strategy would be good. Also, I'm working on adding some more documentation on the general strategy somewhere in the code/package and improving the javadoc for the same as well. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13605354#comment-13605354 ] Shalin Shekhar Mangar commented on SOLR-3755: - Btw, the github fork is at https://github.com/shalinmangar/lucene-solr shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601090#comment-13601090 ] Mark Miller commented on SOLR-3755: --- bq. We wouldn't want any shard assignment/replica addition normally to go to a non-active Slice. I think changing the AssignShard to use getAllSlices may do what we're trying to avoid here. No, I think you are confusing slice/shard state. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601143#comment-13601143 ] Mark Miller commented on SOLR-3755: --- Anshum caught me up in chat - I am actually the one that is confused - because slice state stuff has already been committed. I thought I was looking at pre shard splitting trunk code. The real problem here is how the slice state is being handled in relation to clusterstate.json updates - you can close inactive slices from the clusterstate which will cause havoc. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601175#comment-13601175 ] Mark Miller commented on SOLR-3755: --- So to summarize - to fix this current problem, I think we want to rework the current slice state stuff in trunk - I left open SOLR-4566 for the moment. I think the cleanest thing for this API, and what will help keep the current issue from reoccuring, is if we change getSlices and getSliceMap to return all slices always. Then we add getActiveSlices and getActiveSliceMap, and appropriate calls are changed to that. Then there are likely to be less surprises when we try and copy/update the clusterstate. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601184#comment-13601184 ] Anshum Gupta commented on SOLR-3755: Thanks for the suggestions on that one Mark. I'll put up a patch soon for SOLR-4566 on the lines of what we discussed and what you've mentioned above. Though again, as we're not really using the states anywhere but in the patch for ShardSplitting, it should have no impact. However, just as a note, any future use of getSlices would mean handing inactive slices (or calling getActiveSlices) so the behaviour would change a bit (as we start using more of Slice states). shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601422#comment-13601422 ] Mark Miller commented on SOLR-3755: --- SOLR-4568 is another issue I found while working on this - I'll pull the fix from my patch to SOLR-4568. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601432#comment-13601432 ] Mark Miller commented on SOLR-3755: --- SOLR-4569 is another small improvement issue I'll pull from my patch here. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601439#comment-13601439 ] Mark Miller commented on SOLR-3755: --- SOLR-4570 filed as well - another issue with a solution in my patch. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600489#comment-13600489 ] Mark Miller commented on SOLR-3755: --- I think collection might be a better param name than name for the shard split api shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600774#comment-13600774 ] Mark Miller commented on SOLR-3755: --- I filed SOLR-4566 for the main issue. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600853#comment-13600853 ] Anshum Gupta commented on SOLR-3755: We wouldn't want any shard assignment/replica addition normally to go to a non-active Slice. I think changing the AssignShard to use getAllSlices may do what we're trying to avoid here. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596385#comment-13596385 ] Anshum Gupta commented on SOLR-3755: I'd like to suggest supporting only a single shard through this API. It may be called multiple times for more than one shards. In the future however, we may want to have a split API call which splits all existing shards, but that could be a different thing (if required). shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596853#comment-13596853 ] Shalin Shekhar Mangar commented on SOLR-3755: - bq. I'd like to suggest supporting only a single shard through this API. +1 shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13591412#comment-13591412 ] Shalin Shekhar Mangar commented on SOLR-3755: - bq. How do we know what collection? I assume there will be a collection parameter? Yes, a collection param will also be present. bq. shard.keys is currently used in routing request (and the values are often not shard names), so we probably shouldn't overload it here. After all, it may make sense in the future to be able to use shard.keys to specify which shard you want to split! Yes! That is exactly the thinking behind shard.keys here. It is not being overloaded but used to indicate which shard to split by specifying the key which resolves to a shard name. bq. Related: SOLR-4503 - we now have the capability to use restlet, and should consider doing so for new APIs like this. I'm not familiar with restlet. I'll take a look at it. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13588809#comment-13588809
]
Yonik Seeley commented on SOLR-3755:
{quote}
The collections api may be invoked as follows:
http://host:port/solr/admin/collections?action=SPLITshard=shard_1shard=shard_2
{quote}
Ok, I assume this is for splitting more than one shard (i.e. both shard_1 and
shard_2 in this example will be split?)
How do we know what collection? I assume there will be a collection
parameter?
Related: SOLR-4503 - we now have the capability to use restlet, and should
consider doing so for new APIs like this.
{quote}
Sometimes, shard names are automatically assigned by SolrCloud and it may be
more convenient for users to specify shards by shard keys instead of shard
names e.g.
http://host:port/solr/admin/collections?action=SPLITshard.keys=shardKey1,shardKey2;
{quote}
shard.keys is currently used in routing request (and the values are often not
shard names), so we probably shouldn't overload it here. After all, it may
make sense in the future to be able to use shard.keys to specify which shard
you want to split!
shard splitting
---
Key: SOLR-3755
URL: https://issues.apache.org/jira/browse/SOLR-3755
Project: Solr
Issue Type: New Feature
Components: SolrCloud
Reporter: Yonik Seeley
Attachments: SOLR-3755-CoreAdmin.patch, SOLR-3755.patch,
SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch
We can currently easily add replicas to handle increases in query volume, but
we should also add a way to add additional shards dynamically by splitting
existing shards.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585668#comment-13585668 ] Anshum Gupta commented on SOLR-3755: Any suggestions/feedback on the earlier comment about the Collections API would be good. Here's what the collections API call(s) would look like: The collections api may be invoked as follows: http://host:port/solr/admin/collections?action=SPLITshard=shard_1shard=shard_2 Sometimes, shard names are automatically assigned by SolrCloud and it may be more convenient for users to specify shards by shard keys instead of shard names e.g. http://host:port/solr/admin/collections?action=SPLITshard.keys=shardKey1,shardKey2; shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13582829#comment-13582829 ] Commit Tag Bot commented on SOLR-3755: -- [branch_4x commit] Shalin Shekhar Mangar http://svn.apache.org/viewvc?view=revisionrevision=1447517 SOLR-3755: Do not create core on split action, use 'targetCore' param instead shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13582845#comment-13582845 ] Commit Tag Bot commented on SOLR-3755: -- [trunk commit] Shalin Shekhar Mangar http://svn.apache.org/viewvc?view=revisionrevision=1447516 SOLR-3755: Do not create core on split action, use 'targetCore' param instead shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575205#comment-13575205 ] Commit Tag Bot commented on SOLR-3755: -- [trunk commit] Shalin Shekhar Mangar http://svn.apache.org/viewvc?view=revisionrevision=1444397 SOLR-3755: Test for SolrIndexSplitter shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575207#comment-13575207 ] Commit Tag Bot commented on SOLR-3755: -- [branch_4x commit] Shalin Shekhar Mangar http://svn.apache.org/viewvc?view=revisionrevision=1444398 SOLR-3755: Test for SolrIndexSplitter shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13564201#comment-13564201 ] Shalin Shekhar Mangar commented on SOLR-3755: - We need to introduce shard states into the design. SolrCloud shards are always “active” i.e. no state information is associated with shards presently. I'm planning to add two new states viz. “Construction” and “Recovery” besides the default “Active” state. A shard in “Construction” state has the following properties: * Shard nodes receive no queries * Shard nodes receive no updates except those forwarded by leaders * Overseer does not allocate nodes to such a shard automatically * Leader election is disabled for such a shard * Shard nodes automatically go into recovering state (buffering update mode) A shard in “Recovering” phase is similar to a shard in Construction state except that shard nodes automatically go into recovering state (“APPLYING_BUFFERED” mode and once completed into active state). We could merge the two states together if necessary once we start implementing stuff. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13564207#comment-13564207 ] Shalin Shekhar Mangar commented on SOLR-3755: - Thinking more about the general strategy that Yonik's devised for this feature, here is a rough draft of how it may go. A split operation is triggered via collections API * Overseer Collection Processor (CP) creates new sub-shard in ZK in Construction state s.t. first node to join the shard becomes the leader and thereafter leaders are not elected automatically. Replicas are not automatically created in the “Construction” state * CP creates new cores on leader using the core/cloud descriptors of parent core. ** Such cores are automatically designated as leader for respective sub-shard ** These new cores join sub-shards in buffering-update mode and keep themselves in that mode until the shard changes its state. ** DUPF on parent forwards only relevant updates to sub-shard core. * CP calls CoreAdmin split on leader of shard ** A hard commit is called and index is split and written into correct cores * CP puts shard into Recovery state ** Sub-shard cores go into apply-buffered-updates mode. ** CP puts a watch on sub-shard cores status * Once sub-shard core status becomes active, Overseer creates replicas and watches their state * Once a number of replicas (ceil(numReplicas/2) is enough?) have recovered for all sub-shards, atomically set sub-shard active and parent shard in-active. Suggestions/comments welcome. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553117#comment-13553117 ] Dmitry Kan commented on SOLR-3755: -- Somewhat related: control naming of shards. This could be applicable for both hashing based collections and custom sharding based collections. shardNames=myshard1,myshard2,myshard3? Would this suit to logical (e.g. date based) sharding as well? Do you plan to support such a sharding type in the current shard splitting implementation? Not sure, if this helps: we have implemented our own custom date based sharding (splitting and routing) for solr 3.x and found it to be the most logical way of sharding our data (both from the load balancing and use case point of view). The routing implementation is done via loading a custom shards config file that contains mapping of date ranges to shards. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552307#comment-13552307 ] Yonik Seeley commented on SOLR-3755: OK, after chatting w/ Mark a bit, this seems to be his use-case: A pre-configured cluster w/ no information yet in ZK. Currently implemented via: - configuring the collection shard of each core in solr.xml - bring all of those cores up - start indexing (and in 4.0 style, the correct shard is picked via hashing and splitting up the range according to the currently known shards) This 4.0 behavior could be replicated via a lazyHash router that simply splits the hash range over currently know shards at the time of every request. This is fragile and error prone for many users of course, so it would not be a default. Additionally, we would need code to explicitly specify the router for a collection (assuming the collection had not already been created). Somewhat related: control naming of shards. This could be applicable for both hashing based collections and custom sharding based collections. shardNames=myshard1,myshard2,myshard3? shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13550477#comment-13550477 ] Mark Miller commented on SOLR-3755: --- This has a back compat break that we should address somehow or at least mention in changes - previously you could specify explicit shard ids and still get distributed updates - now if you do that, you won't get distrib updates as shards won't be assigned ranges. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474931#comment-13474931 ] Radim Kolar commented on SOLR-3755: --- Useful theory about rehashing http://en.wikipedia.org/wiki/Consistent_hashing shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13451408#comment-13451408
]
Yonik Seeley commented on SOLR-3755:
bq. Make Slice subclass ZkNodeProps
After a lot of code modification, I've realized that ZkNodeProps was probably
supposed to be the same as Replica. I was fooled by thinking it was generic
properties in ZK on any type of node (slice, replica, or whatever), and that
was reinforced by it's use in other context as generic properties (messages in
the overseer queue use ZkNodeProps as general properties - Overseer.java:125)
Given that Node also has another meaning (A Node is a CoreContainer/JVM that
can contain multiple cores), I'm leaning toward renaming ZkNodeProps to
Replica, and making a truly generic class ZkProps that Replica, Slice, etc, can
subclass from.
Here's an example of the types of code changes I've been making to hopefully
make things more readable:
{code}
-for (Map.EntryString,Slice entry : slices.entrySet()) {
- Slice slice = entry.getValue();
- MapString,ZkNodeProps shards = slice.getShards();
- SetMap.EntryString,ZkNodeProps shardEntries = shards.entrySet();
- for (Map.EntryString,ZkNodeProps shardEntry : shardEntries) {
-final ZkNodeProps node = shardEntry.getValue();
-if
(clusterState.liveNodesContain(node.get(ZkStateReader.NODE_NAME_PROP))) {
- return new ZkCoreNodeProps(node).getCoreUrl();
+for (Slice slice : slices.values()) {
+ for (Replica replica : slice.getReplicas()) {
+if
(clusterState.liveNodesContain(replica.get(ZkStateReader.NODE_NAME_PROP))) {
+ return new ZkCoreNodeProps(replica).getCoreUrl();
{code}
Unfortunately, when I got all done, ZK related tests were no longer passing.
I'm going to try and make another attempt and see if I can make more
incremental changes (so that I can run tests periodically).
shard splitting
---
Key: SOLR-3755
URL: https://issues.apache.org/jira/browse/SOLR-3755
Project: Solr
Issue Type: New Feature
Components: SolrCloud
Reporter: Yonik Seeley
Attachments: SOLR-3755.patch, SOLR-3755.patch
We can currently easily add replicas to handle increases in query volume, but
we should also add a way to add additional shards dynamically by splitting
existing shards.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450033#comment-13450033 ] Yonik Seeley commented on SOLR-3755: I've run into a few impedance mismatch issues implementing the JSON above. Internally we seem to use ZkNodeProps which accepts MapString,String... but a JSON Map is better represented as a MapString,Object. I think I'll try going in the following direction: - Make ZkNodeProps that accepts MapString,Object as properties, and can thus represent integers and more complex types. This will be just like a Map, but add some convenience methods - Make Slice subclass ZkNodeProps - Make a new Replica class (instead of just representing it as a generic ZkNodeProps) In general, to construct these classes from JSON, it seems like we should just pass the MapString,Object generated from the JSON parser and then the constructor can pull out key elements and construct sub-elements. Thoughts? shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447738#comment-13447738 ] Mark Miller commented on SOLR-3755: --- Yeah, I like the first option as well. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447851#comment-13447851 ] Grant Ingersoll commented on SOLR-3755: --- +1 on the first option. I _think_ it's considered good JSON practice to have key names not contain state, but I can't remember where I saw that. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447347#comment-13447347
]
Yonik Seeley commented on SOLR-3755:
Just committed some more progress.
http://svn.apache.org/viewvc?rev=1380287view=rev
I started up a ZK cluster with one shard, one node.
curl http://localhost:8983/solr/admin/cores?core=collection1action=SPLIT;
The cloud state after looks like
{code}
{collection1:{
shard1:{Rogue:8983_solr_collection1:{
shard:shard1,
roles:null,
leader:true,
state:active,
core:collection1,
collection:collection1,
node_name:Rogue:8983_solr,
base_url:http://Rogue:8983/solr}},
shard1_0:{Rogue:8983_solr_collection1_0:{
shard:shard1_0,
leader:true,
roles:null,
state:active,
core:collection1_0,
collection:collection1,
node_name:Rogue:8983_solr,
base_url:http://Rogue:8983/solr}},
shard1_1:{Rogue:8983_solr_collection1_1:{
shard:shard1_1,
roles:null,
leader:true,
state:active,
core:collection1_1,
collection:collection1,
node_name:Rogue:8983_solr,
base_url:http://Rogue:8983/solr
{code}
The original core had 32 docs. After I did a manual commit on both of the new
cores, the first showed 14 docs and the second 18.
shard splitting
---
Key: SOLR-3755
URL: https://issues.apache.org/jira/browse/SOLR-3755
Project: Solr
Issue Type: New Feature
Components: SolrCloud
Reporter: Yonik Seeley
Attachments: SOLR-3755.patch, SOLR-3755.patch
We can currently easily add replicas to handle increases in query volume, but
we should also add a way to add additional shards dynamically by splitting
existing shards.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447400#comment-13447400
]
Yonik Seeley commented on SOLR-3755:
It seems like we need logical shard parameters (i.e. Slice class), but we don't
currently have a place for them.
These parameters would include:
- collection (this is somewhat redundant, but belongs more on a slice than on
a replica)
- replication factor (i.e. in time based sharding, one may want more replicas
of recent shards to handle greater query throughput)
- hash range(s) covered by the slice
- maybe a pointer to the leader, rather than having to search through the
nodes?
You can see the previous structure of cloudstate from my previous message.
One fix is to introduce a nodes or replicas level to contain the nodes and
leave the other properties as top-level:
{code}
shard1: {
replication_factor : 3,
range : -3fff,
nodes : {
Rogue:8983_solr_collection1:{
state : active
}
}
}
{code}
Another way is to introduce a props to store properties:
{code}
shard1: {
props : {
replication_factor : 3,
range : -3fff
},
Rogue:8983_solr_collection1:{
state : active
}
}
{code}
The first option feels more natural to me - properties are directly under the
shard, and the nodes of a shard are simply another property.
shard splitting
---
Key: SOLR-3755
URL: https://issues.apache.org/jira/browse/SOLR-3755
Project: Solr
Issue Type: New Feature
Components: SolrCloud
Reporter: Yonik Seeley
Attachments: SOLR-3755.patch, SOLR-3755.patch
We can currently easily add replicas to handle increases in query volume, but
we should also add a way to add additional shards dynamically by splitting
existing shards.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13446054#comment-13446054 ] Yonik Seeley commented on SOLR-3755: So we need to have new cores up and running, and then install the new indexes in them. We could either do it like replication and use a new index directory (and use a property file to redirect to that latest index), or we could try and make sure that there is no open writer on the new core and then split directly into the normal core index directory. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445362#comment-13445362 ] Yonik Seeley commented on SOLR-3755: Since this doesn't change any existing func, I've committed what I have now to enable easier integration/modification by others. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445371#comment-13445371 ] Uwe Schindler commented on SOLR-3755: - Hi Yonik, looks nice, similar to oal.index.IndexSplitter in misc module just using a HashPartitioned LiveDocs. I am just confused, why does it not use FixedBitSet? The length is fixed and no (int) casts needed. +1 otherwise shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445376#comment-13445376
]
Uwe Schindler commented on SOLR-3755:
-
{code}
// TODO: will many deletes have been removed, or should we optimize?
{code}
The merged indexes will have no deletions at all, because it *merges* not
copies. IndexWriter.addIndexes(IndexReader...) does the same like a standard
Lucene merge, IndexWriter.addIndexes(Directory) just copies the segment files.
This is a plain stupid merge of a segment that has additional, overlaid
deletions.
shard splitting
---
Key: SOLR-3755
URL: https://issues.apache.org/jira/browse/SOLR-3755
Project: Solr
Issue Type: New Feature
Components: SolrCloud
Reporter: Yonik Seeley
Attachments: SOLR-3755.patch, SOLR-3755.patch
We can currently easily add replicas to handle increases in query volume, but
we should also add a way to add additional shards dynamically by splitting
existing shards.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445391#comment-13445391 ] Yonik Seeley commented on SOLR-3755: bq. I am just confused, why does it not use FixedBitSet? Habit... OpenBitSet is just the class I'm used to and my fingers automatically type. bq. The merged indexes will have no deletions at all, because it merges not copies. Cool, thanks for the clarification - I'll update the comment in my local copy. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440698#comment-13440698 ] Yonik Seeley commented on SOLR-3755: We need to associate hash ranges with shards and allow overlapping shards (i.e. 1-10, 1-5,6-10) General Strategy for splitting w/ no service interruptions: - Bring up 2 new cores on the same node, covering the new hash ranges - Both cores should go into recovery mode (i.e. leader should start forwarding updates) - leader does a hard commit and splits the index - Smaller indexes are installed on the new cores - Overseer should create new replicas for new shards - Mark old shard as “retired” – some mechanism to shut it down (after there is an acceptable amount of coverage of the new shards via replicas) Future: allow splitting even with “custom” shards shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
